Skip to content
BranchesTags
 
 

Latest commit

 

History

21 Commits

Folders and files

NameName
Last commit message
Last commit date
readme.md
readme.md
 
 

Repository files navigation

NAS-2026 | MergerFS and SnapRAID

guide-by-example

overview

A NAS that:

  • Allows mixing HDDs of various sizes.
  • Easy to add more storage at any time.
  • A parity drive that protects against disk failure.
  • Free and open source.

But:

  • It's more work than just installing TrueNAS or OMV.
  • Requires a solid knowledge of linux and the terminal.
  • SnapRAID isn't realtime.
    Can only recover data to the state of the last snapraid sync, which usually runs every 24h.
  • Bad for frequently changed data, or millions of small files.
    But that means ideal for media servers - movies, shows, music, photos, audiobooks,...

A notice
Not yet running this setup with actual data. This guide is a way to prepare and document for the deployment. Also AI - chatgpt and claude were used for scripts and sanity checks. But no blind trust, stuff was investigated, checked against documentation and thoroughly tested.

Chapters:

Preparation

linux-distros

Have a Linux installed.
I use Arch, installing it using archinstall, and have ansible playbooks to set it up how I like it.

Format and partition disks and mount them using fstab.

  • Create a new partition table on each disk.
    sudo parted /dev/sdb --script mklabel gpt
    sudo parted /dev/sdc --script mklabel gpt
    sudo parted /dev/sdd --script mklabel gpt
    sudo parted /dev/sde --script mklabel gpt

  • Partition disks.
    sudo parted /dev/sdb --script mkpart primary ext4 0% 100%
    sudo parted /dev/sdc --script mkpart primary ext4 0% 100%
    sudo parted /dev/sdd --script mkpart primary ext4 0% 100%
    sudo parted /dev/sde --script mkpart primary ext4 0% 100%

  • Format the partitions and label them.
    sudo mkfs.ext4 /dev/sdb1 -L disk_1
    sudo mkfs.ext4 /dev/sdc1 -L disk_2
    sudo mkfs.ext4 /dev/sdd1 -L disk_3
    sudo mkfs.ext4 /dev/sde1 -L parity

  • Create directories where drives will be mounted.
    sudo mkdir -p /mnt/disk_1 /mnt/disk_2 /mnt/disk_3 /mnt/parity

  • Edit fstab to mount drives at boot.
    fstab-pic To make things easier, here’s a script that generates the fstab entries.
    It expects ext4, it includes disk sizes and serial numbers, as well as a commented out mergerfs section that can be used later.

    disks-fstab-entries.sh
     
    #!/bin/bash
echo "# ======================================================"
echo "# MERGERFS and SNAPRAID DISKS"
echo "# ======================================================"
echo

# Loop over all partitions
lsblk -ln -o NAME,UUID,FSTYPE,SIZE,PKNAME,LABEL | while read name uuid fstype size pkname label; do

    [[ -n "$pkname" ]] || continue
    [[ -z "$uuid" || "$fstype" =~ ^(swap|zram|iso9660)$ ]] && continue

    serial=$(lsblk -dn -o SERIAL /dev/$pkname 2>/dev/null)
    label=${label:-no-label}
    echo "# $name ($size, label: $label, SN: ${serial:-unknown})"
    echo "UUID=$uuid /mnt/$label $fstype defaults,nofail,errors=remount-ro 0 2"
    echo
done

echo "# ======================================================"
echo "# MergerFS pool combining the disks"
echo "# ======================================================"
echo
echo "# /mnt/disk_* /mnt/pool fuse.mergerfs defaults,category.create=pfrd,func.getattr=newest,minfreespace=20G,fsname=mergerfs 0 0"
    • Create ~/disks-fstab-entries.sh and paste the code
    • Make the script executable: chmod +x disks-fstab-entries.sh
    • Run it: ~/disks-fstab-entries.sh
      It echoes stuff into the terminal for you to copy/paste into fstab.
    • Remove the lines you don't need, edit the mount points as needed, since they are taken from labels

To mount all disks defined in fstab - sudo mount -a or just reboot.
Check if all is fine with lsblk and lsblk -f and duf or dysk.

MergerFS

duf-pic

Merges disks of various sizes in to one combined pool.

  • When writing to that pool, files are spread across the disks
  • Works at the file level, as opposed to block level.
    Means files are plainly accessible, even if any of the disks would be pulled out and placed in another machine, the filesystem and the files are just there.
  • Think of it as a virtual mount point, not a typical filesystem.
  • There is NO redundancy, NO protection. MergerFS is all about just merging disk space.
    That's why we use SnapRAID later.
  • Very simple setup and configuration with a single line in /etc/fstab
  • Easy to add drives, even ones already containing data. No rebuild time.
  • Good performance, near native, with just fuse overhead, low cpu/ram usage.
  • Written in C and C++.

MergerFS setup

  • Install mergerfs.

  • Create a directory for the final mount point.
    sudo mkdir /mnt/pool

  • edit the fstab, add mergerfs mount definition.

    /mnt/disk_* /mnt/pool fuse.mergerfs defaults,category.create=pfrd,func.getattr=newest,minfreespace=20G,fsname=mergerfs 0 0
    The fstab definition options explained
    • /mnt/disk_* - is a wildcard that catches all desired disks mounts.
      Could also be explicitly written "/mnt/disk_1:/mnt/disk_2:/mnt/disk_3"
    • /mnt/pool - where to mount the final combined disk space
    • fuse.mergerfs - defines it as a mergerfs fuse union "filesystem"
    • defaults - bunch of default fstab mount options
    • category.create=pfrd - sets create policy to a random distribution, but the available free space affects the odds.
    • func.getattr=newest - if situation happens where there are two versions of the same file on different disks, this defines which version to pick - newest, it's recommended.
    • minfreespace=20G - disks that have less than 20GB will not be picked to store new files, saves the space for metadata and whatnot.
    • fsname=mergerfs - just defines what name to show in info/disk utilities
    • 0 0 - the first zero is some legacy backup dump, and second is about fsck at boot. Since mergerfs is union filesystem no fsck for it.

    Looking around the internet there are other mount options being used, but digging deeper shown that some were deprecated or were made in to defaults, or were not ideal to pick in the first place.
    Of note is that there is no caching in this setup.

  • Mount it, or reboot.
    sudo mount /mnt/pool

  • Take ownership of the new mount sudo chown -R $USER:$USER /mnt/pool

  • Done.

MergerFS policies details

A major aspect of mergerfs is picking the policy that decides how the data are spread across the drives. Reading the official documentation is a must do if planning to move away from the defaults... but to give some quick idea.

Two types

  • Path Preserving - The top directory anchors everything to a specific drive. Side effect is that if a disk gets full the new write in to that directory fails. You you can also use one of the less strict path preserving polices, the ones starting with msp... that will start putting stuff to another drive instead of failing.
  • Not Path Preserving - Data go to a disk with the most free space, or with the least used space, or the disk is picked randomly, or some variation. The directories structure plays no role.

The default policy is pfrd which picks disk randomly, but the available free space affects the odds.

Possibility to mix policies

MergerFS offers fine control and one can have different policy for directories and for files.

/mnt/disk_* /mnt/pool fuse.mergerfs defaults,func.create=msppfrd,func.mkdir=pfrd,func.getattr=newest,minfreespace=20G,fsname=mergerfs 0 0
  • func.create=msppfrd - path preserving for files
  • func.mkdir=pfrd - directories are spread randomly

Kinda like the idea of having stuff grouped by a directory, but not all the way to the top, just the parent directory. Meaning that files that are together in the same directory end up on the same disk, but directories themselves are spread around.
Feels like this makes the best use of the aspect of mergerfs where you get to keep the data on surviving drives after a failure.
But it kinda depends on the type of data you have.

  • Shows, music, audiobooks,... benefit from this approach as just having some of the files of a season, or audiobook or an album is the same as not having it at all.
  • Photos, documents,.. here this approach might be worse, as a disk failure loses you entire year of photos or an entire vacation because photos were in a directory titled "2019" and "Vacation Iceland". Which might be worse than having at least some of them because they were spread across drives.

Adding a new drive

Run through the Preparation chapter again, but only for the new disk.

  • Partition it, label it, create folder for its mount point /mnt/disk_X
  • Edit the fstab.
    • Add disk_X along the other disks so it mounts on boot.
    • Mergerfs uses a wild card /mnt/disk_*, so the new disk will be included in the pool automatically.
  • Run sudo mount -a or reboot.

For snapraid, if the size of the new drive is smaller or equal to the current parity drive... just add it to the snapraid config as an another data drive.

  • edit /etc/snapraid.conf
  • add disk_X in the data disks section
  • execute sudo snapraid sync or let it run automatically at the next schedule

If the drive is larger...
either accept you are not protecting it, or it's your new parity drive. which just means same stuff as before with partitioning, mounting, labeling, fstab,.. and then parity settings in /etc/snapraid.conf.
And after that it's full parity rebuild with sudo snapraid sync.

SnapRAID

snapraid

Provides parity protection against disk failure and bitrot.
The parity disk must be larger or equal in size to the largest of the data drives it protects, but that one parity drive can protect whatever number of data drives. Adding more parity drives guards against multiple disks failing at once.
Unlike regular raid that is ever going, snapraid needs a scheduled periodic sync, usually once every 24 hours. We can return only to the state when the last sync was run. Also if the data on the the other drives were modified/deleted after the sync, then that change can prevent full recovery as against those data parity is calculated. This behavior makes it good only for data that do not change often, like movies, shows, music, photos, audiobooks, videos,...
Of note is also that snapraid is saving parity information in to a single file - snapraid.parity
Written in C.

A note
Backups are better than parity protection typical for raid setup, or this snapraid stuff. Be sure you understand the difference, pros and cons, understand why people say "raid is not a backup"

SnapRAID setup

  • install snapraid.
  • Create /etc/snapraid.conf 
    # Parity file on a dedicated parity disk
parity /mnt/parity/snapraid.parity

# Data disks, the order matters!
disk disk_1 /mnt/disk_1
disk disk_2 /mnt/disk_2
disk disk_3 /mnt/disk_3

# Content file with metadata for recovery
content /var/lib/snapraid.content
content /mnt/parity/snapraid.content

# Excludes
exclude /lost+found/
exclude .Trash-*/
exclude .recycle/

# autosave every 50GB
autosave 50
  • sudo snapraid sync - the first initial sync

SnapRAID automation and notifications

ntfy

A script that runs daily that checks if disks are mounted, executes sync, scrub, and periodic smartctl disks healh checks and health tests and sends ntfy push notifications.

  • create a file /opt/snapraid-sync-and-maintenance.sh

    snapraid-sync-and-maintenance.sh
     
    #!/bin/bash
# v0.2
set -euo pipefail   # strict mode for bash

# -------------------------------------------------------------
# Configuration
# -------------------------------------------------------------
NTFY_TOPIC="https://ntfy.example.com/NAS" 
MERGERFS_MOUNT="/mnt/pool"

# -------------------------------------------------------------
# Check dependencies
# -------------------------------------------------------------
for cmd in snapraid smartctl curl; do
    command -v "$cmd" >/dev/null 2>&1 || {
        echo "❌ Required command not found: $cmd"
        exit 1
    }
done

# -------------------------------------------------------------
# A function that Sends ntfy notification and logs to journal
# -------------------------------------------------------------
notify() {
    local msg="$1"
    printf "%b" "$msg" | /usr/bin/curl -s -d @- "$NTFY_TOPIC" || true
    echo -e "$msg" # To also log to systemd journal
}

# Preventing concurent runs with a lockfile
LOCKFILE="/run/snapraid_maintenance.lock"
exec 9>"$LOCKFILE" || exit 1
flock -n 9 || {
    notify "❌ Another instance of SnapRAID maintenance script is already running."
    exit 0
}

# -------------------------------------------------------------
echo "=== SnapRAID maintenance script started $(date +"%F %T") ==="
SCRIPT_FAILED=false # variable to check if send success notification or not

# -------------------------------------------------------------
# Check if mergerfs mount point exists
# -------------------------------------------------------------
if ! mountpoint -q "$MERGERFS_MOUNT"; then
    notify "❌ mergerfs mount $MERGERFS_MOUNT not accessible!"
    exit 1
fi

# -------------------------------------------------------------
# Check if disks defined in SnapRAID config are mounted
# -------------------------------------------------------------
SNAPRAID_DATA_DISKS=$(grep '^disk' /etc/snapraid.conf | awk '{print $3}' || true)

if [[ -z "$SNAPRAID_DATA_DISKS" ]]; then
    notify "❌ No disks found in SnapRAID config!"
    exit 1
fi

for disk in $SNAPRAID_DATA_DISKS; do
    if ! mountpoint -q "$disk"; then
        notify "❌ SnapRAID disk not mounted: $disk!"
        exit 1
    fi
done

# -------------------------------------------------------------
# Do SnapRAID sync
# -------------------------------------------------------------
if ! sync_output=$(snapraid sync 2>&1); then
    notify "❌ SnapRAID sync failed:\n$sync_output"
    exit 1
fi

# -------------------------------------------------------------
# Do partial scrub, 15% of data once a month on the 1st
# -------------------------------------------------------------
if [[ $(date +%d) -eq 01 ]]; then
    if ! scrub_output=$(snapraid scrub -p 15 -o 180 2>&1); then
        notify "❌ SnapRAID scrub failed:\n$scrub_output"
        SCRIPT_FAILED=true
    fi
fi

# -------------------------------------------------------------
# SMART health check once a week on sunday
# -------------------------------------------------------------
if [[ $(date +%u) -eq 7 ]]; then
    SMART_DISKS=$(smartctl --scan | awk '{print $1}')
    for disk in $SMART_DISKS; do
        echo "$(date +"%F %T") Starting SMART check $disk"
        if ! smartctl -H "$disk" > /dev/null 2>&1; then
            MODEL=$(smartctl -i "$disk" | grep "Device Model" | cut -d: -f2 | xargs)
            notify "❌ SMART health check failed $disk ($MODEL)"
            SCRIPT_FAILED=true
        fi
    done
fi

# -------------------------------------------------------------
# SMART short test once a month on 10th
# -------------------------------------------------------------
if [[ $(date +%d) -eq 10 ]]; then
    SMART_DISKS=$(smartctl --scan | awk '{print $1}')
    for disk in $SMART_DISKS; do
        echo "$(date +"%F %T") Starting SMART short test $disk"
        smartctl -t short "$disk" > /dev/null 2>&1 || true
    done
    notify "🔍 Monthly SMART short tests started"
    
    sleep 600 # waiting 10 minutes for tests to finish

    ALL_PASSED=true
    for disk in $SMART_DISKS; do
        if ! smartctl -l selftest "$disk" | head -n 8 | grep -q "Completed without error"; then
            MODEL=$(smartctl -i "$disk" | grep "Device Model" | cut -d: -f2 | xargs)
            notify "❌ SMART short test failed: $disk ($MODEL)"
            ALL_PASSED=false
            SCRIPT_FAILED=true
        fi
    done

    if $ALL_PASSED; then
        notify "✅ All SMART short tests passed"
    fi
fi

# -------------------------------------------------------------
# SMART long test once every 4 months - april | august | december | 19th
# -------------------------------------------------------------
if [[ $(date +%d) -eq 19 ]] && (( 10#$(date +%m) % 4 == 0 )); then
    SMART_DISKS=$(smartctl --scan | awk '{print $1}')
    for disk in $SMART_DISKS; do
        echo "$(date +"%F %T") Starting SMART long test $disk"
        smartctl -t long "$disk" > /dev/null 2>&1 || true
    done
    notify "🔍 Triannual SMART long tests started, results in 2 days"
fi

# Results check 2 days later
if [[ $(date +%d) -eq 21 ]] && (( 10#$(date +%m) % 4 == 0 )); then
    SMART_DISKS=$(smartctl --scan | awk '{print $1}')
    ALL_PASSED=true
    for disk in $SMART_DISKS; do
        if ! smartctl -l selftest "$disk" | head -n 8 | grep -q "Completed without error"; then
            MODEL=$(smartctl -i "$disk" | grep "Device Model" | cut -d: -f2 | xargs)
            notify "❌ SMART long test failed: $disk ($MODEL)"
            ALL_PASSED=false
            SCRIPT_FAILED=true
        fi
    done

    if $ALL_PASSED; then
        notify "✅ All SMART long tests passed"
    fi
fi

# -------------------------------------------------------------
# Send success ntfy notification
# -------------------------------------------------------------
if ! $SCRIPT_FAILED; then
    notify "✅ SnapRAID sync and maintenance completed successfully $(date +"%F %T")"
fi

  • Make it executable sudo chmod +x /opt/snapraid-sync-and-maintenance.sh

  • create a systemd unit and a timer files.

    /etc/systemd/system/snapraid-sync-and-maintenance.service

    [Unit]
Description=SnapRAID sync, scrub, smartctl check, smartctl tests

[Service]
Type=oneshot
ExecStart=/opt/snapraid-sync-and-maintenance.sh

    /etc/systemd/system/snapraid-sync-and-maintenance.timer

    [Unit]
Description=SnapRAID sync, scrub, smartctl check, smartctl tests

[Timer]
OnCalendar=*-*-* 00:19:00
Persistent=true

[Install]
WantedBy=timers.target

  • enable the timer
    sudo systemctl enable --now snapraid-sync-and-maintenance.timer

SnapRAID recovery procedure

snapraid-diff

If one of the disks fails, do not run plain command snapraid sync or snapraid might think the files that are gone were deleted on purpose! It should complain and require -E, --force-empty flag for sync to happen when an entire data disk that previously had files is now empty, but still...

Simulating a disk failure.

  • umount one of the disks sudo umount /mnt/disk_2
  • manually execute scheduled snapraid-sync-and-maintenance script
    sudo systemctl start snapraid-sync-and-maintenance.service
  • a notification should come that something is fucky, assuming you have notifications setup
  • ssh in and lsblk and sudo snapraid diff,
    maybe check the last run in the journal systemctl status snapraid-sync-and-maintenance.service
  • check network share and see missing episodes in shows or other files
  • simulate replacing the dead drive by wiping it clean
    • sudo wipefs -a /dev/sdc
    • sudo parted /dev/sdc --script mklabel gpt
    • sudo parted /dev/sdc --script mkpart primary ext4 0% 100%
    • sudo mkfs.ext4 /dev/sdc1 -L disk_2
    • sudo mount /dev/sdc1 /mnt/disk_2
    • do lsblk -f and edit the fstab with the new uuid so it gets mounted on boot
  • run snapraid recovery sudo snapraid fix -d disk_2
  • afterwards check snapraid status and browse if stuff is really back in place
  • fix the ownership and permissions for files and directories
    • sudo chown -R $USER:$USER /mnt/disk_2
    • sudo find /mnt/disk_2 -type d -exec chmod 755 {} \;
    • sudo find /mnt/disk_2 -type f -exec chmod 664 {} \;
  • done

Test if stuff in shares is as it should be.

Network File Sharing - Samba and NFS

samba-nfs

  • Samba - Well supported by all systems - windows, linux, android, macos,...
    also called by the protocol name - SMB or CIFS
  • NFS - Simple, ideal for sharing files between linux machine.
  • iSCSI - Sharing network storage as a block device, literally appears as an unformatted disk on a client machine. Good support, great performance and can be very useful, but impossible to do with file based MergerFS approach.

Samba Setup

samba

Arch Wiki

  • Install samba.
  • Copy the config below in to /etc/samba/smb.conf
    notice the user named bastard being used in it, replace with yours.
    The config needs to exist or other commands wont work.
  • Add your local linux user to samba and set password.
    sudo smbpasswd -a bastard
    Messier alternative, create a new user just for samba
    • sudo useradd -M -s "$(which nologin)" smbuser
    • that user either needs ownership of the shared folder sudo chown -R smbuser:smbuser /mnt/pool, or needs to be added to the group of whoever owns it sudo usermod -aG myspeciallocaluser smbuser and make sure permission allows group to write sudo chmod 0775 /mnt/pool
  • enable smb.service - sudo systemctl enable --now smb.service
  • I don't install nmb.service for the old netbios discovery, it's a dead technology.
  • If windows machines should have the PC appear in network on it's own, install wsdd and enable the service sudo systemctl enable --now wsdd.service

/etc/samba/smb.conf

[global]
   # Security
   security = user
   map to guest = Never
   server min protocol = SMB2

   # Network
   disable netbios = yes
   smb ports = 445
   dns proxy = no
   deadtime = 0

   # MergerFS/FUSE compatibility
   aio read size = 0
   aio write size = 0
   kernel oplocks = no
   posix locking = no
   strict locking = no
   use sendfile = yes
   #use sendfile = no          if transfer issues 
   min receivefile size = 16384
   #min receivefile size = 0   if transfer issues

   # Performance
   socket options = TCP_NODELAY

   # Logging
   log file = /var/log/samba/%m.log
   max log size = 1000

   # Disable printing
   load printers = no

[Pool]
   path = /mnt/pool
   browseable = yes
   writable = yes
   guest ok = no
   valid users = bastard
   create mask = 0664
   directory mask = 0775
   force user = bastard
   force group = bastard

Permanent Samba mount at boot

NFS Setup

nfs

Arch Wiki

Ideal for sharing between linux machines, hypervisor, and also works ok in the apple ecosystem. Windows has some support, but samba is more reliable for regular use.
NFS v3 is picked, as oppose to v4, for simplicity, robustness, and performance.
With v3 the access control to shares is only IP based, allowing in specific IPs, or entire networks.

  • install nfs
    on arch it's nfs-utils

  • edit /etc/nfs.conf and enable nfs version 3

    [nfsd]
vers3 = y
vers4 = n

  • edit /etc/exports adding line that sets up a share
    /mnt/pool 10.0.19.0/24(rw,no_root_squash,fsid=1)

    nfs export options explained
    • rw - read and write allowed
    • no_root_squash - if mounted at linux that requires to write to the share as root, its respected and the files are owned by root, uid 0
    • fsid=1 - manually set filesystem id, with mergerfs it might be useful to have stability
    • async - is now default so skipped, better performance on write, but on power loss or crashes data might be lost
    • no_subtree_check - is now default so skipped, improves reliability and performance at the expense of some potential security issues
    • all_squash - anyone and everyone who writes, that write is squashed to be done by the anonymous user with specific uid/guid
    • anonuid=1000,anongid=1000 - defines uid/guid of the anonymous user

  • enable nfs-server service - sudo systemctl enable --now nfs-server

  • for file permissions:

    • on the server have your user have ownership of the share sudo chown -R $USER:$USER /mnt/pool
    • command id tells uid/gid of that user of yours.
      if the client side user has the same uid/gid stuff will just work
    • if more separate users on the client side, then create those users as local on the server with correct uid/gid and permissions in to the share

Permanent NFS mount at boot

HDD Spindown

spindown-gif

Saves ~3W of power per disk, heat, vibrations, wear, noise,... but the first time accessing the storage takes ~5 seconds.

Disks might or might not spindown on their own on idle without you doing anything.
Most distros don't force it, but the firmware preset of the disks might.

  • command showing the spindown state of every drive:
    for d in /dev/sd[a-z]; do echo -n "$d: "; sudo hdparm -C "$d" | grep state; done
  • See disks activity:
    sudo iotop -ao
    sudo blktrace -d /dev/sdd -o - | blkparse -i -

If you want to take control hdparm can change power saving and idle time.

  • sudo hdparm -B ... - controls APM power saving in drives firmware
  • sudo hdparm -S ... - controls idle standby timer in drives firmware

The execution does not survive reboot so systemd service is used to apply prefered behaviour on boot.

Control spindown

/etc/systemd/system/hdds-spindown-enabled.service

[Unit]
Description=Enable spindown for all HDDs after 2 hours idle
After=local-fs.target systemd-udev-settle.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'for d in /sys/block/sd*; do [ -f "$d/queue/rotational" ] && [ "$(cat $d/queue/rotational)" -eq 1 ] && [ -b "/dev/$(basename $d)" ] && /usr/bin/hdparm -S 244 -B 127 /dev/$(basename $d) 2>/dev/null || true; done'

[Install]
WantedBy=multi-user.target

Enable the service
sudo systemctl enable --now hdds-spindown-enabled.service

The 2 hours is set by hdparm -S 244, hdparm uses bit weird time system.

  • 1 - 240 are multipliers of 5 seconds, so 60=300s=5m, 180=900s=15m,...
  • 241 - 251 are multipliers of 30 minutes, so 241=30m, 242=60m, 243=90m,...

As for the hdparm -B 127 - the APM range is 1 - 254. Lower the number more aggresive power savings. The 127 value is a moderate balanced level that should not override the timer.

Prevent spindown

In a case where you want to prevent spindown.

/etc/systemd/system/hdds-spindown-prevent.service

[Unit]
Description=Prevent spindown for all HDDs
After=local-fs.target systemd-udev-settle.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'for d in /sys/block/sd*; do [ -f "$d/queue/rotational" ] && [ "$(cat $d/queue/rotational)" -eq 1 ] && [ -b "/dev/$(basename $d)" ] && /usr/bin/hdparm -S 0 -B 255 /dev/$(basename $d) 2>/dev/null || true; done'

[Install]
WantedBy=multi-user.target

Enable the service
sudo systemctl enable --now hdds-spindown-prevent.service

Be aware, freshly formated ext4 disks finish their initialization in the background. Kernel runs ext4lazyinit process that zeros inodes and disks can be heard chirping and doing something even when there should be no activity. Though also disks firmware might be doing stuff on its own occasionally.

Monitoring spindowns and spinups

To have a degree of certainty that disks are not spinning up and down 400 times a day, accelerating their degradation.

  • Create the log script in /opt/disks-spin-logger.sh

    disks-spin-logger.sh
     
    #!/bin/bash
LOGFILE="/var/log/disks-spin.log"
STATEFILE="/var/lib/disks-spin.state"
SUMMARY="/var/log/disks-spin-summary.log"
DATE=$(date +"%Y-%m-%d %H:%M:%S")
DAY=$(date +"%Y-%m-%d")

mkdir -p /var/lib

for dev in /dev/sd[a-z]; do
    CURR_STATE=$(hdparm -C "$dev" 2>/dev/null | awk '/drive state/ {print $NF}')
    [[ -z "$CURR_STATE" ]] && continue  # skip if no result

    PREV_STATE=$(awk -v d="$dev" '$1==d {print $2}' "$STATEFILE" 2>/dev/null)

    if [[ "$CURR_STATE" != "$PREV_STATE" ]]; then
        echo "$DATE $dev $CURR_STATE" >> "$LOGFILE"

        # count spinups / spindowns
        case "$CURR_STATE" in
            active*|idle*)  # matches "active", "active/idle", "idle"
                EVENT="SPINUP"
                ;;
            standby)
                EVENT="SPINDOWN"
                ;;
            *)
                EVENT=""
                ;;
        esac

        if [[ -n "$EVENT" ]]; then
            # Update summary: device + day + counters
            awk -v dev="$dev" -v day="$DAY" -v event="$EVENT" '
                BEGIN {found=0}
                {
                    if ($1==day && $2==dev) {
                        if (event=="SPINUP")   $3++
                        if (event=="SPINDOWN") $4++
                        found=1
                    }
                    print
                }
                END {
                    if (!found) {
                        up=(event=="SPINUP")?1:0
                        down=(event=="SPINDOWN")?1:0
                        print day, dev, up, down
                    }
                }
            ' "$SUMMARY" 2>/dev/null > "$SUMMARY.tmp"
            mv "$SUMMARY.tmp" "$SUMMARY"

            if ! grep -q "^DATE" "$SUMMARY"; then
                sed -i '1iDATE DEVICE SPINUPS SPINDOWNS' "$SUMMARY"
            fi

        fi

        # update state file
        grep -v "^$dev " "$STATEFILE" 2>/dev/null > "$STATEFILE.tmp"
        echo "$dev $CURR_STATE" >> "$STATEFILE.tmp"
        mv "$STATEFILE.tmp" "$STATEFILE"
    fi
done

  • Make the script executable: sudo chmod +x /opt/disks-spin-logger.sh

  • create a systemd unit files

    /etc/systemd/system/disks-spin-logger.service

    [Unit]
Description=Log disks spin state

[Service]
Type=oneshot
ExecStart=/opt/disks-spin-logger.sh

    /etc/systemd/system/disks-spin-logger.timer

    [Unit]
Description=Run disks spin logger periodically

[Timer]
OnBootSec=1min
OnUnitActiveSec=5min
AccuracySec=10s
Persistent=true

[Install]
WantedBy=timers.target

  • enable the timer
    sudo systemctl enable --now disks-spin-logger.timer

The log files will be in /var/log/

  • disks-spin.log - state of every disk at the check interval.
  • disks-spin-summary.log - total number of spin downs and ups per day

logrotate

To prevent growth of the log files.

  • install logrotate
  • create a config file
    /etc/logrotate.d/disks-spin 
    /var/log/disks-spin*.log {
    size 20M
    rotate 1
    missingok
    notifempty
    create
}

Can do dry run to see what it would do sudo logrotate -d /etc/logrotate.conf

Hardware

pic-nas-cases

Case

Should be the very first thing you decide on.
How big or small, how expensive, how many 3.5"/2.5" disks positions? What motherboard and PSU will fit in it?
My pick - mATX case from aliexpres - Sagittarius

  • was 150€ with shipping
  • I like the idea of a smaller case, but not too small as ITX motherboards are expensive and much more limiting with PCIE slots. SFX power supplies got much more expensive too.
  • 8x 3.5" hot swappable disk bays, sata / sas compatible
  • great cooling with space for 2x 120fans just for disks, and another 2x 120 for the mobo section

Other popular cases

  • Jonsbo N line has nice stuff
  • Fractal design is often picked, with node line for smaller cases, and Define R5 R6 when big case is not an issues and you want lot of positions.
  • InWin Chopin MAX - when 3.5" disks are not needed, just want a tiny case.

hba-cards-pic

HBA card

If you have more than 4x disks, it might be difficult finding a motherboard with enough sata ports. Or if you plan to to run TrueNAS in a VM you need an HBA card that you passthrough in to it, so that TrueNAS has full access to disks without any abstraction layers.

The ideal solution is to buy a used enterprise-tier raid card in the IT mode.
The IT mode means the raid functionality is disabled and it's just a pcie card that provides you with plenty of sata connection for the drives. But it is of high quality as you don't want to be trying to solve - why sometimes some disks are disconnecting or speeds are weird, or whatever issues...
These cards cost like 400€ new, but you can get them cheap on ebay.

General cards overview - youtube video

The Naming Scheme

  • LSI - the company that manufactures the chips, owned by Broadcom
  • SAS2008; SAS3008; SAS3408; SAS3808 - specific chip
  • 9211-8i; 9300-8i; 9400-8i; 9500-8i - specific model of a card, usually with x8 disks connections
  • Various cable connector types - SFF-8087; SFF-8643; SFF-8654;...

9211-8i used to be the go-to recommendation since they are cheap and the performance is more than enough for spinning drives.
Then the price of 9300-8i started to drop and is about 40€, So why not get newer hardware...

But then theres the power consumption.
A video came out measuring power consumption of these, and the 9300 was idling at highest 11W, the newer and more expensive 9400 was 7W and 9500 was 6W. The old 9211 was 7W. So take that in to consideration.

And to repeat, it must be in the IT mode and ideally ordered straight away with cables. There are sellers with 99%+ positive feedback with 150k+ items sold, so one should not be too scared of buying these.

cpus-mobo-pic

CPU, motherboard, ram

Can be anything you have on hand, can be something really special, all depending on needs, wants and the budget.

Big decision Intel vs AMD.

  • Intel - bit better igpu performance for video transcoding, if planing to run something like jellyfin...
  • AMD - possibility to go ECC ram with some CPUs, which improves stability and reliability a bit

You can also go budget option with something like N100 based motherboard, like ASRock N100M, but not having 2.5gbit network card is kinda deal breaker for me.

Currently in my NAS I have an old pentium G3240, waiting for me to decide what I like to to put in... and then the ram prices happened...

psu-pic

Power supply

PSU tier list

My go-to used to be seasonic, but they started to get really expensive with the PSUs actually manufactured by them.

Last few years I went with ADATA XPG core reactor II (not the VE version). It's manufactured by CWT, has A+ rating on the tier list, 10 years warranty and costs me 80€.
Switching from a very old seasonic that my test NAS had, power consumption went from 26W idle to 22W idle. Fucking stiff cables though.

network-stuff-pic

Network cards

Plan ahead, higher speed NICs require also higher speed switches.

2.5gbit

Started to be really affordable. There are switches from ubiquiti and mikrotik and 2.5gbit NICs are really common in new motherboards now. So it's not that expensive and usually worth it, as even a single spinning HDD is 2x faster than the pathetic 1gbit network speed.

10gbit

People like the idea and the cost of getting there is doable, but if in the planning stage and your home situation allows it... plan not to go for 10GBase-T, meaning not going rj45 copper twisted pair cables and buying 10gbit rj45 switches and NICs. They tend to run very hot, with high power consumption. But I understand, if you already have cat6a cable run... it's bothersome to start to think about cables again.

But if your situation allows, consider going for SFP+ switches and NICs.
For distances under 7m you use cheap and very reliable DAC cables. For longer runs its optical with transceivers.
SFP+ switches run cool enough to be passively cooled. Popular cheap choices are Mikrotik CRS305-1G-4S+IN and CRS309-1G-8S+. But many of their switches have few SFP+ ports.
For 10gbit network cards popular choice is used ebay - intel x520s or x710, or Mellanox ConnectX-3, ConnectX-4, ConnectX-5.
Older cheaper ones, might not support ASPM, which means they will not let CPU reach higher states of power saving. X710 has good reputation on that, but then it also has reputation that it is very VERY picky about cables and transceivers... Some discussion here, some guide here.

Other Guides

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors