tldr: I’m going to set up raid z2 with 4x8TB hard drives. I’ll have photos, documents (text, pdf, etc.), movies/tv shows, and music on the pool. Are the below commands good enough? Anything extra you think I should add?

sudo zpool create mypool raidz2 -o ashift=12 /dev/disk/by-id/12345 ...

zfs set compression=lz4 mypool #maybe zstd?
zpool set autoexpand=on mypool
zpool set autoreplace=on mypool #I might keep this off. I can see myself forgetting in the future
zpool set listsnapshots=on mypool

With ai raising hard drive prices, I over spent on 3x10TB drives in order to reorganize my current pool and have 3 hard drives sitting on a shelf in the event of a failure. My current pool was built over time but it currently consists of 4x8TB drives. They are a mirrored stripe so a usable 16TB. If I understand it correctly, I can lose 1 drive for sure without losing data and maybe a second drive depending on which drive fails. Because of that, I want to move to raid z2 to ensure I can lose 2 drives without data loss. I’m going to move data from my 4x8TB drives, to the 3x10TB, reconfigure the 4x8TB, and move everything back. I run Immich, plex/jellyfin, and navidrome off the pool. All other documents are basically there for long term storage just in case. What options should I use for raid z2 when setting it up?

I know I can look this stuff up. I have been and continue to do so, I was just hoping for some advise from people that are more knowledgeable about this than me. The move from the 4x8TB drives to the 3x10TB is going to take ~3 days so I really don’t want to mess this up and have to start over 😅

Edit:

After looking up each property, this is the command I will probably end up using to create the raid z2 pool, thanks Avid Amoeba:

sudo zpool create
-o ashift=12 -o acltype=posixacl -o xattr=sa
-o compression=lz4 -o dnodesize=auto -o relatime=on
-o normalization=formD
raidz2
mypool
/dev/disk/by-id/12345 …

Edit2:

Above command didn’t work on my machine. The order and uppercase “O” matters. Had to do this:

sudo zpool create \
  mypool \
  raidz2 \
  -o ashift=12 -O compression=lz4 \
  -O normalization=formD -O acltype=posixacl \
  -O xattr=sa -O dnodesize=auto \
  -O relatime=on \
  /dev/disk/by-id/12345 ...

Edit3:

And finally, after all this, I set up my tmp pool of 3x10TB disks as a raid z2 instead of raid z1. Spent a day and a half transferring before I finally saw my mistake after running out of space 🫠

  • greyfox@lemmy.worldEnglish
    2·
    5 hours ago

    You probably shouldn’t enable compression on the root filesystem of the pool. Since you mention movies/TV shows/music those are just going to waste cpu cycles compressing uncompressable data.

    Instead you should consider separate ZFS filesystems for each data type. Since ZFS is a pool you don’t have to pre-allocate space like partitions so there is no harm in having separate filesystems for each data type rather than single large filesystem for everything. You can then turn on compression only for those filesystems that benefit from it.

    Also remember that many permissions like nfs export settings are done on a per filesystem basis so you should lay out your filesystems according to your data type and according to what permissions you want to give out for that filesystem.

    i.e. if you are going to have a Navidrome server to stream your music you don’t want to give that server access to your entire pool, just your music.

    Separate filesystems also means you can have different snapshot schedules/retentions. Documents might need to snapshot more often, and be kept around longer than media snapshots.

    • BlackAura@lemmy.worldEnglish
      2·
      5 hours ago

      If I recall correctly compression, especially lz4, has been shown to impact performance negligably.

      • greyfox@lemmy.worldEnglish
        1·
        3 hours ago

        Yeah it won’t make much difference these days.

        I suppose my point was more so that because ZFS is a pool that can be split up with filesystems new users should be thinking a little differently than they would have been used to with traditional raid volumes/partitions.

        With a normal filesystem partitions are extremely limiting, requiring you to know how much space you need for each partition. ZFS filesystems just being part of the pool means that you can get logical separation between data types without needing that kind of pre-planning.

        So many settings with ZFS that you may want to set differently between data types. Compression, export settings, snapshot schedules, replicating particular data sets to other systems, quotas, etc.

        So I was mostly just saying “you should consider splitting those up so that you can adjust settings per filesystem that make sense”.

        There is also a bit of danger with a single ZFS filesystems if you have no snapshots. ZFS being a copy on write filesystem means that even deleting something actually needs space. A bit counter intuitive but deleting something means writing a new block first then updating the FS to point at the new block. If you fill the pool to 100% you can’t delete anything to free up space. Your only option is to delete a snapshot or delete entire filesystems to free up a single block so that you can cleanup. If you don’t have a snapshot to delete you have to delete the entire filesystem and if you only have one filesystem you need to backup+delete everything… ask me how I know this ;)

        If you have several filesystems you only need to backup and destroy the smallest one to get things moving again. Or better yet have some snapshots you can roll off to free up space or have quotas in place so that you don’t fill the pool entirely.

    • a_fancy_kiwi@lemmy.worldOPEnglish
      1·
      4 hours ago

      Let’s say I did turn on compression on root. I can’t then turn it off on a per file system basis where it isn’t needed?

      • greyfox@lemmy.worldEnglish
        2·
        4 hours ago

        Yep you can certainly do it that way.

        If your concern is just that you already created it you can shut it off after the fact. All new blocks written will have compression disabled.

        • a_fancy_kiwi@lemmy.worldOPEnglish
          1·
          4 hours ago

          No, you actually caught be at the perfect time, the transfer to my temporary pool is almost done. I was just curious how inheritance worked on a pool but after giving it some thought, your recommendation makes more sense; turn it on when I know I need it vs turn it off when I know I don’t. Thanks for the advice.

  • Avid Amoeba@lemmy.caEnglish
    5·
    2 days ago

    I do:

    sudo zpool create \
   -o ashift=12 -O acltype=posixacl -O compression=lz4 \
   -O dnodesize=auto -O normalization=formD -O relatime=on \
   -O xattr=sa \
   mypool \
   raidz2 \
   wwn-0x5000cca284c06395 \
   wwn-0x5000cca295e115f3 \
   wwn-0x5000cca2a1ef9c90 \
   wwn-0x5000cca295c03910 \
   wwn-0x5000cca29dd216b0

    I’m then going to optimize recordsize depending on the workload in datasets. E.g. Immich db might use 8K or 16K recordsize while the library dataset where the files are might be larger so that search is faster. Etc.

      • Avid Amoeba@lemmy.caEnglish
        3·
        2 days ago

        The acl, norm, xattr, dnodesize options are all to make ZFS behave like a Linux filesystem and are pretty standard for using ZFS on Linux. They aren’t default because ZFS’ defaults are for Unix.

    • a_fancy_kiwi@lemmy.worldOPEnglish
      1·
      2 days ago

      Yeah I’m not excited about the write and rebuild times being slower but the read times should still be pretty good. Considering I don’t have any more space for drives in my server and I don’t know how crazy hdd drive prices will get in the next 12 months, the guaranteed 2 drive failure resiliency is more important to me at the moment. My current 1 drive failure resiliency, 2 if I’m lucky, has me worried. My backups are on shucked drives and I don’t want to be put in a situation where I have to rely on them to restore 😅

    • non_burglar@lemmy.worldEnglish
      1·
      2 days ago

      That’s still true, but performance has changed a lot since Jim Salter wrote that. There was a time When 2x mirrored vdevs (the equivalent to raid 10) would have been preferable to raidz2, but performance of both ZFS and disks themselves has improved enough that there wouldn’t be much of a difference in performance between these two in a home lab.

      Personally, I agree with you in that mirrors are preferable, mostly because I don’t really need high availability as much as I want an easier time restoring if a disk fails.