ZFS

The FreeBSD Handbook ZFS chapter captures the conceptual breakthrough: “More than a file system, ZFS is fundamentally different from traditional file systems. Combining the traditionally separate roles of volume manager and file system provides ZFS with unique advantages. The file system is now aware of the underlying structure of the disks. Traditional file systems could exist on a single disk alone at a time.”¹ This single architectural choice (merging the file system and volume manager into one stack) enabled most of ZFS’s distinctive features and replaced the complexity of journaling file systems with cleaner copy-on-write semantics.

The 2001-2006 inception

The Open-E ZFS introduction captures the development timeline: “ZFS stands for Zettabyte File System and is a file system originally developed by Sun Microsystems for building next-generation NAS solutions with better security, reliability, and performance. ZFS was designed in 2001 by Matthew Ahrens and Jeff Bonwick and it was supposed to be a next-generation file system for another Sun Microsystems’ system called OpenSolaris.”² The Wikipedia ZFS documentation describes the productization: “In Solaris 10 6/06 (‘U2’), Sun added the ZFS file system and frequently updated ZFS with new features during the next 5 years.” The five-year design period before initial release was unusually long for a file system; it reflects the depth of architectural rethinking ZFS represented.

The CDDL and the Linux complication

Sun’s open-source strategy and Oracle’s later acquisition shaped ZFS’s modern distribution. The Wikipedia documentation notes: “Solaris was originally developed as proprietary software, but Sun Microsystems was an early commercial proponent of open source software and in June 2005 released most of the Solaris codebase under the CDDL license and founded the OpenSolaris open-source project.” The Open-E documentation captures the post-acquisition turn: “After the purchase of Sun Microsystems by Oracle, OpenSolaris was no longer an open-source project. Also, two-thirds of the system developers left Oracle at that time.”

The CDDL is incompatible with the GPL, which means ZFS cannot be merged into the mainline Linux kernel. The OpenZFS project ships ZFS as a separate kernel module that’s compiled against the kernel’s headers; this is technically legal but creates ongoing tension with each kernel release. Other Unix-like systems (FreeBSD, illumos) don’t have this constraint and ship ZFS as a native part of the system.

The OpenZFS unification

The Open-E documentation captures the cross-platform porting timeline: “A port for FreeBSD was made in 2008. Unlike other systems on the market, ZFS is a 128-bit file system offering virtually unlimited capacity. In turn, ZFS on Linux (ZOL) was created in 2013. Brian Behlendorf, Jorgen Lundman, Aron Xu, and Richard Yao were among those who helped to create and maintain ZOL.” Until 2020, ZFS development happened in parallel across multiple projects (illumos for Solaris-derived OSes, FreeBSD’s tree, ZFS on Linux); the OpenZFS project unified development into a single codebase that all platforms now share. This unification has substantially accelerated ZFS development; new features land across all platforms simultaneously.

The 5 architectural elements

The original Bonwick and Moore CS paper “The Zettabyte File System” identifies the foundational architectural elements: “The key architectural elements of ZFS that solve these problems are pooled storage, the movement of block allocation out of the file system and into the storage pool allocator, an object-based storage model, checksumming of all on-disk blocks, transactional copy-on-write of all blocks, and 128-bit virtual block addresses.”³ Each of these elements addresses a specific limitation of traditional file systems:

• Pooled storage: eliminates the partition-then-file-system rigid layering of traditional storage.
• Block allocation in pool allocator: file system doesn’t manage free space; the pool does.
• Object-based storage: everything is an object; metadata and data use the same primitive.
• Checksumming all blocks: end-to-end integrity verification.
• Transactional COW: atomic updates without journaling complexity.
• 128-bit addressing: capacity beyond any foreseeable need (the “Zettabyte” in the name).

Why “Z”?

The “Z” originally stood for “Zettabyte,” reflecting the 128-bit addressing capability. The itsfoss community guide captures the playful interpretation: “Even though I’m from the US, I prefer to pronounce it ZedFS instead of ZeeFS because it sounds cooler.” Sun later officially adopted “Z File System” as the canonical expansion, dropping the explicit zettabyte tie. The name implicitly conveyed the design ambition: zettabyte-scale storage was decades beyond what any system needed in 2001, but ZFS’s design assumed that scale would eventually arrive.

ZFS’s architecture is layered differently from traditional file systems. Understanding the layering clarifies how ZFS’s features fit together.⁴

Vdevs: virtual devices

The Open-E documentation describes the bottom layer: “Pooled storage is basically a ZFS pool that is a collection of one or more vdevs that are virtual devices storing the data. The ZFS pool, also called Zpool, serves as the highest data container in the whole ZFS system.” A vdev is the basic unit of storage redundancy in ZFS:

• Single disk vdev: a single physical disk; no redundancy.
• Mirror vdev: two or more disks containing identical data; survives N-1 disk failures.
• RAID-Z1 vdev: three or more disks with single parity; survives 1 disk failure.
• RAID-Z2 vdev: four or more disks with double parity; survives 2 disk failures.
• RAID-Z3 vdev: five or more disks with triple parity; survives 3 disk failures.
• Special vdevs: cache (L2ARC SSD read cache), log (SLOG/ZIL synchronous write), spare (hot standby).

The zpool

A zpool is a collection of one or more vdevs that aggregate into a single storage pool. ZFS stripes data across vdevs within the pool (effectively RAID 0 across vdevs), so adding more vdevs to a pool increases both capacity and performance. A pool’s redundancy is limited by its weakest vdev: a pool with one mirror vdev and one single-disk vdev has no redundancy because the single-disk vdev’s failure breaks the entire volume behind the entire pool. This is why ZFS administrators are careful to maintain consistent redundancy across all vdevs in a pool.

Datasets and zvols

The Open-E documentation continues: “The Zpool is used to create one or more file systems (datasets) or block devices (volumes). These file systems and block devices share the remaining pool’s space. Partitioning and formatting operations will be conducted by the ZFS system.” Datasets are the user-facing storage units:

• File system datasets: mountable as directories; contain files and directories like a traditional file system (ext4 or NTFS).
• zvols (block volumes): appear as block devices; can be formatted with other file systems or used as iSCSI targets.
• Snapshots: read-only copies of datasets at points in time.
• Clones: writable datasets created from snapshots; share blocks until divergence.

Inherited properties

Datasets form a hierarchy with property inheritance. Setting compression=lz4 on a parent dataset means all child datasets inherit lz4 compression unless explicitly overridden. The FreeBSD documentation example: zfs create example/compressed followed by zfs set compression=gzip example/compressed creates a compressed dataset. Inherited properties make managing large dataset hierarchies practical; setting encryption=on on a /tank/users dataset means all user home directories are encrypted without per-user configuration.

The caching architecture: ARC, L2ARC, ZIL

The FreeBSD Handbook describes ZFS’s three-tier caching: “ARC is an advanced memory-based read cache. ZFS provides a second level disk-based read cache with L2ARC, and a disk-based synchronous write cache named ZIL.” Each tier serves a specific purpose:

• ARC (Adaptive Replacement Cache): in-memory read cache with sophisticated eviction (combining LRU and LFU). Sized based on available RAM.
• L2ARC (Level 2 ARC): SSD-based extension to ARC for systems with large datasets and limited RAM. Read-only cache.
• ZIL (ZFS Intent Log): in-pool synchronous write log; ensures fsync-style durability.
• SLOG (Separate Log): dedicated SSD for ZIL; reduces synchronous write latency dramatically.

The cache tiers are optional; pools without explicit cache or log devices use ARC only and embed ZIL within the main pool.

Properties as configuration

ZFS treats nearly all configuration as properties on datasets or pools. Common dataset properties include compression (off, lz4, zstd, gzip), encryption (off, aes-256-ccm, aes-256-gcm), deduplication (off, on), quota (size limit), reservation (guaranteed space), recordsize (block size for the dataset), and atime (whether access times are updated). This property-based configuration is uniformly accessible via the zfs command; learning ZFS administration is largely learning the property model.

End-to-end data integrity is what most distinguishes ZFS from traditional file systems. The integrity guarantees apply to both data and metadata, with automatic repair when redundancy exists.

The Wikipedia design quote

The Wikipedia ZFS documentation captures the design philosophy: “ZFS is designed to ensure (subject to sufficient data redundancy) that data stored on disks cannot be lost due to physical errors, misprocessing by the hardware or operating system, or bit rot events and data corruption that may happen over time. Its complete control of the storage system is used to ensure that every step, whether related to file management or disk management, is verified, confirmed, corrected if needed, and optimized, in a way that the storage controller cards and separate volume and file systems cannot achieve.”⁵

The 256-bit checksum mechanism

Every block ZFS writes (data and metadata) is checksummed with a 256-bit hash. The checksum is stored in the parent block’s metadata, not alongside the data it protects; this separation is what makes the integrity end-to-end:

• When ZFS reads a block, it recomputes the checksum and compares it to the stored value.
• If the checksums match, the data is intact.
• If the checksums don’t match, ZFS knows the data is corrupted regardless of what caused the corruption.
• If redundancy exists (mirror or RAID-Z), ZFS automatically retrieves the correct data from a redundant copy.
• ZFS rewrites the corrupted copy to repair it (“self-healing”).

The 256-bit hash space makes accidental collision essentially impossible; if checksums match, the data is intact.

What “end-to-end” means

The end-to-end guarantee is that corruption can be detected and corrected regardless of where in the storage stack it occurs. The Wikipedia documentation lists protected scenarios: “data degradation, power surges (voltage spikes), bugs in disk firmware, phantom writes (the previous write did not make it to the disk), misdirected reads/writes (the disk accesses the wrong block), DMA parity errors between the array and server memory or from the driver, accidental overwrites.” Traditional file systems trust the storage stack; ZFS verifies.

Self-healing on read and on scrub

Self-healing happens both reactively (when corruption is detected during normal reads) and proactively (during scrubs). The zpool scrub command verifies every block in the pool against its checksum, repairing any corrupted blocks from redundant copies. Best practice on production ZFS systems is to run scrubs on a schedule (typically monthly for SAS/SATA disks, weekly for less reliable drives); regular scrubs catch corruption before it accumulates beyond what redundancy can repair.

The “bit rot” prevention claim

ZFS is sometimes described as “bit rot prevention.” Strictly speaking, bit rot still happens at the physical media level; ZFS doesn’t prevent the underlying bit flips. What ZFS does is detect bit rot via checksum mismatches and repair it from redundant copies before the user observes it. The user-visible effect is that bit rot doesn’t lose data on properly-redundant ZFS pools, even though the underlying physical phenomenon still occurs.

Compression and deduplication

ZFS includes built-in compression (lz4, zstd, gzip) and optional deduplication. Compression is generally recommended on (lz4 has minimal CPU overhead and can save substantial space on compressible data); deduplication is more controversial because it requires substantial RAM (typically 5 GB per TB of unique data) and can produce unpredictable performance. Modern ZFS guidance is to enable lz4 compression universally and avoid deduplication unless specific workloads benefit substantially.

RAID-Z is ZFS’s software RAID implementation, designed to address specific weaknesses in traditional RAID 5 and RAID 6. Understanding RAID-Z requires understanding the write hole problem it solves.

The classical RAID write hole

Traditional RAID 5 and RAID 6 implementations have a vulnerability called the write hole: when a stripe is being written, the system must update both the data blocks and the parity block atomically. If a power failure occurs after some blocks have been written but before all are complete, the stripe is left inconsistent: data blocks and parity don’t match. After power restoration, the array can’t tell which blocks were written successfully and which weren’t; subsequent reads can return undetected corrupted data because parity calculation appears valid for a corrupt stripe. The write hole has caused real silent data loss in production environments; battery-backed write caches and journal-based workarounds reduce but don’t eliminate the risk.

How RAID-Z solves the write hole

RAID-Z uses ZFS’s copy-on-write design to eliminate the write hole. Instead of overwriting stripes in place, RAID-Z writes new stripes to free space and updates metadata pointers atomically:

• If power fails before the new stripe write completes, the stripe is incomplete but the metadata still points to the old (intact) stripe.
• If power fails before metadata update completes, same outcome: old data is still pointed to.
• Only after both new stripe write and metadata update complete is the old stripe considered free for reuse.

This atomicity at the transaction level means RAID-Z stripes are either fully written or not written; there’s no partial-write inconsistency to recover from.

RAID-Z variable-width stripes

RAID-Z also addresses RAID 5/6’s space efficiency problem with small files. Traditional RAID 5 always writes full stripes; a 4 KB write to a 5-disk RAID 5 array still uses 4 KB × 5 = 20 KB on disk (4 data + 1 parity). RAID-Z uses variable-width stripes that only span the necessary disks:

• Small files: may use a 2-disk stripe (1 data + 1 parity) regardless of the array size.
• Large files: use full-width stripes for efficiency.
• Write efficiency: small writes don’t waste space on unused stripes.

RAID-Z levels

RAID-Z is available in three variants providing different fault tolerance:

Level	Min disks	Survives	Capacity efficiency	Use case
RAID-Z1	3	1 disk failure	(N-1)/N	Equivalent to RAID 5; small arrays
RAID-Z2	4	2 disk failures	(N-2)/N	Equivalent to RAID 6; medium arrays
RAID-Z3	5	3 disk failures	(N-3)/N	Triple parity; large arrays

When to use which RAID-Z level

The choice depends on disk count and reliability requirements:

• 3-5 disks: RAID-Z1 is typical; mirror is the alternative for smaller arrays.
• 6-10 disks: RAID-Z2 is recommended; modern large drives’ rebuild times make double parity worthwhile.
• 10+ disks: RAID-Z3 increasingly common; rebuild times can exceed 24 hours during which a second failure produces total loss.
• Mirror vdevs: alternative for performance-critical workloads; faster reads but higher cost.

RAID-Z performance characteristics

RAID-Z performance differs from traditional RAID 5/6:

• Sequential reads: excellent; can stripe across all data disks.
• Random reads: limited to single-vdev IOPS; large random workloads benefit from multiple vdevs.
• Sequential writes: good; full-width stripes are written atomically.
• Synchronous writes: require ZIL; SLOG SSD recommended for database workloads.
• Resilver (rebuild) time: proportional to data volume, not array size; more efficient than traditional RAID 5/6 rebuild.

ZFS’s design eliminates many traditional recovery scenarios entirely; the file system’s atomic transactions and end-to-end integrity mean that “running fsck after a crash” simply doesn’t exist as a concept. However, ZFS-specific failure modes do exist and require specialized recovery approaches.

What ZFS makes irrelevant

Several traditional recovery scenarios are largely irrelevant on healthy ZFS pools:

• Crash recovery: ZFS is always consistent on disk; crashes are recovered by simply replaying the most recent transaction group, which takes seconds.
• fsck: ZFS doesn’t have a fsck equivalent because the on-disk format is always consistent. Scrubbing replaces the role of fsck for finding silent corruption.
• Bit rot recovery: automatic on redundant pools; the user never sees corrupted data because ZFS repairs it on read or scrub.
• Partial-write corruption: impossible in ZFS due to atomic transactions.
• Lost-write detection: ZFS detects when writes don’t reach disk via checksum mismatches.

When ZFS recovery is needed

ZFS recovery becomes necessary in specific scenarios:

• Pool import failures: after severe events, the pool may not import normally.
• Multi-disk failures beyond redundancy: RAID-Z1 with 2 disks failed, mirror with both copies lost, etc.
• Accidental destruction: zpool destroy followed by realization that the pool was needed.
• Hardware failures affecting pool metadata: RAM corruption during writes can cause poolwide damage.
• Software bugs: rare but historically have produced data loss in some ZFS versions.
• Encryption key loss: for native-encrypted datasets, lost keys mean data is unrecoverable.

The zpool import -F recovery

For pool import failures, the standard first step is zpool import -F, which attempts to import the pool by rolling back to a recent transaction group that was successfully committed. This often resolves issues from incomplete writes during power loss when the most recent transaction is corrupt but earlier ones are intact. The cost is losing the most recent few seconds of writes; for most scenarios this is acceptable. The -F option preserves earlier data while sacrificing recent uncommitted writes; it’s safe to try and often resolves import problems.

zdb: the ZFS debugger

For deeper recovery, the zdb command provides low-level access to ZFS pool internals. zdb can:

• Display pool structure and dataset hierarchy.
• Read individual blocks by their pool addresses.
• Walk the dataset tree to identify what’s recoverable.
• Extract data from pools that won’t import normally.

zdb is a power tool; it requires substantial ZFS knowledge to use effectively and can damage pools if misused. Recovery work with zdb is typically performed on read-only pool imports or on disk images.

Snapshots as recovery

ZFS snapshots are the first line of defense against user-error data loss. A snapshot taken before a destructive operation can be rolled back trivially:

• zfs rollback dataset@snapshot restores the dataset to the snapshot’s state.
• zfs clone dataset@snapshot newdata creates a writable copy without modifying the original.
• Files within snapshots are accessible via the .zfs/snapshot directory.

Best practice is automated snapshot rotation (Sanoid, syncoid for management); regular snapshots provide cheap, instant rollback for many user-error scenarios.

Send/receive for off-site backup

ZFS includes built-in replication via zfs send and zfs receive. Snapshots are sent as streams that can be received on remote pools, providing efficient incremental backup. Combined with snapshot rotation, send/receive supports comprehensive backup strategies. Off-site send/receive replicas are protection against scenarios where local pool redundancy isn’t sufficient; site-level disasters, ransomware, severe hardware failures.

Professional services for severe scenarios

Severe ZFS recovery scenarios require professional services with ZFS expertise. Specialized tools and disk-image-based reconstruction can sometimes recover data that consumer-level tools can’t reach. Cleanroom recovery applies for physical drive damage; ZFS adds another layer (pool reassembly) on top of standard physical recovery work. Recovery from severely damaged ZFS pools is genuinely specialized work; few recovery services have deep ZFS expertise.