ZFS Performance Tuning: Recordsize, atime, Compression

ZFS out of the box is a good generalist. That means: decent for every workload, optimal for none. If you build a pool for a specific application — VM storage, backup target, file server, database — a handful of dataset properties can deliver noticeably more performance, less space usage and longer SSD life.

This article covers the most important levers in the right order. No invented benchmarks — where tradeoffs exist, we describe them qualitatively. Your own lab or staging-pool tests are irreplaceable.

What cannot be changed after pool setup

Before we get to tunable properties, a reminder of the irrevocable decisions taken at pool create time:

• ashift — sector-size alignment, set when the VDEV is created. Recommendation: ashift=12 (4K sectors) for modern SSDs and enterprise HDDs. For NVMe and some enterprise SSDs, ashift=13 (8K) makes sense. Wrong ashift = write amplification, not changeable later.
• VDEV topology — mirror, RAIDZ1/2/3, dRAID. A RAIDZ configuration cannot be converted into a mirror. RAIDZ expansion (from OpenZFS 2.3) only extends, it does not change the topology. See ZFS RAID levels explained.
• Pool-level encryption properties — not changeable later.

If you get sizing wrong here, no property tuning will save you later. The rest of this article covers what can be changed online and without data loss.

recordsize — the most important lever

recordsize is the maximum logical block size with which ZFS writes data to a dataset. Default: 128K. That is okay for file servers with mixed file sizes, but usually suboptimal for VMs or databases.

Rule of thumb: recordsize should match the typical IO size of the workload.

Workload	Recommended recordsize	Reason
File server (mixed files)	128K (default)	Generalist, good balance
Large media files (video, backups, images)	1M	Less metadata overhead, higher streaming throughput
VM images (Proxmox, qcow2, raw)	64K or 128K	App IO size typically 4–16K, but multi-block operations benefit
Databases (PostgreSQL, MySQL InnoDB)	16K	Match database page size — typically 8K (Postgres) or 16K (InnoDB)
Microsoft SQL Server (data files)	64K	SQL Server writes in 64K extents
Backup targets (Veeam, PBS chunks, Restic)	1M	Long sequential writes, dedup-oriented storage

Important: recordsize applies only to newly written blocks from the moment it is set. Existing data keeps its previous block size until rewritten. To convert a dataset, copy the data fresh (zfs send | zfs receive into a new dataset) or wait until normal writes refresh the dataset contents.

# recordsize for a VM dataset
zfs set recordsize=64K tank/vms

# recordsize for a database dataset (Postgres)
zfs set recordsize=16K tank/postgres

# recordsize for a backup target
zfs set recordsize=1M tank/backups

atime=off — quiet killer removed

atime (access time) is the property that updates a file’s read timestamp on every read — a write per read. That was useful in 1990s Unix workloads. Today it is almost always harmful overhead:

• Every read triggers a metadata write
• SSDs take unnecessary load (write amplification)
• File servers, backup targets, VM storage do not benefit from atime

zfs set atime=off tank

Set at the pool root; inherited by all datasets. Exception: if applications truly depend on atime (mail server in some configurations, some quota tools). In that case, selectively keep atime=on on the datasets holding those applications.

Modern alternative: relatime=on updates atime only when the previous update is older than 24h or before mtime/ctime. Reduces most unnecessary writes without losing atime entirely.

# If atime is needed but should be gentle
zfs set atime=on tank/mailserver
zfs set relatime=on tank/mailserver

Compression — lz4, zstd-3, zstd-19

Compression is a clear win in almost every scenario — less space and often even faster IO because fewer bytes flow to/from disk. The question is only: which algorithm.

Algorithm	CPU cost	Compression ratio (typical)	Latency	Use case
lz4	Very low	1.2:1 – 2.0:1	Very low	Default for VMs, databases, anything latency-sensitive
zstd-3	Low–medium	1.5:1 – 2.5:1	Low	Generalist with better ratio than lz4, barely any CPU overhead on modern EPYC/Xeon
zstd-9	Medium–high	1.8:1 – 3.0:1	Noticeably higher	File servers with read-heavy workload
zstd-19	Very high	2.0:1 – 4.0:1	High	Archive and cold storage when writes are rare and CPU is available
gzip-6	High	similar to zstd-9	High	Legacy — today rarely advantageous over zstd
off	None	1.0:1	Lowest	Only for already compressed data (e.g. Veeam repo with own compression)

Tradeoffs in practice:

• VMs and databases: zstd-3 is often the better choice than lz4 — barely measurable latency increase on modern CPUs, markedly better compression.
• All-flash pools with high IO load: lz4 stays safe; zstd only after lab testing.
• Backup targets: if data arrives already deduplicated and compressed (PBS, Veeam Hardened Repo): compression=off or at most lz4. More gains nothing and costs CPU.
• Cold storage / archive: zstd-19 is worth it. Writes are rare, reads are rare, every saved percent counts.

# Default for most datasets
zfs set compression=zstd-3 tank

# All-flash VM pool with latency priority
zfs set compression=lz4 tank/vms

# Archive
zfs set compression=zstd-19 tank/archive

# Backup target receiving already compressed data
zfs set compression=off tank/veeam

To check the effect: zfs get compressratio dataset shows the actually achieved ratio on existing data.

More on this in ZFS compression in practice.

special_small_blocks — metadata and small files on NVMe

If the pool has a special VDEV (separate mirror VDEV from NVMe), special_small_blocks can ensure that not only metadata but also small data blocks land there. That accelerates file-server workloads with many small files dramatically — hot files live on NVMe, only large files on the slower HDDs.

# Special VDEV must already be attached (zpool add tank special mirror ...)
zfs set special_small_blocks=64K tank/fileserver

Values between 32K and 128K are sensible in practice. Important: the special VDEV must be redundant (at minimum mirror) — if it fails, the entire pool is dead. More in the Proxmox ZFS tuning article.

primarycache / secondarycache

These two properties control what is held in ARC (RAM) and L2ARC (NVMe cache):

Value	Meaning
all (default)	Cache data + metadata
metadata	Cache metadata only
none	Cache nothing

Sensible applications:

• Sequential backup reads: primarycache=metadata — backup data is read once, no value in ARC. RAM stays free for other datasets.
• Databases with their own cache (e.g. PostgreSQL shared_buffers): primarycache=metadata — the DB caches more effectively itself, ARC only for metadata.
• Hot read workloads (file-server profiles): keep primarycache=all.

# Backup dataset: ARC for metadata only
zfs set primarycache=metadata tank/backups
zfs set secondarycache=metadata tank/backups

sync — use with care

sync controls how ZFS handles synchronous writes:

• standard (default): Write waits for SLOG/ZIL confirmation — safe, consistent
• always: Forces sync also for asynchronous writes — very safe, very slow
• disabled: Ignores sync requests — fastest write, but data from the last seconds is lost on power outage

sync=disabled is a trap. Tempting for benchmark numbers but risky in production:

• Safe for: temporary scratch datasets, build artefacts, caches that can always be regenerated
• Risky for: databases, VM images, anything that needs consistency

# Only for provably uncritical scratch data
zfs set sync=disabled tank/scratch

If you need sync performance, add a dedicated SLOG device (Intel Optane, high-end NVMe with power-loss protection) instead — that accelerates sync without losing consistency.

logbias — throughput vs. latency

logbias=throughput changes SLOG behaviour: sync writes go directly into the pool instead of via the SLOG. Useful for database-style workloads with very large sync writes, where the SLOG would otherwise become the bottleneck.

zfs set logbias=throughput tank/large-db

Default logbias=latency is right for most applications.

redundant_metadata

redundant_metadata=most reduces the number of metadata copies from 2 to 1 for most blocks (critical metadata stays duplicated). Saves space, but reduces recoverability for corrupted metadata blocks. Rarely recommended, except in genuinely space-tight, safe environments.

One-time setup checklist for new pools

A pragmatic default setup block for a fresh pool:

# Pool-wide
zfs set atime=off tank
zfs set compression=zstd-3 tank
zfs set xattr=sa tank          # Extended attributes inline, less overhead
zfs set acltype=posixacl tank  # If POSIX ACLs are needed

# VM dataset
zfs create -o recordsize=64K -o compression=lz4 tank/vms

# Database dataset
zfs create -o recordsize=16K -o compression=lz4 -o logbias=latency tank/postgres

# Backup target
zfs create -o recordsize=1M -o compression=off -o primarycache=metadata tank/backups

# General file dataset
zfs create -o recordsize=128K tank/files

These are not universal truths, but experience-based starting values — validate and adjust against real production workloads.

What tuning does not replace

Properties tune what is there. They do not fix:

• Insufficient RAM for ARC. Rule of thumb: 1 GB RAM per 1 TB pool, more with dedup or many small files. Skimp here, you see it in every workload.
• Wrong VDEV topology. A single RAIDZ2 of 12 large HDDs for VM workload will disappoint even with perfect tuning — VMs need mirror VDEVs or all-flash.
• Fragmented pool > 80 % usage. ZFS slows noticeably above ~80 % full. Leave more space or grow.
• Wrong storage stack. Spinning HDDs for high-IOPS workloads stay slow no matter how perfect the tuning.

DATAZONE recommendation

Tuning is not an end in itself — it is the last layer above sensibly sized hardware and the right pool topology. Order:

1. Hardware matched to the application (TrueNAS model selection or Proxmox architecture planning)
2. VDEV topology right at pool create time (ZFS RAID levels)
3. Default properties (atime=off, compression=zstd-3, xattr=sa)
4. Workload-specific datasets (recordsize, compression, sync, primarycache per dataset)
5. Measure. zpool iostat, arcstat, application metrics (see Grafana stack, TrueNAS SMART monitoring)

We help with pool sizing and tuning of existing setups. Experience: untuned SMB pools often yield 20–50 % performance gains without new hardware — provided the sizing is fundamentally sound.

Sources and further reading

• OpenZFS — performance tuning wiki
• ZFS RAID levels explained
• ZFS compression in practice
• Proxmox ZFS tuning: ARC and L2ARC
• TrueNAS performance optimisation
• TrueNAS configurator
• Plan Proxmox architecture