Adding a new vdev vs. expanding an existing vdev in ZFS

There is a difference between adding a new vdev to a ZFS pool and expanding an existing vdev. This difference is important for data recovery. Klennet Recovery has a configuration setting explicitly addressing the expansion, so you should understand the difference.

The difference between expansion and addition

Original

Let's say we start with single 3-wide RAIDZ vdev, with some data written to it, as illustrated below:

raidz1-0 (original)
Disk 1	Disk 2	Disk 3
1	2	3
4	5	6
7	8	9

Original configuration - single 3-wide RAIDZ vdev with data.

Adding a vdev

Adding a new vdev to the pool is just that - you put in several blank disks and run something similar to
zpool add tank raidz1 /dev/sde /dev/sdf /dev/sdg
This creates a new vdev, raidz1-1. You should be careful that the fault tolerance levels match across your vdevs, because the pool is only as fault tolerant as its least fault tolerant vdev. Other than that, the procedure is rather straightforward.

Note two significant facts:

• Adding a vdev does not affect the width of the existing vdevs. The original 3-wide raidz1-0 stays 3-wide.
• The existing data is never moved, and the process is thus near-instantaneuos.

raidz1-0 (unchanged)				raidz1-1 (added)
Disk 1	Disk 2	Disk 3		Disk 1	Disk 2	Disk 3
1	2	3
4	5	6
7	8	9

The pool layout after a vdev is added.

Expanding a vdev

Vdev expansion is performed by adding a single disk to an existing vdev. You do it by running something similar to
zpool attach tank raidz1-0 /dev/sde
This changes the original 3-wide vdev into a 4-wide vdev, and the data is redistributed across all 4 disks. In a traditional RAID parlance, this process is called reshaping.

Note two differences:

• Expanding a vdev changes its width. The original 3-wide raidz1-0 becomes 4-wide.
• All the existing data is moved to match the new layout, and it takes a while (sometimes several days).

From the fault tolerance perspective, ZFS vdev expansion survives power loss. Once the power is back, the expansion continues from the point of interruption. The pool can be used while the expansion is in progress, including for writes, but performance is bad.

The theoretical "clean" result looks as follows:

raidz1-0 (expanded)
Disk 1	Disk 2	Disk 3	Disk 4
1	2	3
		4	5
6
7	8	9

Theoretical vdev layout after expansion.

Realistically, however, during the expansion, ZFS does not move free space. Also, it does not explicitly overwrite the space that becomes free as the data is shuffled around. So the reshaped vdev contains residual data in the original layout. On SSDs, I believe this space is TRIMmed, so there are no recoverable remnants. On HDDs, however, the original data is still intact until the free space is reused.

So on HDDs, the actual result looks more like this:

raidz1-0 (expanded)
Disk 1	Disk 2	Disk 3	Disk 4
1	2	3
		4	5
6	5	6
7	8	9
7	8	9

Actual expansion result, the leftover data is highlighted in gray.

Effect of the expansion on data recovery

Because of the leftover data, the expanded vdevs require special processing during recovery. While normal (never expanded) vdevs only have a single geometry (the number and the order of disks), expanded vdevs have as many geometries as there were expansions. A vdev that was expanded twice has three geometries (the current one, and two old ones) and the data can be read using any one of these.

This additional complexity requires additional computation:

• Determining additional geometries requires additional computation.
• On every read, the reader has to try all possible geometries. During normal operation, ZFS knows which geometry to use, as the information is stored in the disk labels. During recovery the disk labels are considered untrustworthy, so the reader has to try every one in turn, which consumes time on every read. While Klennet Recovery uses various tricks to improve performance here, there is still some penalty involved.

Klennet Recovery has a specific setting to adjust the limit of how many expansions to search for. For the details, refer to the corresponding manual page.

Filed under: ZFS.

Created Thursday, June 11, 2026