Thin provisioning is a storage system design technique that allocates blocks of disk space as needed,
instead of allocating full volume size upfront.
Thin provisioning is widely used in modern storage systems. Its advantages are:
- Overcommit - it is possible to allocate more space than physically exists, then add physical storage over time as needed.
- Flexibility - it is possible to allocate several volumes, then redistribute the physical space between them as needed.
- Use of mixed-size disks.
There are some restrictions imposed by fault tolerance requirements,
but it is much better than a standard RAID requirement for all disks to be the same size.
The downside is that all of the advantages are achieved by splitting data into myriads of small blocks,
chaotically allocated all over the physical disks.
Things were simple before thin provisioning
Before thin provisioning, the allocation of data was regular by design.
Traditional RAIDs and partitions allocate data in large blocks sharing the same set of parameters.
In a traditional RAID you need to know disk order, RAID level, and a single starting address to describe the entire multi-terabyte array.
Every stripe follows the same layout rule, and there is no extra metadata as there is nothing to keep track of.
The conventional RAID reconstruction tools work thanks to that regularity.
They only need to determine a few parameters to reconstruct the entire array.
For arrays with few disks, even a brute-force search through all possible combinations is feasible.
ZFS dRAID is also a regular allocation scheme.
Even though the block map looks random
(see here), the allocation in fact applies regular fixed pattern to the entire disk pack.
The only addition to the parameter set is the unique pattern ID.
As there are only 255 possible patterns, the search space is still manageable.
Thin provisioning implementation
Thin provisioning, in contrast, allocates data in small blocks.
This allows the system to only allocate disk space as needed.
Every time the filesystem demands more space, the thin-provisioned volume allocates a new block.
Somewhere there is a metadata table (also called a block map, or allocation map, or a block mapping) that records where each logical block is physically located.
The same table also accounts for redundancy as needed.
The terminology and block sizes vary between different implementations.
- Linux
LVM
allocates extents, typically 4 MB per extent.
- Microsoft Storage Spaces allocates slabs, typically 256 MB.
- BTRFS allocates chunks and uses different chunk size for metadata (256 MB) and data (1 GB).
- ZFS AnyRAID (projected as of summer 2026) is expected to allocate tiles, 16 GB each.
Despite the differences, the underlying principle is the same - instead of one large, uniformly-described region,
the volume is built from many chaotically-allocated pieces.
Fragmentation of thinly provisioned pools
As blocks are allocated and freed over the life of the volume, the ordering of blocks in the pool becomes increasingly chaotic.
The fragmentation is an inevitable consequence of the allocation requirements.
Even a single file's data can end up scattered across the pool in an order that has nothing to do with the file's logical layout.
The block mapping metadata
Because of the chaotic allocation, the block map metadata (keeping track of where each block is on disks)
is the most important piece of data in a thin-provisioned volume.
The metadata is typically stored in some redundant manner, for reliability.
The metadata redundancy does not protect against the operator error.
If you delete enough files so that some blocks are deallocated,
the volume manager simultaneously updates all the redundant copies of the metadata.
There is now no trace of where the deallocated blocks were.
So, there's the problem.
- Reading regular provisioning requires a small set of parameters.
The number of parameters does not increase with the size of the disk pack.
- Reading thinly-provisioned volume requires a very large set of parameters.
In addition to the RAID parameters, you need a location for every block.
Furthermore, the number of parameters increases with the size of the disk pack, as the number of blocks increases.
Recovery
Easy - Block map is intact
If the block mapping metadata can be recovered in usable form,
the mapping tells you exactly where every block lives, and the recovery is trivial.
You figure out what block contains the address you need, you look up the physical location of that block, and you read it.
Still easy - Block map is intact, but disk map is lost
There are some intermediate cases, where the mapping is available, but it is not clear which disk is which.
This happens, for example, if the disks are referenced by GUIDs from GPT, and GPT is overwritten.
These cases are also easy. There is a limited number of possible permutations, and there is enough input
to verify which permutation is correct.
Intermediate difficulty
There is no intermediate difficulty. The recovery is either simple (block map available), or it is complicated (block map lost).
Difficult - loss of block map
Significant loss of block mapping metadata is mostly caused by an operator error.
Typical example goes like that: the original pool is deleted, because
- of a mix-up - the administrator mis-identifies the pool; or
- someone forgot to copy some important data off the pool;
... and then ...
- the disks were repurposed and significantly overwritten before the mishap is discovered; or
- someone attempted to re-create the pool on the same disks with the same parameters, resulting in the mapping tables being overwritten with blanks.
So now the block map is gone, none of the traditional techniques work.
The search space is far too large to apply any of these.
Entropy-based techniques do not work, filesystem signatures are not dense enough to be useful, and neither are file header-footer signatures.
The only workable approach is to use the filesystem layer checksums to reconstruct the mapping.
This approach:
- Does not work if the filesystem checksums are
- not available at all (as is typical in EXT-over-LVM setup); or
- only available for filesystem metadata, but not for file content (as in some BTRFS setups).
This is because there is not enough density of checksums to have at least one reference per mapping block.
- Requires massive amounts of computation if the filesystem data checksums are available.
More practical complications
Snapshots specifically, and versioning of files in general, often produces different data blocks with similar content.
These blocks are difficult to discriminate when reconstructing a mapping.
Technically, this requires the recovery software to keep a list of candidates for every block,
instead of a single definite mapping, and be ready to re-evaluate the mapping on the fly as read requests come in.
That slows everything down even more.
Conclusion
In fine, thin provisioning is very convenient when it works and very expensive to recover when it fails.
Even if all of your volumes are thickly provisioned, as long as the underlying storage system is
capable of thin provisioning, you still have the same problem.
If the block mapping is completely lost,
- ZFS AnyRAID will be recoverable (when it is eventually released);
- BTRFS RAID is recoverable if the checksums are enabled for file data;
- EXT or XFS over LVM is not recoverable;
- NTFS over MS Storage Spaces is not recoverable;
- ReFS over MS Storage Spaces should be recoverable if checksums for file data (called "integrity streams") are enabled, but I am not aware of any implementation.
Filed under:
NAS,
RAID,
ZFS.
Created
This text is licensed under Creative Commons Attribution 4.0 License