Undelete files on ZFS

What happens if you delete a file on ZFS? Well, customary with most other filesystems, the file disappears. But, can you get it back? The official recommended method is to get the file from a snapshot or a backup. What are the chances if neither are available? Let’s start down the rabbit hole of details.

ZFS and copy-on-write

When a file is deleted, the filesystem needs to update its various metadata in several different places. Not necessarily specific to ZFS, the filesystem typically needs to

  • mark the file record as empty,
  • update directory (or several directories) to remove the file name,
  • update the free space map.

If a crash happens during the update, the filesystem ends up in some partially updated state, which is not good. Copy-on-write (CoW) is the method of dealing with such inconsistencies. Instead of changing the records, ZFS creates a new set of updated records and writes this set to some new location on disk. Then, the root pointers are updated to point to the updated records.

ZFS copy-on-write diagram.

How copy-on-write works
Green shading is accessible data; green lines are old metadata references; red lines are new metadata references

For this trick to work, the old data and the new data cannot overlap. This comes in handy for undelete.

Philosophy of copy-on-write

Maybe it is easier to show the difference between traditional filesystem (like NTFS or EXT) and copy-on-write filesystem (ZFS or ReFS) by stating that there is no deletion in CoW filesystems.

A CoW filesystem like ZFS does not delete file. Instead, it creates a new version of the filesystem, and that new version of the filesystem does not contain the file. Then, a new version of the filesystem is written to the disk. Once the write is complete, new version of the filesystem becomes current, and the file ceases to exist.

To undelete a file we then need to find one of the previous versions of the filesystem which still contains the file. Obviously, the old versions are eventually overwritten, so there is no guarantee that we can find it.

Copy-on-write and undelete on ZFS

In a copy-on-write process, the old version of data remains unmodified, at least for a short while. Therefore, it can be found. There are no pointers to it, it is the deleted file after all, but it still can be found by scanning all disks in the pool. Depending on the filesystem usage patterns and on how full the pool is, the data can survive for quite a long time. It is not unusual to find thousands, or sometimes tens of thousands copies of old metadata. It will eventually be overwritten as the disk space is needed for something else, exactly as it happens with any other filesystem. Thus, the general rule of data recovery stands – the less data written to the filesystem, the better the chance of successful recovery. So, by searching for previous versions of metadata, Klennet ZFS Recovery can find

  • deleted files,
  • previous versions of files which were being edited (because the same copy-on-write rules apply to edits).

Success rate however depends on how much data was written to the pool after the file was deleted.

Created Saturday, January 5, 2019

Updated 20 September 2019

I have a low volume mailing list, for news and tips, which I send out once or twice a month.
Subscribe if you are interested.