Klennet Storage Software

Undelete files on ZFS

What happens if you delete a file on ZFS? Well, customary with most other filesystems, the file disappears. But, can you get it back? The official recommended method is to get the file from a snapshot or a backup. What are the chances if neither are available? Let’s start down the rabbit hole of details.

ZFS and copy-on-write

When a file is deleted, the filesystem needs to update its various metadata in several different places. Not necessarily specific to ZFS, the filesystem typically needs to

  • mark the file record as empty,
  • update directory (or several directories) to remove the file name,
  • update the free space map.

If a crash happens during the update, the filesystem ends up in some partially updated state, which is not good. Copy-on-write (CoW) is the method of dealing with such inconsistencies. Instead of changing the records, ZFS creates a new set of updated records and writes this set to some new location on disk. Then, the root pointers are updated to point to the updated records.

ZFS copy-on-write diagram.

How copy-on-write works
Green shading is accessible data; green lines are old metadata references; red lines are new metadata references

For this trick to work, the old data and the new data cannot overlap. This comes in handy for undelete.

Copy-on-write and undelete on ZFS

In a copy-on-write process, the old version of data remains unmodified, at least for a short while. Therefore, it can be found. There are no pointers to it, it is the deleted file after all, but it still can be found by scanning all disks in the pool. Depending on the filesystem usage patterns and on how full the pool is, the data can survive for quite a long time. It is not unusual to find thousands, or sometimes tens of thousands copies of old metadata. It will eventually be overwritten as the disk space is needed for something else, exactly as it happens with any other filesystem. Thus, the general rule of data recovery stands – the less data written to the filesystem, the better the chance of successful recovery. So, by searching for previous versions of metadata, Klennet ZFS Recovery can find

  • deleted files,
  • previous versions of files which were being edited (because the same copy-on-write rules apply to edits).

Success rate however depends on how much data was written to the pool after the file was deleted.

Created Saturday, January 5, 2019

I have a low volume mailing list, for news and tips, which I send out once or twice a month.
Subscribe if you are interested.