Klennet Storage Software

Recovery of fragmneted MP4, QuickTime, AVI, MPEG-2, and DAV video in Klennet Carver

Recovery of fragmented video files is indispensable capability to have in file carving. In modern mobile phones, video cameras, and dash cameras often more than the half of video files are fragmented, defying traditional data recovery and requiring specialized carving algorithms to recover.

Klennet Carver supports the following video formats

  • MP4, MOV, and 3GP, which are all variations of the same format,
  • AVI,
  • MPEG-2 (AVCHD),
  • DAV (also known as DHAV, video format used by Dahua video recorders)

MP4 and QuickTime (MOV)

MP4 and QuickTime (MOV) videos, by far the most common video file types in use, share a common format, consisting of what is called atoms. Atom is a part of file, defined by atom name (identifying type of data the atom contains) and length. Atom may contain payload data, other atoms, or a combination of both. We will only be concerned with top-level atoms, of which three are important.

  1. ftyp atom, which defines the “file type”, a variation of the format, would it be QuickTime movie, or MP4 video, or MP4 audio, and what is the exact version of it.
  2. moov atom, consisting in turn of multiple atoms, contains various decoder settings and metadata and most importantly, contains a frame map for the mdat atom. Normally, a video decoder cannot seek, or move, to any arbitrary position in the video stream and restart or continue video decoding. The decoder can only seek to a set of defined points, called key frames. So, to allow player application to seek inside the video file, the moov atom contains among other things a list defining which key frame offset inside the mdat atom of the file corresponds to which time point in a video.
  3. mdat atom, containing a video stream. The video stream consists of frames going one after other without (at least in general case) any distinguishing marks between them.

ftyp is always first atom in the file, followed by moov and mdat in any order.

The file is often split into several fragments, not just two fragments, but often five or more. Parts of the file may be overwritten, even if the file is not fragmented, and some fragments may be missing in a fragmented file.

Klennet Carver reassembles these fragments to produce a viewable video whenever possible. Overwritten parts can’t be recovered, and this causes loss of video up to the closest next keyframe. At the keyframe, video decoder synchronizes with the stream again and playback continues normally. Sometimes, Klennet Carver fails to determine fragment boundaries properly. While this causes distortion of a recovered video for a brief period of time, the effect is usually negligible.

If ftyp atom is missing, or if fragmentation is bad, it may be impossible to determine if the order of atoms is ftyp - moov - mdat or ftyp - mdat - moov. If these cases, ftyp atom is rebuilt and the file is regenerated assuming ftyp - mdat- moov atom order.

Missing (overwritten) parts of the file are either filled with zeros or by the best guess of where on this disk they may have been. The total size of all recovered files may well be larger then the capacity of the source media. This happens when there are many video files on the media but only small part is available for each file. Each file is padded with zeros in places where content is missing and this padding may account for significant amount of data.

There are many different video encoding algorithms, called codecs (short for coder-decoder). The recovery implemented in Klennet Carver requires some understanding of the codec used to encode the particular video. If your videos are not recovered, make sure to put forth a support request so that I can look into the case and make any adjustments required to accommodate that specific codec your camera is using.

AVI

AVI video format is similar to MP4, consisting of a set of frames and a frame table. The parts are just called differently, and require somewhat different identification and parsing, but as far as recovery is concerned everything is just about the same. The complexity is about the same, run times are about the same, and last not least good recoveries can be achieved.

Header stealing

In some cases, the header of the video file may be overwritten. Then, there are more video frames than headers on the media . The most important information in the header are codec (decoder) parameters. Without these parameters, the player cannot decode video frames properly, and the resulting video cannot be played. Header stealing is the technique of using headers of one file for frames of some other file. This makes it possible to decode lost parts of the video using parameters from another video as a fallback. There are two drawbacks though:

  1. Header stealing only works well with memory cards. Memory cards typically contain videos made with the same settings, and therefore the codec parameters are identical or at least interchangeable. On a hard drive, or when there are videos taken with different camera settings, it is possible to steal wrong header, thus producing a broken video.
  2. AVI files may have codec parameters stored along with the first frame of the video. Therefore, header stealing also steals the first frame from whatever happens to be the donor video file. The resulting file then displays that stolen frame first when you start playing it, and this may continue for some time until decoder encounters the next keyframe. You should keep this in mind if you are using recovered video in forensic context.

MPEG-2 video

MPEG-2 video uses file extensions .MTS and .M2TS. It may also be called AVCHD video in your digital camera.

MPEG-2 is different from MP4 and AVI. MPEG-2 per se is a stream of fixed-size frames, each frame containing a bit of video, sound, or technical data. Unlike MP4 and AVI, MPEG-2 has no file-wide header. Therefore, MPEG-2 video is just a sequence of frames with no discernable starting or ending points. Klennet Carver will recover all frame sequences longer than specific configurable length, putting each one in a separate file. As MPEG-2 files have no header, header stealing does not apply here.

DAV Dahua video files

DAV is a relatively obscure video format used by Dahua security video recorders. It has three interleaved streams for video, audio, and camera settings like focus. DAV format is something between AVI and MPEG-2, but DAV files do not have indexes. Due to specifics of the format, the recovery quality is fairly good.

Klennet Carver does not support preview for Dahua videos, because no Windows codecs are available for it. You need to copy the files and use Dahua proprietary viewer software to view the files.

DAV files have no header to define parameters for the entire file, and therefore header stealing setting does not apply to DAV files.

Performance considerations

Recovery speed for video files are somewhat faster than for image files. As an example, recovery of about 100 fragmented videos from a memory card takes about three hours with an eight-core i7 CPU.

Media read speed is only significant while initial point is computed. After initial point is done, no further disk reads are required.

I have a low volume mailing list, for news and tips, which I send out once or twice a month.
Subscribe if you are interested.