NTFS Compression and File Recovery/Carving:

The NTFS file system, introduced with Windows NT 3.1, has been around for over 20 years. It is the file system used by all modern Windows operating systems, capturing over 85% market share in desktop and laptop computers, and over 25% market share in servers.

Despite this broad installed base, most users do not take advantage of the file system’s ability to compress files, folders and even entire volumes on the fly. Historically, HDDs have been large enough that there was no great demand for the space savings, not to mention the inherent performance degradation associated with compression.

In recent years, however, moderate interest has been observed in combining NTFS compression with solid state drives (SSDs). The perceived benefits include squeezing more capacity out of these much smaller drives as well as reducing the number of write cycles to extend their endurance lifetimes. [Note: These perceptions are not the opinions or recommendations of the author.] In addition, SSDs are so much faster than HDDs that the fragmentation and performance impacts associated with file system compression can be almost imperceptible.

With all of that said, file system compression presents a unique forensics challenge in the realm of data recovery and file carving. Modern tools do not inherently understand the difference between data that has been stored directly to disk and that which has been compressed first. While correct file signatures can often be identified within a compressed image, any data extracted will be compressed, creating a “corrupted” and useless output file.

This module describes in detail the organization and data structures used by NTFS to store compressed file data. The intent is to show that with a little extra processing, a compressed file system does not have to present a major impediment to a forensic investigation.

Sample Challenge:

Review the compressed .bmp file data shown on slides 9 and 16 in the presentation. Manually decompress all of the data associated with the first three tag bytes (0x40, 0x15 and 0x71). This data resides in offsets 0x03 through 0x24. The color coded view on slide 16 may be helpful in organizing the data, otherwise the raw data is provided below for the adventurous.

Verify that once decompressed, the 35 bytes above represent the same data stream found in the uncompressed .bmp data on slide 9 between offsets 0x00 and 0x36 (54 bytes).

Hint:

While working through the compressed byte stream, be cognizant of your current offset in the cluster, and what effect that has on tuple encodings

For an additional challenge, continue to decompress the data associated with the 4th tag byte 0xFF, and verify that your data stream matches that in the uncompressed .bmp data on slide 9.

A text representation of the compressed challenge data is provided below for copy/paste purposes. In addition, binary images ‘bmp_challenge.bin’ and ‘uncompressed_bmp.bin” are provided for use with a hex editor.

Challenge data:

52 B1 40 42 4D F6 D4 01 00 01 00 36 15 00 40 28
00 30 C8 04 18 01 00 18 71 00 48 00 00 C0 04 7C
02 30 04 10 FF FF FF 03 3F 80 3F 04 3F 04 3F 04
3F 04 3F 04 3F 04 FF 3F 04 3F 04 3F 04 3F 04 3F
04 3F 04 3F 04 3F 04 FF 3F 04 3F 04 3F 04 3F 04
3F 04 3F 04 3F 04 3F 04 FF 3F 04 3F 04 3F 04 3F
04 3F 04 3F 04 1F 02 1F 01 FF 1F 01 1F 01 1F 01
1F 01 1F 01 1F 01 1F 01 1F 01 FF 1F 01 1F 01 1F
01 1F 01 1F 01 1F 01 1F 01 1F 01 FF 1F 01 1F 01
1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 FF 1F 01 1F
01 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 FF 1F 01
1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 FF 1F
01 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 FF
1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01
FF 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 1F
01 FF 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01
1F 01 FF 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01 1F
01 1F 01 FF 1F 01 1F 01 1F 01 1F 01 1F 01 1F 01
1F 01 1F 01 FF 1F 01 1F 01 1F 01 1F 01 1F 01 1F
01 1F 01 1F 01 FF 1F 01 1F 01 1F 01 1F 01 1F 01
1F 01 1F 01 1F 01 7F 1F 01 1F 01 1F 01 1F 01 1F
01 1F 01 1A 01

Binary images:
See attachments below. 

 

Uncompressed_bmp.bin(117.24 KB) marcbudofsky, Jun 25 2014, 8:47 PM
bmp_challenge.bin(341 bytes) marcbudofsky, Jun 25 2014, 8:47 PM

Additional Resources:

The best publically available specifications for the NTFS file system come from the creators of the Linux NTFS driver. At time of writing, the manual can be obtained from the following two addresses:

http://download2.polytechnic.edu.na/pub4/sourceforge/n/nt/ntfsofuefi/NTFS%20Reference%20Documents/ntfsdoc.pdf

http://ftp.kolibrios.org/users/Asper/docs/NTFS/ntfsdoc.html

This well written manual covers every aspect of the file system, with an intended audience of system programmers.

For a programmer interested in traversing the NTFS file system directly, using metadata structures rather than carving raw byte data, the following manual entitled “NTFS Forensics” is a particularly good read:

http://grayscale-research.org/new/pdfs/NTFS%20forensics.pdf

Lastly, for a thorough byte-by-byte deconstruction of important data structures such as the master boot record (MBR), partition table, master file table (MFT), BIOS parameter block (BPB), as well as how these structures have evolved with each version of Windows, visit the Starman’s Realm:

http://thestarman.pcministry.com/asm/mbr/index.html


You must Sign-In to post a comment.