Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Windows NTFS filesystem internals

NTFS is the preferred file system on Windows NT family, including Windows XP, Windows 2000, Windows 2003 and Windows NT. It  address many of the limitations of the earlier FAT16 and FAT32 file systems. The most important features are as follows:

• Data recoverability. NTFS limits the possibility of data corruption by organizing I/O operations by transactions. Transactions are atomic, which means that either the entire I/O operation must complete or none of it can complete. If anything interrupts the transaction in-progress, such as loss of power to the computer or a cancellation of the I/O operation, NTFS does everything possible to guarantee that any changes made to the file system as part of the I/O operation are undone, or rolled back, returning the file system to its condition before the I/O operation began.

  Also, NTFS is a fully recoverable file system. It is designed to restore consistency to a disk after a CPU failure, system crash, or I/O error. NTFS allows the operating system to recover without your having to use disk-checking utilities. However, NTFS provides some disk utilities in case recovery fails or corruption occurs outside the control of the file system.
   

• Storage fault tolerance. Data-redundant storage methods can be used with NTFS to ensure that if data is corrupted on one physical disk, an intact copy can be retrieved from the disk mirror. NTFS always uses data redundancy to protect internal data structures containing metadata vital to the integrity of the file system.
   
• Data security. NTFS implements files and directories as securable objects according to the Windows object security architecture. Access to file and directory objects in NTFS can be restricted to specific users and groups under this architecture. For more information on this, see File Security and Access Rights. Data security features for files and directories are not included in the FAT file systems.

Other advanced features provided by NTFS are the following:

• Multiple data streams. As mentioned in the Files section, NTFS files can consist of more than one stream. The additional streams may contain any kind of data, although typically it is data describing the file, or metadata. For more information about multiple data streams in NTFS files, see the Files section.
   
• Unicode names. Unicode is the standard character set used in NTFS and replaces the older single-byte ASCII character set. Every character used in every major natural language is represented by a unique double-byte number in the Unicode set.

  For more information about Unicode, see About Unicode and Character Sets.
   

• Improved file attribute indexing. NTFS includes the ability to index file attributes as a means of locating and sorting multiple files that share similar data quickly. You can index file names in FAT32 and FAT16 file systems, but not attributes. Also, the file system does not have the functionality to sort the indexed FAT32 and FAT16 file names.
• Dynamic bad-cluster remapping. When a read operation on an NTFS volume that is not fault tolerant encounters corrupted data on a cluster of sectors, each sector in the cluster is flagged as bad, and subsequent attempts to perform read operations on that sector will result in the error being returned. In the same scenario on FAT file systems, the file system itself does not flag bad sectors—the user must run the Chkdsk.exe utility to do this.

  When this scenario occurs on fault-tolerant NTFS volumes, the file system will do the following for each bad sector it encounters:
   

  1. Recover the uncorrupted data from a secondary source on the volume.
  2. Locate a good sector and write the recovered data to it.
  3. Remap the bad sector to the new good sector so that all subsequent attempts to perform I/O operations on the bad sector will be automatically redirected to the new one.
      
• Hard links and junctions. Hard links and junctions are two ways that storage objects can be linked under NTFS. For more information on both, see Hard Links and Junctions.
   
• Compression and sparse file support. NTFS volumes support file compression on an individual file basis. The file compression algorithm used by NTFS is Lempel-Ziv compression. This is a lossless compression algorithm, which means that no data is lost when compressing and decompressing the file, as opposed to lossy compression algorithms such as JPEG, where some data is lost each time data compression and decompression occur.

  For more information on NTFS file compression, see File Compression and Decompression.

  A file in which most of the data is zeros is called a sparse file. NTFS implements a form of file compression specifically for sparse files in which only nonzero data is written to the file and the file system provides the correct amount of zero data to applications at need. For more information on sparse file support, see Sparse Files.
   

• Change journals. NTFS creates and maintains change journals for each volume, to track all changes made to it. For more information on change journals, see Change Journals.
   
• Distributed link tracking. The Windows shell allows a user to create files on their desktop that link to applications that reside in another location in the volume. The Start menu, which the user can configure, contains many instances of this kind of link. Also, the object linking and embedding, or OLE, technology allows applications to embed links to external files within the files they create and maintain. The components of the Office 2000 suite—Word, PowerPoint®, and Excel—are examples of applications that use OLE technology.

  A problem arises in the previous examples when the file being linked to (the link source) is moved, making it inaccessible through the link—also referred to as the link client. Distributed link tracking was first introduced in the version of NTFS shipped with Windows 2000 to enable client applications to track link sources that have been moved. As a result, applications and users that create links do not have to maintain the linkage themselves when the link source is moved.

  For more information on distributed link tracking, see Distributed Link Tracking and Object Identifiers.
   

• Encryption. NTFS provides the Encrypted File System, or EFS, for cryptographic protection of files and directories. For more information on EFS, see File Encryption. For more information on cryptography in general, see Cryptography Reference.
   
• POSIX support. The following POSIX functionality was introduced in Windows 2000:
  1. Files can be accessed in NTFS file systems according to POSIX naming conventions. POSIX conventions allow file names that have trailing spaces, file names that have trailing periods (.), and file names that are identical except for the case of the characters.
  2. Traversal permissions, where the security attributes of each parent directory in the path of a file or directory is used in determining whether a specific user has access to it.
  3. "File change time" time stamps.
  4. POSIX-style hard links. For a code example that shows how to back up and restore these type of links, see Backing Up and Restoring POSIX File Links.
      
• Defragmentation API. A file is stored on a disk drive and other storage media in one or more clusters. Clusters are the atomic unit of data allocation, made up of one or more sectors. Sectors are physical storage units.

  As a file is written to the disk, the file may not be written in contiguous clusters. Noncontiguous clusters slow down the process of reading and writing the file. The farther apart on the disk the noncontiguous clusters are, the slower the process because of the increased time it takes to move the hard drive's read/write head. A file with noncontiguous clusters is said to be fragmented. To optimize files for fast access, a volume may be defragmented.

  Defragmentation is the process of moving a file's clusters on the disk to make them contiguous. NTFS does not perform defragmentation, but with version 5.0 it provides the ability for applications to perform defragmentation by calling an API. This API consists of functions that allow applications to obtain a map of the clusters that are in use and clusters that are not being used, obtain a map of how a file is using its clusters, and move a file.

  For more information on defragmentation, see Defragmenting Files.

• Reparse points. Under NTFS, a file or directory can contain a reparse point, which is a collection of user-defined data. For more information on reparse points, see Reparse Points.
   
• Directories as volume mount points. Volume mount points are directories on a volume that an application can use to "mount" a different volume, that is, to set it up for use at the location a user specifies. In other words, you can use a volume mount point as a gateway to a volume. When a volume is mounted at a volume mount point, users and applications can see the mounted volume by the path of the volume mount point or a drive letter. For example, with a volume mount point set, the user might see drive D as "C:\mnt\Ddrive" as well as "D:".

  Using volume mount points, you can unify into one logical file system disparate file systems such as NTFS, a 16-bit FAT file system, an ISO-9660 file system on a CD-ROM drive, and so on. Neither users nor applications need information about the volume on which a specific file resides. All the information they need to locate a specified file is a complete path. Volumes can be rearranged, substituted, or subdivided into many volumes without users or applications needing to change settings.

  For more information on volume mount points, see Volume Mount Points.

Recommended Links

Services for UNIX - Interoperability

GID on NTFS File System

Can you set group on a file or folder on NTFS file system? - No, ugh... Yes.

This question puzzled me for a long time but since it never really made it to my top priorities, I didn't look up for information on this. I thought of exploring this area more while I was researching something about NFS server.

I started with looking for the utilities (obviously, Windows-based ones) which can set this information for me. My search ended quickly with the chown.exe and chgrp.exe which you can install with Interix/SUA Base Utilities.

Using Process Monitor (replacement of Filemon and Regmon utilities) revealed that group information gets stored on the file system with an IRP_MJ_SET_SECURITY request -

The other interesting fact is this request originating from the POSIX subsystem (psxss.exe) which makes sense because chown.exe and chgrp.exe utilities are POSIX subsystem utilities.

This KB Article says -

In the Windows NT and Windows 2000 NTFS file system, each file also has an owner and a primary group. The primary group of a file is not used by the Win32 subsystem, but is present for programs that make use of the POSIX subsystem. When a file is created, the user who created the file becomes its owner and that user's primary group becomes the file's primary group. Access Control Entries (ACEs) are then added to the DACLs to assign permissions.

That makes it clear that none of other utilities I tried to use, could set this information because they were basically Win32 binaries and Win32 subsystem does not, in anyway, uses this information.

So the best practice would be to set correct primary groups for your users and then use Interix/SUA chown.exe and chgrp.exe utilities to manage them the way you want them to be seen by your UNIX clients.

Additional Note: The ls.exe and chmod.exe are other utilities which can help you do things the UNIX way.

NTFS5 vs. FAT32

Win2K File-System Facts
Compatibility with legacy file systems is an important aspect of OS upgrades. Win2K supports several file systems, and its support is essentially a superset of NT 4.0's support. (For more information about file systems, see "File-System Resources," page 104.) Win2K still supports FAT12, FAT16, and the CD-ROM File System (CDFS), which is the International Organization for Standardization (ISO)-9660-compliant NT file system you use to read CD-ROMs. However, Microsoft doesn't plan to add features or upgrade these systems, except for a possible future addition of support for the UNIX-centric Rock Ridge format in CDFS. NTFS, which is the premier Win2K file system, continues to exist but sports a new file-system structure and new capabilities. Microsoft also included in Win2K a Universal Disk Format (UDF) file system that the company introduced in Windows 98. Currently, Win2K's UDF version, which complies with the ISO-13346 standard, mainly supports DVD-ROM media. However, Microsoft plans to eventually replace the CD-ROM-centric CDFS specification with UDF. Win2K also supports a slightly more recent version of UDF (i.e., UDF 1.50) than Win98 supports (i.e., UDF 1.02).

NTFS.com NTFS File System General Information. Data Recovery

Exploring NTFS On-disk Structures

NTFS FAQ (en)

Microsoft/NTFS overview

NTFS Read-only NTFS support.

Systems Internals -- very useful site with a lot of utilities with source code. Makers of NTFSDOS -- NTFS driver for DOS and other interesting tools for NT recovery

• NTFSInfo v1.0
• NTFSDOS Tools v2.0
• NTFSDOS v2.0
• NTLocksmith v1.0
• Bluescreen of Death Screen saver  -- neat trick

File System Usage in Windows NT 4.0 - by Wemer Vogels - 2.15M .PDF

This paper discusses real world traces of filesystem usage. It shows how certain real world behaviors are important to optimize for and shows how NT succeeds (partially) at providing good filesystem performance.

PartitionMagic - Product Information -- nice utility to convert NTFS into something else

Tutorials

[PPT] Windows Kernel Internals NTFS

ntfsdoc - What is ntfsdoc

NTFS - Wikipedia, the free encyclopedia

Windows NT File System Internals: A Developer's Guide: Chapter 4. ...

Inside NTFS

NTFS streams

NTFS File System Internals - DECUS San Diego

Inside Win2K NTFS, Part 2

NTFS.com NTFS Format. Master File Table (MFT). General Information ...

The Delphi Magazine Issue 98 Sample Article/NTFS Compression And Sparse Filesby Marcel van Brakel This article first appeared in The Delphi Magazine Issue 98 (October 2003)

How could space not be infinite? Is there a sign somewhere
saying ‘Space Ends Here – Mind the Gap’?      (Max Tegmark)

Hard disks keep getting larger every year, but after a while even the largest hard disks just don’t cut it any more. My first PC had a 425Mb hard disk, and I was sure I’d never run out of space. Two years later I found myself looking for a second disk. My second PC had a 4Gb hard disk, and this time I just knew I’d never run out of disk space. Two years later I found myself looking for a second disk. My current PC has a 40Gb hard disk, and now, a year after purchase, it’s 75% full. At most companies I’ve worked for the exact same thing happens. When purchased the storage devices seem to have infinite space, but before long the limits of infinity are reached. Buying additional, or larger, disks or moving unused files to offline storage, are two possible ways to solve this problem. Compression is another.

Ever since Windows NT 3.51, the Windows NT File System (NTFS) has had native support for file compression. While very useful, this generic compression is not without drawbacks. Compression and decompression consume precious processor cycles and, because it uses a generic compression algorithm (Lempel-Ziv), it may not guarantee you the best possible compression ratio. With NTFS version 5 (introduced with Windows 2000) Microsoft has added another, specialised means of compression known as sparse files. In this article we’ll look at both of these compression methods and how you use them from a programmer’s perspective.

NTFS Internals

To fully understand how compression and sparse files work we need to take a peek inside NTFS. NTFS uses disk clusters as its basic unit of storage. Depending on how the volume was formatted, a cluster may span one or more disk sectors. This means that when NTFS reads from or writes to a disk, it always does so using at least one cluster. Even if you create a one-byte file, the file will occupy a full cluster on the disk. Each cluster is identified using a Logical Cluster Number (LCN). The first cluster has LCN 0, the second LCN 1, and so forth.

A file’s data will span one or more clusters on disk, which may not be physically contiguous but could be viewed as such logically. NTFS numbers these logical clusters belonging to a file using a Virtual Cluster Number (VCN). The first logical cluster of a file has VCN 0, the second VCN 1, and so forth. NTFS doesn’t keep track of each individual cluster that belongs to a file but instead manages them using the concept of a run (sometimes called an extend). A run is just a set of one or more clusters that have contiguous LCNs. Each run can be managed by storing its starting LCN and the count of clusters in that run. NTFS does exactly that: for each file it maintains a list of LCN and count pairs identifying the runs in which the file data is stored. The only missing piece is that NTFS needs to be able to translate an offset into a file into the LCN where it’s stored. For this purpose NTFS also stores a VCN to LCN mapping (an offset into a file can be converted to a VCN simply by dividing the offset by the cluster size). The whole concept of LCNs, VCNs and mapping structures is summarised in Figure 1. The file described therein has 8 clusters stored in two runs: four cluster each (runs aren’t required to be four clusters, it’s just an example).