Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Windows NTFS filesystem internals

News Windows filesystems Recommended Links Tutorials Resizing of NTFS partitions Windows Filesystems Recovery exFAT Etc

NTFS is the preferred file system on Windows NT family, including Windows 7, XP, and 2000. It is efficient only on hard drives. It does not perform well (in other words it is too slow) on flash drives. In this case FAT32 or ExFAT should be used. 

NTFS is derived from OS/2 filesystem called HPFS. It was written by Gordon Letwin and others at Microsoft and added to OS/2 version 1.2, at that time still a joint undertaking of Microsoft and IBM.

NTFS  address many of the limitations of the earlier FAT16 and FAT32 file systems. The most important features are as follows:

Other advanced features provided by NTFS are the following:

Recommended Links

Services for UNIX - Interoperability

GID on NTFS File System

Can you set group on a file or folder on NTFS file system? - No, ugh... Yes.

This question puzzled me for a long time but since it never really made it to my top priorities, I didn't look up for information on this. I thought of exploring this area more while I was researching something about NFS server.

I started with looking for the utilities (obviously, Windows-based ones) which can set this information for me. My search ended quickly with the chown.exe and chgrp.exe which you can install with Interix/SUA Base Utilities.

Using Process Monitor (replacement of Filemon and Regmon utilities) revealed that group information gets stored on the file system with an IRP_MJ_SET_SECURITY request -

The other interesting fact is this request originating from the POSIX subsystem (psxss.exe) which makes sense because chown.exe and chgrp.exe utilities are POSIX subsystem utilities.

This KB Article says -

In the Windows NT and Windows 2000 NTFS file system, each file also has an owner and a primary group. The primary group of a file is not used by the Win32 subsystem, but is present for programs that make use of the POSIX subsystem. When a file is created, the user who created the file becomes its owner and that user's primary group becomes the file's primary group. Access Control Entries (ACEs) are then added to the DACLs to assign permissions.

That makes it clear that none of other utilities I tried to use, could set this information because they were basically Win32 binaries and Win32 subsystem does not, in anyway, uses this information.

So the best practice would be to set correct primary groups for your users and then use Interix/SUA chown.exe and chgrp.exe utilities to manage them the way you want them to be seen by your UNIX clients.

Additional Note: The ls.exe and chmod.exe are other utilities which can help you do things the UNIX way.

NTFS5 vs. FAT32

Win2K File-System Facts
Compatibility with legacy file systems is an important aspect of OS upgrades. Win2K supports several file systems, and its support is essentially a superset of NT 4.0's support. (For more information about file systems, see "File-System Resources," page 104.) Win2K still supports FAT12, FAT16, and the CD-ROM File System (CDFS), which is the International Organization for Standardization (ISO)-9660-compliant NT file system you use to read CD-ROMs. However, Microsoft doesn't plan to add features or upgrade these systems, except for a possible future addition of support for the UNIX-centric Rock Ridge format in CDFS. NTFS, which is the premier Win2K file system, continues to exist but sports a new file-system structure and new capabilities. Microsoft also included in Win2K a Universal Disk Format (UDF) file system that the company introduced in Windows 98. Currently, Win2K's UDF version, which complies with the ISO-13346 standard, mainly supports DVD-ROM media. However, Microsoft plans to eventually replace the CD-ROM-centric CDFS specification with UDF. Win2K also supports a slightly more recent version of UDF (i.e., UDF 1.50) than Win98 supports (i.e., UDF 1.02).

NTFS.com NTFS File System General Information. Data Recovery

Exploring NTFS On-disk Structures

NTFS FAQ (en)

Microsoft/NTFS overview

NTFS Read-only NTFS support.

Systems Internals -- very useful site with a lot of utilities with source code. Makers of NTFSDOS -- NTFS driver for DOS and other interesting tools for NT recovery

File System Usage in Windows NT 4.0 - by Wemer Vogels - 2.15M .PDF
This paper discusses real world traces of filesystem usage. It shows how certain real world behaviors are important to optimize for and shows how NT succeeds (partially) at providing good filesystem performance.

PartitionMagic - Product Information -- nice utility to convert NTFS into something else

Tutorials

[PPT] Windows Kernel Internals NTFS

ntfsdoc - What is ntfsdoc

NTFS - Wikipedia, the free encyclopedia

Windows NT File System Internals: A Developer's Guide: Chapter 4. ...

Inside NTFS

NTFS streams

NTFS File System Internals - DECUS San Diego

Inside Win2K NTFS, Part 2

NTFS.com NTFS Format. Master File Table (MFT). General Information ...

The Delphi Magazine Issue 98 Sample Article/NTFS Compression And Sparse Filesby Marcel van Brakel This article first appeared in The Delphi Magazine Issue 98 (October 2003)

How could space not be infinite? Is there a sign somewhere
saying ‘Space Ends Here – Mind the Gap’?      (Max Tegmark)

Hard disks keep getting larger every year, but after a while even the largest hard disks just don’t cut it any more. My first PC had a 425Mb hard disk, and I was sure I’d never run out of space. Two years later I found myself looking for a second disk. My second PC had a 4Gb hard disk, and this time I just knew I’d never run out of disk space. Two years later I found myself looking for a second disk. My current PC has a 40Gb hard disk, and now, a year after purchase, it’s 75% full. At most companies I’ve worked for the exact same thing happens. When purchased the storage devices seem to have infinite space, but before long the limits of infinity are reached. Buying additional, or larger, disks or moving unused files to offline storage, are two possible ways to solve this problem. Compression is another.

Ever since Windows NT 3.51, the Windows NT File System (NTFS) has had native support for file compression. While very useful, this generic compression is not without drawbacks. Compression and decompression consume precious processor cycles and, because it uses a generic compression algorithm (Lempel-Ziv), it may not guarantee you the best possible compression ratio. With NTFS version 5 (introduced with Windows 2000) Microsoft has added another, specialised means of compression known as sparse files. In this article we’ll look at both of these compression methods and how you use them from a programmer’s perspective.

NTFS Internals

To fully understand how compression and sparse files work we need to take a peek inside NTFS. NTFS uses disk clusters as its basic unit of storage. Depending on how the volume was formatted, a cluster may span one or more disk sectors. This means that when NTFS reads from or writes to a disk, it always does so using at least one cluster. Even if you create a one-byte file, the file will occupy a full cluster on the disk. Each cluster is identified using a Logical Cluster Number (LCN). The first cluster has LCN 0, the second LCN 1, and so forth.

A file’s data will span one or more clusters on disk, which may not be physically contiguous but could be viewed as such logically. NTFS numbers these logical clusters belonging to a file using a Virtual Cluster Number (VCN). The first logical cluster of a file has VCN 0, the second VCN 1, and so forth. NTFS doesn’t keep track of each individual cluster that belongs to a file but instead manages them using the concept of a run (sometimes called an extend). A run is just a set of one or more clusters that have contiguous LCNs. Each run can be managed by storing its starting LCN and the count of clusters in that run. NTFS does exactly that: for each file it maintains a list of LCN and count pairs identifying the runs in which the file data is stored. The only missing piece is that NTFS needs to be able to translate an offset into a file into the LCN where it’s stored. For this purpose NTFS also stores a VCN to LCN mapping (an offset into a file can be converted to a VCN simply by dividing the offset by the cluster size). The whole concept of LCNs, VCNs and mapping structures is summarised in Figure 1. The file described therein has 8 clusters stored in two runs: four cluster each (runs aren’t required to be four clusters, it’s just an example).



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019