UFS The Unix File System

Science is facts; just as houses are made of stones, so is science made
of facts; but a pile of stones is not a house and a collection of facts
is not necessarily science.

-- Henri Poincaire

Filesystems is a very interesting area, one of the few areas in Unix where new algorithms still can make a huge difference in performance.

Often the historical view on filesystems is a bit too Unix-centric and states that the Berkeley Fast File System is the ancestor of most modern file systems. This view ignores competitive and earlier implementations from IBM(HPFS), DEC (VAX VMS), Microsoft (NTFS) and others.

Still Unix filesystems became a classic and concepts introduced in ti dominate all modern filesystems It also introduced many interesting features and algorithms into the area. For example a very interesting concept of extended attributes introduced in the 4.4 BSD filesystem have recently been added to Ext2fs:

Immutable files can only be read: nobody can write or delete them. This can be used to protect sensitive configuration files.

Append-only files can be opened in write mode but data is always appended at the end of the file. Like immutable files, they cannot be deleted or renamed. This is especially useful for log files which can only grow. All-in all following attributes are avialable at ext2f:

1. A (no Access time): if a file or directory has this attribute set, whenever it is accessed, either for reading of for writing, its last access time will not be updated. This can be useful, for example, on files or directories which are very often accessed for reading, especially since this parameter is the only one which changes on an inode when it's open read-only.
2. a ( append only): if a file has this attribute set and is open for writing, the only operation possible will be to append data to its previous contents. For a directory, this means that you can only add files to it, but not rename or delete any existing file. Only root can set or clear this attribute.
3. d (no dump): dump (8) is the standard UNIX utility for backups. It dumps any filesystem for which the dump counter is 1 in /etc/fstab (see chapter "Filesystems and Mount Points"). But if a file or directory has this attribute set, unlike others, it will not be taken into account when a dump is in progress. Note that for directories, this also includes all subdirectories and files under it.
4. i ( immutable): a file or directory with this attribute set simply can not be modified at all: it can not be renamed, no further link can be created to it [1] and it cannot be removed. Only root can set or clear this attribute. Note that this also prevents changes to access time, therefore you do not need to set the A attribute when i is set.
5. s ( secure deletion): when such a file or directory with this attribute set is deleted, the blocks it was occupying on disk are written back with zeroes.
6. S ( Synchronous mode): when a file or directory has this attribute set, all modifications on it are synchronous and written back to disk immediately.

Unix filesystem is a classic, but classic has it's own problems: it's actually an old and largely outdated filesystem that outlived its usefulness.  Later ideas implemented in HPFS, BFS and several other more modern filesystems are absent in plain-vanilla implementation of Unix file systems. Balanced trees now serve the base of most modern filesystems including ReiserFs (which started as NTFS clone but aqured some unique features in the process of development):

The Reiser Filesystems by Hans Reiser [and Moscow University researchers], a very ambitious project to not only improve performance and add journaling, but to redefine the filesystem as a storage repository for arbitrarily complex objects. reiserfs. Reiserfs is faster than ext2/3 because it uses balanced trees for it's directory-structures. It was used by Suse and Gentoo.

 Unfortunately the novel feature introduced in HPFS called extended attributes never got traction in other filesystems.  Of course the fundamental decision to make attributes indexable deserves closer examination, given the costs of indexing, but still the fixed set of attributes (like in UFS) created too many problems to ignore this issue. Still I think that extended attributes should be present in a filesystem, and they can replace such kludges as #! notation in UNIX for specifying default processor in executable files.

http://www.scit.wlv.ac.uk/~jphb/spos/notes/ufs.basics.html

These notes describe the basic Unix file system and the kernel structures that support it. For further information the readers should consult The Design of Unix Operating System by M.J.Bach (Prentice-Hall 1986 ISBN 0-13-201757-1) and The Magic Garden Explained by B.Goodheart and J.Cox (Prentice-Hall 1994 0-13-098138-9). The Bach book is probably easier to read but the Goodheart and Cox book is more up-to-date.

Modern Unix systems use a Virtual File System (VFS), this allows the system to use many different actual file systems in a seamless fashion. At a low level, driver software is required for each actual file system. This allows Network File Systems (NFS), High-Sierra File Systems (HSFS - found on CDROMs), MSDOS File Systems (PCFS) amongst others to be included in the Unix view of an integrated hierarchy of files and directories. Included among the various supported file systems are the Unix File System (UFS) and the older System V File System (S5FS). These constitute the traditional Unix file system and will be described in detail in these notes.

• i-nodes
• directory structures
• Kernel buffer structures

When the UFS filesystem was introduced to BSD in 1982, its use of 32 bit offsets ... structure on the disk for use with systems that don't understand GPT. ...
www.freebsd.org/projects/bigdisk/index.html - 14k - Cached - Similar pages

Early versions of Unix used filesystems referred to simply as FS. FS only included the boot block, superblock, a clump of inodes, and the data blocks. This worked well for the small disks early Unixes were designed for, but as technology advanced and disks got larger, moving the head back and forth between the clump of inodes and the data blocks they referred to caused thrashing. BSD optimized this in FFS by inventing cylinder groups, breaking the disk up into smaller chunks, each with its own inode clump and data blocks.

The intent of BSD FFS is to try to localize associated data blocks and metadata in the same cylinder group, and ideally, all of the contents of a directory (both data and metadata for all the files) in the same or nearby cylinder group, thus reducing fragmentation caused by scattering a directory's contents over a whole disk.

Some of the performance parameters in the superblock included number of tracks and sectors, disk rotation speed, head speed, and alignment of the sectors between tracks. In a fully optimized system, the head could be moved between close tracks to read scattered sectors from alternating tracks while waiting for the platter to spin around.

As disks grew larger and larger, sector level optimization became obsolete (especially with disks that used linear sector numbering and variable sectors per track). With larger disks and larger files, fragmented reads became more of a problem. To combat this, BSD originally increased the filesystem block size from one sector to 1k in 4.0BSD, and, in FFS, increased the filesystem block size from 1k to 8k. This has several effects. The chances of a file's sectors being contiguous is much greater. The amount of overhead to list the file's blocks is reduced. The number of blocks representable in a fixed bit width block number is increased (allowing for larger disks).

With larger block sizes, disks with many small files would waste a lot of space, so BSD added block level fragmentation, where the last partial block of data from several files may be stored in a single "fragment" block instead of multiple mostly empty blocks.

UFS file system is made of:

• Boot block, the first block of every file system (block 0)
• Superblock
  Block 1 that contains:
  Total size of the file system (in blocks)
  Number of blocks reserved for inodes
  Name of the file system
  Device identification
  Date of the last superblock update
  Head of the free-block list
  List of free inodes
  Inode blocks and, for assigned inodes:
  File type: regular, device, named pipes, socket, symbolic link
  File owner: UID and GID
  Protection information: rwe for ugo
  Link count: name and inode of master file
  Size of the file in bytes
  Last file access date
  Last file modification date
  Last inode modification date
  Pointers to data blocks: actual location of blocks on physical disk
  Data blocks with user data or system files
  

The superblock is followed by blocks containing inodes and associated inumber pairs. An inode describes an individual file with one inode for each file in the file system. For each file system is allocated a maximum number of inodes and therefore a maximum number of files. The maximum values depend on the the file system size.

Inode 1 on each file system is unnamed and unused. Inode 2 must correspond to the file system root directory that supports all other files in the file system. Inodes after inode 2 are free and can be any file. Inodes and blocks are not allocated in any particular order.

A directory entry, file or link, consists of the name and the inumber representing the file. The link count indicates the number of directory entries that refer to the same file. A file is deleted if the link count is zero. When the file is deleted the associated inode is returned to the free-inode list and its associated blocks are returned to the free-block list.

NEWS CONTENTS

• 20070807 : Linux Replacing atime ( Linux Replacing atime , Aug 7, 2007 )
• 20070807 : http://ufs-linux.sourceforge.net/docs/ufs.txt ( )
• 20070807 : Making Inodes Behave ( Making Inodes Behave, )
• 20070807 : ITworld.com - Solaris sockets, past and present ( ITworld.com - Solaris sockets, past and present, )
• 20070807 : The VERITAS File System ( The VERITAS File System, )
• 20070529 : Srikanth Srinivasan ( softpanorama.org, May 29, 2007 )

Old News ;-)

[Aug 7, 2007] Linux Replacing atime

August 7, 2007 | KernelTrap

Submitted by Jeremy on August 7, 2007 - 9:26am.

In a recent lkml thread, Linus Torvalds was involved in a discussion about mounting filesystems with the noatime option for better performance, "'noatime,data=writeback' will quite likely be *quite* noticeable (with different effects for different loads), but almost nobody actually runs that way." He noted that he set O_NOATIME when writing git, "and it was an absolutely huge time-saver for the case of not having 'noatime' in the mount options. Certainly more than your estimated 10% under some loads." The discussion then looked at using the relatime mount option to improve the situation, "relative atime only updates the atime if the previous atime is older than the mtime or ctime. Like noatime, but useful for applications like mutt that need to know when a file has been read since it was last modified." Ingo Molnar stressed the significance of fixing this performance issue, "I cannot over-emphasize how much of a deal it is in practice. Atime updates are by far the biggest IO performance deficiency that Linux has today. Getting rid of atime updates would give us more everyday Linux performance than all the pagecache speedups of the past 10 years, _combined_." He submitted some patches to improve relatime, and noted about atime:

"It's also perhaps the most stupid Unix design idea of all times. Unix is really nice and well done, but think about this a bit: 'For every file that is read from the disk, lets do a ... write to the disk! And, for every file that is already cached and which we read from the cache ... do a write to the disk!'"

http://ufs-linux.sourceforge.net/docs/ufs.txt

Making Inodes Behave

ITworld.com - Solaris sockets, past and present

As a file on a UFS filesystem, the typical open routine called would be the ufs_open() ... You will find the structure definition for si_user and associated ...
www.itworld.com/Comp/2377/swol-0309-insidesolaris/ - 57k - Cached -

The VERITAS File System

The ufs file system inode structure contains the addresses of 12 direct blocks, one indirect block, and one double indirect block. ...
uw713doc.sco.com/ODM_FSadmin/fssag-4.html - 41k

[May 29, 2007] Developing a file system for AIX by Srikanth Srinivasan

Learn the intricacies of the AIX® file system framework. Every operating system provides a native kernel framework that kernel developers have to understand and adhere to when developing a piece of a kernel component for that operating system. This article sheds some light on the AIX file system framework. You need to understand the framework in order to develop a new file system, or to port an existing file system to the AIX operating system.

Introduction

AIX 5L™ is an award-winning operating system that delivers superior scalability, reliability, and manageability. It is the default operating system that powers some of the most powerful IBM UNIX® servers in the market.

Typically, a file system can be defined as a piece of software that helps in storing, organizing, and retrieving data from a physical storage medium, be it a hard disk drive, CD-ROM, or any other storage device. The code for such data organization, by its very nature, should be portable. In the real world, though, every operating system provides its own interfaces by which it requests a particular file system operation, and it is expected that the underlying piece of software provides results in the format that the operating system expects. The interfaces vary with different flavors of operating systems, and need to be exported by the file system to be supported on the particular operating system.

In this article, you'll learn about the AIX® operating system file system framework. You'll also get an overview of the IO layer and an explanation of some important concepts. Brief explanations are also included of the interfaces and methods when developing a new file system, or when porting an existing file system to the AIX operating system.

AIX, like many UNIX flavors, hosts the file system as a kernel extension. It is assumed that you have basic knowledge of UNIX programming and file system concepts. It would also be helpful to know how to write kernel extensions for AIX.

Understanding the logical file system and the virtual file system

The logical file system layer is the level of abstraction at which users can request the various file operations, such as read, write, stat, and so on. The logical file system interface supports UNIX-type file access semantics. The logical file system layer acts as a superset of the virtual file system, which encapsulates disparate file systems, that provides the kernel with a consistent view of the underlying directory tree. The logical file system is also responsible for managing the kernel's open file table and the per process file descriptor information.

The virtual file system is an abstraction of the underlying physical file system. The virtual file system provides a standard set of interfaces that you should support in order for your file system to be hosted over the AIX operating system. The virtual file system bridges the underlying disparate physical file system to the logical file system, providing a consistent directory tree hierarchy to the rest of the operating system.

Each unique mount instance of a file system object is represented by a virtual file system structure. A virtual file system can be a physical file system, a network file system, or a logical file system (one that does not have a physical backing store, such as ramfs). Figure 1 shows the AIX file system hierarchy.

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

The Sun Solaris UFS implementation chapter of the Solaris™ Internals: Solaris 10 and OpenSolaris Kernel Architecture, Second Edition book by Richard McDougall, Jim Mauro ISBN 0-13-148209-2

Unix File System - Wikipedia, the free encyclopedia

The UNIX ufs filesystem

DYNIX-ptx® System Administration UFS Filesystem

CSC5140 Advanced Topics in Operating Systems

• UNIX Filesystem (part 1)
• UNIX Filesystem (part 2)
• UNIX Interprocess Communication (IPC)

UnixInsider(former SunWorld) paper

• Getting to know the Solaris filesystem, Part 1

• Getting to know the Solaris filesystem, Part 2

Managing filesystem types

From Solaris FAQ

• How can I grow a UFS filesystem (from Solaris FAQ)

• How to remove a UFS filesystem ...

• Creating a UFS filesystem on a floppy disk

Etc

• A Brief UNIX Filesystem Review -- from SCO

• The Unix File System

• Definition of a Filesystem

• Key Parts of the UNIX Filesystem

• About mounting DOS filesystems

• The Unix Filesystem

• Introduction to the UNIX Environment -- contains useful info in Unix filesystem

http://home.lanet.lv/~sd70058/aboutos/os.html

UFS - Wikipedia, the free encyclopedia

by Mark BurgessA short introduction to operating systems

The Solaris UFS File System > Access Control in UFS

The Solaris UFS File System > UFS On-Disk Format

UNIX File System Structure UFS

UNIX File System Structure UFS. A UFS file system is made of:. Boot block, the first block of every file system (block 0); Superblock Block 1 that contains: ...
www.bo.infn.it/alice/alice-doc/mll-doc/duix/admgde/node39.html - 6k - Cached - Similar pages <

http://www.bo.infn.it/alice/alice-doc/mll-doc/duix/admgde/node122.html

Little UFS2 FAQ

Linux userspace UFS2 tools.

Filesystems-HOWTO (part of The Linux Documentation Project, link is not linux specific.)

UFS2 Tools: An open source tool for accessing UFS2 (BSD) slices from within Windows

"Writing AIX kernel extensions" (developerWorks, Aug 2006): This article explains how to write a kernel extensions for AIX.

Open AFS: OpenAFS has an AIX port for the AFS file system.

The Linux Documentation Project's Filesystems HOWTO: FFS. Note that the distinction this draws between FFS and UFS is wrong; both terms are used at present and have been used in the past.

Little UFS2 FAQ: What is the difference between UFS and FFS? Note that this gets the relationship between FFS and UFS backwards; see the "Local Filesystems" chapter of The Design and Implementation of the 4.4BSD Operating System, which refers to the upper layer as UFS, and the "Local Filestores" chapter, which refers to the lower layer as FFS.

Kernel Extensions and Device Support Programming Concepts , SC23-4900-03, to learn about kernel programming and the kernel environment for AIX.

Technical Reference: Kernel and Subsystems, Volume 1 , SC23-4917-03, for detailed information about kernel services, device driver operations, and file system operations for AIX.

Technical Reference: Kernel and Subsystems, Volume 2 , SC23-4918-03, for detailed information about the configuration subsystem, communications subsystem, LFT subsystem, printer subsystems, SCSI subsystem, Integrated Device Electronics, SSA subsystem, and the serial DASD subsystem for AIX.

http://www.uwsg.iu.edu/UAU/filesystem/

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019