|May the source be with you, but remember the KISS principle ;-)|
|Contents||Bulletin||Scripting in shell and Perl||Network troubleshooting||History||Humor|
|News||See also||Recommended Links||I-nodes||Solaris File System Structure||Disk and Filesystems Management in Solaris||Solaris snapshots|
|Solaris File System Structure||Ext2/Ext3||NTFS||Disk Repartitioning||Filesystems Recovery||Humor||Etc|
Science is facts; just as houses are made of stones, so is science
of facts; but a pile of stones is not a house and a collection of facts
is not necessarily science.
-- Henri Poincaire
Filesystems is a very interesting area, one of the few areas in Unix where new algorithms still can make a huge difference in performance.
Often the historical view on filesystems is a bit too Unix-centric and states that the Berkeley Fast File System is the ancestor of most modern file systems. This view ignores competitive and earlier implementations from IBM(HPFS), DEC (VAX VMS), Microsoft (NTFS) and others.
Still Unix filesystems became a classic and concepts introduced in ti dominate all modern filesystems It also introduced many interesting features and algorithms into the area. For example a very interesting concept of extended attributes introduced in the 4.4 BSD filesystem have recently been added to Ext2fs:
Immutable files can only be read: nobody can write or delete them. This can be used to protect sensitive configuration files.
Append-only files can be opened in write mode but data is always appended at the end of the file. Like immutable files, they cannot be deleted or renamed. This is especially useful for log files which can only grow. All-in all following attributes are avialable at ext2f:
Unix filesystem is a classic, but classic has it's own problems: it's actually an old and largely outdated filesystem that outlived its usefulness. Later ideas implemented in HPFS, BFS and several other more modern filesystems are absent in plain-vanilla implementation of Unix file systems. Balanced trees now serve the base of most modern filesystems including ReiserFs (which started as NTFS clone but aqured some unique features in the process of development):
The Reiser Filesystems by Hans Reiser [and Moscow University researchers], a very ambitious project to not only improve performance and add journaling, but to redefine the filesystem as a storage repository for arbitrarily complex objects. reiserfs. Reiserfs is faster than ext2/3 because it uses balanced trees for it's directory-structures. It was used by Suse and Gentoo.
Unfortunately the novel feature introduced in HPFS called extended attributes never got traction in other filesystems. Of course the fundamental decision to make attributes indexable deserves closer examination, given the costs of indexing, but still the fixed set of attributes (like in UFS) created too many problems to ignore this issue. Still I think that extended attributes should be present in a filesystem, and they can replace such kludges as #! notation in UNIX for specifying default processor in executable files.
These notes describe the basic Unix file system and the kernel structures that support it. For further information the readers should consult The Design of Unix Operating System by M.J.Bach (Prentice-Hall 1986 ISBN 0-13-201757-1) and The Magic Garden Explained by B.Goodheart and J.Cox (Prentice-Hall 1994 0-13-098138-9). The Bach book is probably easier to read but the Goodheart and Cox book is more up-to-date.
Modern Unix systems use a Virtual File System (VFS), this allows the system to use many different actual file systems in a seamless fashion. At a low level, driver software is required for each actual file system. This allows Network File Systems (NFS), High-Sierra File Systems (HSFS - found on CDROMs), MSDOS File Systems (PCFS) amongst others to be included in the Unix view of an integrated hierarchy of files and directories. Included among the various supported file systems are the Unix File System (UFS) and the older System V File System (S5FS). These constitute the traditional Unix file system and will be described in detail in these notes.
When the UFS filesystem was introduced to BSD in 1982, its use of 32 bit
offsets ... structure on the disk for use with systems that don't
understand GPT. ...
www.freebsd.org/projects/bigdisk/index.html - 14k -
Early versions of Unix used filesystems referred to simply as FS. FS only included the boot block, superblock, a clump of inodes, and the data blocks. This worked well for the small disks early Unixes were designed for, but as technology advanced and disks got larger, moving the head back and forth between the clump of inodes and the data blocks they referred to caused thrashing. BSD optimized this in FFS by inventing cylinder groups, breaking the disk up into smaller chunks, each with its own inode clump and data blocks.
The intent of BSD FFS is to try to localize associated data blocks and metadata in the same cylinder group, and ideally, all of the contents of a directory (both data and metadata for all the files) in the same or nearby cylinder group, thus reducing fragmentation caused by scattering a directory's contents over a whole disk.
Some of the performance parameters in the superblock included number of tracks and sectors, disk rotation speed, head speed, and alignment of the sectors between tracks. In a fully optimized system, the head could be moved between close tracks to read scattered sectors from alternating tracks while waiting for the platter to spin around.
As disks grew larger and larger, sector level optimization became obsolete (especially with disks that used linear sector numbering and variable sectors per track). With larger disks and larger files, fragmented reads became more of a problem. To combat this, BSD originally increased the filesystem block size from one sector to 1k in 4.0BSD, and, in FFS, increased the filesystem block size from 1k to 8k. This has several effects. The chances of a file's sectors being contiguous is much greater. The amount of overhead to list the file's blocks is reduced. The number of blocks representable in a fixed bit width block number is increased (allowing for larger disks).
With larger block sizes, disks with many small files would waste a lot of space, so BSD added block level fragmentation, where the last partial block of data from several files may be stored in a single "fragment" block instead of multiple mostly empty blocks.
UFS file system is made of:
Block 1 that contains:
Total size of the file system (in blocks)
Number of blocks reserved for inodes
Name of the file system
Date of the last superblock update
Head of the free-block list
List of free inodes
Inode blocks and, for assigned
File type: regular, device, named pipes, socket, symbolic link
Link count: name and inode of master file
Size of the file in bytes
Last file access date
Last file modification date
Last inode modification date
Pointers to data blocks: actual location of blocks on physical disk
Data blocks with user data or system files
The superblock is followed by blocks containing
inodes and associated
inumber pairs. An inode describes an individual file with one inode
for each file in the file system. For each file system is allocated a maximum number
of inodes and therefore a maximum number of files. The maximum values depend on
the the file system size.
Inode 1 on each file system is unnamed and unused. Inode 2 must correspond to the file system root directory that supports all other files in the file system. Inodes after inode 2 are free and can be any file. Inodes and blocks are not allocated in any particular order.
A directory entry, file or link, consists of the
name and the
inumber representing the file. The link count indicates the number
of directory entries that refer to the same file. A file is deleted if the link
count is zero. When the file is deleted the associated inode is returned to the
free-inode list and its associated blocks are returned to the
August 7, 2007 | KernelTrap
Submitted by Jeremy on August 7, 2007 - 9:26am.
In a recent lkml thread, Linus Torvalds was involved in a discussion about mounting filesystems with the
noatimeoption for better performance, "'noatime,data=writeback' will quite likely be *quite* noticeable (with different effects for different loads), but almost nobody actually runs that way." He noted that he set O_NOATIME when writing git, "and it was an absolutely huge time-saver for the case of not having 'noatime' in the mount options. Certainly more than your estimated 10% under some loads." The discussion then looked at using the
relatimemount option to improve the situation, "relative atime only updates the atime if the previous atime is older than the mtime or ctime. Like noatime, but useful for applications like mutt that need to know when a file has been read since it was last modified." Ingo Molnar stressed the significance of fixing this performance issue, "I cannot over-emphasize how much of a deal it is in practice. Atime updates are by far the biggest IO performance deficiency that Linux has today. Getting rid of atime updates would give us more everyday Linux performance than all the pagecache speedups of the past 10 years, _combined_." He submitted some patches to improve
relatime, and noted about
"It's also perhaps the most stupid Unix design idea of all times. Unix is really nice and well done, but think about this a bit: 'For every file that is read from the disk, lets do a ... write to the disk! And, for every file that is already cached and which we read from the cache ... do a write to the disk!'"
As a file on a UFS filesystem, the typical open routine called would be the ufs_open() ... You will find the structure definition for si_user and associated ...
www.itworld.com/Comp/2377/swol-0309-insidesolaris/ - 57k -
The ufs file system inode structure contains the addresses of 12 direct blocks, one indirect block, and one double indirect block. ...
uw713doc.sco.com/ODM_FSadmin/fssag-4.html - 41k
Learn the intricacies of the AIX® file system framework. Every operating system provides a native kernel framework that kernel developers have to understand and adhere to when developing a piece of a kernel component for that operating system. This article sheds some light on the AIX file system framework. You need to understand the framework in order to develop a new file system, or to port an existing file system to the AIX operating system.
AIX 5L™ is an award-winning operating system that delivers superior scalability, reliability, and manageability. It is the default operating system that powers some of the most powerful IBM UNIX® servers in the market.
Typically, a file system can be defined as a piece of software that helps in storing, organizing, and retrieving data from a physical storage medium, be it a hard disk drive, CD-ROM, or any other storage device. The code for such data organization, by its very nature, should be portable. In the real world, though, every operating system provides its own interfaces by which it requests a particular file system operation, and it is expected that the underlying piece of software provides results in the format that the operating system expects. The interfaces vary with different flavors of operating systems, and need to be exported by the file system to be supported on the particular operating system.
In this article, you'll learn about the AIX® operating system file system framework. You'll also get an overview of the IO layer and an explanation of some important concepts. Brief explanations are also included of the interfaces and methods when developing a new file system, or when porting an existing file system to the AIX operating system.
AIX, like many UNIX flavors, hosts the file system as a kernel extension. It is assumed that you have basic knowledge of UNIX programming and file system concepts. It would also be helpful to know how to write kernel extensions for AIX.
Understanding the logical file system and the virtual file system
The logical file system layer is the level of abstraction at which users can request the various file operations, such as read, write, stat, and so on. The logical file system interface supports UNIX-type file access semantics. The logical file system layer acts as a superset of the virtual file system, which encapsulates disparate file systems, that provides the kernel with a consistent view of the underlying directory tree. The logical file system is also responsible for managing the kernel's open file table and the per process file descriptor information.
The virtual file system is an abstraction of the underlying physical file system. The virtual file system provides a standard set of interfaces that you should support in order for your file system to be hosted over the AIX operating system. The virtual file system bridges the underlying disparate physical file system to the logical file system, providing a consistent directory tree hierarchy to the rest of the operating system.
Each unique mount instance of a file system object is represented by a virtual file system structure. A virtual file system can be a physical file system, a network file system, or a logical file system (one that does not have a physical backing store, such as ramfs). Figure 1 shows the AIX file system hierarchy.
Softpanorama hot topic of the month
DYNIX-ptx® System Administration UFS Filesystem
UnixInsider(former SunWorld) paper
From Solaris FAQ
How can I grow a UFS filesystem (from Solaris FAQ)
A Brief UNIX Filesystem Review -- from SCO
Introduction to the UNIX Environment -- contains useful info in Unix filesystem
by Mark BurgessA short introduction to operating systems
UNIX File System Structure UFS. A UFS file system
is made of:. Boot block, the first block of every file system
(block 0); Superblock Block 1 that contains: ...
Linux userspace UFS2 tools.
UFS2 Tools: An open source tool for accessing UFS2 (BSD) slices from within Windows
AIX kernel extensions" (developerWorks, Aug 2006): This
article explains how to write a kernel extensions for AIX.
Open AFS: OpenAFS has an AIX port for the AFS file system.
The Linux Documentation Project's Filesystems HOWTO: FFS. Note that the distinction this draws between FFS and UFS is wrong; both terms are used at present and have been used in the past.
Little UFS2 FAQ: What is the difference between UFS and FFS? Note that this gets the relationship between FFS and UFS backwards; see the "Local Filesystems" chapter of The Design and Implementation of the 4.4BSD Operating System, which refers to the upper layer as UFS, and the "Local Filestores" chapter, which refers to the lower layer as FFS.
Kernel Extensions and Device Support Programming Concepts , SC23-4900-03, to learn about kernel programming and the kernel environment for AIX.
Technical Reference: Kernel and Subsystems, Volume 1 , SC23-4917-03, for detailed information about kernel services, device driver operations, and file system operations for AIX.
Technical Reference: Kernel and Subsystems, Volume 2 , SC23-4918-03, for detailed information about the configuration subsystem, communications subsystem, LFT subsystem, printer subsystems, SCSI subsystem, Integrated Device Electronics, SSA subsystem, and the serial DASD subsystem for AIX.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes. If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner.
ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least
Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.
Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info|
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Last modified: June 04, 2016