Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-) Softpanorama Search

Enterprise Unix System Administration

The KISS rule can be expanded as: Keep It Simple, Sysadmin ;-)

Unix system administrators can administer systems in three different environments

• Small sites have, say,  less then 20 machines, all running the same flavor of Unix OS (for example Solaris). Usually the administrator of a small site has number of users only slightly bigger then the number of the servers. Also typically he/she is a single administrator for all the servers or have just one colleague administrator who often works different shift.  The server maintained are usually small servers often from a low cost provider (Dell, HP). This is a typical environment for startups.

• Medium sites have less then a hundred servers, and may be running two different flavors of Unix. Typically they are installed in a single location. Number of users can exceed the number of servers by factor 2 or 3. There are multiple administrators and there is some level specialization (for example, backup vs. monitoring)Most servers are small or medium but there can be large servers in the environment as well. The latter requires more specialized hardware knowledge. 

• Large sites often called enterprise sites typically have more then a hundred servers and sometimes several datacenters. The team on administrators consists of at least half-dozen sysadmins. At enterprise level there is often a formal hierarchy of administrators:

  • Junior admins typically have BS degree and does not know scripting well although some knowledge of shell is a must.

  • Senior admins typically have MS degree and have in-depth knowledge two or more scripting languages (shell and Perl, or shell and TCL) and one compiled language (C, C++) in the level enough to develop and port small utilities, recompile kernel, compile PHP, apache, bind and such.  In addition they should have solid understanding of networking.

  The level of specialization is also higher with typically specialized admin responsible for backup (say DataProtector-based) and monitoring (say Tivoli based).  All three types of servers are present in the environment.

  Also at enterprise level there is considerably more politics and red tape involved in the sysadmin job in comparison with smaller sites.

  Here is a typical qualification requirements for a senior enterprise Unix administrator:

  • MS degree in computer science or a related technical field.
  • Solaris certification required; 10+ years of Solaris-10 (SPARC/X86) operating system administration
  • Basic networking skills, solid knowledge of DNS (bind), sendmail, firewalls.  Solaris network certification preferred.
  • Minimum 3 years experience with Sun Web Portal
  • Experience with Veritas Volume Manager and Veritas Filesystem
  • Proven understanding of EMC SAN technology
  • Experience with Source Control, CVS or equivalent
  • Experience with relational database and SQL. 
  • Knowledge of system monitoring (such as Tivoli ITM 6.1), availability monitoring and SLA reporting
  • Experience overseeing maintenance of mid range Unix servers including but not limited to Linux, Solaris, and AIX
  • Ability to coordinate special efforts such as emergency upgrades and *deep-dive" technical analysis
  • Experience in administrating of mid-range servers from Sun, Dell and IBM.
  • History of establishing and insuring 24x7 support services, operational standards, processes, and procedures
  • Ability to be flexible with work schedule, including weekends and holidays
  • Ability to prepare, present, and translate technical requirements into successful business cases
  • Must be highly skilled and proficient in problem solving and able to follow a systematic troubleshooting approach
  • Require minimum support or supervision and is able to solve most issues and/or challenges independently
  • Leadership potential

This page is about enterprise system administration.  Enterprise environment has many requirements that  "admin-in-large" should met addition to usual "sysadmins-in-small" staff  typical for small companies. Nearly half of U.S. IT jobs involve the  maintenance of computers and/or software -- approximately  270,000 people.

Among additional requirements the key is support of several flavors of Unix. Often more then two :-(. Other worth mentioning differences might include: 

• Being able to see the picture of integral health of many desperate boxes, often running different flavors of Unix.
• Being able to account for what's in installed on each box and when it was last patched;
• Accurate documentation about other sysadmins actions. Having good log info of who did what;
• Being able to make changes of group of boxes at once;
• Being able to roll back system configurations;

Risk of doing something wrong is also higher for enterprise system administrators and they often need to follow specific procedures to minimize the risk of downtime due to patching of changes in configuration of the severs.

Those risk are often mishandled and lead to dilbertaruization of IT. See Micromanagement -- the real disease in modern data-datacenters.  Outsourcing and the reduction of the absolute numbers of workers due to more automation in Unix administration also has some negative side effects. Technology is not neutral or independent of who controls it.  while degradation of quality sysadmin work environment is less then say degradation of corporate programming jobs it is still pretty visible.  Conflict are common and can be pretty nasty (see for example typical management speak in Learn to resolve and avoid work conflicts)

When you think about simplifying and partially automating administration in the large enterprise environment, then large and expensive systems like Tivoli,  HP OpenView,  Sun Management Center  are naturally come to mind. Among enterprise tools this page provides some info on Tivoli just because this is a system that the author has some hands-on experience with. 

Still but you should not forget about the mini-tools and open source solutions. IMHO one of the greatest tool that simplify Unix administration is an orthodox file manager. If you do not use it, please take a look at OFMs. Windows top OFMs: FAR and Total Commander can be extremely helpful in multiplatform environment.  Unix OFMs are either more limited (deco) or not completely portable (Midnight Commander), but still Midnight Commander make a lot of sense for Unix administrator. The other useful and largely under appreciated tool is folding editor like THE or at least VIM 6 and later. I believe that for sysadmin tools one should stick to tools that use text format in configuration files. They are much more manageable that tools that use binary formats.

There are also pretty good open source system monitoring tools such as Nagios (it has important advantage that the packages are supported for enterprise version of Red Hat and Suse so installation is a breeze)

ssh and VNC are another two cross platform tools that can simplify many tasks without any additional infrastructure. VNC can provide GUI-based environment for remote administration on almost any platform and is very quick and easy to install if you need one ASAP. 

Actually monitoring of servers with open source tools is rather easy if the network connectivity is good. There are a several open source tools that are scalable up to a thousand servers without major problems. If we are talking about Unix servers only, then ssh, Perl and Apache server are enough for pretty sophisticated remote monitoring :-)

Software distribution and configuration management are much more complex things. Here enterprize calss solution like Tivoli Configuration manager might pay off more quickly.

One of the typical configuration management problems that large organizations often have is how to push config files and software updates into multiple boxes after changes in network topology (for example due to acquisition or divesture).  The simplest solution is to have something like a next loop that ssh's into each box and runs a command for us:

for server in `cat machinelist.txt`; do

    echo "running $command on $server"

    ssh user@$server  $command; echo " "

done

Nortel has a nice program called  CLImanager (use to be called CLImax) that runs on Solaris, NT and Linux. It  allows you telnet into multiple servers and run commands in parallel. The program formats data to display. Say you wanted to display "ifconfig/ipconfig" on 50 machines, this would format it, so you have columns of data, easy to read and put in reports. Also, has a "Watch" command that will repeat a command, and format the data. Basically, it logs into multiple machines, parses and displays data, and outputs all errors on another window to keep your main screen clean.

Fermi lab has a  free tool called rgang that does (minus the output formatting) similar things and is written in Python: Here is the Abstract

RGANG is a tool which allows one to execute commands on or distribute files to many nodes (computers). It incorporates an algorithm to build a tree-like structure (or "worm" structure) to allow the distribution processing time to scale very well to 1000 or more nodes.

Because the original "RGANG" executes the commands on the specified nodes serially, execution time was proportional to the number of nodes. A parallel version of "RGANG" has been implemented in Python. This version forks separate rsh/ssh children, which execute in parallel. After successfully waiting on returns from each child or after timing out, this latest version of RGANG displays the node responses in identical fashion to the original "shell" version of RGANG. In addition, the latest RGANG returns the OR of all of the exit status values of the commands executed on each of the nodes. Simple commands can execute via this RGANG on an 80 node cluster in about 3 seconds.

To allow scaling to kiloclusters, the new RGANG can utilize a tree-structure, via an "nway" switch. When so invoked, RGANG uses rsh/ssh to spawn copies of itself on multiple nodes. These copies in turn spawn additional copies.

Recently sysadmin jobs were targeted by outsourcing. See The incredible disappearing sysadmin. But in no way this is an easy task, see

• A Slightly Skeptical View on IT Offshoring
• IT Outsourcing/Offshoring Skeptic: Fighting Outsourcing Myths

There will also always be demand for good sysadmins. You can only do so much from 3000 miles away. Try coordinating something really complex with somebody in another country. The factor of company loyalty is also huge. 

Dr. Nikolai Bezroukov

Old News ;-)

[Jul 21, 2009] FSlint 2.40

Written in Python & Unix Shell

FSlint is a toolkit to find various forms of lint on a filesystem. At the moment it reports duplicate files, bad symbolic links, troublesome file names, empty directories, non stripped executables, temporary... files, duplicate/conflicting (binary) names, and unused ext2 directory blocks

[Jun 10, 2009] System Configuration Collector Server freshmeat.net

System Configuration Collector Server generates summaries of scc-data sent by clients. It offers a Web interface that supports searching the snapshots and the logbooks of the systems. It also supports comparing (parts of) the snapshots of systems.

[Jun 10, 2009] wcd 5.0.2

WCD is a directory changer for DOS and Unix. Another NCD (Norton Change Directory) clone. This program jumps to a (sub)directory anywhere in the directory-tree. The directory to jump to can be given by only the first characters or an expression with wildcards

[May 22, 2009] ttyplayer

ttyplayer is an application with a GUI that can play back typescript files recorded by the "script" program and .rcd files record by ttyrecorder.

[May 20, 2009] 100 open source gems - part 1 TuxRadar

Apache SQL Analyser

Web: http://tinyurl.com/apachesql

Running a server generates an awful lot of two things: heat and logs. If you have a lot of the first one, you're in trouble and you need to invest in expensive cooling solutions. If you have a lot of the second one, you need to either get some in big hard drives to store your burgeoning log burden, archive or delete your logs on a regular basis or make like Google and turn the throwaway data into a business model.

The Apache SQL Analyser (ASQL) is designed to read Apache log files and dynamically convert them to SQLite format so you can analyse them in a more meaningful way. Using the cut, uniq and wc commands, you can parse a log file by hand to figure out how many unique visitors came to your site, but using Apache SQL Analyser is much faster and means that the whole log gets parsed only once. Finding unique addresses is as simple as a SELECT DISTINCT command.

If you're not seeing the advantage yet, remember that every time a visitor comes to your website, Apache writes at least one line in its log file. In fact, every time that a visitor downloads any page, image, CSS file, Flash movie or JavaScript code, Apache notes down the IP address, the file that was requested, the date and time, the browser that was used, and even the URL that the visitor came from. That's a huge amount of data to wade through, and ASQL transforms it all into one standards-readable system that's super fast.

Trying it yourself

To get started with ASQL, first make sure you have Perl installed, as well as Perl support for SQLite - that's the database back-end it uses. Second, you'll need to have an Apache log file ready to use. It doesn't need to be too big for now.

If you compiled ASQL from our disc using ./configure then make. You'll find its program in the bin directory. Change into that directory then run the command ./asql to start it up. The next step is to type load /path/to/your/log.txt, then wait a few seconds while ASQL converts it all to SQLite format. Once that's done, type "show" and press Enter to have ASQL list all the data it has retrieved from your log file. Now you can make with the SQL queries - here are some examples:

SELECT COUNT(*) FROM logs;

That one-liner tells you how many requests came to your server. Remember, requests don't equal page views - a request is a page, yes, but it's also an image, a JavaScript file, or whatever. If you want to count how many unique visitors came to your site, then use this:

SELECT COUNT(DISTINCT source) FROM logs;

The source field contains the IP addresses of all your visitors, and the DISTINCT function means "only list each unique IP address once", so this time you'll get a much lower number back. Another common search is to find out where your visitors are coming from. If someone types your URL into their browser or uses a bookmark, the referer field (yes, it really is spelled like that) is set to . Otherwise (if someone clicked a link to your site), the referer is set to be the URL they came from. Try using this query:

SELECT DISTINCT referer FROM logs;

This will show you all the ways people got to your pages. But it's imperfect, because sometimes people get to your pages by following links on your own site. You'll see lots of your own results in that list. We can fix that by asking ASQL to ignore links from our own pages, like this:

SELECT DISTINCT referer FROM logs WHERE referer NOT LIKE 'http://www.linuxformat%';

The www.linuxformat part will change, but the important thing is the % symbol at the end. It means "anything can follow here" and will match www.linuxformat.com, www.linuxformat.co.uk, www.linuxformat.co.uk/foo/bar, etc.

Making it permanent

If typing on the command line irks you, you should try using ASQL's save command. Normally ASQL converts your log file into an in-memory database that is tossed away as soon as you quit the program. But when you run the save command, that database gets saved to a file on your hard drive (.asql.db in your home directory by default). Because it's a plain old SQLite database, you can load it into any program and analyse it. If you want to bring the file back into Asql, use the restore command.

ArchView

Web: http://archview.sourceforge.net

Have you ever wanted to look inside something before you actually get it? Maybe a boiled egg that you are not sure is done enough, or a lucky bag at the arcade on the pier. Certainly, many people partaking of lunch in the TuxRadar HQ dining hall would like to take a sneaky peek inside the 'pie of the day' before they put one on their plate. There are, it would seem, plenty of occasions where some sort of X-ray vision would help out a lot, and we have only been thinking about the legal ones.

ArchView is like a special magic set of X-Ray goggles for the internet. Its purpose is to grab the important part of an archive file you have found languishing on some FTP site and let you take a look at the contents, before you wind up downloading a huge archive only to find that the file you really wanted isn't inside anyway. It does this by recognising the type of file and grabbing and decoding the parts of the file that has the contents inside it. So far, the recognised file types are RAR, ZIP and ISO files. The last is probably the most useful, but beware: this is by no means an instantaneous proposition. The ISO index portion can easily take five minutes to download on its own.

ArchView is a Firefox extension, and it will run on Mac and Windows as well, so you don't have to give up on it just because the scumbags you work for make you use a different OS at work. The best way to install is through the Firefox add-on site - just do a search for ArchView on the browser/platform of your choice to get the latest version. When it does, depending on the interface options you have chosen (and we're voting XUL, in case you need someone to choose for you) you'll see a browseable directory listing in which you can do all the sorts of things you usually do.

This is one of those extensions that you might not want to use all the time - sometimes you know you want to grab a file without having to wait ages to get a listing first. In this case, it is easy to switch ArchView on and off via the little icon that appears bottom-right of the Firefox window.

Back in Time

Web: www.le-web.org/back-in-time

When Apple released OS X 10.5, people across the globe were surprised to find that backing up their data had become very sexy, all thanks to the introduction of Time Machine - Apple's backup and restore application. Up until that point, backing data up was a mundane affair that you had to give a lot of thought to in order to avoid any loss of data. And it didn't hurt that the method used to retrieve backups involved a great big timeline of snapshots stretching back into the distance, with smooth transitions and effects. The basic premise is to make one large initial backup and then incrementally back up the data that has changed. It's a simple concept, and one that Apple has implemented in a fantastic way, demonstrating the company's usual flair for design and usability.

Thankfully, an alternative to Time Machine for Linux users has appeared - Back In Time. One of the great challenges any back up program faces is making the process as simple as possible for new users, and Back In Time manages to do this with a series of drop-down boxes for users to choose the data that's backed up, along with the frequency and the retention period. It will back up to any attached storage, allowing you to use NFS mounts as well as external storage. From then on in, Back In Time just sits in the system tray and quietly gets on with its job, leaving you to worry about your work and not the safety of your data. If no data has changed, it skips the scheduled backup until it recognises that changes have been made to the folders and files that you've chosen to back up. This all happens silently in the background; Back In Time runs a very small process to keep tabs on its schedule.

Getting it back

Of course, no backup solution is complete without a method of restoring files in case of emergency, and Back In Time offers a similar approach to Time Machine, albeit without the whizzy animations. Instead it opts for the more sober approach of listing the dates and times of each backup, allowing you to select a point in time that you wish to restore a file from. You're then given a directory listing on the right-hand side that you can use to navigate and find the file you're looking for, before right-clicking on it and selecting Restore. It's really that simple, and we're sure that it'll save you many headaches if you chose to work with Back In Time.

Under the hood, the program relies on rsync, the saviour of many a system administrator over the years, and makes use of meld to compare the differences between snapshots. We can't complain at the results and think that wrapping commands such as rsync up in a graphical interface is worthwhile, particularly in this case. The biggest limitation is that it works solely in user-mode, only enabling you to back up and restore areas that you have access to as a user. This means that doing a full system backup and restore isn't possible, but that's not the point of this application; it's more suited to home users, ensuring that their collections of documents, music and images are safe from harm.

Development is moving rapidly on Back In Time, and we would definitely recommend you give it a look, especially if you're looking for a backup and restore solution for technophobes.

Color Explorer

Web: http://billposer.org/Software/ColorExplorer.html

Found a nice shade of purple you want to remember for use in your website about the Romans in Britain? Browser plugins can get the match from a web page, but to just grab the colour from any single desktop pixel you need Color Explorer.

Great for those of us who know what we like but don't always know how to get it, Color Explorer enables you to click on any area anywhere on your desktop and grab the colour, giving you its RGB (Red, Green, Blue) value, for reproducing it on any graphics application or web page.

That's not all. Using the sliders you can generate your own mixes, learning as you go. By default, RGB values are given in hexadecimal, but if you prefer base 10 it's just a click away. It works the other way too - feed a decimal or hex value into the app, and see the resulting colour.

If you'd rather look at colour charts, scroll down the left-hand selection box to pick one of a huge selection of named colours. You can even search this selection using simple regular expressions if you'd prefer a geekier approach. And if you still want more, click for a random colour.

Compare and contrast

The palette displays two colours at once, so you can alter a colour gradually, and still have the original for comparison. One interesting feature is to lock the two highlighted colours together, then alter one by moving a slider. This alters the other by the same value, so that you can explore the relation between two colours. Between this and a colour wheel to start you off on good colour matches, you can have hours of fun, whether designing an interface or planning the paint scheme for your office.

Installation is straightforward, thanks to minimal dependencies. Color Explorer needs the Tcl language and Tk toolkit (and the Xoris program to copy colours from the desktop, which is included in the source and compiled in automatically).

As well as moving up to GPL v3, this latest release improves the help and tooltip pop-ups, while retaining the basic simplicity of the interface. It's a simple tool, but one that many may find useful for a variety of purposes.

DelSafe

Web: http://homepage.esoterica.pt/~nx0yew/delsafe

Accidentally deleted an important file? We've been there, and we feel your pain - but it doesn't happen to us. No, we're not perfect, we just installed DelSafe.

DelSafe moves deleted files to a .Trash folder on your hard drive(s), and a swift undel command brings them back. Put more technically, DelSafe overrides the original unlink, rename, open, and fopen library calls using the Linux LD_PRELOAD mechanism. As a simple renaming process it is blisteringly fast compared with moving files about.

In order to manage this a .Trash file is created at each mount point. As well as using undel, and options to recover older versions from .Trash, you can recover older versions of files directly using a browser that gives you easy batch copying, like Midnight Commander.

To install DelSafe, first run ./delsafe-0.5.0.sh and answer questions on locations and exceptions to DelSafe's protection, then run the Install script, and create the .Trash folders:

./Install
delsafe_create -c

Test it is working with

touch asterix.gaul
rm asterix.gaul
undel asterix.gaul

You'll need to periodically compress or empty the trash. Other than that, sit back and relax, or even lend your PC to other family members, safe in the knowledge that your files can't go astray.

Editra

Web: http://editra.org

These days, it's not unusual to have knowledge of more than one development language, particularly when dealing with the web. But be it PHP, Perl, Ruby or any other language, it's important to be able to quickly generate code in a decent development environment. Staples Vi and Emacs are both powerful, but not everyone is suited to a text-based editor. So it's with pleasure that we introduce Editra: a Python-based GUI code editor with support for a wide range of scripting and programming languages.

As with most other code editors Editra uses a tabbed approach to its code editing area, but it's a remarkably uncluttered space. There's just a smattering of icons in the taskbar and a handful of menus to choose from. Don't let the simplicity put you off, as Editra is designed to be extensible with plugins and there are half a dozen available on its homepage. You can also download the tarball installation file from here, and binary installer packages will be available for Debian and RPM soon.

Generation game

There are a couple of nice touches within Editra, such as the ability to generate HTML and Latex documents, providing you with a common template for each. There's also a keyword palette that provides access to some of the more uncommon keywords, particularly those that change depending on the file type you're working with. Also helpful is the ability to change the code highlighting behaviour, especially if you're migrating to Editra from another authoring program. The program even provides some sample code that you can select and change its formatting styles, allowing you to quickly see how each choice you make changes the way the code is coloured or highlighted. The developers acknowledge that Editra is still in early development, and it's not without flaws, but we'll be keeping an eye on this one going forward.

Eric

Web: http://die-offenbachs.de/eric/index.html

Eric is an IDE, not half a bee. If you are bashing together a simple Python script, you may be happy enough with banging out a few lines in Vi or even Kate. That's fine, but if you have a more complicated application, maybe with multiple source files and a GUI, then an IDE can really make life a lot easier. Eric, we can say, is simply the finest IDE to use for Python because of all the features it manages to cram in, both as part of the application itself and through plugins to other software such as PyLint.

Last time we looked at Eric it was tied to Qt 3.3. The latest release version now makes use of Qt 4, as well as including a huge range of bugfixes and new features. It may be overkill for a simple script, but if you need access to CVS and Subversion repositories, a class browser, unit-test interface, profiling, multithreaded trace and debugging, you need look no further. Earlier versions suffered a little in the responsiveness of the interface, but this seems to be no longer the case, so you really have no excuse. Do it!

One final point: if Ruby is more your bag than the might of the Python, Eric is still probably the best choice of IDE - it includes a Ruby interactive shell and debugging tools.

FastDup

Web: http://sourceforge.net/projects/fastdup

Hard drives are enormous these days and are getting bigger and cheaper all the time, so many people adopt a policy of not deleting files just in case they are needed some day - after all, the space is there, so why not use it? Of course, that all falls apart on Linux, where downloading distros and other large files is common, meaning that even a large hard drive will fill up sooner or later.

What FastDup does is try to delay the need for an upgrade by helping you spot duplicate files. And, as you can guess by the name, the app's unique selling point is its incredible speed, and we think you'll agree that it's almost impossible to conceive just how its creator managed to get some much performance into FastDup.

To get started after you've built FastDup, run the command fastdup /home/yourusername (eg /home/hudzilla). On a good computer it should be able to scan your entire directory and detect all the duplicates in under a second, but if you have a particularly large home directory you may have to wait a short while for the scan to complete.

The key to FastDup's speed is that it doesn't try to hash files to do a difference check, which means that even very large files can be whizzed through quickly. In fact, as far as we can tell, the only situation when FastDup is slow is when you have several large, identical files - these all need to be checked byte-by-byte to be sure they are identical, which is quite a resource drain. Still, the end result is that you get a list of identical files and can clean them up, so in the long run it's for the best.

Even if it does take a few minutes during its first run, FastDup will be a lot quicker the next time it searches, because it already knows which files it can ignore. If there are particular file sizes you want it to target, use the c parameter along with + (greater than), - (less than) and = (equal to), and a file size. For example, fastdup c +1g will only check files that are larger than 1GB. That's the perfect search to help you track down any big space wasters that are lurking on your hard drive and slowing down your system.

GRDC

Web: http://grdc.sourceforge.net

Most people probably don't have any use for remote desktop connections; most of the time their computers are linked to a monitor, keyboard and mouse, so all they have to do is walk up to them to use them. For sysadmins, however, they are an absolute necessity, offering the ability to remotely connect to and therefore manage a system that is located in a different room, office or even country.

There are a handful of remote desktop clients available for Linux, which can in turn give you access via XDMCP for remote X connections, RDP for remote Windows connections and VNC. However, what sets GRDC apart is its flexibility. Supporting both VNC and RDP, it will work on a netbook or other small-screened computer quite easily even if you're working in full-screen mode. It does this by scrolling the screen around within the confines of the local screen, making it a must-have for any sysadmins who want to use their trusty Asus Eee PC or Acer Aspire One away from their desk.

GRDC enables you to build a list of commonly used remote computers, which you can then group into more meaningful batches such as web servers, file servers or client desktops. Not only that, but when any of the connections are active in full-screen mode, they act as a single application, enabling you to switch to another of your virtual desktops and carry on doing something else while it's running in full-screen mode.

The traditional Alt+Tab combination to switch applications is automatically relayed back to the local computer, letting you quickly switch apps without getting caught up with the remote desktop and any applications that are running on it. This means that your remote desktop experience doesn't get in the way of your being productive on your local machine. There's also a rather useful toolbar that becomes available while you're working in full-screen mode, making your life easier by auto-hiding while you're working and re-appearing when you move your mouse to the top of the screen.

graft freshmeat.net

Perl based

Graft provides a mechanism for managing multiple packages under a single directory hierarchy. It was inspired by both Depot (from Carnegie Mellon University) and Stow (by Bob Glickstein). Packages are installed in self-contained directory trees and symbolic links from a common area are made to the package files.

25 Feb 2002 18:51   tycha

Also consider encap

Also don't forget Encap (http://freshmeat.net/projects/epkg) which I think is the oldest of them. I can't compare them since it's been a while. But for Linux users, the main difference between these and rpm is that you can keep multiple installs of the same package in at once (you do ls -la filename instead of rpm -f filename to find out which package is which). The main disadvantage is, you have to leave old packages installed or possibly break package dependancies (since other packages may have been built against old libraries). This leads to possibly large /usr/local (or whatever mount point it's in).

It would be interesting if these packages could read the SRPM spec files but build into it's directories (of course, most RPMs can't install into any directory) and did the autodependency checking that rpm is good for.

makeself

makeself is a small shell script that generates a self-extractable compressed TAR archive from a directory. The resulting file appears as a shell script, and can be launched as is. The archive will then uncompress itself to a temporary directory and an arbitrary command will be executed (for example, an installation script). This is pretty similar to archives generated with WinZip Self-Extractor in the Windows world.

[Apr 2, 2009] Spacewalk

freshmeat.net

Spacewalk is a Linux and Solaris systems management solution. It allows you to inventory your systems (hardware and software information), install and update software on your systems, collect and distribute your custom software packages into manageable groups, provision (Kickstart) your systems, manage and deploy configuration files to your systems, monitor your systems, provision virtual guests, and start/stop/configure virtual guests.

[Mar 10, 2009] rdiff-backup

rdiff-backup backs up one directory to another.

The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special directory so you can still recover files lost some time ago.

The idea is to combine the best features of a mirror and an incremental backup.

rdiff-backup can also operate in a bandwidth-efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back up to a remote location, and only the differences will be transmitted. It can also handle symlinks, device files, permissions, ownership, etc., so it can be used on the entire file system.

[Mar 10, 2009] Cluster SSH

Perl-based

freshmeat.net

Cluster SSH opens terminal windows with connections to specified hosts and an administration console. Any text typed into the administration console is replicated to all other connected and active windows. This tool is intended for, but not limited to, cluster administration where the same configuration or commands must be run on each node within the cluster. Performing these commands all at once via this tool ensures all nodes are kept in sync.

See also: Software Distribution

[Dec 14, 2008] Cool Solutions Using ClusterSSH to Perform Tasks on Multiple Servers Simultaneously By Martijn Pepping

Problem:

As an administrator of SLES/OES Linux clusters or multiple SUSE Linux servers you are probably familiar with that fact that you have to make an identical change on more than one server. Those can be things like editing files, execute commands, collect data or some other administrative task.

There are a couple of way to do this. You can write a script that performs the change for you, or you can SSH into a server, make the change and repeat that task manually for every server.

Now both ways can cost an extended amount of time. Writing and testing a shell script takes some time and performing the task by hand on lets say five or more servers also costs time.

Now, wouldn't it be a real timesaver when you have only one console in which you can perform tasks on multiple servers simultaneously? This solution can be found in ClusterSSH.

Solution:

With ClusterSSH it is possible to make a SSH connection to multiple servers and perform tasks from one single command window, without any scripting. The 'cssh' command lets you connect to any server specified as a command line argument, or to groups of servers (or cluster nodes) defined in a configuration file.

The 'cssh' command opens a terminal window to every server which can be used to review the output sent from the cssh-console, or to edit a single host directly. Commands given in to the cssh-console are executed on every connected host. When you start typing in the cssh-console you'll see that the same command also show up on the commandline of the connected systems.

The state of connected systems can be toggled from the cssh-console. So if you want to exclude certain hosts temporarily from specific command, you can do this with a single mouseclick. Also, hosts can be added on the fly and open terminal windows can automatically be rearranged.

One caveat to be aware of is when editing files. Never assume that file is identical on all systems. For example, lines in a file you are editing may be in a different order. Don't just go to a certain line in a file and start editing. Instead search for the text you want to exit, just to be sure the correct text is edited on all connected systems.

Example:

Configuration files section from the man-page:

/etc/clusters

This file contains a list of tags to server names mappings. When any name is used on the command line it is checked to see if it is a tag in /etc/clusters (or the .csshrc file, or any additional cluster file specified by -c). If it is a tag, then the tag is replaced with the list of servers from the file. The file is formatted as follows:

<tag> [user@]<server> [user@]<server> [...]

i.e.

# List of servers in live

live admin1@server1 admin2@server2 server3 server4

Clusters may also be specified within the users .csshrc file, as documented below.

/etc/csshrc & $HOME/.csshrc

This file contains configuration overrides - the defaults are as marked. Default options are overwritten first by the global file, and then by the user file.

Environment:

ClusterSSH can be used to any system running the SSH daemon.

• ClusterSSH RPM for openSUSE 10.2/10.3 and SLE10 are available through the openSUSE Build-service: http://download.opensuse.org/repositories/home:/martijn/
• The ClusterSSH sourcecode can be downloaded from: http://sourceforge.net/project/showfiles.php?group_id=89139

See also: Software Distribution

[Dec 12, 2008] 10 Ultimate Rules for Effective System Administration by Vivek Gite

The following 10 items are guidelines more than rules, that I have learned over the years doing intensive work on the IT infrastructure. These guidelines are mostly common sense and can be helpful for anybody who administers an IT system, including Linux/Windows Administrator, Network Administrator and DBA.

1. Keep it simple. In technology environment, keeping things simple takes lot more effort and maturity than keeping it complex. As an administrator, when it comes to implementing a particular functionality or solving a problem, there are always several options available. It is best to learn all the available options, including the complex ones to understand how it works. However while implementing, try to keep it as simple as possible. The option you choose should be simple and have the following characteristics:
   • Easy to maintain in a long run
   • Does not add additional over head to the system
   • Solves the primary business/technical problem

   Whenever you are in a dilemma of whether to choose a bleeding edge technology or proven technology that has been around for a while, always go with the proven technology for production implementation.

   Everything should be made as simple as possible, but not simpler. - Albert Einstein

2. Backup regularly. Is both your personal laptop and servers at work, getting backed up regularly? If not, stop everything you are doing now and implement a backup solution on those systems immediately. Seriously! Start planning for your backup right now. Everybody knows that backing up data on a regular basis is critical. Only those who got burnt out on few occasions without having a backup, really understands the importance of having a reliable backup solution. Don't learn the importance of backup after loosing your critical data.

   It is only a matter of time, when you'll be in a situation where a system crashed, data got deleted accidentally or laptop with critical data is lost. Spend quality time and implement a reliable backup solution for both your personal laptop and servers at work.
    

3. Test your backup regularly. I could've combined this as part of rule#2. But, I strongly believe testing the backup deserves special attention. I have seen on several occasions, where administrators thinks they have a valid backup, only to find out during disaster, they couldn't restore from the backup successfully. A backup solution without testing it on a ongoing basis is only as good as not having the backup. Just having faith in the backup that it will work is not good enough. You should have a process to test your production backup every month. You'll have a peaceful sleep at night just by implementing rule#2 and #3.
4. Proactive Monitoring. Are you always working in a fire fighting mode? Is your users calling you to indicate that a system is down or having problem? Experienced administrators knows that they should spend majority of their time implementing solutions to avoid problems, instead of fixing the problems after it happens. Make sure to implement a strong monitoring solution that will monitor and alert you about a problem before it happens. You should never be solving the same issue more than once. Following two points will help you to achieve the proactive monitoring.

   Sit and identify all the equipments, services and applications that needs to be monitored through out the enterprise. Define an acceptable warning and critical levels for those systems. Define who should be notified and how often they should be notified and the method of notification. Once you have these identified, spend time implementing a monitoring system.

   Despite proactive monitoring, there will be times when you'll be putting out a fire. Once you put off the fire, the first question you should ask yourself: How I could've avoided this issue from happening? Once you have the answer for that, make sure to implement an appropriate monitoring solution to prevent this particular incident from happening in the future.
    

5. Document Everything. You should document everything that you perform on the system. This is not a pleasant topic for administrators, as most of us hate to write documentation. An experienced administrators knows that documenting the environment and his work is key for his success and growth. I'm not talking about spending several hours creating a huge document with all fancy formatting.

   Anytime you implement a solution or fix a problem, just scribble down the high level steps that was performed in a text file. You can simply copy/paste the commands you've executed along with one line description. This in itself is a huge step towards documentation for most administrators who are not used to documenting their work.

   Following are some of the primary reasons for documenting every technical activity performed by administrators:

   • Don't learn the same topic twice. When you implement something new, you have spend enough time learning the technology and understanding the steps to implement it on your specific environment. During this process, write down all the steps and refer to those steps the next time you want to perform the same task on a different server.
   • There will be situations when you want to delegate tasks to others. For e.g. when you are going on vacation or when you want to delegate a particular routine task to a junior administrator who is eager to learn. If you had the practice of consistently documenting everything, you can simply pass those text file documentation to the other administrator.
   • Sharing your knowledge with others is one of the efficient ways to grow your knowledge. So, document everything and share with others.
   • Don't waste the valuable RAM space on your brain by remembering everything. Instead off-load some of the items from your brain's RAM to a simple text file and use your brain's RAM to explore new technology.
      
6. Plan and Execute it well. When you are implementing a solution, have a clear plan on what you will do next and when. You should be Project Manager for your own tasks and projects. I.e Analyze all the potential risks involved in implementing a solution. Make sure to give sufficient time to test a particular solution. Come up with a clear test plan and get your users involved in testing process. On your next assignment, try the following and see the benefits for yourself. This forces you to think about all the possible scenarios even before you start the project.
   • Write down the objective of your project. I.e What is the problem you are trying to solve. What is your success criteria on this project/task?
   • List down all the tasks required to complete this particular activity and assign appropriate dates for it.
   • Even when nobody is requesting you to complete a project by certain date, hold yourself responsible by putting a completion date for your project/task.

   When you really get this implemented on the projected date, give yourself a pat on the back and enjoy your accomplishment. Planning and executing projects well on a consistent basis could potentially become one of a huge motivation factor for administrators to start taking up bigger and complex technology projects.
    

7. Use Command Line more than GUI. Use the command line as much as possible. Whether you are configuring a VLAN on a switch or setting up LDAP/NIS authentication on a Linux server, always use the command line instead of GUI. Following are the advantages of using command line.
   • You can do things very quickly on command line.
   • GUI prevents you from understanding and learning the functionality happening behind the scenes.
     Repetitive things can be automated easily using command line.
   • Your brain will have fun and Thank you for it.
      
8. Automate repetitive tasks. If you perform a task more than once, you should find a way to automate it. It may be very tempting to do the repetitive tasks manually, as can complete the task quickly and know the exact steps to perform the task. But, avoid this temptation and spend some extra effort in automating the task, which will free-up your mind from thinking about that routine tasks. Once you've automated the tasks, you can use your time effectively in learning other new fun stuff.
    
9. Support your users and developers. Administrators are technically very sophisticated and sometimes get frustrated with end-users who don't understand technology. But, keep in mind that you have your job mainly because they don't understand technology and need your expertise. When user reports an issue that is totally not related to the system and mainly because of user-error, be nice to the person and explain in a non-technical term about why this is not a system issue.

   Sometimes developers may deploy something on the server causing some undesirable results. Don't get mad on them and blame the problem on the developer. Instead, help the developer to identify the root cause of the problem, by providing sufficient data from the system to narrow down the problem.
    

10. Keep learning and have fun. If you have mastered the skill on how to do all the above 9 items effectively, you'll have more free time on your hand. Keep learning all the times. Anytime someone reports an issue, be curious and treat it as an opportunity to learn something new. Once in a while step aside your computer and spend quality time with your family. On top of all, have fun and enjoy doing the system administration activities.

    Live as if your were to die tomorrow. Learn as if you were to live forever. --Mahatma Gandhi

[Oct 27, 2008] sysprof 1.0.11

About: Sysprof is a sampling CPU profiler that uses a Linux kernel module to profile the entire system, not just a single application. It handles shared libraries, and applications do not need to be recompiled. It profiles all running processes, not just a single application, has a nice graphical interface, shows the time spent in each branch of the call tree, can load and save profiles, and is easy to use.

Release focus: Minor bugfixes

Changes: This version compiles with recent kernels.

Author: Søren Sandmann

[Oct 23, 2008] Another File Integrity Checker 2.12 by Gerbier Eric

Perl-based, so modifiable by admin. See also Integrity Checkers

About: afick is another file integrity checker, designed to be fast and fully portable between Unix and Windows platforms. It works by first creating a database that represents a snapshot of the most essential parts of your computer system. You can then run the script to discover all modifications made since the snapshot was taken (i.e. files added, changed, or removed). The configuration syntax is very close to that of aide or tripwire, and a graphical interface is provided.

Changes: The code now works with perl 5.10. On Windows, afick_planning now sends a report instead of a summary and uses the "LINES" macro. On Unix, a new MOUNT macro allows you to use a remote database in afick_cron. Udev files were removed from scan. The tar.gz installer was recoded to display better diagnostics.

[Sep 22, 2008] werc 0.9.2   by Lost Goblin

About: Werc is a minimalistic content management system implemented as a collection of shell scripts written in the rc shell and taking advantage of other Plan 9 tools available on *nix systems as part of "Plan 9 from User Space". It is designed to be fast, simple, convenient, and easily extensible (you can easily integrate your own dynamic content with the rest of the site). It should handle well both small and big sites, including seamless multi-domain handling. All data is stored as plain text files without need of a database and there is support for blog directories with automatic RSS feed generation (including feed aggregation).

Changes: More minor fixes and improvements were made to prepare for 1.0. The configuration system is now more modular, flexible, and intuitive. Portability was expanded and setup was made easier. A minimal Web server in rc was added. Various cleanups of blog and .txt handling code were made. Many bugs were fixed, especially in sitemap generation code.

[Sep 17, 2008] Logapp 0.10  by Michael Brunner

About: Logapp is a wrapper utility that helps supervise the execution of applications that produce heavy console output (e.g. make, CVS, and Subversion). It does this by logging, trimming, and coloring each line of the output before displaying it. It can be called instead of the executable that should be monitored; it then starts the application and logs all of its console output to a file. The output shown in the terminal is preprocessed, e.g. to limit the length of printed lines and to show the stderr output in a different color. It is also possible to automatically highlight lines that match a certain regular expression. The output is therefore reduced to the necessary amount, and all important lines are easy to identify.

Changes: There have been many code changes and cleanups all over the project. Additionally, multiplier prefixes for numerical parameters have been added, and the log file formatting has been improved.

[Sep 11, 2008] The LXF Guide 10 tips for lazy sysadmins Linux Format The website of the UK's best-selling Linux magazine

A lazy sysadmin is a good sysadmin. Time spent in finding more-efficient shortcuts is time saved later on for that ongoing project of "reading the whole of the internet", so try Juliet Kemp's 10 handy tips to make your admin life easier...

1. Cache your password with ssh-agent
2. Speed up logins using Kerberos
3. screen: detach to avoid repeat logins
4. screen: connect multiple users
5. Expand Bash's tab completion
6. Automate your installations
7. Roll out changes to multiple systems
8. Automate Debian updates
9. Sanely reboot a locked-up box
10. Send commands to several PCs

[Aug 25, 2008] pssh 1.4.0  by Brent N. Chun -

About: pssh provides parallel versions of the OpenSSH tools that are useful for controlling large numbers of machines simultaneously. It includes parallel versions of ssh, scp, and rsync, as well as a parallel kill command.

Changes: A 64-bit bug was fixed: select now uses None when there is no timeout rather than sys.maxint. EINTR is caught on select, read, and write calls. Longopts were fixed for pnuke, prsync, pscp, pslurp, and pssh. Missing environment variables options support was added.

[Aug 22, 2008] LBackup 0.9.8q7  by lucidsystems

About: LBackup is a simple backup system aimed at systems administrators who require reliable backups with minimum fuss. It is configured with configuration files, and the backup is started from the command line. It has been tested for over 2 years. Backups can be to local media, or to remote media via one or more networks. The networks may be private LANs, WANs, or sets of untrusted public networks such as the Internet.

Changes: This release adds improved example backup action scripts, support for rsync v3 and rsync v3 compiled with additional MacOS X metadata support patches, and the ability to specify the sender name and return address used for email reporting within the mail configuration file. Details on compiling and installing rsync3 are available from the download page.

[Jul 20, 2008] Control Your Linux Server Remotely with GnuPG, procmail, and PHP

How many times has this happened to you: you want to access a remote server, but you can't because it is behind a firewall? I frequently found myself in such a situation when I needed to access my Internet-connected server running Linux, so I thought of a system where I could start controlling my server remotely via a simple email.
 

Of course, this solution had one crucial requirement: it had to be secure. The server had to respond only to senders who were identified and authorized, and the command sent to the server, along with its related output, had to travel over the wire in encrypted form. To meet these security requirements, I used the free GNU Privacy Guard (GnuPG) and some asymmetric encryption techniques (See Sidebar 1. Asymmetric Cryptography in This Solution).

GnuPG is the open source implementation of OpenPGP security software. To implement the message encryption, I employed a patent-free algorithm contained in GnuPG called the ElGamal encryption system.

This article demonstrates how my system enables you to remotely control your server in batch mode with signed and encrypted emails. It uses a fictional, authorized e-mail sender (sender@example.com) and an example remote server (machine@example.com) for the sender to inquiry. The server will run Debian Linux.

The Process Schema
The following are the steps involved in the process of controlling a server via email:

1. Create the list of all authorized command senders (e.g., sender@example.com).
2. Let the sender generate private/public key pairs with GnuPG.
3. Generate a private/public key pair with GnuPG for the server, which has the email address machine@example.com.
4. Import the sender's public key on the server keyring and server's public key on the sender's keyring.
5. Let the sender sign and encrypt the command to run remotely on the server, embed it in an email, and send it to the server address.
6. Let the server download the email messages and process them with a script as follows:
   1. Verify whether the sender is authorized.
   2. Decrypt and run the command.
   3. Capture output from stdout and stderr, possibly killing hanging commands after a reasonable period of time.
   4. Sign and encrypt the outputs, embed them in an email, and send the answer back to the sender address.
7. Let the sender read the server outputs, verifying and decrypting its reply.

[Jul 11, 2008] ttyutils 1.1.2  by xiaohu

About: Ttyutils is a suite of UNIX terminal tools. It includes ttyexec, ttylook, ttyadmin, and a few extension programs. ttyexec executes command in a pseudo terminal, captures all stdandard output from the command, and passes it to a built-in virtual terminal emulator and real terminal. ttylook is similar to the BSD watch(1) program, but has fewer limits and can interact with existing ttyexec instances. ttyadmin is a administrator tool which uses an ncurses interface to view and control existing ttyexec instances.

Changes: This version works on AIX.

[ Jul 10, 2008] xhelper 0.0.04  by Jeff Owens

About: xhelper is a tool to automate and control the desktop. It can resize windows, move windows, feed keystrokes to windows, run programs, and automate a group of programs

[Jul 10, 2008] tudu 0.3.1  by meskio

About: ToDo is a list manager in ncurses with a hierarchical representation of tasks. Each task has a title, a long text description, and a deadline (tudu warns you when the date is close). There are categories and priorities.

Changes: The tudurc file was moved from /usr/local/share/tudu to /usr/local/etc.

[May 6, 2008] Key Scripter 1.4 Andrei Romanov

About: Key Scripter listens to key press/release events from a keyboard or a mouse and sends fake key events to an X display. It supports gaming keypads such as the Nostromo SpeedPad and allows the creation and usage of complicated key scripts for games and other applications.

Changes: This release contains a couple of minor bugfixes and an improved example configuration. The development status of Key Scripter is now stable.

[May 6, 2008] Ortro 1.3.0  by Luca Corbo

PHP based

About: Ortro is a framework for enterprise scheduling and monitoring. It allows you to easily assemble jobs to perform workflows and run existing scripts on remote hosts in a secure way using ssh. It also tests your Web applications, creates simple reports using queries from databases (in HTML, text, CSV, or XLS), emails them, and sends notifications of job results using email, SMS, Tibco Rvd, Tivoli postemsg, or Jabber.

Changes: Key features such as auto-discovery of hosts and import/export tools are now available. The telnet plugin was improved and the mail plugin was updated. The PEAR libraries were updated.

[Jan 24, 2008]cgipaf

The package also contain Solaris binary of chpasswd clone, which is extremely useful for mass changes of passwords in mixed corporate environments which along with Linux and AIX (both have native chpasswd  implementation) include Solaris or other Unixes that does not have chpasswd utility (HP-UX is another example in this category).   Version 1.3.2 now includes Solaris binary of chpasswd which works on Solaris 9 and 10.

cgipaf is a combination of three CGI programs.

• passwd.cgi, which allow users to update their password,
• viewmailcfg.cgi, which allows users to view their current mail configuration,
• mailcfg.cgi, which updates the mail configuration.

All programs use PAM for user authentication. It is possible to run a script to update SAMBA passwords or NIS configuration when a password is changed. mailcfg.cgi creates a .procmailrc in the user's home directory. A user with too many invalid logins can be locked. The minimum and maximum UID can be set in the configuration file, so you can specify a range of UIDs that are allowed to use cgipaf.

[Nov 27, 2007] Classification of Corporate Psychopaths

Unix administrators need to be aware of this pretty widespread danger.

• The Danger of Micromanagers and Micromanagement

• The Manipulator Bosses (aka Mayberry Machiavellians )

• Narcissistic Managers

• Paranoid Managers: Type 2 Corporate Psychopaths

• Bully Managers in IT Workplace: Type 1 of Corporate Psychopaths

[Jun 11, 2007] developerWorks AIX and UNIX Technical library view

System Administration Toolkit: Standardizing your UNIX command-line tools Examine methods for standardizing your interface to simplify movements between different UNIX(R) systems. If you manage multiple UNIX systems, particularly in a heterogeneous environment, then the hardest task can be switching between the different environments and performing the different tasks while having to consider all of the differences between the systems. This article does not cover specific differences, but you'll look at ways that can provide compatible layers, or wrappers, to support a consistent environment.
System Administration Toolkit: Backing up key information Most UNIX(R) administrators have processes in place to back up the data and information on their UNIX machines, but what about the configuration files and other elements that provide the configuration data your machines need to operate? This article provides detailed information on techniques for achieving an effective and efficient backup system for these key files.
Take a closer look at OpenBSD OpenBSD is quite possibly the most secure operating system on the planet. Every step of the development process focuses on building a secure, open, and free platform. UNIX(R) and Linux(R) administrators take note: Without realizing it, you probably use tools ported from OpenBSD every day. Maybe it's time to give the whole operating system a closer look.
System Administration Toolkit: Managing NIS services for authorizations Examine how to set up, configure, and update a Network Information System (NIS) installation for sharing information, and learn how NIS can be merged with other solutions, such as files and Domain Name System (DNS), to provide subnet, network, and worldwide data sharing facilities. In a large UNIX(R) network, the ability to share information among the many systems helps to alleviate many problems, such as sharing permissions across different systems with Network File System (NFS), or simply providing a single login for the entire network.
System Administration Toolkit: Migrating and moving UNIX directory trees Occasionally, you need to copy around an entire UNIX(R) directory tree, either between areas on the same system or between different systems. There are many different methods of achieving this, but not all preserve the right amount of information or are compatible across different systems. This article discusses the various options available for UNIX and how best to make them work.
System Administration Toolkit: Migrating and moving UNIX filesystems Learn how to transfer an entire file system on a live system, including how to create, copy, and re-enable the new file system. If you have a UNIX(R) disk or system failure or simply fill up your file system, then you need to create a new partition and file system and copy over the contents. You might even need to mount the new partition in place to preserve the location of vital files and components. To add further complications, you need to do this on a live system, where you'd need to preserve file permissions, ownership, and possibly named pipes and other components. Effectively transferring these components and retaining all of this information is a vital part of the migration process.
System Administration Toolkit: Monitoring disk space and usage Look at methods for determining disk usage across multiple UNIX(R) systems and how to create a simple warning system to alert you of potential problems. Keeping an eye on your file systems and ensuring they don't fill up is a trivial, but vital, process in the day-to-day management of your UNIX systems. In this article, you'll look at methods for keeping an eye on disk space, discovering which files, users, or applications are using up the most space, and how to make use of quotas and other solutions to find the information you need.

[Sep 5, 2006] LISA '06 Technical Sessions

Is Entropy Winning? Drowning in the Data Tsunami
Lee Damon, Sr. Computing Specialist, University of Washington; Evan Marcus, CTO and Founder, Aardvark Technologies, Ltd

We're drowning under a wave of data and are oblivious to it. As data space expands we will start losing track of—and thus losing—our data. Archival backups add complexity to this already confusing situation. Then we toss in security and availability issues for some spice. Where is this going, and how can we handle it in the face of millions of gigabytes of "old cruft"?

Lee Damon has been a UNIX system administrator since 1985 and has been active in SAGE since its inception. He assisted in developing a mixed AIX/SunOS environment at IBM Watson Research and has developed mixed environments for Gulfstream Aerospace and QUALCOMM. He is currently leading the development effort for the Nikola project at the University of Washington. He is past chair of the SAGE Ethics and Policies working groups and he chaired LISA '04.

Improv for Sysadmins
Bob Apthorpe, St. Edward's University; Dan Klein, Consultant

Have you ever seen "Whose Line Is It Anyway?" and marveled at the actors? Although it may not be obvious, improvisors and sysadmins have a lot in common! We both have to think on our feet, often "winging it," and both groups actively practice ad hoc problem-solving. Management calls it "thinking outside of the box," and we say "welcome to our world."

From the outside, good improv looks like a lot of fun (it is!), and good system administration looks easy and fun (why else do we have toys in our cubes?). Both groups have fun because they both create environments to bring people together and make good things happen. At its core, improvisation is not about being funny so much as it is about carefully listening, clearly expressing oneself, and confidently making decisions and taking action. So is system administration. Our goal is to get paid to play.

This session will relate improvisational acting concepts to system administration. Improv can show us how our responses to others can be misinterpreted and, more important, how to change that by producing a constructive dialogue. Understanding your audience and their context can make everything move much more smoothly! Other topics will include the role body language plays in communication, especially in the communication of status, and the importance of observation and attention to detail, with an emphasis on "active listening," saying "yes, and . . . ," and other observation/communication techniques.

The session concludes with a question-and-answer period and additional improv demonstrations as time permits. We won't try to be funny, but we know that you'll enjoy learning some incredibly valuable improvisational techniques.

Bob Apthorpe is a system administrator at St. Edward's University in Austin, Texas. He first attended LISA in 1998 and transferred from Web development to international operations at Excite.com shortly thereafter. His current interests include risk assessment, operations-friendly software development, and improvisational theatre. Bob is a proud member of the troupe "Improv for Evil" but his wife loves him anyway.

Dan Klein began his life of crime in 2nd grade, when he was caught with a pack of firecrackers. Since then his brushes with authority have been sporadic but relentless, but have not managed to deny him a security clearance, a job, or his well deserved reputation as an off-the-wall maverick. His computer experience has included simulation and process control, the internals of almost every UNIX kernel released in the past 28 years, and graphical user interface management systems

The Future of System Administration: How to Stop Worrying and Learn to Love Self-Managing Systems
Alva L. Couch, Associate Professor of Computer Science, Tufts University

The profession of system administration is currently threatened by many forces, including self-managing products that seem to obsolete the system administrator, a lack of upward mobility paths for professional system administrators, and a growing trend toward outsourcing system administration and related tasks. In this talk, I explore how ongoing changes in the systems we manage can drive positive changes in the profession. The bad news is that the way we prepare system administrators today is woefully inadequate for managing the systems of the future, and we must also rise to the challenge by learning to interact with the systems we manage at a very different level than we are currently trained to do.

System Administration: Drowning in Management Complexity
Chad Verbowski, Software Architect, Microsoft Research

Systems management is challenging because it requires administrators to understand and specify the desired state of each system based on their knowledge of the network, hardware, security, distributed applications, and workloads in their environment. Yearly increases in the variation, complexity, and volume of systems management tasks are outpacing our ability to hire qualified administrators to maintain our IT environments.

This talk presents a new black-box approach for reducing the complexity of systems and security management faced by administrators. The goal is to show this as a scalable alternative compared with current signature and declarative management approaches. Real world data, examples, and solutions are used to illustrate the scope and impact of troubleshooting, malware detection, and change management problems, as faced by today's systems administrators.

Chad's research on network management led to a job offer from MFS Datanet (eventually swallowed by Worldcom) in Silicon Valley. After that, a stint at Cisco Systems followed and then he took a leap (of faith) to a network management start-up-->. He eventually arrived at Microsoft in 1998.

Originally hired to work on the notorious Java VM, he worked on the headless support in Windows 2000, then ran the development team for the first release of Microsoft Operations Manager before finding his niche at Microsoft Research. At MSR Chad cofounded the Cybersecurity and Systems Management research group, where he focuses on his area of interest: reducing complexity in software.

alphaWorks IBM Autonomic Task Manager for Administrators Overview

IBM Autonomic Task Manager for Administrators (ATMA) is a spreadsheet-based scripting environment for quickly composing and automating system management tasks. With this environment, administrators can execute management commands and combine these commands to create ad hoc scripts and visualizations of system management information. The basic building blocks for these tasks are spreadsheet templates that are customized with a simple drag-and-drop interface. Autonomic Task Manager for Administrators enables the insertion of GUI, visualization, or system management components into cells of a spreadsheet and customization of the cells in order to insert control logic for a system management solution. Just as in spreadsheets, data in the cells are automatically processed and updated; this feature allows real-time system data feeds.

Currently, Autonomic Task Manager for Administrators supports a variety of system management plug-ins, including Simple Network Management Protocol (SNMP), Secure Shell (SSH), and Java™ Management Extension (JMX). Using ATMA's component plug-in API, developers can build custom components that can be used to develop tools using different management APIs; one such API interfaces to IBM Autonomic Integrated Runtime Environment, which allows communication with resources based on Web Services Distributed Management (WSDM). ATMA can also interact with any Java object.

Autonomic Task Manager for Administrators significantly reduces script creation time with its familiar spreadsheet interface and building blocks made from templates. The package includes the executable, relevant plug-ins, installation instructions, and user documentation.

How does it work?

System administrators or value-added re-sellers (VARs) can use the spreadsheet-based scripting environment to build scripts incrementally, potentially starting from templates and using standard components. After a script is developed, it can immediately become available for use or further customization.

Depending on the specific administrative function being addressed, this technology helps to tie together the various underlying components. The cells may contain numbers and text, as in most spreadsheets, as well as GUI objects such as buttons and checkboxes, visualization objects such as plots and pie charts, programming objects such as collections and timers, and system objects such as JMX, SNMP, etc. These objects can be either created by the user or assigned to cells as a result of evaluating expressions that define the functional relationship between objects in various cells.

Examples are included in the documentation provided with this package.

alphaWorks Remote System Management Tool Overview

Remote Server Management Tool is an Eclipse plug-in that provides an integrated graphical user interface (GUI) environment and enables testers to manage multiple remote servers simultaneously. The tool is designed as a management tool for those who would otherwise telnet to more than one server to manage the servers and who must look at different docs and man pages to find commands for different platforms in order to create or manage users and groups and to initiate and monitor processes. This tool handles these operations on remote servers by using a user-friendly GUI; in addition, it displays configuration of the test server (number of processors, RAM, etc.). The activities that can be managed by this tool on the remote and local server are divided as follows:

• Process Management: This utility lists the process running on UNIX and Windows® servers. One can start and stop processes. Along with process listing, the utility also provides details of the resources used by the process.
• User Management: This utility facilitates creation of users and groups on UNIX servers; it also provides options for listing, creating, deleting, and modifying the attributes of users and groups.
• File Management: This utility acts as a windows explorer for any selected server, irrespective of its operating system. One can create, edit, delete, and copy files and directories on local or remote servers. Testers can tail the remote files.

How does it work?

This Eclipse plug-in was written with the Standard Widget Toolkit (SWT). The tool has a perspective named Remote System Management; the perspective consists of test servers and a console view. The remote test servers are mounted in the Test Servers view for management of their resources (process, file system, and users or groups).

At the back end, this Eclipse plug-in uses the Software Test Automation Framework (STAF). STAF is an open-source framework that masks the operating system-specific details and provides common services and APIs in order to manage system resources. The APIs are provided for a majority of the languages. Along with the built-in services, STAF also supports external services. The Remote Server Management Tool comes with two STAF external services: one for user management and another for proving system details.

[Mar 20, 2006] Linux Today - Dell, Novell to Manage Red Hat, SUSE Dell, Novell to Manage Red Hat, SUSE by Jacqueline Emigh

At Novell BrainShare today, Novell and Dell joined hands in launching a software product for remote management of servers running either Novell's own SUSE Linux or a competing Linux distribution put out by Red Hat, Novell's long-time archrival.

Pegged for availability on April 19, the jointly developed software for Dell PowerEdge servers will be dubbed Novell Zenworks 7 Linux Management - Dell Edition, said Jason Werner, a Novell product marketing manager, during a pre-briefing with Linux Today.

The upcoming software package "takes our Zenworks Linux management product and adds a layer of Dell-specific management," according to Werner.

The new Dell Edition of Zenworks will be geared mainly to organizations with multiple remote PowerEdge servers, "where you wouldn't necessarily have Linux expertise (on site) at all locations," Werner said.

Target customers include organizations engaged in server consolidation as well as those that are migrating servers from Microsoft Windows to either SUSE or Red Hat.

The Dell Edition will be the first iteration of Zenworks tailored to managing both of these two major distributions of Linux. Novell did not work directly with Red Hat in creating the product, he said.

But together with Dell, a long-time Red Hat ally, Novell has been tweaking Zenworks to support Red Hat environments.

Already tested by Novell on both SUSE and Red Hat Linux, the product will bring together Zenworks features such as remote provisioning and inventory management with capabilities specific to Dell's PowerEdge platform. The Dell-specific tools will deal with areas ranging from bios administration to remote access management.

Novell Zenworks 7, Linux Management - Dell Edition will not replace the Dell OpenManage software that has shipped for some time with PowerEdge servers, Werner said.

"But [the Zenworks] software will cover the entire [server] lifecycle, including pre-OS and RAID," he told Linux Today.

Through the new Dell edition, administrators in remote locations will have access to detailed bios and firmware information. "You'll be able to run queries to find out what has been deployed on a server," he added.

Administrators will also be able to make configuration changes remotely, repurposing a system "simply by changing it from a Web server to a storage server, for example," according to the Novell executive.

Configuration changes made on one server can be quickly promulgated among other servers that perform the same roles, reside in the same geographies, or have the same models and makes.

"You can even adjust the utility partition on the hard drive when no OS is present," Werner said. Consequently, he suggested, organizations can be more certain that configuration settings will remain consistent among a group of servers.

On the other hand, the product will also support capabilities built into Zenworks for assigning administrative rights only to authorized individuals, Werner said.

The Dell edition of Zenworks will be sold separately from PowerEdge servers. The product will not be available through Novell or its resellers.

Instead, sales will be performed exclusively through Dell, according to Werner.

As some analysts see it, today's announcement by Novell and Dell reflects an increasingly visible industry-wide trend toward better Linux management tools.

"It's really obvious that [Linux management] tools are getting broader, more sophisticated, and better able to integrate with outside systems," said Andy Mann, a senior analyst at Enterprise Management Associates (EMA), during another interview.

But although Hewlett-Packard and IBM Tivoli have accomplished some penetration of the Linux management market, much of the innovation so far has come from smaller vendors such as Levanta, Velocity Software, and Opsware, according to Mann.

But many Linux administrators have relied mainly on tools from Novell and Red Hat. "So it's good to see a company such as Novell getting behind some new management software," added the analyst, who is also the author of a recently released report from EMA called "Get the Truth on Linux Management."

Co-sponsored by Open Source Development Labs (OSDL) and Levanta, one of the OSDL's members, the study of over 200 Linux companies dismisses earlier claims that Linux has a higher Total Cost of Ownership (TCO) as "no longer true."

Mann also told Linux Today that support for other Linux distributions could prove useful to Novell. "Zen is [basically] open source software, [but] with some proprietary components. It should be in Novell's best interests to support as many other distributions of Linux as it can, to further the growth of Linux," he said.

"Support for other distros could only help Novell. It certainly couldn't hurt," concurred David Dennis, Levanta's director of marketing.

Dennis noted that many Linux customers are now seeking multi-distro support as a way of avoiding "vendor lock-in."

Levanta's management tools support both SUSE and Red Hat Linux, along with a "second tier" of distros such as CentOS and Asianux, according to the marketing director.

But Dennis also maintained that Linux management tools vary along a number of other lines, based on the administrative capabilities needed in particular types of deployments.

Novell has already been providing hefty Linux management support through its multiplatform Zenworks lineup, observed Fred Broussard, an IDC analyst, in another interview with Linux Today.

Broussard also pointed out that it isn't at all unusual for competitors in the computer industry to cooperate on some levels.

"We've heard a lot over the years about Novell and Red Hat having an adversarial relationship," according to the IDC analyst.

"But at the end of the day, Novell is going to do what its customers want. Novell is a very customer-centric company," Broussard told Linux Today.

Novell's Werner declined to comment one way or the other on whether other products supporting multiple Linux distributions are also in the works at Novell. "Not that we've made public comments on," Werner told Linux Today.

The upcoming Novell Zenworks 7 Linux Management - Dell Edition will be priced at $69 per license.

[May 23, 2005] NewsForge The Fifth Commandment of system administration By: Brian Warshawsky

If you're a good administrator, you pride yourself on developing a fundamental understanding of the systems you build. After a while, as you begin to comprehend the complete complexity that goes along with building and maintaining your infrastructure, the commands and procedures to control them become second nature. You have to look at the documentation less and less, until eventually people refer to you as a guru. Having this kind of understanding of your servers is important, but it does no good if you aren't available when something crashes. By creating detailed written policies detailing the ins and outs of your systems in advance, you can provide critical background information to your backup admin who can use it to restore functionality in your absence.

V. Thou shalt document complete and effective policies and procedures

In the past I found documented policies useful especially at two different times. The first is at the inception of a project. Before the system goes into production, sometimes even before the hardware is bought, detail in writing exactly what you need the server to accomplish, where its performance bottlenecks will be, and what your intentions are to correct these issues. This will allow you (and upper management!) to know that your time is not being spent chasing a fantasy implementation that will never work. It also helps you to better understand the nature of the beast you're building. If anything goes wrong during the installation and configuration process (and something always does) you'll be better prepared to deal with it simply due to the better understanding you've obtained by mapping everything out beforehand. At this point you don't need anything more than an outline (sometimes in the form of a project plan) and a few diagrams to guide you. If it's a much larger-scale implementation though, you'll need a detailed project plan dividing the entire process into phases. For instance, a large-scale Beowulf cluster would require a detailed project plan, while a new intranet Web server might only require a brief outline of configuration tasks and a diagram showing how it's integrated into network.

The second time that these policies are important is after the server has finished configuration and is ready to go into a production environment. At this point, before it is rolled out, you should take some time to create some detailed step-by-step documents explaining the backup restoration process, the steps necessary to restart a service (or just make a list of important services that might need to be restarted, depending upon the experience of your back admins) and anything else that might be helpful. Just remember that you won't always be available to fix something; having detailed instructions for common problems or routine exercises can make the difference between 10 minutes of downtime and a week and a half if you are unavailable.

The commandments so far:
I. Thou shalt make regular and complete backups
II. Thou shalt establish absolute trust in thy servers
III. Thou shalt be the first to know when something goes down
IV. Thou shalt keep server logs on everything
V. Thou shalt document complete and effective policies and procedures

[Nov 02, 2005] MValent to release updated apps management software - Computerworld

Two companies offer products that are similar to mValent's offering: Relicore Inc. in Burlington, Mass., and Collation Inc. in Redwood City, Calif. But mValent seems to be unique because it focuses on the entire life cycle of managing applications, from predeployment into deployment, Drogseth said.

 (COMPUTERWORLD) - MValent Inc. next week plans to take the wraps off the latest version of mValent Integrity, software that's designed to automate the configuration and management of application and Web servers and other application components.

The software, which will be available on Monday, starts at $60,000 and will be deployed by several existing mValent customers, including WorldWinner Inc., an online gaming company in Newton, Mass., and State Street Corp., a financial services company in Boston, according to executives at the companies.

... ... ...

State Street expects to upgrade to mValent Integrity soon, said Joseph Kennedy, State Street's vice president of IT. State Street wants to stay current with the product, since prior versions have reduced the time needed to debug new application configurations. State Street uses MValent software to monitor a variety of systems that affect its 19,000 employees, he said.

Kennedy said mValent has helped the financial services firm add scalability to its application infrastructure and resolve problems with configurations, something he called "invaluable." State Street has been able to expand the application environments it maintains without adding system administrators, he said.

MValent Integrity appears to be unique in the market, said two analysts, Jean-Pierre Garbanim at Forrester Research Inc. and Dennis Drogseth at Enterprise Management Associates.

[Oct 7, 2005] mValent ¦ Powerful Change Control

mValent Integrity tracks changes to deployed servers and monitors configuration drift alerting IT teams to potentially critical problems. By comparing application environments in mValent Integrity for differences in granular configuration items, IT teams rapidly isolate root causes of production incidents. These teams can then model fixes to problems to validate their impact and automatically deploy them.

• Rich Compare Capabilities – mValent Integrity's Compare function aids troubleshooting by quickly pinpointing differences between multiple server instances or across infrastructure stacks representing different application environments.
• Versioning and Rollback - Running 'snapshots' of application infrastructure environments, plus the ability to recover quickly from unwanted changes.
• Tracking and Alerts - Knowing when a change has been made - no matter what changed - and accurately reporting on the specific properties before and after the change, gives IT teams early warning on potential problems.
• Point-in-Time Views - By keeping a running record of changes to a granular level, mValent Integrity reports on all changes that occurred between two points in time, or show that no unapproved changes took place.
• Audit Reports – Show the changes made to an individual server or a whole production environment by time period or by user.

The Book of Webmin

Webmin is a web-based interface for system administration for Unix. Using any browser that supports tables and forms (and Java for the File Manager module), you can setup user accounts, Apache, DNS, file sharing and so on.

Webmin consists of a simple web server, and a number of CGI programs which directly update system files like /etc/inetd.conf and /etc/passwd. The web server and all CGI programs are written in Perl version 5, and use no non-standard Perl modules.

Linux.com Automounter madness

"The world of automounters is a confusing one. For one thing, a single automounter wasn't enough, so there are two of them for Linux, called 'amd' and 'autofs'. While it's easy to say 'well, just pick one and go on your way', many environments have demands that will require both, and both serve different purposes and have different strengths and weaknesses. The automounter world is not so cut and dry. In this article, I'll give a light overview of what amd and autofs look like, what their respective purposes are in life, and go over some of the common configuration options for each. Later, I'll spice things up by including use cases where one will work better than the other and more advanced features available to you as an administrator.

"Automounters can be a real pain in the neck. Admins don't like things that are a pain in the neck, so one should be able to gather that if we're putting up with these things, they must be pretty darn useful--and they are. For one thing, automounters can get their mounting information from centralized 'maps', which can be flat files, or even NIS maps or sections of an LDAP directory. This is far easier than editing 300 /etc/fstab files for different static NFS mounts..."

Related Stories:
LinuxWorld.com.au: Mounting file systems(Feb 17, 2001)
LinuxFocus.org: automount and autofs (Feb 11, 2001)

[Dec 17, 2003] Slashdot Automating Unix and Linux Administration

Learn to script (Score:4, Interesting) by holden_t (444907) <holden_t@NOspAm.yahoo.com> on Thursday October 09, @03:09PM (#7175570)

Certainly I haven't read the book but it looks as if Kirk is offering examples of how to write scripts to handle everyday gruntwork. Good idea. But I say to those that call themselves sys.admins, Learn how to script!!! I work at a large bankrupt telcom :) and it's amazing the amount of admins that don't have the slightest idea how to write the simplest loop. Or use ksh, bash, or csh's cmd history. Or vi. Maybe this is just a corporate thing. They were raised, in a sense, in a setting where all they had to do was add users and replace disks. Maybe they never learned how to do anything else. Back in '83 I took manuals home and poured over every page, every weekend for months. That didn't make me a good admin but it gave me a good foundation. From there I had to just halfway use my head (imagination?) and start writing scripts. Ugly? Sure. Did they get better? Of course! Now I play admin on 110+ machines, and I stay bored. Why? Because I've written a response engine in Expect that handles most of my everyday problems. I call it AGE, Automated Gruntwork Eliminator. There's no way I could have done this if I had just sat back and floated, not put in a bit of effort to learn new things.

Multiple Machines (Score:5, Interesting) by BrookHarty (9119) on Thursday October 09, @01:48PM (#7175005) (http://www.ironwolve.com/)

One of the problems we have, is when you have clusters with 100+ machines, and need to push configs, or gather stats off each box. On solaris, we run a script called "shout" that does a for/next loop that ssh's into each box and runs a command for us. We also have one called "Scream" which does some root privilege ssh enabled commands. Nortel has a nice program called CLIManager (use to be called CLImax), that allows you telnet into multiple passports and run commands. Same idea, but the program formats data to display. Say you wanted to display "ipconfig" on 50 machines, this would format it, so you have columns of data, easy to read and put in reports. Also, has a "Watch" command that will repeat a command, and format the data. Say you want to display counters. I have not seen an opensource program that does the same as "CliManager" but its has to be one of the best idea's that should be implemented in opensource. Basically, it logs into multiple machines, parses and displays data, and outputs all errors on another window to keep your main screen clean. Think of logging into 10 machines, and doing a tail -f on an active log file. Then the program would parse the data, display it in a table, and all updates would be highlighted. I havnt spoken to the author of CliManager, but I guess he also hated logging into multiple machines, and running the same command. This program has been updated over the years, and is now the standard interface to the nodes. It just uses telnet and a command line, but you can log into 100's of nodes at once. Wish I could post pics and the tgz file, maybe someone from Nortel can comment. (Runs on Solaris, NT and linux)

Re:Multiple Machines (Score:2) by Xzzy (111297) <sether@ t r u 7 h.org> on Thursday October 09, @04:21PM (#7176481) (http://tru7h.org)

> Nortel has a nice program called CLIManager (use > to be called CLImax), that allows you telnet into > multiple passports and run commands. Fermilab has available a tool called rgang that does (minus the output formatting) something like this: http://fermitools.fnal.gov/abstracts/rgang/abstrac t.html We use it regularily on a cluster of 176 machines. It's biggest flaw is it tends to hang when one of the machines it encounters is down. But it is free so I won't complain. :)

Multiple Machines in Parallel (Score:1) by cquark (246669) on Thursday October 09, @04:29PM (#7176572)

One of the problems we have, is when you have clusters with 100+ machines, and need to push configs, or gather stats off each box. On solaris, we run a script called "shout" that does a for/next loop that ssh's into each box and runs a command for us. We also have one called "Scream" which does some root privilege ssh enabled commands. While the serial approach of looping through machines is a huge improvement over making changes by hand, for large scale environments, you need to use a parallel approach, with 16 processes or so contacting machines in parallel. I wrote my own script, but these days the Parallel::ForkManager [cpan.org] module for perl does the process management part for you.

Re:Multiple Machines (Score:2) by Sevn (12012) on Thursday October 09, @04:57PM (#7176807) (http://www.dangpow.com/~sevn | Last Journal: Tuesday April 01, @07:18PM)

I do pretty much the same thing this way: Generate ssh key file. Put pub key file in $HOME/.ssh/authorized_keys2 on the remote machines. Have a text file with a list of all the names the machines resolve to. for i in `cat machinelist.txt`; do echo "running blah on $i"; ssh user@$i 'some command I want to run on all machines'; echo " "; done It comes in handy for stuff like checking the mail queues or doing a tail -50 on a log file. Mundane stuff like that. Everyone once in a while I'll do basically the same thing with scp instead. It can get as complicated as you want. I used a for loop like this to remount 150 /tmp dirs noexec and make the edits to fstab.

Re:Multiple Machines (Score:2) by drinkypoo (153816) <drink@hypeDEBIANrlogos.org minus distro> on Thursday October 09, @10:00PM (#7179637) (http://slashdot.org/ | Last Journal: Friday November 21, @04:31PM)

IBM also owns Tivoli Systems, which made something called TME10, the current name of which escapes me at the moment. TME10 uses CORBA (their ORB is now Java, but it used to be basically ANSI C plus classes, compiled with the microsoft compiler on windows and gcc on most other platforms. Lots of it was perl, some of it was shell, plenty of it was C. Methods called Perl scripts pretty damn frequently. The interface was completely configurable and not only could you customize them without purchasing any additional products (if you felt froggy) but they also sold products to make this easier to do. Last I checked this package ran with varying degrees of ability (but most operating systems were very well suppored) on all major Commercial Unices, BSDi, Linux, OS/2, NT, Novell, and a bunch of random Unices that most people have never heard of, and never had to. It was sometimes problematic but the fact is that it was incredibly cross-platform. It was a neat way to do system monitoring. It would be nice to develop something open source like that. I think that today it would not be all that difficult a task. I'd like to see all communications be encrypted, with arbitrary shapes allowed in the network in terms of who talks to who, and who has control over who, to reflect the realities of organizations.

Re:Multiple Machines (Score:0) by Anonymous Coward on Thursday October 09, @04:14PM (#7176396)

IBM has two solutions depending on the environment. PSSP under AIX will allow you to run distrbuted command across nodes with either a correct RSH config or SSH Keys with no passphrase. PSSP, also, allow for parrallel copy. Under Linux( and AIX actually) there is CSM which also allows for DSH with the same config requirements. You can do Parallel copy under CSM, but you have to be tricky with something like, "dsh headnode:/file /file" .

Re:Learn to script (Score:2) by Wolfrider (856) <kingneutron@yahoTOKYOo.com minus city> on Friday October 10, @08:10PM (#7187085) (http://wolfrdr.tripod.com/linuxtips.html)

O'Reilly's book helped me quite a bit. http://www.oreilly.com/catalog/bash2/ In addition, Debian has a new package called abs-guide that I haven't checked out yet. http://packages.debian.org/unstable/doc/abs-guide. html --I've written a bunch of helpful bash scripts to help me with everyday stuff, as well as aliases and functions. If you want, email me - kingneutron at yahoo NOSPAM dot com and put "Request for bash scripts" in the subject line, and I'll send you a tarball.

Might be useful... (Score:2) by Vrallis (33290) on Friday October 10, @12:22AM (#7180451) (http://krynn.penguinpowered.com)

This might very well be a book I'll pick up sometime. I'm always looking for more ideas. I maintain about ~170 remote Linux boxes (in our company's retail stores and warehouses), as well as our ~30 or so inhouse servers. I went through a lot of work to enable our rollout and conversion to go more smoothly. The network and methodology for users, printers, etc. is extremely simplified and patterened. For each of the 3 'models' of PCs we use, I have a master system that I produced. I used Mondo Rescue [mondorescue.com] to produce CD backups of these systems. These systems act as serial terminal controllers, print spoolers, routers, desktop system usage (OpenOffice, Mozilla, Kmail under KDE), and other functions as needed. When we need to replace a system, or rollout a new location, we grab a system, pop in the Mondo CD, and do a nuke restore. When done, we have a standard configuration user that we log in as. It runs a quick implementation script where you answer anywhere from 3-8 questions (depending on the system type and options), and it configures everything. All networking, users, sets up Kmail, configures all printers and terminals (we use Comtrol Rocketport serial boards), and so on. If the system is physically ready, we can have it ready software-wise in about 20 minutes (2 CDs to restore). Updates are done via a couple different methods. I use SSH (over our internal VPN, using key authentication) in scripts to do most updates. If I need to do anything major, such as recently updating Mozilla, we do a CD distribution. The users have a simple menu to take care of running the update for them, even with autorun under KDE. Just pop in the CD, and it automatically takes them into the menu they need. All logs are duplicated across the network to a central server, but intrusion is less likely as these systems sit on a private frame network. They do, however, have fully secured network setups, as we use cheap dial-up internet access as a backup in case the frame circuit goes down. I can't help but feel every day like this is just one big hack/kludge, but it works, works damned well, and was about half the cost of any other solution (i.e. higher end Cisco routers to handle various functions, and using Equinox ELS-IIs or the like...those pieces of crap never would work right, we finally pulled only 2 we had in use, and they are currently collecting dust in a storage cabinet). Needless to say, I am *always* looking for ideas to improve upon this.

[Oct  25, 2003] Cultured Perl Application configuration with Perl

The simple approach: Do it yourself (DIY)
Theoretically (and with the right tools!) anyone can build a configuration parser, right? The Perl Cookbook, for one, shows a quick implementation that provides a good start. So how hard can it be to write a configuration file parser if you begin with this kind of implementation?

Quite hard, actually, because this kind of project raises several more complex issues like these:

• Blank lines and comments in the configuration file
• Erroneous lines (like misspelled keywords), and the question of which are critical and which can be ignored
• The probability that you may have to write your own parser, because you are likely to need a variety of different data structures (booleans, scalars, arrays, and hashes, for example)
• Multiple configuration files
• Variable defaults
• Integrating command-line options with the file configuration and controlling how they interact
• Educating users in yet another DIY configuration file format (This usually goes something like: "This will work, as long as you have no '=' on a line by itself. Oh, and comments begin with '#' but they have to be by themselves. Don't forget to use uppercase for the keywords and lowercase for the values. Come back! Come back! I didn't tell you about the mandatory keywords!")
• Rewriting or copying possibly buggy configuration code instead of reusing a module
• Making the configuration an object with a consistent interface instead of the usual DIY haphazard hash of keywords

Scared yet? That's why we have AppConfig. It can handle all these concerns. It's more than likely that DIY is not what you should be using.

[Sept 15, 2003] The road to better programming Chapter 6. Developing cfperl, from the beginning

[Sept 15, 2003] Fix-It Fatigue  By John Foley, George V. Hulme. Patched became a problem and not only on Microsoft operating systems ;-)

With the Blaster worm seeming to be under control, alleged virus-author Jeffrey Parson under house arrest in Minnesota, and hacker Adrian Lamo under the watchful eye of the feds, business-technology managers may have enjoyed a few hours of peace and quiet last week. But it was short-lived. On Sept. 10, Microsoft issued a security bulletin warning of three new critical vulnerabilities in the Windows operating system, sending systems administrators rushing to patch their computers. It's become an all-too-common scenario--and one that's causing some businesses to re-evaluate their heavy reliance on Microsoft products.

A year-and-a-half after Bill Gates declared that trustworthy computing had become Microsoft's No. 1 priority, the software bugs keep coming. The latest vulnerabilities involve the Remote Procedure Call service in Windows, making it possible for a malicious hacker to take control of a target system, introduce an infectious worm, or launch a denial-of-service attack. A week earlier, Microsoft issued five other warnings, four involving the omnipresent Office applications suite. For the year, the tally stands at 39.

And those are just the holes that have been uncovered by others and reported to Microsoft. In addition, the software vendor is combing through its code, finding holes, and issuing patches without publicizing the flaws. No one knows how many more are yet to be uncovered. "There's no way to wrap your hands around that," says Dan Ingevaldson, engineering manager with security vendor Internet Security Systems Inc.

Some business and technology professionals are running out of patience. "The issues around these vulnerabilities are escalating to the point where it's not just CIOs or CTOs, it's corporate officers, it's boards of directors asking: 'What are we going to do?'" says Ruth Harenchar, CIO of Bowne & Co., which last week scrambled to patch 4,500 Windows PCs and 500 servers in the United States and more overseas. "The situation appears to be getting worse, not better."

The patching work has thrown Bowne & Co.'s technology projects off schedule. Now, the specialty-printing-services company is assessing its options. Among them: redesigning its network around a thin-client model to reduce the number of PCs running Windows and, on other machines, migrating to Linux. "It's getting to be enough of a burden that you have to seriously start thinking about alternatives," Harenchar says.

Raymond James & Associates has assembled a team of IT staffers to manage the constant patching. "Organizations have to mobilize and realize this is going to be a way of life for the foreseeable future," says VP of IS Gene Fredriksen.

The financial-services firm, with offices around the world, last week began the arduous task of patching 10,000 PCs and 1,000 servers. "The pressure is on," Fredriksen says. "Anybody that isn't patched by the weekend is going to have trouble." The fear is that the latest vulnerability leaves Windows computers open to a Blaster-like worm. "There's a very good chance that a worm is going to be developed" to take advantage of the latest security holes, says ISS's Ingevaldson.

"People are getting fed up," says Lloyd Hession, chief information security officer at financial-network provider Radianz, adding that the number of Windows patches is reaching "epic proportions." The situation is causing more than just a few disgruntled customers to re-evaluate how much they use Microsoft products. Says Gartner security analyst John Pescatore, "There's definitely a very large trend towards that."

O'Reilly Network Top Five Open Source Packages for System Administrators I do not like Cfengine. IMHO although the idea is good implementation does not substantially improves sysadmin productivity in comparison with rsh and regular scripts in Ksh93 and Perl. 

• #5: Amanda
• #4: LDAP
• #3: GRUB
• #2 Nagios
• #1 Cfengine

Sysadmin Tales of Terror

Solaris 8 Administrator's Guide Chapter 4 Network Configuration By Paul Watters January 2002 ISBN 0-596-00073-1,400 pages

After undertaking the complex tasks required to configure a single host, planning and setting up an entire network can be daunting. In this chapter, you'll learn how to configure a Solaris-based network, including the configuration of single or multiple network interfaces, static and dynamic routing, and network troubleshooting. In addition, examples for enabling devices and testing interfaces will be provided.

Developer Todo

About: Developer Todo is a program to assist developers in maintaining a list of outstanding tasks in a heirarchical, colourised, and prioritised list. Additionally, it can automatically list outstanding items when you change into a directory.

Changes: Fixed more GCC 3.x compilation problems, and a problem when running without the TERM environment variable set.

f2w Helpdesk at Sourceforge

About: f2w helpdesk is a Web-based helpdesk package. It allows requests to be categorised to an arbitrary level of detail using a expert-system-like question and answer method. Advice and problem-specific information can be associated with the request categories, thus building up a knowledge base to speed the resolution of frequently occurring problems. Users can also add their own tasks, thus using it as a todo list or for workflow within teams, and notes can be added to each request at any time.

Changes: Minor user interface improvements, bugfixes in the Oracle configuration script, a new MS SQL configuration, and the automatic addition of new help desk operators to at least one team (without this, they can't do much).

About: Atop is an ASCII full-screen performance monitor similar to the command top. For every interval (default 10 seconds), it shows system-level activity related to the CPU, memory, swap, disks and network layers, and it shows for every active process the CPU utilization in system and user mode, the virtual and resident memory growth, priority, username, state, and exit code. The process level activity is also shown for processes which finished during the last interval (for this reason process accounting is switched on), to get a complete overview about the consumers of things such as CPU time. Atop only shows the active system-resources and processes, and only shows the deviations since the previous interval (e.g., the memory growth rather than total memory usage per process). Unfortunately, the standard kernel does not maintain counters about the number of disk and network accesses issued per process. Later on, kernel patches will be made available to add these process level counters. The current version of atop is already prepared to display these counters.

Changes: Process accounting is now handled securely by creating a separate subdirectory in /tmp. Disk types other than sda and hda are now recognized. Other bugfixes and modifications include allowing users to run without process accounting, avoiding flag list and buffer overflows, and correct access times for disks in SMP systems.

SSGDOC - System Administration at cs.unm.edu

This document contains documentation, procedure, and policy for the Systems Support group. Reading and maintaining it is a required element to employment in the Systems Support Group - it is vital in order for us to provide consistent (hopefully excellent) service to the CS department. It should be kept as terse as possible (otherwise no one will read it) while providing sufficient documentation so that all (especially new) members will have a good running start at understanding the technical composition of the site, and the group's procedures and policies of operation. CS.UNM.EDU's technical composition has been largely modelled after the LISA paper available at http://www.infrastructures.org. If you hope to understand the document you are currently reading, you really must first read the Infrastructures paper upon which this practical document is modelled after.

[Jan 28, 2002] Sun BluePrints[tm] OnLine - Articles January 2002Data Center Design Philosophy

Editor's Note - This article is the complete first chapter of the Sun BluePrints[tm] book, Enterprise Data Center Design and Methodology, by Rob Snevely (ISBN 0-13-047393-6), which is available through www.sun.com/books, amazon.com, fatbrain.com and Barnes & Noble bookstores.

The detailed process of data center design appears on the outset to be a purely mechanical process involving the layout of the area, computations to determine equipment capacities, and innumerable other engineering details. They are, of course, essential to the design and creation of a data center, however, the mechanics alone do not a data center make. The use of pure mechanics rarely creates anything that is useful, except perhaps by chance.

There are, in fact, some philosophical guidelines that should be kept in mind during the data center design process. These are based on the relatively short history of designing and building practical data centers, but are also based on design concepts going way back. This chapter looks at some of these philosophies.

This chapter contains the following sections:

• Look Forward by Looking Back
• A Modern Pantheon
• Fundamentals of the Philosophy
• Top Ten Data Center Design Guidelines

[Aug 8, 2001] Several useful papers from SysAdmin Magazine

• A Perl Package for Monitoring Traffic by John Shearer
  http://www.sysadminmag.com/articles/2001/0109/0109k/0109k.htm 
  The rtr-graph package described in Shearer's article is a set of Perl scripts for polling routers (or other SNMP-enabled devices) to gain specific traffic information.
• A Framework for Automated File Transfer by George Callaway http://www.sysadminmag.com/articles/2001/0109/0109l/0109l.htm 
  Callaway shares a set of portable scripts for file transfer.
• Checking Your System Logs with awk by Jose Nazario
  http://www.sysadminmag.com/articles/2001/0109/0109m/0109m.htm 
  Nazario provides some scripts to facilitate the log-checking
  process.

[July 27, 2001] System Administrator Appreciation Day -- help yourself ;-)

Advice to employees the proper use of the System Administrator's valuable time

Why You Can't Find Your System Administrator
http://www.cybernothing.org/cno/sysadmin.html

A Week in the Life of a System Administrator
http://www.avdf.com/jan98/hum_h003.html

101 Things You Do Not Want Your System Administrator To Say
http://www.brenna.net/notes/sys-adm.html

Tech Tales - The funniest tech support stories on the Internet
http://www.techtales.com

Understanding and Preventing System Slowdowns

Submitted by <Jamie Wilson> on Friday at 08:44:55 (EDT))

A Sunworld article analyzes performance issues on Sun systems and gives advice on how to track them down. The article discusses the use of tools, such as sar, netstat and top to analyze the location of a bottleneck.

[Aug 20, 2000] Linux Today - IBM developerWorks Tutorial Compiling and installing software from sources -- small tutorials for newbies who want to learn how to download, unpack, compile, and troubleshoot apps.

LinuxLookup.com: Ownership & Permissions Guide(Jul 08, 2000)
LinuxPapers.org: File Permissions in Unix and Linux(Apr 14, 2000)
LinuxPowered.com: Linux file permissions & ownership(Dec 19, 1999)
Ext2: File Permissions Made Easy(Nov 20, 1999)
apcmag: Linux File Permissions(Jun 24, 1999)
LinuxPower: Permissions and the ext2 Filesystem(Jan 05, 1999)

LinuxFreak Monitoring your desktop machine

Monitoring your desktop machine
Cat: Feature, Posted June 04, 2000 by gh0ul

Many people who start out with Linux using just X, never notice a lot of the things that can go on with a Linux system. Some of those things should be monitored or atleast kept an eye on.In this article we will go over a couple methods of watching over your Linux box from the X windowing System.First off, we will go over a traffic monitor called 'trafshow'. trafshow is a light and easy traffic monitor that displays information about connections to your system, for example, if someone telnets to you, it will display their ip, the protocol, and port.. very simple information for just keeping an eye open. You can get it from here. Grab trafshow-1.3.tar.gz, un-tar and compile it, you should be able to simply untar it and issue a make && make install in the directory (must be root for the make install) After that, go ahead and open an xterm, I suggest using the flags:
xterm -bg black -fg white Then you can su and start trafshow, or some may prefer to give it root permissions so they would not need to su anymore, but others might prefer to keep it root only, so that if you have users on your system whom you don't want to see your connections, you'll still be good to go. It's suggested that you keep the terminal open with trafshow running at all times, so if you are ever curious about a connection, you can simply take a look and see what's going where.

Another pretty popular traffic monitor is iptraf, it does a lot of the same as trafshow, yet can be configured more, and also will log data. You can find it here: ftp://ftp.cebu.mozcom.com/pub/linux/net/iptraf-2.2.1.tar.gz

Some other helpful network monitoring tools:

• (tcpflow, tcpdump, and trafshow+rvnamed below all require libpcap, which can be obtained at ftp://ftp.ee.lbl.gov/libpcap-0.4.tar.Z)
• tcpflow is a utility for showing TCP packets contents in cleartext.. in an easy to view manner (layer 2)
  URL: ftp://ftp.circlemud.org/pub/jelson/tcpflow/tcpflow-0.12.tar.gz
• tcpdump is a layer 3 traffic monitor and data acquisition.
  URL: ftp://ftp.ee.lbl.gov/tcpdump-3.4.tar.Z
• trafshow+rvnamed is a version of trafshow (like above) merged with nvnamed from iptraf.
  URL: http://www.unix.gr/firesoft/trafshow-angelos.tar.gz
• Portsentry is a wonderful tool for detecting port scans, you can even set it up to execute scripts like perhaps a notify-by-email script.linuxnewbie also has a pretty good review on setting portsentry up.
  URL: http://www.psionic.com/tools/portsentry-1.0.tar.gz
• Another neat application (for monitoring log files) is root-tail, you can find it here: http://goof.com/pcg/marc/data/root-tail-0.0.6.tar.gz root-tail, similar to the normal Unix command 'tail' (will tail the end of a file) , yet root-tail prints it to your desktop (directly to the X11 root window).
  you can customize it quite nicely also, and tail things to your root window like syslog, or any other logs that might be of interest to you.

  Example Usage:
  root-tail -g 80x25+100+50 /var/log/messages,green /var/log/secure,red,'ALERT'

These are NOT full proof anti-hack methods to take on your system, but these should help you out a bit, and they are always handy to go along with normal system administration.

Linux Magazine January 2000 GURU GUIDANCE Big Brother Is Watching

Fortunately, this Big Brother is truly your friend. However, like its Orwellian namesake, it is constantly on the lookout for things it doesn't like, waiting to sound an alarm. I am talking about a systems-monitoring tool developed by Sean MacGuire of The MacLawran Group (http//www.maclawran.ca/bb-dnld/) called Big Brother.

Big Brother does most things that you'll find in commercial monitoring tools; it can let you know when a machine on your network is down or becoming overloaded or when a filesystem is getting too full; it can tell you when specific processes are or are not running on clients; it can even page you when a specific event occurs. It can be used to monitor Unix, Linux, Windows NT, and NetWare clients.

One of the main reasons you'll want to try out Big Brother is because of its simplicity. It is composed of just a handful of scripts and programs, which collect information and report it to a central server, which displays everything in an accessible HTML format. Big Brother's scripts are easy to change and reconfigure, allowing you to customize the software to suit your network.

Although it is not covered directly by the GNU General Public License, you can download Big Brother for free from the MacLawran Group's Web site. It is covered by a "fair use" license, which requires written permission from the MacLawran Group to redistribute it.

[Jul 29, 2000] Slashdot Are Buffer Overflow Sploits Intel's Fault -- interesting discussion about problems with C

[Jul 29, 2000] Slashdot Preventing Vendors From Playing The Blame Game -- several insightful posts about AIX, Websphere, etc.

Issue #92 Mailing From Scripts - Focus On Linux - 07-23-00

The old Unix mailer at /bin/mail is one of the programs that really helped to launch the Internet as we know it today. In days gone by, it

was one of the most used binaries on any Unix workstation. Today, the /bin/mail binary can still be used, but is horribly outdated -- it's a text-only mailer with almost no features at all.

It is good, however for one thing: automated mailing, such as sending e-mail from a shell script. To send mail with the /bin/mail program, use this syntax:

  mail -s "subject" touser@address < body_text

For example, if your e-mail address is joehalliway@nowhere.com, and you needed to have the login information from the last command on a given system sent to you every night at a given time, you could create a cron job which called this command:

  last | mail -s "Login information" joehalliway@nowhere.com

When executed, this command would send the output from the last command to your e-mail box with the subject given. There are many uses for the /bin/mail program when employed this way in scripts; use your imagination.

Focus on Linux Forum /bin/mail other than text

You can use bin mail to also mail attachments. This was not mentioned
in the article so I thought I might help out a few hapless souls.

To use /bin/mail to mail a attachment you must also have uuencode
installed. Not to worry though, most distributions include this by
default. Anyhow on to the good stuff.

So I'm in my home directory /home/possum and there is a file in there
called attachment.txt. I can send this file two ways:

The first makes the attachment.txt show in the mail, in other words the mail will contain the contents of attachment.txt. To do this I enter the command "mail whoever@wherever.com < attachment.txt".

The second way send attachment.txt as an attachment. I had wanted to
do this many times and was unable to until I found uuencode could make this happen. To send it as an attachment I would enter the command "uuencode attachment.txt attachment.txt | mail whoever@wherever.com". In the command line the first attachment.txt is the name of the file I want to send. The second attachment.txt is the name of the attachment the recipient will get. Pipe that to mail and Viola you have sent an attachment from a command line or shell script. Happy scripting, Paul

Issue #92 Calling Commands When Booting - Focus On Linux - 07-23-00

Linux Today - Linuxuser.co.za Tricks with -etc-issue

When your Linux System boots it starts "getty" processes on a number of virtual terminals on your system. When getty starts, it prints the contents of /etc/issue to the terminal. To customize the look of your console, all you have to do is edit /etc/issue.

You can also add colour by using the colour capabilities of the Linux console. The Linux console, like almost any other existing terminal, has escape sequences that can be used to change the appearance of text on screen. Piping the following script into /etc/issue:

	#!/bin/sh

	spaces(){
	        COUNT=0;
 	       while [ $COUNT -lt $1 ]; do
 	               echo -ne " ";
 	               COUNT=$[$COUNT+1]
 	       done
	}

	esc="\033["
	echo -ne "${esc}H${esc}J\n${esc}44;37;1m"
	WELCOME="Welcome to "`hostname`" running Linux "`uname -r`
	CHARS=$[(80-`echo $WELCOME | wc --chars`)/2]
	spaces $CHARS
	echo -ne $WELCOME
	spaces $CHARS
	echo -ne "${esc}0m\n\\l "

should produce this result:

... ... ... ... ...

Linux Today - O'Reilly Network What is a Network Administrator Anyway

TechRepublic: Establishing quotas for users on a Linux network(Jul 22, 2000)

RootPrompt.org: Using expect for System Administration(Jul 12, 2000)

LinuxPR: Linux Network Administrator's Guide Revised and Expanded (Jul 11, 2000)

RootPrompt.org: Trust and the System Administrator(Jun 19, 2000)

32BitsOnline: Book [Review]: Essential System Administration(May 21, 2000)

LinuxWorld: Relief for sysadmin headaches(May 02, 2000)

O'Reilly Network: CYA for System Administrators; Things to keep in mind in our litigious society(Apr 21, 2000)

SunWorld: Xvfb - A conversation every system administrator should hear(Mar 26, 2000)

Sys Admin: Policy Routing in Linux(Mar 18, 2000)

Linux Journal: Book Review --Linux System Administration(Feb 27, 2000)

osOpinion: The newbie network administrator and open source: I fear Skippy(Jan 11, 2000)

BW: DigitalThink Announces Completion of Linux/UNIX System and Network Administration Series(Dec 29, 1999)

(May 9, 2000, 07:27 UTC) (425 reads) (0 talkbacks) (Posted by marty)
"The file /etc/inetd.conf is vitally important to your system's security and well-being -- especially if your system has a 24x7 connection to the Internet."

[Jan 27, 2000] Administering Linux using CVS.

Much emphasis has of late been put on making Linux easier to manage. Typically the model used it that of a single user managing one or more systems. However, once mutiple administrators and automated scripts are all altering the configuration for a single machine the sutiation becomes somewhat more complex. This problem is in essence very similar to the problem software developers face when a team is working on a source tree. There are many solutions to this problem, one of the most popular being Concurrent Versions System or CVS.

Twinkle-Toes Release 4th February 1999.

• html
• dvi
• postscript
• uncompressed postscript
• latex source

[Jan 23, 2000] Version 2.0 of the Netware file system has been announced by the Timpanogas Group. It is currently only available in binary form. Source is evidently forthcoming, but has been delayed due to some weirdness that is best read directly from the announcement.

[Jan 23, 2000] System Administration Made Simpler, Part 4  -- VNC (Virtual Network Computing)

Who says system administration can't be fun? My lovely and wonderful wife, Sally, is busy using this week's system administration feature to play Kpoker, the K Desktop Environment poker game. She's not busy checking on logs, administering print queues, or even running a process analyzer. She's playing poker. Of course, earlier on, I took control of her Windows 95 PC and started to edit a letter she was working on, so it probably serves me right. In the interest of telling you all about this marvelous tool, I decided to connect to her Windows 95 session and see how the KDE poker game she was playing on our server was doing. Turns out she was losing all our money. Well, that's gambling for you.

Oh--did I mention I am talking about a fantastic remote administration tool that works with not only Windows, Solaris, DEC Alphas running OSF1, but even that old favorite of the desktop publishing world, the MacIntosh?

This great tool is VNC (Virtual Network Computing) from AT&T Laboratories in Cambridge, England. VNC is a package that allows you to view other computer desktops from your own desktop. For instance, I could be running an X server on a Linux machine from a Windows 95 or NT box, or doing the reverse. I can do it from my internal network or across the Internet.

The system administrator in a large company with a number of Windows workstations knows the headaches of all the simple "operator" error calls that nonetheless require a great deal of work and time as you walk the user through the right steps to solve their problems. Wouldn't it be great if you could take control of their desktops and do it for them while they watch and learn? Now, I know there are commercial packages that can do this, but not necessarily from your Linux desktop. They also cost more than VNC.

That's right. VNC is distributed free of charge.

[Jan 3, 2000]   Portable Unix Toolkit (unix scripts)

Most of these scripts are Korn shell scripts, a few are Perl or Expect scripts. The Demo Edition, which is distributed as freeware, contains 11 scripts. The Professional Edition contains a total of 49 scripts and includes all the scripts in the Demo Edition as well as the login environment files. The Enterprise Edition contains a total of 78 scripts and includes all the scripts in the Demo Edition as well as the login environment files. (Some of the Enterprise Edition scripts work only on Solaris, AIX or HP-UX - see the list below.) The Master Edition includes the Professional Edition and the Enterprise Edition for a total of 106 scripts and over 7,000 lines of code. Click on the script names (below) to view the manual pages for the scripts.

The Portable Unix Shell Environment

The Portable Unix Shell Environment (PUSE) is a set of Korn shell login environment files and about 80 Korn shell, Perl and Expect scripts that have been ported to several versions of Unix. It is distributed at no charge as open source software. The scripts, which can be used independently of the login environment files, include general utility scripts and systems administration scripts.

The Portable Unix Shell Environment allows the user to, for example:

• Substitute one string for another in all the text files in a directory hierarchy.
• List all the subdirectories in one or more directories, including symbolic links to directories.
• Copy a file to multiple remote hosts, with backup of the target file on each host.
• List the city or region for a telephone area code or vice-versa.
• Use a single set of Korn shell login environment files (.profile, .kshrc) in a heterogeneous Unix network.

The PUSE has been downloaded 825 times since February 1999. It has been tested on Solaris 2.5.1, 2.6, HP-UX 10.00, 10.20, AIX 3.2.5, 4.1.4, 4.3.2, Ultrix 4.3 and UWin-NT 1.6, 1.68. The scripts were written from scratch; they contain no viruses, worms, Trojan horses, trapdoors, etc. The PUSE is Y2K compliant.

[Dec. 21, 1999]  The FreeBSD Diary -- System tools - toys I have found -- short discussion of  last, swapinfo, systat, tops and z-tools.

[Dec. 21, 1999] Tom Limoncelli's Published Papers --several interesting papers. Recommended.  See Articles for more details

[Dec. 17, 1999] Linux PR Common UNIX Printing System 1.0.3 Released

Hollywood, MD (December 15, 1999) -- Easy Software Products today announced the 1.0.3 release of the Common UNIX Printing System ("CUPS"), an IPP-based printing system for UNIX®

CUPS 1.0.3 is a bug fix release, includes improvements to the HP-GL/2 and image file filters, and provides better error recovery when printing to networked printers. CUPS 1.0.3 can be downloaded from the CUPS web site at http://www.cups.org.

The Common UNIX Printing System provides a portable printing layer for UNIX® operating systems. It has been developed by Easy Software Products to promote a standard printing solution for all UNIX vendors and users. CUPS provides the System V and Berkeley command-line interfaces.

CUPS uses the Internet Printing Protocol (IETF-IPP) as the basis for managing print jobs and queues. The Line Printer Daemon (LPD, RFC1179), Server Message Block (SMB), and AppSocket protocols are also supported with reduced functionality.

CUPS adds network printer browsing and PostScript Printer Description ("PPD")-based printing options to support real world applications under UNIX.

CUPS also includes a customized version of GNU GhostScript (currently based off GNU GhostScript 4.03) and an image file RIP that can be used to support non-PostScript printers.

Sample drivers are provided for HP DeskJet and LaserJet printers. Drivers for over 1600 printers are available in our ESP Print Pro software.

CUPS is licensed under the GNU General Public License. Please contact Easy Software Products for commercial support and "binary distribution" rights.

[June 25, 1999] Linux File Associations in Linux -- important news

You are no longer restricted to native executable file formats (ELF, a.out, etc), in Linux. With kernel 2.2 onwards, there’s support for multiple file formats, that is, you can make the kernel recognize any file format provided you’ve an interpreter for it. These files can then be run just by typing their name at the prompt, like any pure executable. You could, for instance, associate all text files with the vi editor. Whenever you type the name of a text file on the shell prompt, the vi editor will automatically load with this file.

[June 24, 1999] open source IT - Tutorials Using RPM

[June 23, 1999] sysads.com.ph Unix System Administration

[April 21, 1999] Linux Administration Made Easy  by Steve Frampton, <3srf@qlink.queensu.ca> v0.99u.01 (PRE-RELEASE), 21 April 1999. A new LDP book.

[April 21, 1999] ISystem and Network Performance Tuning, Hal Stern, Sun Microsystems

[April 10, 1999] The Sys Admin.  -- a nice tale about sysadmins

[ March. 10, 1999] The Best Windows File Server Linux!

NetBench 5.01 shows how well a network operating system does at the mundane task of file serving, by measuring Wintel file input/output. Natively, Linux doesn't work with DOS/Windows files, but Samba, an open-source Server Message Block (SMB) client and server that ships with all commercial Linuxes, provides that capacity. And how!

You might think that Linux would operate at a disadvantage here, but Linux kicks NT's butt. Only at the lightest loads does NT hold any advantage over the Linuxes. Once the load moves to 12 clients, all the Linux platforms take commanding leads over NT. At 32 clients, SuSE, the weakest Linux, has more than double NT's throughput, and Red Hat, the leader, extends its lead to almost 250 percent of NT's performance.

[ Feb. 4, 1999] Certification page added

E-books, Courses, Tutorials

Online Libraries

• Linux Today - Online book -- 100 Linux Tips and Tricks

  Update: Vladimir Vuksan has posted another mirror site located at http://www.unm.edu/~vuksan/100_linux_tips_and_tricks.pdf

• LinuxPR: Andamooka adds two open content books: LKMPG and Linux From Scratch
• Vendors
  • HP-UX Books  
    • HP-UX Operating System
      • HP-UX 11.0 Release Documentation
      • HP-UX 10.- Release Documentation
    • Networking & Communications
    • Internet & Security Solutions
    • Network & Systems Management
  • AIX System and Related Product Documentation AIX System and Related Product Documentation
    • AIX Version 4.3 Base Documentation
      • AIX Versions 3.2 and 4 Performance Tuning Guide
      • AIX Version 4.3 Guide to Printers and Printing
      • AIX Version 4.3 System Management Guide: Communications and Networks
      • AIX Version 4.3 System Management Guide: Operating System and Devices
      • Distributed SMIT 2.2 for AIX: Guide and Reference
      • Performance Toolbox Version 1.2 and 2 for AIX: Guide and Reference
    • AIX Version 4.3 Extended Documentation
    • AIX & RS-6000 "How-To" Redbooks -- "How-To" books on a wide range of AIX software and RS6000 hardware topics. Includes topics such as operating systems, system management, communications, hardware, application solutions and development, databases, Internet, high availability clusters, and scalable POWERparallel systems.
    • Other IBM Redbooks -- extremely valuable e-library
  • DIGITAL UNIX Version 4.0D Online Books
    • Digital Unix System Administration  see also Table of content (local)
  • CERN UNIX User GuideBRVersion 1.02BRCNDCI164
  • SCO System Administration Guide -- very good
• R870 Unix System Administration - A Survival Course
  • Table of content (local)

Mark Burgess

Principles of system administration - Table of Contents

USAIL

USAIL can be freely mirrored. A very useful resource...

• USAIL: Unix system administration independent learning
  • Some Useful Tools
  • File and Directory Permissions
  • Users, Groups and Passwords
  • Adding new users
  • Introduction to Security
  • Syslog notes
  • Other Important Files Where Information is Logged
  • Recalling Commands: history
  • Unix System Security
  • Administration Tasks Lab
  • Administration Tasks Quiz
• Integrating Your Machine With the Network
• Configuring and Administrating Mail
• Administrating Peripherals
• Backups
• Automating Tasks
• Administrating Software
• Monitoring Performance

LDP e-books

[April 21, 1999] Linux Administration Made Easy  by Steve Frampton, <3srf@qlink.queensu.ca> v0.99u.01 (PRE-RELEASE), 21 April 1999. A new LDP book.

The Network Administrators' Guide   by Olaf Kirk

Other E-books

• System administration procedures of CS and Icarus
• Introduction to Unix Systems Administration
• Unix Systems Administration Notes -- by Paul A. Farrell.
• Unix System Administration  --  Frank G. Fiamingo fgf+@osu.edu  University Technology Services The Ohio State University
• Working With UNIX table of contents -- New Mexico Suppercomputing Challenge Summer Teacher Training Session
• 85321 Systems Administration

Recommended Links

---

In case of broken links please try to use Google search. If you find the page please notify us about new location

Seven Sisters ;-)

• **** Open Directory - Computers Software Operating Systems Unix Administration

• ****+ Lycos Directory Computers Operating Systems Unix Administration -- very good

• **+ Yahoo! Computers and InternetSoftwareOperating SystemsUnix -- pretty week

• ***+ HotBot Directory Computers & Internet Operating Systems Unix Administration

Search engines:

• **** Google Search Unix administration

• ***+ GoTo.com Search Results Unix administration

• AltaVista Simple Query Unix administration -- good, but low signal to noise ratio

Professional societies:

• www.usenix.org

• SAGE - Web resources for SysAdmins
• SAGE-AU Member Resources -- + ftp site with tools

Portals and collections of links

• ****  BigAdmin -- a really useful web site for Solaris administrators

• docs.sun.com - SunOS / Solaris Documentation Online.
• **** Unix Guru Universe -- one of the best collection of links
  • System Administration
• ***+Unix System Administrator's Resources(http://www.stokely.com/unix.sysadm.resources/)
• Cameron Laird's personal notes on system administration
• Unix SysAdm Resources FAQs, Patches and Other Info [A-M]
• Steven Winikoff System Administration Links
• DIGITAL UNIX Version 4.0D Online Books
• Search Unix oriented databases at Internet Sleuth.
• Tony's UNIX administrators Home Page
• DDBs Sysadmin Page
• Know your Unix System Administrator.
• UNIX System Administration Links essentially links.   Good
• Systems Administration Course
• Shadow Passwords
• System Administration
  World Wide Web Security FAQ
  CERT tech tips
  Linux Newbie Administrator Guide
  Newsgroup: Linux Setup
  Linux Installation
  Linux RPM's described
  Linux HOWTO
  Expanding I-Opener
  Lube & Tune: Partitioning, TCP/IP, and more
  UMSDOS - Linux and MS-DOS on one partition
  IDE/Fast-ATA FAQ

Forums

• nixCraft Linux Forum

Recommended Articles

Burnout, and Other Social Isuues

• Classification of Corporate Psychopaths
  • The Danger of Micromanagers and Micromanagement
  • The Manipulator Bosses (aka Mayberry Machiavellians )
  • Narcissistic Managers
  • Paranoid Managers: Type 2 Corporate Psychopaths
  • Bully Managers in IT Workplace: Type 1 of Corporate Psychopaths
• Linux.com: Personal Side of Being a Sysadmin

  "The unfortunate reality of being a Systems Administrator is that sometime during your career, you will most likely run into a user (or lus3r if you prefer) whom has an IQ of a diced carrot and demands that you drop everything to fix their system/email/whatever. This article focuses on how to deal with these kinds of issues."
  

• alt.sysadmin.recovery FAQ
• Softpanorama Information Overload Annotated Webliography
• Softpanorama Humor Archive. Unique Collection of Open Source Related Humor

Etc

[Jul 13, 2001] Duct Tape and Design: Applying Extreme Programming to System Administration By Pat Eyler. Article is provided courtesy of New Riders.

Extreme programming is 80% idiotism...  Just ignore it despite the fact that they include some rational recommendation in the mess they propose...

Lately I've been reading a lot about extreme programming, often abbreviated XP (see http://www.extreme programming for more details). I like what I've been reading and I'd like to start using the ideas, but I'm a sys admin, not a developer. Do the ideas work in my domain? I think they do. In this article I'll lay out the basic tenets of XP, look at how the practices might fit into system administration, and discuss some possible problems.

XP seems to affect people rather strongly. They either love it or hate it. If you hate XP, don't stop reading here. I'm not advocating system administration Wild West style; then again, XP doesn't advocate cowboy coders either. There seem to be many ideas in XP that would do quite well in improving the work of many sys admins.

If you do like XP, you shouldn't stop here either.

At the core of XP is the idea that there are some rules that lead to better, more efficient code. If these rules work better, why not do them all, all the time, all the way? XP isn't about hanging off a sheer rock wall—it's about doing things extremely well.

NOTE

The following terms come from the book Extreme Programming Explained: Embrace Change by Kent Beck (Addison-Wesley, 1999, ISBN 0-201-61641-6). See the "Resources" section for more places to look for information.

This central theme expresses itself in four core values:

• Communication. If the developers and their customers aren't talking, problems will come up quickly and often.
• Simplicity. "Do the simplest thing that could possibly work." The simpler your code, the less likely it is to break.
• Feedback. Use frequent feedback to steer your work.
• Courage. You need to have the courage to do what needs to be done. Inaction can be just as bad as the wrong action.

To keep the four values going, XP uses a set of twelve practices for the developer. Every programmer on the team should be using all of these practices throughout the life of the project. Keeping all of the knobs cranked is what XP is all about.

1. The planning game. One of the biggest complaints about XP is that it throws design out the window. This isn't really true. XP relies on a rough sketch of the overall shape with frequent short-term planning sessions to keep everything moving in the right direction.
2. Testing. Before any new code is written, a test should exist to ensure that the code works correctly. Before any project is declared complete, it has to meet all acceptance tests as well. The combination of constant (automated) unit testing and frequent acceptance testing helps to ensure a quality product.
3. Pair programming. Pairs of programmers write all the production code. One developer is responsible for the tactical end of things (getting compilable, correct code written). The other is the strategist and makes sure that the long-term impact of the code is in sync with what the project needs.
4. Refactoring. This is often termed "refactor mercilessly." Consistent cleaning and improvement of the code makes it easier to maintain and extend. Unit testing and pair programming help ensure that refactored code doesn't break existing functionality.
5. Simple design. Don't write functionality into a program where it isn't needed. Because you're refactoring code regularly (and because you unit-test everything all the time), adding functionality later is easier. When you find out you need something, you'll know just what to add.
6. Collective code ownership. Each member on the team should know enough about the system to make any needed changes, and should have the authority to do so.
7. Continuous integration. After each new bit of functionality is added by a pair of programmers, it should be tested and integrated back into the main tree. If everything is tested before it's integrated, you rarely need to worry about the system breaking. If integration happens often, no one is surprised by changes.
8. On-site customer. Having the customer right there allows minor course corrections to happen in near real-time. Without it, development is slowed while programmers wait for feedback.
9. Small releases. Releases should be as small as possible while still doing something valid. Many projects shoot for a two-week release cycle.
10. 40-hour week. If you work too many hours for too many days, you'll burn out. Stay away from "death marches."
11. Coding standards. If everyone is using the same coding style, it's easier to refactor, the code is easier to understand, and it's easier to switch partners as you move along.
12. Metaphor. Everyone working on a project needs to understand how it all fits together.

This is a very short description of XP. To really understand it, you probably need to take a look at some of the sites listed down in the "Resources" section of this article. Hopefully, you've got enough information to look at XP through a sys admin's glasses.

Extreme System Administration

Okay, so system administration is less like traditional development and more like duct taping things. Does this mean we shouldn't design things? Or that we should run from fire to fire putting out flames? I don't think so. If we spend our time making sure that our work is done right the first time, we'll all be better off.

I think this is where XP can come in. Some of the XP practices seem to fit right into system administration. Certainly, the ideas of maintainable code (and configurations), common ownership, and clean design are laudable goals for anyone dealing with computers. But how do the practices work for a sys admin?

Collective code ownership, simple design, 40-hour weeks, and coding standards all seem to fit without a great deal of fuss and bother. Recognize that configuration files are just another body of code, and the fit is even better.

The planning game is a bit more of a stretch. Every sys admin has a wide and varied set of customers, and not all of them agree on what needs to be done. It's important to work in light of the basic needs of customers, though. Talking with them about the end results of your work and developing plans in concert with them will help ensure that you're on track with what needs to happen.

Building a body of tests for your network and/or system(s) allows you to make changes in a controlled fashion. Testing gives you the reassurance that everything is as it should be.

Refactoring should be a part of every sys admin's toolkit. If you can consistently simplify and/or improve your systems, they become easier to maintain. You can refactor configuration files. You can refactor crontabs, scripts, and even your architecture. Just make sure that you can test everything to verify that you haven't broken anything.

Having an on-site customer allows you to check the impact of changes you want to make, verify solutions you've implemented, and better foresee needs that will come up. Some methods of bringing your customers on site include email lists, a users group, and just walking around to talk to people.

Small releases allow you to put a change into place incrementally. Making changes slowly lets you back out problems quickly. You can work with greater confidence because you're doing fewer things at any given time.

Metaphor (understanding the whole system) allows each member of the sys admin team to respond quickly to problems. It also allows you to describe proposed changes more readily, and understand the potential effects of user requests.

Caveat Admin

Some aspects of XP don't seem to fit completely into the world of system administration. The most difficult seems to be pair programming. While pair programming might not be a viable option for most sys admins, the ideas underlying it fit very well. Another pair of knowledgeable eyes to look over plans, configuration files, scripts, and other products of our craft can be a boon. (I've had more than a few brainos caught by coworkers.)

I think wholesale adoption of XP into system administration probably isn't the way to go. Reading up on the methodology and stealing the concepts that work seems a much better way to add XP to your daily regimen.

Anyone want to pair up to go clean up sendmail.cf?

Resources

• Extreme Programming web site
• XP portal
• RoleModel Software
• Extreme Programming Explained: Embrace Change by Kent Beck (Addison-Wesley, 2000, ISBN 0-201-61641-6)
• Extreme Programming Installed by Ronald E. Jeffries, Ann Anderson, and Chet Hendrickson (Addison-Wesley, 2001, ISBN 0-201-70842-6)
• Planning Extreme Programming by Kent Beck and Martin Fowler (Addison-Wesley, 2001, ISBN 0-201-71091-9)

The FreeBSD Diary -- System tools - toys I have found -- short discussion of  last, swapinfo, systat, tops and z-tools.

Tom Limoncelli's Published Papers

• "Creating a Network for Lucent Bell Labs Research South"Published in Proceedings of the Eleventh Systems Administration Conference (LISA '97), Usenix, October 1997
• "Turning the Corner: Upgrading Yourself from ``System Clerk'' to ``System Advocate''"Published in Proceedings of the Eleventh Systems Administration Conference (LISA '97), Usenix, October 1997
• "Providing Reliable NT Desktop Services by Avoiding NT Server"Published in Proceedings of the Second Usenix NT Systems Administration Conference (LISA-NT '98), August 1998
• "Tricks You Can Do If Your Firewall Is a Bridge"Published in Proceedings of the First Usenix Network Administration Conference (NETA '99), April 1999

  Published in Proceedings of the Thirteenth Systems Administration Conference (LISA '99), Usenix, November 1999

• "Deconstructing User Requests and the Nine Step Model" (coming soon)
• "Adverse Termination Procedures" (coming soon).