Softpanorama
(slightly skeptical)
Open Source Software Educational Society
May the source be with you, but remember the KISS principle ;-)
Softpanorama Search
|
Softpanorama |
May the source be with you, but remember the KISS principle ;-)
Softpanorama Search
|
by · 90 commentsAnyone who has never made a mistake has never tried anything new. -- Albert Einstein.
Here are a few mistakes that I made while working at UNIX prompt. Some mistakes caused me a good amount of downtime. Most of these mistakes are from my early days as a UNIX admin.
userdel Command
The file /etc/deluser.conf was configured to remove the home directory (it was done by previous sys admin and it was my first day at work) and mail spool of the user to be removed. I just wanted to remove the user account and I end up deleting everything (note -r was activated via deluser.conf):
userdel fooRebooted Solaris Box
On Linux killall command kill processes by name (killall httpd). On Solaris it kill all active processes. As root I killed all process, this was our main Oracle db box:
killall process-nameDestroyed named.conf
I wanted to append a new zone to /var/named/chroot/etc/named.conf file., but end up running:
./mkzone example.com > /var/named/chroot/etc/named.confDestroyed Working Backups with Tar and Rsync (personal backups)
I had only one backup copy of my QT project and I just wanted to get a directory called functions. I end up deleting entire backup (note -c switch instead of -x):
cd /mnt/bacupusbharddisk
tar -zcvf project.tar.gz functions
I had no backup. Similarly I end up running rsync command and deleted all new files by overwriting files from backup set (now I’ve switched to rsnapshot)
rsync -av -delete /dest /src
Again, I had no backup.Deleted Apache DocumentRoot
I had sym links for my web server docroot (/home/httpd/http was symlinked to /www). I forgot about symlink issue. To save disk space, I ran rm -rf on http directory. Luckily, I had full working backup set.
Accidentally Changed Hostname and Triggered False Alarm
Accidentally changed the current hostname (I wanted to see current hostname settings) for one of our cluster node. Within minutes I received an alert message on both mobile and email.
hostname foo.example.comPublic Network Interface Shutdown
I wanted to shutdown VPN interface eth0, but ended up shutting down eth1 while I was logged in via SSH:
ifconfig eth1 downFirewall Lockdown
I made changes to sshd_config and changed the ssh port number from 22 to 1022, but failed to update firewall rules. After a quick kernel upgrade, I had rebooted the box. I had to call remote data center tech to reset firewall settings. (now I use firewall reset script to avoid lockdowns).
Typing UNIX Commands on Wrong Box
I wanted to shutdown my local Fedora desktop system, but I issued halt on remote server (I was logged into remote box via SSH):
halt
service httpd stop
Wrong CNAME DNS Entry
Created a wrong DNS CNAME entry in example.com zone file. The end result - a few visitors went to /dev/null:
echo 'foo 86400 IN CNAME lb0.example.com' >> example.com && rndc reloadConclusion
All men make mistakes, but only wise men learn from their mistakes -- Winston Churchill.
From all those mistakes I’ve learnt that:
- Backup = ( Full + Removable tapes (or media) + Offline + Offsite + Tested )
- The clear choice for preserving all data of UNIX file systems is dump, which is only tool that guaranties recovery under all conditions. (see Torture-testing Backup and Archive Programs paper).
- Never use rsync with single backup directory. Create a snapshots using rsync or rsnapshots.
- Use CVS to store configuration files.
- Wait and read command line again before hitting the dam [Enter] key.
- Use your well tested perl / shell scripts and open source configuration management software such as puppet, Cfengine or Chef to configure all servers. This also applies to day today jobs such as creating the users and so on.
Mistakes are the inevitable, so did you made any mistakes that have caused some sort of downtime? Please add them into the comments below.
- 06.21.09 at 2:42 am
- My all time favorite mistake was a simple extra space:
cd /usr/lib
ls /tmp/foo/bar
I typed
rm -rf /tmp/foo/bar/ *
instead of
rm -rf /tmp/foo/bar/*
The system doesn’t run very will without all of it’s libraries……
- 06.21.09 at 9:15 am
- never type anything such as:
rm -rf /usr/tmp/whatever
maybe you are going to type enter by mistake before the end of the line. You would then for example erase all your disk starting on /.if you want to use -rf option, add it at the end on the line:
rm /usr/tmp/whatever -rf
and even this way, read your line twice before adding -rf
- 06.21.09 at 9:26 am
- public network interface shutdown … done
typing unix command on wrong box … done
Delete apache DocumentRoot … done
Firewall lockdone … done with a NAT rule redirecting the configuration interface of the firewall to another box, serial connection saved meI can add, being trapped by aptitude keeping tracks of previously planned — but not executed — actions, like “remove slapd from the master directory server”
- 06.21.09 at 11:03 am
- Rebooted the wrong box
While adding alias to main network interface I ended up changing the main IP address, the system froze right away and I had to call for a reboot
Instead of appending text to Apache config file, I overwritten it’s contents
Firewall lockdown while changing the ssh port
Wrongfully run a script contained recursive chmod and chown as root on / caused me a downtime of about 12 hours and a complete re-installSome mistakes are really silly, and when they happen, you don’t believe your self you did that, but every mistake, regardless of it’s silliness, should be a learned lesson.
If you did a trivial mistake, you should not just overlook it, you have to think of the reasons that made you did it, like: you didn’t have much sleep or your mind was confused about personal life or …..etc.I like Einstein’s quote, you really have to do mistakes to learn.
fsck /dev/sda2
the root filesystem was rendered unreadable. system down. dead. users really
pissed off.
fortunately there was a full backup and the machine rebooted within an hour.
Yea, I did that one too. Wanted to shut down my own vm but I issued init 0 on a remote server which I accessed via ssh. And oh yes, it was the production server.
Locking myself out while configuring the firewall – done – more than once. It wasn’t really a CLI mistake though. Just being a n00b.
georgesdev, good one. I usually:
ls -a /path/to/files
to double check the contents
then up arrowkey homekey hit del a few times and type rm. I always get nervous
with rm sitting at the prompt. I’ll have to remember that -rf at the end of
the line.
I always make mistakes making links. I can never remember the syntax. :/
Here’s to less CLI mistakes… (beer)
I am still not sure *why* I did when I have read about all the warnings about using this command. (A dyslexic moment with the syntax?)
Ah well, a good lesson learned. At least it was not the disaster it could of been. I shall be *very* paranoid about this command in the future.
sudo rm /etcinstead of
sudo rm etc
Without the entire /etc folder the computer didn't work anymore (which was
to be expected ofcourse) and I ended up reinstalling my computer.
And the worst: update and upgrade while some
important applications were running, of
course on a production server.. as someone mentioned the system doesn’t run
very well
without all of its original libraries :)
Just assuming that a [-v] option stands for –verbose
Yep, most of the time. But not on a [pkill] command.
[pkill -v myprocess] will kill _any_ process you can kill — except those whose
name contains “myprocess”. Ooooops. :-!
(I just wanted pkill to display “verbose” information when killing processes)
Yes, I know. Pretty dumb thing. Lesson learned ?
I would suggest adding another critical rule to your list:
” Read The Fantastic Manual — First” ;-)
My Friend tried to see last reboot time and mistakenly executed command “last | reboot” instead of “last | grep reboot”
It made a outage on Production DB server.
Another guy, wants to see the FQDN on solaris
box and executed “hostname -f”
It changed the hostname name to “-f” and clients faced lot of connectivity issues
due to this mistake.
[ hostname -f is used in Linux to see FQDN name but it solaris its usage is
different ]
Luckily i had backups but took a little while to restore,
crontab -r instead of -e
e for edit
r for remove..
now i always use -l for list before editing…
cd /tmp/files
rm ./*
Of course, eventually someone removed /tmp/files..
On My RHEL 5 sever I create /tmp mount point to my storage and tmpwatch script that run under cron.daily removes files which have not been accessed 12 hours !!!
# ps -aef | grep -v foo | awk {print’$2′} | xargs kill -9
Typed “grep -v” instead of “grep -i” and u can guess what happened :(
rm -Rf /var/* in the wrong box. Recovered
in few minutes by doing scp root@healty_box:/var . – the ssh session
on the just broken box was still open . This saved my life :-PI understand where pkill gets its -v functionality from (pgrep and thus from grep), but honestly I don’t see what use of -v would be for pkill. When do you really need to say something like kill all processes except this one? Seems reckless. Maybe 1 in a million times you’d use it properly, but probably most of the time people just get burned by it. I wrote to the author of pkill about this but never heard anything back. Oh well.
Worst thing that ever happened to me:
Our oracle database runs some rdbms jobs at midnight to clean out very old rows
from various tables, along the line of “delete from XXXX where last_access <
sysdate-3650". One sunday i installed ntp to all machines, made a start script
that does an ntpdate first, then runs ntpd. Tested it:
$ date 010100002030; /etc/init.d/ntpd start; date
Worked great, current time was ok.
$ date 010100002030; reboot
After the machine was back up i noticed i had forgotten the /etc/rc*.d symlinks.
But i never thought of the database until a lot of people were very angry monday
morning. Fortunately, there's an automated backup every saturday.
# cd /folder
# chmod 000 .. -R
used two points instead of one, and of course the system used the upper folder
witch is / for modifying attributes
ended up getting out of my home and go the the server to reset the permissions
back to normal. I got lucky because i just did a dd to move the system from
one HDD to another and I haven’t deleted the old one yet :)
And of course the classical configuring the wrong box, firewall lockout :)
rm -rf *
I intended to remove all files under a site, after changing the current working
directory, then replacing with the stable one
wrong window, wrong server, and I did it on production server xx((
just aware the mistakes 1.5 after typing [ENTER]
no backup. maybe luckily, the site was keep running smooth..
it seems that the deleted files were such images, or media contents
1-2 secs incidental removal in heavy machine gave me loss approx. 20 MB
ls ~usr/tar/foo/bar then rm -f ~usr/tar/foo/bar – make sure you know what you will delete
When working with SSH, always make sure what system you are on. Modifying system prompts generally eliminates all confusion there.
It’s all just creating a habit of doing things safely… at least for me.
note the /etc. It was supposed to be rm -rf etc
We were developing a website, and I was in charge of developing the CGI scripts which generated a lot of temporary files, so on pure routine i worked in “/var/www/web/” and entered “rm temp/*” which i misspelled at some point as “rm tmp/ *”. I kind of wondered, in my overtired brain, what took so long for the delete to finish, it should only be 20 small files that is should delete.
The very next morning the paying client was to fly in and pay us a visit, and get a demonstration of the project.
P.S Thanks to Subversion and opened files in Emacs buffers I managed to get almost all files back, and I had rewritten the missing files before the morning.
rm * in one of my project directory (no
backup). I planned to do rm *~ to delete backup files but used international
keyboard where space was required after ~ (dead key for letters like õ)..postsuper -d ALL (instead of -r ALL, adjacent keys – 80k spooled mails gone). No recovery possible – ramfs :/
Had a .pl script to delete mails in .Spam directories older than X days, didn’t put in enough error checking, some helpdesk guy provisioned a domain with a leading space in it and script rm’d (rm -rf /mailstore/ domain.com/.Spam/*) the whole mailstore. (250k users – 500GB used) – Hooray for 1 day old backup
chown -R named:named /var/named when there was a proc filesystem under /var/named/proc. Every running process on system got chown.. /bin/bash, /usr/sbin/sshd and so on. Took hours of manual find’s to fix.
.. and pretty much all the ones everyone else listed :)
You break it, you fix it.
# cd /usr/local/bin
# ls -l -> that displayed some binaries that I didn’t need / want.
# cd ..
# rm -Rf /bin
– Yeah, you guessed it – smoked the bin folder ! The system wasn’t happy after
that. This is what happens when you are root and do something without reading
the command before hitting [enter] late at night. First and last time …
never try this, if u do so even root will not be able to login
instead I typed :
rm -rf /var
unfortunatley I didnt spot it for a while, and not until after did I realize
that my on-site backups were stored in /var/backups …
IT was a truly miserable few days that followed while I pieced together the
box from SVN and various other sources …
symlinks… ack! I was cleaning up space and thought weird.. I don’t remember having a bunch of databases in this location.. rm -f * unfortunately, it was a symlink to my /db slice, that DID have my databases, friday afternoon fun.
I did a similar with being in the wrong directory… deleted all my mysql binaries.
This was also after we had acquired a company and the same happened on one of their servers months before.. we never realized that, and the server had an issue one dady… so we rebooted. Mysql had been running in memory for months, and upon reboot there was no more mysql. Took us a while to figure that out because no one had thought that the mysql binaries were GONE! Luckily I wasn’t the one who had deleted the binaries, just got to witness the aftermath.
I never forget what I lost??
rm -rf *
… but did not press enter on the last line (as a joke). I expected them to come
back and see it as a joke and rofl….back space… The unthinkable happened, the
screen went to sleep and they banged the enter key to wake it up a couple of
times. We lost 3 days worth of business and some new clients. estimated cost
$50,000+
I never thought about using CVS to backup config files. After doing some really dumb things to files in /etc (deleting, stupid edits, etc), I started creating a directory to hold original config files, and renaming those files things like httpd.conf.orig or httpd.conf.091709
As always, the best way to learn this operating system is to break it…however unintentionally.
Oddly, this kind of update works great in Ubuntu.
“rm -fr /home” instead of “rm -fr home/”
It was a sleepless night, but thanks to Carlo Wood and his ext3grep I rescued about 95% of data ;-)
and it deleted the foo directory from disk :(…lost all my code just before the dead line :(
# ps -aef | grep -v foo | awk {print’$2′} | xargs kill -9
Typed “grep -v” instead of “grep -i” and u can guess what happened :(
rm -Rf /var/* in the wrong box. Recovered
in few minutes by doing scp root@healty_box:/var . – the ssh session
on the just broken box was still open . This saved my life :-PI understand where pkill gets its -v functionality from (pgrep and thus from grep), but honestly I don’t see what use of -v would be for pkill. When do you really need to say something like kill all processes except this one? Seems reckless. Maybe 1 in a million times you’d use it properly, but probably most of the time people just get burned by it. I wrote to the author of pkill about this but never heard anything back. Oh well.
Worst thing that ever happened to me:
Our oracle database runs some rdbms jobs at midnight to clean out very old rows
from various tables, along the line of “delete from XXXX where last_access <
sysdate-3650". One sunday i installed ntp to all machines, made a start script
that does an ntpdate first, then runs ntpd. Tested it:
$ date 010100002030; /etc/init.d/ntpd start; date
Worked great, current time was ok.
$ date 010100002030; reboot
After the machine was back up i noticed i had forgotten the /etc/rc*.d symlinks.
But i never thought of the database until a lot of people were very angry monday
morning. Fortunately, there's an automated backup every saturday.
# cd /folder
# chmod 000 .. -R
used two points instead of one, and of course the system used the upper folder
witch is / for modifying attributes
ended up getting out of my home and go the the server to reset the permissions
back to normal. I got lucky because i just did a dd to move the system from
one HDD to another and I haven’t deleted the old one yet :)
And of course the classical configuring the wrong box, firewall lockout :)
rm -rf *
I intended to remove all files under a site, after changing the current working
directory, then replacing with the stable one
wrong window, wrong server, and I did it on production server xx((
just aware the mistakes 1.5 after typing [ENTER]
no backup. maybe luckily, the site was keep running smooth..
it seems that the deleted files were such images, or media contents
1-2 secs incidental removal in heavy machine gave me loss approx. 20 MB
ls ~usr/tar/foo/bar then rm -f ~usr/tar/foo/bar – make sure you know what you will delete
When working with SSH, always make sure what system you are on. Modifying system prompts generally eliminates all confusion there.
It’s all just creating a habit of doing things safely… at least for me.
note the /etc. It was supposed to be rm -rf etc
We were developing a website, and I was in charge of developing the CGI scripts which generated a lot of temporary files, so on pure routine i worked in “/var/www/web/” and entered “rm temp/*” which i misspelled at some point as “rm tmp/ *”. I kind of wondered, in my overtired brain, what took so long for the delete to finish, it should only be 20 small files that is should delete.
The very next morning the paying client was to fly in and pay us a visit, and get a demonstration of the project.
P.S Thanks to Subversion and opened files in Emacs buffers I managed to get almost all files back, and I had rewritten the missing files before the morning.
rm * in one of my project directory (no
backup). I planned to do rm *~ to delete backup files but used international
keyboard where space was required after ~ (dead key for letters like õ)..postsuper -d ALL (instead of -r ALL, adjacent keys – 80k spooled mails gone). No recovery possible – ramfs :/
Had a .pl script to delete mails in .Spam directories older than X days, didn’t put in enough error checking, some helpdesk guy provisioned a domain with a leading space in it and script rm’d (rm -rf /mailstore/ domain.com/.Spam/*) the whole mailstore. (250k users – 500GB used) – Hooray for 1 day old backup
chown -R named:named /var/named when there was a proc filesystem under /var/named/proc. Every running process on system got chown.. /bin/bash, /usr/sbin/sshd and so on. Took hours of manual find’s to fix.
.. and pretty much all the ones everyone else listed :)
You break it, you fix it.
lesson learned: keep your passwords secure, but handy
# cd /usr/local/bin
# ls -l -> that displayed some binaries that I didn’t need / want.
# cd ..
# rm -Rf /bin
– Yeah, you guessed it – smoked the bin folder ! The system wasn’t happy after
that. This is what happens when you are root and do something without reading
the command before hitting [enter] late at night. First and last time …
never try this, if u do so even root will not be able to login
instead I typed :
rm -rf /var
unfortunatley I didnt spot it for a while, and not until after did I realize
that my on-site backups were stored in /var/backups …
IT was a truly miserable few days that followed while I pieced together the
box from SVN and various other sources …
My all time classic:
- rm -rf /foo/bar/ * [space between / and *]
Be carefull with clamscan’s:
–detect-pua=yes –detect-structured=yes –remove=no –move=DIRECTORY
I chose to scan / instead of /home/user and I ended with a screwed apt, libs,
and missing files from allover the place :D I luckily had –log=/home/user/scan.log
and not console output, so I could restore the moved files one by one
next time I use –copy instead of move and never start with /
these 2 happened at home, while working I’ve learned a long time ago (SCO Unix times) to backup files before rm :D
symlinks… ack! I was cleaning up space and thought weird.. I don’t remember having a bunch of databases in this location.. rm -f * unfortunately, it was a symlink to my /db slice, that DID have my databases, friday afternoon fun.
I did a similar with being in the wrong directory… deleted all my mysql binaries.
This was also after we had acquired a company and the same happened on one of their servers months before.. we never realized that, and the server had an issue one dady… so we rebooted. Mysql had been running in memory for months, and upon reboot there was no more mysql. Took us a while to figure that out because no one had thought that the mysql binaries were GONE! Luckily I wasn’t the one who had deleted the binaries, just got to witness the aftermath.
sudo chmod 777 /
# du -sh /home/myuser/oldserver/var
32G /home/myuser/oldserver/var
# cd /home/myuser/oldserver
# rm -rf /var
The box ran for 6 months after doing this, by the way, until I had to shut it down to upgrade the RAM…although of course all the mail, Web content, and cron jobs were gone. *sigh*
Had to get thru serial to undo vlan config.
Oh, the funny thing is that some hours later my boss just made the same mistake lol
shutdown commands I decided to use init.
I typed init 0…No one at work to hit the power switch for me so
I had to make the 30 minute drive into work.reboot on the commandline.
I was logged in as root…[root@localhost ~]#
pkg_delete bash
The next time i tried to log in, it kept on telling me access denied… hmmmm… ow sh#t
(since my default shell in /etc/passwd was still pointing to a non-existent /usr/local/bin/bash, i would never be able to log in)
In all fairness it was labeled wrong on the box and kvm ID.
Now I’ve learned to check hostname before decom’ing anything.
Not fully inserting a memory module on my home machine which shortcircuited my motherboard.
On several occasions i had to use a rdesktop session to windows machine and use putty to connect to a machine (yep.. i know it sounds weird ;-) ) Anyway.. text copied in windows is stored differently than text copied in the shell. Why changing a root passwd on a box, (password copied using putty) i just control v-ed it and logged off. I had to go to the datacenter to boot into single user mode to acces the box again.
Using the same crappy setup, i copied some text in windows, accidently hit control-v in the putty screen of the box i was logged into as root, the first word was halt, the last character an enter.
Configuring nat on the wrong interface while connected through ssh
Adding a new interface on a machine, filled in the details of a home network in kudzu which changed the default gateway to 192.168.1.1 on the main interface. Only checking the output of ifconfig but not the traffic or gateway and dns settings.
fsck -y on filesystem without unmounting it
My best and most recent FUBAR was when I was backing up my system(I have horrible, HORRIBLE luck with backups to the point I don’t bother doing them any more for the most part); I was using mondorescue and backing the files up to an NTFS partition I had mounted under /mondo and had done a backup that wouldn’t restore anything because of an apostrophe or single quote in one of the file names was backing up, so I had to remove the files causing the problem which wasn’t really a biggie and did the backup, then formatted the drive as I had been planning………..only to discover that I hadn’t remounted the NTFS partition under /mondo as I had thought and all 30+ GB of data was gone. I attempted recovery several times but it was just gone.
crontab -r
au lieu de :
crontab -e
ce qui a eu pour effet de vider la liste crontab…
shutdown -n
(i thaught -n meant “now”)
which had for consequence to reboot the server without networking…
#rm /some/directory/FilesToBeDele[TAB]*
Thinking there were multiple files that began with FilesToBeDele. Instead, there was only one and pressing tab put in the extra space. Luckily I was in my home dir, and there was a file with write only permission so rm paused to ask if I was sure. I ^C and wiped my brow. Of course the [TAB] is totally unneccesary in this instance, but my pinky is faster than my brain.
Jul 9, 2009
If you need to move your Linux installation to a different hard drive or partition (and keep it working) and your distro uses grub this tech tip is what you need.To start, get a live CD and boot into it. I prefer Ubuntu for things like this. It has Gparted. Now follow the steps outlined below.
Copying
- Mount both your source and destination partitions.
- Run this command from a terminal:
$ sudo cp -afv /path/to/source/* /path/to/destinationDon’t forget the asterisk after the source path.- After the command finishes copying, shut down, remove the source drive, and boot the live CD again.
Configuration
- Mount your destination drive (or partition).
- Run the command “gksu gedit” (or use nano or vi).
- Edit the file /etc/fstab. Change the UUID or device entry with the mount point / (the root partition) to your new drive. You can find your new drive’s (or partition’s) UUID with this command:
$ ls -l /dev/disk/by-uuid/- Edit the file /boot/grub/menu.lst. Change the UUID of the appropriate entries at the bottom of the file to the new one.
Install Grub
- Run sudo grub.
- At the Grub prompt, type:
find /boot/grub/menu.lstThis will tell you what your new drive and partition’s number is. (Something like hd(0,0))- Type:
root hd(0,0)but replace "hd(0,0)" with your partition’s number from above.- Type:
setup hd(0)but replace "hd(0)" with your drive's number from above. (Omit the comma and the number after it).That’s it! You should now have a bootable working copy of your source drive on your destination drive! You can use this to move to a different drive, partition, or filesystem.
Related Stories:
Linux - Compare two directories(Feb 18, 2009)
Cloning Linux Systems With CloneZilla Server Edition (CloneZilla SE)(Jan 22, 2009)
Copying a Filesystem between Computers(Oct 28, 2008)
rsnapshot: rsync-Based Filesystem Snapshot(Aug 26, 2008)
K9Copy Helps Make DVD Backups Easy(Aug 23, 2008)
Useful command-line secrets for increasing productivity in the office
Level: Intermediate
Michael Stutz (stutz@dsl.org), Author, Consultant
19 Sep 2006
Updated 21 Sep 2006Using UNIX in a day-to-day office setting doesn't have to be clumsy. Learn some of the many ways, both simple and complex, to use the power of the UNIX shell and available system tools to greatly increase your productivity in the office.
More dW content related to: AWK tips The language of the UNIX® command line is notoriously versatile: With a panorama of small tools and utilities and a shell to combine and execute them, you can specify many precise and complex tasks.
But when used in an office setting, these same tools can become a powerful ally toward increasing your productivity. Many techniques unique to UNIX can be applied to the issue of workplace efficiency.
This article gives several suggestions and techniques for bolstering office productivity at the command-line level: how to review your current system habits, how to time your work, secrets for manipulating dates, a quick and simple method of sending yourself a reminder, and a way to automate repetitive interactions.
The first step toward increasing your office productivity using the UNIX command line is to take a close look at your current day-to-day habits. The tools and applications you regularly use and the files you access and modify can give you an idea of what routines are taking up a lot of your time -- and what you might be avoiding.
You'll want to see what tools and applications you're using regularly. You can easily ascertain your daily work habits on the system with the shell's
historybuilt-in, which outputs an enumerated listing of the input lines you've sent to the shell in the current and past sessions. See Listing 1 for a typical example.
Listing 1. Sample output of the shell history built-in
$ history 1 who 2 ls 3 cd /usr/local/proj 4 ls 5 cd websphere 6 ls 7 ls -l $
The actual history is usually kept in a file so that it can be kept through future sessions; for example, the Korn shell keeps its command history hidden in the .sh_history file in the user's home directory, and the Bash shell uses .bash_history. These files are usually overwritten when they reach a certain length, but many shells have variables to set the maximum length of the history; the Korn and Bash shells have the HISTSIZE and HISTFILESIZE variables, which you can set in your shell startup file.It can be useful to run
historythroughsortto get a list of the most popular commands. Then, useawkto strip out the command name minus options and arguments, and pass the sorted list touniqto give an enumerated list. Finally, callsortagain to resort the list in reverse order (highest first) by the first column, which is the enumeration itself. Listing 2 shows an example of this in action.
Listing 2. Listing the commands in the shell history by popularity
$ history|awk '{print $2}'|awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c|sort -r 4 ls 2 cd 1 who $
If your history file is large, you can run periodic checks by piping totailfirst -- for example, to check the last 1,000 commands, try:
$ history|tail -1000|awk '{print $2}'|awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c|sort -r
Review the files you access or modify
Use the same principle to review the files that you've modified or accessed. To do this, use the
findutility to locate and review all files you've accessed or changed during a certain time period -- today, yesterday, or at any date or segment of time in the past.You generally can't find out who last accessed or modified a file, because this information isn't easily available under UNIX, but you can review your personal files by limiting the search to only files contained in your home directory tree. You can also limit the search to only files in the directory of a particular project that you're monitoring or otherwise working on.
The
findutility has several flags that aid in locating files by time, as listed in Table 1. Directories aren't regular files but are accessed every time you list them or make them the current working directory, so exclude them in the search using a negation and the-typeflag.
Table 1. Selected flags of the find utility
Flag Description -daystartThis flag starts at the beginning of the day. -atimeThe time the file was last accessed -- in number of days. -ctimeThe time the file's status last changed -- in number of days. -mtimeThe time the file was last modified -- in number of days. -aminThe time the file was last accessed -- in number of minutes. (It is not available on all implementations.) -cminThe time the file's status last changed -- in number of minutes. (It is not available on all implementations.) -mminThe time the file was last modified -- in number of minutes. (It is not available on all implementations.) -typeThis flag describes the type of file, such as dfor directories.-userXFiles belonging to user X. -groupXFiles belonging to group X. -newerXFiles that are newer than file X. Here's how to list all the files in your home directory tree that were modified exactly one hour ago:
$ find ~ -mmin 60 \! -type d
Giving a negative value for a flag means to match that number or sooner. For example, here's how to list all the files in your home directory tree that were modified exactly one hour ago or any time since:
$ find ~ -mmin -60 \! -type d
Not all implementations offindsupport theminflags. If yours doesn't, you can make a workaround by usingtouchto create a dummy file whose timestamp is older than what you're looking for, and then search for files newer than it with the-newerflag:
The special-daystartflag, when used in conjunction with any of the day options, measures days from the beginning of the current day instead of from 24 hours previous to when the command is executed. Try listing all of your files, existing anywhere on the system, that have been accessed any time from the beginning of the day today up until right now:
$ find / -user `whoami` -daystart -atime -1 \! -type d
Similarly, you can list all the files in your home directory tree that were modified at any time today:
$ find ~ -daystart -mtime -1 \! -type d
Give different values for the various time flags to change the search times. You can also combine flags. For instance, you can list all the files in your home directory tree that were both accessed and modified between now and seven days ago:
$ find ~ -daystart -atime -7 -mtime -7 \! -type d
You can also find files based on a specific date or a range of time, measured in either days or minutes. The general way to do this is to usetouchto make a dummy file or files, as described earlier.When you want to find files that match a certain range, make two dummy files whose timestamps delineate the range. Then, use the
-newerflag with the older file, and use "\! -newer" on the second file.For example, to find all the files in the /usr/share directory tree that were accessed in August, 2006, try the following:
Finally, it's sometimes helpful when listing the contents of a directory to view the files sorted by their time of last modification. Some versions of thelstool have the-coption, which sorts by the time of file modification, showing the most recently modified files first. In conjunction with the-l(long-listing) and-t(sort by modification time) options, you can peruse a directory listing by the most recently modified files first; the long listing shows the file modification time instead of the default creation time:
$ ls -ltc /usr/local/proj/websphere | less
Time your workAnother useful means of increasing office productivity using UNIX is to time commands that you regularly execute. Then, you can evaluate the results and determine whether you're spending too much time waiting for a particular process to finish.
Is the system slowing you down? How long are you waiting at the shell, doing nothing, while a particular command is being executed? How long does it take you to run through your usual morning routine?
You can get concrete answers to these questions when you use the
date,sleep, andechocommands to time your work.To do this, type a long input line that first contains a
datestatement to output the time and date in the desired format (usually hours and minutes suffice). Then, run the command input line -- this can be several lines strung together with shell directives -- and finally, get the date again on the same input line. If the commands you're testing produce a lot of output, redirect it so that you can read both start and stop dates. Calculate the difference between the two dates:
Test your typing speedYou can use these same principles to test your typing speed:
$ date;cat|wc -w;date
This command works best if you give a long typing sample that lasts at least a minute, but ideally three minutes or more. Take the difference in minutes between the two dates and divide by the number of words you typed (which is output by the middle command) to get the average number of words per minute you type.You can automate this by setting variables for the start and stop dates and for the command that outputs the number of words. But to do this right, you must be careful to avoid a common error in calculation when subtracting times. A GNU extension to the
datecommand, the%sformat option, avoids such errors -- it outputs the number of seconds since the UNIX epoch, which is defined as midnight UTC on January 1, 1970. Then, you can calculate the time based on seconds alone.Assign a variable,
SPEED, as the output of anechocommand to set up the right equation to pipe to a calculator tool, such asbc. Then, output a newechostatement that outputs a message with the speed:
You can put this in a script and then change the permissions to make it executable by all users, so that others on the system can use it, too, as in Listing 3.
Listing 3. Example of running the typespeed script
Know your datesThe
datetool can do much more than just print the current system date. You can use it to get the day of the week on which a given date falls and to get dates relative to the current date.Another GNU extension to the
datecommand, the-doption, comes in handy when you don't have a desk calendar nearby -- and what UNIX person bothers with one? With this powerful option, you can quickly find out what day of the week a particular date falls on by giving the date as a quoted argument:
$ date -d "nov 22" Wed Nov 22 00:00:00 EST 2006 $
In this example, you see that November 22 of this year is on a Wednesday.So, when it's suggested that the big meeting be held on November 22, you'll know right away that it falls on a Wednesday -- which is the day you're out in the field office.
The
-doption can also tell you what the date will be relative to the current date -- either a number of days or weeks from now, or before now (ago). Do this by quoting this relative offset as an argument to the-doption.Suppose, for example, that you need to know the date two weeks hence. If you're at a shell prompt, you can get the answer immediately:
$ date -d '2 weeks'
There are other important ways to use this command. With thenextdirective, you can get the day of the week for a coming day:
$ date -d 'next monday'
With the
agodirective, you can get dates in the past:
$ date -d '30 days ago'
And you can use negative numbers to get dates in reverse:
$ date -d 'dec 14 -2 weeks'
This technique is useful to give yourself a reminder based on a coming date, perhaps in a script or shell startup file, like so:
Give yourself remindersUse the tools at your disposal to leave reminders for yourself on the system -- they take up less space than notes on paper, and you'll see them from anywhere you happen to be logged in.
When you're working on the system, it's easy to get distracted. The
leavetool, common on the IBM AIX® operating system and Berkeley Software Distribution (BSD) systems (see Resources) can help.Give
leavethe time when you have to leave, using a 24-hour format: HHMM. It runs in the background, and five minutes before that given time, it outputs on your terminal a reminder for you to leave. It does this again one minute before the given time if you're still logged in, and then at the time itself -- and from then on, it keeps sending reminders every minute until you log out (or kill theleaveprocess). See Listing 4 for an example. When you log out, theleaveprocess is killed.
Listing 4. Example of running the leave command
You can give relative times. If you want to leave a certain amount of time from now, precede the time argument with a+. So, to be reminded to leave in two hours, type the following:
$ leave +0200
To give a time amount in minutes, make the hours field 0. For example, if you know you have only 10 more minutes before you absolutely have to go, type:
$ leave +0010
You can also specify the time to leave as an argument, which makes
leavea useful command to put in scripts -- particularly in shell startup files. For instance, if you're normally scheduled to work until 5 p.m., but on Fridays you have to be out of the building at 4 p.m., you can set a weekly reminder in your shell startup file:
You can put a plain
leavestatement, with no arguments, in a startup script. Every time you start a login shell, you can enter a time to be reminded when to leave; if you press theEnterkey, giving no value, thenleaveexits without setting a reminder.Send yourself an e-mail reminder
You can also send yourself a reminder using a text message. Sometimes it's useful to make a reminder that you'll see either later in your current login session or the next time you log in.
At one time, the old
elmmail agent came bundled with a tool that enabled you to send memorandums using e-mail; it was basically a script that prompted for the sender, the subject, and the body text. This is easily replicated by the time-honored method of sending mail to yourself with the command-linemailxtool. (On some UNIX systems,mailx.)Give as an argument your e-mail address (or your username on the local system, if that's where you read mail); then, you can type the reminder on the Subject line when prompted, if it's short enough, as in Listing 5. If the reminder won't fit on the Subject line, type it in the body of the message. A
^Don a line by itself exitsmailxand sends the mail.
Listing 5. Example of sending yourself a reminder with the mailx command
Back to top
Automate your repetitive interactions
The Expect language (an extension of Tcl/Tk, but other variations are also available) is used to write scripts that run sessions with interactive programs, as if the script were a user interacting directly with the program.
Expect scripts can save you a great deal of time, particularly when you find yourself engaging in repetitive tasks. Expect can interact with multiple programs including shells and text-based Web browsers, start remote sessions, and run over the network.
For example, if you frequently connect to a system on your local intranet to run particular program -- the
test-serverscommand, for instance -- you can automate it with an Expect script namedservmaint, whose contents appear in Listing 6.
Listing 6. Sample Expect script to automate remote system program execution
Now, instead of going through the entire process of having to run
telnetto connect to the remote system, log in with your username and password, run the command(s) on that system, and then log out. You just run theservmaintscript as given in Listing 6; everything else is done for you. Of course, if you give a password or other proprietary information in such a script, there is a security consideration; at minimum, you should change the file's permissions so that you're the only user (besides the superuser) who can read it.Any repetitive task involving system interaction can be programmed in Expect -- it's capable of branching, conditionals, and all other features of a high-level language so that the response and direction of the interaction with the program(s) can be completely automated.
ConclusionIn an office setting, UNIX systems can handle many of the tasks that are normally handled by standalone computers running other operating systems -- and with their rich supply of command-line tools, they're capable of productivity boosters that can't be found anywhere else.
This article introduced several techniques and concepts to increase your office productivity using UNIX command-line tools and applications. You should be able to apply these ideas to your own office situations and, with a little command-line ingenuity, come up with even more ways to save time and be more productive.
Learn
- "Expect exceeds expectations" (developerWorks, April 2002): For a concise introduction to the Expect language, read Cameron Laird's article.
- "Hone your regexp pattern-building skills" (developerWorks, July 2006): This article gives examples of powerful, real-world regular expressions that can increase your office productivity.
- "Tcl your desktop" (developerWorks, June 2006): Make your work desktop uncluttered and efficient by following the steps outlined in this article.
- "Text processing with UNIX" (developerWorks, August 2006): This article demonstrates the power of UNIX command-line tools for processing text.
- "Working smarter, not harder" (developerWorks, August 2006): Part 2 of the "Speaking UNIX" series contains a primer on the
historybuilt-in.
- AIX and UNIX: Visit the developerWorks AIX and UNIX zone to expand your UNIX skills.
- New to AIX and UNIX: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.
- developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
- Podcasts: Tune in and catch up with IBM technical experts.
Get products and technologies
Discuss
- leave utility: If your UNIX system doesn't come with the
leaveutility, you can download a free copy from netbsd.org.
- date tool: The GNU Project's implementation of the
datetool contains an extension for outputting the number of seconds since the UNIX epoch. Download a free copy from the GNU Project Web site.
- IBM trial software: Build your next development project with software for download directly from developerWorks.
- Participate in the AIX and UNIX forums:
- AIX 5L -- technical forum
- AIX for Developers Forum
- Cluster Systems Management
- IBM Support Assistant
- Performance Tools -- technical
- Virtualization -- technical
- More AIX and UNIX forums
- Participate in the developerWorks blogs and get involved in the developerWorks community.
Michael Stutz is author of The Linux Cookbook, which he also designed and typeset using only open source software. His research interests include digital publishing and the future of the book. He has used various UNIX operating systems for 20 years. You can reach him at stutz@dsl.org.
Monitoring Processes with pgrep By Sandra Henry-Stocker
This week, we're going to look at a simple bash script for monitoring processes that we want to ensure are running all the time. We'll use a couple cute scripting "tricks" to facilitate this process and make it as useful as possible. The basic command we're going to use is pgrep. For those of you unfamiliar with pgrep, it's a very nice Solaris command that looks in the process queue to see whether a process by a particular name is running. If it finds the requested process, it returns the process id. For example: % pgrep httpd 1345 1346 1347 1348 This output tells us that there are four httpd processes running on our system. These processes might look like this if we were to execute a ps -ef command: % ps -ef | grep httpd output The pgrep command, therefore, accomplishes what many of us used to do with strings of Unix command of this variety: % ps -ef | grep httpd | grep -v grep | awk '{print $2}' In this command, we ran the ps command, narrowed the output down to only those lines containing the word "httpd", removed the grep command itself, and then printed out the second column of the output, the process id. With pgrep, extracting the process ids for the processes that we want to track is faster and "cleaner". Let's look at a couple code segments. First, the old way: for PROC in [ proc1 proc2 proc3 proc4 proc5 ] do RUNNING = `ps -ef | grep $PROC | grep -v grep | wc -l` if [ $RUNNING ge 1 ]; then echo $proc1 is running else echo $proc1 is down fi done For each process, we generate a count of the number of instances we detect in the ps output and, if this number is one or more, we issue the "running" output. Otherwise, we display a message saying the process is down. Now, here's out replacement code using pgrep: for PROC in [ proc1 proc2 proc3 proc4 proc5 ] do if [ `pgrep $PROC` ]; then echo $PROC is running else echo $PROC is down fi done In this case, we've simplified our code in a couple of ways. First, we rely on pgrep to give is output (procids) if the process is running and nothing if it isn't. Second, because we're not using ps and grep, we don't have to remove the output that isn't relevant to our task. We don't have to remove the ps output relating to the other running processes and to the process generated by our grep command. The process for killing a set of processes would be quite similar. In fact, we could use both pgrep and a "sister" command, pkill in a similar manner. for PROC in [ proc1 proc2 proc3 proc4 proc5 ] do if [ `pgrep $PROC` ]; then pkill $PROC else echo $PRIC is not running fi done The pgrep command is more predictable because we know we're going to get only the process id and that we won't be matching on other strings that just happen to appear in the ps output (e.g., if someone were editing the httpd.conf file). The pgrep, pkill and related commands are not only easier to use. The code is easier to read and understand. One of the reasons for using sequences of commands such as this: ps -ef | grep $PROC | grep -v grep | wc -l was to ensure that we knew what our answer would have to look like. If we left off the final "wc -l", we might get one or a number of pieces of output and have to deal with this fact when we went to check it. In addition, we could use similar logic when the number of processes, rather than just some or none, was important. We would just check the number against what we expected to see. Even so, anyone reading this script a year later would have to stop and think through this command. This is not true for pgrep. The command "pgrep httpd" is easy and quick to interpret as "if httpd is running". The "if [ `pgrep $PROC` ]" is especially efficient as well. This statement tests whether there is output from the command and is compact and readable. Much as I love Unix for the way it allows me to pipe output from one command to the other, I love it even more when I don't have to.sh -x By S. Lee Henry Whenever you enter a command in a Unix shell, whether interactively or through a script, the shell expands your commands and passes them on to the Unix kernel for execution. Normally, the shell does its work invisibly. In fact, it so unobtrusively processes your commands that you can easily forget that it's actually doing something for you. As we saw last week, presenting the shell with a command like "rm *" can, on rare occasion, results in a complaint. When the shell balks, producing an error indicating that the argument list is too long, it suddenly reminds us of its presence and that it is subject to resource limitations just like everything else. Invoking the shell with an option to display commands as it processes them is another way to become acquainted with the shell's method of intercepting and interpreting your commands. The Bourne family shells use the option -x. If you enter the shell using a -x, then commands will be displayed for you before execution. For example: boson% /bin/ksh -x $ date + date Mon Jun 4 07:11:01 EDT 2001 You can also see file expansion as the shell provides it for you: $ ls oops* + ls oops1 oops2 oop3 oops4 oops5 oopsies oops1 oops2 oop3 oops4 oops5 oopsies This is all very exciting, of course, but of limited utility once you get a solid appreciation of how hard the shell is working for you command line after command line. The sh -x "trick" can be very useful when you are debugging a script though. Instead of inserting lines of code like "echo at end of loop" to help determine your code is failing, you can change your "shebang" line to include the -x option: #!/bin/sh -x Afterwards, when you run the script, each line of code will display as it is processed so you can easily see which of the commands are working and where your breakdown is occurring. This is far more useful than looking at no output or little output and wondering where processing is hanging up -- especially true for a complex script where execution follows numerous paths. Being able to watch the executed commands and the order in which they are executed while the script is running can be an invaluable debugging aid -- particularly for complex scripts that don't write much output to the screen while running.How Many is Too Many? By Sandra Henry-Stocker I surprised myself recently when I issued a command to remove all the files in an old, and clearly unimportant, directory and received this response: bin/rm: arg list too long I seldom encounter this response when cleaning up server directories that I manage, so seeing it surprised me. When I began listing the directory's contents, I wasn't surprised that my command had failed. The directory contained more than 200,000 small, old, and meaningless files, which would take a long time to list, consumes quite a bit of directory file space, and would comprise a very long command line if the shell were about to manage it. Even if every file name had only eight characters, then a line containing all of their names (with blank characters separating the names) would be nearly 1.8 million bytes long. Not surprisingly, my shell balked at the task. Situations like this remind us that, even though Unix is flexible, powerful, and fun, each of the commands has built in limits. My shell could not allocate adequate space to "expand" the asterisk that I presented in my "rm *" command to a list of all 200,000+ files. Of course, Unix offers several ways to solve every problem and running out of space to expand a command merely invites one to solve the problem differently. In my case, the easiest solution was to remove the directory along with its contents. The rm -r command, since it doesn't require any argument expansion, is "happy" to comply with such a request. Had I not wanted to remove every file in the directory, I would have gone through a little more trouble. I could have removed subsets of the files, using commands like "rm a*" or "rm *5" until I had removed all of the unwanted files. A third approach would have been the appropriate for preserving only a small number of the directory's files ? especially files that are easily described by substring or date. I would have tarred up the interesting files using tar and a wild card or a find command to create an include file. You will not often encounter situations where the shell will be unable to expand your file names into a workable command. Few directories house as many files as the one that I was cleaning up, and the Unix shells allocate enough buffer space for most commands that you might enter. Even so, limits exist and you might happen to bump into one of them every few years.
So you're new to Linux and wondering how this virtual terminal stuff works. Well you can work in six different terminals at a time. To move around from one to another:
To change to Terminal 1 - Alt + F1 To change to Terminal 2 - Alt + F2 ... To change to Terminal 6 - Alt + F6That's cool. But I just did locate on something and a lot of stuff scrolled up. How do I scroll up to see what flew by?Shift + PgUp - Scroll Up Shift + PgDn - Scroll Down Note: If you switch away from a console and switch back to it, you will lose what has already scrolled by.If you had X running and wanted to change from X to text based and vice versa
To change to text based from X - Ctrl + Alt + F(n) where n = 1..6 To change to X from text based - Alt + F7Something unexpected happened and I want to shut down my X server.
Just press:Ctrl + Alt + Backspace