Softpanorama
(slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Softpanorama Search

logrotate

News	Perl Log Analysis and Management Tools	Recommended Links	Log analyzers	Log Colorizing
Perl-Logrotate	Log rotation in Solaris	logger	Humor	Etc

Most large well-written Unix programs produce activity and error logs. These logs are important sources of information when debugging system problems, investigating security incidents, and monitoring system workload.

log files grow and can eventually consume all of the free space in a filesystem and interfere with the normal operation of the system.

The logrotate is a small C-written utility (there is also Perl rewrite of Logrotate) script helps balance record-keeping with storage limitations by periodically saving log files and starting a new ones. A pre-determined number of log files (of each type) are kept on disk, and older logs are deleted automatically.

Both RHEL and Suse include logrotate and it is preconfigured for basic log-rotation. If you have some logs under /var/log that are suffixed with a dot and a number (/var/log/daemon.log.1) or even compressed (/var/log/mysql.log.5.gz), it means they were rotated by logrotate.

Logrotate needs a simple configuration file to help it manage the logs. The options used in the logrotate.conf file are documented in the manpage for logrotate. Many options are available, including options to to configure rotation to take place only after the logfile reaches a certain size, to have old logfiles e-mailed to you, and to compress saved logfiles to reduce storage requirements.

You can have multiple configuration files, one for every log, or maintain multiple logs through one configuration file. The default configuration file is /etc/logrotate.conf. If you have services that keep a log under /var/log like Apache and MySQL, you'll find their configuration files under /etc/logrotate.d.

Here is a sample logrotate.conf:

weekly rotate 4 create include /etc/logrotate.d /var/log/wtmp { missingok monthly create 0664 root utmp rotate 1 }

The first four keywords, weekly, rotate 4, create and include are global directives, whereas the directives missingok, monthly, create and rotate are specific to the log file /var/log/wtmp. A local directive takes precedence over a global directive if it appears in both, like rotate in the listing above.

Working with Logrotate

If you have a program that keeps a log, you can save some serious effort by using logrotate to keep them in order for you. Logrotate can also work on logs of services that are run by a normal user. It also comes in handy if you have a program that keeps multiple logs and writes a lot to them.

Let's assume we have a new service "foobar" which is configured to log under /home/bodhi/logs/foobar and runs as a normal user's process. We keep its logrotate configurations under /home/bodhi/config in a file named foobar. Like this one:

/var/log/foobar/*.log { daily missingok rotate 7 compress delaycompress create 640 bodhi bodhi mail bodhi@localhost sharedscripts postrotate /etc/init.d/foobar restart endscript }

We use daily to ensure that the logs are rotated everyday and rotate 7 to make sure no more than 7 days of logs are kept. When a log expires, it's emailed to bodhi@localhost before being deleted as specified by mail. compress shrinks rotated logs except yesterday's because of delaycompress. Logrotate doesn't complain if the log files are missing thanks to missingok. After the logs have been rotated, a new one is created by create as being owned by the user and group bodhi and the service is restarted.

Putting it to work

Now that we have our configuration file in place, it's time to create a cron job for it to actually make logrotate work. You do this by running crontab -e and adding a new cron job to the file.

00 00 * * * /usr/sbin/logrotate -s /home/bodhi/config/logrotate.status /home/bodhi/config/foobar

Cron ensures that the command runs at midnight everyday. The command has three parts. /usr/sbin/logrotate is the path to logrotate. The -s /home/bodhi/config/logrotate.status option specifies where logrotate keeps its status information. This file has to be writeable by the user running the cron. The last argument /home/bodhi/config/foobar tells logrotate to use our custom configuration file.

To test the configuration file, issue the command specified in the cron job with an additional -f switch. This will force the logs to be rotated. It helps to run logrotate with this switch every time the configuration files are changed.

Logrotate is a very simple but extremely useful tool. Many people fail to realize that logrotate can work for them as well. In this article, the configuration file instructs logrotate to take charge of a process running under a user.

We used several configuration options, but a visit to man logrotate for more options is a good idea before writing your own. If you have logrotate managing logs on your machine, reviewing its configuration files of the services its maintaining is a good way to learn logrotate's common usage.

Logrotate

In Red Hat and Suse log rotation it is handled by the logrotate utility and configured using files in /etc/logrotate.d.

By default, logs are kept for 4 weeks.

To change the time your logs are kept for, edit /etc/logrotate.conf and change the rotate parameter. You can also override this for specific logs by editing the files in /etc/log/logrotate.d/. For more details see the logrotate man page.

If you have more that one Linux/Unix server then you should probably set one machine up as a log server and send all your logs to a central machine as well as storing them locally. This makes is easier to process your logs and means you have a second copy should a machine have it's security compromised.

To make a server accept log messages from other machines you need to enable syslog remote reception. This can be done by editing /etc/sysconfig/syslog and adding "-r" to the SYSLOGD_OPTIONS so it looks like this

SYSLOGD_OPTIONS="-m 0 -r"

Then restart syslog by doing

# /etc/rc.d/init.d/syslog restart

You should see a message in the log files about remote reception

To make a machine send it's log messages to your log server you need to add an entry to /etc/syslog.conf. Something like

*.info @loghost

should be all that is needed. See the man page for more details (man 5 syslog.conf)

You also need to restart syslog after this change. The procedure is the same as above.

Restricting Access

Using tcp wrappers is restrict access is simple to set up. Most services are already configured to use tcp wrappers. The most secure way to configure tcpwrappers is to deny everything be default.

/etc/hosts.deny should contain only the line

ALL: ALL

Then you can add services to /etc/hosts.allow as appropriate. For example

swat: 192.168.5. : severity notice
sshd: KNOWN : severity notice
ALL: LOCAL : severity notice

You do not need to restart anything after changing these files. They are reread every time a connection is made.

Following are a few points of interest in the logrotate tour:

/usr/sbin/logrotate -- Utility residence
/etc/logrotate.conf -- Default configuration file
/etc/logrotate.d -- Default directory for other configuration files
/var/lib/logrotate.status -- Default state file

On RH, logrotate is run from a daily job directory /etc/cron.daily/logrotate. When logrotate is called, it obtains information about which files it is processing by using the default configuration file /etc/logrotate.conf. Alternate configuration files can be used if optioned. The configuration file defines the file to rotate and the parameters to apply. A configuration file entry may look like this:

#daily
"/var/log/logfile.log" {
        daily
        rotate 7
        compress
        }

Using the configuration shown here, the file logfile.log would be compressed and rotated and given the name of logfile.log.1.gz. After day two, the file logfile.log.1.gz would be rotated to logfile.log.2.gz and new copy of logfile.log.1.gz produced from logfile.log. At the end of the configuration file schedule, the directory would contain seven copies of the database. On the eighth day, the copy with the extension logfile.log.7.gz would be replaced with another rotated copy. A long listing (ls -al) in the directory would display seven copies of the database files, with dates preserved from when they were first created.

The man pages for logrotate explain the parameters used to control logrotate. The remainder of this article describes how logrotate can be used to produce a backup schema.

When planning the backup process, I used a variation on the grandfather-father-son backup tape rotation method. The backup set consists of grandfather(monthly), father(weekly), and son(daily) as shown in Figure 1.

Under this grandfather-father-son implementation method, a full backup is performed every day (Monday-Sunday); a full backup of the oldest daily is preserved every Sunday; a full backup of the oldest weekly is preserved the last day of a four-week cycle. This assumes you start the daily cycle on a Monday. After 1 year, 6 dailies, 3 weeklies, and 12 monthly copies are preserved.

Logrotate works similarly to a stack in programming. As each file is rotated, it is pushed onto the stack and forces other files down the stack. The size of the stack is determined by the rotate value in the logrotate configuration file. Once the stack capacity is reached, the first file in gets pushed out. Figure 1 illustrates the idea.

To produce the backup schema shown in Figure 1, create the configuration file /etc/logrotate_backup.conf from the following listing:

##############################
#
#Name:logrotate_backup.conf
#
#Description:script designed to produce backups
#grandfather(monthly),father(weekly),son(daily)
#
#Author: Sean D. Conway
#
#Wed Aug 28 18:12:59 CDT 2002
#
#Revision:
#
#NOTE:
#replace the path to be backed up.
#don't remove "" good for files that have spaces.
#maintain the "." file extensions this will preserve the frequency
################################

#daily (son)
"/home/oracle/backup/database" {
        daily
        rotate 7
        compress
        }

#weekly (father)
"/home/oracle/backup/database.7.gz" {
        weekly
        rotate 4
        }

#monthly (grandfather)
"/home/oracle/backup/database.7.gz.4" {
        monthly
        rotate 12
        }

This configuration file is divided into three sections: daily, weekly, and monthly. The following listing provides some comments to explain the parameter lines found in the script:

#daily (son)                         <--comments line
"/home/oracle/backup/database" {     <--file to rotate
        daily                        <--rotation schedule
        rotate 7                     <--rotation interval
        compress                     <--compress the file
        }

The parameters here enable the logrotate program to rotate and compress daily a file called database. The rotation interval is seven. At the end of one week, the storage directory will contain the following files: database.1.gz, database.2.gz, database.3.gz, database.4.gz, database.5.gz, and database.6.gz. The weekly configuration would rotate the file database.7.gz to database.7.gz.1. The file database.1.gz would be the most current.

The logrotate utility can access the configuration files using two methods:

1. The configuration file can be placed in the default configuration file directory /etc/logrotate.d. This directory is listed in logrotates default configuration file /etc/logrotate.conf. All files in /etc/logrotate.d are additional configuration files that will be processed by the program.

2. An alternate method is to force the logrotate program to use an alternate configuration file from a command line. The following is the cron entry to support logrotate running each day at 07:00 using a configuration file called /etc/logrotate_backup.conf.

00 07 * * * /usr/sbin/logrotate -f /etc/logrotate_backup.conf

I prefer the second method because it allows more flexible scheduling.

The logrotate backup mechanism is used to support three database servers as shown in Figure 2. Each database server produces a backup file and sends it to a server called a tape_gate. On the tape_gate server, logrotate runs daily and takes care of producing the rotation schema. The complete backup series is exported to tape for offsite storage.

Tape_gate also runs a licensed tape client for accessing the tape silo. This configuration allows the administrators to use the client software to schedule and access the backups. The administrator can use the backups stored locally or pull the file back from the tape storage device. All the file creation dates are preserved, and a simple grep on a month name will produce a list of backup files available for that month.

To get support for this backup solution, I had to address some design limitations. This implementation requires a server to process the backups. The horsepower required for the server is minimal. For this installation, I used a reclaimed Pentium desktop with an additional cost of two hard drives. It does provide an alternate point for accessing backups. I find it easier to access a backup from a disk rather than searching a tape library.

Backup files are moved from the source to tape_gate using secure ftp. At selective network slow periods the additional network traffic is acceptable. Secure ftp provides access to a ready backup from tape_gate if needed.

Table 1 shows the complete set of filenames after a one-year cycle. The drive capacity must be sufficient to support the number of system backups being handled. The cost for hard drive capacity today is lower than it was ten years ago. Sixty-gig IDE drives are common. At 14-MB compressed, I was able to get all the backups on a 60-GB drive. The three-database server backups total 500 MB compressed.

The feature that sold this design was cost. A client license for accessing the tape silo costs $1200. Each of the databases runs a different operating system, so each would have needed a different client license. Instead, two drives cost $300 and $1200 for one client license for tape_gate; I saved $2100.

The alternative was to create NFS mount on all the servers.

I have been asked to add additional backups to the server. It seems this common approach to backups for a variety of systems can make life easier for administrators. Certainly, one common backup mechanism for all systems is easier to remember than a different backup method for each system.

Sean D. Conway is a former college instructor turned Network System Specialist for a regional telecommunications company in Canada. Working with administrators, they are responsible for the care and feeding of operational support systems inside the telecommunication cloud. He still dabbles in network theory lectures. His formal electronic engineering technology training and education background are valuable tools in achieving a goal -- making learning about computers and networks easier.

Notes:

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Some amount of grammar and spelling errors should be expected.
The site contain some broken links as it develops like a living tree... Please try to use Google, Open directory, etc. to find a replacement link (see HOWTO search the WEB for details). We would appreciate if you can mail us a correct link.

Google Search
Open directory	Research Index

Old News ;-)

Rotating Linux Log Files MDLog-sysadmin

Comments Chris Riney - June 7, 2006
I’m glad that he mentioned that this was based on a Debian systems, since the RedHat & SuSE systems that I work with do not have the syslog “Feature” that is described (must be part of syslogNG which I’m not familiar with). Both of these major commercial distributions manage the syslog files via logrotate.

Actually I’m just starting to work with SuSE Enterprise, and I’m noticing the differences in the way RedHat & SuSE implement scheduling Hourly, Daily, Weekly, & Monthly rotations.

A feature of of LogRotate 3.7.x that I found, (and was looking for) is the ‘dateext’ which appends the rotation date to the end instead of a sequential number. The problem I have with the sequential number, is that logrotate moves the older log’s up the sequence, which causes those file to be backed-up multiple times with differential backups.

One item that you didn’t cover, is how logrotate knows when it is the proper time to rotate a daily, weekly, or monthly file. Part of that is controlled by the status file (/var/lib/logrotate.status on most Linux system) which tracks when a particular file was last rotated. The other part is that Monthly rotations are done with the first run of logrotate that month, and Weekly’s are run if the last run was later in the previous week, or the last rotation was over 7 days prior (Sunday=0 Saturday=6). So if you are running logrotate daily, then it works itself out to the first day of the month, and Sunday’s for the weeklies.

logrotate notes on usage

These are my personal notes on how to use `logrotate' on a linux system.
I'm writing these notes to help me make sense of how things work.
2005-5-5: I don't know where the logrotate source went to. All of my links to logrotate are broken. The source in about September 2001 was logrotate-3.3.tar.gz. That probably isn't very useful though.

How logrotate is invoked from cron.

On one of my RedHat 5.2 systems, `crond' is started from the script /etc/rc.d/init.d/crond. The manual `man cron' says that cron
searches for /etc/crontab and the files in the /etc/cron.d/ directory.
My /etc/crontab file contains the following.
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
0  0 1 * * root run-parts /etc/cron.monthly
And the /etc/cron.d directory contains nothing.
So clearly, any logrotate commands are being run out of one or more of the /etc/cron.* files. The manual `man 5 crontab' indicates that the above commands mean that the program `run-parts' will be run as user `root' for each of the scripts.

The command `run-parts' turns out to be the script /usr/bin/run-parts, which is very short, as follows.
#!/bin/bash

# run-parts - concept taken from Debian

# keep going when something fails
set +e

if [ $# -lt 1 ]; then
        echo "Usage: run-parts <dir>"
        exit 1
fi

if [ ! -d $1 ]; then
        echo "Not a directory: $1"
        exit 1
fi

for i in $1/* ; do
        [ -d $i ] && continue
        if [ -x $i ]; then
                $i
        fi
done

exit 0
In essence, this just runs all of the scripts in the specified directory. E.g. each hour, the executable plain files in /etc/cron.hourly are run.
It turns out that the `logrotate' program is invoked from /etc/cron.daily. This is the contents of the file /etc/cron.daily/logrotate.
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
This means that the only file that `logrotate' gets its instructions from directly is /etc/logrotate.conf, which contains the following lines.
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 8 weeks worth of backlogs
rotate 8

# send errors to root
errors root

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own lastlog or wtmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    rotate 1
}

# system-specific logs may be configured here
This is all explained in the manual `man logrotate'.

Operation of logrotate.
The directory /etc/logrotate.d contains the following files.
-rw-r--r--   1 root     root          354 Oct 13  1998 apache
-rw-r--r--   1 root     root          108 Aug 28  1999 cron
-rw-r--r--   1 root     root          188 Oct 14  1998 linuxconf
-rw-r--r--   1 root     root          156 Oct 13  1998 mgetty
-rw-r--r--   1 root     root          327 Aug 12  1998 syslog
-rw-r--r--   1 root     root          457 Sep 10  1998 uucp
The file /etc/logrotate.d/apache is the one I'm interested in for this exercise. This file contains the following.
/var/log/httpd/access_log {
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}

/var/log/httpd/agent_log {
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}

/var/log/httpd/error_log {
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}

/var/log/httpd/referer_log {
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
}
When I installed the latest version of Apache to get the PHP3 and PostgreSQL to work (around 20 March 2000) on my web server machine, I installed Apache so that the log files were in /home2/apache/logs instead of /var/log/httpd.
Therefore what I need to do now is to modify the /etc/logrotate.d/apache file so that the files referred to are all in the directory /home2/apache/logs instead. My new /etc/logrotate.d/apache script is as follows, and I saved the old one in directory /etc/logrotate.d/old1.
# The new improved logrotate script for apache on fox.

/home2/apache/logs/*-access_log {
    rotate 9
    monthly
    errors ak@fox.txpxlxgy.org
    create
    ifempty
    olddir /home2/apache/logs/oldlogs
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
    }

/home2/apache/logs/*-combref_log {
    rotate 9
    monthly
    errors ak@fox.txpxlxgy.org
    create
    ifempty
    olddir /home2/apache/logs/oldlogs
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
    }

/home2/apache/logs/*-error_log {
    rotate 9
    monthly
    errors ak@fox.txpxlxgy.org
    create
    ifempty
    olddir /home2/apache/logs/oldlogs
    postrotate
        /usr/bin/killall -HUP httpd
    endscript
    }
For a log file of an ad-hoc utility I wrote, I have now created this configuration file /etc/logrotate.d/dping for `logrotate'.
# The new improved logrotate script for dping on fox.

/var/log/dping.txt {
    rotate 9
    monthly
    compress
#    delaycompress
    copytruncate
    ifempty
    errors ak@fox.txpxlxgy.org
    postrotate
    endscript
    }
Since the creating process cannot be HUPped to make it work with a new log file, I've used `copytruncate' to copy the file and then truncate it. The `delaycompress' is not useful because the original log file is copied rather than moved. Creation of .cvsignore file

Now on Sunday 30 September 2001, it's time to get `logrotate' going on my new SuSE 7.1 web server `dog'. The problem is that SuSE 7.1 does not come with logrotate software on it. So I had to go and look for it. The link at redhat was wrong, but by looking around a bit, I finally found it at ftp://ftp.redhat.com/pub/redhat/linux/code/logrotate/. I downloaded this file: logrotate-3.3.tar.gz.
[2005-5-5: I think it has disappeared again now. But here's a backup copy of logrotate-3.3.tar.gz.]
This can be compiled very simply with `make' and `make install'. There's a manual logrotate.8 with it too. The binary is installed as /usr/sbin/logrotate. So now all I have to do is write a configuration file for the logrotate program and then write a cron script for it.

(Note that since my main initial consideration is to rotate the httpd logs for the beginning of October 2001, I could just use the Apache tool for this purpose, which is /usr/local/apache/bin/rotatelogs on my system. However, I can't understand the documentation for this. So I'm playing safe and using the more flexible redhat `logrotate' tool instead.)

On my SuSE 7.1 machine `dog', the manual says that after reading the per-user crontab files in /var/spool/cron/tabs on start-up, the cron process reads the file /etc/crontab. On my machine as configured, I find the following.
root@dog# more /etc/crontab
SHELL=/bin/sh
PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin
MAILTO=root

#-* * * * *    root  test -x /usr/sbin/atrun && /usr/sbin/atrun
0 21 * * *     root  test -x /usr/sbin/faxqclean && /usr/sbin/faxqclean
5 22 * * *     root  test -x /usr/sbin/texpire && /usr/sbin/texpire
25 23 * * *    root  test -e /usr/sbin/faxcron && sh /usr/sbin/faxcron | mail FaxMaster

#
# check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly
#
-*/15 * * * *   root  test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons
59 *  * * *     root  rm -f /var/spool/cron/lastrun/cron.hourly
14 0  * * *     root  rm -f /var/spool/cron/lastrun/cron.daily
29 0  * * 6     root  rm -f /var/spool/cron/lastrun/cron.weekly
44 0  1 * *     root  rm -f /var/spool/cron/lastrun/cron.monthly
There's nothing here to help me with initiating my daily script. (By the way, the fax commands are a bit worrying. I'll get rid of those when I understand exactly what they do. They obviously produce many meaningless message which root receives every day!)
So my next step is to look at the files in /etc/cron.d, because the cron manual says that all scripts in this directory are read next. On my machine the only file in /etc/cron.d is a script `seccheck', which produces copious useless messages to root every day and week. (I'll see if I can get rid of that some day too!)

The directory /etc/cron.daily contains a script `aaa_base_rotate_logs' which contains a complex set of rotation rules, but how are the scripts in this directory invoked? Hmmm... Maybe they're invoked from that `/usr/lib/cron/run-crons' script. Yes!! That's where it's invoked from. Yet another big, incomprehensible script. The core of that script is the following Bourne-shell loop.
SPOOL=/var/spool/cron/lastrun
for CRONDIR in /etc/cron.{hourly,daily,weekly,monthly} ; do
    test -d $CRONDIR || continue
    BASE=${CRONDIR##*/}
    test -e $SPOOL/$BASE && {
        case $BASE in
          cron.hourly)  TIME="-cmin  +60 -or -cmin  60" ;;
          cron.daily)   TIME="-ctime +1  -or -ctime 1"  ;;
          cron.weekly)  TIME="-ctime +7  -or -ctime 7"  ;;
          cron.monthly) TIME="-ctime +`date -u +%d`"    ;;
        esac
        eval find $SPOOL/$BASE $TIME | xargs -r rm -f
    }
    if test ! -e $SPOOL/$BASE ; then
        touch $SPOOL/$BASE

        # keep going when something fails
        set +e
        for SCRIPT in $CRONDIR/*[^~,] ; do
            test -d $SCRIPT && continue
            test -x $SCRIPT || continue
           case "$SCRIPT" in
               *.rpm*)         continue ;;
               *.swap)         continue ;;
               *.bak)          continue ;;
               *.orig)         continue ;;
               \#*)            continue ;;
           esac
           /sbin/checkproc $SCRIPT && continue
           nice -15 $SCRIPT
        done
    fi
done
Now what does this mean?
The command `BASE=${CRONDIR##*/}' means that BASE is set to the `longest substring of $CRONDIR which matches pattern "*/"', according to my Bash reference card. This just means that the leading path components are removed. (This is done more simply in C-shell!) In the case of the daily cron job, if there is a file /var/spool/cron/lastrun/cron.daily (which is true), then the following command is run.
eval find /var/spool/cron/lastrun/cron.daily -ctime +1 -or -ctime 1 | xargs -r rm -f
The `xargs' command (which I have never seen before) builds and executes a command line from standard input. Weird! The `xargs' manual says this.
If  the  standard  input  does not contain any non-blanks,
do not run the command.  Normally, the command is run once
even if there is no input.
So in this case, the command `rm -f' is executed for each of the files with modification times within 24 hours of the current time. I don't really understand this.
It looks like the file removal commands in the file /etc/crontab are designed to synchronise the operation of the quarter-hour operations. The commands are only executed if the files in /var/spool/cron/lastrun have been removed. What a convoluted way of achieving a simple objective!!

Next any file in the directory /etc/cron.daily which do not end in the characters `~' or `,' (presumed to be edited files) are executed if they are executable and do not have the endings .rpm, .swap, .bak or .orig as follows.
           /sbin/checkproc $SCRIPT && continue
           nice -15 $SCRIPT
The loop continues is the process is already running. Otherwise it is run with nice level 15.
What this all means finally is that any script in the directory /etc/cron.daily will be run at 00:00 on each day. More to the point, the /etc/cron.monthly scripts are run at 00:00 at the beginning of each month. It all looks a bit dodgy because the `date' test is `date -u', which gives the current UTC day. But this should work, although the motivation for the `-u' is not quite clear.

All I have to do now is write a script and put it in /etc/cron.monthly. No problem!

Sys Admin Logrotate -- A Backup Solution
Sean D. Conway

After a system failure, tracking down the media, determining the last known good backup, and starting the recovery process can be tricky. I implemented the solution described in this article to reduce the stress of performing backups. Using a program found in Linux called logrotate, I created a backup solution to consolidate and automate the backup process. Logrotate's configuration file contains the details to create the backups, compress the backup files, provide automatic rotation, and remove stale files.

Logrotate is a utility designed to handle the management of log files. Logrotate maintains a series of log files, automatically rotates the files, and removes the files based on the parameters provided in a configuration file. You can specify the name of any file in the logrotate configuration file, it is treated the same as a log file.

I used logrotate when I worked with Red Hat (RH) 6.0. Since then, I have found it to be part of a standard RH install. The current version is available from:
http://www.rpmfind.net/linux/RPM/
The program can be installed with an rpm -Uvh logrotate-3.6.4-1.i386.rpm (current RH 7.3 version) command. Following are a few points of interest in the logrotate tour:
/usr/sbin/logrotate -- Utility residence
/etc/logrotate.conf -- Default configuration file
/etc/logrotate.d -- Default directory for other configuration files
/var/lib/logrotate.status -- Default state file
On RH, logrotate is run from a daily job directory /etc/cron.daily/logrotate. When logrotate is called, it obtains information about which files it is processing by using the default configuration file /etc/logrotate.conf. Alternate configuration files can be used if optioned. The configuration file defines the file to rotate and the parameters to apply. A configuration file entry may look like this:
#daily
"/var/log/logfile.log" {
        daily
        rotate 7
        compress
        }
Using the configuration shown here, the file logfile.log would be compressed and rotated and given the name of logfile.log.1.gz. After day two, the file logfile.log.1.gz would be rotated to logfile.log.2.gz and new copy of logfile.log.1.gz produced from logfile.log. At the end of the configuration file schedule, the directory would contain seven copies of the database. On the eighth day, the copy with the extension logfile.log.7.gz would be replaced with another rotated copy. A long listing (ls -al) in the directory would display seven copies of the database files, with dates preserved from when they were first created.
The man pages for logrotate explain the parameters used to control logrotate. The remainder of this article describes how logrotate can be used to produce a backup schema.

When planning the backup process, I used a variation on the grandfather-father-son backup tape rotation method. The backup set consists of grandfather(monthly), father(weekly), and son(daily) as shown in Figure 1.

Under this grandfather-father-son implementation method, a full backup is performed every day (Monday-Sunday); a full backup of the oldest daily is preserved every Sunday; a full backup of the oldest weekly is preserved the last day of a four-week cycle. This assumes you start the daily cycle on a Monday. After 1 year, 6 dailies, 3 weeklies, and 12 monthly copies are preserved.

Logrotate works similarly to a stack in programming. As each file is rotated, it is pushed onto the stack and forces other files down the stack. The size of the stack is determined by the rotate value in the logrotate configuration file. Once the stack capacity is reached, the first file in gets pushed out. Figure 1 illustrates the idea.

To produce the backup schema shown in Figure 1, create the configuration file /etc/logrotate_backup.conf from the following listing:
##############################
#
#Name:logrotate_backup.conf
#
#Description:script designed to produce backups
#grandfather(monthly),father(weekly),son(daily)
#
#Author: Sean D. Conway
#
#Wed Aug 28 18:12:59 CDT 2002
#
#Revision:
#
#NOTE:
#replace the path to be backed up.
#don't remove "" good for files that have spaces.
#maintain the "." file extensions this will preserve the frequency
################################

#daily (son)
"/home/oracle/backup/database" {
        daily
        rotate 7
        compress
        }

#weekly (father)
"/home/oracle/backup/database.7.gz" {
        weekly
        rotate 4
        }

#monthly (grandfather)
"/home/oracle/backup/database.7.gz.4" {
        monthly
        rotate 12
        }
This configuration file is divided into three sections: daily, weekly, and monthly. The following listing provides some comments to explain the parameter lines found in the script:
#daily (son)                         <--comments line
"/home/oracle/backup/database" {     <--file to rotate
        daily                        <--rotation schedule
        rotate 7                     <--rotation interval
        compress                     <--compress the file
        }
The parameters here enable the logrotate program to rotate and compress daily a file called database. The rotation interval is seven. At the end of one week, the storage directory will contain the following files: database.1.gz, database.2.gz, database.3.gz, database.4.gz, database.5.gz, and database.6.gz. The weekly configuration would rotate the file database.7.gz to database.7.gz.1. The file database.1.gz would be the most current.
The logrotate utility can access the configuration files using two methods:

1. The configuration file can be placed in the default configuration file directory /etc/logrotate.d. This directory is listed in logrotates default configuration file /etc/logrotate.conf. All files in /etc/logrotate.d are additional configuration files that will be processed by the program.

2. An alternate method is to force the logrotate program to use an alternate configuration file from a command line. The following is the cron entry to support logrotate running each day at 07:00 using a configuration file called /etc/logrotate_backup.conf.
00 07 * * * /usr/sbin/logrotate -f /etc/logrotate_backup.conf
I prefer the second method because it allows more flexible scheduling.
The logrotate backup mechanism is used to support three database servers as shown in Figure 2. Each database server produces a backup file and sends it to a server called a tape_gate. On the tape_gate server, logrotate runs daily and takes care of producing the rotation schema. The complete backup series is exported to tape for offsite storage.

Tape_gate also runs a licensed tape client for accessing the tape silo. This configuration allows the administrators to use the client software to schedule and access the backups. The administrator can use the backups stored locally or pull the file back from the tape storage device. All the file creation dates are preserved, and a simple grep on a month name will produce a list of backup files available for that month.

To get support for this backup solution, I had to address some design limitations. This implementation requires a server to process the backups. The horsepower required for the server is minimal. For this installation, I used a reclaimed Pentium desktop with an additional cost of two hard drives. It does provide an alternate point for accessing backups. I find it easier to access a backup from a disk rather than searching a tape library.

Backup files are moved from the source to tape_gate using secure ftp. At selective network slow periods the additional network traffic is acceptable. Secure ftp provides access to a ready backup from tape_gate if needed.

Table 1 shows the complete set of filenames after a one-year cycle. The drive capacity must be sufficient to support the number of system backups being handled. The cost for hard drive capacity today is lower than it was ten years ago. Sixty-gig IDE drives are common. At 14-MB compressed, I was able to get all the backups on a 60-GB drive. The three-database server backups total 500 MB compressed.

The feature that sold this design was cost. A client license for accessing the tape silo costs $1200. Each of the databases runs a different operating system, so each would have needed a different client license. Instead, two drives cost $300 and $1200 for one client license for tape_gate; I saved $2100. The alternative was to create NFS mount on all the servers.

I have been asked to add additional backups to the server. It seems this common approach to backups for a variety of systems can make life easier for administrators. Certainly, one common backup mechanism for all systems is easier to remember than a different backup method for each system.

Sean D. Conway is a former college instructor turned Network System Specialist for a regional telecommunications company in Canada. Working with administrators, they are responsible for the care and feeding of operational support systems inside the telecommunication cloud. He still dabbles in network theory lectures. His formal electronic engineering technology training and education background are valuable tools in achieving a goal -- making learning about computers and networks easier.

Changing Apache log rotation behaviour on CentOS / RHEL
March 22, 2008

On a default install of CentOS or Red Hat Enterprise Linux, the log rotation script will automatically rotate the Apache log file each day and then reload the httpd service. This post looks at how to prevent this action from occuring automatically, or to change the behaviour to rotate the log files if your naming convention for log files is different from the default.

The cron daemon on a CentOS or Red Hat Enterprise Linux server by default runs the scripts in the directory /etc/cron.daily on a daily basis. This includes running the "logrotate" script which runs /usr/sbin/logrotate passing it the /etc/logrotate.conf configuration file. The default version of this logrotate.conf file is as follows:
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4



# create new (empty) log files after rotating old ones
create


# uncomment this if you want your log files compressed
#compress


# RPM packages drop log rotation information into this directory
include /etc/logrotate.d


# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    minsize 1M
    create 0664 root utmp
    rotate 1
}


# system-specific logs may be also be configured here.
These default variables are fairly straight forward and give you control over how frequently the log files are rotated, how many lots of previous log files to keep before deleting them, whether or not to compress them and so on. It also tells logrotate to process all the files in the directory /etc/logrotate.d

The file which controls log rotation of Apache on CentOS and Red Hat Enterprise Linux is named /etc/logrotate.d/httpd, and the default contents of this file are as follows:
/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}
What this file is saying, is to rotate all files that match the pattern /var/log/httpd/*.log. The default access and error log files in Apache on CentOS and RHEL are named "access_log" and "error_log" so they match this pattern.

If you have changed the default names of the log files and they don't match this pattern, then you can have logrotate rotate your log files by making sure the pattern is in this file in place of the /var/log/httpd/*.log line. For example, if your log files were named error.foo and access.foo then /var/log/httpd/*.foo would match them.

If you do not wish the logrotate script to rotate your Apache log files at all, then you can simply delete this file like so, logged in either as root or using sudo:
rm -f /etc/logrotate.d/httpd
The changes will take effect the next time the logrotate script is run.
Recommended Links

https://fedorahosted.org/logrotate/ index of the latest versions of the logrotate package

The Tao of Mac - HOWTO Use logrotate

Logrotate - Linux Config Wiki

Copyright © 1996-2009 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

Disclaimer:

The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with.

We do not warrant the correctness of the information provided or its fitness for any purpose

In no way this site is associated with or endorse cybersquatters using the term "softpanorama" with other main or country domains (e.g. softpanorama.com) with bad faith intent to profit from the goodwill belonging to someone else.

Last modified: August 11, 2009