Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Enterprise Logs Collection and Analysis Infrastructure

A system log is a recording of certain events. The kind of events found in a system log is determined by the nature of the particular log and any configurations used to control those events that are logged. System logs are usually text files containing a timestamp and other information specific to the message or subsystem.

This page presents several approaches to collect and monitor system logs, and first of all based on traditional Unix syslog facility.

Analysis of logs is much more important part of security infrastructure then Network Intrusion detection system -- a black hole that consumes untold number of millions of dollars each year in most developed countries.

Some important issues include:

• Logfile content: Improving the quality of the data in your logs. How much data you need in logs and what events are useful for logging and which are not.
• Logfile centralization: building a central loghost, and integrating MS Windows systems into your UNIX log system
• Log management: Centralization, parsing, and storing all that data possibly using such tools as Tivoli
• Log archiving: how long you need to store your logfiles

Notes: Those pages are written by people for whom English is not a native language. Some amount of grammar and spelling errors should be expected. This is a Spartan WHYFF (We Help You For Free) site. It cannot replace the best teachers and the best books. The site contain some obsolete pages as it develops like a living tree... Some links on older pages are broken. Please try to use Google, Open directory, etc. to find a replacement link (see HOWTO search the WEB for details). We would appreciate if you can mail us a correct link. Search Amazon by keywords:     Open directory Research Index

Old News ;-)

Help - IBM WebSphere Help System / Adapter configuration file samples

Adapters created using the Generic Log Adapter framework can be used for building log parsers for the Log and Trace Analyzer. The following adapter configuration files are provided as examples for creating rules-based adapters and static adapters.

Log file type	Adapter sample available	Directory
AIX errpt log	V4.3.3(rules), V5.1.0(rules), V5.2.0(rules)	<base_dir>\AIX\errpt\v4
AIX syslog	regex.adapter	<base_dir>\AIX\syslog\v4
Apache HTTP Server access log	regex.adapter, static.adapter	<base_dir>\Apache\access\v1.3.26
Apache HTTP Server error log	regex.adapter	<base_dir>\Apache\error\v1.3.26
Common Base Event XML log	regex.adapter	<base_dir>\XML\CommonBaseEvent\v1.0.1
ESS (Shark) Problem log	regex.adapter	<base_dir>\SAN\ESS(Shark)\ProblemLog
IBM DB2 Express diagnostic log	regex.adapter	<base_dir>\DB2\diag\tool
IBM DB2 Universal Database Cli Trace log	static.adapter	<base_dir>\DB2\cli_trace\V7.2,8.1
IBM DB2 Universal Database diagnostic log	regex.adapter(v8.1), static.adapter(v8.1), regex.adapter(v8.2)	<base_dir>\DB2\diag\v8.1, <base_dir>\DB2\diag\v8.2
IBM DB2 Universal Database JDBC trace log	static.adapter	<base_dir>\DB2\jcc\v8.1
IBM DB2 Universal Database SVC Dump on z/OS	static.adapter	<base_dir>\DB2\zOS\SVCDump
IBM DB2 Universal Database Trace log	static.adapter	<base_dir>\DB2\trace\V7.2,8.1
IBM HTTP Server access log	regex.adapter, static.adapter	<base_dir>\IHS\access\v1.3.19.3
IBM HTTP Server error log	regex.adapter, static.adapter	<base_dir>\IHS\error\v1.3.19.3
IBM WebSphere Application Server activity log	static.adapter	<base_dir>\WAS\activity\v4
IBM WebSphere Application Server activity log	regex.adapter, regex_example.adapter, regex_showlog.adapter, regex_showlog_example.adapter, static.adapter	<base_dir>\WAS\activity\v5
IBM WebSphere Application Server error log for z/OS	static.adapter	<base_dir>\WAS\zOSerror\v4
IBM WebSphere Application Server plugin log	regex.adapter, regex_example.adapter	<base_dir>\WAS\plugin\v4,5
IBM WebSphere Application Server trace log	static.adapter	<base_dir>\WAS\trace\v4, <base_dir>\WAS\trace\v5, <base_dir>\WAS\trace\v6
IBM WebSphere Commerce Server ecmsg, stdout, stderr log	regex.adapter, regex_example.adapter	<base_dir>\WCS\ecmsg\v5.4, <base_dir>\WCS\ecmsg\v5.5
IBM WebSphere Edge Server log	static.adapter	<base_dir>\WES\v1.0
IBM WebSphere InterChange Server log	static.adapter	<base_dir>\WICS\server\4.2.x
IBM WebSphere MQ error log	static.adapter	<base_dir>\WAS\MQ\error\v5.2
IBM WebSphere MQ FDC log	regex.adapter, regex_example.adapter	<base_dir>\WAS\MQ\FDC\v5.2,5.3
IBM WebSphere MQ for z/OS Job log	static.adapter	<base_dir>\WAS\MQ\zOS\v5.3
IBM WebSphere Portal Server appserver_err log	static.adapter	<base_dir>\WPS\appservererr\v4
IBM WebSphere Portal Server appserverout log	regex.adapter, regex_example.adapter	<base_dir>\WPS\appserverout\v4,5
IBM WebSphere Portal Server run-time information log	regex.adapter, regex_example.adapter	<base_dir>\WPS\runtimeinfo\V5.0
IBM WebSphere Portal Server systemerr log	static.adapter	<base_dir>\WPS\systemerr
Logging Utilities XML log	static.adapter	<base_dir>\XML\log\v1.0
Microsoft Windows Application log	regex.adapter, regex_example.adapter	<base_dir>\Windows\application
Microsoft Windows Security log	regex.adapter, regex_example.adapter	<base_dir>\Windows\security
Microsoft Windows System log	regex.adapter, regex_example.adapter	<base_dir>\Windows\system
Rational TestManager log	static.adapter	<base_dir>\Rational\TestManager\2003.06.00
RedHat syslog	regex.adapter, regex_example.adapter	<base_dir>\Linux\RedHat\syslog\v7.1,8.0
SAN File system log	static.adapter	<base_dir>\SAN\FS
SAN Volume Controller error log	regex.adapter	<base_dir>\SAN\VC\error
SAP system log	static.adapter, example.log	<base_dir>\SAP\system
Squadrons-S-HMC log	static.adapter	<base_dir>\SAN\Squadrons-S\HMC
SunOS syslog	regex.adapter, regex_example.adapter	<base_dir>\SunOS\syslog\v5.8
SunOS vold log	regex.adapter, regex_example.adapter	<base_dir>\SunOS\vold\v5.8
z/OS GTF Trace log	static.adapter	<base_dir>\zOS\GTFTrace
z/OS Job log	static.adapter	<base_dir>\zOS\joblog\v1.4
z/OS logrec	static.adapter	<base_dir>\zOS\logrec\v1.4
z/OS master trace log	static.adapter	<base_dir>\zOS\mtrace\v1.4
z/OS System log	static.adapter	<base_dir>\zOS\syslog\v1.4
z/OS System trace log	static.adapter	<base_dir>\zOS\systemtrace

<base_dir> is the directory where the Agent Controller is installed:

<Agent_Controller_DIR>\plugins\com.ibm.etools.logging.parser\config

Related concepts
Overview of the Log and Trace Analyzer

Related tasks
Creating a log parser for the Log and Trace Analyzer
 

Related references
Supported log file types

[Oct 27, 2007] UNIX System Administration Tools

Centralized Logging with syslog-ng and SEC - October 2007 (PDF, 309 KB)

Monitoring with Simple Event Correlator Page 1

Frequently, it is useful for security professionals, network administrators and end users alike to monitor the logs that various programs in the system write for specific events -- for instance, recurring login failures that might indicate a brute-force attack. Doing this manually would be a daunting, if not infeasible, task. A tool to automate log monitoring and event correlation can prove to be invaluable in sifting through continuously-generated logs.

The Simple Event Correlator (SEC) is a Perl script that implements an event correlator. You can use it to scan through log files of any type and pick out events that you want to report on. Tools like logwatch can do much the same thing, but what sets SEC apart is its ability to generate and store contexts. A context is an arbitrary set of things that describe a particular event. Since SEC is able to essentially remember (and even forget) these contexts, the level of noise generated is remarkably low, and even a large amount of input can be handled by a relatively small number of rules.

Looking for root login attempts

For instance, let's start with something basic, like looking for direct ssh root logins to a machine (security best practice is to completely not allow such logins, but let's not follow that for the sake of this example):

   Feb  1 11:54:48 192.168.22.1 sshd[20994]: [ID 800047 auth.info] Accepted publickey for root 
     from 192.168.15.3 port 33890 ssh2

Ok, so we can create an SEC configuration file (let's call it root.conf) that contains the following:

   type=Single
   ptype=RegExp
   pattern=(^.+\d+ \d+:\d+:\d+) (\d+\.\d+\.\d+\.\d+) sshd\[\d+\]: \[.+\] Accepted (.+) for root 
     from (\d+\.\d+\.\d+\.\d+)
   desc=direct ssh root login on $2 from $4 (via $3) @ $1
   action=add root-ssh_$2 $0; report root-ssh_$2 /usr/bin/mail -s "Direct root login on $2 from $4" 
     foo@example.com

This is an example of a rule in SEC. The first line describes the type, in this case, "Single" which tells SEC that we just want to deal with single instances of this event. The second line, ptype, tells SEC how we want to search for patterns. In this case we've chosen "RegExp" which says to use Perl's powerful regular expression engine. We can choose other types of matches, such as substring matches, tell the rule the utilize a Perl function or module, or tell it to look at the contents of a variable you can set.

The next line in this rule, the pattern in this case, is a big regular expression (regex) that would match on log entries where someone is logging in directly as root. We've grouped the timestamp, the IPs for both the source and destination and the method used to login for us to use later in an email. (If you're familiar with Perl, you can see SEC uses a similar regex grouping.)

The next line is the description of this rule. The final line is the action we intend to take. In this case, we add the entire log entry to a context called root-ssh_$2, where $2 will expand out to be the IP address of the machine being logged into. Finally, the rule will send mail out to foo@example.com with the contents of the context, which will include the matching log entry.

To run this thing we do:

   sec -detach -conf=root.conf -input=/var/log/messages

It will start up and begin looking for direct root logins in the background. We can tell SEC to watch multiple files (using Perl's glob() function):

   sec -detach -conf=root.conf -input=/var/log/incoming/logins*

Say this rule chugs away and sends you e-mail every morning at 5am when your cron job from some machine logs into another machine (as root!) to run backups. You don't want to get email every morning, so we can suppress those using the aptly named suppress rule type. To do that, we insert the following rule above our existing "look for root logins" rule:

   type=Suppress
   ptype=RegExp
   pattern=^.+\d+ \d+:\d+:\d+ \d+\.\d+\.\d+\.\d+ sshd\[\d+\]: \[.+\] Accepted .+ for root from 192.168.55.89

Then we can send SIGABRT to the sec process we started previously:

   kill -SIGABRT `ps ax | grep sec | grep root.conf | awk '{print $1}'`

which will tell that SEC process to reread its configuration file and continue.

Looking for brute force attacks

Now let's look at using SEC to watch for a brute force attack via ssh:

   # create the context on the initial triggering cluster of events
   type=SingleWithThreshold
   ptype=RegExp
   pattern=(^.+\d+ \d+:\d+:\d+) (\d+\.\d+\.\d+\.\d+) sshd\[\d+\]: \[.+\] Failed (.+)
     for (.*?) from (\d+\.\d+\.\d+\.\d+)
   desc=Possible brute force attack (ssh) user $4 on $2 from $5
   window=60
   thresh=5
   context=!SSH_BRUTE_FROM_$5
   action=create SSH_BRUTE_FROM_$5 60 (report SSH_BRUTE_FROM_$5 /usr/bin/mail -s 
     "ssh brute force attack on $2 from $5" foo@example.com); add SSH_BRUTE_FROM_$5
     5 failed ssh attempts within 60 seconds detected; add SSH_BRUTE_FROM_$5 $0
   # add subsequent events to the context
   type=Single
   ptype=RegExp
   pattern=(^.+\d+ \d+:\d+:\d+) (\d+\.\d+\.\d+\.\d+) sshd\[\d+\]: \[.+\] Failed (.+)
     for (.*?) from (\d+\.\d+\.\d+\.\d+)
   desc=Possible brute force attack (ssh) user $4 on $2 from $5
   context=SSH_BRUTE_FROM_$5
   action=add SSH_BRUTE_FROM_$5 "Additional event: $0"; set SSH_BRUTE_FROM_$5 30

This actually specifies two rules. The first is another rule type within SEC: SingleWithThreshold. It adds two more options to the Single rule we used above: window and thresh. Window is the times pan this rule should be looking over and thresh is the threshold for number of events that need to appear within the window to trigger the action in this rule. We're also using the context option, which tells this rule to trigger only if the context doesn't exist. The rule will trigger if it matches 5 failed login events within 60 seconds. The action line creates the context ($5 representing the IP of the attacker) which expires in 60 seconds. Upon expiration it sends out an e-mail with a description and the matching log entries. The second rule adds additional events to the context, and extends the context's lifetime by 30 seconds, as long as the context already exists; otherwise it does nothing.

The flexibility of SEC

The creation and handling of these contexts which are created dynamically lies at the heart of SEC's power and what set it apart from other "log watcher" style programs.

For example, a printer having a paper jam may issue a lot of incessant log messages until someone gets over to the printer to deal with it, and if a log watcher was set to send an e-mail every time it matched on the paper jam message, that's a lot of e-mail, most of which will get deleted. It would be worse if it was an e-mail to a pager. SEC can create a context stating, "I've seen a paper jam event and have already sent out a page," which the rule can check for in the future and suppress further e-mails if the context already exists.

Another good example of this is included with SEC, a simple horizontal portscan detector, which will trigger an alarm if 10 hosts have been scanned within 60 seconds, which has been traditionally a difficult thing to detect well.

John P. Rouillard has an extensive paper in which he demonstrates much of the power of SEC's contexts and we highly recommend reading it for much more of the gory details on log monitoring in general and SEC in particular.

In addition to contexts, SEC also includes some handy rule types beyond what we've shown so far (from the sec manual page):

SingleWithScript - match input event and depending on the exit value of an external script, execute an action.

SingleWithSuppress - match input event and execute an action immediately, but ignore following matching events for the next t seconds.

Pair - match input event, execute an action immediately, and ignore following matching events until some other input event arrives. On the arrival of the second event execute another action.

PairWithWindow - match input event and wait for t seconds for other input event to arrive. If that event is not observed within a given time window, execute an action. If the event arrives on time, execute another action.

SingleWith2Thresholds - count matching input events during t1 seconds and if a given threshold is exceeded, execute an action. Then start the counting of matching events again and if their number per t2 seconds drops below the second threshold, execute another action.

Calendar - execute an action at specific times.

The Calendar rule type, for instance, allows us to look for the absence of a particular event (e.g. a nightly backup being kicked off). Or, you can use it to create a particular contexts, like this example from the SEC man page:

   type=Calendar
   time=0 23 * * *
   desc=NightContext
   action=create %s 32400

This way, you can have your other rules check to see if this context is active and take different actions at night versus during the day.

More examples

Let's say we want to analyze Oracle database TNS-listener logs. Specifically, we want to find people logging into the database as one of the superuser accounts (SYSTEM, SYS, etc), which is a Bad Thing (tm):

   24-FEB-2005 00:26:52 * (CONNECT_DATA=(SID=fprd)
     (CID=(PROGRAM=O:\FPRD\FS750\bin\CLIENT\WINX86\PSTOOLS.EXE)(HOST=PSRPT3)(USER=report))) 
     * (ADDRESS=(PROTOCOL=tcp)(HOST=10.222.33.44)(PORT=4042)) * establish * fprd * 0

In my environment, we chop up the listener logs everyday and we run the following rules on each day's log:

   type=Single
   ptype=RegExp
   pattern=^(\d{2}-\p{IsAlpha}{3}-\d{4} \d{1,2}:\d{1,2}:\d{1,2}).*CID=\((.*)\)\(HOST=(.*)\)\
     (USER=(SYSTEM|INTERNAL|SYS).*HOST=(\d+.\d+.\d+.\d+).*
   desc=$4 login on $5 @ $1 from $3 ($2)
   action=add $4_login $0; create FOUND_VIOLATIONS
   #
   type=Single
   ptype=substr
   pattern=SEC_SHUTDOWN
   context=SEC_INTERNAL_EVENT && FOUND_VIOLATIONS
   desc=Write all contexts to stdout
   action=eval %o ( use Mail::Mailer; my $mailer = new Mail::Mailer; \
   $mailer->open({ From => "root\@syslog", \
				To => "admin\@example.com", \
				Subject => "SYSTEM Logins Found",}) or die "Can't open: $!\n";\
   while($context = each(%main::context_list)) { \
	print $mailer "Context name: $context\n"; \
	print $mailer '-' x 60, "\n"; \
	foreach $line (@{$main::context_list{$context}->{"Buffer"}}) { \
	print $mailer $line, "\n"; \
	} \
	print $mailer '=' x 60, "\n"; \
   } \
   $mailer->close();)

We run this configuration using the following Perl script that will pick out today's logfile to parse:

   #!/usr/bin/perl
   use strict;
   use Date::Manip;
   my $filedate = ParseDate("yesterday");
   my $fileprefix = UnixDate($filedate, "%Y-%m-%d");
   my $logdir = "/var/log/oracle-listener";
   opendir(LOGDIR, $logdir) or die "Cannot open $logdir! $!\n";
   my @todaysfiles = grep /$fileprefix/, readdir LOGDIR;
   if (scalar(@todaysfiles) > 1 ) { print "More than one file matches for today\n"; }
   closedir LOGDIR;
   foreach (@todaysfiles) {
    my $secout = `sec -conf=/home/tmurase/sec/oracle.conf -intevents 
     -cleantime=300 -input=$logdir/$_ -fromstart -notail`;
       print $secout, "\n";
   }

The Perl script invokes SEC with the -intevents flag which generates internal events that we can catch with SEC rules. In this case, we wish to catch that SEC will shutdown after it finishes parsing the file. Another option, -cleantime=300 gives us 5 minutes of grace time before the SEC process terminates.

Here we are using the first rule to simply add events to an automatically named context, much as we did above, and creating the context FOUND_VIOLATIONS as a flag for the next rule to evaluate. The second rule will check for the existence of FOUND_VIOLATIONS and the SEC_INTERNAL_EVENT context which is raised during the shutdown sequence, and we look for the SEC_SHUTDOWN event come across input using a simple substring pattern. (This technique of dumping out all contexts before shutdown is pulled from SEC FAQ 3.23.)

As you can see, the action line of the second rule has a lot going on. What we're doing is calling a small Perl script from within SEC that will generate an email with all of the database access violations the first rule collected.

Another thing that we often wish to monitor closely are the nightly backups. Namely, we want to make sure they've actually started, and that they actually managed to finish.

Say that a successful run looks like this in the logs:

   Apr  9 00:01:10  localhost /USR/SBIN/CRON[15882]: (root) CMD ( /root/bin/backup.pl / )

time passes...

   Apr  9 03:14:15  localhost backup.pl[15883]: finished successfully

An unsuccessful run would be, for our purposes, the absence of these two log entries. We can kick off a Calendar rule to set a context that indicates we are waiting for the first log entry to show up:

   type=Calendar
   time=55 23 * * *
   desc=Wait4Backup
   action=create %s 3600 shellcmd /usr/local/scripts/backup-failure.sh

Here we create the context "Wait4Backup" and set it to expire at 55 minutes after midnight, whereupon it executes a shell script that will presumably do some cleanup actions and notifications. The time parameter for the calendar rule uses a crontab-esque format with ranges and lists of numbers allowed.

We'll want to delete the Wait4Backup context and create a new context when the log entry for the start of the backup shows up:

   type=Single
   ptype=RegExp
   pattern=(^.+\d+ \d+:\d+:\d+) (.+?) .*CRON[(.+?)]: (root) CMD ( /root/bin/backup.pl / )
   context=Wait4Backup
   desc=Nightly backup for $1 starting at $0 pid $2
   action=delete Wait4Backup; create BackupRun_$1_$2 18000 shellcmd /usr/local/scripts/backup-failure.sh

With this rule, we've created a five-hour window in which the backup should finish before this new context expires and reports a failure.

Now for the last part: what to do when the backup finishes.

   type=Single
   ptype=RegExp
   pattern=(^.+\d+ \d+:\d+:\d+) (.+?) .*backup.pl[(.+?)]: finished successfully
   action=delete BackupRun_$1_$2; shellcmd /usr/local/scripts/backup-finished.sh

   type=Single
   ptype=RegExp
   pattern=(^.+\d+ \d+:\d+:\d+) (.+?) .*backup.pl[(.+?)]: error: (.*)
   action=delete BackupRun_$1_$2; shellcmd /usr/local/scripts/backup-failure.sh $4

The first rule takes care of what to do when it does finish successfully. The latter takes care of what happens when the backup script has errors. With these four rules, we have SEC covering the various possible states of our simple backup script, catching even the absence of the script starting on time.

Go forth and watch logs!

SEC is a powerful tool that builds on simple statements to handle the type of application monitoring and log monitoring that rivals commercial tools such as Tivoli or HP Openview. It does not have a GUI frontend or convenient reports, however, so a little more time must be spent on generating and formatting output of SEC's information. For those looking for more examples, a new rules collection has been started up at http://www.bleedingsnort.com/sec/.

Event correlation and data mining for event logs

The Simple Event Correlator (SEC) is a Perl script that implements an event correlator. You can use it to scan through log files of any type and pick out events that you want to report on. Tools like logwatch can do much the same thing, but what sets SEC apart is its ability to generate and store contexts. A context is an arbitrary set of things that describe a particular event. Since SEC is able to essentially remember (and even forget) these contexts, the level of noise generated is remarkably low, and even a large amount of input can be handled by a relatively small number of rules.

• Event logging and event log monitoring
• Event correlation – concept and existing solutions
• Simple Event Correlator (SEC)
• Frequent itemset mining for event logs
• Data clustering for event logs
• Discussion

[Feb 20, 2007] Microsoft Log Parser Toolkit  by Gabriele Giuseppini, Mark Burnett, Jeremy Faircloth, Dave Kleiman

Universal query tool to text-based data (log files, XML and CSV files), Event Logs, Registry and Active Directory

• Paperback: 437 pages
• Publisher: Syngress Publishing; 1 edition (February 1, 2005)
• Language: English
• ISBN-10: 1932266526
• ISBN-13: 978-1932266528

Dream Book on Dream Tool, October 3, 2006
 

Reviewer:

Joaquin Menchaca (San José, CA USA) - See all my reviews

This tool is amazing in that it supports a variety input and output formats including reading in syslog and outputting into databases are pretty Excel charts. The filtering uses an SQL syntax. The tool comes with a DLL that can be registered, so that scripters (VBScript, Perl, JScript, etc.) can access the power of this tool.

This book not only covers the tool (alternative being to scrape the network for complex incomprehensible snippets), but shows real world practical solutions with the tool, from analyzing web logs, system events, security and network scans, etc.

This tool is just heavensend for analysis and transforming of any data in a variety of formats. The book and tool go hand-in-hand, and I highly recommend incorporating this into your tool (and book) into your tool kit and/or scripting endeavors immediately.

[Feb 20, 2007] NIST Guide to Computer Security Log Management, September 2006 Adobe .pdf (1,909 KB)

[Dec 5, 2007] freshmeat.net Project details for log4sh

log4sh is a logging framework for shell scripts that works similar to the other wonderful logging products available from the Apache Software Foundataion (eg. log4j, log4perl). Although not as powerful as the others, it can make the task of adding advanced logging to shell scripts easier, and has much more power than just using simple "echo" commands throughout. In addition, it can be configured from a properties file so that scripts in a production environment do not need to be altered to change the amount of logging they produce.

Release focus: Major feature enhancements

Changes:
This release finally flushes out nearly all of the planned features for the 1.3 development series. It will hopefully be the last release in the 1.3 series before moving to the 1.4/1.5 series. In this release, the SyslogAppender is now fully functional, several bugs have been fixed, and there are additional unit tests to verify functionality. There is also a new Advanced Usage section in the documentation.

Author:
Kate Ward [contact developer]

[Dec 15, 2006]  Interview syslog-ng 2.0 developer Balázs Scheidler by: Robert Fekete

December 13, 2006 (Linux.com ) syslog-ng is an alternative system logging tool, a replacement for the standard Unix syslogd system-event logging application. Featuring reliable logging to remote servers via the TCP network protocol, availability on many platforms and architectures, and high-level message filtering capabilities, syslog-ng is part of several Linux distributions. We discussed the highlights of last month's version 2.0 release with the developer, Balázs Scheidler.

NewsForge: How and why did you start the project?

Balázs Scheidler: Back in 1998 the main Hungarian telecommunication company was looking for someone on a local Linux mailing list to port nsyslog to Linux. nsyslog -- developed by Darren Reed -- was at that time incomplete, somewhat buggy, and available only for BSD. While at university, I had been working for an ISP and got often annoyed with syslogd: it creates too many files, it is difficult to find and move the important information, and so on. Developing a better syslog application was a fitting task for me.

NF: Why is it called syslog-ng?

BS: syslog-ng 1.0 was largely based on nsyslog, but nsyslog did not have a real license. I wanted to release the port under GPL, but Darren permitted this only if I renamed the application.

NF: What kind of support is available for the users?

BS: There is a community FAQ and an active mailing list. If you are stuck with the compiling or the configuration, the mailing list is the best place to find help. My company, BalaBit IT Security, offers commercial support for those who need quick support.

NF: Documentation?

BS: The reference guide is mostly up-to-date, but I hope to improve it someday. I am sure there are several howtos floating around on the Internet.

NF: Who uses syslog-ng?

BS: Everyone who takes logging a bit more seriously. I know about people who use it on single workstations, and about companies that manage the centralized logging of several thousand devices with syslog-ng. We have support contracts even with Fortune 500 companies.

NF: What's new in version 2.0?

BS: 1.6 did not have any big problems, only smaller nuances. 2.0 was rewritten from scratch to create a better base for future development and to address small issues. For example, the data structures were optimized, greatly reducing the CPU usage. I have received feedback from a large log center that the new version uses 50% less CPU under the same load.

Every log message may include a timezone. syslog-ng can convert between different timestamps if needed.

It can read and forward logfiles. If an application logs into a file, syslog-ng can read this file and transfer the messages to a remote logcenter.

2.0 supports the IPv6 network protocol, and can also send and receive messages to multicast IP addresses.

It is also possible to include hostnames in the logs without having to use a domain name server. Using a DNS would seriously limit the processing speed in high-traffic environments and requires a network connection. Now you can create a file similar to /etc/hosts that syslog-ng uses to resolve the frequently used IP addresses to hostnames. That makes the logs much easier to read.

syslog-ng 2.0 uses active flow control to prevent message losses. This means that if the output side of syslog-ng is accepting messages slowly, then syslog-ng will wait a bit more between reading messages from the input side. That way the receiver is not flooded with messages it could not process on time, and no messages are lost.

NF: Is syslog-ng available only for Linux, or are other platforms also supported?

BS: It can be compiled for any type of Unix -- it runs on BSD, Solaris, HP-UX, AIX, and probably some others as well. Most bigger Linux distributions have syslog-ng packages: Debian, SUSE, Gentoo.... I think Gentoo installs it by default, replacing syslogd entirely.

NF: What other projects do you work on?

BS: syslog-ng is a hobby for me; that is why it took almost five years to finish version 2.0. My main project is Zorp, an application-level proxy firewall developed by my company. Recently I have been working on an appliance that can transparently proxy and audit the Secure Shell (SSH) protocol.

During development I stumble into many bugs and difficulties, so I have submitted patches to many places, such as glib and the tproxy kernel module.

NF: Are these projects also open source?

BS: No, these are commercial products, but the Zorp firewall does have a GPL version.

NF: Any plans for future syslog-ng features?

BS: I plan to support the syslog protocol that is being developed by IETF.

I would like to add disk-based buffering, so you could configure syslog-ng to log into a file if the network connection goes down, and transmit the messages from the file when the network becomes available again.

It would be also good to transfer the messages securely via TLS, and to have application-layer acknowledgments on the protocol level.

[Nov 11, 2006] Sisyphus 1.1 is now available at http://www.cs.sandia.gov/sisyphus
 

[Oct 23, 2006]  [PDF] Guide to Computer Security Log Management -- NIST publication.

[Oct 22, 2006] NIST 800-92.A Guide to Computer Security Log Management

Mostly fluff but all-in-all not bad for a government publication :-). Compare with Solaris™ Operating Environment Security

[Oct 22, 2006] Five mistakes of log analysis by Anton Chuvakin

October 21, 2004 (Computerworld) -- As the IT market grows, organizations are deploying more security solutions to guard against the ever-widening threat landscape. All those devices are known to generate copious amounts of audit records and alerts, and many organizations are setting up repeatable log collection and analysis processes.
However, when planning and implementing log collection and analysis infrastructure, the organizations often discover that they aren't realizing the full promise of such a system. This happens due to some common log-analysis mistakes.
This article covers the typical mistakes organizations make when analyzing audit logs and other security-related records produced by security infrastructure components.
No. 1: Not looking at the logs
Let's start with an obvious but critical one. While collecting and storing logs is important, it's only a means to an end -- knowing what 's going on in your environment and responding to it. Thus, once technology is in place and logs are collected, there needs to be a process of ongoing monitoring and review that hooks into actions and possible escalation.
It's worthwhile to note that some organizations take a half-step in the right direction: They review logs only after a major incident. This gives them the reactive benefit of log analysis but fails to realize the proactive one -- knowing when bad stuff is about to happen.
Looking at logs proactively helps organizations better realize the value of their security infrastructures. For example, many complain that their network intrusion-detection systems (NIDS) don't give them their money's worth. A big reason for that is that such systems often produce false alarms, which leads to decreased reliability of their output and an inability to act on it. Comprehensive correlation of NIDS logs with other records such as firewalls logs and server audit trails as well as vulnerability and network service information about the target allow companies to "make NIDS perform" and gain new detection capabilities.

Some organizations also have to look at log files and audit tracks due to regulatory pressure.
No. 2: Storing logs for too short a time
This makes the security team think they have all the logs needed for monitoring and investigation (while saving money on storage hardware) and then leading to the horrible realization after the incident that all logs are gone due to its retention policy. The incident is often discovered a long time after the crime or abuse has been committed.
If cost is critical, the solution is to split the retention into two parts: short-term online storage and long-term off-line storage.

... ... ...

[Mar 16, 2006] Slashdot Host Integrity Monitoring Using Osiris and Samhain

Looking at logging output in an enterprise environment can be very difficult. To make this really useful you need to aggregate information in a central repository, from all different servers/apps running on many machines. For true heavy duty log analysis you need to resort to tools such as SenSage [sensage.com]'s log storage/analysis tool.

Any other tool will choke on the volume of information you'll be chugging through in an enterprise environment, unless you pay for a multi-million-dollar Oracle deployment.

A Linux-based product used by Blue Cross/Blue Shield, Yahoo, Lehman Brothers, etc. For true enterprise security you need something like this.

[Dec 12, 2005] http://syslog-win32.sourceforge.net A very interesting open source implementation of syslog daemon for windows by Alexander Yaworsky. Quality coding.

This is another syslog for windows, it includes daemon and client. Features:

• RFC 3164-compliant;
• high performance;
• message forwarding;
• log rotation;
• completely open-source.

Unix Log Analysis Program

Logalizer is a Perl script that will analyze log files and email you the results.

One of the most important things a system administrator should do is to examine his system log files. However, most system administrators, myself included, have neither the time nor inclination to to do so. After setting up appropriate site filters (a bit of work), logalizer will automatically analyze log files and send the reports to you by email so that you can more easily detect attempted break-ins or problems with your systems.

This program is very easy to install. Customization to minimize the amount of noise is done by writing regular expressions to include and exclude parts of the log files that are interesting. Sample filter files are provided, but they will need to be tuned for your site.

[Apr 15, 2003] Bhavin Vaidya bhavin at cadence.com

• Previous message: sulog
• Next message: Summary: Sendmail
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Thanks to ...

devnull at adc.idt.com
Cody
Rob Rankin
Carsten Hey
Tom Yates

devnull at adc.idt.com and Tom Yates came with the upto the point solution for me.

I have added following line to my /etc/syslog.conf file and then touched /var/log/sulog
file.
----
auth.info                                               /var/log/sulog
#                     ^^^^ this white space is TABs, not spaces
-----
and then,

/etc/init.d/syslog restart

We wanted to do this as following line was creating too large a file and we also write
our logs to syslog server.
----
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
----
So, we changed our above line to ...      (which made a loose su log info)
------
*.notice;mail.none;authpriv.none;cron.none              /var/log/messages
-------

Sorry for the late summary posting as was sick at home since Friday after making the
changes on Thursday. Hope this helps all.

Regards, Bhavin



Bhavin Vaidya wrote:

> Hello,
>
> We would like to customise the sulog activity accross all (Solaris, HP-UX, AIX and
> Red Hat) OSs.
> Red Hat logs the login and sulogin activity under /var/log/messages file.
>
> We would like to customise the sulog written to it's own log file say /var/log/sulog,
> like it does on rest of the other OSs.  I have tried looking at /etc/pam.d/su but
> didn't find any config statement there.
>
> Will appriciate if any one let me know how I can achive this.
>
> BTW, I'm the Solaris, HP-UX and AIX tech and very new to Red Hat Linux.
>
> Thanks in advance and with regards,
> Bhavin

Re Solaris log files

I vote for  /var/log/cron.19990220, /var/log/ftp.19990220, /var/log/authlog.199902, etc.  Do you have so many logs online that they need more than one flat directory?  Then go one more level down, but not 4. Also, putting the timestamp in the filename makes restores and greps of the files less confusing.

But I think the problem is even bigger than that.

Some log files grow VERY RAPIDLY -- many megabytes per day. Some grow very slowly.  authlog comes to mind. It's best to keep individual log files under some certain size. 1MB is great.  10MB is OK.  50MB is getting kinda big.

But with these different growth rates, the tendency is to age some of them daily, others weekly, others yearly(!).

Then there's the annoying ones like wtmp that are binary.

And let's not forget that some processes need to be restarted after a logfile move, while others don't.

And some programs follow the paradigm "my logfile must exist and be writable by me or else I will silently log nothing".

I've always considered writing some tool that would allow you to manage and age all your log files from one config file.  Maybe the config file would be a table that lists the base logfile name, the interval at which it gets aged, the number of logs or amount of space to keep online before deleting them, etc.

Anybody know of any such program?  It might be too much work for too little gain.

The ultimate would be an ADAPTIVE process that keeps fewer old logs online if space is getting tight, etc.  Personally I think an adaptive news expire program would be nice, too.

I'll get right on these, as soon as I get this other stuff done for my boss... :-)

Todd Williams                     Manager, Computer and Communication Systems
MacNeal-Schwendler Corp. ("MSC"),  815 Colorado Blvd.,  Los Angeles, CA   90041
todd.williams@macsch.com          (323)259-4973          http://www.macsch.com/
geek  n. : a carnival performer often billed as a wild man whose act usu.
includes biting the head off a live chicken or snake  -Webster's New Collegiate

CERT/Understanding system log files on a Solaris

Solaris systems use the /var directory to store logs and other local files so that the operating system can support other directories being mounted as read only, sometimes from file servers elsewhere on the network. The /var directory is thus often on a partition that is local to the system.

All of the log files described below can be found in subdirectories under /var. There may be other application-specific log files that you will also need to inspect. However, it is beyond the scope of this implementation to describe all of the log files that you might want to inspect for your specific Solaris installation.

Because log files often provide the only indication of an intrusion, intruders often attempt to erase any evidence of their activities by removing or modifying the log files. For this reason, it is very important that your log files be adequately protected to make it as difficult as possible for intruders to change or remove then. See the practice "Managing logging and other data collection mechanisms" for more information on this topic.

[PDF] Solaris™ Operating Environment Security

Recommended Links

Open Directory - Computers: Software: Internet: Site Management ...

NIST 800-92. A Guide to Computer Security Log Management. Not very well written but still useful publication from NIST.

loganalysis - system log analysis, infrastructure, and auditing -- mail list

LogAnalysis.Org

ONLamp.com System Logging

Unix Log Analysis

Log Analysis Basics

Basic Steps in Forensic Analysis of Unix Systems

Aggregate Logs from Remote Sites

Central Loghost Mini-HOWTO Syslog related links

• [1] "File Formats - syslog.conf(4)." 22 January 1997. URL:
  http://www.bama.ua.edu/cgi-bin/man-cgi?syslog.conf+4 (23 December 2000).
• [2] Internet Engineering Task Force. "Security Issues in Network Event Logging (syslog)." 23 October 2000. URL: http://www.ietf.org/html.charters/syslog-charter.html   (23 December 2000).
• [3] Carnegie Mellon University. "Manage logging and other data collection mechanisms." 18 October 2000. URL: http://www.cert.org/security-improvement/practices/p092.html   (23 December 2000).
• [4] Orebaugh, Angela. "Securing Solaris." October 2, 2000. URL: http://www.sans.org/infosecFAQ/sec_solaris.htm (23 December 2000).
• [5] Bezroukov, Nikolai. "Softpanorama University Pages: Solaris Hardening and Security." URL: http://www.softpanorama.org/Security/sos.shtml (23 December 2000).
• [6] "Maintenance Commands - syslogd(1M)." 27 February 1997. URL: http://www.bama.ua.edu/cgi-bin/man-cgi?syslogd+1M (23 December 2000).
• [7] "Tucows Linux Man Pages syslogd.8." 12 October 1998. URL: http://howto.tucows.com/man/man8/syslogd.8.html (23 December 2000).
• [8] Carnegie Mellon University. "Configure firewall logging and alert mechanisms." CERT Security Improvement Modules. 2 August 1999. URL:
  http://www.cert.org/security-improvement/practices/p059.html (23 December 2000).
• [9] Adiscon. "EventReporter." URL:
  http://www.eventreporter.com/en/Product/Integrate-NT-Event-Log-into-Unix-Syslogd.asp (23 December 2000).
• [10] Kiwi Enterprises. "Kiwi’s Software." 23 December 2000. URL:
  http://www.kiwi-enterprises.com/products.htm (23 December 2000).
• [11] Cisco Systems. "Logging." Improving Security on Cisco Routers. URL: http://www.cisco.com/warp/public/707/21.html#logging (23 December 2000).
• [12] Cisco Systems. "Troubleshooting Commands." 19 December 2000. URL: http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/12cgcr/fun_r/frprt3/frtroubl.htm (23 December 2000).
• [13] Cisco Systems. "Managing the System." URL: http://www.cisco.com/univercd/cc/td/doc/product/software/ios111/mods/1mod/1cbook/1csysmgt.htm (23 December 2000).
• [14] Cisco Systems. "Logging and Counter Caveats." Characterizing and Tracing Packet Floods Using Cisco Routers. URL: http://www.cisco.com/warp/public/707/22.html#3f (23 December 2000).
• [15] Carnegie Mellon University. "Using newsyslog to rotate files containing logging messages on systems running Solaris 2.x." 2 March 2000. URL:
  http://www.cert.org/security-improvement/implementations/i041.09.html (23 December 2000).
• [16] Carnegie Mellon University. "Understanding system log files on a Solaris 2.x operating system." 2 March 2000. URL:
  http://www.cert.org/security-improvement/implementations/i041.12.html (23 December 2000).
• [17] Reed, Darren. "Nsyslogd." URL: http://coombs.anu.edu.au/~avalon/nsyslog.html (23 December 2000).
• [18] Scheidler, Balázs. "syslog-ng." 6 November 2000. URL: http://www.balabit.hu/products/syslog-ng (23 December 2000).
• [19] "Core FreeSoft." URL: http://www.core-sdi.com/english/freesoft.html (23 December 2000).
• [20] Conover, Matt. "Index of /files/SRS." URL: http://www.w00w00.org/files/SRS/ (23 December 2000).
• [21] Stokely, Celeste. "Celeste’s Tutorial On Solaris 2.x Modems & Terminals." 20 May 2000. URL: http://www.stokely.com/unix.serial.port.resources/modem.html (23 December 2000).
• [22] Spitzner, Lance. "Watching Your Logs." 19 July 2000. URL: http://www.enteract.com/~lspitz/swatch.html   (23 December 2000).
• [23] Rowland, Craig. "Psionic Logcheck Version 1.1.1." 10 May 2000. URL: http://www.psionic.com/abacus/logcheck/ (23 December 2000).

Reference

NIST Guide to Computer Security Log Management, September 2006 Adobe .pdf (1,909 KB)
 

Man Pages

• last(1)

• loginlog(4)

• login(1)

• logger(1)

• sulog(4)

• sysidtool(1M)

• syslog(3)

• syslogd(1)

• syslog.conf(4)

• sys-unconfig(1M)

• vold(1M)

Recommended Papers

CERT/Understanding system log files on a Solaris

Solaris systems use the /var directory to store logs and other local files so that the operating system can support other directories being mounted as read only, sometimes from file servers elsewhere on the network. The /var directory is thus often on a partition that is local to the system.

All of the log files described below can be found in subdirectories under /var. There may be other application-specific log files that you will also need to inspect. However, it is beyond the scope of this implementation to describe all of the log files that you might want to inspect for your specific Solaris installation.

Because log files often provide the only indication of an intrusion, intruders often attempt to erase any evidence of their activities by removing or modifying the log files. For this reason, it is very important that your log files be adequately protected to make it as difficult as possible for intruders to change or remove then. See the practice "Managing logging and other data collection mechanisms" for more information on this topic.

How to perform network-wide security event log monitoring

This white paper explains the need to monitor security event logs network-wide and how you can achieve this using GFI LANguard S.E.L.M.

Case Study: Implementing a Centralized Logging Facility

During the past several years I have found that there is an increase use in the number of Windows based systems appearing in our predominately all UNIX environment.

Remote Syslogging - A Primer

The syslog daemon is a very versatile tool that should never be overlooked under any circumstances.

Effective Logging & Use of the Kiwi Syslog Utility 04/15/2004

This paper will familiarize the reader with the basics of syslog as defined by RFC 3164, describe some variations of syslog as implemented by various network hardware vendors, provide an overview specifically of Kiwi's syslog utility and its' functionality, demonstrate basic configuration of the syslog utility, and finally provide examples of some advanced configurations of the syslog utility that will offer specific automated functionality tailored toward specific needs. Screenshots and other information will be presented in order to provide a clearer understanding of how to accomplish these tasks using the utility. After reading this document, a security professional should have a good understanding of how Kiwi's syslog utility could be implemented to provide an effective means of providing network information used for a wide range of tasks.

Log Analysis as an OLAP Application - A Cube to Rule Them All

This paper discusses a specific implementation of using OLAP technology on log analysis, in particular by using the Seagate Analysis OLAP client. The Seagate Analysis OLAP client, which is released free to registered users since February 2000, snuggly fits into this role for log analysis. This tool is free and powerful enough to be the first step for practitioners to explore OLAP�s utility. We will discuss how OLAP alleviates the log analysis problem, basic ideas on OLAP and related database design concepts. There is also an iteration through a mini project that uses the Seagate Analysis on Windows NT Event Logs.

Detecting Intrusions with your Firewall Log and OsHids

In this article we are going to talk about one of the basics, but powerful, methods of Intrusion Detection: Firewall's Log analysis. Although a firewall generates a lot of log, being difficult to analyze it, you can use the OsHids tool to monitor your logs (generating an easy to view log in html with an PHP interface) and help you visualize any attempt to bypass your firewall policy.

How to detect hackers on your web server

A discussion of the methods used by hackers to attack IIS web servers, and how you can use event log monitoring on your web server to be alerted to successful attacks immediately.

Practical Implementation of Syslog in Mixed Windows Environments for Secure Centralized Audit Logging

The Event log service is by design a distributed system, and there are no native Windows tools available to facilitate centralization of logging functions. In addition, the failure to conform to any external logging format standard makes it impossible to interoperate with the logging functions of other operating systems or network devices. The Windows Event viewer application offers only basic functionality and is inadequate for monitoring the audit log files of any medium to large size network. In this paper, I survey some of the options available to access the Windows Event log and demonstrate how to implement a versatile centralized remote logging solution using a commercially available Win32 implementation of the Syslog protocol.

Setting up a Linux Log Server to enhance System Security

If a break-in occurs and you want to track the cracker down, the system administrator will first check the log files for evidence of a break-in, so she must be 100% SURE that the log files are valid and haven't been tampered with.

Complete Reference Guide to Creating a Remote Log Server

A remote log server is nothing more then a system preconfigured at install-time to provide hard drive space for other systems to log to. This system must be completely secured and locked down. No unencrypted remote access should be allowed, all RPC Daemons and other misc. services should be turned off as well. The only data allowed to the machine should be UDP/Port 514. We will be walking you through a step-by-step process that details how to configure, install, and deploy a remote log server. Utilizing some of the most renowned security experts across the globe for input, I've compiled a comprehensive, and easy to understand guide on ensuring this to be a successful launch.

Case Study: Implementing a Centralized Logging Facility

During the past several years I have found that there is an increase use in the number of Windows based systems appearing in our predominately all UNIX environment. This has been a downfall especially since UNIX and Windows systems are so different with regards to logging facilities, UNIX with its syslog facilities and Windows Eventlog; therefore I needed to find a way so that our Windows and UNIX systems could utilize a more robust logging facility. With budget concerns, being a major contributing factor, I needed to find a solution that was inexpensive. Therefore all the items that I chose to implement at this time are freeware and applications that already exist in our environment. The Windows systems needed to be configured so that they would audit the proper events and then forward that onto a UNIX system for storage and eventually analysis. Next, the UNIX systems needed a bit of tuning to get syslog to log the correct items. Finally, the logs needed to be retained and rotated.

Event Logs: Defining Their Purpose in Today's Network Security Environment 03/28/2004

The purpose of this research topic is to identify the purpose of the event log in today's network security environment. This topic came about to solve an every day business problem. Simply, there is not enough time in the day to perform all security analyst tasks and adequately monitor all network security devices. However, expectations were that monitoring all components of network security is essential. It's the way things had been done and anything short of that may render a device or component of network security as "insecure". It was clear that something must be done.

Using ISA Server Logs to Interpret Network Traffic 04/08/2004

Firewalls are necessary for a defense-in-depth strategy. Microsoft entered the firewall market with Internet Security and Acceleration Server (ISA Server). ISA Server was a follow-on release of Microsoft Proxy Server and part of the .Net Family. As with most Microsoft products, logging capabilities are included. ISA Server contains detailed security and access logs. You can install ISA Server in three different modes: firewall mode, web caching mode, or integrated mode. In firewall mode, you can secure communication between an internal network and the Internet using rules. You can publish internal servers so that their services are available to Internet users. In web caching mode, you can decrease network bandwidth with ISA Server storing commonly accessed objects locally. You can route web requests from the Internet to an internal Web Server. In integrated mode, all of these features are available.

Custom System Statistics Monitoring

As a systems administrator, knowing the status of the systems in your network is a must. This can be quite a challenge when you are dealing with tens, hundreds or thousands of different systems. A multitude of options are available for obtaining systems statistics, many of them freely available. For my purposes, I had a number of different requirements that most packages did not meet. In the end, I opted to build my own. This article explains the design of my solution, as well as its installation and configuration for use in your own environment.

One of my first requirements for systems statistics monitoring was any solution has to include a SQL backend. At the very least, it needs to have some way to export its data easily to allow it to be loaded into a SQL database. I wanted all data to be stored in a set of tables, but for simplicity and speed I decided to store only a single set of current data in the primary data table. Historical data then would be archived to an archive table, allowing reporting tools to use much simpler queries to obtain either current or historical data and to increase the speed and efficiency of queries against the current data table.

I debated whether to use a central data collection system or to allow each client to collect its own data. In the end, I decided to have each client collect the data and update the database. This made it much simpler to do asynchronous status updates without using a threaded server using a pull model. The agent needs to run on each host to collect the data, so why not have it make the database updates too?

The data gathering was a much more difficult problem to tackle due to portability issues. After spending some time implementing data-gathering functions, I came upon libstatgrab, a cross-platform systems statistics gathering library written in C. Many of the statistics I needed were handled by libstatgrab, so the first version uses libstatgrab exclusively to collect the data.

Figure 1 contains a simple graph showing the data flow from the OS level to ssclient via libstatgrab and then from ssclient to the database via libmysqlclient.

Linux.com: SysAdmin to SysAdmin: Using Jabber as a Log Monitor(Jun 10, 2004)
Linux.com: SysAdmin to SysAdmin: Perl's Tie::File Module(Jun 03, 2004)
Linux.com: SysAdmin to SysAdmin: Linux is the Unix Reference Implementation(May 13, 2004)
Linux.com: SysAdmin to SysAdmin: Using Ximian Tools for System Maintenance(May 07, 2004)

Epylog is a log notifier and parser that periodically tails system logs on Unix systems, parses the output in order to present it in an easily readable format (parsing modules currently exist only for Linux), and mails the final report to the administrator. It can run daily or hourly. Epylog is written specifically for large clusters where many systems log to a single loghost using syslog or syslog-ng.

Author:
Konstantin Ryabitsev [contact developer]

CCZE is a robust and modular log colorizer with plugins for apm, exim, fetchmail, httpd, postfix, procmail, squid, syslog, ulogd, vsftpd, xferlog, and more.

Author:
Gergely Nagy [contact developer]

Homepage:
http://bonehunter.rulez.org/CCZE.phtml
Changelog:
ftp://bonehunter.rulez.org/pub/ccze/stable/ChangeLog
Debian package:
http://packages.debian.org/unstable/utils/ccze.html
 

Complete System Resource Monitor and Task Organizor consists of a daemon, a client, and a WWW-server (within the daemon). The daemon can run tasks and handle client-daemon and WWW-daemon requests. Clients can receive statistics and issue commands. Monitoring and maintenance is performed as prescheduled tasks. The system includes a process monitor and handler, a basic WWW-client, and a console client.

http://csrtd.sourceforge.net/csrtd/csrt-daemon-latest.tar.gz

About:
FSHeal aims to be a general filesystem tool that can scan and report vital "defective" information about the filesystem like broken symlinks, forgotten backup files, and left-over object files, but also source files, documentation files, user documents, and so on. It will scan the filesystem without modifying anything and reporting all the data to a logfile specified by the user which can then be reviewed and actions taken accordingly.

http://sourceforge.net/projects/fsheal/

http://ganglia.sourceforge.net/

Etc

/var/log/utmp; /var/log/utmpx

These logs keep track of users currently logged into the system. Using the who command, check the users logged in at the current time:

Look for user logins that are unexpected (e.g., for staff on vacation), occur at unusual times during the day, or originate from unusual locations.

/var/log/wtmp; /var/log/wtmpx

These logs keep track of logins and logouts. Using the last command, do the following:

Look for user logins occurring at unusual times.

Look for user logins originating from unusual places (locations, addresses, and devices).

Look for unusual reboots of the syst

reboot system boot Sun Mar 23 05:36

/var/log/syslog

By default, the syslog file will contain only messages from mail (as defined in the /etc/syslog.conf file). Look for anything that looks unusual.

/var/adm/messages

This log records system console output and syslog messages. Look for unexpected system halts.

Look for unexpected system reboots.

Look for failed su and login commands.

Mar 30 09:14:00 <hostname> login: 4 LOGIN FAILURES ON 0, <userid>

Mar 31 12:37:43 <hostname> su: 'su root' failed for <userid> on /dev/pts/??

Look for unexpected successful su commands.

Mar 28 14:31:11 <hostname> su: 'su root' succeeded for <userid> on /dev/console

/var/adm/pacct

This log records the commands run by all users. Process accounting must be turned on before this file is generated. You may want to use the lastcomm command to audit commands run by a specific user during a specified time period.

compile <userid> ttyp1 0.35 secs Mon Mar 31 12:59  

/var/adm/aculog

This log keeps track of dial-out modems. Look for records of dialing out that conflict with your policy for use of dial-out modems. Also look for unauthorized use of the dial-out modems

Other log files   Solaris  includes Basic Security Module (BSM), but it is not turned on by default. If you have configured this service, review all the files and reports associated with BSM for the various kinds of entries that have been described in the practice "Inspect your system and network logs."

 If your site has large networks of systems with many log files to inspect, consider using tools that collect and collate log file information. As you learn what is normal and abnormal for your site, integrate that knowledge into your specific procedures for inspecting log files.

Last modified: February 28, 2008