Perl-based Event Correlation

SEC has been primarily tested on Linux and Solaris, but since it is written in Perl and does not use any platform-dependent subroutines, it should work on most operating systems.

Key features:

• input: regular files, named pipes, and standard input, and can thus be employed as an event correlator for any application that is able to write its output events to a file stream.
• rules, each rule specifying an event matching condition, an action list, and optionally a Boolean expression whose truth value decides whether the rule can be applied at a given moment. Regular expressions, Perl subroutines, etc. are used for defining event matching conditions. Event correlation rule types:
  • Single - match input event and execute an action list.
  • SingleWithScript - match input event and execute an action list, if an external script or program returns a certain exit value.
  • SingleWithSuppress - match input event and execute an action list, but ignore the following matching events for the next t seconds.
  • Pair - match input event, execute an action list, and ignore the following matching events until some other input event arrives. On the arrival of the second event execute another action list.
  • PairWithWindow - match input event and wait for t seconds for other input event to arrive. If that event is not observed within the given time window, execute an action list. If the event arrives on time, execute another action list.
  • SingleWithThreshold - count matching input events during t seconds and if a given threshold is exceeded, execute an action list and ignore the following matching events during the remaining time window. The window of t seconds is sliding.
  • SingleWith2Thresholds - count matching input events during t1 seconds and if a given threshold is exceeded, execute an action list. Then start the counting of matching events again and if their number per t2 seconds drops below the second threshold, execute another action list. Both event correlation windows are sliding.
  • Suppress - suppress matching input event (used to keep the event from being matched by later rules).
  • Calendar - execute an action list at specific times.

    Rules allow not only shell commands to be executed as actions, but they can also:

    • create and delete contexts that decide whether a particular rule can be applied at a given moment,
    • associate events with a context and report collected events at a later time (similar feature is supported by logsurfer),
    • generate new events that will be input for other rules,
    • reset correlation operations that have been started by other rules,
    • spawn external event, fault, or knowledge analysis modules,
    • etc.
    This makes it possible to combine several rules and form more complex event correlation schemes.
• output events can be produced by executing user-specified shell scripts or programs by writing messages to pipes or files, and by various other means. The applications SEC has been used or integrated with include HP OpenView NNM and Operations, CiscoWorks, BMC Patrol, Nagios, SNMPTT, Snort IDS, Prelude IDS, etc.

Old News ;-)

[Jan 30, 2006] "Simple Event Correlator for real-time security log monitoring" - a paper about SEC that was published in Hakin9 Magazine

Simple Event Correlation installation and configuration

Risto Vaarandi Publications

• Risto Vaarandi. Simple Event Correlator for real-time security log monitoring. Hakin9 Magazine 1/2006 (6), pp. 28-39, 2006
• Risto Vaarandi. Tools and Techniques for Event Log Analysis. PhD Thesis, Tallinn University of Technology, 2005
• Risto Vaarandi. A Breadth-First Algorithm for Mining Frequent Patterns from Event Logs. Proceedings of the 2004 IFIP International Conference on Intelligence in Communication Systems, LNCS Vol. 3283, © Springer-Verlag, pp. 293-308, 2004
• Risto Vaarandi. A Data Clustering Algorithm for Mining Patterns From Event Logs. Proceedings of the 2003 IEEE Workshop on IP Operations and Management, pp. 119-126, 2003
• Risto Vaarandi. A Clustering Algorithm for Logfile Data Sets. Technical Report, University of Kuopio, 2003
• Risto Vaarandi. SEC - a Lightweight Event Correlation Tool. Proceedings of the 2002 IEEE Workshop on IP Operations and Management, pp. 111-115, 2002
• Risto Vaarandi. Platform Independent Event Correlation Tool for Network Management. Proceedings of the 2002 IEEE/IFIP Network Operations and Management Symposium, pp. 907-910, 2002
• Risto Vaarandi. Platform Independent Tool for Local Event Correlation. Acta Cybernetica 15(4), pp. 705-723, 2002
• Risto Vaarandi. Platform Independent Tool for Local Event Correlation. Technical Report, University of Szeged, 2001
• Risto Vaarandi. Hajusad failisüsteemid ("Distributed File Systems"). MSc Thesis, University of Tartu, 1996 (see also the references page)