Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

TEC Event Adapters

The TEC event adapters are processes, which are written to convert other sources of event information to TEC events and send it to the TEC server. An event adapter monitors the resource of interest and detects when a reportable event has occurred. Next, it formats that information into a form understood by the event server. Finally, it sends the formatted event instances to the event server. Event adapters do not have a graphical representation on the Tivoli desktop. An event adapter usually resides on the machine where the resource it is monitoring resides. Sometimes it is monitoring a device incapable of running an adapter such as a printer or router. In that case, SNMP or other  protocol is used.

Event adapter does not necessary need to reside on a Tivoli managed node. All that is required is a TCP/IP connection. The Tivoli adapters are documented in the IBM Tivoli Enterprise Console Adapters Guide Version 3.8.

Tivoli ships several event adapters with the TEC. The most important are:

• logadapter
• snmp_adapter
• postmsg adapter utilities.

Third parties also write event adapters for their products. Using the Event Integration Facility (EIF, see Tivoli Event Integration Facility User's Guide - Contents), you can write a custom event adapter.

Although not an adapter, the TEC gateway is similar in that it is software that uses the TME interface of Tivoli Event Integration Facility to communicate with the event server. Like an adapter, it can be configured with a configuration file, and the configuration file can be distributed with an adapter configuration profile (ACP) entry using the Adapter Configuration Facility (ACF).

The event adapters consist of three major components which correspond to three main functions:

• Detecting event instances. The detection of event instances depends upon the source being monitored. There are two types of adapters:
  • Sensors which monitor passively, receiving information directly from a monitored source.
  • Probes which are active, either periodically checking a file for updates or running a command (process) or application at configurable lengths to gather information.
     
• Converting the event instance into TEC format (formatiing of the event).

  The interface between event adapters and the event server is the event definition. Both the sender (event adapter) and receiver (event server) must agree on which events, and which attributes of those events, are defined. If the event adapter sends an event instance not in conformance with the definition, the event server will leave it in the reception log marked as PARSING_FAILED and not evaluate it against rules.

  Event definitions are written in the BAROC (Basic Recorder of Objects in C) language. BAROC will be presented in detail later. In essence, it defines the kinds of events, the attributes of the events and the data types for those attributes. A BAROC definition is similar to a struct in the C programming language or a record in the PASCAL programming language.
   

• Sending them to TEC.

One layer of the adapter interfaces with the source of the events detecting when events of interest have occurred.

This may be a separate program or part of the event adapter. Its purpose is to capture events so they can be formatted and sent to the TEC server.

The interface between event adapters and the event server is the event definition. Both the sender (event adapter) and receiver (event server) must agree on which events, and which attributes of those events, are defined. If the event adapter sends an event instance not in conformance with the definition, the event server will leave it in the reception log marked as PARSING_FAILED and not evaluate it against rules.

Event definitions are written in the BAROC (Basic Recorder of Objects in C) language. Event information is formatted as a set of attributes. Each attribute is predefined and contains a name and value. Adapters separate information into event classes, format this information into attributes, and send this information to the event server. The event server then processes this information.

Event classes are a classification of events, and should not be confused with the term "classes" in OO programming. There are purely data structures and does not contain any executable components (methods). Event classes can be divided into subclasses to facilitate a further breakdown of information so that more detailed rules can be applied to the information. In essence, event classes are an agreement between the adapter and the event server about what information the adapter will send to the event server for a given class.

Suppose that an adapter is monitoring the event type Su_Failure it need to exract from the source the atteribles that are present in the BAROC definition for Su_Failure. In this case that includes but not limited to attributes from_user, to_user, date, hostname.  Those values should be extracted from the record of syslogd related to the event.

If the adapter creates an instance which doesn't match the definition, the instance will not be processed when it is sent. Incorrectly structured event will be simply dropped with no warning.

On the event server side, the BAROC file must be formally imported into the rule base and compiled. Syntax errors in the BAROC definition file are detected on the server side.

The event adapter relies on a configuration file to specify parameters such as which TEC server to use, which port to use, whether to hold a communication link or to renegotiate when an event is ready to be sent, the maximum size of an event instance, which file to use to store events if the event server is down, and more. The advantage of a configuration file is that parameters can be changed and the event adapter can simply be stopped and started rather than recompiled. Another advantage is that the same adapter code can execute in more than one place customized by different configuration files.

The underlying communication mechanism is TCP/IP.

The event adapter relies on several condiguration files. Among them:

• The BAROC definition file defines what events are known, the attributes of those events, and the data type of each attribute. The BAROC file must be imported into the rule base of the event server. It need not reside on the adapter side, but the event instances generated there must be in conformance with the definitions. The conventional extension for a BAROC file is .baroc.
• The configuration file defines parameters regarding the communication between the event adapter and the event server. Each TEC gateway can also have a configuration file. A global configuration file is shared by more than one adapter while an adapter specific configuration file is specialized. The conventional extension for a configuration file is .conf.
• Error file. The file which configures the error logging and tracing level. It also specifies the name of file into which errors should be written. The conventional extension for an error file is .err.
• The rules file contains the rules which are executed by the event server. The rules specify what actions to take in response to incoming events. While not required for the operation of the event adapter, rules files are often supplied by the author of the adapter as they are closely related to the events being collected by the adapter -- here's what to look for and here's what to do if you find it. Additional rules can be written to respond to events already being collected. The conventional extension for a rules file is .rls.

The configuration file presents configuration keywords followed by filters. Keywords are presented in pairs--parameter keyword and value for that parameter. For example, PollInterval=30. Filters typically follow all of the keywords. The comment character for configuration files is #.

Some keywords are meaningful for all adapters while others are event adapter specific (The configuration keywords for each adapter are documented in the TEC Adapters Guide).

PreFilterMode=OUT
#
ServerLocation=@EventServer
EventMaxSize=4096 BufEvtPath=C:\WINNT\system32\drivers\etc\Tivoli/tec/tecad_nt.cache
PollInterval=30
Pre37Server=no
SpaceReplacement=TRUE
LanguageID=ENGLISH
WINEVENTLOGS=Application,System,Security
#  

The ServerLocation tells the event adapter to which server to send events. A ServerLocation can be specified in each adapter's configuration file and in the gateway's configuration file. If the local event server is not available, then the entry in the tec gateway configuration file will be checked

The following value means the TEC event server is in the local TMR.

  ServerLocation=@EventServer 

Another TMR can be specified by using a fully qualified name.

  ServerLocation=@EventServer#bebop-region 

Event adapters may reside on non-Tivoli managed machines. In that case, use the name or IP address of the machine on which the event server executes.

  ServerLocation=146.78.29.11 

It is possible to configure alternative servers for use in case the primary is unavailable. Up to eight server names can be provided on the ServerLocation line in the gateway's configuration file (not in an adapter's configuration file). If the first server listed cannot make or maintain a connection with the event adapter, the gateway will try sending event instances to the second. If that is unsuccessful, the adapter will try the third and so on. The adapter will automatically revert to the primary server after sending a fixed number of event instances to the alternate. At any given time, events are only being sent to one server.

This optional keyword specifies the file in which events are buffered if they can't be delivered. When the server becomes available again, the buffered events are automatically sent.

BUFEVTPATH specifies the full path name of the adapter cache file. On endpoint adapters, the BufEvtPath keyword uses the $TIVOLIHOME variable to resolve file location and drive letter differences over different environments by using a path relative to the endpoint installation. The ACF defines $TIVOLIHOME on each endpoint; you cannot change its value.

Filtermode. A filter can be specified in a configuration file. By default, if an event matches the filter criteria, it will not be sent. This default behavior is expressed as FilterMode=OUT. If the majority of events generated by an adapter are uninteresting, then it is possible to reverse the filtering so that events which match the filter criteria are the only ones sent. This is expressed as FilterMode=IN.

If you set FilterMode=IN, make sure you have one or more Filter or FilterCache statements defined as well, or you will not receive any events from the adapter.

If TestMode=Yes, events are not sent to the event server specified in the ServerLocation. They are written into a file. The file is given the name specified as the ServerLocation parameter. That file is located in the current directory if an absolute path isn't given. This may vary depending on how the adapter was started.

Event filtering is very important. Filters in the configuration files provide a way of specifying which events not to generate. This minimizes network traffic, lowers demand on the reception engine, minimizes computation cost at the event server, and allows more important event instances to use cache space.

The authors of the event adapters made some educated judgements concerning what events might be of interest. A given installation may not care about all of the kinds of events that an adapter is capable of generating. The event filter allows each site to customize which of the available events they are interested in receiving.

The general form of a filter is:

Filter:Class= class_name ;attribute= value ;...;attribute= value

A filter must be no longer than 512 characters and fit on one line. Filter statements appear in the configuration file following the keyword value pairs. If an event instance matches the criteria specified, for instance, is of the named class with attribute values as specified, it will not be sent to the server. (As noted above, the FilterMode keyword can be used to reverse the sense of filtering so events matching the criteria are sent and the default is to suppress events.) Every event class to be suppressed must be specified in a filter which means regular expressions or superclasses cannot be used to create groupings of classes.

An example of a particular filter is:

Filter:Class=Su_Success; hostname=bebop

This would suppress event instances reporting Su_Success on the machine bebop. It could be further qualified by additional attribute values or the attribute values could be omitted entirely.

If BufferEvents is set to yes and the event adapter is unsuccessful in sending event instances to the server, it will buffer them locally or at the gateway. All events will be stored if BufferEvents=yes and FilterMode=OUT. Storing all these events could cause the buffer file to grow large. There is another kind of optional filter which suppresses the buffering of event instances in such a case called FilterCache. Together with FilterMode, it is used to determine which are stored in the cache. The buffer filter is specified in the same way as the filter mentioned above. If a gateway is being used along with an adapter on an endpoint, they must both specify the same FilterCache, if it is used.

You can also use Tcl regular expressions in a filter statement. This example of a filter statement using a regular expression matches all events with a class name that begins with TEC_:

  Filter:Class=re:'TEC_.*'

(In Tcl regular expressions ".*" matches anything including the empty string.)

The following example shows a filter statement with a more narrow range. This filter statement matches all events with a class name that begins with TEC_ and has a severity of critical:

 Filter:Class=re:'TEC_.*';severity=CRITICAL

It is important to note that Tivoli uses an exception to the Tcl regular expression syntax. The back slash character (\) in Tivoli means the literal character that follows, not some special character such as a tab. For example, "\t" means the tab character in Tcl but means the letter t in Tivoli.This is useful for taking away the special meaning of some characters such as wildcard specifiers.

For more information on Tcl regular expressions, see a Tcl user's guide.

Tivoli Adapter Configuration Facility (ACF)

The Tivoli Adapter Configuration Facility (ACF) makes it easier to configure multiple adapters in an enterprise. It presents a graphical user interface to the adapter configuration file which can then be distributed via subscription. This is easier than having to edit the configuration file on every machine hosting an adapter. Adapter configuration profiles (ACP) works like other profiles in the Tivoli product set.

State correlation monitors information from incoming events and associates this information with user-defined patterns. State correlation analyzes the incoming events for user-defined states to suppress duplicate events, identify event thresholds, and collect or group similar events.

Note: State correlation does not work with managed node adapters built with the C API. Instead, use an endpoint adapter or a non-TME adapter.

You can use state correlation with your custom adapters or TEC gateways. When using state correlation with a gateway, you can enhance your filtering by correlating events across multiple adapters. For more information
about TEC gateways, see the TEC Adapters Guide.

State correlation helps minimize event traffic by identifying similar events and consolidating their information into summary events where possible. Those events that cannot be included in a summary are single events. The event server then receives these summary events, as well as the single events. Accordingly, the event server does not receive the redundant, individual events for each summary.

Correlation is achieved with state-based and stateless rules. You specify these rules by using XML syntax, defined by the supplied DTD file, tecsce.dtd. The rules also have non-XML elements that define the associated rule predicates. The location of the default XML file is $BINDIR/TME/TEC/default_sm/tecroot.xml. This same directory also contains other samples of state correlation XML files. These files are only found on the system where you installed Tivoli Event Integration Facility or the Adapter Configuration Facility. They are not distributed with the default profile to other systems. For more information about the additional sample files, see the readme.txt file in the same location.

Edit the winnt\system32\drivers\etc\Tivoli\tec\tec_gateway.conf file to set UseStateCorrelation=YES. By default, UseStateCorrelation is set to NO.

You can also set LogLevel=ALL and LogFileName=c:/tmp/gateway.log (or some other location), so that any problems parsing the XML file will be logged to the given log file.

Identify the XML configuration file by setting StateCorrelationConfigURL=file:C:/tmp/bvt.xml (or other XML configuration file).

The match rule will discard any Bvt_general events with a message of discard.

The Rule ID is testcases.match.discard.

The eventType or event class is Bvt_general.

The match tag tells what predicates are to be evaluated to discern whether a match is made. In this example, we observe that if a msg slot is equal to the string "discard," then we have a match and the trigger actions below will be executed.

The action Discard, which discards the event, will be executed

This drops all duplicates of the event class Bvt_general with a msg slot of duplicate.

The duplicate tag specifies the time interval in milliseconds. In this example, the time interval is 30000 ms. This is the valid time interval for this duplicate rule.

Only the first such event during the time interval will be forwarded

• Tivoli Plus Partners are third-party vendors which integrate their products with the Tivoli software, including TEC. Part of this integration is to send events from their applications to TEC.
• IBM Tivoli Monitoring can be extended to send anything it detects to the TEC.
• There is a command included with the TEC distribution to send event instances to the Tivoli Enterprise Console, and this can be invoked as a single command or from within a script.
• The NT and logfile adapters can be reconfigured to read from any logfile, so if an application detects events and writes information into a file, the adapter will format an event instance and send it to the TEC.
• Finally, custom adapters can be written.

Part of the integration means that the partners define events and rules so their product can be monitored with TEC. Integration usually also means Distributed Monitoring monitors are defined, an icon for the Partner appears on the Tivoli desktop, and the software is distributed with Tivoli Software Distribution. Other areas of integration are defined depending on the product.

Old News ;-)

Configuring and Troubleshooting the SNMP Adapter for TEC 3.9 fixpack 6
 

Tivoli Enterprise Console UNIX Logfile Adapter

[Aug 20, 2007] OpenEsm - What is OpenESM

Zabbix-based monitoring solution. Has Tivoli event adapter written in Perl: OpenESM Universal Tivoli Enterprise Console Event Adapter

Right now, OpenESM has OpenESM for Monitoring v1.3. This release of the software is a combination of Zabbix, Apache, Simple Event Correlation and MySQL. Out of the box, we provide monitoring - warehousing of monitoring data - SLA reporting - correlation and notification. We offer the source code, but we also have a VMWARE based appliance.

Recommended Links

IBM Tivoli Enterprise Console Adapters Guide version 3.8

Review Questions

1. What are the three main functions of an event adapter?
   
   The main functions of event adapters are to detect, format, and send event instances to the event server.
    
2. Besides event adapters, name three other possible sources of events.
   
   Any three of -- Tivoli Plus Partners, Tivoli Monitoring, the command line ( wpostemsg and postemsg ) and the reconfigured logfile.
    
3. What is the importance of the BAROC file?
   
   The BAROC file is the interface between the event adapter and the event server. It defines the kind of events possible and their attributes.

Last modified: June 02, 2008