Softpanorama
(slightly skeptical)
Open Source Software Educational Society
May the source be with you, but remember the KISS principle ;-)
Softpanorama Search
|
Softpanorama |
May the source be with you, but remember the KISS principle ;-)
Softpanorama Search
|
John Morrell
InfoManagement Direct, November 3, 2006Complex event processing (CEP) involves the continuous processing and analysis of high-volume, high-speed data streams from inside and outside an organization to detect business-critical issues as they happen. In comparison to traditional intelligence processes, which provide delayed analysis, CEP software processes data streams and detects business events in real-time. Some examples of CEP applications are:
- Real-time financial market data analysis and enrichment,
- Financial trade auditing and compliance,
- IT security event correlation,
- Asset management and tracking using RFID, and
- Manufacturing process, power grid or energy pipeline monitoring.
The vast majority of event processing applications today are custom-coded. Much of this custom coding effort, however, can be eliminated by using CEP software; the level of time and cost savings corresponds with the complexity of the event processing application. The remainder of this article will articulate a framework by which you can understand where and to what degree CEP software can offer cost savings over custom development.
What Does CEP Offer?
CEP software offers two major components: a high-level language for programmers to easily describe how to process the streams, and an infrastructure engine for processing and analyzing high-volume data streams. Although CEP software performs different functions, the component structure is mildly analogous to database software, where there is a language (SQL) and an engine (the database server).
Because some of the operations a programmer wants to perform on data streams are similar to a relational model, a select number of CEP vendors offer a language that is based on SQL. This provides a familiar programming environment, speeding the creation of event processing applications.
The engine provides the core components to execute the analysis at run-time. The engine takes on many complex tasks typical in data management infrastructure software as well as those unique to event processing:
- Stream management: Data streams are analogous to a database table of infinite size, with each new event appending a row onto the table. As streams often travel over a network, there can be issues such as dropped, delayed or out-of-order messages. A good CEP engine will automatically handle all these issues without requiring programmer intervention, ensure reliable message delivery and generate a valid, dependable stream for processing.
- Memory management: Data streams can become very large and have many queries running against them. A good CEP engine needs to optimize how memory is managed to ensure high throughput. Special care must be taken to avoid copying and ensure that every piece of data is only stored once.
- Parallel execution and synchronization: To maintain performance, a CEP engine will perform operations in parallel and synchronize data between the threads. Excess synchronization can hurt performance. Thus, a CEP engine not only has to automatically perform state synchronization for the programmer, but it must also balance the synchronization rates for efficient execution.
- Windows: Processing on data streams is performed in "windows," typically, units of time. An efficient CEP engine must be able to expire messages properly, both on new events and timer events.
- Indexing: Fast-moving data streams require indexes to be continually updated at a similar high rate for efficient processing. A good CEP engine will automatically manage these indexes so the programmer does not have to deal with such issues.
These and many more functions are abstracted from the programmer, making the development of CEP applications easier.
Types of Event Processing Applications
If a developer were to create a custom-coded event processing application, he or she would need to code some if not all of the CEP engine features mentioned above, depending on the complexity of the event processing application.
To simplify the framework for determining the applicability of CEP software, let's examine event processing applications in four tiers:
- Tier One: simple event processing applications,
- Tier Two: event processing applications involving multiple streams and/or stored data,
- Tier Three: complex analysis and pattern matching across event streams, and
- Tier Four: multiple, enterprise-class event processing applications.
Abstract: Yemanja is a model-based event correlation engine for multi-layer fault diagnosis. It targets complex propagating fault scenarios, and can smoothly correlate low-level network events with high-level application performance alerts related to quality of service violations. Entity models that represent devices or abstract components encapsulate entity behavior. Distantly associated entities are not explicitly aware of each other, and communicate through event propagation chains.
Cited by: More
Non-deterministic Diagnosis of End-to-End Service Failures in a.. - Steinder (2001) (Correct)
The present and future of event correlation: A need for.. - Steinder, Sethi (2001) (Correct)
Combinatorial Designs In Multiple Faults Localization For.. - Fecko, Steinder (2001) (Correct)
Active bibliography (related documents): More All
0.8: End-to-end Service Failure Diagnosis Using Belief Networks - Steinder, Sethi (2002) (Correct)
0.6: Increasing Robustness of Fault Localization Through Analysis.. - Steinder, Sethi (2002) (Correct)
0.3: IP Fault Localization Via Risk Modeling - Ramana Rao Kompella (2005) (Correct)
Similar documents based on text: More All
0.4: Intelligent Search of Correlated Alarms for GSM Networks.. - Zheng, Xu, Lv, Ma (2002) (Correct)
0.3: A Conceptual Framework for Network Management Event.. - Masum Hasan Binay (Correct)
0.3: GulfStream - a System for Dynamic Topology.. - Fakhouri.. (2001) (Correct)
Related documents from co-citation: More All
6: IFIPIEEE International Symposium Integrated Network Management (context) - IFIP, Symposium et al. - 2001
3: Alarm correlation (context) - Jakobson, Weissman - 1993
3: High speed and robust event correlation (context) - Yemini, Kliger - 1996
Citations (may not include all citations):
107 Remote Network Monitoring Management Information Base - Waldbusser - 1995
46 Oceano -- SLA-based management of computing utility (context) - Appleby, Fakhouri et al.
36 Alarm correlation (context) - Jakobson, Weissman - 1993
30 Schemes for fault identification in communication networks - Katzela, Schwartz - 1995
25 High speed and robust event correlation (context) - Yemini, Kliger et al. - 1996
23 Event correlation using rule and object based techniques (context) - Nygate - 1995
22 and Internetworking Protocols (context) - Perlman, Second et al. - 1999
20 GEM -- a generalised event monitoring language for distribut.. (context) - Mansouri-Samani, Sloman - 1997
17 A Complete Guide to DB2 Universal Database (context) - Chamberlin - 1998
15 Definition of Managed Objects for Bridges (context) - Decker, Langille et al. - 1993
14 A case-based reasoning approach to the resolution of faults .. (context) - Lewis - 1993
11 Event correlation in heterogeneous networks using the OSI ma.. (context) - Jordaan, Paterok - 1993
10 Layered model for supporting fault isolation and recovery (context) - Gopal - 2000
10 A conceptual framework for network management event correlat.. - Hasan, Sugla et al. - 1999
9 Scaling Internet services by dynamic allocation of connectio.. (context) - Goldszmidt, Hunt - 1999
9 Composite events for network event correlation - Liu, Mok et al. - 1999
8 Towards a practical alarm correlation system (context) - Houck, Calo et al. - 1995
6 Alarm correlation engine (context) - Wu, Bhatnagar et al. - 1998
3 Service Level Agreements : Managing Cost and Quality in Serv.. (context) - Hiles - 1993
2 Value-oriented network management (context) - Schwartz, Zager - 2000
1 A modeling framework for integrated distributed systems faul.. (context) - Katker - 1996
1 A Simple Network Management Protovol (context) - Case, Fedor et al. - 1990
1 IBM Internal Article (context) - Appleby, Fakhouri et al.
1 Management Information Base Network Mangement TCPIP based in.. (context) - Rose, Base et al. - 1991
http://www.adventnet.com
Documents on the same site (http://www.cis.udel.edu/~steinder/PAPERS/index.html): More
End-to-end Service Failure Diagnosis Using Belief Networks - Steinder, Sethi (2002) (Correct)
Increasing Robustness of Fault Localization Through Analysis.. - Steinder, Sethi (2002) (Correct)
The present and future of event correlation: A need for.. - Steinder, Sethi (2001)
With the increasing complexity of enterprise networks and the Internet, event correlation is playing an increasingly important role in network as well as integrated system management systems. Even though the timing of events often reveals important diagnostic information about event relationships and should therefore be represented in event correlation rules or models, most extant approaches lack a formal mechanism to define complex temporal relationships among correlated events. In this paper, we discuss the formal use of composite events for event correlation and present a composite event specification approach that can precisely express complex timing constraints among correlated event instances, for which efficient compilation and detection algorithms have been developed in [13, 14]. A Java implementation of this approach, called Java Event CorrelaTOR (JECTOR), is described, and some preliminary experimental results of using JECTOR in an experimental network management environment are also discussed in the paper.
Abstract: This paper addresses the problem of efficient management of events, in particular in those environments where events carry information useful to multiple applications, possibly operating in different domains and at different levels of abstraction. We investigate the problems and opportunities offered by such environments, and define a framework that enables a semantic mapping of events, i.e., enables the processing and successive refinement of events at different levels of abstraction, so that they can be understood and efficiently consumed by business applications. We identify the requirements of an event mapping system and present a specification language, integrating high-level Petri nets and database query languages, which provides the required expressive power to specify complex event processing functions and includes a set of constructs that support the design process and allows efficient implementations.
Event correlation simplifies and speeds the monitoring of network events by consolidating events and error logs into a short, easy-to-understand package. A network administrator can deal with, say, 25 events based on cross-referencing intrusion events against firewall entries and host/asset databases much more efficiently than when he must scan 10,000 mostly normal log entries.The benefits can be very real: more efficient use of staff time and skills, as well as the prevention of revenue loss resulting from downtime.
According to Marcus Ranum, an independent computer and communications security consultant in Woodbine, Md., "Correlation is something everyone wants, but nobody even knows what it is. It's like liberty or free beer -- everyone thinks it's a great idea and we should all have it, but there's no road map for getting from here to there." Still, a variety of technologies and operations are associated with event correlation:
Compression takes multiple occurrences of the same event, examines them for duplicate information, removes redundancies and reports them as a single event. So 1,000 "route failed" events become a single events that says "route failed 1,000 times."
Counting reports a specified number of similar events as one. This differs from compression in that it doesn't just tally the same event and that there's a threshold to trigger a report.
Suppression associates priorities with events and lets the system suppress an alarm for a lower-priority event if a higher-priority event has occurred.
Generalization associates events with some higher-level events, which are what's reported. This can be useful for correlating events involving multiple ports on the same switch or router in the event that it fails. You don't need to see each specific failure if you can determine that the entire unit has problems.
Time-based correlation can be helpful establishing causality -- for instance, tracing a connectivity problem to a failed piece of hardware. Often more information can be gleaned by correlating events that have specific time-based relationships. Some problems can be determined only through such temporal correlation. Examples of time-based relationships include the following:
• Event A is followed by Event B.
• This is the first Event A since the recent Event B.
• Event A follows Event B within two minutes.
• Event A wasn't observed within Interval I.
Winning Users Over
"Event correlation, in its basic form, is becoming almost a commodity product," says Drogseth. "Where you want to reduce the number of events and events and have some level of topological awareness to eliminate duplicates -- that's pretty standard and working today." Buyers are skeptical, but Drogseth says many event-correlation products work well out of the box or with minimal customization.
"There are any number of more sophisticated approaches that are all about diagnostics, finding out what is the real cause of a problem," Drogseth says. "Here, you have to address a lot more complexity in network infrastructure." When you start trying to isolate a problem and get at the true root cause, he says, "you have a high level of investment and complexity, but also a high level of value."
Copyright © 1996-2009 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Site uses AdSense so you need to be aware of Google privacy policy. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
Disclaimer:
Last modified: April 26, 2009