Logs Management, Security & Integrity

The information that is routinely collected about specific events occurring within information technology (IT) systems and networks can be used by organizations to improve the security of their operations. This information is recorded as an entry in a log, and each log entry can be linked to a particular event. Log entries, which can be analyzed when organizations need to identify security incidents and operational problems, provide valuable information to managers who are responsible for the operations and security of systems.   

Log entries are recorded by the systems’ software and applications. The entries containing information related to system security are produced by several sources. Some log entries are created by security software, such as antivirus software, firewalls, intrusion detection systems, and intrusion prevention systems. Other sources of security-related log entries are the operating systems on an organization’s servers, workstations, and networking equipment, and the applications on the systems.

NIST’s Information Technology Laboratory recently issued Special Publication (SP) 800-92, Guide to Computer Security Log Management, by Karen Kent and Murugiah Souppaya, to help organizations develop, implement, and maintain effective processes for managing logs with security-related information. It contains basic information about computer security logs, the usefulness of these logs, and the challenges of managing them. Briefly metioned are the components of the log management infrastructure; the planning processes that enable the organization to carry out consistent, reliable, and efficient log management practices; and the operational processes that aid organizations in successfully managing logs.  See  http://csrc.nist.gov/publications/nistpubs/index.html.

Logs can be used for many purposes:   

1. optimizing system and network performance; to record the actions of users;

2. identifying security incidents, policy violations, fraudulent activities, and operational problems;

3. performing audits and forensic analyses;

4. supporting internal investigations;

5. establishing baselines; and

6. identifying operational trends and long-term problems.

 

NIST’s guide focuses on helping organizations manage the use of logs to improve IT security. While many logs are created by IT systems and could provide data that is useful for security, NIST SP 800-92 focuses on the logs that are closely related to computer security. For example, audit logs track user authentication attempts, and security device logs record possible attacks on systems. In managing computer security-related log data, organizations have to create, transmit, store, analyze and dispose of the data correctly.  The computer security records should be stored in sufficient detail for an appropriate period of time and be available for routine log analysis. Federal organizations have to take into account the requirements of laws, regulations, and organizational policies. For example, federal organizations may need to analyze the log information for compliance with federal legislation and regulations, including: 

• Federal Information Security Management Act of 2002 (FISMA) - requires federal agencies to develop, document, and implement an organization-wide program to provide information security for the information systems that support its operations and assets.  

• Health Insurance Portability and Accountability Act of 1996 (HIPAA) - mandates safeguarding the confidentiality, integrity, and availability of electronically protected health information. 

• Sarbanes-Oxley Act of 2002 (SOX) - applies to financial and accounting practices and the IT functions that support these practices. 

• Gramm-Leach-Bliley Act (GLBA) - requires financial institutions to protect their customers’ information against security threats. 

One of the challenges to the effective management of computer security logs is balancing the availability of large amounts of log information with the limited availability of organizational resources for analysis of the data. A large amount of information is collected daily by a large number of logs, and there are increasing threats to networks and systems. Organizations could realize benefits in using the data to reduce risks, but the staff time and resources needed to perform the analyses and to manage the log information have to be taken into consideration. 

The large number of log information sources may produce inconsistent and incompatible content, formats, and time stamp information, making it difficult for analysts to understand the meaning of the data collected. Organizations may have to utilize automated methods to convert logs with different content and formats to a single standard format with consistent data field representations.

Another challenge is protecting the confidentiality, integrity, and availability of log information. Information such as users’ passwords and the content of e-mails may be captured by logs. This raises security and privacy concerns involving both the individuals that review the logs and others that might be able to access the logs through either authorized or unauthorized means. Logs that are secured improperly in storage or in transit might also be susceptible to alteration and destruction by both intentional and unintentional techniques. As a result, malicious activities might go unnoticed and evidence could be manipulated to conceal the identity of a malicious party. 

Log information should be analyzed on a regular basis and in a timely fashion by security, system, and network administrators. These staff members need support for their exacting tasks. They especially need training on how to carry out the log analysis procedures and how to prioritize their activities effectively. They should also be provided with tools that can automate portions of the analysis process, such as scripts and security software tools. These tools can be helpful in finding patterns that humans cannot easily perceive, such as correlating entries from multiple logs that are related to the same event.  Analysis of logs by staff members has to be an ongoing activity so that organizations can predict future problems and prevent them. In the past, many logs have not been analyzed in a timely manner. When organizations do not institute sound processes for analyzing logs, the value of the logs is significantly reduced.

Organizations also need to protect the availability of their logs. Many logs have a maximum size; for example, the software is limited to storing the 10,000 most recent events, or keeping 100 megabytes of log data. When the size limit is reached, the log might overwrite old data with new data or completely stop collecting log information.  Both of these outcomes result in the loss of availability of log data. To meet data retention requirements, organizations might need to keep copies of log files for a longer period of time than the original log sources can support. It may be necessary to establish processes to archive the log information. 

Because of the volume of logs and the costs of archiving log data, it can be appropriate in some cases to reduce the logs by filtering out log entries that do not need to be archived.  The confidentiality and integrity of the archived logs also need to be protected.

NIST recommends that organizations carry out the following actions for more effective and efficient log management processes: 

Establish policies and procedures for log management. Organizations should develop standard processes for performing log management. In the planning process, logging requirements and goals should be defined. Based on those goals and requirements, an organization can then develop policies that clearly define mandatory requirements and suggested recommendations for log management activities, including log generation, transmission, storage, analysis, and disposal. An organization should also ensure that related policies and procedures incorporate and support the log management requirements and recommendations. The organization’s management should provide the necessary support for the efforts involving log management planning, policy, and procedures development.  Policies and procedures help to assure a consistent approach and implementation of laws and regulatory requirements throughout the organization. Audits, testing, and validation procedures can help to assure that the logging standards and guidelines are being followed.  Requirements and recommendations for logging should be created in conjunction with an analysis of the technology and resources needed to implement the log management process. Generally, organizations should require logging and analyzing the data that is of the greatest importance, and should also have non-mandatory recommendations for the other types and sources of data that should be logged and analyzed if time and resources permit. In some cases, organizations can choose to have all or nearly all of its log data generated and stored for at least a short period of time in case it is needed. This policy gives greater weight to security considerations than to usability and resource usage. Also this policy can support better decision making in some cases. When establishing requirements and recommendations, organizations should strive to be flexible since each system is different and will log different amounts of data than other systems within the organization.The organization’s policies and procedures should also address the preservation of original logs. Many organizations send copies of network traffic logs to centralized devices. In addition, they may use tools that analyze and interpret network traffic. In cases where logs may be needed as evidence in proceedings, organizations may wish to acquire copies of the original log files, the centralized log files, and interpreted log data.  This policy is useful in case there are any questions regarding the fidelity of the copying and interpretation processes. Retaining logs for evidence may involve the use of different forms of storage and different processes, such as putting additional restrictions on access to the records.

Prioritize log management appropriately throughout the organization. After an organization defines its requirements and goals for the log management process, it should then prioritize its requirements and goals based on the organization’s perceived reduction of risk and the expected time and resources needed to perform log management functions.  An organization should also define roles and responsibilities for log management for key personnel throughout the organization, including establishing log management duties at both the individual system level and the log management infrastructure level.

 Create and maintain a log management infrastructure. A log management infrastructure consists of the hardware, software, networks, and media used to generate, transmit, store, analyze, and dispose of log data. Log management infrastructures normally perform several functions that support the analysis and security of log data.  After establishing an initial log management policy and identifying roles and responsibilities, an organization should develop one or more log management infrastructures that can effectively support the policy and roles. Organizations should consider implementing log management infrastructures that includes centralized log servers and log data storage. When designing infrastructures, organizations should plan for both the current and future needs of the infrastructures and the individual log sources throughout the organization. Major factors that should be considered in the design include the volume of log data to be processed, network bandwidth, online and offline data storage, the security requirements for the data, and the time and resources needed for staff to analyze the logs.

Provide proper support for all staff with log management responsibilities. To ensure that log management for individual systems is performed effectively throughout the organization, the administrators of those systems should receive adequate support.  This should include disseminating information to log management staff, providing training, designating points of contact to answer questions, providing specific technical guidance, and making tools and documentation available.

Establish standard log management operational processes. The major log management operational processes include configuring log sources, performing log analysis, initiating responses to identified events, and managing long-term storage. In addition, administrators have other responsibilities, such as:

• Monitoring the logging status of all log sources;

• Monitoring log rotation and archival processes;

• Checking for upgrades and patches to logging software, and acquiring, testing, and deploying them;

• Ensuring that each logging host’s clock is synchronized to a common time source;

• Reconfiguring logging as needed, based on policy changes, technology changes, and other factors; and

• Documenting and reporting anomalies in log settings, configurations, and processes.

 Other NIST publications that support log management processes include: 

• NIST SP 800-31, Intrusion Detection Systems (IDS), provides information about hardware and software systems that automate the processes of monitoring events occurring in computer systems and networks, and of analyzing them for signs of security problems.

• NIST SP 800-40 version 2, Creating a Patch and Vulnerability Management Program, provides guidance on creating a security patch and vulnerability management program, and on testing the effectiveness of the program. 

• NIST SP 800-41, Guidelines on Firewalls and Firewall Policy, provides guidance on the development of policies to guide the selection, installation and maintenance of firewalls that protect systems connected to the Internet and to other networks. 

• NIST SP 800-83, Guide to Malware Incident Prevention and Handling, discusses how to protect the confidentiality, integrity, and availability of data, applications, and operating systems by preventing and handling incidents involving the insertion of malicious code and software into systems.