Effective Log Management

www.contextis.com
About Introduction
About Context Information Security Log files are historical records of the running state of hardware and software,
storing information on how they are used, errors that occur and application-
specific events which detail how users interact with them. Routine review of this
information can provide system administrators and computer security teams with
insight into how effectively the business is operating and where configuration
Context has a client base including some of the worlds most high profile blue chip errors may be causing issues on the network so that they can be remediated
companies and government organisations. Our strong track record is based above before they have wider impact.
all on the technical expertise, professionalism, independence and the integrity of
our consultants. Our comprehensive portfolio of technical services sets the Log records are also an immensely valuable source of information for computer
standard for the security industry. security purposes, but their value as part of a corporate intrusion detection and
incident response process is largely misunderstood by many organisations;
Context has a dedicated incident response and investigation team, which works logs are either not collected at all or are collected without consideration for
with clients to detect, investigate, understand, respond to, and protect against how they might be used should an incident occur. The Effective Log Management
nefarious activity on their networks. This team does not focus on generic malware, project has been commissioned by the UK Centre for the Protection of National
but offers services designed to counter the most sophisticated attacks targeted Infrastructure (CPNI) to demonstrate the value of this data to the reader and
against our clients. Our team comprises experts who can provide substantial discuss how it can be used to support an efficient response to a network
experience and expertise on a business as well as a technical level. intrusion. This booklet accompanies the full whitepaper Effective Log
Context is proud to be part of the Cyber Incident Response scheme run by Management, available from www.cpni.gov.uk. The advice within these
CPNI and CESG, the Information Assurance arm of GCHQ. documents is intended to be used in conjunction with the 20 Critical Security
Controls for Effective Cyber Defence coordinated by the Council on CyberSecurity.
About This Booklet
Context would like to acknowledge the help and support of CPNI in researching Only 6% of network breaches
this topic and producing the accompanying products. discovered by log review
While 86% of victims held evidence
of the breach in log files*
Developing a log management strategy to enhance an organisations computer
security posture is not easy and there are a number of hurdles to overcome.
The number of devices that generate logs on a network can be overwhelming
and storage strategies can be both expensive and complex to implement.
What logs should be collected? For how long should they be stored? How should
they be stored to best ensure the security and integrity of the data? How can log
files be used proactively to look for the malicious needles in the haystack
of data available?
The lesson for organisations is simple: logs are an evidence source of potentially
vital importance, but effort is required to exploit them for maximum value.
They are a dataset which requires little effort to start collecting at some level,
but are greatly helpful during the investigation of security breaches. They can be
This material is provided for general information purposes only. You should make your own judgement as
regards use of this material and seek independent professional advice on your particular circumstances. used by forward facing organisations to identify, track and mitigate attacks before
Neither the publisher, nor the author, nor any contributors assume any liability to anyone for any loss or any damage is caused. However, in all cases a clear log management strategy is
damage caused by any error or omission in the work, whether such error or omission is the result
of negligence or any other cause.
essential to ensure the data is of sufficient quality and is organised appropriately.

Context Effective Log Management * source: http://www.verizonenterprise.com/resources/reports/rp_2010-data-breach-report_en_xg.pdf

Challenges Advantages
When defining a log management strategy within an organisation, a balance will The Role of Log Management in Incident Response
likely have to be struck between breadth and depth of data collection and how
it will be stored and analysed. While the ideal policy may be to fully log all Correlation of log events across a range of devices is a critical part of any incident
possible data sources and retain this information for extended periods whilst response activity as it assists an organisation in assessing the extent and impact
simultaneously analysing them for anomalies, this is rarely practical for many of a network compromise, as well as informing what steps may be necessary
organisations. Therefore, it is critical that the inclusion of each data source is for mitigation. Network security appliances may provide elements of the picture
blended into an overall incident response process to create a layered approach during an attack, but may not have full visibility over a network or possess the
that ensures that the best possible view of an intrusion can be determined as correlation capabilities to fully describe the attackers activities.
quickly as possible For example, a security product may identify when an intruder performs a remote
Common challenges associated with effective log creation, analysis and attack against an organisations web application, but it may not have the alert rules
management include the number of devices capable of generating log data and to identify the attackers propagation through a network from the initial point of
the associated resources required for retention, the security of log data storage compromise. By correlating the original alert event with log files from the web,
and the effective analysis of data once it has been captured. An additional database and authentication servers as well as events generated on the hosts,
challenge is the involvement of third-party IT service providers and their ability greater visibility into the extent of the compromise may be established.
to supply responders with the required data in a timely fashion.
Comprehensive logging over the longest timescale possible will allow incident The Role of Log Management in Intrusion Detection
responders to generate a detailed picture of an intruders activities. While log files are typically used to respond to incidents by determining the
Many compromises are only discovered weeks, months or in extreme cases, how, when and what of an intruders activities, they can also be used proactively
years after the initial breach. to monitor the security posture of a network. By analysing the data generated by
devices on a regular basis, security teams may be able to detect and respond to
the initial stages of a security incident as they occur.
Top three primary challenges to attack detection Intrusion detection requires greater up-front resource commitment both in
staffing and time allocation, but proactive analysis can detect events that may
using logged events:
not be highlighted by automated systems and can be used to limit the wider
1. Organisations not collecting the appropriate operational impact and cost of a security incident.
and security-related data to enable analysis
2. A lack of system and vulnerability awareness for network assets
3. A lack of relevant event context to observe normal behaviour
and detect anomalies
(Source: https://www.sans.org/reading-room/analysts-program/security-analystics-survey-2013)

Context Effective Log Management

Log Management Preparation Log Sources
The generation, storage and analysis of log data must be defined by a process While log data can be generated by large numbers of devices, they generally relate
that first identifies the assets that need protection, establishes data sources that to one of four source classes:
can contribute to defending these assets and then validates their inclusion in Services that provide functionality to users
the log management strategy. Validation of log sources should be iterative and
be informed by the process of analysis; if the captured data is of limited value or Infrastructure supporting the network
stores too many records for the projected retention period, the inclusion of the Host devices
data source as it currently exists and length of time the data is stored for should
Remote connection services
be re-evaluated and incorporated back into the overall strategy.
The focus of this booklet is to identify appropriate sources of log file data within
an organisation in order to facilitate intrusion detection and incident response
Identify Risks processes. An increase in the availability and popularity of cloud and managed
services means that logging for these data sources may not be readily controlled
Understand the network assets that need most protection
by local network administration. The following recommendations will still be
relevant in situations where a third-party supplier provides a service, although
the availability and accessibility of particular records may vary. Therefore, it is
important that the availability of log data from these services is discussed and
Identify Log Sources agreed with the provider in advance of potential security incidents.
Identify sources that can capture logs relevant to those assets For all log sources, both failures and successes should be logged wherever
possible, as some attacker activities may not be considered anomalous and
therefore would not be stored as exceptions. Without logging successful actions,
it will be extremely difficult to compile an accurate picture of how an attack was
Identify Data Access executed and what follow-on activities were conducted.
Determine how the data can be collected from these sources
Without logging successful actions,
it will be extremely difficult to
Identify Format and Fields compile an accurate picture of how
Define the most appropriate record format and fields required for analysis
an attack was executed and what
follow-on activities were conducted
Validate Value to Analysis Process When establishing a log management policy, it is important to consider what data
Analyse data and measure value to security of assets is appropriate to store at a minimum to enable intrusion detection and response
procedures. While storing all data for every possible source may be the ideal for
full response purposes, this might not be practical in every situation. The following
pages provide a product-agnostic view of the primary log sources on a typical
network, describing how they might be used during analysis to contribute to the
Validate and Adjust Retention Period overall picture of a security incident.
Measure quantity of data stored against value to analysis,
modifying retention period in accordance with resources and policies

Context Effective Log Management

Servers Servers
Proxy Server Web Server
Proxy servers are intermediate devices through which remote hosts such as Web servers represent a potential attack vector for an attacker either via the
web servers are accessed from within a network. If employed on a network, exploitation of vulnerabilities in the applications they are serving or through the web
web proxies can provide a source of log data for outbound network traffic as they server software itself. Servers that are connected directly to an external network
store information on requests to web servers including the Uniform Resource such as the internet can provide an initial foothold into a network, whereas internal
Locator (URL) requested and the status of the response. This information can web servers may be exploited for lateral propagation purposes. Web servers are
provide an indication of outbound requests sent from malware to obtain tasking also commonly compromised to act as watering holes, where an attacker will use
and to send data back to an attacker, along with the IP address of the affected their access to serve exploit code to visitors of corporate websites. Examination of
host on the network. log data relating to web servers can identify abnormal requests that might indicate
attacker reconnaissance or exploitation activities, as well as providing insight into
Mail Server outbound requests that may show exfiltration activity from servers being used as
data staging host.
A common attack vector for host compromise is spear phishing or content
delivery attacks, where an attacker sends emails to a target containing malicious
software as an attachment. Logging of metadata and content pertaining to inbound Database Server
email traffic can help establish which users have received emails with malicious SQL injection (SQLi) attacks are a common method for exploiting weaknesses in
content, when the attacks occurred and other information that may be useful for web application software to obtain information from, or control over, a database
ongoing detection purposes, such as email subject lines that may be reused or server. Depending on the objectives of the attacker, SQLi attacks can be used to
could provide insight into the methodologies used to entice users into opening display the contents of protected database tables, add new user accounts to
the attachments. enable direct remote access, provide the ability to upload and execute attack
In some cases, malicious software may have the capability to use Simple Mail tools and can even provide full remote desktop access to the server.
Transfer Protocol (SMTP) as a command and control or data exfiltration channel. Database transaction logs can be used to determine what commands were sent
When investigating incidents involving this type of malware, logs of outbound email to the database and whether or not they were successful. For attacks where
sessions can also be useful to determine the frequency and type of data sent from the primary objective is to obtain the information stored within the databases
the network. themselves, log data can provide insight into what access the attacker has
obtained and the impact of the activity.
If an intruder attempts to upload additional tools to the server via SQLi, database
logs can potentially be used in conjunction with web server logs to extract and
analyse the malicious payloads even if the original files have been deleted from
the host.

Context Effective Log Management

Network Hosts
Authentication Servers Program Execution Auditing
Authentication servers are used to authorise clients to join networks and access Most operating systems provide some administrator-level logging of program
resources held on them. Although the implementations and the protocols used execution, such as Audit Process Tracking within the Microsoft Windows
for authentication vary depending on the network environment, they typically log environment. When enabled, this data can allow analysts to track when
each user authentication attempt along with a set of metadata about the applications start and terminate as the host is running, potentially allowing
connection. Log data from authentication servers can be used during incident them to identify the execution of malware
response to determine how an intruder is connecting to the network and to
track their movements if they are propagating between hosts.
Control of an authentication server may represent a very attractive target to an
Process Execution Logging can
intruder as they can potentially leverage this to have unrestricted access to a allow analysts to track when
number of resources on the network. In addition to the logging of authentication
attempts to the server, it is important to ensure that host logging is activated to applications start and terminate,
detect direct compromise of the authentication server.
potentially allowing them to identify
Firewalls the execution of malware
Firewalls can provide a rich source of data for network traffic types that contravene
Other logging options can provide opportunities to detect malware persistence
company policies based upon the rule sets they are configured with. Most devices
mechanisms as they can track services being started and scheduled jobs
can also track the state of traffic flows passing through them in order to perform
being registered. These log sources are best used in conjunction with an
content inspection of the sessions to detect, alert on and block malicious activity.
understanding of the typical configuration of hosts on the network so that
The value of firewall log data will vary depending on where it is deployed within a
anomalous behaviour can be identified.
network and the type of attack activity undertaken; firewalls on the border of a
network may be able to detect incoming attacks and data exfiltration events,
but will have no visibility of lateral movement between hosts. File Access Auditing
The creation of log records relating to user and application access of sensitive
Routers and Switches files can provide an indication of potentially what data may have been lost in
a network intrusion event. Within the Microsoft Windows environment, this is
Routers may be configured to permit or deny particular network traffic types and
achieved through Windows Object Access Auditing, which logs accesses to files
to log information about traffic that falls outside of these policies. In addition to
in administrator-defined folders and network shares.
this, many device models will log information regarding their system state and
attempts to authenticate to their administrative interfaces. Some routers and In addition to file access logging within the operating system itself, Document
switches have the ability to generate additional logging on network protocols Management Systems may also provide log data to track which documents are
that may be in use on the network. This information may be used in an intrusion accessed by which users.
detection capacity or as part of a proactive approach to improving network
security by detecting implementation issues.

Context Effective Log Management

Security Software Remote Access
Network Intrusion Detection Systems Virtual Private Networks
Network Intrusion Detection Systems (NIDS) can record detailed information Virtual Private Networks (VPNs) provide a mechanism to remotely connect
on suspicious network traffic flows based upon signature detection routines. devices to a network and use associated resources as if they were connected to
Logging rates for NIDS will vary depending on the type and number of signatures the corporate environment. While VPN technology is advantageous for connecting
deployed on the system and therefore the effectiveness of detection capability disparate computing resources and enabling remote working, the compromise of
can be reviewed over time by examining log records to determine if the alerts VPN access credentials can represent a significant vulnerability for a network as
are relevant and actionable. a remote attacker can interact with any host on the network that the user account
has access to.
Host Security Software VPN logs can provide a good data set for behavioural analysis purposes and can be
Security products such as anti-virus software are often deployed to provide a used to identify sessions that are markedly different from typical user interactions
local level of protection to the host. Log records generated by this software can be with the network.
useful for identifying intruder tools, although their value will strongly depend on
the quality of signature sets and heuristic rules available to the application and Remote Desktop Services
whether the malware has been specifically designed to evade detection. Even if an Remote desktop services allow users to connect remotely to machines within
attackers malware is not detected by anti-virus applications, additional tools such a network, either by establishing a new session on the host or taking control of
as password dumpers may be identified through heuristic detection routines. an existing one, depending on the software used. While the remote host is not
In addition to the analysis of anti-virus detection logs, log files of the software logically connected to the network as is the case with VPN connections, remote
itself can be useful as attackers may attempt to disable scanning routines, desktop services still provide many of the capabilities available to a local user such
leaving evidence as log events as access to network shares and applications installed on the machine. They also
allow data to be sent between the remote host and the client through interactions
between clipboards (copy and paste) and, depending on the software and
configuration, drag and drop file transfer.
Remote desktop connections are often favoured by intruders as a mechanism for
controlling a compromised host after valid user credentials have been obtained
by malware or key loggers. Once connected via a remote desktop connection,
attackers have the opportunity to use the graphical user interface of the host to
install new tools, disable host-based security software and perform further
reconnaissance of the network.
Analysis of log records pertaining to remote desktop connections can indicate which
hosts within a network are controlled by an attacker based off known indicators of
compromise, or can be used in a proactive manner to detect abnormal activity such
as access to hosts outside of business hours.

Context Effective Log Management

The Kill Chain Storage and Retention
The Kill Chain is a conceptual model used to define the stages through which an Once the log source identification and selection process has been completed,
adversary has to progress to achieve an objective. When considering how log sources the appropriate method for storing this information must be determined.
should be deployed and analysed, the stages of the Intrusion Kill Chain (Reference: There are three main log storage strategies that can be employed: local storage
http://papers.rohanamin.com/?p=15) can be a useful model for determining how a on each device, centralised storage on a log server or a combination of the two
layered approach to log data generation can maximise the opportunities to detect depending on the required detail and accessibility of log records.
an intruder through proactive analysis of a range of data sources. Local logging distributes storage requirements across the enterprise as each
device is responsible for supplying its own data store and therefore no separate
Reconnaissance database provision is required. Centralised logging can take the form of a
Web server logs Remote connection device logs
networked file store that various agents can write their logs to, or a purpose-built
Network Intrusion Detection System (NIDS) alert logs Network firewall logs solution in the form of a Security Information and Event Management (SIEM) tool.
Although some form of centralised logging is generally recommended as it
ensures that larger amounts of log data can be pre-collected and correlated to
Weaponisation enable real-time analysis for both intrusion detection and response purposes,
it comes with a greater up-front provisioning requirement for primary storage and
NIDS alert logs
log backups. A combination of the two approaches may be considered as a method
for overcoming many of these disadvantages at the expense of increasing overall
administration overhead. In this case, the most important log sources and data
Delivery
fields are automatically forwarded to a centralised location for ingestion and
real-time analysis, while more in-depth data is stored for shorter periods on
Server logging (web, database, email)
NIDS alert logs
individual devices and is interrogated on-demand when an incident is detected.

Suggested Log Record Retention Periods

Exploitation
Log Type Retention Time Notes
Host Intrusion Detection System (HIDS) alert logs
Proxy, web, authentication, 2 years+ Longer retention periods are required
remote access, firewall for these high value data sources that
can be used to identify the impact of
new and existing persistent threats
Installation
over typical attack timescales.
Host file system and process execution logging
HIDS alert logs Anti-virus alert logs Email, IDS, anti-virus, 6 months While valuable, these data sources
database, network 1 year are often coupled with active alerting
infrastructure facilities that can notify security teams
of incidents as they are detected.
Command & Control
Host process execution 4 weeks Depending on usage, host logging can
Web proxy logs and file access 6 months generate large numbers of events that
Domain Name System (DNS) logs
are mainly used for verification after
an intrusion has been discovered.

Activities on Target
The full spectrum of logging capabilities including log files
pertaining to interactions with hosts and network services

Context Effective Log Management

Security Utilising Log Data
However your organisation chooses to store log files, it is important to ensure that Responding to Incidents
they are suitably protected from manipulation or destruction by a malicious party
intent on covering the tracks of their intrusion. While early systems for logging The role of log analysis in incident response involves the retrospective examination
events often did not consider intentional or inadvertent modification of log records of log data following a security incident. Post-event filtering and analysis of
as a primary consideration in their design, additional security measures are now these data sets using indicators of compromise can help develop a picture of an
appropriate as log stores present an attractive target for attackers. intruders activity. This understanding can then be used to remediate the attack
by identifying and cleaning the affected systems, determining and correcting the
Although the security capabilities vary between log management solutions, intrusion vector used and ensuring that no backdoors have been installed to
the following general recommendations can be made: maintain access to the network. As part of this, log analysis can be used to gain
Ensure the integrity of the processes generating log records an understanding of the impact of the attack, including clarifying the origin and
Unauthorised users should not be permitted to modify or interrupt the quantity of any data taken from the network.
processes that are used to create log files on devices. For example, The log analysis process during incident response will vary with the nature and
users should not be able to turn off anti-virus software logging. extent of the attack. Often, a company will become aware of an incident through the
Limit access to local log files detection of malicious activity by security software, a notification from an external
If log data is stored on hosts, user accounts should not be given access to log organisation (private sector or government department) with wider knowledge of
files unless it is to append to existing records. If possible, no read, rename or an attack campaign, or through proactive anomaly detection conducted by internal
delete privileges should be granted on log files. security teams.
Implement security on centralised log files In some situations, the scope and complexity of a security incident may warrant
Centralised log stores can represent an attractive target for attackers wishing more in-depth analysis of log data that is beyond the resources of in-house
to modify log data across a range of devices. For log data that is sent to a network security teams. If this occurs, the expertise of an external incident
centralised location, security measures such as digital signing or encryption response company may be required to assist in the handling of an intrusion event,
can be explored to ensure the integrity of this data, although for large during which time it is advantageous to have readily available and well organised
volumes this may have unacceptable efficiency or administrative overheads. log stores so that data can be quickly passed to incident handlers to enable rapid
analysis and response.
Secure log data transmission
If possible, any data transmitted between log sources and a centralised log
Proactive Analysis for Intrusion Detection
store should be secured to prevent unauthorised modifications. While some
devices and log agents support encrypted communication channels natively, Although log management is most commonly associated with incident response
other log sources that use plaintext protocols can be protected through the processes where log files are queried retrospectively based upon one or more
use of additional layers of encryption such as Internet Protocol Security indicators of compromise, log data can be used as part of an ongoing process
(IPsec) tunnels. This may also be impractical for many organisations but is of review and analysis wherever possible to identify attacks as they occur.
worth considering when high-value log sources have to transmit their data The most effective way to begin this process is to establish a baseline
to centralised locations. understanding of the typical activities on a network and to dedicate regular analyst
resources to reviewing log data to identify anomalous events. The process to
determine which events are anomalous will vary greatly between organisations
and must be defined on a case-by-case basis by internal security teams or in
consultation with external incident response providers.
Once a solid understanding of this baseline has been obtained, additional log
sources can be introduced to provide a richer understanding of the state of the
network. In addition, developments in analyst tradecraft such as heuristic
identification of anomalous events (abnormal traffic volumes, remote connections
from unusual locations or at unusual times, unexpected communication between
hosts) will enhance an organisations ability to proactively identify attacks against
the network and limit any damage caused.
Context Effective Log Management
Utilising Log Data Case Study
Security Information and Event Management software In order to illustrate how an effective log management strategy can contribute
to a successful incident response investigation, we present a case study where
Security Information and Event Management (SIEM) software offers the capability comprehensive logging enabled a detailed picture of the incident and a high level
to aggregate, store and display log data in a way that can be used to enable near of post-investigation assurance that the incident was contained and remediated.
real-time analysis of the security of a network from the perspective of each log
source. If resourced and used appropriately, SIEM can help solve some of the Company X received a victim notification issued through the Cyber Incident
most complex problems facing an organisation looking to improve the security Response scheme informing them that a legitimate third-party website had been
of their network. compromised and was serving malware (via a watering hole attack) from a
suspected state-sponsored actor. There were indications that hosts on the network
SIEM deployments generally consist of one or more database and log analysis had been compromised after visiting the site.
servers and can acquire data either through agent-based or agentless
mechanisms. While available products differ in the overall capabilities they Company X had a dedicated incident response and investigative capability and
offer, common functionality includes: as such was able to stand up an internal team of significant size to conduct the
investigation, with external experts providing oversight and direction.
Log record aggregation
SIEM software provides a centralised location to aggregate logs from
multiple data sources. Investigation via log analysis
Data retention
Centralised storage also enables long-term retention of log data beyond limits
revealed that the attacker had not
that may be in place on each device. actively developed their foothold
Event correlation
SIEM software often includes the capability to integrate multiple log data
any further
sources and correlate events between them.
Visualisation dashboards During the investigation analysis of significant volumes (approximately nine billion
Many SIEM tools offer a range of customisable visualisation dashboards that records) of log data took place, primarily consisting of web proxy logs and host
can provide a high-level overview of the condition of the network and can data logs. While the watering hole attack was found to have been successful and
highlight abnormalities in the data. some malware was found to be active on the network, further investigation via log
analysis revealed the attacker had not actively developed this foothold any further
and there was no evidence of attacker activity elsewhere on the network. If the
deployed malware had been activated by the attacker, the log store would have
proved even more valuable as commands sent to the particular malware used
in this attack could be decoded from data stored within proxy logs, allowing
investigators to understand the attackers activities more thoroughly.
As a result of the initial notification, the organisation was able to rapidly identify
seven compromised hosts. Forensic analysis of the machines and reverse
engineering of the identified malware gave a thorough understanding of the initial
infection, provided further development of indicators such as IP addresses and
traffic signatures, but not whether it had spread beyond those seven machines.

Context Effective Log Management