CSIRTS

The Operational Role of Security Information

and Event Management Systems

Sandeep Bhatt, Pratyusa K. Manadhata, and Loai Zomlot | Hewlett-Packard Laboratories

An integral part of enterprise computer security incident response teams, a security operations center
(SOC) monitors security incidents in real time. Security incident and event management systems play
a critical role in SOCs—collecting, normalizing, storing, and correlating events to identify malicious
activities—but face operational challenges.

C omputer security incident response teams

(CSIRTs) are responsible for receiving, review-
ing, and responding to computer security incident
SOC receives event information from the sensors and
log files and triggers alerts indicating possible malicious
behavior, both at the perimeter of the network and in
reports and activity. Each CSIRT has a defined constit- the enterprise.
uency, such as a corporate, government, or educational When an alert is triggered, SOC personnel deter-
organization; a region; or a country. A CSIRT’s first task mine whether it was triggered inadvertently—perhaps
is to monitor security events related to its organization’s in response to routine network maintenance—and is
IT assets. Performing this task is the security operations harmless, or if the events indicate a strong likelihood of
center (SOC), which is typically a centralized unit in an malicious activity. In the latter case, the alert is escalated
enterprise IT organization. to a team that coordinates incident response and foren-
Security incident and event management (SIEM) sic activities with the owners of the involved servers and
systems are an important tool in SOCs—collecting, applications. In extreme cases, the team must also coor-
normalizing, and analyzing security events from diverse dinate with internal human resources, legal and market-
sources—but they must evolve to overcome future scal- ing executives, and law enforcement.
ability issues. A SOC’s effectiveness depends on its analytic and
forensic capabilities, access to actionable threat intel-
Security Incident ligence, awareness of the enterprise networks and sys-
and Event Management Systems tems, and internal processes to coordinate responses
A SOC’s goal is to monitor security-related events from from organizations across the enterprise.
enterprise IT assets, including the IT network, perim- In the early days, there were relatively few IT secu-
eter defense systems such as firewalls and intrusion rity tools, including firewalls for perimeter protection
prevention devices, application servers, databases, and and intrusion detection (IDSs) and antivirus systems
user accounts. Each asset might be monitored using a (AVSs) for monitoring hosts. Each of these systems
variety of sensors and maintain log files of activity. The came with its own vendor-specific user interface. As

1540-7993/14/$31.00 © 2014 IEEE Copublished by the IEEE Computer and Reliability Societies September/October 2014 35

j5man.indd 35 9/18/2014 11:55:34 PM

 CSIRTS

Event sources www.computer.org

Events Specialized connectors

www.computer.org

Authentication device
Security
management Security alerts
platform

Firewall

Normalized Terminal
Network device events
stream
alerts

Web application firewall

Forensic
analysis
Application server database

Hosts

Figure 1. A typical security incident and event management (SIEM) system architecture. The SIEM system accepts inputs from various security
devices and sensors. Connectors receive events, parse them, and convert them into a common format.

such tools became more widely used and more tools for post hoc forensic analysis as well as investigating and
appeared, two problems arose: first, there were too detecting slow and stealthy attacks, including advanced
many user interfaces to manage, and second, there were persistent threats (APTs).
no tools to correlate events across different security
tools. Correlating events across tools became a necessity Structuring SOCs around SIEM Systems
because individual tools that operate with little or no Figure 1 illustrates an SIEM system’s basic architectural
awareness of the IT architecture trigger too many false- components. The SIEM system accepts inputs from var-
positive alerts, even after careful tuning. On the other ious security devices and sensors, including perimeter
hand, when multiple sensors trigger alerts in response defense systems (network firewalls and intrusion pre-
to an action, the action is more likely to be malicious. vention systems), host sensors (IDSs and AVSs), appli-
SIEM systems were designed to meet these chal- cations (Web application firewalls and authentication
lenges: they collect events from diverse sources, each systems), and network sensors. Each device and sen-
of which might represent events using a vendor-spe- sor is configured to output security events with unusual
cific schema; normalize these disparate schemata into or anomalous behavior that might indicate malicious
a common representation; and store these normalized intent. These events are represented in vendor-, device-,
events. Their rule engine triggers alerts from the stored and version-specific schema. So, the SIEM system’s
events; the rules allow correlation of events from differ- first task is to normalize the different representations
ent sensors. SIEM systems also include auxiliary con- into a common format to ease further processing and
textual information, such as up-to-date information on to simplify rule creation and maintenance. As the figure
enterprise assets that can be used to write better, con- shows, SIEM system connectors—customized for each
text-aware rules and prioritize alerts. SIEM systems’ version, device type, and vendor—receive the events.
main strength is their ability to cross-correlate logs The connectors parse input events and convert them
from diverse sources using common attributes to define into a common format, and do so in a scalable manner
meaningful attack patterns and scenarios, which when to keep up with the event source.
they occur, can alert security analysts (SAs). Thus, SIEM Once normalized, events are forwarded to the secu-
systems are like radar, detecting objects in a timely man- rity management platform and the archival forensic
ner. Their long-term event retention capability is useful analysis database. The platform maintains and analyzes

36 IEEE Security & Privacy September/October 2014

j5man.indd 36 9/18/2014 11:55:35 PM

 a window of events, typically those seen in the past few and asset management systems, and external sources,
hours. If necessary, the archival database stores events such as threat activity alerts from public agencies and
for a longer term—for instance, three to six months— private corporations. If a breach is revealed, level 2 SAs
for further forensic investigation. Although different create a case and forward it to forensics teams and secu-
platforms have different capabilities, a typical archi- rity engineers (levels 3 and above) to determine the
val database can ingest up to 100 Kbytes per second, attack’s extent and impact. If the alert is a false alarm,
whereas a typical management platform can ingest level 2 analysts work with the security engineering team
approximately 10 Kbytes per second. Specialized hard- to fine-tune the rules that created the alert so that it’s
ware can further improve ingestion rates. less likely to trigger false alarms.
The management platform’s rule engine applies its
rules periodically to events in the window. Whenever a An Example SOC
rule triggers a new alert, the alert is sent to the SIEM The Hewlett-Packard (HP) enterprise network spans
system terminal for review by a SOC analyst. Each rule 166 countries and supports more than 300,000 employ-
captures information about malicious behavior. For ees. Its Cyber Defense Center (CDC) continuously
example, a rule might look for a large number of failed monitors the network—HP’s version of a state-of-the-
login attempts within a time window or look for HTTP art SOC—and is staffed with level 1 and 2 analysts,
requests to known malicious websites. The rules are who are supported by dedicated forensics analysts. The
generated from two sources: an analyst can create rules, CDC’s structure and its deployment of SIEM systems
and the SIEM system can algorithmically generate rules are representative of SOC deployments for large, global
from events, for example, via pattern mining—that is, enterprises and government organizations.
identifying sets of events that occur together frequently We estimate that HP’s enterprise network generates
within a time window.1 A rule engine might also use 100 billion to 1 trillion security events each day. Col-
anomaly detection—triggering alerts when observed lecting, storing, and analyzing all these events is nearly
events differ from normal events. impossible, so the CDC focuses on several important
event sources such as HTTP proxy, DNS, and antivirus
SOC Structure logs. Its elaborate SIEM system infrastructure involves
Modern enterprise SOCs are hierarchically structured hundreds of load-balanced connectors, more than 100
around an SIEM system. SOCs typically consist of instances of archival databases, and multiple instances
several levels of SAs—the higher the level, the more of the SIEM system manager. The infrastructure cur-
experienced the SA and the more specialized his or her rently processes approximately 3 billion events per day;
function. The lowest-level SAs (usually called level 1) this could grow to 30 billion events in the near future.
mainly monitor the SIEM system alert screen and tri-
age events, deciding whether they’re potential attacks Cost-Benefit Analysis
or false alarms. If a rule triggers many false positives, SIEM systems’ popularity in SOCs is due primarily to
level 1 SAs escalate the rule to a SOC engineer or a their ability to handle a large number of events from
higher-level SA for further tuning to reduce false posi- many different sources. When enterprise networks
tives. When level 1 SAs can’t classify an alert as either were smaller and generated fewer events, SIEM sys-
an attack or a false positive, they escalate the alert for tems weren’t very popular among enterprise network
further investigation. administrators. As enterprise networks started growing
The large volume of alerts flowing into an enterprise due to the addition of new devices, applications, and
SOC leads to overwhelming amounts of work and long ­employees, the number of events generated also grew.
work shifts. Although the number of alerts generated In addition, network administrators started collecting
depends heavily on the network environment, SOCs events from more sources. Hence, over the past decade,
typically aim for 1,000 to 3,000 alerts per day per level SIEM systems have become an indispensable tool for
1 SA for manageability. If a network generates more handling enterprise security events. SIEM systems are
alerts, then SAs sample the alerts. SOC teams usu- arguably the most important tools in SOCs today, and
ally work around the clock in 8- to 12-hour shifts. This we expect the trend to continue.
tough schedule and workload generally result in rela- However, operating large-scale SIEM systems
tively short job retention periods for level 1 SAs, often requires a large budget. A typical management platform
less than two years including training. might cost US$80,000, and an archival database
Level 2 SAs are tasked with investigating alerts that might cost $20,000. Hence, a large SIEM system
level 1 SAs identify as potential attacks. To carry out with hundreds of connectors, a few hundred archival
deeper analysis, level 2 SAs access a wider range of infor- databases, and multiple platforms might require $3 to
mation, including internal sources, such as system logs $5 million in up-front hardware cost and additional

www.computer.org/security 37

j5man.indd 37 9/18/2014 11:55:35 PM

 CSIRTS

yearly maintenance cost. If a SOC staffs 20 SAs, the severe time restrictions force SAs to sample alerts from
yearly operating cost might be upwards of $5 million. the events list. Although the number of alerts on SA
Enterprise-scale SIEM systems need significant screens is proportional to the size of the logs flowing
investment in both hardware and manpower, and SIEM into the SIEM system, some professionals claim that by
systems and SOCs must continue to deliver to justify writing the right SIEM system rules and applying the
the investment. right management techniques, they can dampen the
relation between the volume of logs and alerts. This
Operational Challenges remains speculative; with the explosive growth in data
SOCs confront various operational challenges when using rates, it’s difficult to see how SIEM system processing
SIEM systems, driven primarily by the scale and complex- rates can keep up under cost constraints.
ity of the enterprise being monitored and the rate at which
events arrive from security devices and sensors. Lack of Contextual Information
Another challenge SOCs face is isolation from enter-
Rule Creation and Management prise network operations. SOC personnel aren’t
Having all the network and host logs at the SAs’ finger- involved in the details of configuring, testing, and
tips is attractive because the more information a SOC maintaining enterprise assets. Routine activities such
has, the better its situational awareness. However, this as patching, backup, and testing might trigger alerts
comes at the cost of trans- in SIEM systems designed
forming an SIEM to detect security
system’s data manage- Communicating the right kind and breaches, and track-
ment to big data man- amount of information between ing down the cause
agement, which turns of such alerts creates
enterprise operations and the SOC
storage, search, shar- unnecessary overhead.
ing, transfer, analy- in an automated way is essential. In an interview,
sis, and visualization a senior SA pointed
into challenges. One out the importance
aspect of this problem is the system’s inability to effi- of automatically collecting detailed host configurations,
ciently execute complex queries, severely limiting SAs’ servers, devices, and user information. In principle, this
ability to write c omplex correlation rules. A more information can be correlated with SIEM system alerts
problematic aspect is the number of false alarms that to significantly reduce false-alarm rates. However, col-
the SIEM system rules tend to trigger. Because benign lecting and maintaining this information, especially in
events outnumber malicious ones, even a low false- large networks, is challenging. Instead, such informa-
positive rate will produce many false alarms,2 which the tion is often communicated in an informal, even ad hoc
SOC might not have the capacity to deal with. There- manner, either verbally or via email. In one incident, an
fore, SIEM system rules must have extremely low false- SA had to contact the network operations team about
positive rates to be usable in practice. potential malicious activity in the internal network,
Most of the time, SIEM system analysts need to write which turned out to be a spurt in traffic from a patching
very specific rules to capture an attack, but this means server. The SOC saw probing alerts on its screens; these
the system might miss other forms of that attack. Thus, were manually tracked to the patching server and even-
there’s always a tradeoff between false-positive and tually declared false positives.
false-negative rates. To prevent false negatives—that Information communicated informally usually falls
is, detection misses from overly specific attack rules— through the cracks when SOC analysts change shifts,
engineers resort to generic rules, so that an activity with thus exacerbating the problem. Instead of storing cru-
even a remote possibility of indicating an attack will cial contextual information in SIEM systems, all too
trigger an alert. Then, analysts are responsible for moni- often, SOCs rely on SAs to maintain this information.
toring the SIEM system to distinguish the true alarms Unfortunately, this information is lost when SAs leave
from the enormous number of false ones. Many SOC and replacements are hired.
teams have limited resources to process overwhelming In general, although isolating a SOC from the
volumes of events. Thus, the SOC enters a vicious cycle enterprise systems’ routine maintenance activities is
of accumulating more and more alerts that SAs must a reasonable objective, communicating the right kind
process each hour. and amount of information between enterprise opera-
In talking with many SOC teams, we found it’s tions and the SOC in an automated way is essential to
acceptable to triage an event in 10 minutes; some teams reduce the SOC’s load and to achieve more effective
would like to reduce this to one minute or less! Such security monitoring.

38 IEEE Security & Privacy September/October 2014

j5man.indd 38 9/18/2014 11:55:35 PM

 Ad Hoc Use of Long-Term Data systems might need their own event generation and fil-
Current SOC operations use SIEM system features in tering solutions to cope with problems of scale.
an unbalanced manner. Most teams focus on short-term SIEM systems also face input validation chal-
alerting functionality and ignore the long-term reten- lenges. As we collect events from many sources, the
tion feature. SOCs usually monitor a rolling and narrow data becomes increasingly noisy. For example, an event
time window of events—typically an hour or two. This source might be unavailable or might not be able to
limits their ability to detect stealthy, slow-advancing populate all fields in an event. The data might also con-
attacks, especially APTs. In our interviews with SOC tain malicious events. If attackers compromise a data
analysts, we found that many teams were aware of this source or a communication channel, they might be able
issue and tried to mitigate it by sampling the raw logs to inject malicious events into SIEM systems. Hence,
randomly with the hope of finding attack patterns. SIEM systems must evolve to prevent attackers from
SOC analysts should give more attention to the SIEM generating malicious events and detect and filter out
system’s retention feature and develop analytic solu- malicious and noisy events.
tions that help in visually revealing the patterns of the Ultimately, the challenge is to determine the “right”
slow and stealthy attacks. Running these analytics also events to collect. Given a security problem to solve, col-
helps to guide the SIEM system rule-writing process by lecting and correlating the right events is much easier
unveiling unknown attacks. This practice shifts the SOC than dealing with all enterprise events. But how do we
team from a reactive to a proactive state. determine the right events a priori? Are there problems
for which event sampling is sufficient? There will always
Technical Challenges and Opportunities be the fear of missing important events if not everything
We believe that SIEM systems will continue to be an is collected.
integral part of SOCs; however, they must evolve to deal
with event collection, storage, analysis, and visualization Event Storage
challenges. The key challenge SIEM systems face is the SIEM systems must balance storage costs with analy-
scale of events enterprise networks generate from hard- sis requirements. For example, if one event requires
ware devices, software applications, and actions of people approximately 400 bytes of storage and we achieve
in the network. More complex applications and devices 10x data compression, then 1 trillion events per day
tend to generate more events; for example, accessing a will require 40 Tbytes of storage. Although storing
modern webpage might generate several HTTP requests the data for perpetuity would be ideal for analysis and
for content embedded in the webpage. Therefore, an forensics, storage costs make this impractical. Thus,
enterprise’s number of events is proportional to the num- we need to define a data retention period that makes
ber of employees, the number of network devices and an optimal tradeoff between storage costs and analysis
applications, and the devices’ and applications’ com- requirements. In addition, regulatory compliance might
plexity. The number of events will grow as enterprise require sensitive events to be deleted after a time period.
administrators enable event logging in more devices and SIEM systems also face a tradeoff in storage archi-
applications; the number will also grow each year as cor- tecture, as forensic analysis and real-time analysis have
porations hire more employees and add new and com- different characteristics. In forensic analysis, we make
plex devices and software applications to their network, infrequent queries on large data volumes; thus, write-
for instance, bring-your-own-device scenarios. optimized databases with high ingestion rates might be
suitable. As long as SIEM systems can store events at a
Event Collection high rate, a reasonably higher query response time or
Collecting events in a scalable manner is a challenge. a small delay in data availability is tolerable. However,
For example, if we decide to collect 1 trillion events per read-optimized databases might work better for real-
day, our SIEM system must support an ingestion rate of time analysis. We might prefer the ability to read stored
12 million events per second. The required rate is orders data quickly for analysis, even though we have to tolerate
of magnitude greater than the state of the art; support- a low ingestion rate or a small delay in data availability.
ing this rate requires significant investment, such as Finally, the large event dataset might contain private
additional connectors. SIEM systems also face logistical and sensitive information such as employee Web brows-
challenges in data collection—an event source of inter- ing history. Hence, SIEM systems face the challenge of
est might generate too few events or too many benign securing stored events from unauthorized access.
events that are of no use to SOC operations. For exam-
ple, DNS servers often log only DNS queries and ignore Event Correlation and Analysis
DNS responses. Similarly, laptop and desktop syslogs SIEM systems correlate events across multiple event
might contain too many benign events. Hence, SIEM streams and look for known event patterns to identify

www.computer.org/security 39

j5man.indd 39 9/18/2014 11:55:36 PM

 CSIRTS

attacks and other security-relevant events. For example, effective and efficient decisions, for instance, identi-
they might correlate HTTP proxy and antivirus prod- fying a new attack or deciding which security alerts
uct logs to detect malware downloaded to endpoint to respond to. SIEM systems must develop visualiza-
devices. However, performing correlations and iden- tion techniques that aid humans in gathering infor-
tifying patterns at the scale of large enterprises remain mation from large quantities of data, provide context
challenges. Moreover, SIEM systems must perform information in a timely manner, and work at different
more sophisticated analysis to derive true value from organizational levels, such as system administrator and
the collected events. higher-level management.
Scalable analysis algorithms that handle 1 trillion or
more events per day face significant challenges. First, Toward Addressing the Challenges
identifying attacks from event streams is more art than Multiple SIEM system vendors have offered different
science—no definition of attacks exists, so SOC ana- approaches to improve SIEM systems’ capabilities to
lysts use heuristics derived from past experience to collect, store, and correlate events in large enterprise
identify attacks and other relevant events. Analysis algo- networks; however, progress is necessary to address
rithms should automatically identify patterns of interest scalability issues. For example, because complex
from large event streams, but automating a heuristics- correlation is time consuming, analysts typically avoid
driven process is difficult. creating feature-rich correlation rules that incorporate
Second, even if an algorithm can identify attacks many information sources to capture sophisticated and
today, it might not work tomorrow as adversaries stealthy attacks.
adapt, enterprise networks change, and employees’ To the best of our knowledge, no research directly
behaviors change. Hence, the algorithm must learn addresses SIEM system challenges. However, we
and evolve continuously. believe that advances in many fields of computer
Third, the problem of false positives becomes more science will significantly impact SIEM systems. For
acute as SIEM systems collect more data. Because example, advances in storage systems—especially
benign events outnumber malicious events by orders of nonvolatile memory—will help with storing more
magnitude, an extremely low false-positive rate might event data at lower cost. Similarly, advances in paral-
still produce too many false positives to be usable in lel and distributed computing, especially in big data
practice. Hence, analysis algorithms might not be able analysis, will provide the platform for scalable anal-
to make a scalability–accuracy tradeoff. ysis. For example, a distributed correlation engine
Fourth, even if an analysis algorithm produces no might handle more complex rules than traditional
false positives, it might produce more true positives SIEM systems. There’s also recent work on using big
than SOC analysts can handle. Hence, SIEM sys- data analysis to identify actionable security informa-
tems will have to prioritize the true positives for SOC tion from very large event datasets. Ting-Fang Yen
­analyst consumption. and her colleagues analyze HTTP proxy logs to iden-
Fifth, more events might lead to statistically signifi- tify suspicious host activities—they extract features
cant but ultimately meaningless correlations.3 When from logs, then use clustering to find outlying suspi-
dealing with high-dimensional data, many unrelated cious activities.4
variables can have high correlations, which will manifest There’s a long line of research on alert correlation
as false positives. Hence, analysis approaches should be as a way to increase the features available to make deci-
able to filter out spurious correlations. sions, building on the assumption of the impractical-
Finally, the hardest challenge is inferring human ity of achieving meaningful results on the basis of a
intent from machine logs—analysis algorithms will single event such as a network packet.2,5,6 However,
have to infer attacker and user intent from event streams alert correlation solutions tend to have false correla-
to identify true attacks, as both malicious attacker tions from the large amount of low-quality events that
actions and benign user actions might generate the SIEM systems handle. This has led to research on alert
same event patterns. ­prioritization—that is, identifying higher-quality alerts
that analysts should focus on. Researchers introduced
Visualization multiple alert prioritization approaches, some using
We believe that SIEM systems will never reach the matu- probability theory and Dempster-Shafer theory.7,8
rity level needed to replace human analysts in SOCs. At Big data visualization is a very active research area.9
best, they’ll be tools in analysts’ and network adminis- Data visualization specifically for security has also
trators’ decision-making processes. Hence, SIEM sys- been explored.10 Advances in these two areas will help
tems face the challenge of summarizing analysis results address the big data visualization problem that SIEM
and presenting them so that humans can make more systems face.

40 IEEE Security & Privacy September/October 2014

j5man.indd 40 9/18/2014 11:55:36 PM

 A lthough SIEM systems provide a solid techni-
cal foundation for SOCs, we believe further
advances are needed to create a more adaptive, context-
Sandeep Bhatt is a researcher at Hewlett-Packard Labo-
ratories. His research interests include network data
security analysis, managing end-to-end access control
aware, flexible, holistic, and social solution. This solu- in enterprise networks, and algorithms for parallel
tion should capture organization-specific knowledge computation, network communication, and VLSI lay-
as feedback from the SOC to guide the investigation out. Bhatt received a PhD in computer science from
processes and reduce the volume of events. The goals MIT. Contact him at [email protected].
should be to
Pratyusa K. Manadhata is a researcher at Hewlett-Pack-
■■ give the SOC the freedom to design and adapt its pro- ard Laboratories. His research interests include secu-
cesses per incident, rity and privacy, with an emphasis on big data analysis
■■ manage incidents by collecting all related information for security. Manadhata received a PhD in computer
and communications in one place and help the sys- science from Carnegie Mellon University. He’s a
tem learn from previous incidents (to reduce repeti- member of the ACM, IEEE, and Usenix. Contact him
tive investigations triggered by similar false alarms, for at [email protected].
example), and
■■ automate and customize the repetitive aspects of each Loai Zomlot is a postdoctoral research associate at
role in the SOC team by presenting contextual infor- Hewlett-Packard Laboratories. His research interests
mation to SAs along with the alert (as opposed to the include cybersecurity, with a special focus on intru-
current practice of having SAs manually ferret out rel- sion detection and analysis. Zomlot received a PhD in
evant information from multiple sources). computer science from Kansas State University. Con-
tact him at [email protected].
References
1. R. Agrawal, T. Imielinski, and A. Swami, “Mining Asso- Selected CS articles and columns are also available for free
ciation Rules between Sets of Items in Large Databases,” at http://ComputingNow.computer.org.
Proc. ACM SIGMOD Int’l Conf. Management of Data
(SIGMOD 93), 1993, pp. 207–216.
2. S. Axelsson, “The Base-Rate Fallacy and the Difficulty
of Intrusion Detection,” ACM Trans. Information System
Security, 2000, pp. 186–205.
3. O. Ogas, “Beware the Big Errors of ‘Big Data,’” Wired, 2013;
www.wired.com/opinion/2013/02/big-data-means

On Computing
-big-errors-people.
4. T.-F. Yen et al., “Beehive: Large-Scale Log Analysis for
Detecting Suspicious Activity in Enterprise Networks,”
Proc. 29th Ann. Computer Security Applications Conference podcast
(ACSAC 13), 2013, pp. 199–208. www.computer.org/oncomputing
5. F. Cuppens and A. Miege, “Alert Correlation in a Cooper-
ative Intrusion Detection Framework,” Proc. IEEE Symp.
Security and Privacy, 2002, pp. 202–215.
6. B. Morin et al., “A Logic-Based Model to Support Alert
Correlation in Intrusion Detection,” Information Fusion,
2009, pp. 285–299.
7. L. Zomlot et al., “Prioritizing Intrusion Analysis Using
Dempster-Shafer Theory,” Proc. 4th ACM Workshop Arti-
ficial Intelligence and Security (AISec 11), 2011, pp. 59–70.
8. Y. Zhai et al., “Reasoning about Complementary Intru-
sion Evidence,” Proc. 20th Ann. Computer Security Applica-
tions Conf. (ACSAC 04), 2004, pp. 39–48.
9. S. Liu et al., “A Survey on Information Visualization:
Recent Advances and Challenges,” The Visual Computer,
Springer, 2014, pp. 1–21.
10. R. Marty, Applied Security Visualization, Addison-Wesley,
2009.

www.computer.org/security 41

j5man.indd 41 9/18/2014 11:55:36 PM