See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/326445987

LogLens: A Real-time Log Analysis System

Conference Paper · July 2018

DOI: 10.1109/ICDCS.2018.00105

CITATIONS READS
85 2,756

10 authors, including:

Biplob Debnath Mohiuddin Solaimani

NEC Laboratories America University of Texas at Dallas
44 PUBLICATIONS 1,369 CITATIONS 14 PUBLICATIONS 238 CITATIONS

SEE PROFILE SEE PROFILE

Nipun Arora Cristian Lumezanu

NEC Laboratories America NEC Laboratories America
18 PUBLICATIONS 302 CITATIONS 56 PUBLICATIONS 1,700 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Biplob Debnath on 02 January 2019.

The user has requested enhancement of the downloaded file.

 2018 IEEE 38th International Conference on Distributed Computing Systems

LogLens: A Real-time Log Analysis System

Biplob Debnath∗ , Mohiuddin Solaimani† , Muhammad Ali Gulzar† , Nipun Arora† ,
Cristian Lumezanu∗ , Jianwu Xu∗ , Bo Zong∗ , Hui Zhang§† , Guofei Jiang§† , and Latifur Khan‡
∗ NEC
Laboratories America, Inc., Princeton, New Jersey, USA
‡ CS
Department, The University of Texas at Dallas, USA
§ Ant Financial, Hangzhou, China
Email: [email protected], [email protected], [email protected], [email protected]
{lume,jianwu,bzong}@nec-labs.com, shengchu.zh@antfin.com, [email protected], [email protected]

Abstract—Administrators of most user-facing systems depend formats. As more new devices and data formats enter the
on periodic log data to get an idea of the health and status market (Gartner, Inc. forecasts that 20.4 billion IoT units will
of production applications. Logs report information, which is be in use worldwide by 2020 [15]), it becomes increasingly
crucial to diagnose the root cause of complex problems. In this
difficult for the supervised log analysis tools to keep track and
paper, we present a real-time log analysis system called LogLens
that automates the process of anomaly detection from logs with adapt to new log structures and identify unknown anomalies.
no (or minimal) target system knowledge and user specification. In this paper, we describe LogLens, a log analysis sys-
In LogLens, we employ unsupervised machine learning based tem to automatically detect operational problems from any
techniques to discover patterns in application logs, and then software system logs. Rather than taking the log structure
leverage these patterns along with the real-time log parsing for
as an input, LogLens automatically learns structures from
designing advanced log analytics applications. Compared to the
existing systems which are primarily limited to log indexing and the “correct logs” and generates models that capture normal
search capabilities, LogLens presents an extensible system for system behaviors. It subsequently employs these models to
supporting both stateless and stateful log analysis applications. analyze production logs generated in real-time and detects
Currently, LogLens is running at the core of a commercial log anomalies. Here, we define anomaly as a log or group of
analysis solution handling millions of logs generated from the
logs that do not match the normal system behavior models.
large-scale industrial environments and reported up to 12096x
man-hours reduction in troubleshooting operational problems LogLens requires no (or minimal) user involvement and adapts
compared to the manual approach. automatically to new log formats and patterns as long as users
can provide a set of logs for building models against which
I. I NTRODUCTION anomalies are detected.
Log analysis is the process of transforming raw logs – writ- LogLens classifies anomaly detection algorithms into two
ten records of software systems events – into information that major groups: stateful and stateless. Stateless anomalies arise
helps operators and administrators to solve problems [1, 2]. from analyzing a single log instance, while stateful anomalies
Log analysis is used in a variety of domains such as detecting appear when a combination of multiple logs does not match the
security threats [3, 4, 5], compliance auditing [6], power plant trained model. For example, identifying errors or warnings in
fault detection [7], or data center operations [8, 9, 10, 11, 12]. operational logs do not require keeping state about each log. In
The ability to analyze logs quickly and accurately is critical to contrast, identifying maximum duration violation of a database
reduce system downtime and to detect operational problems transaction requires storing start event time of the transaction
before or while they occur. so that when an end event of the same transaction comes,
A critical aspect of a log that enables fast and accurate anomalies can be detected by calculating the duration of the
analysis is its structure. Recognizing the structure of a log transaction. LogLens presents one exemplary stateless algo-
greatly helps in easy extraction of specific system information, rithm and one exemplary stateful algorithm. The exemplary
such as the type, time of creation, source of a specific stateless algorithm is a log parser, which parses logs using
event, the value of key performance indicators, etc. Without a patterns discovered during system normal runs and reports
known log structure, log analysis becomes a simple keyword- anomalies if streaming logs cannot be parsed using discovered
based text search tool. In fact, most commercial log analytics patterns. This stateless parser can parse logs up to 41x faster
platforms today [13, 14] allow users to directly specify log than the Logstash [16], which is a widely used log parsing
patterns or to generate models based on domain knowledge. tool. The exemplary stateful algorithm discovers relationships
While supervised log analysis can help extracting important among log sequences representing usual operational workflows
insights without ambiguity, it also has several shortcomings: from the system normal runs and reports anomalies in the
a) it is specific to what the user seeks and focuses on known streaming logs. This stateful algorithm can handle heteroge-
errors and b) it cannot easily adapt to new data sources and neous log streams and can automatically discover ID fields to
link multiple logs corresponding to an event.
† Work done while worked at NEC Laboratories America, Inc. To analyze massive volumes of logs with zero-downtime,

2575-8411/18/$31.00 ©2018 IEEE 1052

DOI 10.1109/ICDCS.2018.00105
we deploy LogLens on top of the Spark [17] framework. • Handling data drift. System behavior typically evolves
While Spark provides a low latency and high throughput over time. Hence, log data characteristics and behavior
data processing platform, our experience in building large models may also change. To this end, LogLens periodically
scale log analysis system reveals that Spark lacks several relearns models to adapt to system behavior change.
key features needed to deploy LogLens as a real-time service • Expediting stateful anomaly detection. Real-time anomaly
with zero-downtime. In particular, its immutable broadcasting detection algorithms are generally event-driven. Thus, in the
feature [18] forces us to restart service in order to update the absence of logs, some anomalies cannot be detected in time.
learned models in a running system, thus we can not guarantee LogLens ensures that all anomalies are reported in time by
zero-downtime. Moreover, stateful algorithms need external leveraging an external heartbeat controller which generates
stimuli for efficient memory management and timely anomaly dummy messages periodically.
detection. As a remedy, LogLens introduces a rebroadcasting • Deploying log analysis as a service. We aim to design a
mechanism. In addition, it proposes to add an external heart- system which can handle high volume and high velocity of
beat controller for efficiently managing open states and for the log streams in real-time. However, we want to leverage
immediately reporting anomalies. existing open-source data processing frameworks to min-
The rest of this paper is organized as follows: Section II imize implementation and deployment effort. In addition,
describes our LogLens architecture. Section III describes our we want to guarantee zero-downtime (i.e., no service dis-
stateless log parsing algorithm. Section IV describes our ruption). To this end, LogLens chooses Spark [17] big data
stateful log sequence anomaly detection algorithm. Section V processing framework because of its maturity, huge echo
describes the challenges we faced and solutions adopted for system and community support, and widespread adoption
deploying LogLens as a service. Section VI shows our ex- in the industry and academic realms. However, we find that
perimental results. Section VII describes two case-studies of even Spark (as well as similar frameworks, i.e., Flink [19],
LogLens deployment in solving real-world problems. Finally, Samza [20], etc.) does not have all necessary features to
Section VIII states the conclusion and lesson learned followed deploy a log analysis service (see Section V). Finally,
by a bibliography. LogLens enhances the Spark framework to satisfy our design
goals.
II. L OG L ENS A RCHITECTURE
In this section, we present the system architecture of B. Architectural Components
LogLens and the rationale behind our design choices. Figure 1 illustrates the architecture of LogLens. Now, we
briefly describe each component.
A. Design Goals
Agent is a daemon process which collects heterogeneous logs
The design of LogLens is driven by the following goals: from multiple sources and sends them to the log manager.
• Handling heterogeneous logs. Logs may have a variety
of formats depending on their sources and what they are Log Manager receives logs from agents. It controls incoming
trying to convey. An automated log analyzer should be able log rate and identifies log sources. It forwards incoming logs
to handle any log formats irrespective of its origin. to the parser. It also stores them into the log storage.
• Minimizing human involvement. Ideally, an automated Log Storage is the main storage or archival component. It
log analyzer should work from scratch without any prior organizes logs based on the log source information. Stored
knowledge. For logs from the new sources, it should not logs can be used for building models during log analysis. They
mandate any human involvement. To this end, LogLens can also be used for future log replaying to perform further
leverages unsupervised machine learning based techniques. analysis, or for post-facto querying when troubleshooting
Human interaction is limited to providing “training” logs, operational problems.
which captures “correct” behaviors. LogLens learns various Model Builder generates models for the anomaly detection.
models from this training dataset and uses them later to It takes a set of training logs assuming that they represent
detect anomalies. In addition, LogLens provides options to normal behavior and uses unsupervised machine learning
the users to incorporate their domain knowledge in order to based techniques to build models. To adapt to system behavior
improve the accuracy of the anomaly detection algorithms. change, periodically it collects logs from the log storage for
• Providing a generic system. We aim to design a generic relearning models and stores them on the model storage.
system which captures most real-world use cases and
challenges. To this end, LogLens presents two exemplary Model Storage stores models. All the anomaly detectors read
anomaly detection algorithms. The first algorithm is state- models directly from the model storage.
less, while the second algorithm is stateful. LogLens presents Model Manager retrieves model information from the model
a stateless log parser, which is a core component to design storage and notifies the controller to update a model. LogLens
any log analysis algorithm. Usually, stateful algorithms are supports both automatic and human interaction inside model
more complex and need quite an effort to implement effi- manager. For example, users can configure LogLens to au-
ciently – LogLens presents a log sequence anomaly detector tomatically instruct model builder every midnight to rebuild
to demonstrate various real-world challenges. models using the last seven days logs. In addition, model

1053
 Agent
Log Sequence
Agent Log Log Parser Anomaly Viszualization
Anomaly Detector
Manager (Stateless) Storage Dashboard
Agent (Stateful)
Log

Heartbeat
Model
Controller
Controller

Model
Manager

Log Model Model

Storage Builder Storage

Fig. 1: LogLens architecture showing major components and operational workflows.

manager allows human experts to inspect models and edit them big data processing framework. It uses Kafka [21] for shipping
to incorporate domain knowledge. logs and communicating among different components. For the
Model Controller gets notifications from the model manager storage, it uses Elasticsearch [14] a NoSQL database. Elastic-
and sends control instructions to the anomaly detectors. Mod- search provides a very useful query facility that can be used
els can be added or updated or deleted, and each operation for data exploration. Furthermore, it has close integration with
needs a separate instruction which contains detail information Kibana [22], which provides a tool for building visualization
about the steps that need to be executed. Anomaly detectors front-ends and writing interactive ad-hoc queries.
read control instructions and take action accordingly. Now, we describe our exemplary anomaly detection al-
gorithms in Section III and Section IV, and deployment
Log Parser takes streaming logs and log-pattern model from challenges and solutions in Section V
the model manager as input. It parses logs using patterns
and forwards them to the log sequence anomaly detector. III. S TATELESS : L OG PARSER
All unparsed logs are reported as anomalies and presented
to the user for further review. Log parser is an example For an automated log analysis system, a core step is to
implementation of the stateless anomaly detection algorithm. parse raw logs and make them structured so that various
We describe it in detail in Section III. log analysis tasks could be carried out by leveraging the
structured form of the raw logs. LogLens parses logs using
Log Sequence Anomaly Detector detects anomalous log patterns learned from the systems normal runs. Here, we define
sequence of an event (or transaction), which consists of a pattern as a GROK expression [23]. For example, for the log
sequence of actions and each action is represented by a log. It “Connect DB 127.0.0.1 user abc123”, one of the matching
is a stateful algorithm which detects malfunctioned events by GROK patterns is “%{WORD:Action} DB %{IP:Server} user
analyzing abnormal log sequences based on an automata-based %{NOTSPACE:UserName}” and after parsing LogLens pro-
model. We describe it in detail in Section IV. duces {“Action”: “Connect”, “Server”: “127.0.0.1”, “User-
Heartbeat Controller periodically sends heartbeat (i.e., echo Name”:“abc123”} as a parsing output in JSON format. Parsed
or dummy) messages to the log sequence anomaly detector. outputs can be used as a building block for designing various
These messages aid to report anomalies in time and to identify log analysis features. For example, our stateful algorithm (see
open states in a transaction. Section IV) uses them to detect log sequence violations.
Anomaly Storage stores all anomalies for human validation. Challenges. Automatically parsing heterogeneous logs with-
Each anomaly has a type, severity, reason, timestamp, asso- out human involvement is a non-trivial task. LogLens parses
ciates logs, etc. logs in two phases: 1) it discovers a set of GROK patterns
from a set of logs representing system normal runs and 2) it
Visualization Dashboard provides a graphical user interface parses logs using these GROK patterns.
and dashboard to the end users. It combines information from Existing log analysis tools either use predefined regular
log storage, model storage, and anomaly storage to present expressions (RegEx) or source-code level information for log
anomalies to the users. Users can easily view anomalies and parsing [11, 16, 24]. Thus, these tools are supervised and
take actions to rebuild or edit models. It also allows users to need human involvement – they cannot be used for the first
run complex analysis by issuing ad-hoc queries. phase. Our earlier work, LogMine [25], shows how to discover
Most components described above can be implemented us- patterns without any human involvement by clustering similar
ing many different open-source products. LogLens uses Spark logs. LogMine uses tokenized logs and datatypes of the tokens

1054
during the similarity computation step. However, identifying LogLens allows users to specify formats to identify timestamp
some tokens, especially timestamp identification is a very related tokens. It uses Java’s SimpleDateFormat [27] notation
challenging task. In addition, LogMine may fail to meet user to specify a timestamp format. However, if users do not specify
needs as it is very hard to automatically infer semantics of a any format, LogLens identifies timestamps based on a set of
field in the GROK pattern. predefined formats (for example, MM/dd HH:mm:ss, dd/MM
In the second phase, we need a tool to parse incoming logs. HH:mm:ss:SSS, yyyy/MM/dd HH:mm:ss.SSS etc.). Users can
We can use Logstash [16], an industrial-strength open-source also add new formats in the predefined list. The worst case
log parsing tool, which can parse logs using GROK patterns. time complexity of identifying timestamp is O(k), where k
However, we find that Logstash suffers from two severe is the total number predefined formats or the total number of
scalability problems: 1) it cannot handle a large number of user-specified formats.
patterns and 2) it consumes huge memory (see Section VI-A). Solution. LogLens uses the following two optimizations to
Since LogLens discovers patterns with no (or minimal) human quickly identify tokens related to the timestamp formats:
involvement, it can generate a huge number of patterns which
• Caching matched formats. LogLens maintains a cache to
is very problematic for the Logstash.
track the matched formats. Caching reduces the amortized
Solution. LogLens provides an efficient solution for identify- time complexity to O(1). To identify timestamp related
ing timestamps and to meet user expectation it allows users tokens in a log, first, LogLens finds if there is a cache hit.
to edit/modify automatically generated GROK patterns. For In case of a cache miss, LogLens checks the non-cached
the fast parsing, LogLens transforms both logs and patterns formats and if a match found, then the corresponding format
into their underlying datatypes and builds an index for quickly is added to the cache. This simple caching strategy works
finding the log-to-GROK mapping. Now, we describe log well in practice as logs from the same (or similar) sources
parsing workflow in detail. use same formats, and every source uses only a few different
formats to record timestamps.
A. Model Building • Filtering. LogLens maintains a set of keywords based on
1) Tokenization: LogLens preprocesses a log by splitting the most common form of specifying month (i.e., jan-
it into individual units called tokens. Splitting is done based dec, january-december, 01-12, 1-9) , day (i.e., 01-31), and
on a set of delimiters. The default delimiter set consists of hour (i.e., 00-59), day of the week (i.e., mon-sun, monday-
white space characters (i.e., space, tab, etc.). LogLens also sunday), etc. It uses these keywords to filter out tokens
allows users to provide delimiters to overwrite the default which cannot be related to a timestamp. If a token cannot be
delimiters in order to meet their needs. In addition, users can filtered, then only LogLens checks the predefined formats.
provide regular expression (RegEx) based rules to split a single
3) Pattern Discovery By Clustering Similar Logs: In this
token into multiple sub-tokens. For example, to split the token
step, LogLens clusters preprocessed logs based on a similarity
“123KB” into sub-tokens “123” and “KB”, user can provide
distance using LogMine [25] algorithm. All logs within a
the following RegEx rule: “[0-9]+KB” =⇒ “[0-9]+ KB”.
cluster are merged together to generate one final pattern in
2) Datatype Identification: During this step, for every token
the form of a GROK expression. LogLens assigns a field
LogLens identifies its datatype based on the RegEx rules.
ID for each field. The field ID consists of two parts: 1) the
Table I shows the sample RegEx rules for identifying different
ID of the log pattern that this field belongs to and 2) the
datatypes in LogLens.
sequence number of this field compared to other fields in the
Datatype Regular Expression (RegEx) Syntax same pattern. The log format pattern IDs can be assigned
WORD [a-zA-Z]+ with the integer number 1, 2, 3, ... m for a log pattern
NUMBER -?[0-9]+(.[0-9]+)?
IP [0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3} set of size m. The field sequence order can be assigned
NOTSPACE \S+ with the integer number 1, 2, 3, ... k for a log pattern
DATETIME [0-9]{4}/[0-9]{2}/[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}
ANYDATA .*
with k variable fields. For example, for the log “2016/02/23
09:00:31 127.0.0.1 login user1” the corresponding generated
TABLE I: Syntax for various data types. Notation is adopted GROK pattern would be “%{DATETIME:P1F1} %{IP:P1F2}
from the Java Pattern API [26]. %{WORD:P1F3} user1”.
4) Incorporating Domain Knowledge: LogLens automati-
Challenge. LogLens identifies timestamps and unifies them cally generates patterns, therefore it may not always meet
into a single format “yyyy/MM/dd HH:mm:ss.SSS” corre- user needs. In addition, users may want to generate patterns
sponding to the DATETIME datatype. However, we find from one system and later want to apply them to a different
that it is a very cumbersome process due to the hetero- system with some minor modifications. A user may even
geneity of timestamp formats used in various logs. For ex- want to delete some patterns or add new patterns or edit
ample, timestamp “2016/02/23 09:00:31” can be expressed datatypes. To solve these issues, LogLens allows users to
in “2016/23/02 09:00:31” or “2016/23/02 09:00:31.000” or edit automatically generated patterns. It supports the following
“Feb 23, 2016 09:00:31” or “2016 Feb 23 09:00:31” or editing operations:
“02/23/2016 09:00:31” or “02-23-2016 09:00:31” and so on. • LogLens allows users to add the semantic meaning of a field

1055
 by renaming its generic name. For example, LogLens may (in terms of number of tokens). If no candidate pattern
assign “P1F1” as a generic field name for the “logTime” is found, then the candidate-pattern-group is set to empty.
field, thus it may be difficult for users to interpret the Next, LogLens adds this group in a hash index using log-
parsed output. By renaming “P1F1” to “logTime”, users can signature as the “key”, and candidate-pattern-group as the
easily fix this problem. To ease renaming effort, LogLens “value”. Finally, it follows Step 3.
uses a heuristic based approach to leverage commonly 3) Scanning the candidate-pattern-group. If a candidate-
used patterns found in the logs. For example, LogLens pattern-group is found, LogLens scans all patterns in that
automatically renames “PDU = %{NUMBER:P1F1}” as group until the input log is parsed. If an input log cannot be
“PDU = %{NUMBER:PDU}”. If none of the heuristics parsed or group has no patterns (i.e., empty), then LogLens
matches, then only LogLens assigns a generic name. reports it as an anomaly.
• LogLens allows users to specialize a field. For example, a
Pattern-Signature Generation. LogLens generates a pattern-
user can specialize “%{IP:P1F2}” by replacing it with the
signature from each GROK pattern as follows. First, it
fixed value “127.0.0.1”.
splits a pattern into various tokens separated by white space
• LogLens allows users to generalize a specific token
characters. Next, it replaces every token by its datatype.
value. For example, a user can generalize “user1” to
For example, the token “%{DATETIME:P1F1}” is replaced
“%{NOTSPACE:userName}” in order to convert it into a
by its datatype “DATETIME”. If datatype is not present
variable field.
in the token, then LogLens finds out the datatype of the
• LogLens allows users to edit datatype definition to include
token’s present value. For example, the token “user1” is
multiple tokens under one field. To support this feature, it
replaced by “NOTSPACE” by using the RegEx rule defined
introduces the ANYDATA (i.e., wildcard) datatype, which
in Table I. Thus, the pattern-signature of the GROK pat-
is defined in Table I.
tern “%{DATETIME:P1F1} %{IP:P1F2} %{WORD:P1F3}
user1” would be “DATETIME IP WORD NOTSPACE”.
B. Parsing Logs and Anomaly Detection
LogLens uses patterns discovered during modeling stage for How to compare log-signature with pattern-signature? If
parsing logs. If a log does not match with any patterns, then a log-signature is parsed by a pattern-signature, then corre-
it is reported as an anomaly. sponding GROK pattern is added to the candidate-pattern-
group. There are two cases to consider for the pattern-
Problem Definition. Log parsing problem using a set of signature: without and with the ANYDATA datatype (i.e.,
patterns can be formalized as follows: given a set of m wildcard). The first case (i.e., without) is easy to handle,
GROK patterns and a set of n logs, find out the log-to-pattern while the second case is challenging due to the variability
mappings. A naı̈ve solution scans all m patterns to find a match arising from the presence of wildcard. LogLens solves this
for every log. This simple algorithm needs on the average m 2 problem with a dynamic programming algorithm. It can be
comparisons for the matched logs, while for the unmatched formally defined as follows: given a log-signature of length
logs it incurs m comparisons. So, the overall time complexity r tokens, L =< l1 , l2 , ..., lr > and a pattern-signature of
is O(mn). LogLens aims to reduce the number of comparisons length s tokens, P =< p1 , p2 , ..., ps >, we have to find
to O(1), thus the overall time complexity reduces to O(n). out if L can be matched by P . Let us define T [i, j] to the
Solution Sketch. LogLens leverages the fact that logs and pat- boolean value indicating whether < l1 , l2 , ..., li > is parsed by
terns have the common underlying datatypes representing their < p1 , p2 , ..., pj > or not. This matching problem has optimal
structures, thus it can build an index based on these structures substructure, which gives the following recursive formula:
to quickly find the log-to-pattern mapping. LogLens maintains ⎧
⎨ true if i = 0 and j = 0
an index in order to reduce the number of comparisons by T [i, j] = T [i − 1, j − 1] if li = pj or isCovered(li , pj )
using the following three steps: ⎩ T [i − 1, j]  T [i, j − 1] if pj = ∗
1) Finding candidate-pattern-group. To parse a log, Here, isCovered(li , pj ) is a function, which returns true if
LogLens first generates a log-signature by concatenating the RegEx definition corresponding to li ’s datatype is covered
the datatypes of all its tokens. For example, for the by the RegEx definition of the pj ’s datatype. For example,
log “2016/02/23 09:00:31.000 127.0.0.1 login user1” the isCovered(“WORD”, “NOTSPACE”) returns true. In contrast,
corresponding log-signature would be “DATETIME IP isCovered(“NOTSPACE”, “WORD”) returns false. Based on
WORD NOTSPACE”. Next, LogLens finds out if there is a the above formulation, LogLens uses dynamic programming
candidate-pattern-group which can parse the log-signature. to compute the solution in a bottom-up fashion as outlined in
2) Building candidate-pattern-group. If no group found, Algorithm 1. If T [r, s] is true, then LogLens adds the GROK
first LogLens builds a candidate-pattern-group by compar- pattern corresponding to P in the candidate-pattern-group.
ing an input logs log-signature with all GROK m patterns
using their pattern-signatures (explained later) to find IV. S TATEFUL : L OG S EQUENCE A NOMALY D ETECTOR
out all potential candidate patterns and put all candidate Log sequence anomaly detector detects abnormal log se-
patterns in one group. In a group, patterns are sorted in quence in an event (or transaction). Here, we define an event
the ascending order of datatype’s generality and length as follows: an event is an independent operational work unit of

1056
Algorithm 1 Dynamic Programming Algorithm
procedure IS M ATCHED
Input: String logSignature, String patternSignature
Output: boolean (i.e., true/false)
String L[] = logSignature.split(“ ”);
String P[] = patternSignature.split(“ ”);
boolean T[][] = new boolean[L.length+1][P.length+1];
T[0][0] = true;
for (int i = 1; i < T.length; i++) do
for (int j = 1; j < T[0].length; j++) do
if (L[i-1].equals(P[j-1])) then Fig. 2: Sample event trace logs.
T[i][j] = T[i-1][j-1];
else if (P[j-1].equals(“ANYDATA”)) then
 Handling wildcard case
T[i][j] = T[i-1][j]  T[i][j-1];
else if (isCovered(L[i-1], P[j-1])) then
 Is log-token covered by pattern-token?
T[i][j] = T[i-1][j-1];
end if
end for Fig. 3: Sample automaton for an event from the logs of Figure
end for 2. It has the rule of min/max occurrence of each state s and
return T[L.length][P.length]; min/max time duration of an event. Each state corresponds to
end procedure a log in that event.

phases: learning and detection. During the learning phase,

a business process with a finite action sequence. For example, it builds a model that captures the normal behavior of an
cloud data center provides users to access, instantiate virtual event. First, it discovers event ID Fields automatically from
machines, assign resources, and so on. It executes each of the heterogeneous logs by leveraging the fact that log parser
the above tasks by coordinating multiple internal services has already identified the log format and parsed logs into
distributed on different servers. It generates logs for each of the various fields. Next, it builds automata which have rules that
action (i.e., start VMs, contact scheduler, resource manager, represent the normal event log sequences. For example, Figure
hypervisors, etc.) forming an event. Thus, each event has logs 2 shows some logs representing two events with event ID Field
from multiple sources following a sequence. The malfunc- discovered by LogLens. Figure 3 shows corresponding au-
tioning event follows unusual/deviated action sequence, which tomata with discovered rules, where each state corresponds to
may lead to system failure. However, traditional stateless a log in an event. During detection phase, LogLens uses this
anomaly detection techniques dealing with individual logs automatically discovered automata model to detect anomalies.
do not catch these failures because individual logs may not Now, we briefly describe these two phases.
be anomalous although altogether they follow an abnormal
sequence. This requires a stateful algorithm to dig abnormal A. Model Building
event log sequences by storing all intermediate log sequence 1) Automatic Event ID Field Discovery: Log Parser (de-
information in memory/state. scribed in Section III) parses logs and sends them to the log
Challenge. Logs in an event may not be always homogeneous. sequence anomaly detector. Each parsed log has a log pattern
Thus, detecting anomalous log sequence from incoming logs and a field set. LogLens discovers a unique ID Field from these
is a challenging problem as it requires to discover events and parsed logs in an event by leveraging the fact that ID appears
to preserve log sequence information (i.e., state) in memory. the same in multiple logs in an event. LogLens uses a variant
Most of the log sequence anomaly detectors [11, 28, 29, 30, of the Apriori based [32] technique, however the key challenge
31] are supervised as they need human input for discovering for LogLens is to discover events from a large volume of logs
events and do not work for heterogeneous logs. Some research with varying formats. LogLens has following two main steps:
works use unsupervised learning [4, 5, 10], but they are not • Building a reverse index. LogLens builds a reverse index
domain-agnostic to handle heterogeneous logs. Here, our goal of log fields based on their field content. First, it extracts all
is to design a generic unsupervised algorithm for handling field contents from a parsed log. Next, it builds a reverse
heterogeneous logs. index table. Each field content is a key and a list of logs
Solution. LogLens describes a log sequence based anomaly with (log pattern, field) pair as a value in the table.
• ID Field discovery. LogLens discovers ID Field for all
detection algorithm that discovers event automatically using a
finite state automaton (FSA) based model. LogLens has two possible log patterns by scanning the reverse index. For each

1057
 possible event ID content, it builds a list of (log pattern, 1) it introduces a downtime of several seconds, if not minutes,
field) pair for all logs that have this ID content. This gives depending on the size of the cluster; 2) restarting the cluster
multiple lists and LogLens builds a set of unique lists. If any requires rescheduling and redistribution of data and memory
list covers all log patterns discovered in the training logs, leading to significant decrease in the throughput of the cluster;
then LogLens assigns it to an event ID Field. and 3) if a stateful Spark streaming service is terminated, all
2) Event Automata Modeling: In this step, LogLens pro- the state data is lost and losing states can have significant
files automata with rules from logs using ID Fields. It scans impact on the efficacy of the anomaly detection algorithms.
through each log and extracts its ID Field and content. To eliminate any possibility of downtime or loss of state, the
LogLens also keeps track of the log arrival time. For an model update mechanism should meet at least the following
ID Field content, it keeps a sorted list of log patterns with their two requirements: 1) service must be up, and running all the
fields. Finally, it merges them and builds automata with rules. time and 2) states must be preserved during model updates.
An automaton is built with states. Each log pattern with its Solution. In LogLens, to update a broadcast variable (BV) at
ID Field is a state which stores log arrival time, the number of runtime, we modify Spark internals with minimum possible
occurrences, etc. Each automaton has a begin, an end, and mul- changes. Our solution is capable of rebroadcasting the im-
tiple intermediate states. LogLens also tracks the occurrence of mutable BVs at runtime without job termination. BVs is a
the intermediate states and the duration between the begin and serializable data object that is a virtual data block containing
the end state. After building automata, LogLens profiles the a reference to the actual disk block where the variable resides.
minimum and maximum of those statistics (min/max duration When a BV is used in a Spark program, it is shipped to
of an event, min/max occurrence of the intermediate states, each individual worker. During execution, whenever a worker
etc.) and uses them as rules for detecting anomalies. requests the value of a BV using getValue() method, Spark
first looks into the local data-block-cache of the worker for the
B. Anomaly Detection
variable. If there is a cache miss, it sends a pull request to the
LogLens collects incoming logs in real-time. First, it extracts driver (where the variable is initially stored) to get the value
log pattern and ID from each log. It groups all logs that have a over the network. Once this variable is received, it is stored
common Field ID. After that, it sorts logs in each group based on the local-disk-block-cache of that worker. From now and
on their arrival time – this gives incoming log sequence in an so on, this cached value of the variable will be used whenever
event. Next, it scans logs in each group and validates them a getValue() method is called.
against the automata rules discovered during model learning. To rebroadcast a BV which already resides in the local-
Logs in a sequence will be flagged as anomalies if they violate disk-block-cache of individual workers, LogLens invalidates
any of these rules. Table II shows various anomaly types all locally cached values. Thus, whenever getValue() method
reported by LogLens. is called for that BV, a pull request is made to the driver. At
Type Anomaly the driver, when a pull request is received rather than handing
1 Missing begin/end state over the old value, the driver sends the updated value. The
2 Missing intermediate states worker then receives the updated value and stores it in the
Min/Max occurrence violation
3
of the intermediate stats
local cache. From now and so on, the newly fetched local
Min/Max time duration violation copy of the BV will be used.
4
in between the begin state and the end state Whenever a new model is issued from the model manager, it
TABLE II: Sample log sequence anomalies. is read from the model storage and enrolled into a queue. The
scheduler then waits for the current job to finish. LogLens’s
dynamic model update implementation communicates with the
V. LogLens AS A S ERVICE : C HALLENGES AND S OLUTIONS block manager of each worker as well as the driver. It also
In this section, we highlight two key real-world deployment tracks all BV identifiers to maintain the same ID for the
challenges that we encountered when implementing LogLens updated BV which is otherwise incremented at each update.
as a service using Spark [17]. We believe that these challenges This allows workers to retrieve the original BV after cache
and our proposed generic solutions will offer insights for invalidation. Furthermore, LogLens’s implements a thread-safe
building similar services in the near future. queuing mechanism to avoid any race conditions due to the
extreme parallelization of the Spark jobs.
A. Supporting Dynamic Model Updates Spark data processing is a queue-based execution of the
Challenges. Spark’s data parallel execution model uses broad- data received in every micro-batch. In LogLens, model update
cast variables to load models and distribute data to all workers. operation runs between these micro-batches in a serialized
However, broadcast variables have been designed to be im- lock process. The model data is loaded into memory, and
mutable and can only be updated before data stream processing an in-memory copy operation loads the data to the BV. The
is started. The only possible way to update a model in Spark is execution proceeds as normal and whenever the broadcast
to re-initialize and re-broadcast the model data to all workers. value is required, workers fetch a fresh copy from the master.
Unfortunately, this process can lead to drastic consequences: The only blocking operation is the in-memory copy operation,

1058
and hence the overhead is directly dependent on the size of the state object. This method returns the reference to the state-
the model. In practice, we find that this overhead is negligible map object where all the states of that partition are stored.
and it does not incur any slow-down on LogLens. For anomaly detection, this state-map is enumerated to find
the states that are open and expired with respect to the current
B. Implementing Stateful Algorithms Efficiently log time. Although LogLens does not have the key to an open
Expedited Anomaly Detection. LogLens focuses on the real- state, it can still access that state and reports anomalies which
time anomaly detection, thus it is essential to report anomalies would otherwise go entirely undetected. However, because of
as soon as they occur. At the same time to allow for scalable event-driven nature of the Spark’s stream processing, LogLens
and fast execution, LogLens uses data-parallel algorithms to still needs a trigger on all partitions to handle the infrequent
distribute the processing workload of incoming logs across log arrival scenario.
worker nodes. Data partitioning logic is only constrained
Solution. As a remedy, LogLens also leverages the external
by grouping together logs which have an inherent causal
heartbeat controller and a custom partitioner. LogLens peri-
dependency on each other (i.e., same model, source, etc.) –
odically receives heartbeat messages from this controller to
this allows LogLens to optimize performance and to avoid
trigger the expired state detection procedure. This external
performance bottlenecks as much as possible.
message is sent to the same data channel (where logs arrive)
In a stateful anomaly detection, each log is independent of
with a specific tag to indicate that it is a heartbeat message. If
the other, thus if a log comes, then anomalies can be reported
such a message is observed in the program logic, the custom
to the user. However, several real-world issues are potentially
partitioner kicks in and broadcasts the same heartbeat message
problematic especially in the case of stateful anomalies, which
to all partitions. Whenever a heartbeat message is received,
depend on the previous states. Two of these issues are:
an anomaly detection algorithm iterates over its states to
1) What if a transaction fails and no log comes at all from a detect anomalies and clean up expired states. This procedure
source or for a particular key or pattern of the model? Es- is performed at all the partitions on every worker since the
sentially, the saved state is already “anomalous”, but cannot heartbeat message is duplicated and broadcast to each partition
be reported since we have no concrete log as evidence. In on the data channel.
this case, the anomaly would never be reported.
2) Similarly if logs of certain automata are coming very
infrequently (several hours apart). This could be because VI. E XPERIMENTAL R ESULTS
of overload in the target system. In such a scenario,
the anomaly may not be reported immediately for any The goal of this section is to show experimental results to
countermeasures to be taken. evaluate the functionality and effectiveness of LogLens.
Traditional timeout based approaches cannot be used as they Dataset. We use six different datasets covering various data-
use system time, which can be very different from “log time”. center operations for evaluation as shown in Table III. In the
The log timestamps may come faster or slower than the actual table, we have proprietary dataset D1 of trace log of a data
time progress within the LogLens system. Hence only the log center (Figure 2 shows sample logs), a synthetic dataset D2,
rate of embedded timestamps within the logs can be used to storage server based dataset D3, OpenStack based dataset D4
predict timestamps in the absence of logs. Furthermore, the for infrastructure as a service deployment, PCAP based dataset
key based mapping of states only allows similar keys to access D5, and proprietary D6 dataset covering network operations.
or modify the state. Even if somehow LogLens receives an We simulate these datasets as streams in our LogLens system.
event that informs the program logic to flush the unnecessary
states, there is currently no way to access the states without Total logs
Dataset Type
their keys. Training Testing
Solution. To allow for expedited real-time anomaly detection, D1 Trace log 16,000 16,000
LogLens uses an external heartbeat controller. This controller D2 Synthetic 18,000 18,000
generates a heartbeat message for every log source and period- D3 Storage Server 792,176 NA
ically sends it to the anomaly detectors if the corresponding log D4 OpenStack [33] 400,000 NA
agent is still active. The heartbeat message is embedded with a D5 PCAP [34] 246,500 NA
timestamp based on the last log observed and the rate of logs D6 Network 1,000,000 NA
from that source. Hence in the absence of logs, the heartbeat
message provides the current time of the target systems and TABLE III: Evaluation Dataset.
allows LogLens to proceed with the anomaly detection.
Efficient State Management. To enable efficient memory Experimental setup. We perform our tests on a Spark cluster
management of the open states, LogLens extends the Spark with Spark Streaming. Our cluster has one master and eight
API (v1.6.1) to expose the reference of the state in a partition worker nodes. We use Spark version 1.6.1 with Kafka version
to the program logic. Within the program logic, the state-map 0.9.0.1. For replaying log data, we have developed an agent,
can be accessed by calling the getParentStateMap() method on which emulates the log streaming behavior.

1059
A. Log Parser 30
Ground Truth

Anomaly Count
Fast Timestamp Identification. LogLens has 89 predefined 21 21 LogLens
20
timestamp formats in the knowledge-base. From our experi-
13 13
ments using datasets in Table III, we find that by combining
both caching and filtering, LogLens can detect timestamps up 10
to 22x faster than the linear scan-based solution – 19.4x is
contributed by caching, and the rest is contributed by filtering. 0

2
D

D
Fast Log Parsing. We compare LogLens against
Fig. 4: Log sequence anomaly detector results accuracy.
Logstash [16], a popular open-source log parsing tool,
to show its efficiency. For these experiments, we use D3, D4,
D5, and D6 datasets which use the same set of logs in both anomaly immediately. With HB controller, we expect to re-
training and testing phases for sanity checking – a correct port more anomalies as soon as they occur. Figure 5 shows
parser does not produce any anomalies for these datasets. performance result of our HB controller. For a certain time
Using LogMine [25] algorithm, first, we generate a set of period, we run our anomaly detector on D1 and D2. If we
GROK patterns from the training logs; next, we parse testing do not use HB controller, we detect 20 anomalies for D1 and
logs using these patterns; and we expect that every testing 10 anomalies for D2. However, when with HB controller, we
log matches a GROK pattern as testing logs are same as the detect 21 anomalies for D1 and 13 anomalies for D2, and all
training logs. Table IV shows that LogLens runs up to 41x of these extra anomalies are related to the missing end states.
times faster than Logstash (v5.3.0), and handles large number These results demonstrate that HB controller is effective in
of patterns. Both LogLens and Logstash parse all training immediately reporting anomalies.
logs and produce same parsing results. For the D4 and D6
datasets, Logstash did not generate any output even running 30
for more than 48 hours, and eventually we stopped it. The Anomaly Count
Ground Truth
21 21 LogLens w/o HB
main reason is as follows: D4 and D6 datasets produce 3234 20
20 LogLens w/ HB
and 2012 patterns, respectively and Logstash is not suitable
13 13
for handling such large pattern-sets. 10
10
Running Time
Dataset Total Patterns
LogLens Logstash Improvement 0
D3 301 109 sec 4550 sec 4074.31%
1

2
D

D
D4 3234 72 sec NA NA
D5 243 34 sec 588 sec 1629.41% Fig. 5: Anomaly detection with and without heartbeats
D6 2012 170 sec NA NA
Model Controller. LogLens provides a key feature like the
TABLE IV: Results:LogLens vs. Logstash.
model update as a service. It can add, update, or delete
models without restarting the running service. The goal of this
experiment is to show that the number of anomaly changes
B. Log Sequence Anomaly Detector
after the model update in order to verify model controller’s
The effectiveness of LogLens in easing human log analysis functionality. Now, we run two set of experiments. First, we
burden requires detecting anomaly accurately. We also need build models using training logs of D1 and D2. D1’s model
to verify that heartbeat controller helps to report anomalies has two automata, while D2 has three automata. Using these
in real-time, and model controller instantly reflects system models, we detect 21 anomalies for D1, and 13 anomalies for
behavior changes after a model update operation. Since log D2. Next, we modify both models by deleting an automaton
sequence anomaly detector uses the output from the log parser, from them, update models through the model controller with-
consequentially our evaluation also demonstrates log parser’s out service interruption, and rerun testing. Table V shows that
efficacy in building advanced log analytics applications. deleting an automaton reduces the number of anomalies from
21 to 13 for D1, and 13 to 9 for D2. This behavior matches
Accuracy. We use D1 and D2 to evaluate the accuracy of
with our intuition as the second set of experiments will
the log sequence anomaly detector because we have ground
produce fewer anomalies because they have fewer automata
truth form them. Figure 4 shows that D1 has originally 21
rules. Therefore, these two set of experiments validate the
anomalous sequences, and our detector identifies all of them;
functionality of the model update operation.
D2 has originally 13 anomalous sequences, and our detector
identifies all of them (red bar). Thus, for both datasets, we get VII. R EAL - WORLD C ASE S TUDIES
100% recall. A. Analyzing Custom Application Logs
Heartbeat Controller. In LogLens, heartbeat (HB) controller In this case-study, users want to analyze logs from a custom
controls open states and helps to report missing end state application. These logs record SQL queries issued in the

1060
 Total Anomaly
Total Anomaly
Dataset Automata Count
Automata Count
(after delete) (after delete)
D1 2 21 1 13
D2 3 13 2 9

TABLE V: Anomaly detection using model updates.

system to get various statistics used by other applications.

Table VI shows a set of sample logs. These logs are extremely
complex, and it took one week for the users to manually
generate patterns to parse all logs in order to understand log Fig. 7: SS7 attacks revealed by LogLens anomalies
characteristics. In contrast, LogLens generated 367 patterns in
50 seconds, which reduces humans effort by 12096x and aids
logs of other colors are normal ones. As shown in these
them to instantly focus on the log analysis tasks.
examples, normal logs follow SS7 protocol with a sequen-
(0):GetFormControl():2[25 21:39:21] SQL SELECT TABLE: tblFormControl WHERE: tblFormControl.oFCID=’6a602aaa-9afd-4e2c-95e9-ee900dde4b50’
tial pattern “InvokePurgeMs” → “InvokeSendAuthentication-
Info” → “InvokeUpdateLocation”. Unlike normal logs, the
(0): GetObjects():2[25 21:39:21] SQL SELECT TABLE: tblContent WHERE: oPID=’ad1aa290-01ae-4edd-989c-1cee2ba63707’
AND ((((((oID IN (SELECT oID FROM tblFormData WHERE oFORMINSTID=tblContent.oID AND oFCID=’6a602aaa-9afd-4e2c-95e9-ee900dde4b50’
AND ((tblFormData.tValue IS NOT NULL AND tblFormData.tValue ¿ ’1799-01-01T00:00:00.000’ AND tblFormData.tValue ¡ ’2200-01-01T00:00:00.000’
)))))AND (((nType!=15 OR oID IN (SELECT oFORMINSTID FROM tblFormInstance WHERE tblFormInstance.oFORMID=
’3ebee358-2087-43d4-908b-df9ed04e74cc’)) AND (nType!=14 OR tblContent.oID=’3ebee358-2087-43d4-908b-df9ed04e74cc’))) AND ((oID IN
(SELECT oID FROM tblFormData WHERE oFORMINSTID=tblContent.oID AND oFCID=’7e68b547-0869-4a56-a664-26b32d0b5804’ AND traces left by spoofing attacks form a different sequence
“InvokePurgeMs” → “InvokeSendAuthenticationInfo” with-
((tblFormData.tValue¡=’2017-10-26T03:59:59.000’ OR tblFormData.tValue IS NULL))))) AND ( (oID IN (SELECT oID
FROM tblFormData WHERE oFORMINSTID=tblContent.oID AND oFCID=’e28c6d82-532d-4618-a0a8-d62a15e00731’ AND
(tblFormData.sValue=N’dadf4506-2995-42c4-8616-cb43786fa382’)))) AND ((oID IN (SELECT oID FROM tblFormData WHERE oFORMINSTID=tblContent.oID
AND oFCID=’2a004b8d-16ef-4973-8ec8-be7db392e436’ AND ((tblFormData.sValue¡¿N’Y’ OR tblFormData.sValue IS NULL)))))))) AND (nType!=15 OR oID IN
(SELECT oFORMINSTID FROM tblFormInstance WHERE oFORMID=’3ebee358-2087-43d4-908b-df9ed04e74cc’)) AND 1=1 AND (nType=15)) AND(oID IN
(SELECT oID FROM tblPerm WHERE (oGrantID=’dadf4506-2995-42c4-8616-cb43786fa382’ OR oGrantID=’[Authenticated]’ OR oGrantID=’[Anonymous]’
out ending “InvokeUpdateLocation” because attackers only
OR oGrantID IN (SELECT oParent FROM tblMembership WHERE oChild=’dadf4506-2995-42c4-8616-cb43786fa382’))
AND fRead=1) ) AND (nSubType!=2 AND nSubType!=1 AND nSubType!=4 AND nSubType!=5) AND (nType!=15 OR nVersion!=0) want to guess credentials of the legitimate users without
(0): GetObjects():2[25 21:39:21] SQL SELECT TABLE: tblFormData WHERE: oFORMINSTID = ’418f38ce-a35e-47db-8e1c-88fc7eb09de3’ AND oFCID
IN (’fe53e626-13ae-4206-8bc7-178cbc69b866’, ’6a602aaa-9afd-4e2c-95e9-ee900dde4b50’, ’1bfb5785-4f29-488b-8d09-c42faef48fee’,
’611e6c07-c8ba-44c5-b745-485e9faddcb4’, ’3d8dfd3d-2c62-4c19-8cb5-a3bec8bf729b’, ’7e68b547-0869-4a56-a664-26b32d0b5804’)
finishing up the full protocol. Anomaly clusters are formed
because attackers launched intensive spoofing attacks, which
TABLE VI: Custom application logs sample result in a large number of log sequence violations in a short
period of time.
It is very difficult even for the domain experts to manually
B. Discovering Security Attacks
check all 1 million SS7 logs for investigating security attacks.
In this case-study, we report spoofing attacks [35] dis- In this empirical study, it took them 2 days to perform such
covered by the log sequence anomaly detector of LogLens investigation. In contrast, LogLens reported abnormal logs that
from a set of Signaling System No. 7 (SS7) logs [36]. This were related to the spoofing attacks in 5 minutes without any
dataset contains 2.7 million SS7 logs spanning three hours domain knowledge, thus it saved 576x of man-hours.
from 2016/05/09 10:00:00 to 2016/05/09 13:00:00. For the
model learning, users take the first two hours of SS7 logs as VIII. C ONCLUSION AND L ESSON L EARNED
training data. With this learned model, they discover anomalies LogLens provides a blueprint for implementing a real-
in the remaining one hour of SS7 logs. time log analysis system by leveraging unsupervised machine
learning based techniques. It can detect anomalies with no
(or minimal) human involvement and can easily adapt to
the system behavior change. It deploys log analysis as a
service by enhancing the Spark [17] big data processing
framework. LogLens classifies anomaly detection algorithms
into stateless and stateful categories and provides a reference
implementation for both categories. The stateless log parser
runs up to 41x faster than the state-of-the-art Logstash [16]
tool and the log analytic applications save up to 12096x man-
Fig. 6: LogLens detects anomalies from SS7 security log data: hours in diagnosing real-world operational problems. Finally,
four clusters of anomalies highlighted by red circles cover logs we share one of the key lessons learned from our interaction
that indicate spoofing attacks. with the real users: when designing an automated system
As shown in Figure 6, the anomalies form multiple clusters instead of no human involvement, we need to focus on the
including in total 994 anomalies. In each cluster, its anomalies minimization of human involvement. One of the main problems
are temporally close to each other. In practice, such anomaly of learning models from a training dataset (i.e., data-driven
clusters usually serve as indicators for the significant system approach) is that in many cases this dataset may not cover all
events. By manually investigating the logs in the clusters, users the possible use-cases that users want to monitor. Therefore,
find that the abnormal logs are traces left by potential SS7 we have to provide options to users for incorporating their
spoofing attacks. domain knowledge during model building as well as allow
Figure 7 shows a few examples of abnormal logs. In them to edit automatically generated models to improve the
particular, the abnormal logs are marked as red, while the accuracy of the anomaly detection results.

1061
 R EFERENCES
[1] S. Alspaugh, B. Chen, J. Lin, A. Ganapathi, M. Hearst, and R. Katz, “Analyzing
log analysis: An empirical study of user log mining,” in LISA14, 2014, pp. 62–77.
[2] G. Lee, J. Lin, C. Liu, A. Lorek, and D. Ryaboy, “The unified logging infrastructure
for data analytics at twitter,” VLDB, vol. 5, no. 12, pp. 1771–1780, 2012.
[3] LATK, “Log Analysis Tool Kit,” http://www.cert.org/digital-intelligence/tools/latke.
cfm, Aug. 2017.
[4] M. Du, F. Li, G. Zheng, and V. Srikumar, “DeepLog: anomaly detection and
diagnosis from system logs through deep learning,” in ACM Conference on
Computer and Communications Security (CCS), 2017.
[5] C. C. Michael and A. Ghosh, “Simple, state-based approaches to program-based
anomaly detection,” ACM Transactions on Information and System Security, vol. 5,
no. 3, Aug. 2002.
[6] E. Analyzer, “An IT Compliance and Log Management Software for SIEM,” https:
//www.manageengine.com/products/eventlog/, Aug. 2017.
[7] PlantLog, “Operator Rounds Software,” https://plantlog.com/, Aug. 2017.
[8] LogEntries, “Log Analysis for Software-defined Data Centers,” https://
blog.logentries.com/2015/02/log-analysis-for-software-defined-data-centers/, Feb.
2015.
[9] X. Yu, P. Joshi, J. Xu, G. Jin, H. Zhang, and G. Jiang, “Cloudseer: Workflow
monitoring of cloud infrastructures via interleaved logs,” in ASPLOS’16. ACM,
2016, pp. 489–502.
[10] Q. Fu, J. G. Lou, Y. Wang, and J. Li, “Execution anomaly detection in distributed
systems through unstructured log analysis,” in Data Mining, 2009. ICDM’09. Ninth
IEEE International Conference on, 2009.
[11] W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan, “Detecting large-scale
system problems by mining console logs,” in ACM SIGOPS. ACM, 2009, pp.
117–132.
[12] S. He, J. Zhu, P. He, and M. R. Lyu, “Experience report: System log analysis
for anomaly detection,” 2016 IEEE 27th International Symposium on Software
Reliability Engineering (ISSRE), pp. 207–218, 2016.
[13] Splunk, “Turn Machine Data Into Answers,” https://www.splunk.com, Aug. 2017.
[14] ElasticSearch, “Open-Source Log Storage,” Aug. 2017. [Online]. Available:
https://www.elastic.co/products/elasticsearch
[15] Gartner, “Iot forecast,” http://www.gartner.com/newsroom/id/3598917, Aug. 2017.
[16] Logstash, “Log Parser,” Aug. 2017. [Online]. Available: https://www.elastic.co/
products/logstash
[17] A. Spark, “Lightning-fast cluster computing,” http://spark.apache.org/, Aug. 2017.
[18] F. Yang, J. Li, and J. Cheng, “Husky: Towards a more efficient and expressive
distributed computing framework,” VLDB Endowment, vol. 9, no. 5, pp. 420–431,
2016.
[19] A. Flink, “Scalable Stream and Batch Data Processing,” https://flink.apache.org/,
Aug. 2017.
[20] Samza, “Distributed stream processing framework,” http://samza.apache.org/, Aug.
2017.
[21] Message-Broker, “Apache kafka,” http://kafka.apache.org/, Aug. 2017.
[22] Kibana, “Visualization tool,” Aug. 2017. [Online]. Available: https://www.elastic.
co/products/kibana
[23] GROK, “Pattern,” Aug. 2017. [Online]. Available: https://www.elastic.co/guide/
en/logstash/current/plugins-filters-grok.html
[24] D. Yuan, H. Mai, W. Xiong, L. Tan, Y. Zhou, and S. Pasupathy, “Sherlog: error
diagnosis by connecting clues from run-time logs,” in ACM SIGARCH, vol. 38,
no. 1. ACM, 2010, pp. 143–154.
[25] H. Hamooni, B. Debnath, H. Xu, Jianwu amd Zhang, G. Jiang, and A. Mueen,
“Logmine: Fast pattern recognition for log analytics,” in CIKM. ACM, October
2016.
[26] “Java regex,” https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html,
Aug. 2017.
[27] RegEx-Format, “Java SimpleDateFormat,” https://docs.oracle.com/javase/7/docs/
api/java/text/SimpleDateFormat.html, Aug. 2017.
[28] C. Ezeife and D. Zhang, “Tidfp: mining frequent patterns in different databases with
transaction id,” in International Conference on Data Warehousing and Knowledge
Discovery. Springer, 2009, pp. 125–137.
[29] “DISCO,” https://fluxicon.com/disco/.
[30] G. Cugola and A. Margara, “Processing flows of information: From data stream to
complex event processing,” vol. 44, no. 3. ACM, 2012, p. 15.
[31] A. Margara, G. Cugola, and G. Tamburrelli, “Learning from the past: automated
rule generation for complex event processing,” in Proceedings of the 8th ACM
International Conference on Distributed Event-Based Systems. ACM, 2014, pp.
47–58.
[32] I. Tudor, “Association rule mining as a data mining technique,” Seria Matematic
Informatic Fizic Buletin, vol. 1, pp. 49–56, 2008.
[33] IaSS, “Openstack,” https://en.wikipedia.org/wiki/OpenStack, Aug. 2017.
[34] PCAP, “Packet Capture,” https://en.wikipedia.org/wiki/Pcap, Aug. 2017.
[35] Spoofing-Attack, “Spoofing attack,” https://en.wikipedia.org/wiki/Spoofing attack,
Aug. 2017.
[36] SS7, “Signaling system no. 7,” https://en.wikipedia.org/wiki/Signalling System
No. 7, Aug. 2017.

1062

View publication stats