p-shopping.com/category.screen?category_id=GIFTS” “Mozilla/4.0 (compatible; MSIE 6.

0; Windows NT
317 27.160.0.0 - - [07/Jan 18:10:56:156] “GET /oldlink?item_id=EST-26&JSESSIONID=SD5SL9FF1ADFF3
1.1” 200 1318 “http://buttercup-shopping.com/cart.do?action=purchase&itemId=EST-26&product_id=K9-
” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)” 468 125.17.14.100
07/Jan 18:10:56:147] “POST /category.screen?category_id=SURPRISE&JSESSIONID=SD9SL4FF4ADFF7 HTTP
200 2423 “http://buttercup-shopping.com/cart.do?action=addtocart&itemId=EST-16&product_id=RP-
” “Opera/9.20 (Windows NT 6.0; U; en)” 564 130.253.37.97 - - [07/Jan 18:10:55:189] “GET /cart.
tion=changequantity&itemId=EST-18&product_id=AV-CB-01&JSESSIONID=SD5SL7FF6ADFF10 HTTP 1.1” 404
“http://buttercup-shopping.com/oldlink?item_id=EST-18” “Opera/9.20 (Windows NT 6.0; U; en)” 766
53.37.97 - - [07/Jan 18:10:55:187] “GET /oldlink?item_id=EST-6&JSESSIONID=SD10SL8FF2ADFF9 HTTP
THE ESSENTIAL
200 3865 “http://buttercup-shopping.com/cart.do?action=purchase&itemId=EST-6&product_id=AV-

MACHINE DATA
” “Opera/9.01 (Windows NT 5.1; U; en)” 553 62.216.64.19 - - [07/Jan 18:10:55:111] “GET /category.
n?category_id=FLOWERS&JSESSIONID=SD8SL8FF1ADFF6 HTTP 1.1” 200 3971 “http://buttercup-shopping.
art.do?action=remove&itemId=EST-15&product_id=FL-DSH-01” “Opera/9.01 (Windows NT 5.1; U; en)”
30.253.37.97 - - [07/Jan 18:10:55:108] “GET /category.screen?category_id=SURPRISE&JSESSIONID=SD-
F9ADFF9 HTTP 1.1” 404 474 “http://buttercup-shopping.com/oldlink?item_id=EST-21” “Mozilla/4.0
atible; MSIE 6.0; Windows NT 5.1)” 606 195.69.160.22 - - [07/Jan 18:10:54:192] “GET /cart.do?ac-
remove&itemId=EST-15&product_id=AV-SB-02&JSESSIONID=SD4SL1FF7ADFF7 HTTP 1.1” 200 205 “http://
rcup-shopping.com/cart.do?action=remove&itemId=EST-15&product_id=AV-SB-02” “Mozilla/4.0 (compati-
MSIE 6.0; Windows NT 5.1; SV1)” 163 131.178.233.243 - - [07/Jan 18:10:54:171] “GET /oldlink?item_
T-17&JSESSIONID=SD1SL9FF9ADFF1 HTTP 1.1” 200 1976 “http://buttercup-shopping.com/cart.do?ac-

GUIDE TO
purchase&itemId=EST-17&product_id=K9-CW-01” “Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3;
) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.38 Safari/533.4” 544 86.9.190.90 - - [07/
8:10:54:166] “POST /cart.do?action=addtocart&itemId=EST-7&product_id=FI-SW-01&JSESSIONID=SD10SL3F-
F4 HTTP 1.1” 404 2258 “http://buttercup-shopping.com/cart.do?action=addtocart&itemId=EST-7&prod-
d=FI-SW-01” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)” 189 82.245.228.36 - - [07/
8:10:54:165] “GET /category.screen?category_id=TEDDY&JSESSIONID=SD1SL1FF8ADFF2 HTTP 1.1” 200 488
://buttercup-shopping.com/cart.do?action=purchase&itemId=EST-27&product_id=FL-DLH-02” “Mozil-
0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)” 969 128.241.220.82 - - [07/
8:10:54:145] “GET /cart.do?action=view&itemId=EST-13&product_id=RP-SN-01&JSESSIONID=SD7SL8F-
F2 HTTP 1.1” 200 1901 “http://buttercup-shopping.com/cart.do?action=view&itemId=EST-13&product_
-SN-01” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)” 393 130.253.37.97 - - [07/Jan
:54:121] “GET /category.screen?category_id=BOUQUETS&JSESSIONID=SD10SL1FF4ADFF10 HTTP 1.1” 406 1437
://buttercup-shopping.com/cart.do?action=addtocart&itemId=EST-27&product_id=AV-SB-02” “Mozilla/5.0
ntosh; U; Intel Mac OS X 10_6_3; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.38
i/533.4” 571 141.146.8.66 - - [07/Jan 18:10:53:118] “GET /category.screen?category_id=SURPRISE&-
IONID=SD7SL3FF9ADFF10 HTTP 1.1” 200 3814 “http://buttercup-shopping.com/category.screen?cate-
id=SURPRISE” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)” 932 141.146.8.66 - - [07/Jan
:53:104] “POST /category.screen?category_id=BOUQUETS&JSESSIONID=SD3SL1FF7ADFF2 HTTP 1.1” 200 2567
://buttercup-shopping.com/product.screen?product_id=AV-SB-02” “Mozilla/4.0 (compatible; MSIE 6.0;
ws NT 5.1)” 920 130.253.37.97 - - [07/Jan 18:10:52:182] “GET /category.screen?category_id=TED-
ESSIONID=SD8SL2FF5ADFF2 HTTP 1.1” 200 1649 “http://buttercup-shopping.com/category.screen?cat-
_id=TEDDY” “Googlebot/2.1 ( http://www.googlebot.com/bot.html) “ 302 125.17.14.100 - - [07/Jan
DIGITAL EXHAUST.
But some of the most important insights you can gain—across
IT and the business—are hidden in this data: where things went
wrong, how to optimize the customer experience, the fingerprints

TIME-SERIES DATA. of fraud. All of these insights can be found in the machine data
generated by the normal operations of your organization.

BIG DATA. Machine data is valuable because it contains a definitive record of

all the activity and behavior of your customers, users, transactions,
applications, servers, networks and mobile devices. It includes
Whatever you call it, machine data is one of the configurations, data from APIs, message queues, change events,
the output of diagnostic commands, call detail records and sensor
most underused and undervalued assets of any data from industrial systems and more.

organization. And, unfortunately, it’s usually kept

The challenge with leveraging machine data is that it comes in a
for some minimum amount of time before being dizzying array of unpredictable formats, and traditional monitoring
and analysis tools weren’t designed for the variety, velocity,
tossed out and never looked at again.
volume or variability of this data. But there’s a tremendous upside
for organizations that take advantage of this data—including
quickly diagnosing service problems, detecting sophisticated
security threats, understanding the health and performance of
remote equipment and demonstrating compliance.

2 3
USING MACHINE DATA SOURCES

Order
MACHINE DATA

IN PRACTICE
Processing

Middleware
Error

Using machine data requires three

(seemingly) simple steps: Care IVR

Twitter

1 2 3 Figure 1: Machine data can come from any number of sources, and at first
INGEST CORRELATE ANALYZE glance, can look like random text.

SOURCES MACHINE DATA

CUSTOMER ID ORDER ID PRODUCT ID
Order
Processing
The organizations that get the most value from machine data
are able to take disparate data types, link them together, and Middleware ORDER ID CUSTOMER ID
Error
gain value from the result. But one of the biggest challenges is
understanding what data you should ingest. ON HOLD TIME

Care IVR
CUSTOMER ID

By defining the use cases you’re attempting to resolve – be it TWITTER ID CUSTOMER TWEET

security, IT operations, business analytics or the Internet of Twitter

Things – you can start to identify the data sources you should
COMPANY TWITTER ID
ingest and begin correlating.
Figure 2: The value of machine data is hidden in this text.

So how does machine data provide value? See the example to

the right. SOURCES MACHINE DATA
CUSTOMER ID ORDER ID PRODUCT ID
Order
Processing
In this example, analyzing the machine data makes the story clear:
Middleware ORDER ID CUSTOMER ID
Error

1. A customer’s order didn’t go through

ON HOLD TIME

2. The customer called Support to try to resolve the issue Care IVR
CUSTOMER ID

3. After some time on hold, the customer sent a negative tweet TWITTER ID CUSTOMER TWEET

Twitter
about the company
COMPANY TWITTER ID

By linking together the machine data, the company can see the
Figure 3: By correlating different types of machine data together, you can start
original issue and get a full view of the customer experience. to gain real insight into what’s going on in your infrastructure, see security
threats or even use the insights to drive better business decisions.
4 5
 Table of Contents

THE ESSENTIAL GUIDE User Data................................................. 8

Authentication........................................ 8
Proxies...................................................... 72
VoIP........................................................... 74

TO MACHINE DATA Virtual Private Networks (VPN).......10

Operating System Data.................76
Application Data............................... 12 System Logs........................................... 76
This book provides a high-level overview of the most common Antivirus................................................... 12 System Performance........................... 78
types of machine data that are found in organizations of nearly (APM) Tool Logs....................................14
any size. While each organization’s needs and data sources will Virtual Infrastructure Data.......... 80
Custom Application Logs..................16
vary by vendor, product and infrastructure, this book details where & Debug Logs Amazon Web Services (AWS)........80

you should look for machine data and the value it can provide to CRM, ERP and Other............................ 18 Microsoft Azure.................................... 82

IT, security, business analytics and Internet of Things use cases. Business Applications VMware Server Logs,..........................84
Code Management..............................20 Configuration Data and
Performance Metrics
Many of the data sources listed in this book can support multiple Vulnerability Scanning....................... 22

use cases – this is a major part of what drives machine data’s Mail Server..............................................24 Physical Infrastructure
tremendous value. The use cases supported by each data source Test Coverage Tools............................26 & IoT Data............................................ 86

can be easily identified with the icons below. Automation, Configuration and...... 28 Physical Card Readers........................86
Deployment Tools (Platforms) Sensor Data............................................88
Build Systems (Platforms)................30 Server Logs........................................... 90
Binary Repositories............................. 32 Backup..................................................... 92
Container Logs & Metrics..................34 Storage.....................................................94
Mainframe...............................................96
Middleware Data.............................. 36
Patch Logs..............................................98
Middleware.............................................36
Telephony..............................................100
Web Server.............................................38
Point of Sale Systems....................... 102
SECURITY IT OPS, APP DELIVERY Application Server...............................42
& COMPLIANCE & DEVOPS RFID/NFC/BLE...................................104
Mobile Device Data.............................44
Smart Meters....................................... 106
Network Data..................................... 46 Transportation..................................... 108
SNMP........................................................46 Medical Devices................................... 110
Deep Packet Inspection Data..........48 Environmental Sensors......................112
DHCP........................................................50 Industrial Control Systems.............. 114
Endpoint.................................................. 52 Wearables.............................................. 116
Firewall.....................................................54
Additional Data Sources..............118
FTP............................................................56
Database.................................................118
Intrusion Detection/Prevention......58
Third-Party Lists................................. 120
INTERNET BUSINESS Load Balancer...................................... 60
OF THINGS ANALYTICS Social Media Feeds.............................122
DNS...........................................................62
Human Resources.............................. 124
Network Access Control (NAC)......64
Business Service Transaction &.... 126
Network Switches................................66
Business Service Performance Data
Network Routers..................................68
Network Protocols...............................70
6 7
 USER DATA
AUTHENTICATION
DATA
Use Cases: Security & Compliance, IT Operations, Application Delivery
Examples: Active Directory, LDAP, Identity Management,
Single-Sign On

Authentication data provides insight into users and identity Use Cases:
activity. Common authentication data sources include:
IT Ops & Application Delivery: Authentication data supports
• Active Directory: a distributed directory in which organiza- IT operations teams as they troubleshoot issues related to
tions define user and group identities, security policies and authentication. For example, application support can be tied to
logins, enabling IT operations to see whether users are struggling
content controls.
to log in to applications. For IT operations teams that support
• LDAP: an open standard defined by the IETF and is typically Active Directory, logs can be used to troubleshoot and understand
used to provide user authentication (name and password). the health of Active Directory.

It has a flexible directory structure that can be used for a

Security & Compliance: For security, authentication data provides
variety of information such as full name, phone numbers,
a wealth of information about user activity, such as multiple login
email and physical addresses, organizational units, workgroup
failures or successes to multiple hosts in a given time window,
and manager. activities from different locations within a given amount of time,
and brute force activities. Specifically:
• Identity Management: identity management is the method
of linking the users of digital resources—whether people, IoT
• Active Directory domain controller logs contain information
devices, systems or applications—to a verifiable online ID.
regarding user accounts, such as privileged account activity,
• Single Sign-On (SSO): a process of using federated identity as well as the details on remote access, new account creation
management to provide verifiable, attestable identities from a and expired account activity.
single source to multiple systems. SSO significantly increases
• LDAP logs include a record of who, when and where users log
security by tying user credentials to a single source, allowing
in to a system and how information is accessed.
changes to user rights and account status to be made once,
and reflected in every application or service to which the user • Identity Management data shows access rights by user, group
has access. SSO is particularly important for users with elevat- and job title (e.g., CEO, supervisor or regular user). This data
ed security rights such as system or network administrators can be used to identify access anomalies that could be po-
that have access to a large number of systems. tential threats—for example, the CEO accessing a low-level
networking device or a network admin accessing the CEO’s
account.
8 9
 USER DATA
VIRTUAL PRIVATE
NETWORKS (VPN)
Use Cases: Security & Compliance
Examples: Citrix NetScaler Nitro, Citrix NetScaler IPFIX, Cisco

Virtual private networks (VPNs) are a way of building a secure Use Cases:
extension of a private network over an insecure, public one. VPNs
can be established either between networks, routing all traffic Security & Compliance: VPN logs help in analyzing users coming
between two sites, or between a client device and a network. onto the network. This information can be used in a number
Network-to-network VPNs typically are created using strong of ways, including situational awareness, monitoring foreign IP
credentials such as certificates on each end of the connection. subnets, and compliance monitoring of browsers and applications
Client-to-network VPNs rely on user authentication, which can of connected hosts. VPN data can also help identify:
be as simple as a username and password. VPNs use network
tunneling protocols such as IPSec, OpenVPN plus SSL or L2TP • Activities from different locations, such as changes in location
with cryptographically strong algorithms to scramble information within a given amount of time
in transit and ensure end-to-end data integrity.
• Access from risky countries or locations

• User sessions at odd times, such as late evenings or weekends

• User land speed violations

• Abnormal frequency of sessions based on each user profile

10 11
 APPLICATION DATA
ANTIVIRUS
Use Cases: Security & Compliance
Examples: Kaspersky, McAfee, Norton Security, F-Secure, Avira,
Panda, Trend Micro

The weakest link in corporate security is an individual, and Use Cases:

antivirus is one way to protect employees from performing
inadvertently harmful actions. Whether it’s clicking on an Security & Compliance: Antivirus logs support the analysis
untrustworthy web link, downloading malicious software or of malware and vulnerabilities of hosts, laptops and servers; and
opening a booby-trapped document (often one sent to them by can be used to monitor for suspicious file paths. This data can
an unsuspecting colleague), antivirus can often prevent, mitigate help identify:
or reverse the damage.
• Newly detected binaries, file hash, files in the filesystem
So-called advanced persistent threats (APTs) often enter through and registries
a single compromised machine attached to a trusted network.
While not perfect, antivirus software can recognize and thwart • When binaries, hash or registries match threat intelligence
common attack methods before they can spread.
• Unpatched operating systems

• Known malware signatures

12 13
 APPLICATION DATA
APM
TOOL LOGS
Use Cases: IT Operations, Application Delivery, Security & Compliance
Examples: Dynatrace, New Relic, App Dynamics, MMSoft Pulseway,
LogicMonitor, Stackify, Idera, Ipswitch

Application Performance Management (APM) software provides Use Cases:

end-to-end measurement of complex, multi-tier applications to
provide performance metrics from an end user’s perspective. IT Ops & Application Delivery: By providing end-to-end
APM logs also provide event traces and diagnostic data that can measurement of complex, multi-tier applications, APM logs can
assist developers in identifying performance bottlenecks or error show infrastructure problems and bottlenecks that aren’t visible
conditions. The data from APM software provides both a baseline when looking at each system individually, such as slow DNS
of typical application performance and a record of anomalous resolution causing a complex web app to bog down as it tries to
behavior or performance degradation. Carefully monitoring access content and modules on many different systems.
APM logs can provide an early warning to application problems
and allow IT and developers to remediate issues before users Security & Compliance: Security teams can use APM logs to
experience significant degradation or disruption. APM logs also perform post-hoc forensic analysis of incidents that span multiple
are required to perform post-hoc forensic analysis of complex systems and exploit vulnerabilities. The data can be used to
application problems that may involve subtle interactions between correlate security indications between the system and application
multiple machines, network devices or both. activities. It also helps to identify SQL/API calls/CMD made in
relation to suspicious activity, or abnormal amounts of sessions or
CPU load in relation to security activity.

14 15
 APPLICATION DATA
CUSTOM APPLICATION
& DEBUG LOGS
Use Cases: IT Operations, Application Delivery, Security & Compliance
Examples: Custom applications

Best practices for application developers require the inclusion of Use Cases:
debugging code in applications that can be enabled to provide
minute details of application state, variables and error conditions IT Ops & Application Delivery: Debug output can expose
or exceptions. Debug output is typically logged for later analysis application behavior that causes inefficient use of system
that can expose the cause of application crashes, memory leaks, resources or application failures that can be addressed by
performance degradation and security holes. Furthermore, since developers and operations teams. Debug output is useful for
the events causing a security or performance problem may be unraveling the internal state of an application that exhibits
spaced over time, logs—along with the problem software—can performance problems or has been shown to have security
help correlate and trace temporally separated errors to show how vulnerabilities, and the data can be helpful in identifying
they contribute to a larger problem. root cause

Application debug logs provide a record of program behavior Security & Compliance: Security breaches are often the result of
that is necessary to identify and fix software defects, security improper handling of unexpected inputs, such as buffer overflow
vulnerabilities or performance bottlenecks. While test logs record exploits or data injection used in cross-site scripting attacks.
the output results of application usage, debug logs provide This type of low-level vulnerability is almost impossible to detect
information about an application’s internal state, including the without logging the internal state of various application variables
contents of variables, memory buffers and registers; a detailed and buffers.
record of API calls; and even a step-by-step trace through a
particular module or subroutine. Due to the performance overhead Similar to APM logs, custom application and debug logs can be
and amount of data produced, debug logs typically are enabled used to correlate security indications between the system and
only when a problem can’t be identified via test or event logs. application activities. It also helps to identify SQL/API calls/CMD
made in relation to suspicious activity, or abnormal amounts of
sessions or CPU load in relation to security activity.

16 17
 APPLICATION DATA
CRM, ERP AND OTHER
BUSINESS APPLICATIONS
Use Cases: Application Delivery, Security & Compliance, Business Analytics, IoT
Examples: SAP, SFDC, SugarCRM, Oracle, Microsoft Dynamics

Business Applications can create a wealth of data as part of tion across back-end digital business processes. ERP systems
normal operations. Two examples are CRM and ERP applications: are typically built on a relational database management sys-
tem with a variety of modules and customizations for specific
• Customer relationship management (CRM) systems have
functions such as supplier relationship management or supply
become an essential part of every organization, providing a
chain management. Due to their complexity, ERP systems
central database of all customer contact information, commu-
often are installed and managed by product specialists.
nications and transaction details. CRM systems have evolved
from simple contact management systems to platforms for
Use Cases:
customer support and engagement by providing personalized
Application Delivery: CRM databases can provide a complete
sales and support information. The same customer support
record of all information and events leading up to a customer
data repository can be used to develop customized marketing
escalation. When combined with other data sources, CRM can
messages and sales promotions. CRM systems are also useful provide indicators of deeper issues.
for application support and enhancement by recording details
Like other application records, ERP logs are necessary when
about customer problems with a particular system or applica-
debugging performance and reliability problems due to
tion along with their eventual solution—details that can inform
the complex interactions between many systems in an ERP
future application or service updates. implementation. Logs are also useful in capacity planning.

• Enterprise resource planning (ERP) applications are a critical Security & Compliance: CRM records can help security teams
back-office IT service that provides systematic, automated unravel incidents that involve multiple customers and problem
collection and analysis of a variety of product, supply chain episodes over a long time span. They can also provide evidence
and logistics data. ERP is used in product planning, tracking of a breach, should records be modified outside normal business
purchases of components and supplies, inventory manage- processes. In addition, the data can be used to audit access
records of customer or internal user information.
ment, monitoring and regulating manufacturing processes,
managing logistics, warehouse inventory and shipping and to Business Analytics & IoT: CRM and ERP data is a crucial source of
monitor and measure the effectiveness of sales and market- referential and transactional data that helps drive much needed
ing campaigns. ERP software also integrates with CRM, HR, context to machine data in business use cases. For instance, when
combined with point-of-sale data and mobile application data
finance/accounting/payroll and asset management systems,
from loyalty applications, retailers can drive real-time 1:1 targeted
with bidirectional data flows that provide consistent informa-
marketing campaigns, and then use machine learning to predict
18 customer purchasing behavior and revenue trends. 19
 APPLICATION DATA
CODE
MANAGEMENT
Use Cases: Application Delivery

For all but the most trivial implementations, application source Use Cases:
code is comprised of dozens if not hundreds of interrelated files.
The complexity and volatility of code—particularly when using Application Delivery: The version records of code management
agile development methodologies and changes are made daily— can help IT operations teams identify application changes that are
makes keeping track of it virtually impossible without a structured, causing system problems, such as excessive resource consumption
automated source code management and revision control system. or interference with other applications.

Originally built as client-server applications where developers

checked in code to a central repository, today’s systems (such
as Git) are often distributed, with each developer working from
a local copy of the full repository and changes synchronized
across all subscribers to a particular project. Code management
systems provide revision control (the ability to back out changes
to an earlier version), software build automation, configuration
status records and reporting, and the ability to branch or fork all
or part of a source-code tree into a separate subproject with its
own versioning.

20 21
 APPLICATION DATA
VULNERABILITY
SCANNING
Use Cases: Security & Compliance
Examples: ncircle IP360, Nessus

An effective way to find security holes is to examine infrastructure Use Cases:

from the attacker’s point of view. Vulnerability scans probe an
organization’s network for known software defects that provide Security & Compliance: Vulnerability scans yield data about open
entry points for external agents. These scans yield data about ports and IP addresses that can be used by malicious agents to
open ports and IP addresses that can be used by malicious agents gain entry to a particular system or entire network. The data can
to gain entry to a particular system or entire network. used to identify:

Systems often keep network services running by default, even • System misconfiguration causing security vulnerability
when they aren’t required for a particular server. These running, • Outdated patches
unmonitored services are a common means of external attack, • Unnecessary network service ports
as they may not be patched with the latest OS security updates.
• Misconfigured filesystems, users or applications
Broadscale vulnerability scans can reveal security holes that could
be leveraged to access an entire enterprise network. • Changes in system configuration
• Changes in various user, app or filesystem permissions

22 23
 APPLICATION DATA
MAIL
SERVER
Use Cases: IT Operations, Security & Compliance
Examples: Exchange, Office 365

Email remains the primary form of formal communication in Use Cases:

most organizations. As such, mail server databases and logs are
some of the most important business records. Due to their size IT Ops: Email messages and activity logs can be required to
and tendency to grow without bounds, email data management maintain compliance with an organization’s information security,
typically requires both data retention and archival policies so that retention and regulatory compliance processes. Mail server
only important records are held and inactive data is moved to low- transaction and error logs also are essential debugging tools for IT
cost storage. problem resolution and also may be used for usage-based billing.

Security & Compliance: Mail server data can help identify

malicious attachments, malicious domain links and redirects,
emails from known malicious domains, and emails from unknown
domains. It can also be used to identify emails with abnormal or
excessive message sizes, and abnormal email activities times.

24 25
 APPLICATION DATA
TEST COVERAGE
TOOLS
Use Cases: Application Delivery & DevOps
Examples: Static Analysis & Unit Testing logs (SonarQube, Tox, PyTest, RubyGem
MiniTest, Bacon, Go Testing), build server logs and performance metrics

Typical test coverage includes functional, statement, branch Use Cases:

and conditional coverage. The idea is to match what percentage
of code can be exercised by a test suite of one or more Application Delivery and DevOps: Test coverage data
coverage criteria. Coverage tests are usually defined by rule or monitoring helps release managers, application owners and
requirements. In addition to coverage testing, software delivery others understand:
teams can utilize machine data to understand the line count, code
• How much technical debt and issues are they resolving?
density and technical debt.
• How ready is their next release?
• From unit testing – how many tests were performed per
hour and what tests are being run?

If test coverage data is combined with build data, release

managers can start monitoring build and release performance and
start understanding the release quality. They can understand the
trends in error percentage and make decisions on if the build is
ready for production. Understanding code quality can also help
support teams get prepared for any additional volume of calls or
any particular issues that may arise.

26 27
 APPLICATION DATA
AUTOMATION, CONFIGURATION,
DEPLOYMENT TOOLS (PLATFORMS)
Use Cases: Application Delivery & DevOps
Examples: Puppet Enterprise, Ansible Tower, Chef, SaltStack, Rundeck,
machine data ingested through APIs, webhooks or run logs

Automated configuration and deployment tools, also known Use Cases:

as “infrastructure as code,” allow IT and DevOps practitioners
to practice continuous application delivery in the cloud or on Application Delivery: Automation and configuration machine
premises. When infrastructure is treated as code, it’s easy to share, data monitoring helps application delivery teams deliver
collaborate, manage version control, perform peer unit testing, applications faster without sacrificing stability or security.
automate deployments, check the status of deployment and more.

Tools like Rundeck are platforms that take automation frameworks

like Salt Stack and enable teams to automate states or playbook
to make sure the code is released and reported back to a central
reporting tool.

28 29
 APPLICATION DATA
BUILD SYSTEMS
(PLATFORMS)
Use Cases: Application Delivery & DevOps
Examples: Jenkins, Bamboo, TravicCI, TeamCity, machine data ingested
through APIs, logs, webhooks

Build platforms, like Jenkins and Bamboo, enable a continuous Use Cases:
integration practice that allows application delivery teams—
including developers, DevOps practitioners, QA, and release Application Delivery and DevOps: Build systems monitoring
engineering—to build artifacts, trigger new builds and helps release managers, test and QA teams understand the health
environments, automate tests and more. of their build environment, the status of tests, get insights into
stack traces and build queues. This visibility helps remediate build
or test bottlenecks and increase the application delivery velocity
and quality.

30 31
 APPLICATION DATA
BINARY
REPOSITORIES
Use Cases: Application Delivery & DevOps
Examples: Data from Nexus, Artifactory, delivered through APIs, webhooks; Yum,
Pacman and Aptly data delivered through logs

A binary repository is a tool for downloading and storing of binary Use Cases:
files used and created in software development. It’s used to store
software binary packages, artifacts and their corresponding Application Delivery and DevOps: Analyzing binary
metadata. They’re different than source code repositories, as repository data helps application delivery teams and release
binary repositories do not store source files. Searching through managers to ensure that the final deployment of code to
these repositories is possible by analyzing associated metadata. production is successful.

32 33
 APPLICATION DATA
CONTAINER LOGS
AND METRICS
Use Cases: Application Delivery & DevOps
Examples: Docker

Container logs are an efficient way to acquire logs generated by Use Cases:
applications running inside a container. By utilizing logging drivers,
output that is usually logged is redirected to another target. Since Application Delivery and DevOps: Acquiring container log files
logging drivers start and stop when containers start and stop, this gives developers and operations teams insight on errors, issues
is the most effective way of capturing machine data, given the and availability of applications running inside containers. Logs
often limited lifespan of a container. and metrics at the container level also call attention to containers
whose performance is outside of expected parameters. As a result,
admins can “kill” or “stop” a container instance and “run” a new
Container metrics contain details related to CPU, memory, I/O and
container in its place.
network metrics generated by a container. By capturing this data,
you have the opportunity to spot specific containers that appear
to consume more resources than others – enabling faster, more
precise troubleshooting.

34 35
 MIDDLEWARE DATA
MIDDLEWARE
Use Cases: IT Operations, Application Delivery, Security & Compliance,
Business Analytics
Examples: Tibco, Software AG

Middleware describes a software layer of the prototypical Use Cases:

three-tier enterprise application that typically implements
data transformations, analysis and business logic. Middleware IT Ops & Application Delivery: Middleware data can help
accesses databases for persistent storage and relies on web operations teams diagnose problems with three-tier applications
apps for the user interface. Middleware is often developed on that involve the interaction between web, middleware and
the J2EE platform. database servers.

Security & Compliance: Since middleware generally accesses

network services and sensitive databases, security teams can
use log data to vet application integrity, identify suspicious
behavior and specific vulnerabilities. It can also be used for user
and customer transaction monitoring and to identify abnormal
transactions, unknown user interaction with third party accounts,
and the sequence of exact transaction patterns that match known
fraudulent profiles.

Business Analytics: Middleware messaging data is crucial to

understanding transactional business process execution. Using this
data can lead to a better understanding of the overall performance
of a business process, bottlenecks and the opportunities of
process improvement, as well as real-time reaction to potential
failures. When used with machine learning, this data can help
predict problems in a process or provide early indication to where
a process is failing.

36 37
 MIDDLEWARE DATA
WEB SERVER
Use Cases: IT Operations, Application Delivery, Security & Compliance,
Business Analytics
Examples: Java J2EE, Apache, Application Usage Logs, IIS logs, nginx

Web servers are the backend application behind every website • Application Usage Logs: Like Apache web logs, collecting
that delivers all content seen by browser clients. Web servers application usage logs can provide valuable information
access static HTML pages and run application scripts in a variety to multiple stakeholders including developers, IT, sales and
of languages that generate dynamic content and call other
marketing. Depending on how granular the measurement,
applications such as middleware.
usage tracking can assist developers in identifying application
Web servers can vary widely, and can include: features that are most and seldom used, those that users have

• Java – J2EE: Java is the most popular programming language trouble with and areas for future enhancement. For custom-

due to its versatility, relative ease of use and rich ecosystem of er-facing applications, usage logs provide sales and marketing

developer tools. Via the J2EE platform, which includes APIs, teams insight into the effectiveness of online and app-based

protocols, SDKs and object modules, Java is widely used for sales channels and promotions, data about sell-through and

enterprise apps including web applets, middle-tier business transaction abandonment, and information for potential cross-

logic and graphic front ends. Java is also used for native sales promotions.

Android mobile apps.

Use Cases:
• Apache: Apache is one of the oldest and most-used web
IT Ops & Application Delivery: Web logs are critical in debugging
servers on the internet, powering millions of enterprise, gov- both web application and server problems, but also are used to
ernment and public sites. Apache keeps detailed records of generate traffic statistics that are useful in capacity planning. Web
every transaction: every time a browser requests a webpage, server data can provide varying information for IT operations
Apache log details include items such as the time, remote IP teams:
address, browser type and page requested. Apache also logs
various error conditions such as a request for a missing file, • J2EE data can help operations teams diagnose problems with
attempts to access a file without appropriate permissions or three-tier applications that involve the interaction between
problems with an Apache plug-in module. Apache logs are web, middleware and database servers.
critical in debugging both web application and server prob-
• In aggregate, Apache web logs can show activity of a web
lems, but are also used to generate traffic statistics, track
service. Drilling into details can reveal infrastructure bottle-
user behavior and flag security attacks such as attempted
necks and indicate downstream issues.
unauthorized entry or DDoS.

38 39
 MIDDLEWARE DATA
WEB SERVER
(Continued)

• Application usage logs can help IT operations teams with Business Analytics: Web logs are crucial to understanding the
infrastructure capacity planning, optimization, load balanc- interaction between a company’s customers and the overall
ing and usage-based billing by providing detailed records of journey they’re taking across multiple channels of the business.
This data can help companies understand what customers are
resource consumption.
looking for, where they are spending time, or what they were
looking for before they called your call center for help. When
Security & Compliance: Web logs record error conditions such as
correlated with CRM and point-of-sale data, companies can gain a
a request to access a file without appropriate permissions and also
better overall understanding of customer journeys, driving higher
track user activity that can flag security attacks such as attempted
conversions from browse to act, and act to purchase.
unauthorized entry or DDoS. It can also help to identify SQL
injections and support correlating fraudulent transactions.

• Since Java apps frequently access network services and

sensitive databases, security teams can use log data to vet
the integrity of J2EE apps, identify suspicious application
behavior and application vulnerabilities.

• Apache web logs can alert to security attacks such as

attempted unauthorized entry, XSS, buffer overflows
or DDoS.

• Like web logs, generic application usage logs can alert

security teams to unauthorized access such as a someone
consuming more resources than normal, or using applications
at odd hours.

40 41
 MIDDLEWARE DATA
APPLICATION
SERVER
Use Cases: IT Operations, Application Delivery, Security & Compliance
Examples: log4j, log4php

Whether building a multi-tier web application or using a traditional Use Cases:

client-server design, application servers run the backend software
that handles user requests. Today, these are typically deployed as IT Ops & Application Delivery: The value of application server
virtual machines on a multi-tenant hypervisor. logs depends on what they collect; however, these may include
customer information useful in troubleshooting or application
state transitions similar to, but less verbose than debug output
that can provide clues to application crashes, memory leaks and
performance problems.

Security & Compliance: Security breaches are often the result of

improper handling of unexpected inputs, such as buffer overflow
exploits or data injection used in cross-site scripting attacks.
This type of low-level vulnerability is almost impossible to detect
without logging the internal state of various application variables
and buffers. Since the events causing a security or performance
problem may be spaced over time, logs, along with the problem
software, can help correlate and trace temporally separated errors
to show how they contribute to a larger problem. Anomalies in the
logs can indicate potential failures or compromised attempts. The
data can also help:

• Monitor user or customer transactions

• Identify abnormal volume/amount/session of transactions

• Identify unknown user interaction with third accounts,

users or both

• Sequence the exact transaction patterns matching

fraudulent profiles

42 43
 MIDDLEWARE DATA
MOBILE DEVICE
DATA
Use Cases: IT Operations, Application Delivery,
Security & Compliance

Given the array of always-active sensors on mobile devices, they Use Cases:
are veritable gushers of data that can include:
IT Ops & Application Delivery: Since mobile apps invariably
• Physical parameters such as location, network MAC ID, device connect to one or more backend services, data from the client’s
GUID, device type and OS version point of view can provide insight into the app’s condition and
state when investigating issues such as crashes, performance
• Network settings such as address, AP or cell-base station degradation or security leaks. Mobile data shows the sequence of
location, link performance events and the application conditions leading up to and during
a problem. If the source of the problem is the mobile application
• Application-specific telemetry such as time in app, features itself, getting insight on mobile application data can help
used and internal state and debug parameters similar to those developers deliver a better performing mobile app.
provided by conventional application servers
Security & Compliance: Security teams can expand the threat
landscape by monitoring mobile device data for abnormal activity
in regards to authentication, location and application usage.

44 45
 NETWORK DATA
SNMP
Use Cases: IT Operations, Security & Compliance
Examples: LogicMonitor, ManageEngine, Spiceworks,
Ruckus Idera, Ipswitch

The simple network management protocol (SNMP) is one of the Use Cases:
oldest, most flexible and broadly adopted IP protocols used for
managing or monitoring networking devices, servers and virtual IT Ops: SNMP data can provide current information about
appliances. This includes network devices such as routers and performance, configuration and current state. This allows the
switches, as well as non-networking equipment such as server monitoring of the “normal” state of the environment, which is vital
hardware or disk arrays. when using a service-level approach to monitoring of health of any
environment. This could include current speed of all of the ports
SNMP supports two different methods of obtaining data. on a switch, the number of bytes sent (per port or in aggregate)
through a router, the CPU temperature of a server, and any other
• SNMP Traps are essentially alerts, set to send an alert on a
information made available by the vendor per the SNMP MIBs for
state change, critical threshold, hardware failure and more. that device.
Traps are initiated by the SNMP device, and the trap is sent to
Many environments rely on SNMP traps for alerting when a critical
an SNMP collector.
state is reached (e.g., CPU temperature is critical) or when a failure
• SNMP Polling is an interactive query/response approach. Un- occurs (e.g., RAID disk failure). SNMP traps are not only sent by
devices to monitoring systems, in some environments SNMP traps
like traps, polling is initiated by the SNMP collector in the form
are the de-facto method for multiple monitoring and alerting
of a request for certain­—or all—SNMP data available on the
systems to aggregate errors to a single console.
SNMP device.

Although many now provide vendor-specific APIs for remote Security & Compliance: SNMP traps and alerts from network
management and data collection, SNMP is still valuable in devices can help security teams identify abnormal activity over
troubleshooting due to its ubiquity (nearly every device supports the network. SNMP Polling helps a security analyst to see the
it) and inherently centralized design (a single instance of SNMP data transmission rates for a network-connected device that is
management software can collect data from every device on an suspected of malicious activity.
internal network, even across route domains).
The data can also help identify abnormal amounts of traffic to
a certain site or domain, an abnormal amount of specific SNMP
traps from a certain host, and an abnormal number of unique
SNMP traps from hosts compared to normal profiles.

46 47
 NETWORK DATA
DEEP PACKET
INSPECTION DATA
Use Cases: IT Operations, Security & Compliance
Examples: Stream, PCAP, bro

Deep Packet Inspection (DPI) is a fundamental technique used by Use Cases:

firewalls to inspect headers and the payload of network packets
before passing them down the network subject to security rules. IT Ops: Data on the network wire is authoritative and difficult
DPI provides information about the source and destination of the to spoof (although encryption, steganography and advanced
packet, the protocol, other IP and TCP/UDP header information deception techniques can evade DPI). For example, DPI provides
and the actual data. raw information of everything transmitted over a network,
including things that aren’t necessarily part of or difficult to
extract from a log, such as database query results.

Security & Compliance: Packet Capture logs (PCAP) see

everything traversing a network and are required to identify
security attacks and incidents such as advanced persistent threats,
data exfiltration, DDoS and malware. DPI also can be used to filter
content subject to an organization’s terms of service. PCAP data
can also be used to provide and identify:

• DNS session analysis for malicious domain communications

from each endpoint

• Abnormal amounts of traffic or sessions

• Abnormal amounts of domain and host communications

• Known malicious traffic from a host

• Expired SSL certification analysis

• Abnormal host communications (internal and external)

48 49
 NETWORK DATA
DHCP
Use Cases: IT Operations, Security & Compliance
Examples: DHCP Insight, Linux DHCP

DHCP is the network protocol most client devices use to associate Use Cases:
themselves with an IP network. Implemented via a DHCP server,
which could be standalone or embedded in a router or other IT Ops: DHCP logs can be used when troubleshooting a client
network appliance, DHCP provides network clients with critical device that is having network problems since it provides a
network parameters including IP address, subnet mask, network definitive record of the device’s primary IP parameters. The data
gateway, DNS servers, WINS or other name servers, time servers may show the DHCP server itself is at fault; for example, by not
(NTP), a host and domain name and the address of other optional properly vending addresses, renewing IP leases or giving the same
network services. address to two separate devices.

Security & Compliance: DHCP logs show exactly which systems

are connecting to a network, their IP and MAC addresses, when
they connect and for how long. This information is useful in
establishing the state of a network when a security incident occurs
and tracing an attacker’s address back to a time of access and
type of device by looking at the MAC ID and vendor identification
string. The data can also be used to support user network access
verification.

50 51
 NETWORK DATA
ENDPOINT
Use Cases: Security & Compliance
Examples: McAfee ePO, Symantec SEP

Endpoint security is used to protect corporate networks from Use Cases:

inadvertent attacks by compromised devices using untrusted
remote networks such as hotspots. By installing clients on laptops Security & Compliance: Endpoint data can be used for a variety
or other wireless and mobile devices, endpoint security software of security uses, including identifying newly detected binaries,
can monitor activity and provide security teams with warnings of file hash, files in the filesystem and registries. It can also help
devices attempting to spread malware or pose other threats. with identifying binary and hash registries that match threat
intelligence, as well as unpatched operating systems and binaries,
and to detect known malware.
In this context, endpoint refers to the security client software
or agent installed on a client device that logs security-related
activity from the client OS, login, logout, shutdown events and
various applications such as the browser (Explorer, Edge), mail
client (Outlook) and Office applications. Endpoints also log their
configuration and various security parameters (certificates, local
anti-malware signatures, etc.), all of which is useful in post-hoc
forensic security incident analysis.

52 53
 NETWORK DATA
FIREWALL
Use Cases: IT Operations, Security & Compliance
Examples: Palo Alto, Cisco, Check Point

Firewalls demarcate zones of different security policy. By Use Cases:

controlling the flow of network traffic, firewalls act as gatekeepers
collecting valuable data that might not be captured in other IT Ops: When network applications are having communication
locations due to the firewall’s unique position as the gatekeeper problems, network security policies may be the culprit. Firewall
to network traffic. Firewalls also execute security policy and thus data can provide visibility into which traffic is blocked and which
may break applications using unusual or unauthorized network traffic has passed through – helping identify if you have an app or
protocols. network issue.

Security & Compliance: Firewall logs provide a detailed record

of traffic between network segments, including source and
destination IP addresses, ports and protocols, all of which are
critical when investigating security incidents. The data may also
reveal gaps in security policy that can be closed with tighter
construction of firewall rules. Firewall data can help identify and
detect:

• Lateral movement
• Command and Control traffic
• DDoS traffic
• Malicious domain traffic
• Unknown domain traffic
• Unknown locations traffic

54 55
 NETWORK DATA
FTP
Use Cases: IT Operations, Security & Compliance
Examples: OSSEC, Getwatchlist, UTBox, Security Onion,
iSeries - AS400, Traffic Ray

FTP is one of the oldest and most rudimentary network protocols Use Cases:
for copying data from one system to another. Before websites
and HTTP, FTP was the best way to move large files across the IT Ops: FTP traffic logs record the key elements of a file
internet. FTP is still used in organizations that need reliable, transmission, including source (client) name and address and
deterministic internet file transfer. remote user name if the destination is password-protected. This
and other data are crucial when troubleshooting FTP problems,
regardless of the application.

Security & Compliance: Analyzing FTP servers can help security

teams identify when compromised credentials are used, when
abnormal traffic is coming from different locations or at odd times,
and when sensitive files and document are being accessed.

56 57
 NETWORK DATA
INTRUSION
DETECTION/PREVENTION
Use Cases: Security & Compliance
Examples: Tipping Point, Juniper IDP, Netscreen Firewall, Juniper NSM IDP,
Juniper NSM, Snort, McAfee IDS

IDS and IPS are complementary, parallel security systems that Use Cases:
supplement firewalls – IDS by exposing successful network and
server attacks that penetrate a firewall, and IPS by providing more Security & Compliance: IDS logs provide security teams detailed
advanced defenses against sophisticated attacks. IDS is typically records of attacks including the type, source, destination and
placed at the network edge, just inside a perimeter firewall, port(s) used that provide an overall attack signature. Special
although some organizations also put a system outside the firewall signatures may trigger alarms or other mitigating actions. IPS
to provide greater intelligence about all attacks. Likewise, IPS is provide the same set of attack signature data, but also may
typically placed at the network perimeter, although it also may be include a threat analysis of bad network packets and detection of
used in layers at other points inside the network or on individual lateral movement. This data can also detect command and control
servers. IPS usually works by dropping packets, resetting network traffic, DDoS traffic, and malicious or unknown domain traffic.
connections and blacklisting specific IP addresses or ranges.

58 59
 NETWORK DATA
LOAD BALANCER
Use Cases: IT Operations
Examples: Local Traffic Manager, Cisco Load Balancer, Citrix,
Kemp Technologies, Radware AppDirector OnDemand

Load balancers allocate external network traffic bound for Use Cases:
a particular server or application across multiple redundant
instances. There are two categories of load balancer: local, in IT Ops: Load balancer logs provide operations teams with a record
which all resources in a load-balanced pool are on the same of overall traffic to systems or particular applications and provide
subnet; and global or distributed, where the resource pool is indicators of each system’s traffic-handling capacity and health,
spread across multiple sites. Load balancers use several user- along with the status and health of the load balancer itself.
selectable algorithms to allocate traffic including:

• Round robin (systems get an equal number of connections

allocated sequentially)

• Weighted round robin (where the load is assigned according

to the percentage weight assigned each system in a pool)

• Least connections (where new connections go to the system

with the fewest number of existing clients)

• Weighted least connections (where the connection

handling capacity of each system is taken into account when
determining the least busy system for new connections)

• Random (connections are randomly assigned to each member

of a pool)

60 61
 NETWORK DATA
DNS
Use Cases: IT Operations, Security & Compliance
Examples: BIND, PowerDNS, Unbound, Dnsmasq, Erl-DNS

The domain name system (DNS) is the internet’s phone book, Use Cases:
providing a mapping between system or network resource names
and IP addresses. DNS has a hierarchical name space that typically IT Ops: DNS server logs provide operations teams with a record
includes three levels: a top-level domain (TLD) such as .com, .edu of traffic, the type of queries, how many are locally resolved either
or .gov; a second-level domain such as “google” or “whitehouse;” from an authoritative server or out of cache, and a picture of
and a system level such as “www” or “mail.” DNS nameservers overall system health.
operate in this hierarchy either by acting as authoritative sources
for particular domains, such as a company or government agency, Security & Compliance: Security teams can use DNS logs to
or by acting as caching servers that store DNS query results for investigate client address requests such as correlating lookups
subsequent lookup by users in a specific location or organization; with other activity, whether requests are made for inappropriate
for example, a broadband provider caching addresses for its or otherwise suspicious sites and relative popularity of individual
customers. sites or domains. Since DNS servers are a frequent target of DDoS
attacks, logs can reveal an unusually high number of requests
from external sources. Likewise, since compromised DNS servers
themselves are often used to initiate DDoS attacks against other
sites, DNS logs can reveal whether an organization’s servers have
been compromised. DNS data can also provide detection of
unknown domains, malicious domains and temporary domains. 

62 63
 NETWORK DATA
NETWORK ACCESS
CONTROL (NAC)
Use Cases: Security & Compliance
Examples: Aruba ClearPass, Cisco ACS

Network access or admission control is a form of Use Cases:

client/endpoint security that uses a locally installed software
agent to pre-authorize connections to a protected network. NAC Security & Compliance: NAC software collects data about the
screens client devices for contamination by known malware and connecting clients such as an inventory of installed client software,
adherence to security policies such as running an approved OS compliance with security policies, OS and application patch
with the most recent patches. Clients failing NAC screens are versions, accessibility by remote access clients and user access
rerouted to an isolated quarantine network until any detected to protected networks. NAC logs provide security teams with a
problems are corrected. detailed profile of a client’s state and activity. It can provide details
into unauthorized device connections and be used to correlate
users/IP to a physical network location.

64 65
 NETWORK DATA
NETWORK SWITCHES
Use Cases: IT Operations, Security & Compliance
Examples: Ethernet Switch, Virtual Switches

Switches are network intersections, places where packets move Use Cases:
from one network segment to another. In their purest form,
switches work within a particular IP subnet and can’t route Layer IT Ops: Operations teams use switch logs to see the state of traffic
3 packets to another network. Modern data center designs flow, such as source and destination, class of service and causes
typically use a two-tier switch hierarchy: top-of-rack (ToR) of congestion. Logs can show traffic statistics in the aggregate,
switches connecting servers and storage arrays at the edge, by port and by client, and whether particular ports are congested,
and aggregation or spine switches connecting to the network failing or down.
core. Although ethernet switches are far more widespread, some
organizations also use fiber channel or infiniband for storage Security & Compliance: Switch data, often captured as NetFlow
area networks or HPC interconnects, each of which has its own records, is a critical data source for flagging advanced persistent
type of switch. threats, analyzing traffic flows for unusual activity and identifying
potential data exfiltration. As a wire-level data source, switch
statistics are almost impossible to spoof and thus a crucial source
of security data. This data can also be used to correlate users or IP
addresses to a physical network location.

66 67
 NETWORK DATA
NETWORK ROUTERS
Use Cases: IT Operations, Security & Compliance
Examples: Routers from Cisco, Juniper, Linksys, Arista,
Extreme Networks, Avaya

If switches are network intersections, then routers are the signal Use Cases:
lights and traffic cops—the devices responsible for ensuring
that traffic goes to the right network segment. Unlike switches IT Ops: Network engineers use router logs and statistics to
that operate at Layer 2, routers work at Layer 3, directing traffic monitor traffic flow and ensure that traffic is being correctly
based on TCP/IP address and protocol (port number). Routers forwarded between network segments. Data from routing
are responsible for particular Layer 3 address spaces and manage protocol updates can show whether your routers are appropriately
traffic using information in routing tables and configured policies. exchanging route tables with other locations, that external traffic
Routers exchange information and update their forwarding tables can reach you, and that internal traffic is correctly forwarded to
using dynamic routing protocols. external routers.

Security & Compliance: Routers collect the same sort of traffic

logs and statistics as switches; thus, their data is equally valuable
to security teams as a source for flagging advanced persistent
threats, analyzing traffic flows for unusual activity and identifying
potential data exfiltration. As a wire-level data source, router
statistics are almost impossible to spoof and thus a critical
source of security data. Router data can also be used to detect
configuration changes and error or failure alerts correlating with
security indicators.

68 69
 NETWORK DATA
NETWORK PROTOCOLS
Use Cases: IT Operations, Security & Compliance
Examples: HTTP, Cisco NetFlow, Ntop, Flow-tools, FlowScan,
EHNT, BPFT

Network protocols describe the structure of data that flows Use Cases:
through networks. In most cases, network ports are assigned to
specific protocols for both security and performance reasons. IT Ops: Network protocol traffic analysis can help determine
Some protocols operate at a lower level of the computing stack the network’s role in overall availability and performance of
and are used to direct packet routing, such as TCP, UDP or IP. critical services. Application traffic can be monitored for usage,
Other protocols, such as HTTP, HTTPS and TNS describe how performance and availability, and can provide visibility into
packets are structured for applications – such as web services, specific user data. For applications that cannot be instrumented
databases and a wide range of client-based applications. By on the servers, network traffic may be the only way to acquire
capturing, decrypting and analyzing network protocol data, performance data.
you can better understand the kinds of applications, their
usage, performance and even payload (content of the data) Security & Compliance: Network protocols are an important
of applications. Since this data can be gathered directly from source for identifying advanced persistent threats, analyzing
a network tap, or with specialized software, it provides a traffic flows for unusual activity and identifying potential data
perspective on applications and how they interoperate that may exfiltration. Aggregating and analyzing flow records also can
not be otherwise available. show anomalous traffic patterns and flow destinations that
are indicative of a breach, such as an APT phoning home to a
command and control server for instructions, additional malware
code, or copying large amounts of data to an attacker’s system.
The data can also be used to detect traffic related to DDoS,
malicious domains, and unknown domains or locations.

70 71
 NETWORK DATA
PROXIES
Use Cases: IT Operations, Security & Compliance
Examples: Blue Coat, Fortinet, Juniper IDP, Netscreen Firewall,
Palo Alto Networks, nginx

Network proxies are used in several ways in IT infrastructure: Use Cases:

as web application accelerators and intelligent traffic direction,
application-level firewalls and content filters. By acting as a IT Ops: Operations teams often use proxies embedded in an
transparent ‘bump-in-the-wire’ intermediary, proxies see the entire application delivery controller (ADC), a more advanced Layer
Layer 7 network protocol stack, which allows them to implement 7-aware version of a load balancer. In this context, proxy logs
application-specific traffic management and security policies. can provide information about incoming requests and traffic
distribution among available resources.

Security & Compliance: Proxy records can identify details about

specific content traversing network control points including file
names, types, source and destination, and metadata about the
requesting client such as OS signature, application and username/
ID (depending on the proxy implementation). The data can also
be used to help detect command and control traffic, malicious
domain traffic and unknown domain traffic.

Web proxies and some next generation firewalls may act in a

transparent or explicit mode communicating with HTTP(s)
servers on behalf of a client. Using a number of related
technologies, the request and response can be inspected and
permitted, or blocked, based on user role, site or resource
category or attack indicator. Data logged in the events can
potentially be used in detective correlation.

72 73
 NETWORK DATA
VoIP
Use Cases: IT Operations, Security & Compliance
Examples: Asterisk CDR, Asterisk event, Asterisk messages

Voice over IP refers to several methods for transmitting real-time Use Cases:
audio and video information over an IP-based data network. Unlike
traditional phone systems using dedicated, point-to-point circuits, IT Ops: VoIP logs provide troubleshooting and usage data
VoIP applications use packet-based networks to carry real-time similar to that of other network applications. Details include
audio streams that are interspersed with other ethernet data source, destination, time and duration of calls, call quality metrics
traffic. Since TCP packets may be delivered out of order due to (e.g., packet loss, latency, audio fidelity/bit rate) and any error
data loss and retransmission, VoIP includes features to buffer and conditions. Integrating VoIP source/destination records with an
reassemble a stream. Similarly, VoIP packets are usually tagged employee database such as AD or LDAP and a DHCP database
with quality of service (QoS) headers to prioritize their delivery allows linking call records to actual people and IP addresses to
through the network. physical locations; information that can assist in troubleshooting
and billing.

Security & Compliance: VoIP deployments may expose

organizations to potential security threats, and analyzing VoIP
logs can help identify and prevent these exploits.

74 75
 OPERATING SYSTEM DATA
SYSTEM LOGS
Use Cases: IT Operations, Application Delivery, Security & Compliance
Examples: Unix, Windows, Mac OS

Every OS records details of its operating conditions and Use Cases:

errors, and these time-stamped logs are the fundamental and
authoritative source of system telemetry. Depending on the OS, IT Ops & Application Delivery: System logs often are the first
there may be separate logs for different classes of events, such as place operations teams turn when troubleshooting system
routine informational updates, system errors, boot loader records, problems, whether with the OS, hardware or various I/O interfaces.
login attempts and debug output. Error logs often aggregate Since a particular problem often manifests itself with errors in
records from multiple subsystems and OS services or daemons, multiple subsystems, correlating log entries is one of the best
and, thus, are a definitive source of troubleshooting information. ways of identifying the root cause of a subtle system failure.

Security & Compliance: System logs include a variety of security

information such as attempted logins, file access and system
firewall activity. These entries can alert security teams to network
attacks, a security breach or compromised software. They also
are an invaluable source of information in forensic analysis of a
security incident. For example, the data can be used to identify
changes in system configurations and commands executed by
users or privileged users.

76 77
 OPERATING SYSTEM DATA
SYSTEM PERFORMANCE
Use Cases: IT Operations, Application Delivery, Security & Compliance
Examples: PERFMON, Windows Events Logs, sar, vmstat, iostat

Measures of system activity such as CPU load, memory and disk Use Cases:
usage, and I/O traffic are the IT equivalent of EKGs to a doctor:
the vital signs that show system health. Recording these measures IT Ops & Application Delivery: Performance logs provide a
provides a record of system activity over time that shows normal, real-time indication of system health by showing resource usage
baseline levels and unusual events. By registering myriad system that, when compared with historical norms, flags performance
parameters, performance logs also can highlight mismatches problems. When measurements deviate from standard or typical
between system capacity and application requirements, such parameters, it’s a warning for IT admins to do further investigation.
as a database using all available system memory and frequently
swapping to disk. Security & Compliance: While primarily used for keeping
infrastructure up and running, monitoring system performance can
also be used to uncover potential security incidents by detecting
abnormal activity in performance. One example is abnormal
system resource usage in correlation with a security indication.

78 79
 VIRTUAL INFRASTRUCTURE DATA
AMAZON WEB
SERVICES
Use Cases: IT Operations, Security & Compliance
Examples: CloudTrail, CloudWatch, Config, S3

AWS is the largest and most widely used public cloud Use Cases:
infrastructure, providing on-demand compute, storage, database,
big data and application services with consumption-based IT Ops: AWS services provide similar types of system and
pricing. AWS can be used to replace traditional enterprise virtual service data as traditional IT infrastructure, much of which is
server infrastructure in which software runs on individual virtual consolidated by the CloudWatch service. These include service
machines (VM) or to host cloud-native applications built from monitoring, alarms and dashboards for metrics, logs, and events
a collection of AWS services. AWS includes a host of service generated by other AWS resources and applications. Typical
management, automation, security, network and monitoring events and measures include when instances are instantiated
services used to deploy, scale, decommission, audit and administer and decommissioned, CPU usage, network traffic and storage
one’s AWS environment, subscriptions and hosted applications. consumption.

Security & Compliance: Security data from AWS services includes

login and logout events and attempts, API calls and logs from
network and web application firewalls.

80 81
 VIRTUAL INFRASTRUCTURE DATA
MICROSOFT AZURE
Use Cases: IT Operations, Security & Compliance
Examples: WADLogs, WADEventLogs, WADPerformanceCounter,
WADDiagnostInfrastructure

Azure is a popular and widely used public cloud infrastructure, Use Cases:
providing on-demand compute, storage, database, big data and
application services with consumption-based pricing. Azure IT Ops: Azure services provide detailed logs for monitoring one’s
can be used to replace traditional enterprise virtual server infrastructure across entire technology stack, VMs, containers,
infrastructure in which software runs on individual VMs, or to host storage and application services. The data is useful in maintaining
cloud-native applications built from a collection of Azure services. application delivery quality and service levels, measuring user
Azure includes a host of service management, automation, behavior, resource utilization and for capacity planning and cost
security, network and monitoring services used to deploy, scale, management.
decommission, audit and administer one’s Azure environment,
subscriptions and hosted applications. Security & Compliance: Security teams can use Azure service
logs to audit and attest to compliance with established policies.
Log data also is invaluable for incident forensic analysis, such
as identifying unauthorized access attempts from access
logs, tracking resources and configuration change events and
identifying vulnerabilities in hosts or firewalls.

82 83
 VIRTUAL INFRASTRUCTURE DATA
VMware SERVER LOGS,
CONFIGURATION DATA AND
PERFORMANCE METRICS
Use Cases: IT Operations, Security & Compliance
Examples: vCenter, ESXi

VMware vSphere ESXi is the most commonly used enterprise and over 100 other metrics fall into this category. As with the
server virtualization platform. The VMware management inventory information, this information is not present in the
platform, whether one of the vSphere products, vCloud or log files and must be viewed through the vSphere client or
standalone hypervisor, produce a variety of data, fall into four
polled through the vSphere API.
main categories:

• vCenter logs - vCenter is the “control center” of a vSphere

Use Cases:
environment. The vCenter logs show information including:
IT Ops: Operations teams can use VMware data to measure the
who is logging in to make changes, which individuals made
health of the overall hypervisor environment and underlying guest
changes and authentication failures.
operating systems. Admins can use this data for capacity planning
• ESXi logs - every vSphere environment includes one or more and for troubleshooting of ongoing performance issues, such as
datastore latency issues.
ESXi hypervisors; these are the systems that host the virtual
machines. ESXi logs contain information that is useful when
This data also records hardware resource usage that can be used
troubleshooting hardware and configuration issues.
to optimize VM deployments across a server pool to maximize
• Inventory information - the vCenter environment tracks resource consumption without having workloads overwhelm any
given server.
configuration about a number of items including: hypervisors,
virtual machines, datastores, clusters and more. This includes
Security & Compliance: The uncoupled nature of virtual resources
the configuration of each item, and how a given item relates
and underlying physical hardware can cause complex challenges
to any other. This information is not represented in the log during incident investigations, capacity analyses, change
files from either the vCenter or ESXi servers. This information tracking and security reporting. One common security use case
can be viewed using the vSphere client or by using vSphere for VMware data comes from the vCenter logs, which audit the
APIs to pull this information. In both cases this information is activity of individuals using the vSphere interface to re-assign user
permissions within the VMware environment. 
pulled from the vCenter servers.

• Performance information - for each configuration item, the

vCenter server tracks a number of performance metrics about
that item. Datastore latency, virtual or physical CPU utilization,

84 85
 PHYSICAL INFRASTRUCTURE & IoT DATA
PHYSICAL CARD
READERS
Use Cases: Security & Compliance

Most organizations use automated systems to secure physical Use Cases:

access to facilities. Historically, these have been simple magnetic
strips affixed to employee badges; however, locations with Security & Compliance: For IT security teams, the data from card
stringent security requirements may use some form of biometric readers provide the same sort of access information for physical
reader or digital key. Regardless of the technology, the systems locations as a network firewall log. The data can be used to detect
compare an individual’s identity with a database and activate attempted breaches and be correlated to system and network logs
doors when the user is authorized to enter a particular location. to identify potential insider threats and provide overall situational
As digital systems, badge readers record information such as awareness. It can also be used to detect access at unusual times
user ID, date and time of entry and perhaps a photo for each and locations or for unusual durations.
access attempt.
 

86 87
 PHYSICAL INFRASTRUCTURE & IoT DATA
SENSOR DATA
Use Cases: IT Operations, Security, Business Analytics, Internet of Things
Examples: Binary and numeric values including switch state, temperature,
pressure, frequency, flow, from MQTT, AMQP and CoAP brokers,
HTTP event collector

Industrial equipment, sensors and other devices often have Use Cases:
embedded processors and networking that allows them
to record and transmit a vast array of information about IT Ops: Some of the most important parameters for operations
operating conditions. Regardless of device, their data provides teams to monitor are environmental conditions such as
unprecedented detail about performance parameters and temperature, humidity, airflow and voltage regulation in a data
anomalies that can indicate larger problems—for example, a center. Similar readings are available from individual servers and
device ready to fail or issues with another system. Aggregating network equipment that, when correlated, can highlight problems
and correlating data from multiple devices and subsystems in the facility or equipment ready to fail.
provides a complete picture of equipment, system, factory or
building performance. Security & Compliance: Sensor data can help protect mission-
critical assets and industrial systems against cybersecurity threats
by providing visibility into system performance or set points that
could put machines or people at risk. Data can also be used to
satisfy compliance reporting requirements.

Internet of Things:

Preventative Maintenance and Asset Lifecycle Management:

Sensor data can provide insights into asset deployment, utilization
and resource consumption. Operational data can also be used to
proactively approach long-term asset management, maintenance
and performance.

Monitoring and Diagnostics: Monitoring sensors can help ensure

that equipment in the field operates as intended, for example,
monitoring and tracking unplanned device or system downtime.
The data can also be used to understand the cause of failure on
a device to improve efficiency and availability, and to identify
outliers and issues in device production or deployment.

88 89
 PHYSICAL INFRASTRUCTURE & IoT DATA
SERVER LOGS
Use Cases: IT Operations, Security & Compliance, Application Delivery

Server operating systems routinely record a variety of operational, Use Cases:

security, error and debugging data such as system libraries loaded
during boot, application processes open, network connections, file IT Ops & Application Delivery: Server logs provide a detailed
systems mounted and system memory usage. The level of detail record of overall system health and forensic information about the
is configurable by the system administrator; however, there are exact time of errors and anomalous conditions that are invaluable
sufficient options to provide a complete picture of system activity in finding the root cause of system problems.
throughout its lifetime. Depending on the subsystem, server logs
are useful to system, network, storage and security teams. Security & Compliance: Server logs include data from security
subsystems such as the local firewall, login attempts and
file access errors that security teams can use to identify
breach attempts, track successful system penetrations
and plug vulnerabilities. Monitoring server logs such as file
access, authentication and application usage can help secure
infrastructure components.

90 91
 PHYSICAL INFRASTRUCTURE & IoT DATA
BACKUP
Use Cases: IT Operations

Despite the use of data replication to mirror systems, databases Use Cases:
and file stores, data backup remains an essential IT function by
providing for long-term, archival storage of valuable information, IT Ops: Backup systems routinely log activity and system
much of which has legal and regulatory requirements regarding conditions, recording information such as job history, error
its preservation. Backups also can be used to store multiple conditions, backup target and a detailed manifest of copied files
versions of system images and data, allowing organizations to or volumes. This data allows operations teams to monitor the
reverse changes, accidental deletions or corrupted data quickly, health of backup systems, software and jobs; triggers alerts in the
restoring the system or database to a known good state. Backup case of errors; and assists in debugging backup failures. It also
software can use different types of storage media depending on allows teams to locate where specific data may be stored, when a
the likelihood of needing the data: external disks or virtual tape recovery is required.
libraries for active data and tape, optical disks or a cloud service
for long-term storage.

92 93
 PHYSICAL INFRASTRUCTURE & IoT DATA
STORAGE
Use Cases: IT Operations
Examples: EMC, Netapp, IBM

Data center storage is provisioned in two general ways: built into Use Cases:
servers and shared using various network storage protocols, or
via a dedicated storage array that consolidates capacity for use IT Ops: Shared storage logs record overall system health (both
by multiple applications that access it using either a dedicated hardware and software), error conditions (such as a failed
storage area network (SAN) or ethernet LAN file-sharing controller, network interface or disks) and usage (both capacity
protocol. The activity of internal, server-based storage is typically used per volume and file or volume accesses). Collectively, the
recorded in system logs, however storage arrays have internal information can alert operations teams to problems, the need for
controllers/storage processors that run a storage-optimized more capacity and performance bottlenecks.
OS and log a plethora of operating, error and usage data. Since
many organizations have several such arrays, the logs often are
consolidated by a storage management system that can report on
the aggregate activity and capacity.

94 95
 PHYSICAL INFRASTRUCTURE & IoT DATA
MAINFRAME
Use Cases: IT Operations, Security & Compliance

Mainframes are the original business computer: large, centralized Use Cases:
systems housing multiple processors, system memory (RAM)
and I/O controllers. Despite their 60-year legacy, mainframes IT Ops: Like other servers, mainframes measure and log numerous
still are widely used for mission-critical applications, particularly system parameters that show their current status, configuration
transaction processing. Although they usually run a proprietary and overall health. Since most mainframe subsystems are
OS, mainframes also can be virtualized to run Unix and Linux or, redundant, system logs also show non-disruptive hardware failures
with add-on processor cards, Windows Server. Mainframes are or anomalous behavior that is predictive of an impending failure.
valued for their bulletproof reliability and security, using highly Due to their use for critical applications, mainframes often record
redundant hardware and resilient, stringently tested software. application performance data such as memory usage, I/O and
As such, they appeal to organizations wanting to consolidate transaction throughput, processor utilization and network activity.
workloads onto a small number of systems and that need the
added reliability and versatility. Security & Compliance: Mainframes contain critical operational
data. In a security context, mainframe data is treated like any
other enterprise data that requires visibility and monitoring for
data confidentiality and integrity, compliance and audits with
regulatory requirements, and for access monitoring.

96 97
 PHYSICAL INFRASTRUCTURE & IoT DATA
PATCH LOGS
Use Cases: IT Operations, Security & Compliance

Keeping operating systems and applications updated with the Use Cases:
latest bug fixes and security patches is an essential task that can
prevent unplanned downtime, random application crashes and IT Ops: Operations teams use patch logs to verify the timely and
security breaches. Although commercial apps and operating correct application of scheduled patches, identify unpatched
systems often have embedded patching software, some systems and applications, and alert to errors in the patching
organizations use independent patch management software process. Correlating errors to patch logs can indicate when an
to consolidate patch management and ensure the consistent error is due to a patch.
application of patches across their software fleet and to build
patch jobs for custom, internal applications. Security & Compliance: Security teams can use patch logs to
monitor system updates and determine which assets could be at
Patch management software keeps a patch inventory using a risk, due to failed or out-of-date patches.
database of available updates and can match these against an
organization’s installed software. Other features include patch  
scheduling, post-install testing and validation and documentation
of required system configurations and patching procedures.

98 99
 PHYSICAL INFRASTRUCTURE & IoT DATA
TELEPHONY
Use Cases: IT Operations
Examples: Cisco Unified Communications Manager, Shoretel, Twilio

Real-time business communications are no longer limited to Use Cases:

voice calls provided by plain old telephone service (POTS);
instead, voice, video, text messaging and web conferences are IP IT Ops: Like VoIP, telephony logs provide an overview of system
applications delivered over existing enterprise networks. Unlike health along with troubleshooting and usage data similar to that
traditional client-server or web applications, telephony and other of other network applications. Details include source, destination,
communications applications have strict requirements on network time and duration of voice/video calls, web conferences and
quality of service, latency and packet loss, making service quality text messages, call-quality metrics (e.g., packet loss, latency,
and reliability much more sensitive to network conditions and audio fidelity/bit rate), error conditions and user attendance at
server responsiveness. Traditional POTS has conditioned people web conferences. By integrating telephony records of source/
to expect immediate dial tone when picking up the phone and destination address with an employee database such as AD or
be intolerant of noise, echo or other problems that can plague LDAP and a DHCP database, organizations can link call records to
IP telephony; as such, the systems and supporting infrastructure actual user IDs and IP addresses to physical locations; information
require careful monitoring and management to assure quality and that can assist in troubleshooting and billing. Logs also can
reliability. reveal any network segments experiencing congestion or other
performance problems that may indicate equipment problems or
the need for an upgrade.

100 101
 PHYSICAL INFRASTRUCTURE & IoT DATA
POINT-OF-SALE SYSTEMS
(POS)
Use Cases: Internet of Things, Business Analytics, Security & Compliance
Examples: IBM, LightSpeed, NCR, Revel Systems, Square, Toshiba, Vend

Point-of-sale (POS) systems are most often associated with Business Analytics: POS system contain information about what’s
transactions generated at a retail outlet. However, thanks to the sold, how it’s paid for, as well as the pace at which it’s being sold.
rise of mobile POS solutions, many of these systems are starting to Organizations can use this data to monitor revenue in real time,
be deployed in temporary locations, such as a community fair or a which can feed into how to better market 1:1 against customers,
high school event. track product placement and sales in a store, or detect potentially
fraudulent transactions in real time. This type of real-time big
The typical POS system incorporates a cash register based on data analysis can have profound impact on cross- and up-sell
a PC or embedded system, monitor, receipt printer, display, opportunities. POS data also delivers visibility into the customer
barcode scanner and debit/credit card reader. Machine data experience, such as which coupons are most popular or the
generated by POS systems provides organizations with real-time combinations of products that are selling together. When enriched
insight into everything from what’s sold, to the amount of cash with geolocation data, it can also drive valuable insights into
being generated per transaction, to what payment methods are location-based analytics.
being used.
Security & Compliance: POS systems are typically used for
financial transactions and are often targeted since they contain
Use Cases: account, payment and financial information. Because the POS
transaction information is highly sought after for its value to
Internet of Things: Historically, POS systems were either not
attackers, and the POS can be used as an entry point to the
connected or managed on a dedicated private network. Thanks
network, it’s critical to protect these systems. Furthermore, POS
to the rise of the Internet of Things (IoT), these systems are
systems are usually unmanned, run an underlying operating
being connected directly to cloud platforms that make remotely
system, and versioning/monitoring typically fall outside of IT’s
administering these devices from a central location much simpler.
purview – adding additional complexity to their security. Visibility
There’s no longer a need to dispatch IT personnel to manually
and analysis of POS systems and data can provide insights that
update each system. This is critical because a POS failure can
are critical to protecting financial information, detecting fraud and
result in longer lines that inconvenience customers and potentially
securing vulnerabilities. 
lead to lost revenue. A negative customer experience can easily
translate to customers opting to shop somewhere else in a retail
industry that is intensely competitive.

102 103
 PHYSICAL INFRASTRUCTURE & IoT DATA
RFID/NFC/BLE
Use Cases: Internet of Things, Business Analytics
Examples: Alien Technology, BluVision, Check Point Systems, Gimbal, MonsoonRF,
Radius Networks, STMicroelectronics, TAGSYS RFID, ThingMagic

RFID, NFC and BLE are the three primary wireless methods Use Cases:
organizations use today to keep track of objects and interact with
customers in retail stores. NFC is a subset of RFID and is designed Internet of Things: RFID is arguably one of the first instances
to be a more secure form of data exchange, and allows devices of an Internet of Things (IoT) application. Deployed in place of
to communicate peer-to-peer. Common use cases of RFID are traditional barcode readers, RFID tags are used in everything from
asset tracking, inventory management, even attendee tracking. shipping to keeping track of farm animals. IoT deployments make
NFC is commonly used for contact-less payments, exchanging it possible to capture RFID data in way that makes it simpler to
information between two parties (such as smartphones), and even track events involving anything that has an attached RFID tag.
badge readers that unlock doors. Data insights from RFID can help improve overall supply chain,
order processing and inventory management.

At the same time, organizations are adopting Bluetooth Low

Energy (BLE) wireless connectivity solutions that can broadcast BLE, meanwhile, is used to engage customers more directly as
signals to other devices. BLE is used most widely in beacons that they move about a specific location, which in turn creates data
are employed, for example, to inform shoppers of new sales in that can be used to optimize the customer experience.
retail stores on their smartphones or to update fans on events that
might be occurring during a sporting event. Business Analytics: Whether it’s inventory tracked using RFID
tags or customers and employees moving around specific
locations, new classes of analytics applications are using the
data generated by these devices to serve up actionable business
insights in near real time. Retailers can leverage this data for
several use cases, such as making sure that inventory is located
as close as possible to the locations where customers are most
likely to want to purchase.

104 105
 PHYSICAL INFRASTRUCTURE & IoT DATA
SMART METERS
Use Cases: Internet of Things, Business Analytics
Examples: ABB, GE, Google, eMeter, IBM, Itron, Schneider Electric, Siemens

Smart meters record consumption of energy, usage of water, or Use Cases:

usage of natural gas so that the information can be continually
processed and shared. Typically, smart meters allow for bi- Internet of Things: Smart meters are deployed across critical
directional communication in real time in a way that allows a systems at large utilities companies, for example, power, gas and
gauge of some type to be adjusted. water utilities. These systems are the lifeblood of infrastructure
and failure can lead to catastrophic outcomes. Real time
monitoring of smart meters can help organizations better analyze
failures remotely, by way of remotely detecting line down failures.
Equally important is securing the devices from tampering that
could lead to malicious attacks and breaches.

Energy companies and water utilities make extensive use of smart

sensors to track everything from oil reserves to the quality of the
water supply.

Business Analytics: A wide variety of industries are applying

analytics to the data being collected by smart meters to optimize
service. For example, an oil or gas company no longer needs
to physically send a worker to a location to read a meter. The
provider already knows how much fuel has been consumed and
how much remains.

In the future, smart meters will be used in everything from

modern traffic control systems to defense systems designed to
protect critical infrastructure. Aggregating data from these smart
meters can give utilities critical insights into the demand. Heavily
regulated utilities are required to meet established SLA’s during
demand response events, and machine data from smart meters
can drive visibility into how they are responding.

106 107
 PHYSICAL INFRASTRUCTURE & IoT DATA
TRANSPORTATION
Use Cases: Internet of Things, Business Analytics
Examples: Boeing, BMW, Ford, GE, General Motors, Daimler-Benz,
John Deere, Volkswagen

Vehicles of all sizes and types generate massive amounts of Use Cases:
machine data. This data can be used to gain real-time visibility into
the health and performance of the vehicle and to drive predictive Internet of Things: Vehicle manufacturers are attaching sensors to
maintenance applications. Armed with that data, an airplane or every mechanical and electronic component they use. This allows
automobile manufacturer can follow a maintenance regime that is companies to gain a unified view of assets to quickly identify
more data driven than driven “by the book.” and diagnose operational issues and to monitor, track and avoid
unplanned asset downtime. This helps to ensure that equipment
is operating as intended. Manufacturers can also detect anomalies
That information can then be used to improve availability and
and deviations from normal behavior to take corrective action –
reliability, and extend the lifecycle of a vehicle that has not been
improving uptime, asset reliability and longevity.
extensively used or, conversely, replace components that have
seen extensive wear and tear sooner.
Business Analytics: With access to machine data, vehicle
manufacturers are applying analytics in ways that fundamentally
change their business models. Instead of selling a vehicle,
manufacturers increasingly prefer to lease vehicles based on
actual usage. The longer that vehicle can be used between repairs,
the more profitable that leasing service becomes. The key to
providing this type of service economically is advanced analytics,
which are applied to all the aggregate data that’s collected.

108 109
 PHYSICAL INFRASTRUCTURE & IoT DATA
MEDICAL DEVICES
Use Cases: Internet of Things, Business Analytics, Security & Compliance
Examples: Abbot Laboratories, Apple, Baxter, Boston Scientific, GE,
Siemens, St. Jude Medical

Everything from intensive care units to wearable devices Use Cases:

generates multiple types of machine data. In fact, just about every
aspect of patient care inside and out of a hospital setting can be Internet of Things: Most devices inside a hospital are connected
instrumented. While the primary goal is to save lives, a crucial to local monitoring applications. But it’s possible to monitor
secondary goal is to reduce healthcare costs by reducing the both patient care remotely using sensors that communicate with either
the number of potential visits to a hospital as well as the length a wearable device or some other system for monitoring patients in
of stay. their homes.

Business Analytics: Machine data also makes it simpler for

medical professionals to analyze both patient and anonymous
data across a broader range of geographically distributed
regions—for example, to see how certain diseases are affecting
one group of people more than another. These insights can also be
used to help improve patient experience and deliver better care.

Security & Compliance: Medical devices run on operating systems

and applications, and since they run in a controlled manner, the
devices can be behind on patching levels making them susceptible
to vulnerabilities. Analyzing this data can provide visibility into
potential malicious activity or compromised protected health
information (PHI).

110 111
 PHYSICAL INFRASTRUCTURE & IoT DATA
ENVIRONMENTAL
SENSORS
Use Cases: Internet of Things, Business Analytics
Examples: Bosch Sensortec, Mouser Electronics, Raritan, Schneider Electric,
TSI, Vaisala

Environmental sensors provide data on barometric air pressure, Use Cases:

humidity, ambient air temperature and air quality. They are applied
in everything from combating pollution and detecting gasses to Internet of Things: Environmental sensors are a class of smart
keeping data centers from overheating. meters that have been optimized to monitor the environment.
In some instances, such as a data center, the information provided
by these sensors is used to automatically alter temperature
setting and heat flow.

Business Analytics: Environmental sensor data collect can be used

in retail applications to answer predictive questions, such as “what
impact will inclement weather have on foot traffic in a mall?”

112 113
 PHYSICAL INFRASTRUCTURE & IoT DATA
INDUSTRIAL CONTROL
SYSTEMS (ICS)
Use Cases: Internet of Things, Business Analytics, Security & Compliance
Examples: ABB, Emerson Electric, GE, Hitachi, Honeywell, Rockwell Automation,
Siemens, Toshiba

Within the context of a manufacturing environment, industrial Use Cases:

control systems make use of programmable logic controllers to
both acquire data and execute supervisory functions. Much of Internet of Things: Machine data from ICS can be used to gain
the process automation employed in a manufacturing facility is real-time visibility into the uptime and availability of critical assets.
enabled by the industrial control systems. This enables companies to detect an issue, perform root cause
analysis and take preventive action to prevent certain events from
happening in the future. Companies are also leveraging machine
data from ICS systems to secure these mission-critical assets.

Business Analytics: Organizations can apply machine learning

algorithms against the machine data created by industrial control
systems to increase productivity, uptime and availability. ICS data
can also drive visibility into complex manufacturing processes,
helping identify bottlenecks and remove inefficiencies.

Security & Compliance: Industrial control systems play a critical

role in delivering services to industry and municipalities across
the world. These systems live on top of traditional IT infrastructure
and – while typically separate from enterprise IT – digital
transformation is driving organizations to provide connectivity
to these systems, increasing exposure to attacks. These systems
tend to be unmanned from a security perspective. Regardless of
how ICS might get attacked or infected, data from ICS devices
can provide visibility and can be used to analyze and identify
malicious activity and potential threats. This visibility enables
companies to measure impact and risk, and associate them with
business processes.

114 115
 PHYSICAL INFRASTRUCTURE & IoT DATA
WEARABLES
Use Cases: Internet of Things, Business Analytics
Examples: ARM, Intel, Lenovo, Microsoft, Samsung

From smartwatches that double as fitness aids to medical Use Cases:

devices that enable physicians to remotely monitor vital statistics,
wearable devices have proven they are here to stay. Wearable Internet of Things: Beyond merely syncing with smartphones,
devices are one of the most recognizable parts of the Internet the latest generation of smartwatches is taking advantage of
of Things. geo-positioning systems and application programming interfaces
to give device owners an optimal application experience that
includes both their location and often time of day.

Going forward, there soon will be whole new classes of

wearable devices taking advantage of everything from virtual
reality applications delivered via a headset to sensors embedded
in the latest fashion. Use cases include IoT security to ensure
that wearable data is secure and personal information is
not compromised.

Business Analytics: As more people become comfortable

with sharing data via wearable devices, many are experiencing
the power of analytics firsthand. Developers of applications
optimized for wearables are making recommendations concerning
everything from how to improve life expectancy to where to find a
meal. Analytics from wearables can help improve user experience
and drive product innovation. For example, product managers can
understand how consumers are interacting with devices to build
better features.

116 117
 ADDITIONAL DATA SOURCES
DATABASE
Use Cases: IT Operations, Application Delivery, Security & Compliance
Examples: MySQL, Postgres, Other Relational Databases

Databases are the fundamental elements of information collection, Use Cases:

storage and analysis of digital information. Databases are
categorized as either relational, in which data is organized in IT Ops & Application Delivery: Database logs can be aggregated
spreadsheet-like tables of columns and rows, or NoSQL (non- and analyzed to show the overall performance of a particular
relational), where information is organized purely by columns database system, and also provide visibility into database issues.
(column store) as key-value pairs, by unstructured documents or Metrics useful to IT operations teams include queries per second
interconnected graphs linking related data elements. and query response time, both measured against a baseline
standard made from historical data.

Security & Compliance: Database logs provide security teams

information about the accounts or systems accessing tables
or other database elements. Correlating database access and
transaction logs with identity management system records
can flag unauthorized access or access attempts to databases.
Database logs can also expose security holes such as open ports
or dormant, unused admin accounts, and help identify abnormal
queries or users, and abnormal database/table access.

118 119
 ADDITIONAL DATA SOURCES
THIRD-PARTY LISTS
Use Cases: Security & Compliance
Examples: Threat Lists, OS Blacklist, IP Blacklists, Vulnerability Lists,
Google Analytics

One of the methods that IT security vendors use to detect and Use Cases:
flag security problems is one or more database of known threats
and vulnerabilities. These include malware code signatures, OS Security & Compliance: By aggregating data from users around
and application patch versions, the source IP address of previous the world, third-party security lists provide security teams with
attacks and spam and reputation databases using real-time real-time information about nascent threats and vulnerabilities
aggregation of malware, spam and compromised websites that allow updating security policies, firewall rules and vulnerable
collected from millions of users. Third-party lists provide an early- software before an attack. Lists also are used to identify known
warning system for new methods or sources of attack. sources of spam, both commercial and malware-infested, to
improve the effectiveness of filters on internal email systems.

120 121
 ADDITIONAL DATA SOURCES
SOCIAL MEDIA FEEDS
Use Cases: IT Operations

Social networks are some of the most heavily trafficked sites Use Cases:
on the internet. By allowing users to communicate and share
information among friends and colleagues, social media has IT Ops: Due to their interactivity, convenience and ubiquity,
become an important outlet for news, entertainment, photo- social media feeds provide organizations with an unfiltered and
sharing and real-time reaction to public events. As such, social instantaneous view of customer opinion. By analyzing feeds from
media feeds are an increasingly effective advertising medium and the most popular sites, organizations can quickly identify potential
source of customer contact, feedback and support. problems with a product or service, mishandled customer support
incidents or other sources of customer dissatisfaction about an
organization’s products or online presence. Proactively addressing
these online complaints allows the organization to turn unhappy
and potentially lost customers into delighted and loyal ones.

122 123
 ADDITIONAL DATA SOURCES
HUMAN RESOURCES
Use Cases: Security & Compliance
Examples: BambooHR, Fairsail HRMS, Namely, Zenefits

Human Resources records include information relating to the Use Cases:

entire employee life cycle. HR records provide the definitive
source of employee information for identity management systems Security & Compliance: HR records can show if someone no
and enterprise directories, making them an important source longer employed still has active accounts, and can also provide
for authentication and authorization data. Although HR data evidence of disciplinary action that might be useful in security
traditionally has been textual, it increasingly includes images and investigations.
biometric information such as an employee’s portrait, fingerprints
and iris scans.  

124 125
 ADDITIONAL DATA SOURCES
BUSINESS SERVICE
TRANSACTION & BUSINESS
SERVICE PERFORMANCE DATA
Use Cases: IT Operations, Application Delivery, Security & Compliance
Examples: Payments Status, Batch Upload Status, Customer Order Status

Transaction records provide an auditable trail of activity for every Use Cases:
part of every business process. Whether for financial transactions
such as payments and orders, or tasks such as customer support IT Ops & Application Delivery: IT can use process logs to identify
and service calls, business process logs are required to verify flaws in their support or admin processes, or problems that have
activity in case of disputes, to certify compliance with regulations fallen through gaps in existing process flows.
and terms of service, and to provide detailed evidence of business
transactions. A technique called business process mining uses Security & Compliance: Hackers are good at covering their tracks
sophisticated software to analyze logs and identify process, by altering common log files, but business process logs that track
control, data, organizational and social structures. These might activity across multiple systems used in a particular process can
include mapping the flow of patients through a hospital or highlight anomalies that may indicate security issues.
customer problems through a support organization to optimize
process flow, measure performance and identify outlier incidents
for further investigation.

126 127
p-shopping.com/category.screen?category_id=GIFTS” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
317 27.160.0.0 - - [07/Jan 18:10:56:156] “GET /oldlink?item_id=EST-26&JSESSIONID=SD5SL9FF1ADFF3
1.1” 200 1318 “http://buttercup-shopping.com/cart.do?action=purchase&itemId=EST-26&product_id=K9-
” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)” 468 125.17.14.100
07/Jan 18:10:56:147] “POST /category.screen?category_id=SURPRISE&JSESSIONID=SD9SL4FF4ADFF7 HTTP

BK-Splunk-Essential-Guide-to-Machine-Data-101
© 2017 Splunk Inc. All rights reserved. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data,Splunk Cloud, Splunk
200 2423 “http://buttercup-shopping.com/cart.do?action=addtocart&itemId=EST-16&product_id=RP-
” “Opera/9.20 (Windows NT 6.0; U; en)” 564 130.253.37.97 - - [07/Jan 18:10:55:189] “GET /cart.
tion=changequantity&itemId=EST-18&product_id=AV-CB-01&JSESSIONID=SD5SL7FF6ADFF10 HTTP 1.1” 404
“http://buttercup-shopping.com/oldlink?item_id=EST-18” “Opera/9.20 (Windows NT 6.0; U; en)” 766

Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries.
53.37.97 - - [07/Jan 18:10:55:187] “GET /oldlink?item_id=EST-6&JSESSIONID=SD10SL8FF2ADFF9 HTTP

valuable to everyone. Join millions of passionate users by

ABOUT SPLUNK.
200 3865 “http://buttercup-shopping.com/cart.do?action=purchase&itemId=EST-6&product_id=AV-

Splunk Inc. makes machine data accessible, usable and

” “Opera/9.01 (Windows NT 5.1; U; en)” 553 62.216.64.19 - - [07/Jan 18:10:55:111] “GET /category.
n?category_id=FLOWERS&JSESSIONID=SD8SL8FF1ADFF6 HTTP 1.1” 200 3971 “http://buttercup-shopping.

trying Splunk for free: www.splunk.com/free-trials.

art.do?action=remove&itemId=EST-15&product_id=FL-DSH-01” “Opera/9.01 (Windows NT 5.1; U; en)”
30.253.37.97 - - [07/Jan 18:10:55:108] “GET /category.screen?category_id=SURPRISE&JSESSIONID=SD-
F9ADFF9 HTTP 1.1” 404 474 “http://buttercup-shopping.com/oldlink?item_id=EST-21” “Mozilla/4.0
atible; MSIE 6.0; Windows NT 5.1)” 606 195.69.160.22 - - [07/Jan 18:10:54:192] “GET /cart.do?ac-
remove&itemId=EST-15&product_id=AV-SB-02&JSESSIONID=SD4SL1FF7ADFF7 HTTP 1.1” 200 205 “http://but-
p-shopping.com/cart.do?action=remove&itemId=EST-15&product_id=AV-SB-02” “Mozilla/4.0 (compatible;
6.0; Windows NT 5.1; SV1)” 163 131.178.233.243 - - [07/Jan 18:10:5
] “GET /oldlink?item_id=EST-17&JSESSIONID=SD1SL9FF9ADFF1 HTTP 1.1” 200 1976 “http://butter-
hopping.com/cart.do?action=purchase&itemId=EST-17&product_id=K9-CW-01” “Mozilla/5.0 (Macintosh;
tel Mac OS X 10_6_3; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.38 Safari/533.4”
6.9.190.90 - - [07/Jan 18:10:54:166] “POST /cart.do?action=addtocart&itemId=EST-7&prod-
d=FI-SW-01&JSESSIONID=SD10SL3FF1ADFF4 HTTP 1.1” 404 2258 “http://buttercup-shopping.com/
do?action=addtocart&itemId=EST-7&product_id=FI-SW-01” “Mozilla/4.0 (compatible; MSIE 6.0;
ws NT 5.1; SV1)” 189 82.245.228.36 - - [07/Jan 18:10:54:165] “GET /category.screen?cate-
id=TEDDY&JSESSIONID=SD1SL1FF8ADFF2 HTTP 1.1” 200 488 “http://buttercup-shopping.com/cart.
tion=purchase&itemId=EST-27&product_id=FL-DLH-02” “Mozilla/4.0 (compatible; MSIE 6.0; Windows
1; SV1; .NET CLR 1.1.4322)” 969 128.241.220.82 - - [07/Jan 18:10:54:145] “GET /cart.do?ac-
view&itemId=EST-13&product_id=RP-SN-01&JSESSIONID=SD7SL8FF3ADFF2 HTTP 1.1” 200 1901 “http://but-
p-shopping.com/cart.do?action=view&itemId=EST-13&product_id=RP-SN-01” “Mozilla/4.0 (compatible;
6.0; Windows NT 5.1; SV1)” 393 130.253.37.97 - - [07/Jan 18:10:54:121] “GET /category.screen?-
ory_id=BOUQUETS&JSESSIONID=SD10SL1FF4ADFF10 HTTP 1.1” 406 1437 “http://buttercup-shopping.com/
do?action=addtocart&itemId=EST-27&product_id=AV-SB-02” “Mozilla/5.0 (Macintosh; U; Intel Mac OS X
3; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.38 Safari/533.4” 571 141.146.8.66 -
/Jan 18:10:53:118] “GET /category.screen?category_id=SURPRISE&JSESSIONID=SD7SL3FF9ADFF10 HTTP 1.1”
814 “http://buttercup-shopping.com/category.screen?category_id=SURPRISE” “Mozilla/4.0 (compatible;
6.0; Windows NT 5.1)” 932 141.146.8.66 - - [07/Jan 18:10:53:104] “POST /category.screen?catego-
=BOUQUETS&JSESSIONID=SD3SL1FF7ADFF2 HTTP 1.1” 200 2567 “http://buttercup-shopping.com/product.
n?product_id=AV-SB-02” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)” 920 130.253.37.97 - -
an 18:10:52:182] “GET /category.screen?category_id=TEDDY&JSESSIONID=SD8SL2FF5ADFF2 HTTP 1.1” 200