ESM200-700 StudentGuide
ESM200-700 StudentGuide
181129
ArcSight ESM Administration and Analyst 7.0.0p1
Student Guide ESM200-700p1
November 29, 2018
https://software.microfocus.com/en-us/software/security-operations
Micro Focus. The information contained herein is subject to change without notice.
The only warranties for Micro Focus products and services are set forth in the
express warranty statements accompanying such products and services. Nothing
herein should be construed as constituting an additional warranty. Micro Focus
shall not be liable for technical or editorial errors or omissions contained herein.
Trademark acknowledgments if needed.
Module 0 ‐ Course introduction
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Class Logistics
Facility Emergency Procedures Rules
‐ Alarms/sirens ‐ Make sure to have fun!
‐ Evacuation routes from this room ‐ Ask questions
‐ Off‐site assembly area ‐ Please do not talk during presentations or
demonstrations
Facility information ‐ No cell phones, pagers, or recordings
‐ Restroom ‐ Use Internet during breaks and lunch
Course Hours
‐ Course Hours
‐ Breaks ‐ Approx. every hour
‐ Lunch
Class Introductions
Instructor Introductions
‐ Name
‐ Email
Student Introductions
‐ Name
‐ Company
‐ Role
‐ ArcSight product experiences
‐ Expectations
Course Objectives (1 of 2)
Upon completion of this course, you will be able to:
‐ Discuss where ArcSight ESM fits in a modern day SOC
‐ Describe the basic architecture of an ArcSight ESM installation
‐ Articulate how ArcSight ESM uses both context and content
‐ Use the Event Lifecycle as a framework to become familiar with how ArcSight resources
interact with event data
‐ Identify, analyze, and report on event data using ArcSight ESM
‐ Install, troubleshoot, and update ArcSight context and content
Course objectives (2 of 2)
Upon completion of this course, you will be able to:
‐ Install, troubleshoot, and update ArcSight Context and Content
‐ Use workflow management tools to provide real‐time incident response and escalation tracking
‐ Cases
‐ Annotations
‐ User Management
‐ Build and modify basic reporting within ESM to provide metrics data
‐ Establish ESM peering across multiple ESM instances to
‐ Identify events quickly
‐ Create quick status reports
‐ Provide basic content management
Course agenda (1 of 2)
Topic Duration Day
Course agenda (2 of 2)
Topic Duration Day
Module 10 – User administration Thursday
Customer feedback
A great part of our success is because of YOU and your feedback!
You will receive a course evaluation survey
Feedback is vital to improving our course offerings
Please complete the evaluation
Net Promoter Score (NPS) Customer Satisfaction
10 Great Job 5 (Strongly Agree)
9 Good Job 4 (Agree)
Certificate of completion
A great part of our success is because of YOU
and your feedback!
Please complete the course evaluation
‐ You will receive an email reminder from Micro Focus
Education
‐ Follow the link provided and complete all
questionnaires
‐ You will receive the Certificate of Completion email
upon submitting the evaluation
Questions
Course Registration Contact:
‐ ESP‐[email protected] ‐ Americas
‐ training‐[email protected] ‐ EMEA
‐ esp‐edu‐[email protected] ‐ APJ
Enterprise Security Learning Management System:
‐ Security LMS
Other training delivery methods
‐ eLearning
‐ Virtual Instructor‐led
Any Questions?
10
Slide intentionally left blank
11
Module 1 ‐ ESM Overview
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Objectives
Discuss what ArcSight ESM is and how it fits into a SOC
List the problems ESM can solve
Discuss basic processes to make an ESM installation successful
Describe the basic ArcSight components (10’ ‐ 100,000’ view)
Identify basic user roles within an ArcSight Installation
Topics
What is ArcSight ESM
• Technical Definition
• Problem it Solves
• How it fits into a modern day SOC
Process
• Incident Remediation
• Policy Compliance
• Metrics Reporting
Technology
• Components that make up ESM
• The event flow into ESM
• How new technologies integrate with ESM
People
• SOC Users
• ArcSight SMEs
• Stakeholders
What is ArcSight ESM
4
What is a SOC and what does it do?
Information is needed for the SOC to do its job
What Do We Need to Address These Challenges?
Data Enrichment and Powerful Real‐Time Correlation Solution
ArcSight Enterprise Security Manager (ESM)
Solution Overview
• Enriched data from multiple sources provides more than 400+ event data points
• Increases event data points by more than 4x for through threat detection
• Real‐time data correlation from multiple input sources (integration with ADP)
• Powerful event correlation of up to 75,000 events per second
• Support for large enterprises through multi‐tenancy with centralized console
• Ability to enforce central roles, rights, and responsibilities permissions matrix
• Simplified SOC workflow and triage management through ArcSight Command Center
• Rule development and continued improvement of rule‐based threat detection engine
9
HPE CONFIDENTIAL
So what is needed to successful?
People
‐ Highly Skilled individuals who work at the SOC
Technology
Process
‐ A series of mature, repeatable steps for
accomplishing something
Process
Technology
People
‐ Tools that provide data or responses to
accomplish the SOC’s objectives
10
People
11
People: Classic ArcSight Roles
Management
Stakeholders
Business Users
ArcSight SMEs
L3
Administrator Author
L2
SOC Users
Analyst
L1
Operator
12
People: ArcSight Roles
Management
Stakeholders
Business Users
ArcSight SMEs
L3
Profile:
Administrator Author • Junior Security Team Member
Job:
• Triage Events of Interest
• Follow SOPs
L2
• Route case to expert/SME
SOC Users
Analyst
Tasks:
• Monitor for Events of Interest
• Triage Events of Interest
• Update Workflow
L1
Command Center
• Create Cases
Operator • Provide SOP Feedback
13
People: ArcSight Roles
Management
Stakeholders
Business Users Profile:
• Mid Level/Senior team member
• SME in a specific area in the SOC
ArcSight SMEs
Job:
L3
• Investigates Incidents/Cases that
have been forwarded to them
Console
Tasks:
*Also Could be known as • Investigate Events of Interest
• Follow Workflow
Operator/Analysts
L2
• Update/Resolve Cases
• Provide SOP Feedback
SOC Users
Command Center
• May Build/Maintain limited
content
14
People: ArcSight Roles Profile:
• Senior team member
Management
Stakeholders
• Knowledge of SQL and Boolean
Logic
• Knowledge of security and
compliance goals
Business Users
Job:
• Evaluate, Develop, and Manage
ArcSight SMEs
*Also Could be known as ArcSight Content to fulfill security
Analyzer Administrators requirements
L3
• Maintain/Update SOPs,
SOC Users
Analyst Documentation
• Work with Administrator to bring
in necessary log sources
• Work With Management to
understand the SOCs security and
L1
compliance goals
Operator
15
People: ArcSight Roles Profile:
• Mid‐level/Senior team member
• Knowledge of UNIX
Management
Stakeholders
• Knowledge of Ports/Protocols
Business Users Job:
• Manage User Access
• Manage ESM Health
ArcSight SMEs
Console
Tasks:
L3
• Monitor/Maintain ESM Health
Administrator Author • Monitor/Maintain Connector
Health
Command Center • Bring in new data sources
• Manage Archiving processes
• Manage User Access
L2
• Update system configuration
Documentation
SOC Users
Analyst • Update/Patch ESM and
Connectors
L1
Operator
16
People: ArcSight Roles Profile:
• Management Position (pref.
senior)
Management
Stakeholders • Understands in corporate
compliance and security policies
• Understands SOC Operations
Business Users Command Center
Job:
• Works with other stakeholders to
ArcSight SMEs
obtain resources/funding needed
to achieve SOC security and
L3
compliance goals
Email
Administrator Author
Tasks:
• Secure funding and resources
necessary to achieve SOC security
and compliance goals
L2
• Work with Author to develop
Requirements for Usecases
SOC Users
Analyst • Develop
• Review ArcSight Reports and
Metrics to determine/Confirm
compliance
L1
Operator
17
Process
18
Process: Finding Events of Interest
Context
Logs
Objectives
Events of Interest
19
Other Processes:
Incident Response
Policy Compliance
Metrics Reporting
20
ESM Topology
21
ESM Basic Topology
SmartConnector CORRe DB
Device
CEF
1. Collect
‐ Polling
‐ Listening (port)
2. Normalize Correlation Eng.
‐ Parse (CSV, REGEX, SQ, XML…) CEF
‐ Categorize – Device Vendor, Product, DECID
‐ Zone tag – Network Model
‐ Options – Filter, Aggregate, DSM…
22 3. Forward / Cache – 70% / 30%
Correlation
Audit
Serverr Monitoring
Events
23
ESM Topology – Syslog Listener
Device SmartConnector
24
ESM Topology – Windows Unified Connector
Devices
SmartConnector
Win
Win
Win
Win
25
ESM Topology – Command Center, Console & Archiving
Command Center
Browser
Console
SmartConnector
Device
NFS
26
ESM Topology – ArcSight Management Center (ArcMC)
Devices
ArcMC
Application
O/S
Win O/S
27
Application
Logger
NYC
Application
Logger
Dallas
Application Logger
LA
28
ESM Topology – Disaster Tolerant
Devices SmartConnectors
Application HQ
Logger
NYC
Application
Dallas
Remote
Application Bunker
Logger
LA
29
EMS & Event Broker
30
ESM Distributed Correlation Clusters
31
Technology
32
ArcSight Resources &
Packages
33
ESM Resources
Active Actors
Users Cases
Channels Assets Connectors
Stages
Customers
Search
Filters Dashboards
Saved
Searches
Files
ESM Manager
Rules
Filters
Reports Integration
Commands
Knowledge
Query Pattern Base
Viewers Discovery
Lists
Notifications
34
Packages
Containers for related resources
‐ Install or uninstall as a unit
‐ Import or export as a resource bundle file
‐ CIPs (Compliance Insight Packages) are created
by ArcSight and distributed as Packaged
resources
35
Use Cases packages
View, configure, and transport ArcSight provided sets of related resources
36
ArcSight Architecture
37
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
38
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
39
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
40
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
ArcMC
ADP
41
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
• Collects Raw Logs
• Normalizes and adds context
• Sends events to destination
• ArcSight ESM
• ArcSight Logger
ArcMC
• ArcSight Event Broker
• Syslog
• CSV File
ADP
42
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
• Built on Apache Kafka
• Centralizes Event Processing
ArcMC
• Helps Scale your ArcSight Environment
• Opens up Event Data to 3rd Party Systems
• Prerequisite for setting up Investigate
ADP
43
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
ArcMC
• Provides a path to compliance for Event
Monitoring/Review Regulations
• Provides historical analysis‐quality litigation data
that is easily searchable
ADP • Optimized for high event throughput
• Not really covered in this Course
44
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
• Used to manage Connector, Event Broker and
Logger Configurations
ArcMC
• Provides Event Flow Visualizations and Health
status monitoring of managed nodes
• Not covered in this course
ADP
45
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
• Java based technology that is
ESM core to the ESM
• Receives event feeds from
Manager Connectors and Event Broker
• Completes normalization of
Events
CORRe
• Correlates Normalized
ArcMC
Events
• Processes all User Requests
• Writes Event Data to
ADP Database
46
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
ESM
Manager
• Events are compressed at 10:1 ratio
• Optimized for high speed
CORRe performance and storage efficiency
ArcMC
• Manager and CORRe are installed
on one Server
ADP
47
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
• Thick Java based client installed on
desktopESM ESM
• The majority of authoring activity
Manager
must be done here
Console
• Administrators may need access as
well depending on ESM version
CORRe
ArcMC
Command Center
ADP
48
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
ESM ESM
• Manager
Web‐based interface allowing:
• Limited content creation
• Storage Management Console
• Peer Management
Content Sync
• CORRe
ArcMC
Command Center
• Keyword/Field‐Based Search
ADP
49
ArcSight Architecture
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
ESM ESM
Manager
• Next generation high‐speed
search and hunt capability Console
• Based on Vertica
CORRe
ArcMC
• Provides Search 10x faster than Command Center
competition
• Provides an intuitive UI and
visualization capabilities
• Seamless integration with 3rd party
ADP
data lake solutions (Hadoop)
50
ArcSight Connectivity
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
ESM ESM
Console
ArcMC
Command Center
User Interfaces
ADP
51
ArcSight Connectivity
Connectors Event Manager CORRe Console Command Investigate
Broker Center
ADP ESM ESM User Interfaces Investigate/Remediate Capabilities
ESM ESM
Manager
Console
CORRe
ArcMC
Command Center
User Interfaces
ADP
52
Summary
Learned how ESM solves a problem within a SOC environment
Learned the different Roles that may be involved in ESM
Learned about the ArcSight architecture
53
Module 1 Lab Overview
Open …\etc\hosts – ping esm1 :& esm2
Open Command center – license must be current
Putty login: to /opt/arcsight/manager/bin
‐ ./arsight deploylicense
‐ ./arcsight reenableuser ad
‐ ./arcsight tempca ‐i
‐ /arcsight resetpwd
‐ ./arcsight manager‐reload‐config
‐ Manager Setup: ./arcsight managersetup
54
Demo & Lab Exercises
55
55
DEMO & Lab Exercises
Open …\etc\hosts – ping esm1 :& esm2
Open Command center – license must be current
Putty login: to /opt/arcsight/manager/bin
‐ ./arsight deploylicense
‐ ./arcsight reenableuser ad
‐ ./arcsight tempca ‐i
‐ /arcsight resetpwd
‐ ./arcsight manager‐reload‐config
‐ Manager Setup: ./arcsight managersetup
56
Module 2 ‐ ESM Command Center
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Objectives
Discuss how to navigate to the user interface
Identify the users appropriate users for this interface
Discuss what information can be obtained from this interface
Discuss the help function
Use ArcSight Command Center to gain information about the ESM deployment
Topics
What is ArcSight Command Center
• Technical Definition
The Interface
• Menus
• Features/Limitations
People
• SOC Users
• ArcSight SMEs
• Stakeholders
What is ArcSight Command
Center?
4
4
Definition
Definition
Accessible Information
7
7
Menus
Menu Bar • Access to:
Dashboards • Dashboards you granted by
Events your ACL
Reports • Your Home Screen
Cases
Applications
Administration
Stats
Dark/Light Mode
User
Site Map
8
Menus
Menu Bar • Access to:
Dashboards • Access Channels your ACL
Events grants you to
Reports • Grants access to Event
Cases Search (Logger Type
Applications Search)
Administration
Stats
Dark/Light Mode
User
Site Map
9
Menus
Menu Bar • Access to:
Dashboards • Run and access reports
Events • Limited configuration using
Reports runtime parameters
Cases
Applications
Administration
Stats
Dark/Light Mode
User
Site Map
10
Menus
Menu Bar • Access to:
Dashboards • Create Cases
Events • Manage Cases
Reports • Delete Cases
Cases
Applications
Administration
Stats
Dark/Light Mode
User
Site Map
11
Menus
Menu Bar • Access to:
Dashboards • Integration tools can be
Events added here
Reports
Cases
Applications
Administration
Stats
Dark/Light Mode
User
Site Map
12
Menus
Menu Bar • Access to:
Dashboards • Configurations for Peered
Events search/Content Sync
Reports • Access to Log Retrieval
Cases information for Support
Applications Requests
Administration • Modify Event Search
Stats Behavior
Dark/Light Mode • Configure ESM Storage
User • View ESM License
Site Map Entitlements
13
Menus
Menu Bar • Access to:
Dashboards • Usage Stats for ESM
Events
Reports
Cases
Applications
Administration
Stats
Dark/Light Mode
User
Site Map
14
Menus
Menu Bar • Access to:
Dashboards • View and Acknowledge
Events Notification Status
Reports • Update Notification Status
Cases
Applications
Administration
Stats
Notifications
User
Site Map
15
Menus
• Access to:
• Change User Password
• Access Help (admin
dropdown menu)
16
Menus
Menu Bar • Access to:
Dashboards • All menu options lined up
Events in one place
Reports
Cases
Applications
Administration
Stats
Notifications
User
Site Map
17
Features/Limitations
Resource Capabilities Resource Capabilities
Active Channels Read/Write Events ArcSight Command Center only
Filters Read/Write Content ArcSight Command Center only
Management
Field Sets Read/Write*
Peers ArcSight Command Center only
Fields Read*
Saved Searches ArcSight Command Center only
Data Monitors Read
Search Filters ArcSight Command Center only
Dashboards Read
Storage and ArcSight Command Center only
Rules Not Available Archives
List Not Available Log Retrieval ArcSight Command Center only
Queries Not Available License ArcSight Command Center only
Query Viewers Read Stats/Site map ArcSight Command Center only
18
People
19
19
SOC Users
• Access Dashboards/Reports
Management
Stakeholders
quickly
• Create and use Active Channels
Business Users
• Integrate with Investigate
• Create/Manage Cases
ArcSight SMEs
L3
Administrator Author
L2
SOC Users
Analyst
L1
Operator
20
• Access Dashboards/Reports
quickly
ArcSight SMEs • Create and use Active Channels
• Make Administration Changes
Stakeholders to:
• Storage
• Archiving
Business Users • Event Search
• Manage Peer
Relationships
ArcSight SMEs
• View Connector Status
L3
Administrator Author
L2
SOC Users
Analyst
L1
Operator
21
Stakeholders • Access Dashboards/Reports
quickly
Management
Stakeholders
Business Users
ArcSight SMEs
L3
Administrator Author
L2
SOC Users
Analyst
L1
Operator
22
Demo & Lab Exercises
23
23
24
Module 3 ‐ ESM Console
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Module topics
Installing the ArcSight Console
Logging on and navigation
ESM Console Window
‐ Toolbar Commands
‐ Navigator Panel
‐ Viewer Panel Views
‐ Active Channels and Dashboards
‐ Edit/Inspect Panel
‐ ESM Console Help
Reference Resources
ESM Console Preferences – password and programs pane
Objectives
Upon successful completion of this module, you will be able to:
‐ Install the ArcSight ESM Console
‐ Create a Knowledge Base article in the console and view it in a browser
‐ Create a new reference file
‐ Customize the ESM console
‐ Format date and time
‐ Add a shortcut key to a resource in the ESM
Installing and configuring the console
Install on Linux, Windows or Mac
‐ Linux machines: install as non‐root user
Transfer settings if have existing installation
Run in default mode or FIPS mode
‐ FIPS mode: cannot revert to default mode
‐ FIPS cipher suite:
‐ FIPS 140‐2
‐ FIPS with Suite B 128 bits
‐ FIPS with Suite B 192 bits
Choose direct connection or proxy server
Installing and configuring the console
Authenticate to ESM manager
‐ Password
‐ Password and SSL
‐ supports only client keystore for SSL based authentication
‐ PKCS#11 token as SSL client based authentication not currently supported
‐ Password or SSL
‐ Only option to use PKCS#11 authentication
‐ SSL only
Single user or multiple users
‐ Single recommended
Starting the console
Use shortcuts
Command window from console’s bin directory
‐ Windows: Arcsight console
‐ Unix: ./arcsight console
Working in the console
Navigator Inspect/Edit
panel panel
Viewer
panel
Toolbar commands • network model
• use case
• nslookup
• ping
• portinfo
• traceroute
channel control • web search
notifications • who is
navigator panel Inspect/edit panel
viewer panel categorize event
scheduled jobs
Navigator panel
Resources
Packages
Use Cases
Viewer panel
float panel
Open resources
10
Inspect/Edit panel
multiple tabs open
11
12
Knowledge Base
13
File Resource
Acts as a common secure share
repository to store information
Can contains scripts, utilities, data files,
templates, graphics and any general purpose
files
Allows permission to Read/Write or no
access
14
Reference Pages
Pointers to an internal or external web page:
‐ Resource Groups
‐ Individual Events
‐ Vulnerabilities
Right‐click accessible from resource tree or viewer
grid
15
Console preferences
16
Module summary (1 of 2)
In this module, you learned that
‐ The ESM console is a client application
‐ Used to identify, investigate, and review security data collected and correlated
‐ Contains three main panels
‐ Navigator panel ‐ locate, view, and use ESM resources
‐ Viewer panel ‐ displays events, assets and search results
‐ Inspect/Edit panel ‐ view event properties and modify ESM resources attributes
17
Module summary (2 of 2)
‐ Access help three different ways
‐ Consists of a navigation panel and topic display window
‐ ESM reference resources
‐ Knowledge Base enables you to post site‐specific data, such as protocols, to a web viewer
‐ File Resource can contain non‐ESM objects, which users can access to obtain information
‐ Reference pages are pointers to an internal or external web page
‐ Console preferences are set using the 8 panes in the preferences dialog box
18
19
Lab Exercises
20
Module 4 ‐ SmartConnectors
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Module topics
What is a connector?
What is normalization?
SmartConnector functions
What is the network model?
Deploying and Configuring SmartConnectors
What is a connector?
Interface to objects on network
Generate event data
Normalize data
SmartConnectors
‐ Execute commands like telling a scanner to run a scan
‐ Lookup IP address, host names
What is normalization?
Parse data to pull out values from events
Populate fields in schema
What is the network model?
How many phone numbers know from memory?
Add contact name when entering phone number
Network model represents nodes
‐ Assets: individual nodes, such as servers, routers, and laptops
‐ Asset Ranges: contiguous block of IP addresses/nodes
‐ Zones: contiguous block of addresses
‐ Networks: way differentiate private address spaces
‐ Customers: represent cost centers or separate BUs
SmartConnector functions
Collect data from source device
Filter out data not needed
Parse individual events
Normalize into common schema
Aggregate events
Categorize events
Pass events to ESM Manager after processing
Can issue commands to devices
300+ SmartConnectors Overview
CounterAct
SmartConnector
Asset Import
SmartConnector
Firewall Events
Events
ArcSight
Smartconnector ESM Console
NEW YORK
Syslog ArcSight ESM
Manager & CORRE,
ArcSight San Francisco
Smartconnector
SmartConnector Functions
1. Collects
‐ Active ‐ polling
‐ Passive ‐ listener
2. Normalizes – parse and map raw events into Common Event Format (CEF)
‐ Zone Tagging – add for each IP (Zone Name)
‐ Categorization – add Category field values – based Device Event Class ID
‐ Translates Time Zones ‐ GMT
‐ Map Device Severity to Agent Severity for Threat Priority Calculation
‐ Customer (optional) – add for each record
‐ Other Options – Filter, Aggregate, Turbo Mode, DSM
3. Forwards or Cache/Forwards – 70 / 30 Cache Flush
‐ Cache – when cannot send to a destination – one cache per destination
‐ Multiple Destinations – two or more in parallel
‐ Failover Destination – one only
SmartConnectors by Task
Event Log Connectors
‐ Retrieve security log events from devices and applications
‐ (Cisco Pix, Checkpoint NG (OPSEC), Cisco IDS, McAfee EPO…)
Scanner Connectors
‐ Vulnerability data ‐ Scanner devices
‐ (Nessus, foundscan, NCircle, Internet Scanner)
‐ Assets Import – SmartConnector and FlexConnector
‐ Identity Import – Identity View
‐ (Microsoft Active Directory …)
CounterACT Connectors
‐ Execute commands in the device to retrieve, modify or analyze its configuration
‐ (Cisco Pix Shell, Checkpoint NG(SAM), Solsoft, NCM, NRM)
SmartConnector Data Sources
Log Files or Folders of Log Files (Folder Follower)
‐ Fixed Delimited
‐ REGEX
Database Reader (ODBC, JDBC)
‐ Time Based
‐ ID Based
‐ Multi‐Database
Syslog – listener (port) or flat file concentrator
SNMP (Simple Network Management Protocol) – listener “Trap” events
XML (Extensible Markup Language) – Folder Log File Reader
API (Application Programming Interface) – device or application‐specific API used to pull events
10
SmartConnector ‐ Configuration Options
11
SmartConnectors – Active Collection (Polling)
Connectors can also pull events from
the security devices using protocols like
Events are sent to the
RDEP, JDBC/ODBC, OPSEC, eStreamer…
Destination
IDS, Firewall, Events are
Router, OS, normalized
Antivirus, etc…
12
SmartConnectors – Passive Collection (Listener)
Connector receives
events from Events are sent to
third party device the Destination
13
SmartConnector Cache Scenarios (1 and 2)
Activated for any of following conditions:
1. Destination cannot be reached
Cache
2. Burst of events that the destination must throttle
Cache Events that must
be throttled
14
SmartConnector Cache Scenarios (3 and 4)
3. Transport configured to cache – paused or scheduled delivery or bandwidth
4. Manager cannot process events – spike to Manager or DB
Cache
Cache Concepts:
– All events in cache ‐ already filtered and/or aggregated
– One Cache per transport destination
– Cache Flush ‐ 70% live events and 30% cached events
51,102,0
– Cache Overflow –
– Maximum exceeded
15
– First In First Out (FIFO) ‐ drops 20MB of events at a time
‐ How many events are generated during an average day
‐ Aggregation applied
‐ Filters applied
‐ Turbo Mode of the SmartConnectors
16
SmartConnector Platforms
17
Platform File
Linux ArcSight‐w.x.y.nnnn.z‐Connector‐Linux.bin
ArcSight‐w.x.y.nnnn.z‐Connector‐Linux64.bin
Microsoft Windows ArcSight‐w.x.y.nnnn.z‐Connector‐Win.exe
ArcSight‐w.x.y.nnnn.z‐Connector‐Win64.exe
Solaris ArcSight‐w.x.y.nnnn.z‐Connector‐Solaris.bin
ArcSight‐w.x.y.nnnn.z‐Connector‐SolarisIA.bin
AIX ArcSight‐w.x.y.nnnn.z‐Connector‐AIX.bin
ArcSight Update Pack ArcSight‐w.x.y.nnnn.z‐Connectors.aup
ArcSight‐w.x.y.nnnn.z‐opensource.tgz
Configuration Guides SmartConnectorConfigGuides‐w.x.y.nnnn.zip
Release Notes SmartConnectorReleaseNotes‐w.x.y.nnnn.pdf
18
SmartConnector Installation Check List
Type of SSL certificate
Manager Host Name/IP Address/Port used
ArcSight Username/Password of capable user Install
Connectors
Required parameters
Connector to install
Connector Name – as it appears Console
Connector Location – group folder in Console
Device Location and Comment – not required but Best Practice
Run Connector as a service or not – testing
19
SmartConnector Installation ‐ GUI mode
Run self‐extracting binary – extraction and configuration continuous set of panels
‐ Microsoft Windows
‐ ArcSight‐w.x.y.nnnn.z‐Connector‐Win.exe
‐ Unix/Linux with X11
‐ ./ArcSight‐w.x.y.nnnn.z‐Connector‐Linux.bin
20
SmartConnector Installation ‐ Command Line/Console Mode
2. Configuration ‐ run from ARCSIGHT_HOME>/bin runagentsetup
‐ ./runagentsetup.sh
‐ runagentsetup.bat
21
SmartConnector Installation ‐ Silent Mode
Deploying large number of identical Connectors
1st install
‐ Run – extraction only – CLI or GUI mode
‐ ArcSight‐w.x.y.nnnn.z‐Connector‐Win.exe
‐ ./ArcSight‐w.x.y.nnnn.z‐Connector‐Linux.bin
‐ Run – configuration setup using recording properties file …/bin
‐ runagentsetup –i recorderui
Subsequent multiple installs –
‐ Run with edited recorded properties file
‐ runagentsetup –i silent –f <recorded/edited properties file>
22
SmartConnector – ArcSight Command Scripts
Function ./arcsight connectorsetup
Install as Windows Service ./arcsight connectorsvc ‐i
Modify any parameters ./runagentsetup
23
Upgrading SmartConnectors ‐ Overview
Connector upgrade file
‐ ArcSight‐w.x.y.nnnn.z‐Connectors.aup
Connector Appliance/ArcSight Management Center AUP Repository
‐ Maintains a number of connector AUP files
‐ Supports multiple version upgrade/rollback capability
ESM Console
‐ Uses secure connections
‐ Launch, manage, review ‐ status of upgrades
‐ Copy .aup ‐ /opt/arcsight/manager/updates
‐ Remotely Update ‐ newer version
‐ Remotely Rollback – previous version
24
Upgrade and Rollback Processes
Administrative permission required
Individually select and launch upgrade
Upgrade, restart, and send results
‐ If successful, SmartConnector starts and reports
successful status
‐ If failed, original SmartConnector restarts with
last known good configuration and reports
failed status
25
Learning Check
1. To invoke the ArcSight SmartConnector Configuration Wizard, run which command from the
Connector directory:
a. arcsight wizardsetup –w
b. arcsight setup –i
c. arcsight connector –w
d. arcsight connectorsetup –w
2. True or False. Upgrading Connectors can be accomplished through the ESM Console.
3. True or False. Connectors can be configured to have multiple destinations.
4. True or False. Connectors can have only a single Failover Destination.
5. True or False. If there is not a vendor‐specific SmartConnector available, the device cannot be
connected to Arcsight ESM.
27
Learning Check
6. _____________ files provide a way to collect a set of files together and update
ArcSight resources as well as distribute parsers to SmartConnectors.
a. .zip
b. .aup
c. .lic
d. .bin
7. True or False. The AUP Master Destination flag should be set to “true” for only one
ESM destination at a time.
28
Learning Check
2. Status
c. ./arcsight connectors
3. Install as Windows Service
d. ./arcsight connectorup
4. Modify any parameters
6. SSL Certificates f. ./runagentsetup
7. Edit SSL Trust Store
g. ./arcsight connectorsvc ‐i
29
Module 5 Learning Check ‐5
9. When deploying Connectors throughout your organization, consider the following:
(select all that apply).
a. Daily generated Events
b.Applied configuration options
c. Turbo Mode setting
d. Amount of connector cache available
10. True or False. Deploying ArcSight Management Center in an ESM environment
centralizes SmartConnector upgrade, log management, and other component
configuration tasks.
30
Module 5 Learning Check Answers
1. $ arcsight connectorsetup –w
2. True ‐ also possible to rollback to the previous version
3. True
4. True
5. False
6. AUP
7.T rue
8.‐
9. Daily generated Events
Applied configuration options
Turbo Mode setting
Amount of connector cache available
10. True
31
Lab Exercises
32
Module 5 ‐ ArcSight Marketplace
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Module topics
What is the marketplace?
Marketplace packages/use cases
What is the marketplace?
Like the app store for use cases
Security use cases
Referred to as use cases or packages
Standard Content
‐ Predefined content
‐ Installed by default
‐ Ex: Filters available in
Shared/All Filters/ArcSight System/
‐ Locked content, not to be modified
Download from ArcSight Marketplace
Marketplace packages
Lab exercise
5
Slide intentionally left blank
Slide intentionally left blank
Module 6 – Active Channels, Field Sets & Schema
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Module topics
What is schema?
What are field sets?
What is an Active Channel?
What is the Event Life Cycle
Module objectives
Upon successful completion of this module, you will be able to
‐ Describe an Active Channel
‐ Describe what a field set is
‐ Describe the ArcSight Event Schema
‐ Describe the Event Life Cycle
What is Schema?
4
What is schema?
Data fields
Over 400 data fields
Divided into 17 groups
Common Event Format CEF
Event data fields
Event schema: devices and assets
Terms EMS uses to identify items on your network
Network Node – physical location with unique network address
Endpoint – reference or description of a network node
‐ Includes IP address
‐ Fully qualified host name
‐ MAC address
Sensor – detects activity
‐ Produces stream of event data
‐ Produces a stream of network node descriptions
Asset – network node with unique identifier (IP or MAC address, host name, zone, or external ID)
Schema: devices and connectors
Device
Network node
Reports to SmartConnector
Can be individual sensor or software that collects, then reports
What is a field set?
9
What is a field set?
A field is a collection of one or more columns
Way to limit columns displayed
Collection of columns or fields = field set
Save and apply field sets
Over 400+ fields
Pre‐defined, user‐defined, shareable and modifiable
Sortable fields and unsortable
10
Date and Time Stamps
ESM uses multiple fields to tag date and time of each event
‐ Start
‐ End
‐ Device Receipt
‐ Agent Receipt
‐ Manager Receipt
11
What is an Active Channel?
Like tuning into a TV channel
See only information defined by parameters
Can stream events
Three Channel Types
‐ Event Active Channel – can be continuously refreshed or a snapshot
‐ Rules Verify Channel – replay events for testing rules
‐ Resource Channel – status of certain resources
13
Dynamic and Static Active Channels
Continuously evaluate Evaluate once at attach time
‐ Runs a query at a pre‐defined refresh interval ‐ Single query against the ArcSight Data Store
‐ Results are constantly refreshing as each query ‐ Results are static
completes
‐ Intended to inspect historical events
‐ Intended for real‐time monitoring
14
Identifying Dynamic and Static Active Channels
Navigator Panel
‐ Yellow lightning bolt shows
“Continuously evaluate” channels
‐ Static channels have no bolt
All Active Channels
‐ Display events that can be examined
in detail
‐ Are sortable if the displayed fields allow
‐ Use Field Sets to determine pre‐defined
columns
15
ArcSight Event
Schema and Lifecycle
16
Event Lifecycle – schema processing in 7 phases
1. Data Collection and initial schema population
‐ Acquisition, Filtering, Normalization, and Aggregation of Event Data
‐ Apply Event Categories
‐ Apply Customer and Zone from Network Model
2. Network Model Lookup and Priority Evaluation Phase
3. Correlation Evaluation
4. Monitoring and Investigation
5. Workflow
6. Incident Analysis and Reporting
7. Storage and Archive
17
18
Event Normalization
Transforms raw event data into a Common Event Format
19
19
Event Schema Groups
Simplify event field identification and access
‐ Organize endpoint device and asset data into common field definitions
‐ Add event categorization and object modelling to enhance evaluation
‐ Provide navigation reference for resource editors and right‐click menus
20
Event Schema Groups (1 of 3)
Timestamp, event identification
and classification groups
• Event (root)
• Category
• Threat
21
Event Schema Groups (3 of 3)
File State, Request, Workflow and Custom Groups
File
Old File
Request ‐ HTML
Event Annotation
Device Custom
Flex
ArcSight
Forwarding
Connector
23
Seven Phases Event Lifecycle – Overview
1. Data Collection and Event Processing
2. Network Model Lookup and Priority Evaluation
3. Correlation Evaluation
4. Monitoring and Investigation
5. Workflow
Phase 7
6. Incident Analysis and Reporting
7. Storage and Archive Phase 6
Phase 5
Phase 4
Phase 3
Phase 2
Phase 1
24
Phase 1 – Data Collection and Event Processing
ArcSight Connector functions
‐ Converts raw event data into normalized events
‐ Tags each normalized event with event categories – Customer and Zone
‐ Filters and aggregates normalized, categorized events
‐ Sends base events to ESM Manager
25
Collection and Normalization of Event Data
Devices generate event data and send event data to the ArcSight SmartConnector
ArcSight SmartConnector normalizes event data
‐ Extracts values from log data
‐ Maps values to corresponding ArcSight event fields
‐ Evaluates and makes comparisons on normalized events
26
Juxtaposing...
1 2 3 4 5 6 7 8
Src IP Src Port Dest IP Dest Port D/T Stamp Payload Action Rule
Number
27
Event Data Normalization ‐ Example 2
Raw event being received by the Connector
‐ Jan 3 2008 22:15:09: CP FW In Action: drop Service:27444 (Rule5) from
144.32.56.211/1422 to 10.1.25.155/1152
Values entered into the schema after the normalization process is complete
Device
Source IP Source Destination IP Destination
Receipt Name
Address Port Address Port
Time
CP FW In
Jan 3 2008 Action: drop
144.32.56.211 1422 10.1.25.155 1152
22:15:09 Service:2744
4 (Rule5)
28
29
Applying Event Categories
SmartConnectors use the Event Category Model to describe normalized events
Based on Lookup – Device Vendor, Device Product and Device Event ID
Category Object entity being targeted
Category Behavior what is being done to the object
Category Outcome result of the Behavior on the object
Category Technique nature of the behavior represented
Category Device
type of device generating the event
Group
Category
relative security risk of the event
Significance
30
Categorization Examples
Object Behavior Technique Device Group Outcome Significance
• A security information management solution reported a hostile, failed brute force attack
targeting the login of a service
31
Customer and Zone Look Up in Network Model
Normalized events are tagged with usage‐related labels
‐ Customer – an organization containing one or more networks
‐ Zone – contiguous block of IP addresses within a network
Helps ArcSight Manager identify source and destination of events
32
Network Model Customer and Zone Example
33
Customer and Zone – Example
Privileged User Monitoring
34
Filtering and Aggregating Events
Filtering – deletes events
Aggregating ‐ merges events with similar values to a single aggregated event
ArcSight Connector sends base events to ESM Manager
35
36
37
Priority Rating
Takes into account Priority Formula Factors and Agent Severity
Displayed in Priority column of Active Channel
Easy‐to‐identify events that need immediate attention
Color‐coded and numbered
0, 1, 2 Green – very low priority
3, 4 Blue ‐ low priority
5, 6 Yellow ‐ medium priority
7, 8 Orange ‐ high priority
9, 10 Red ‐ very high priority
Priority Formula Evaluation – Example
Priority calculated by Manager
Displayed in Active Channel 1 ‐ 10
39
Tools Used to Correlate
Filters – conditions to reduce the number of events processed
Rules – evaluate events against condition sets and initiate responses
‐ Pre‐persistent
‐ Lightweight
‐ Simple
‐ Join
Data Monitors ‐ summarize events in a tabular or graphical format
‐ Event Based
‐ Non‐Event Based
‐ Correlation
Software add‐ons
‐ ArcSight Pattern Discovery ‐‐ scans millions of events to find event matches previously overlooked
‐ ArcSight Interactive Discovery ‐‐ displays relationships between events using pre‐built interactive
graphics
41
42
Phase 4 – Monitoring and Investigation
Investigate and monitor Events
43
Dashboards
• View problem areas on your network using pie‐charts, bar charts, or tables
Event Graph Data Monitors
• Transform multiple network security data in Active Channels into graphics
Query Viewers
• High‐level summaries of network activity to investigate events
Event Search
• Extracts events across multiple ESM and Logger peers
44
45
46
47
Phase 5 – Workflow
Escalate incidents to other users
48
ESM Workflow
• Make immediate investigations
• Inform and escalate incidents to users
• Track responses
• Various Workflow Resources
• Annotations
• Cases
• Stages
• Notifications
• Knowledge Base
• Reference Pages
49
ESM Workflow – Example
50
51
Reporting Tools
• Capture views or summaries of event data
• Use Queries and Trends to gather data
• Three types
• Focused Reports
• Standard Reports
• Delta Reports
52
Reporting Tools – Example
53
Phase 7 – Storage and Archive
CORR‐Engine storage management
Event Active Retention
Event Archives
Active
All normalized events
Retention
54
Module 6 Summary
In this module, you learned that:
• Active Channels – allow selection of all events
• Fields Sets – subset of the total events in the Schema
• Schema – logical table of all events received
• Event Lifecycle – Seven Phases
55
Lab Exercises
56
Module 7 ‐ ESM Filters
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
1
1
What are filters?
Boolean conditions that select events
‐ Based on Field values and/or Variable values
‐ Asset – Using Asset ID
‐ Asset Category – Using Asset ID
‐ Field Value(s) – Using Active List or Session List
Applied in 2 ArcSight components
‐ ESM Manager
‐ Connectors
Filters in ESM Manager
Determine events ESM Manager will process based on Filter conditions
Used by various resources during Event Lifecycle phases
‐ In Rules and Data Monitors during the Correlation phase
‐ In Active Channels and Query Viewers during the Monitoring and Investigation phase
‐ In Reports and Queries during the Reporting and Analysis phase
Applying filters in connectors
Filters out events that match conditions
‐ These events are not forwarded to the
destination
Non‐matching events are forwarded to the
destination
Types of filters
and or not
Common Conditions Editor
Filters in Active Channels
Filters can be used by many ESM Resources
‐ Get a powerful resource set when combining them with Active Channels
Benefits
‐ Display a stream of historical or live events defined by Filter conditions
‐ Narrow down your displayed Events, improve your searches and save time
Filters in Active Channels
Filters can be applied several ways in Active channels
‐ Filter resource
‐ Unnamed local filter condition
‐ Inline filters
‐ Event‐based filters in Investigate command
Filters in Active Channels – Resources
When you create an Active Channel, you can
select a Filter resource from a list of existing
named filters
Conditions expressed in that Filter are
applied to all events coming into this Active
Channel
10
Filters in Active Channels – Unnamed local filter condition
In the Active Channel Editor, under the Filter tab, you can specify an unnamed
condition that is applied only to the current active channel
All events coming into the active channel are evaluated against these conditions, but
the conditions are not reusable by any other resource
11
Filters in Active Channels – Inline Filters
Apply a limited set of conditions to individual columns in a grid
Are added to a local filter condition using an AND operator
Used to further refine the current conditions already set for the channel
12
Filters in Active Channels – Investigate
Analyze in Channel
Event‐based filters in Investigate
command
‐ Right‐click an event attribute in an
Active Channel view
‐ Choose Analyze in Channel –
There are filtering options that
vary based on the data involved
13
Visualizing Inline Filters
Work Around
‐ Data Stream Analysis InActiveList()
Add explicit conditions in Inline Filter
For example: targetAddress = 10.10.111.129
Or
Build a Query and Query View using the
filter with InActiveList() condition.
Active Channel
Filter Inline Filter *name
*Start
*End
*ET or MRT
14
Debugging Filters
Test whether a selected Filter matches a
certain type of event
Identify conditions that do not match the
event details
15
Putting it All Together
Shown as a Manual Input Flow Chart shape
= Contains Internal CCE
*ET or MRT = Time Stamp End Time or MRT
*Start = Start Time
*End = End Time
Active Channel
Filter *name
*Start
*name *End
*ET or MRT
Field Set
Field Set
Events
16
Integration Commands
Perform additional investigation tasks on
events in Active
Channel grid
Access investigation options, such as
‐ Nslookup
‐ Ping
‐ Traceroute
‐ Web Search
‐ Whois
‐ Logger Search
17
Lab Exercises
18
Slide intentionally left blank
19
Module 8 ‐ Data Monitors & Dashboards
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Topics Covered
Data Monitors
Dashboards
Event Monitoring using Dashboards
Upon completion of this module, you should be able to:
Identify Data Monitor types and functions
Access and Use Dashboards
Modify Dashboard Data Monitor Layouts
Event Lifecycle Phases
1. Data Collection and Event Processing 5. Workflow
2. Network Model Lookup and Priority 6. Incident Analysis and Reporting
Evaluation
7. Storage and Archive
3. Correlation Evaluation
4. Monitoring and Investigation
Data Monitors
Drive display elements within Dashboards
Evaluate event streams and system health stats
‐ Gather data when enabled
Consolidate events with common elements
Summarize event data graphically
Provide different types of analysis
The same data monitor can be displayed in multiple dashboards or displayed differently
in the same dashboard
‐ Chart or table displays
Data Monitor Types
Event based – Graphic or tabular event summaries
Correlation – Statistical values and moving averages
Non‐event based – System health component summaries
Event Based Data Monitors
7
Types of Event‐based Data Monitors
Asset Category Count
Event Graph
Geographic Event Graph
Hierarchy Map
Hourly Counts
Last “n” Events
Last State
Top Value Counts (Bucketized)
Event‐based Data Monitors – Asset Category Count
Counts and displays the number of events
that occur per Asset Category
Event‐based Data Monitors – Event Graph
Displays a real time diagram of selected
event activity
10
Event‐based Data Monitors – Geographic Event Graph
Displays a real time geographic map of selected event activity
11
Event‐based Data Monitors – Hierarchy Map
Displays an image made up of proportionally
sized panels
‐ Each panel represents a group of events
These events are selected by group fields
that are selected in the Source Node
Identifier
12
Event‐based Data Monitors – Hourly Counts
Displays total count of events on an hourly
basis along with their priority
13
Displays most recent events, which are categorized by Priority, Name, Protocol, and
Category
14
Event‐based Data Monitors – Last State
Displays complex values in simple, rapidly observable graphic results
‐ Green, red, and yellow signal lights or checkmarks, exclamation symbols, asterisks
15
Event‐based Data Monitors – Top Value Counts (Bucketized)
Displays events with maximum values for a
selected data field
‐ Displays the total number of events and event
severity
16
Event Based Correlation Data
Monitors
17
Types of correlation Data Monitors
Event Correlation
Event Reconciliation
Moving Average
Session Reconciliation
Statistics
18
Explaining Data Monitor Correlation
Data Monitor
*name
Filter
Moving
Average
Dashboard
*name
19
Correlation Data Monitors – Event Correlation
Provides flow volume correlation between
two different event streams
Helps confirm attacks reported by different
systems
20
Correlation Data Monitors – Event Reconciliation
Correlates events
between two sensors
using Filters and matching
fields
21
Correlation Data Monitors – Moving Average
Displays moving average of events based on
a selected data field
22
Correlation Data Monitors – Session Reconciliation
Correlates events based on their occurrence within a relevant time period
Typically used to watch network devices involving long term concerns
23
Correlation Data Monitors – Statistics
Enables you to select other statistical
methods in addition to moving average
Additional statistical methods
‐ Average
‐ Standard deviation
‐ Skew
‐ Kurtosis
24
Non‐Event Based Data
Monitors
25
Types of non‐event based Data Monitors
System Monitor
‐ Displays measurements based on ESM Manager’s internal systems, Java classes, and
attributes
System Monitor Attribute
‐ Displays specific attributes of a given internal ArcSight Java class
Rules Partial Match
‐ Displays Rules that have partial matches and the total number of partially matched events
within a specified time frame
26
Non‐event Based Data Monitors – System Monitor
Displays measurements based on ESM Manager’s internal systems, Java classes, and
attributes
27
Non‐event Based Data Monitors – System Monitor Attribute
Displays specific attributes of a given internal ArcSight Java class
28
Non‐event Based Data Monitors – Rules Partial Match
Displays Rules that have partial matches and
the total number of partially matched events
within a specified time frame
29
30
Dashboards are driven by Data Monitors
Data Monitor shown as a flowchart predefined process shape
Correlation
Data
Filter Monitor
Data *name
Monitor Type
Filter
*name
Type
Dashboard
Filter
*name
Data Viewer
Monitor Panel Display
Filter
*name Or Command
Type Center
31
Dashboards Layouts
Data monitors/Query Viewers included in Dashboards can drilldown into Active
Channels for further investigation
32
Monitoring Events Using Dashboards
Display system and network conditions as reported by data sources
Visualize event flow and analysis, utilizing drill‐down capabilities
33
Stock Content Dashboard Resources
ArcSight Administration – system health and performance monitoring of Connectors, ESM instances and
Loggers
Core Security – essential monitoring of Microsoft Windows, firewall, and intrusion detection and
prevention activity
Foundation – standard content packages selected at installation
34
Module summary
In this module, you learned that:
‐ Data Monitor types and function are
‐ Event‐Based
‐ Correlation Based
‐ Non‐Event Based
‐ Dashboards allow
‐ Navigating, viewing, and drilling down of events
‐ Modifying Displays
35
Lab Exercises
36
Module 9 ‐ Rules and Lists
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Topics Covered
Rules Overview
Types of Rules
Rule Conditions
Rule Aggregation
Rule Actions
Rule Triggers
Light Weight and Pre‐persistence Rules
Active Lists
Session Lists
Upon completion of this module, you should be able to:
Create and validate the following:
‐ Rule behavior
‐ Brute Force Login Attempt and Successful rules
‐ Active and Session List integration rules
Rules Overview
Standard Rule – Simple or Join
5
Two Types of Standard Rules
Simple Join
Real‐time and Scheduled Rules
Real‐time – Only work with real‐time events
Scheduled – Run at scheduled time intervals and work with real‐time, batch, and
historical events
Rule Conditions
Created using Common Conditions Editor
Rely on basic Boolean Logic principles
Rules Aggregation
Sets required number of event matches
within a specified timeframe
Defines attack patterns
Types of Rule Actions (1 of 2)
Set Event Field
Send to Open View Operations
Send Notification
Execute Command
Execute Connector Command
10
Types of Rule Actions (2 of 2)
Export to External System
Create New or Add to Existing Case
Add to or remove from Active list
Add to or terminate Session list
Add or remove Asset Category from Asset
11
Rule Actions, Cases and Notifications
Case
Create or Cases
add to
RULE
Name Notification
Match Count & Time Acknowledgement?
Aggregation Unique
Aggregation
Identical
Trigger?
Set Event Fields
Set Event Fields!
Active List
Rule Triggers
Identifies when actions should be carried out
Types
‐ Event
‐ Threshold
‐ Time
13
Lightweight Rules
Provide simpler, faster, less resource‐
intensive rule processing
‐ Processed earlier in the event flow than
standard rules
‐ Intended for populating and modifying active
lists and session lists
Active
List
Active
LW
Filter List
Rule
Session
List
Session
List
15
Lightweight Rule Restrictions
Simple event conditions only
‐ No joins
‐ No negated event conditions
Aggregation is disabled – data fields are not
aggregated
‐ Action only on the On Every Event trigger
‐ Limited to Active List and Session List Actions
‐ Add to or Remove Entry from Active List
‐ Add to or Terminate entry in Session List
Does not generate any correlation or audit
events
‐ Failure logging only
16
Pre‐persistence Rules
17
Pre‐persistence Rules
Match and modify event fields before CORR
storage
‐ Processed earlier in the event flow than
lightweight and standard rules
‐ Intended for early populating of base event
fields
‐ Accelerates analytical processing such as threat‐
level calculation
18
Pre‐persistence Rule Restrictions
Simple event conditions only
‐ No joins, aggregation, or correlation
‐ Failure logging only
Action only on the On Every Event trigger
‐ Limited to “Set Event Field” activity
Event cannot be scheduled or replayed
No modification once persisted to database
19
Enabling and Disabling Rules and Rule Actions
Rules are enabled/disabled from the
Navigator rule resource tree
Rules actions are selectively
enabled/disabled from the rule editor
20
Active Lists
21
Active Lists
A configurable data store that can hold information derived from events or other
sources
22
Active Lists – Types
Event‐based
Fields‐based
Key Fields
23
Active List Attributes
Name – Active List Name
Optimize Data – For large lists, store only
entry hashing, count and timestamps in
memory for faster searching
Capacity – Number of entries
TTL – “Time to live” for entry
Data – List schema fields
24
Manipulating Active Lists
Active List
Filter 1 Rule 1 *Name
Add Audit Event
Remove
Rule Trigger Action TTL
Add to Active List
TTL t/o
Rule Trigger Action Filter 2 Rule 2
Remove from Active List
Filter Condition
InActiveList()
Data
Filter 3
Filter Condition Monitor
NOT InActiveList()
25
Cumulative Fields in Active Lists (1 of 2)
Cumulative sub‐types are available for numeric data type fields
‐ SUM
‐ MAX
‐ MIN
26
Cumulative Fields in Active Lists (2 of 2)
Main Benefits
‐ Cumulative values calculated consistently
by multiple rules and events in parallel
‐ Better performance reading a value,
computing new values in a rule, and
storing it back
27
Cumulative Fields in Active Lists
Restrictions
‐ Only available in fields‐based Active Lists
‐ Numeric fields cannot be used as key fields
‐ Manual edit – List entries for numeric subtypes (value entered is the final value)
‐ Not Supported – Numeric subtypes in multi‐mapped Active Lists
‐ Trends cannot act on these lists
28
Time‐Partitioned Active List
Captures data partitioned over time using
timestamp field
In‐memory cache segregates data sets into
timestamp‐based partitions
Latest partition data kept in memory
‐ Oldest partition first to age‐out of list
Time‐Partitioned Restrictions
‐ No multi‐mapped lists
‐ Partially cached must be enabled
‐ List must be fields‐based (not event‐based)
‐ Fields must include (at least) one date field
29
Session Lists
30
Session Lists
Stores data similar to Active Lists
‐ A configurable data store that can hold information derived from events or other sources
31
Differences between Session Lists and Active Lists
Session List features
‐ Entries are “terminated” instead of “removed”
‐ Entries have a start‐time, end‐time, and creation‐time
‐ Entire session list does not have to be resident in memory
‐ Data uses partitions because session lists can grow very large over time
‐ Session lists are optimized for efficient time‐based queries
32
Session List Configurable Components
Name – Session list name
Overlapping Entries – Optional multiple instances of key pairings
In Memory Cap – Maximum number of entries in memory
Entry Expiration Time – Time after which entries are end‐dated if termination event not
received
33
Session List Fields
Fields determine list schema
Data types for fields
‐ Address
‐ IP or MAC
‐ Date
‐ Numeric
‐ Double
‐ Integer
‐ Long
‐ Resource Reference
‐ Asset, Report, Actor, etc.
‐ String
‐ Key field
34
Session List
Filter Rule *name Session
List
Rule Trigger Action Session List
Add to Session List *name
Rule Trigger Action
Terminate Session List
35
Module summary
Rules are programmed procedures that evaluate events against a set of conditions and patterns
To process events at a given time, Rules can be
‐ Real‐time
‐ Scheduled
Actions are automatic procedures executed when all Rule conditions and aggregation
requirements are met
Rule Triggers identify when Rule actions should be carried out
An Active List is a configurable data store that can hold information derived from events or
other sources
‐ Create, read, and remove entries within Active Lists dynamically
Session Lists – Store data similar to Active Lists and terminate entries instead of removing them
36
Lab Exercises
37
Slide intentionally left blank
38
Slide intentionally left blank
39
Module 10 ‐ User Administration
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Module Topics
Administration of Users
‐ Features and functions of user groups
‐ Creating users and user groups
‐ Administration of ACLs (Access Control Lists)
‐ ArcSight password policy
Module objectives
Upon completion of this module, you will be able to:
From the ArcSight Command Center
‐ Add, edit, clone, delete user groups
‐ Add, edit, copy, delete users
‐ Edit advanced permissions
From the ArcSight Console
‐ Create, edit, rename, delete user groups
‐ Create, edit, move, delete users
‐ Manage resource permissions
From within your ESM installation, access and modify global user password properties
Revisiting ArcSight User Roles
Admin Oversees installation, manages users, and overall ArcSight system
health, manages operations within SOC
Operator Responsible for event monitoring and investigating incidents to
triage level
Analyst Responsible for specialized investigation and remediation in
response to notifications from Operators
Business User Uses ArcSight to learn and communicate network conditions
ArcSight User Accounts
Controls authentication and authorization
‐ Set User login passwords and User Types
‐ Enable or disable User login functionality
‐ When a User is created, a personal resource group (folder) is assigned
‐ The Admin User (admin) is created during ArcSight Manager installation
User groups resource structure
Organize and grant permissions of Resources to individual users
Default user group list
‐ Users
‐ Shared
‐ Administrators
‐ Custom user groups
‐ Default user groups
Configuring user groups in the console
User groups associate related users to resources and permissions required to perform
their ArcSight user roles
‐ Customizable based on organization, permission levels, or roles
‐ Permit access specific ArcSight resources
‐ Enforce control of events a User can view
User Type Capabilities
Normal User – Full privileges to use the Command Center, the ArcSight Console, and
ArcSight Web client, and all tools
Management Tool – Limited privileges to run network management tools on behalf of
software applications (no console access)
Archive Utility – Limited privileges to run the archive command for resources (no
console access)
Forwarding Connector – Limited privileges to run the Forwarding Connector
Connector Installer – Limited privileges to add SmartConnectors
Web User – Privileges to use the Command Center and ArcSight Web, but not the
ArcSight Console
User Group and Access Control
Access Control Lists enforce access control for Console,
Command Center and ArcSight Web users, and customize what
content is accessible
‐ Manage User Group permissions
‐ Define which groups of users have which
access to which specific resources
User Group Console Views
Group Startup (Default) Views
‐ Active Channels
‐ Dashboards
‐ Query Viewers
‐ Use Cases
Startup Views Modification
‐ Add or edit alternate or additional
startup/default console display resources
10
ArcSight Password Policy
ArcSight Manager provides administrators the ability to enforce strong passwords
Use ArcSight's built in authentication
Modify server.properties file using elements from server.defaults.properties located in
the
ARCSIGHT_HOME\config directory
‐ Sets password length and complexity
‐ Restricts passwords containing Users name
‐ Checks passwords with Regex
‐ Password uniqueness
‐ Password expiration
11
Module summary
In this module, you learned to
‐ Define ArcSight ESM Users and User Groups
‐ Create a User and User Group
‐ Describe User Types, Administration of ACLs, and Password policy
12
Lab exercise
13
Slide intentionally left blank
14
Slide intentionally left blank
15
Module 11 ‐ ESM Notifications
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Module Objectives
Upon completion of this module, you should be able to
‐ Describe the operation of ArcSight notifications
‐ Configure ArcSight notifications
ArcSight notification flows
Notification Group
Escalation Level 1
Destination
If acknowledgement is required,
the notification engine will wait
the specified wait time and send
Notification the notification to the next
Escalation Level within
the same Group
Escalation Level 2
Destination
Notification process
When a notification action triggers and an acknowledgement is required:
‐ Notification engine alerts all active destinations
‐ Waits a specified acknowledgement time
‐ If not acknowledged in specified time, notification is sent to the next escalation level
‐ Process repeats until acknowledgements are received, or there are no further escalation
levels
In the ESM console
‐ Normally blue toolbar icon turns red with exclamation point
‐ Message also appears in console status bar
In ArcSight Command Center ‐ toolbar notification icon illuminates a circled number
indicating the number of pending notifications
5
Notification components
Notification Groups and Escalation Levels
‐ Shared
‐ All Destinations
‐ Multiple administration groups
‐ Unassigned
Notification Destinations
notification
‐ Endpoint Attributes – Consoles, email, pagers, groups
cellphone
‐ Start/End Times escalation notification
levels destination
Notification templates
Notification templates
Standard velocity macro template types are available in ARCSIGHT_HOME\config\notification
‐ Cell Phone
‐ Console
‐ Drop Notification
‐ Email
‐ Informative
‐ Pager
Configuring notifications
Three facilities provided
‐ During Initial installation (Configuration Wizard)
‐ ESM Console – Notifications resource
Enabled during initial manager installation
Reconfigurable
‐ Managersetup wizard
‐ ArcSight ESM Console (destinations)
ESM Console Email Notification Settings
ESM Console Notification Escalation Wait Times
10
Notification Group Setup
ArcSight ESM admin user is required for defining Destination Notification
11
Rule Action Configuration
Real‐Time Rules can be configured to send notifications
‐ Send notification action disabled
by default
‐ Edit Action
‐ Selects destination group
‐ Sets Ack Required
‐ Modifies default message
12
ArcSight System Monitor – Functions
Monitors and warns about failures or potential failures
Only sends important announcements and errors
Performs these functions
‐ Sends a message to the admin Console
‐ Writes to server.std.log and server.log files
‐ Sends email to an administrative email address
‐ Bypasses notification subsystem because it might be down!
‐ Creates an internal event
13
ArcSight Whine Daemon – Process
Monitors critical functions
‐ CORR‐Engine down?
‐ NIC blown?
‐ All gone?
Sends notifications under these conditions
‐ Subsystem failures and warnings
‐ CORR‐Engine free space shortage
‐ Archiving failures
14
CORR‐Engine Storage Thresholds
Alerts when disk free space drops below warning and error thresholds
Configurable within ArcSight Command Center
‐ System and Event Storage space warning and error threshold percentage settings
‐ Email notification destination lists for warning and error thresholds
15
Module Summary
In this module, you learned:
‐ The architecture and operation of ArcSight notifications, notification templates, and the
Whine Daemon
‐ Configuring and using ArcSight notifications
16
Lab Exercises
Slide intentionally left blank
18
Slide intentionally left blank
19
Module 12 ‐ ESM Workflow and Cases
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Topics
Cases Overview
The Cases Page
‐ Working with existing cases
‐ Viewing events in a case
‐ Adding a new case
‐ Follow up on a case
‐ Finalizing a case
‐ Adding attachments
‐ Adding notes
Module objectives
Define a Case
Access and manage cases
Event Lifecycle Phases
1. Data Collection and Event Processing 5. Workflow
2. Network Model Lookup and Priority 6. Incident Analysis and Reporting
Evaluation
7. Archiving and Long‐Term Storage
3. Correlation Evaluation
4. Monitoring and Investigation
Command Center Cases Page
Default display columns shown
‐ Customize… allows you to add, remove, and reorder columns
Add events to a case
Automatically through a rule
Manually from active channel, data monitor,
query viewer grid
Manually from another case’s event channel
Viewing a case’s events
Case Groups:
‐ <user name>’s cases
‐ Shared cases
‐ Public cases
‐ Unassigned
‐ All Cases – only shown to admins
Case Details Channel
Working with cases
Cases must be locked to edit
Configure columns
Cases page shows only cases assigned to the current user
The follow up tab allows you to note additional actions, recommend actions to others, and/or note follow‐
up contacts.
Can add attachments
Can add notes
Case Organization and Management
Click on an individual case to expose its detail pane
Security classification letter codes
Field Description
Attack Mechanism Options include: P (Physical), O (Operational), I (Informational), and U
(Unknown)
Attack Agent Options include: I (Insider), C (Collaborative), O (Outsider), and U (Unknown)
Incident Source 1 Editable text.
Incident Source 2 Editable text.
Vulnerability Options include: D (Design), O (Operational), E (Operational Environment), and
U (Unknown)
Sensitivity Options include: U (Unclassified), C (Confidential), S (Secret), and T (Top Secret)
Associated Impact Options include: A (Availability), C (Confidentiality), I (Integrity), and U
(Unknown)
Action Selections include: B (Block/Shutdown), M (Monitoring), and O (Other)
Code Value automatically calculated from other Security Classification field entries.
10
Finalizing a Case
11
Module 12 Summary
In this module, you have learned:
‐ Cases are entries of one or more selected events of interest collected together with notes,
attachments, and information about actions taken at each stage in the workflow process
‐ How to
‐ Work with existing cases
‐ View events in a case
‐ Add a new case
‐ Follow up on a case
‐ Add attachments
‐ Add notes
‐ Finalize a case
12
Lab Exercises
13
Slide intentionally left blank
Slide intentionally left blank
Module 13 ‐ ESM Queries & Query
Viewers
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Topics Covered
Query
Query Viewers
‐ Drilldowns
‐ Baselines
‐ Reports
‐ Dashboard views
Module objectives
Explain Queries
Define Query Viewers
Explain the advantages of using Query Viewers
Create the following functions with Query Viewers:
‐ Drilldowns
‐ Baselines
‐ Reports
‐ Dashboard views
What are Query Viewers?
Query Viewer
As an option,
Dashboard
you can add
query viewer
results to
Report dashboards,
and reports.
View Query results
Query in interactive charts
and tables
ActiveLists
Trend
Session Lists
ESM Event
Database Cases
Data Assets
Summary Notification
Sources
Defining and using baselines
Compare result data from a query viewer ran
at different times and dates
Only derived from table views
To create:
‐ Query Viewer > View Data as > Table
‐ Analyze in Channel > Add as baseline…
‐ Select columns to display
Sort or filter baseline data
Data drilldowns
Drilldowns give a focused view about particular aspects of a single item
ESM allows you to define and set the options for the drilldown using the Drilldowns tab
Query Viewer report
quickly share result data
Query Viewer Editor – Local Variables Tab
Local Variables ‐ define embedded variables
within the Query Viewer
Adding Query Viewers to Dashboards
Module 13 Summary (1 of 2)
In this module, you learned that:
‐ A Query Viewer is a resource that is used for defining and running SQL Queries on events and
other ESM Resources such as
‐ Trends
‐ Assets
‐ Cases
‐ You can use Query Viewers to
‐ Generate high‐level summaries
‐ Baselines
‐ Simple Reports
‐ To create a Query Viewer, you can define various options in the Query Viewer Editor
10
Module 13 Summary (2 of 2)
You can choose to view the results of a Query Viewer as a
‐ Table
‐ Bar chart
‐ Horizontal bar chart
‐ Pie chart
You can get a focused view about particular aspects of a single item, such as an Asset in a Query
result by adding a drilldown capability to a Query Viewer
ESM allows you to compare the results of the same Query run at different times and in different
contexts using Baselines
You can
‐ Add a Query Viewer result to a Dashboard
‐ Generate Reports from Query Viewer results
11
Lab Exercises
12
Module 14 ‐ ESM Reports
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Topics covered
Report Runtime Definitions
‐ Time span
‐ Filters
‐ Output format
‐ Distribution
Running and Archiving Reports
Managing Archived Reports
Focused Reports
Delta Reports
Report Scheduling
Course objectives
Define a report
Run, view, and save a report
Manage archived reports
Event Lifecycle Phases
1. Data Collection and Event Processing 5. Workflow
2. Network Model Lookup and Priority 6. Incident Analysis and Reporting
Evaluation
7. Storage and Archive
3. Correlation Evaluation
4. Monitoring and Investigation
Running and viewing reports
Run on‐demand or scheduled
‐ Query and trend data inputs
‐ Standard or customized templates
‐ Part of ESM standard content, use as is
or copy and modify
‐ Save, distribute or discard after viewing
Save formats
‐ PDF
‐ HTML
‐ XLS (Excel Spreadsheet)
‐ RTF
‐ CSV
Archive within ESM or export
Report Workflow
Report data (within a Query resource)
can include data from:
Determine
• Active lists Input Data
• Session lists
• Notifications
• Cases Reports Select/Develop
Workflow Reports
• Assets
• Events Analyst
• Trends Run/Distribute
Reports
Select standard report or modify/develop
report
Run report
‐ Optionally schedule execution
‐ Determine distribution and archiving
Running Reports
Archiving reports at runtime
Saves generated report to ESM Manager as
Archived
‐ Click Save Output to Archive (Command Center)
‐ Click Save Output (ESM console)
‐ Enter location, name, and expiration date
Archived reports
Focused report
Same as other reports
Variation of a report
Example:
‐ run the same report on the different
subdivisions of data
‐ Don’t have to copy and modify the master
report every time
10
Delta report
A single report that compares two data sets
‐ Supports single bar chart reports only
11
Report job scheduling
Scheduling recurring report jobs is only
accomplished from the ArcSight Console
Reports can be run from ArcSight Command
Center or the ArcSight Console
12
Module summary
In this module, you learned that
‐ Reports are summaries of data which communicates the state of enterprise security
‐ Report Runtime is defined by Time span, Filters, Output format, and Distribution
‐ You can run, view, archive, and manage archived Reports
‐ Focused Reports enable you to run the same report definition without having to copy and
modify the master report every time
‐ Use the ArcSight Admin user when scheduling reports
13
Lab exercise
14
15
Module 15 ‐ Content Management &
Peering
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Module topics
Configure a peer
Create sync packages
Share content
Module objectives
On completion of this module, you will be able to:
‐ Peer ESMs
‐ Perform a search on a peer
‐ Create a package and sync to a peer
‐ Manually push a package
‐ Verify successful distribution of a package
Why peer?
Distribute searches
Share content
Peering guidelines
Time and date set for appropriate time zone of each Manager or Logger
Cannot edit peer; only delete & re‐add
Username/password authentication not affected by changes to credentials once
peered
Same privileges on peer as you have on system logged into
Need license that includes peering
Enable port 9000
No peering if running FIPS Suite B mode
Admins: view, create, & edit peers;
Other users: controlled by permissions
5
Configuring peers
Target
Authentication ID and code authentication Initiator
‐ Generate authorization ID and code on Target
(Peer Authorization) Manager B
Manager A
‐ Enter host name of initiator and port
(9000 is default)
‐ On initiator, enter ID and code
(Peer Configuration)
Username and password authentication
‐ Enter username/password of target from
initiator
(Peer Configuration)
‐ External IP Address of initiator
Configuring peers screenshots
Peer authentication considerations
LDAP or RADIUS
‐ Must use username and password of local machine
SLL Client authentication (CAC)
‐ Must use authorization ID and code
Authorization ID and code expire
‐ Establish peer configuration within a few minutes
‐ Peer configuration is good until cancelled
Content management
Admin users only
Peer relationship required Edit Content Resources
Same ESM versions
ESM managers in San Francisco, London & Tokyo Build
Packages
‐ Update rules on Tokyo manager
‐ Create custom content package Manual or
Manual or
Scheduled Push
‐ Synchronize package to San Francisco & London Scheduled Push
Master
Manually push or schedule a push (Publisher)
ESM
Publisher is source of custom packages Subscriber Subscriber
ESM A ESM D
Subscribers receive packages
Subscriber Subscriber
ESM B ESM C
16.
9
Share content
Command Center: Administration > Content Management
Packages – list current user’s packages
Subscribers – peered ESM active destinations
Schedule – setup recurring push time
10
ArcSight packages
Packages are user‐created bundles of one or
more resource types, often exported as an
.arb file
Created and modified within the ArcSight
console
11
Subscribers
Select/deselect active subscribers for a content push
Click a subscriber to see package push history
12
Content push options
Manually using the push button
Scheduled – set a timed/recurring push schedule for packages with the “Follow
Schedule” checkbox selected
13
Module summary
In this module, you learned to:
‐ Configure a peer
‐ Create sync packages
‐ Share content
14
Lab exercise
15
Module 16 – ESM Event Search
Administrator and Analyst ESM 7.0.0 P1
ESM200‐700p1
Module topics
ArcSight Command Center Search Interface
Event Search Input
Search Results Display
Search Facilities
‐ Full Text Keyword search
‐ Field‐based search
‐ Pipeline operators
‐ Peer Search and Constraints
‐ Filters and Saved Searches
Module objectives
On completion of this module, you will be able to:
‐ Describe how keyword, field‐based and pipeline searches are performed
‐ Describe how search results are displayed
‐ Use the unified Search page to initiate any type of search
‐ Use Search Helper and Search Builder features to save time constructing search expressions
‐ Load, modify, and save search filters and saved searches
‐ Enable peer ESM and Logger instances for searching
Search Query Elements
A search query consists of search string, time range, and field set
Folder and disk icons provide filter selection and save
Advanced Search query options include
‐ Boolean trees, Full Text, and Field‐based tools
‐ Storage Group criteria
‐ Peer ESM/Logger criteria (if configured)
Searching peers
User Search Session
Search Interface
(Command Center or
Logger GUI)
Peer Event Peer Event
Search Search
Local
ESM/Logger
Peer Peer
ESM/Logger ESM/Logger
Peer Peer
ESM/Logger ESM/Logger
Search Controls
To search, load a saved query filter or create a new search by entering terms in the Search text box or by
using Advanced Search Builder
Uncheck “Local Only” to perform distributed search on Peer ArcSight Hosts
‐ Checkbox only appears if Peer ESM/Loggers are configured
Specify the time range, field sets, and any search constraints
Click Go! to start search
Load Save query
saved as filter Display Find Key‐Values Custom time range fields
filter Local/Peer CEF Field in raw events Select time
Search Summary range
Clear search text
Search
query text
Access Search Builder
Search
Helper
Display Toggle Go!
Select field set to search,
for results display Auto Update Hits Events Elapsed
Cancel to stop
and Timer Scanned Time
6
How Search Queries Work
When you execute a search query
‐ If constraints are applied, only data within constraint limits are searched
‐ Specified time range is matched against Event timestamps
‐ Terms in the query expression are matched against each event scanned
If an event matches criteria specified in the filter (query) and its timestamp falls within
the time range
‐ Event is considered a “hit”
‐ Event is displayed in Search Results table
‐ Event is incorporated into an event histogram
Only those fields selected in the current field set appear in the Search Results
Search results can be further processed by pipeline operators
Search Results
Display in a table with a self‐ranging histogram
‐ Default: 25 events displayed per page
‐ Expand/collapse raw event data Histogram
legend and toggle
Field Summary
Select a field of interest in the Field Summary Panel for more options
Field Summary Drill‐down and Charting
Selecting a Summary panel option
adds the specific field search terms
to the query and reruns the search
Exporting Search Results
Save or distribute in CSV or PDF format using the “Export Results” button
11
Search Query Expressions
Syntax
<Search Expression> | <Search Operators>
Search Expressions
‐ Full‐text keywords
‐ Field‐based values
Search (Pipeline | ) Operators
‐ Match patterns of data within search results
‐ Apply additional constraints to narrow the search and/or format event display
Boolean Operators (AND, OR, and NOT) are used to connect multiple keyword and/or field
conditions
12
Search Performance Factors
Besides query complexity, search performance is affected by
‐ Time range
‐ Searching archived data
‐ Searching one or more peers
‐ Searching by Storage Group
‐ Concurrent ESM event processing
13
Unified (Mixed) Search Efficiency
ESM event data is all indexed, both in primary storage and in archive
Logger CORRe contains both indexed and non‐indexed event data
‐ Logger Indexed searches apply to any combination of indexed fields and full‐text keywords
‐ Logger considerations when only some events searched are indexed:
‐ First, run search using indexed data fields to reduce data set (faster)
‐ Then, add search terms for non‐indexed data on the resulting smaller data set
‐ “fast path > slow path” methodology improves performance on all mixes of search expressions,
especially when including regex
Fastest search results are obtained when searches are performed on data that has all
been indexed (as in ESM)
14
Event Search Techniques
Users can search using any of these methods:
‐ Load a pre‐configured Filter or Saved Search
‐ Type a query in the Search text box
‐ Create a query using Search Builder
‐ Add additional search terms and/or Pipeline Search Operators to refine results of the query
15
Loading a Saved Search or Filter Query
16
Saving a Search Filter
A modified search query can be renamed and
saved for reuse
The save button launches a Save Query pane
‐ Name the new filter
‐ Choose to save as either Search Filter or Saved
Search
‐ Saved Search includes
a time range
‐ Schedule it includes both
time range and a recurring
hourly, daily, or weekly
run time with export location
17
Saving a Search Filter as a Saved Search
Saved Search option ‐ additional entries
‐ Schedule Options:
Selecting Saved Search
and Schedule it makes
these settings available
‐ Export Options:
18
Managing Scheduled Saved Searches
Command Center Administration tracks scheduled search activity
Once configured, a scheduled search must be enabled to run
19
Entering a Keyword Search
Keyword Time Range
20
Keyword Query Example
Keyword (full text) queries consist of
‐ Keyword(s) to be searched
‐ Boolean operators defining multiple keyword relationships
Example: a simple keyword search looking for “failed login” in an event would be
expressed by this query
failed AND login
21
Keyword Search Syntax
Keywords in quotation search for an exact match
‐ “file” returns only file
‐ file (no quotation marks) returns File, file, files, etc.
NOT case sensitive
A space between two keywords is an implicit AND of the two terms
‐ Infected file returns events containing both infected AND file, in any order
Spaces and other delimiters are not allowed
‐ “infected file” is invalid
Keyword terms can be combined or nested using Boolean operators
‐ (infected OR bad) AND file
22
Field‐based Query
Similar to keyword search, you can search on the content of one or more specific fields in a CEF event
Search results display all events that match the field criteria, time range, filters, and other constraints
applied
23
24
Field Query Syntax
Default ‐ Field search IS case sensitive
‐ agentSeverity = “High” returns all instances of High in the agentSeverity field, but not high
‐ This setting can be modified using Administration Search Options
Field values not enclosed in quotation marks with a space between terms is an implicit
AND of the two terms
‐ name CONTAINS infected file returns events as: name field contains (the words) infected AND
file
‐ name CONTAINS “infected file” returns events as: specific string infected file, in that order
Field search criteria can be combined with keywords in search query, using Boolean
operators
‐ Example: Infected AND agentSeverity=High
25
Regular Expression Queries
Regex is a string of characters (word, phrase, or text string) that match what you’re looking for in a series
of events
Regular Expressions (Regex)
‐ Can be used in combination with full text or field query methods
‐ Can be used with CEF or non‐CEF data
‐ IS NOT case sensitive
‐ Uses Boolean logic to combine statements in more complex ways
‐ Requires specific format and characters, including wildcards
‐ Is based on java Regex syntax
Search Pipeline Operators (1 of 3)
Provide a flow‐based event processing facility that refines results of an initial field‐
base/full‐text query in a pipeline format
cef ‐ extracts values for specified fields from matching CEF events
chart ‐ displays as a chart of a count of unique values
‐ Includes aggregation functions: average, count, mean, minimum, maximum, span, standard
deviation, and sum; multi‐series plotting functions; and span (time grouping) function
dedup ‐ removes duplicate events
eval ‐ displays events that match the resultant of a specified expression
extract ‐ displays key‐value pairs from raw events
27
Search Pipeline Operators (2 of 3)
fields ‐ includes or excludes specified fields
head ‐ displays first <n> lines of search results (default 10)
keys ‐ identifies keys in raw events based on specified delimiters (used with extract)
parse ‐ applies a named parser to the matching events of a search query
rare ‐ lists the least common values for a specified field in tabular form
regex ‐ selects events that match a specified regular expression
rename ‐ renames a cef or rex extracted field
replace ‐ replaces a specified string in one or more specified fields with a new specified
string
28
Search Pipeline Operators (3 of 3)
rex ‐ extracts values based on a specified regular expression
sort ‐ order specified as descending or ascending (default)
tail ‐ displays last <n> lines (default 10)
top ‐ lists in tabular form the most common values for a specified field
transaction ‐ group events that have the same values in specified fields
where ‐ displays events matching criteria specified in “where” expression
29
Using Wildcards and Special Characters
An asterisk wildcard character ( * ) can be used in search queries with some
restrictions:
‐ For Keyword expressions ‐ cannot be the first character
‐ log* or lo*g* are both valid, and return log, login, logged, etc.
‐ *log is invalid and returns an error
‐ For Field or Regex expressions ‐ can be in any position
‐ name=*log returns log, blog, myblog, etc. in name field
Special characters: spaces, tabs, commas, /, |, and so on
‐ For Keyword ‐ can NOT be used
‐ For Field or Regex ‐ may be used with certain restrictions
30
Advanced Search Options
Used to tune search
expression preferences and results display
options
‐ Local system level – applies to all users
‐ Default settings shown
‐ “Yes” settings incur extra processing and can
impact search performance
31
Field Sets
Determines field (column) display for events in search results
There are pre‐defined field sets and custom (editable) field sets
Custom field sets can be created temporarily for a search session, or saved and used by
other users as shared field sets
Field set selection within a search result is specific to each user’s interface session
‐ Each user has control over visible field sets
32
Creating Custom Field Sets
Click the Customize icon or select “Customize”
from the Fields drop down
‐ Browse Categories for the fields of choice
‐ Drag and drop or use the positioning arrows to
place the fields within the Selected Fields list
33
Viewing Field Set Definitions
Only custom (Shared) field sets can be removed or overwritten
Default Fields and Running Tasks
Default CEF Fields – names, types, and length
Running Tasks – search job status
35
Search Helper (Auto‐Suggest)
List of Fields and Operators
Autocomplete
Search History Help
Syntax examples Auto‐open ON/OFF
Open Search
Helper
36
Advanced Search ‐ Search Builder Tool
Load or save filters, clear input, help
Choose Boolean operators and display type
Logic layout of query being created
Create field and Full Text query terms
(similar to ESM Common Condition Editor )
Add pipeline query terms
Constrain search by
storage groups and/or peers
37
Search Builder Alternate Displays
Display drop‐down menu selects logical
operator view options
‐ Color Block View (shown here)
‐ Tree View (default, previous slide)
‐ Both Tree View and Color Block View can display
field selection beside or below filter diagram
38
Search Constraints
Control search range by storage group and peer host selections
‐ By default, event search includes all storage groups on the local host
‐ If configured, peer hosts are searched if “Local Only” checkbox is reset
‐ To search for events in specific storage group(s), use the _storageGroup metadata identifier:
storageGroup IN ["Internal Event Storage Group”, “SG1"]
‐ To perform a distributed search across specific peer hosts, use the _peerLogger metadata identifier:
_peerLogger IN [“192.0.2.10”, “192.0.2.11”]
39
Regex Helper
Generates Regex parsing for a selected Non‐CEF raw event
‐ Click on the Extract Fields icon next to the word RAW to invoke the Regex Helper
40
Processing Search Results with Regex Helper
Opening a non‐CEF raw event in search results also exposes the Extract Fields (Regex Helper) icon
Mouse over a field name to show
Clicking the RAW icon displays the
its location in the Raw Event and
Extract Fields dialog for the event
its corresponding Regex syntax
41
Regex Helper ‐ Extract Fields pane
The raw event is displayed in a parsed Regex expression
‐ Parsed fields are given default names and listed in a value table
‐ Selecting one or more fields in the value table inserts corresponding parsing syntax into the search box
‐ Optionally, rename field(s) as you want the columns displayed
42
Module Summary (1 of 3)
In this module, you learned that:
‐ Search provides a common user interface for constructing
‐ Queries based on Field, Keyword, and/or regex expressions
‐ Pipeline operators to further process search results
‐ Search queries are limited by a user‐specified time range
‐ Optional constraints select configurable search groups and peer hosts
‐ Search results are displayed
‐ In table form
‐ On same page as search terms
‐ 25 per page ‐‐ default
‐ Using columns selected by the current pre‐configured or user‐specified field set
43
Module Summary (2 of 3)
‐ Search results can be exported as either
‐ .csv
‐ .pdf
‐ The Search Helper tool provides
‐ Selectable list of fields and operators
‐ Search history list
‐ Syntax examples
‐ Context help
‐ Auto‐complete function
‐ Suggested next search operators
‐ The Search Builder tool provides
‐ Boolean tree view and editor
‐ Field and full‐text expression editor
‐ Constraint selection
44
Module Summary (3 of 3)
‐ Queries can be saved as
‐ Filters
‐ Saved searches
‐ Scheduled searches
‐ Filters ‐ save the query expression and constraints, but not time range
‐ Saved searches ‐ save the query expression, constraints, and time range
‐ Saved search jobs ‐ can be scheduled to run at specific fixed times or intervals
‐ Advanced Search Options can customize
‐ System‐level search expression case sensitivity
‐ Delimiter behavior
‐ rawEvent population
‐ Field Summary operation
45
Lab exercise
46
Slide intentionally left blank
47