© 2020 SPLUNK INC.

© 2020 SPLUNK INC.

Starting Your
Splunk Journey –
Get Your Data In
PLA1906C

Ben Marcus
Sr. Staff IT Engineer | Qualcomm
Forward- During the course of this presentation, we may make forward‐looking statements regarding
future events or plans of the company. We caution you that such statements reflect our

Looking current expectations and estimates based on factors currently known to us and that actual
events or results may differ materially. The forward-looking statements made in the this

Statements presentation are being made as of the time and date of its live presentation. If reviewed after
its live presentation, it may not contain current or accurate information. We do not assume
any obligation to update any forward‐looking statements made herein.

In addition, any information about our roadmap outlines our general product direction and is
subject to change at any time without notice. It is for informational purposes only, and shall
not be incorporated into any contract or other commitment. Splunk undertakes no obligation
either to develop the features or functionalities described or to include any such feature or
functionality in a future release.

Splunk, Splunk>, Data-to-Everything, D2E and Turn Data Into Doing are trademarks and registered trademarks of Splunk Inc. in the United States
and other countries. All other brand names, product names or trademarks belong to their respective owners. © 2020 Splunk Inc. All rights reserved
 © 2020 SPLUNK INC.

Ben Marcus
Sr. Staff IT Engineer | Qualcomm

https://www.linkedin.com/in/heybigben
 © 2020 SPLUNK INC.

Agenda 1. General Data Onboarding

Overview
2. Splunk Universal Forwarder
Will discuss common ways of
getting data into Splunk and 3. Remote Syslog
give ideas for example data
sources and why different types 4. Splunk HTTP Event Collector (HEC)
of sources are useful.

5. Scripted Inputs via Universal Forwarder (UF)

6. Modular Inputs

7. Database (DBX) Connect

8. Other transform, routing, onboarding tools

 © 2020 SPLUNK INC.

General Data Onboarding

How to interface with data - push vs pull Event breaking

What are you going to do with the data Max length (truncation)

Sizing Host, source, sourcetype, index

Timestamp extraction Common Information Model (CIM)

Compatible apps compatible fields

Splunk Docs - Getting Data in

Use test Splunk
environment
 © 2020 SPLUNK INC.

Splunk Universal Forwarder (UF)

Monitor Log Files
Linux syslogs
• /var/log/syslog, /var/log/authlog, /var/log/sudo.log, /var/log/syslogs/local1-7
• /root/.bash_history

Application Logs /opt/splunkforwarder/etc/apps

• Webservers (Apache/Nginx) /linuxsyslogs/default/inputs.conf
/clearcase/default/inputs.conf
• Revision Control /lsf/default/inputs.conf
/webserver/default/inputs.conf
• Database (mysql, postgres) /dns/default/inputs.conf
• DNS /osquery/default/inputs.conf
/database/default/inputs.conf
• LDAP
• OS Query (/var/log/osquery/osqueryd.snapshots.log,
/var/log/osquery/osqueryd.results.log)
Package ”apps”

Splunk Docs - Universal Forwarder Manual

 © 2020 SPLUNK INC.

Remote Syslog
Easiest for appliances and devices where you can’t run the Universal Forwarder directly

Appliances send remote syslog via UDP/TCP 514 port

• Firewalls (Fortinet, PaloAlto)
• Routers (Cisco, Arista)
• VMware
• VPN’s
• Web proxies, web security gateways
• Load balancers (Citrix Netscaler, F5 BigIP)
• Servers (HP ILO, Dell IDRAC, Supermicro BMC)

Check
Timezones
 © 2020 SPLUNK INC.

Remote Syslog Architectures

Classic Splunk Connect 4 Syslog (SC4S)

Spunk Search Head

Indexers

Indexers

Syslog Servers with HTTP/HTTPS

Universal Forwarders Enterprise LB

TCP/UDP SC4S SC4S

Enterprise Load Balancer

Sources Splunk docs - SC4S

Performant and Scalable Syslog Data Ingest

 © 2020 SPLUNK INC.

Splunk HTTP Event Collector (HEC)

Great method for application developers
Collectd.conf
Easy for scripts and custom applications to post data to Splunk
• collectd, telegraf – linux agents send metrics to Splunk HEC.
Splunk App Infra
• Webhooks (Zoom, Github, Plex)
• Custom - Data center power stats

curl -k https://hecserver.yourdomain.com:443/services/collector/event -H "Authorization: Splunk

secrettoken" -d '{"event": "hello world via Splunk HEC"}'

Splunk Docs - HEC

Log in JSON
 Scripted Inputs
via Splunk UF [script://./bin/nfsstatjson.sh]
interval = 1800
Linux commands sourcetype = json_nfsmountstat
source = json_nfsmountstat
(Splunk *nix TA)
index = nfsmounts
• top disabled = 0
Careful with
• ps userid
• netstat [powershell://conninfo]
• uptime script="$SplunkHome\etc\apps\myap\bin\Conn.ps1”
schedule = */20 * 9-16 * 1-5
• df
• lsof
Windows commands
• Powershell

© 2020 SPLUNK INC.

 © 2020 SPLUNK INC.

Windows Data
Splunk UF on Windows Splunk Docs - Windows UF

WinEventLog WinPerfMon WinHostMon ADMon

Logs Performance Counters Host Properties Active Directory

System CPU OperatingSystem

Security Memory Processor AD baseline
Application Network NetworkAdapter AD changes
Apps and Services LogicalDisk Service
Process
Disk
 © 2020 SPLUNK INC.

Modular Inputs (Splunk Heavy Forwarder)

App or add on extending Splunk Enterprise framework to define a custom input as if it
were a native input.

App has special input to pull/obtain data from cloud or via app API

REST API modular input – obtain data via remote application REST endpoint.

Splunk Apps - Splunkbase

 © 2020 SPLUNK INC.

Cloud Connectors
AWS Azure GCP
Splunk app AWS Splunk Add-on for Microsoft Splunk Add on for Google
Kinesis Firehose Cloud Services, Cloud
(sends directly to Splunk HEC) Splunk Add-on for Office 365 Splunk HEC
Splunk Universal Forwarder Splunk HEC

CloudTrail, CloudWatch Azure AD User Audit, G Suite Admin Console,

(metrics/logs/VPC flow), Storage Accounts, Virtual Stackdriver Logging, Cloud
Config, Billing, ALB/NLB Machines, Subscriptions, Security, Command Center,
(load balancers), Billing GCP, GKE and Anthos,
Cloudfront, S3 Stackdriver Metrics
 © 2020 SPLUNK INC.

Splunk Database Connect

• Database connect–dbx v3 type inputs

– Most JDBC/ODBC drivers will work

• SQL input – periodically run query and index results

• Tail table with unique key

• Enrichment via Splunk database lookups

| dbxlookup lookup=“userdept” | table host, user, title, dept

Splunk Docs - DBX

 © 2020 SPLUNK INC.

Other Transform, Routing, Onboarding Tools

• Splunk Data Stream Processor – DSP
• Fluentd (fluentd.org) – Open source data collector/router
• Kafka (kafka.apache.org) – Distributed streaming platform
– Event Streaming Platform for Kafka – confluent.io
• Cribl (cribl.io) – LogStream processor, Ingest at scale
– Universal receiver: Receive data from multiple sources, send to Splunk
– Pull data at high scale from Kafka, S3, Azure Event hubs,
and other pull sources
– Receive data on standard protocols such as Syslog and SNMP Traps
– Batch ingest of historic event logs, on file systems or in S3

Know More – Data Stream Processor

 © 2020 SPLUNK INC.

Other Data Sources

• Splunk Stream (packet data)
• Netflow (summary metadata on connections)

Any
Machine
Data
 © 2020 SPLUNK INC.

Please provide feedback via the

SESSION SURVEY