Best Practices and Better Practices For Admins Latest Slides: Collaborate: #Bestpractices Sign Up at HTTP://SPLK - It/slack

© 2019 S P LUNK INC.
Best Practices and Better

Practices for Admins
Load survey!
Latest Slides:
https://splk.it/conf19-FN1054
Collaborate: #bestpractices
Sign Up @ http://splk.it/slack
Best Practices &

Better Practices
for Admins
Tuesday, October 22 | 03:00 PM - 03:45 PM
Thursday, October 24 | 02:15 PM - 03:00 PM
burch
Forward- During the course of this presentation, we may make forward‐looking statements regarding
future events or plans of the company. We caution you that such statements reflect our
Looking current expectations and estimates based on factors currently known to us and that actual
events or results may differ materially. The forward-looking statements made in the this
Statements presentation are being made as of the time and date of its live presentation. If reviewed after
its live presentation, it may not contain current or accurate information. We do not assume
any obligation to update any forward‐looking statements made herein.
In addition, any information about our roadmap outlines our general product direction and is
subject to change at any time without notice. It is for informational purposes only, and shall
not be incorporated into any contract or other commitment. Splunk undertakes no obligation
either to develop the features or functionalities described or to include any such feature or
functionality in a future release.
Splunk, Splunk>, Turn Data Into Doing, The Engine for Machine Data, Splunk Cloud,
Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the
United States and other countries. All other brand names, product names, or trademarks
belong to their respective owners. © 2019 Splunk Inc. All rights reserved.
burch
Prince of Puns
Manager, Product Best Practices | Splunk, Inc.
@SloshBurch
Comp Sci + MBA
Middleware Eng
2012: Customer
2014: Employee
Produced by Product Best Practices

Splunk Success Framework

Upgrade Cheat Sheet

https://docs.splunk.com/Documentation/Splunk/X/Installation/HowtoupgradeSplunk
Target 8.0 7.3 7.2 7.1

Min 7.0 6.6 6.5 6.5
https://www.splunk.com/en_us/legal/splunk-software-support-policy.html
Target 7.0 6.6 6.5 6.4 6.3 6.2 6.1 6.0 4.3
6.0
Min 6.0 6.0 6.0 6.0 6.0 6.0 4.3 uninstall
UF: 5.0
Upgrades and Python

Banner Notifications
Search “docs.splunk.com Splunk Web messages”
Examples:
• Scheduled restart
• Ongoing issues
• Cool KO to check out
Specific audiences
• Role
• Capability
Create a Lab!
search “splunk ssf sandbox lab”
Indent Config
Example: Benefit:
[general] Easily see system vs hand edits

pass4SymmKey = $1$ShiC+P0X
sessionTimeout = 30m
serverName = elBurcho Detect hand config updated by system
when conf just isn’t working

search “docs.splunk.com btool”
btool <configuration> list <stanza|> <--debug|>
Add to your env path! (source a profile file from an app)

• Linux: export LD_LIBRARY_PATH=$SPLUNK_HOME/lib
• Mac: export DYLD_LIBRARY_PATH=$SPLUNK_HOME/lib
No “.conf”
Use --debug with | grep –v “system/default”
Not current runtime

Utilities Tier DR
search “splunk ssf dns”
Impact of failures on the utility tier
Preserve component's state
Preserve networking using DNS

Bootstrap
Minimal system/local
1. Install Splunk Enterprise

2. Bootstrap
• Point to DS/Master/Deployer
• system/local overwritten by apps
• Centralized control
• Global App < Function Apps
3. Download app with scripted input

• Non config changes
• Risky!
Keep It Clean: Naming Conventions

search “splunk ssf naming convention”
Template:
<summary|>_<company>_<function>_<environment>
<company>
• Yours or from a 3rd party/splunk app
<function>
• Nothing that changes (i.e. organization/teams)
<environment>
• PROD, DR, QA, TEST, DEV, etc…
<summary|>
• Exists as a modifying of corresponding index
App Management
What practices do you notice?
Burch_CustomerOverview Burch_license_server_ta Burch_splunkAdmin_win_ta

Burch_configbackup_ta Burch_loginui_ta Burch_splunkUpgrade_ta
Burch_datacollection_ta Burch_master_ta Burch_stopdeploymentclient_ta
Burch_deployer_ta Burch_mc_ta Burch_utility_ta
Burch_deploymentserver_ta Burch_multisite_site1_ta Burch_workspace_admin
Burch_dreamhost_ta Burch_multisite_site2_ta Burch_workspace_bptw
Burch_es_ta Burch_plex_ta Burch_workspace_default
Burch_forwarder_ta Burch_sandbox_ta Burch_workspace_developer
Burch_handbrake_ta Burch_searchhead_distributed_ta Burch_workspace_power
Burch_heavyforwarder_ta Burch_searchhead_ta Burch_workspace_user
Burch_indexer_ta Burch_searchheadcluster_ta Burch_zglobal_ta
Burch_indexlisting_ta Burch_searchtimeko_ta
Burch_license_client_ta Burch_splunkAdmin_nix_ta
App Management

App Management

Simple Version Control

Search for “splunkbase Stateful Snapshot for Splunk”
Good: Scripted Input Targets

• Specific Diag (or just etc dir) • Utilities
• Clean old copies • SHC Working Folder
Better: Scripted Input Source Control != High Availability

• Check in to git • VMotion type stuffs
Best: Custom Built Solution

• Source Control
SHC Need 2 Knows

“So…I can’t just treat it like a Deployment Server?!”
Benefits Caveats
Deployer not critical path Min 3+ SHs

• Odd number for consensus
Config -> default by default
• Search “splunk Choose a deployer push mode” Same specs
More effective hardware utiliz. No manual conf edits on SHs
• Split Brain
Eliminates dedicated alerting SHs
• A.K.A. Job Servers deployerPushThreads = auto
Search Head limits.conf

(1 of 2)
Configuration Benefit
[scheduler] Defaults to 50%

max_searches_perc Ad Hod takes precedent regardless
auto_summary_perc Additional controls for scheduling
shc_role_quota_enforcement Quota cluster wide

• Default is instance specific
shc_syswide_quota_enforcement
Search Head limits.conf

(2 of 2)
Configuration Benefit
[realtime] When up-to-the-second accuracy is not needed
indexed_realtime_use_by_default Better resource usage

search “docs.splunk.com about real time
searches”
Some events may not be available to browse in

the UI.
[search]
remote_timeline_fetchall Onus on users to write less verbose searches
Indexer Performance Improvement

Search “splunk web.conf startwebserver”
No splunkweb in the indexer room Turn down splunkweb for what

Index Definitions
[volume:home] volume:
path = $SPLUNK_DB
maxVolumeDataSizeMB = x
maxVolumeDataSizeMB
[volume:cold] • Indexes compete for storage
path = $SPLUNK_DB
maxVolumeDataSizeMB = x [default]
[default]
homePath = volume:home/$_index_name/db [newindex]
coldPath = volume:cold/$_index_name/colddb
thawedPath = $SPLUNK_DB/$_index_name/thaweddb $_index_name
[newindex]
Cluster of One
“We lost that data even though we had replication”
Benefits Challenges
Baby step to clustering ONLY IF YOU PLAN TO NEED REPLICATION

“Retroactive” data replication
No additional disk Administratively difficult
• If factors are still 1 • Higher chance of errors
• Conceptually abstract
summary_replication
Indexer Discovery
Search docs.splunk.com for “indexerdiscovery”
Pros Cons
Dynamic indexer listings Network access to master node

indexerWeightByDiskCapacity Min forwarder 6.3.7 OR 6.4.4
• Indexers with different volume sizes
Total Disk != Free Space
Seed with traditional tcpout group
Lead to uneven data distribution
Data Distribution Quirks
Consolidated data == serial search
Forwarders:Indexers Ratio
autoLBVolume + autoLBFrequency
Security Through Obscurity

docs.splunk.com “Securing Splunk Enterprise”
Security Through Obscurity

• Change default ports
• Change default system account ($SPLUNK_HOME/etc/default/user-seed.conf)
Auditable Logins
• Empty $SPLUNK_HOME/etc/passwd and $SPLUNK_HOME/etc/.ui_login
• Distribute authentication.conf
“Best practices for Splunk Enterprise security” in docs.splunk.com
Treat pass4symkey seriously. No rogue Search Heads.

MC Deployment
Buddy with License Server
Standalone instance
Conceptually “Admin Console”

• No user stuff
• Only MC apps/jobs
Health Check
wwwaaaaaahhhhh?!
TimeZones
search “splunk answers 776614”
Prevent drift with NTP
Timezones for Splunk infra
U.T.C.
Timezones for forwarders
IT Policy
Boot Start
init.d -> systemd
Introduced with 7.2.2
Default is systemd until 7.3
systemd is the future
sudo OR PolKit
Workload Management
Support Tickets
docs.splunk.com “How to file a great Support case”
Open Cases
• break/fix only
• Details, details, details
• Diags everywhere!
– Remote
– Upload to case
Schedule webex
• Delay and much lost in email
Define Scope
Search “splunk ssf scope”
Service
Catalog

Hiring Practices
search “splunk ssf staffing model”
Complexity
• Distributed Deployment
• Indexer Clustering
• Search Head Clustering
• Data Collection Tier
• Complex Utility Deployments
Work Expectations
• Platform HA means People HA
• SSF staff vs End Users workload
Architect
Developer
Engineer
Executive Sponsor
Search Expert
Knowledge Manager
Program Manager
Project Manager
User Community

Data Management
“Compare QA & PROD…D’oh!”
Non PROD data -> PROD SPLUNK!

• “If a single team depends on it, then it’s production” – Terry Martin
• Or, Search Head traverses
Logical Separation:
• Role Based Access Control
• Separate indexes per env
• Use eventtypes/tags
forwardedindex.filter.disable
docs.splunk
“logging
best
practices”

Tighter Coupling
Use Case with Data
Data in Splunk
That data shown back to you as a use case thereby providing
insights into your business or technology resulting in machine
data being accessible, usable and valuable to everyone.
Tighter Coupling
Use Case with Data
Data in Splunk
That data shown back to you as a use case thereby providing
insights into your business or technology resulting in machine
data being accessible, usable and valuable to everyone.
Onboarding != Ingestion
A David Paper Joint!
SME != admin && admin != SME
Technical SME Product SME

“What index do you need? “I like black turtlenecks.”
Onboarding Phases Ingestion
Fetch from source:

• Read access
• Data volume estimate
• Sample
Use sample for:

• Event Breaks
• Time Stamps
Give ‘em the first hit free!

They’ll be hooked on Splunk!
What to create?
Dashboards and reports and scheduled searches, Oh My
Workflow Phase: Use Case Definition

Alerts vs Dashboards vs Searches
Root Cause Unknown Root Cause Known
Unaware
Issue Exists
Aware
Issue Exists

Listening
Unaware
• Dashboards & Glass Tables
Issue Exists
• Odd symptom combo
Aware
Issue Exists

Listening
Unaware
Issue Exists
Investigating
Aware
• Go Spelunking!
Issue Exists
• Hunting for RC

Listening
Unaware Monitoring
Issue Exists • Scheduled Searches & Alert Actions
Investigating
Aware
• Go Spelunking!
Issue Exists
• Hunting for RC

Listening
Unaware Monitoring
Issue Exists • Scheduled Searches & Alert Actions
Investigating
Aware Attacking
• Go Spelunking!
Issue Exists • Adaptive Response
• Hunting for RC
Live like a ‘user’
Use non-admin account

• Prevents accidents
• Live with limitations
• Appreciate user experience
Admin on MC
User Education & Enablement

Outsource it to us!
Search Tutorial
Splunk Books, documentation
Capture unique things

Community Q&A
answers.splunk.com
E-mail notifications
Fast answers
Larger distribution
Who wants
to role play?
Choose Your Own
Adventure!
• New employee at Buttercup Games

• Lied on your resume about Splunk
experience (no experience)
Scenario
• Company has no HR. Punishment is Pony
Diaper Duty (pun intended)
• Given same Splunk access as peer.

• You log in to see…
All the Dashboards!

Doug M.
D. Merritt
index=*
Splunk Admin
You
Grand Moff Tarkin
Your throat
“eliminating
consumer
choices can
greatly
reduce anxiety”
Oh, the Places You’ll Go

Too many options!
Same Challenge. Different Platforms.

What did this button do for user design?
Mislead?
Restrict?
Guidance!
Confidence!
Comfort!
Eureka! Welcome Page!

Effective material presented at every log in
Burch’s Experience
Same questions and confusions over and over
What is Splunk?
What report/dashboard to use?
What data available?
Want to learn more!

Welcome Email
Lost in their mailbox…
Lost in their mailbox
Static == Ineffective
Requires effort from user

Workspace
Do you keep everyone’s work on everyone’s desk?
…so why do we do that in Splunk?

App as a Workspaces
Discovery, collaboration, resource usage
Incentives
Is EDU Required?
“Yea, I took education”

“But I didn’t care, nor pay attention”
Incentives
Alternative Approach: No Requirements

But limit impact…
You can’t stop splunk-thusiasm… …so shape it in your favor!

Incentive Driven User Onboarding

“I can’t believe those users did those things I let them do!"
Don’t be a data butler
Identify & coach & promote to power
Work with you to implement and learn

best practices
Result
Curiosity and exploration
Rinse & Repeat
Admin teachers power user Power user teaches user

Find Impacting Searches
Search Activity:
• Top 20 Memory-Consuming Searches
Search Usage Statistics

• Long-Running Searches
Great for
• Clean up
• Identifying users to mature
Sub-Concept: For The Nguyen (FTN)

search “splunk ssf role-based data”
Data Access
Search Constraints
Product Feature Capabilities
Knowledge Object Permissions
Default App
• New employee at Buttercup Games

• Lied on your resume about Splunk
experience (no experience)
Scenario
REMIX • Company has no HR. Punishment is Pony
Diaper Duty (pun intended)
• Splunk Admins attended this session!

• You log in to see…
…only what you need

With clear information on where to go/learn next
Follow!
Thank
You!
Go to the .conf19 mobile app to
RATE THIS SESSION

Best Practices and Better Practices For Admins Latest Slides: Collaborate: #Bestpractices Sign Up at HTTP://SPLK - It/slack

Uploaded by

Copyright:

Available Formats

Best Practices and Better Practices For Admins Latest Slides: Collaborate: #Bestpractices Sign Up at HTTP://SPLK - It/slack

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Best Practices and Better Practices For Admins Latest Slides: Collaborate: #Bestpractices Sign Up at HTTP://SPLK - It/slack

Uploaded by

Copyright:

Available Formats

© 2019 S P LUNK INC.