Pushing Configuration Bundles in

an Indexer Cluster

Kartheek Kolla | Software Engineer, Splunk

Meema Esguerra | Software Engineer, Splunk
 Forward-Looking Statements
During the course of this presentation, we may make forward-looking statements regarding future events or
the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results could
differ materially. For important factors that may cause actual results to differ from those contained in our
forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live
presentation. If reviewed after its live presentation, this presentation may not contain current or accurate
information. We do not assume any obligation to update any forward looking statements we may make. In
addition, any information about our roadmap outlines our general product direction and is subject to change
at any time without notice. It is for informational purposes only and shall not be incorporated into any contract
or other commitment. Splunk undertakes no obligation either to develop the features or functionality
described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in
the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2017 Splunk Inc. All rights reserved.
 Agenda

▶ Overview of Indexer Clustering

▶ Configuration Management in an Indexer Cluster.
• What is a bundle? And why should I care?
• Creating and distributing the bundle to all peers
• Other bundle operations (validate, check restart, and roll back)
• To restart or not to restart
• Phased bundle downloads
▶ Demo
▶ Troubleshooting and Best Practices
▶ QA
Indexer Cluster
Overview
Pushing Configuration Bundles in
an Indexer Cluster
Indexer Clustering
 Why Use Clustering?
Index Replication, Of Course

​Data ​Data ​Data ​Disaster ​Search

Availability Fidelity Resiliency Recovery Affinity
 Elements of an Indexer Cluster

▶ Cluster Master
• Manages the cluster activities (peer addition, configuration distribution)
• Maintains an in-memory state of all the peers & their corresponding buckets, configs
• Orchestrates remedial activities during peer failures
• Tells search heads where to search
▶ Cluster Peer (Indexer)
• Receive and index incoming data (typically from forwarders)
• Replicate data to other peers for data availability
• Respond to the incoming searches by providing search results
• Update cluster master on any state change (peer, buckets, configs etc.)
 Configuration
Management in the
Cluster
Pushing Configuration Bundles in
an Indexer Cluster
 Configuration Bundles
What is a bundle? And why should you care?

The Mysterious Bundle

▶ The set of configuration files and
apps common to all peers
▶ Managed from the master

▶ Distributed to the peers in a single

operation
• i.e. indexes.conf, props.conf,
and transforms.conf
• apps will also include versions of these
configuration files
 Where do the Bundles reside?
Both in the Cluster Master and the Cluster Peers

Cluster Master Cluster Peer

$SPLUNK_HOME/etc/master-apps $SPLUNK_HOME/etc/slave-apps

_cluster _cluster

default default

local local

app1 app1

app2 app2
 © 2017 SPLUNK INC.

To distribute new or edited

configuration files or apps across
peers, execute a Bundle Push
Operation.
 © 2017 SPLUNK INC.

Bundle Pushing
It’s not like replication

Master

bundle push bundle push bundle push

replication replication

Indexer Indexer Indexer

12
 States of a Bundle Push
States in a Bundle Push Status Report

2. Master Validation

1. Creation 3. Peer Validation

5. Rolling Restart 4. Reload

 Validating a Bundle

▶ Validate a bundle with the peers if

bundle is acceptable
▶ $ ./bin/splunk validate cluster-bundle
 Rolling Back a Bundle

use previous bundle

▶ Did something go wrong with the
bundle that was just pushed? You
can rollback the latest bundle
▶ $ ./bin/splunk rollback cluster-bundle
 To Restart or Not To Restart

▶ Configuration changes which need a restart to apply

• Changes to indexes.conf (deleting an index), inputs.conf
• Change home path in index.conf
• Delete an existing App
▶ Configuration changes which do not need a restart to apply
• Adding a new index in indexes.conf, or a new app with reloadable configuration
• Changes or additions to transforms.conf or props.conf
 Checking for Restart

▶ Do you want to know whether your

config changes require a restart?
▶ Allows admins to plan ahead when
the configuration changes will trigger
a rolling restart.
▶ $ ./bin/splunk validate cluster-bundle
–check-restart

Reload and Restart Simulation

 © 2017 SPLUNK INC.

Oh No! Large Bundles

Network Congestion and Timeouts

Master
>1gb

bundle push bundle push bundle push

timeout

Indexer Indexer Indexer

18
 © 2017 SPLUNK INC.

Phased Bundle Downloads

Sending to One Peer at a time

Master
>1gb

bundle push bundle push bundle push

Indexer Indexer Indexer

19
 Enabling Phased Bundle Downloads

▶ This is recommended if a cluster master is pushing large bundles or even a

smaller bundle but to a large cluster (20 or more indexers).
▶ To enable, set the following configuration within server.conf
#inside server.conf
[clustering]
mode=master
max_peers_to_download_bundle = 1
 © 2017 SPLUNK INC.

1. Configuration Bundle
2. Bundle Operations: Apply, Validate,
Check for Restart, Rollback
Recap
Bundle Push and Other 3. States during Bundle Push
Operations

4. Phased Bundle Downloads

 © 2017 SPLUNK INC.

DEMO
Skip Screenshots
 Bundle Operations
Cluster Master UI – Indexer Clustering
Bundle Operations
Configuration Bundle Actions
Bundle Operations
Configuration Bundle Actions
 Bundle Operations
Configuration change to effect bundle push
Bundle Operations
Bundle Push from the UI
 Bundle Operations
Bundle Push from the UI - Stages
 Bundle Operations
Bundle Push from the UI - Stages
 Bundle Operations
Bundle Push from the UI - Stages
 Bundle Operations
Bundle Push from the UI - Stages
 Bundle Operations
Bundle Push from the UI - Complete
Bundle Operations
UI – Before Bundle Push
Bundle Operations
UI – After Bundle Push
 Bundle Operations
UI – Validate and Check Restart
 Bundle Operations
Configuration change to effect bundle Check Restart
 Bundle Operations
Validate and Check Restart from the UI
 Bundle Operations
Validate and Check Restart from the UI
 Bundle Operations
Validate and Check Restart from the UI
 Bundle Operations
Validate and Check Restart from the UI
Troubleshooting
and
Best Practices
Pushing Configuration Bundles in
an Indexer Cluster
 Apply Bundle
Cluster Master – splunkd.log
INFO CMBundleMgr - apply bundle status transitioning from='None', to='Bundle Creation is
in progress.
INFO CMMaster - Creating a new bundle on the master.
INFO CMBundleMgr - apply bundle status transitioning from='None', to='Bundle validation
is in progress.’
INFO CMMaster - Bundle validation is in progress for peers=8A0E551F-227C-48DC-9018-
515A63F971E8,F78272B2-24DC-43BD-A402-A101123F2B4F.
Bundle validation success reported by [F78272B2-24DC-43BD-A402-A101123F2B4F
peer_name=peer1] successful for bundleid=1E704E11131BC274111CD8991E0DE621
INFO CMPeer - Bundle validation success reported by [8A0E551F-227C-48DC-9018-
515A63F971E8 peer_name=peer2] successful for bundleid=1E704E11131BC274111CD8991E0DE621
 Apply Bundle
Cluster Master – splunkd.log
INFO CMBundleMgr - apply bundle status transitioning from='Bundle validation is in
progress.', to='Bundle reload is in progress. Waiting for all peers to return the
status.’
INFO CMMaster - Peers have indicated they are reload capable, issuing bundle reload.
INFO CMPeer - Bundle reload status peer=8A0E551F-227C-48DC-9018-515A63F971E8
peer_name=peer2 restart_required=1 rolling_restart=1 dryrun=0 reasons=[One or more
configs require a restart to take effect. Configs=inputs]
INFO CMPeer - Bundle reload status peer=F78272B2-24DC-43BD-A402-A101123F2B4F
peer_name=peer1 restart_required=1 rolling_restart=1 dryrun=0 reasons=[One or more
configs require a restart to take effect. Configs=inputs]
INFO CMMaster - Restart required to apply the bundle.
INFO CMBundleMgr - apply bundle status transitioning from='Bundle reload is in progress.
Waiting for all peers to return the status.', to='Rolling restart of the peers is in
progress.’
INFO CMMaster - Starting a rolling restart of the peers.
 Apply Bundle
Viewing the cluster bundle

▶ $ ./bin/splunk show cluster-bundle-status

 Check Restart
Cluster Master – splunkd.log
INFO CMMaster - Changing the dryrun bundle on the master,
dryrun='815D4F439CA468F282164B7E456154E7'.
CMBundleMgr - apply bundle status transitioning from='None', to='Bundle validation is in
progress.’
CMMaster - Tracking bundle validation status for peers=F78272B2-24DC-43BD-A402-
A101123F2B4F,8A0E551F-227C-48DC-9018-515A63F971E8
INFO CMPeer - Bundle validation success reported by [F78272B2-24DC-43BD-A402-
A101123F2B4F peer_name=peer1] successful for bundleid=815D4F439CA468F282164B7E456154E7
INFO CMPeer - Bundle validation success reported by [8A0E551F-227C-48DC-9018-
515A63F971E8 peer_name=peer2] successful for bundleid=815D4F439CA468F282164B7E456154E7
INFO CMMaster - CMMaster issuing a dry run of bundle push
 Check Restart
Cluster Master – splunkd.log

INFO CMBundleMgr - apply bundle status

transitioning from='Bundle validation is
in progress.', to='Bundle dryrun reload
is in progress. Waiting for all peers to
return the status.’

INFO CMMaster - Dry Run Complete

 Rollback
Cluster Master – splunkd.log
INFO CMBundleMgr - apply bundle status transitioning from='None', to='Bundle Creation is
in progress.’
INFO CMMaster - Initiating rollback bundle process
INFO CMBundleMgr - Rolling back bundle on the master to previousActive
bundle=ED4C6043FBF5345DBDCD3DD304636F9C previousActive
bundlePath=/Users/kkolla/Work/instances/master/var/run/splunk/cluster/remote-
bundle/971e64c31b6bbb0c0e7479cf500e0647-1501714888.bundle
CMMaster - Tracking bundle validation status for peers=F78272B2-24DC-43BD-A402-
A101123F2B4F,8A0E551F-227C-48DC-9018-515A63F971E8
INFO CMBundleMgr - setting previousActive bundle=ED4C6043FBF5345DBDCD3DD304636F9C to
active bundle=1E704E11131BC274111CD8991E0DE621
INFO CMMaster - Peers have indicated they are reload capable, issuing bundle reload.
INFO CMMaster - Starting a rolling restart of the peers.
 Keeping Track of Bundles
which bundle again?

▶ The bundles folder

$SPLUNK_HOME/var/run/splunk/cluster/remote-bundle

815D4F439CA468F282164B7E456154E7

_cluster

app1

1E704E11131BC274111CD8991E0DE621.tar.gz.previous

ED4C6043FBF5345DBDCD3DD304636F9C.tar.gz.active
 © 2017 SPLUNK INC.

Best Practices

1. Find a reasonable time to do your bundle push

2. Do NOT change slave configurations locally

3. When in doubt, use “—check-restart”

4. Enable phased bundle downloads for large

bundles or large clusters

URLs:
https://docs.splunk.com/Documentation/Splunk/6.6.
3/Indexer/Managecommonconfigurations
https://docs.splunk.com/Documentation/Splunk/6.6.
3/Indexer/Configurationbundleissues
 © 2017 SPLUNK INC.

Thank You
Don't forget to rate this session in the
.conf2017 mobile app