Basic Setup of Splunk for GPG13 & any other environment requiring verifiable protective monitoring

This document outlines the basic steps required after install to bring Splunk up to standard for GPG13 environments, these steps are also commonplace in non GPG13 environments where log data needs to be verified genuine.

Assumptions:y y y Splunk has been installed A listener has been created on port 514 for Syslog and/or Splunk Universal Forwarders are sending data to the indexer (usually on port 9997) A default config is in use.

Things to do (to meet PMC1 of GPG13): y y y Enable IT Data Block Signing Enable Audit Event Signing Enable Event Hashing

Enabling Audit Event Signing The audit processor in Splunk signs audit events by applying a sequence number ID to the event, and by creating a hash signature from the sequence ID and the event's timestamp. Once you've enabled audit signing, you can search for gaps in the sequence of these numbers and find out if your data has been tampered with. For each processed audit event, Splunk's auditing processor computes an SHA256 hash on all of the data. The processor then encrypts the hash value and applies Base64 encoding to it. Splunk then compares this value to whatever key (your private key, or the default keys) you specify in audit.conf. Configuring Configure the following settings of Splunk's auditing feature through audit.conf: Turn on and off audit event signing. Set default public and private keys.

Configure audit.conf Create your own audit.conf. Edit this file in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. For more information on configuration files in general, see how configuration files work. Generate your own keys using genAuditKeys.py in $SPLUNK_HOME/bin/:
# ./splunk cmd python genAuditKeys.py

This creates your private and public keys, y y $SPLUNK_HOME/etc/auth/audit/private.pem $SPLUNK_HOME/etc/auth/audit/public.pem.

To use these keys, set privateKey and publicKey to the path to your keys in your
$SPLUNK_HOME/etc/system/local/audit.conf:

[auditTrail] privateKey = $PATH_TO_PRIVATE_KEY publicKey = $PATH_TO_PUBLIC_KEY

Note: If the [auditTrail] section is missing, audit events are still generated, but not signed. If the publicKey or privateKey values are missing, audit events will be generated but not signed. Once you've configured audit event signing, the sequence number ID that the audit processor assigns to each event lets you detect gaps in data which can identify tampering with the system. You can search the audit events to determine if gaps are detected:

Enabling IT Data Block Signing A signature for a block of IT data involves three things: A hash is generated for each individual event. The events are grouped into blocks of a size you specify. A digital signature is generated and applied to each block of events. Splunk can also encrypt the hash to create a digital signature if you have configured the public and private keys in audit.conf. More details on this are in the Audit Event Signing section. Digital signatures are stored in a database you specify and can be validated as needed. Splunk can demonstrate data tampering or gaps in the data by validating the digital signature at a later date. If the signature does not match the data, an unexpected change has been made. Configuration You configure IT data signing in indexes.conf. Edit this file in $SPLUNK_HOME/etc/system/local/ or in your custom application directory, in $SPLUNK_HOME/etc/apps/. Do not edit the copy in default. For more information on configuration files in general, see "About configuration files". You can: Enable IT data signing and specify the number of events contained in your IT data signatures. Disable IT data signing. Specify the database to store signing data in. We are going to specify the DB & Enable signing in this step. By default, IT data signing is disabled for all indexes. To enable IT data signing, set the blockSignSize attribute in indexes.conf to an integer value greater than 0. This attribute specifies the number of events that make up a block of data to apply a signature to. You must set this attribute for each index using IT data signing. This example enables IT data signing for for the main index and sets the number of events per signature block to 100:
[main] blockSignSize=100

The maximum number of events for the blockSignSize attribute is 2000.

Once this is set, you unfortunatelyl now must reindex your data for this change to take effect, CD t the location of the Splunk binaries & execute:
./splunk stop ./splunk clean all ./splunk start

Depending on what stage you are at & what data is already in Splunk, this might take some considerable time to complete. We don t recommend you continue configuration until all data is reindexed as the experience will be very slow. To view the integrity of indexed data at search time, open the Show source window for results of a search. To bring up the Show source window, click the drop-down arrow at the left of any search result. Select Show source and a window will open displaying the raw data for each search result. The Show source window displays information as to whether the block of IT data has gaps, has been tampered with, or is valid (no gaps or tampering). The status shown for types of events are: Valid Tampered with Has gaps in data

Configuring Event Hashing Event hashing provides a way to detect if events have been tampered with between index time and search time. Event hashes aren't cryptographically secure. Someone could tamper with an event if they have physical access to a machine's file system. You should use event hashing only if you don't have the capability to run Splunk's IT data block signing feature; individual event hashing is more resource intensive than data block signing. Both may be used to provide for added security & verification. When event hashing is enabled, Splunk hashes events with a SHA256 hash just before index time. When each event is displayed at search time, a hash is calculated and compared to that event's index time hash. If the hashes match, the event is decorated in the search results as "valid". If the hashes don't match, the event is decorated as "tampered" (For the CLI: the value of the decoration is stored in the internal field: _decoration). Configuring Turn on event hashing by adding an [eventHashing] section to audit.conf. This is a very simple step Event hashing in search results Splunk provides different visual indicators for your search results depending on the interface you use. In Splunk Web Search results are decorated in Splunk Web with decorations showing whether an event is valid or has been tampered with.

Indicates Valid data

Indicated data which may have been corrupted or tampered with

You now need to:y y y y Configure Mail settings in Splunk Manager Configure Searches Configure Alerts Configure Reports o Searches, alerts & report alongside mapping of Event IDs & SysLog text to be done by SMEs & CLAS consultants, then presented for rule writing & implementation