Splunk Training

Manual

Lab #1 Start Splunk Enterprise and Launch Splunk Web

1. Launch Splunk Web
Please enter the following url: http://www.uxcreate.com/guacamole

2. Login Screen
Please enter your username and password to be obtained by the instructor

Lab 2 Navigating Splunk Web

About SplunkHome

Splunk Home is your interactive portal to the data and apps accessible from this
Splunk instance. The main parts of Home include the Splunk Enterprise navigation
bar, the Apps menu, the Explore Splunk Enterprise panel, and a custom default
dashboard (not shown here).

Apps
The Apps panel lists the apps that are installed on your Splunk instance that you
have permission to view. Select the app from the list to open it.
For an out-of-the-box Splunk Enterprise installation, you see one App in the
workspace: Search & Reporting. When you have more than one app, you can drag
and drop the apps within the workspace to rearrange them.

You can do two actions on this panel:

Click the gear icon to view and manage the apps that are installed in your
Splunk instance.
Click the plus icon to browse for more apps to install.

Explore Splunk Enterprise

The options in the Explore Splunk Enterprise panel help you to get started using
Splunk Enterprise. Click on the icons to open the Add Data view, browse for new apps,

open the Splunk Enterprise Documentation, or open Splunk Answers.

About the Splunk bar

Use the Splunk bar to navigate your Splunk instance. It appears on every page in
Splunk Enterprise. You can use it to switch between apps, manage and edit your
Splunk configuration, view system-level messages, and monitor the progress of
search jobs.

The following screenshot shows the Splunk bar in Splunk Home.

The Splunk bar in another view, such as the Search & Reporting app's Search
view, also includes an App menu next to the Splunk logo.

Return to Splunk Home

Click the Splunk logo on the navigation bar to return to Splunk Home from any
other view in Splunk Web.

Settings menu

The Settings menu lists the configuration pages for Knowledge objects, Distributed
environment settings, System and licensing, Data, and Authentication settings. If you do
not see some of these options, you do not have the permissions to view or edit them.

User menu
The User menu here is called "Administrator" because that is the default user name
for a new installation. You can change this display name by selecting Edit account
and changing the Full name. You can also edit the time zone settings, select a
default app for this account, and change the account's password. The User menu is
also where you Logout of this Splunk installation.

Messages menu
All system-level error messages are listed here. When there is a new message to
review, a notification displays as a count next to the Messages menu. Click the X to
remove the message.

Activity menu
The Activity menu lists shortcuts to the Jobs, Triggered alerts, and System
Activity views.

Click Jobs to open the search jobs manager window, where you can view and
manage currently running searches.
Click Triggered Alerts to view scheduled alerts that are triggered. This
tutorial does not discuss saving and scheduling alerts. See "About alerts" in
the Alerting Manual.
Click System Activity to see Dashboards about user activity and status of the
system.

Help
Click Help to see links to Video Tutorials, Splunk Answers, the Splunk Support

Portal, and online Documentation.

Find
Use Find to search for objects within your Splunk Enterprise instance. Find performs
non-case sensitive matches on the ID, labels, and descriptions in saved objects. For
example, if you type in "error", it returns the saved objects that contain the term
"error".

These saved objects include Reports, Dashboards, Alerts, and Data models. The
results appear in the list separated by the categories where they exist.

You can also run a search for error in the Search & Reporting app by clicking
Open error in search.

Lab 4 Searching the tutorial data

Start searching
In this section, you start searching that tutorial data. This topic discusses
searches that retrieve events from the index.
Before you can start this section, you need to first download and add the tutorial
data.
What to search
1. Click Search in the App navigation bar.
2. In the Search landing page, Look at the What to search panel.

3. Click Data Summary.

Review the tutorial data, which represents a fictitious online game store, called
Buttercup Games. The data summary tells you where the data comes from and
what type of data it is. There are five hosts, eight sources, and three source

types. The three source types are Apache web access logs
(access_combined_wcookie), Linux secure formatted logs (secure), and the
vendor sales log (vendor_sales).
Most of this tutorial covers searching the Apache web access logs and
correlating it with the vendor sales logs.
Search assistant
You have data for an online store that sells a variety of games. Try to find out
how many errors have occurred on the site.
1. Open Splunk Search, and type buttercupgames into the search bar.

As you type, the Search Assistant opens. There are two parts to search
assistant: the matching search history and search help.
Search assistant suggests completions for your search based on terms it
matches in your event data. These search completions are listed under
Matching terms or Matching searches. It does not list terms or phrases that do
not exist in your event data. Search assistant also displays the number of
matches for the search term. This number gives you an idea of how many search
results Splunk will return. Here, buttercupgames appears in 36,819 events.

10

Here, search assistant also provides Steps to help you learn How to Search.
Step 1 explains searches to retrieve events with examples for searching with
terms, quoted phrases, boolean operators, wildcards, and field values. Step 2
introduces how to use search commands.
Search assistant has more uses after you start learning the search language.
When you type in search commands, search assistant displays the command
syntax and usage.
If you do not want search assistant to open automatically, click Auto Open to
toggle it off. You can click the down arrow below the search bar to open it back
up again.
Retrieve events from the index
1. Type in keywords to find errors or failures and use Boolean operators: AND,
OR, NOT.
buttercupgames (error OR fail* OR severe)

Boolean operators need to be capitalized. The AND directive is implied between

terms, so you do not need to write it. You can use parentheses to group terms.
When evaluating boolean expressions, precedence is given to terms inside
parentheses. OR clauses are evaluated before AND or NOT clauses.
The asterisk wildcard is used to match terms that start with "fail". These terms
can include: failure, failed, and so on.

11

This search retrieves 427 matching events.

The search command
Each time you type keywords and phrases, you implicitly use the search
command to retrieve events from a Splunk index. The search command lets you
use keywords, quoted phrases, field values, boolean expressions, and
comparison expressions to specify which events you want to retrieve.
You can also explicitly invoke the search command later in the pipeline to filter
search results. Read "Use the search command" in the Search manual.

12

Use fields to search

You can not take full advantage of the more advanced search features in Splunk
Enterprise without understanding what fields are and how to use them.
About fields
When you look at the Data Summary in the search view, you see tabs for the
Hosts, Sources, and Source Types that described the type of data you added to
your Splunk index.

These are also default fields (host, source, sourcetype) that Splunk Enterprise
extracts from the data during indexing. They help to specify exactly which events
you want to retrieve from the index.

What are fields?

Fields exist in machine data in many forms. Often, a field is a value (with a fixed,
delimited position on the line) or a name and value pair, where there is a single
value to each field name. A field can be multivalued, that is, it can appear more
than once in an event and has a different value for each appearance.
Some examples of fields are clientip for IP addresses accessing your Web
server, _time for the timestamp of an event, and host for domain name of a
server. One of the more common examples of multivalue fields is email address

13

fields. While the From field will contain only a single email address, the To and Cc
fields have one or more email addresses associated with them.
In Splunk Enterprise, fields are searchable name and value pairings that
distinguish one event from another because not all events will have the same
fields and field values. Fields let you write more tailored searches to retrieve the
specific events that you want.

Extracted fields
Splunk extracts fields from event data at index-time and at search-time.

Default and other indexed fields are extracted for each event that is processed
when that data is indexed. Default fields include host, source, and sourcetype..
Splunk Enterprise extracts different sets of fields, when you run a search..
You can also use the field extractor to create custom fields dynamically on your
local Splunk instance. The field extractor lets you define any pattern for
recognizing one or more fields in your events.

Find and select fields

1. Go to the Search dashboard and type the following into the search bar:

sourcetype="access_*"
Search for fields use the syntax: fieldname="fieldvalue" . Field names are
case sensitive, but field values are not. You can use wildcards in field values.
Quotes are required when the field values include spaces.
This search indicates that you want to retrieve only events from your web access
logs and nothing else.

14

This search uses the wildcard access_* to match any Apache web
access sourcetype, which can be access_common, access_combined, or
access_combined_wcookie.

2. In the Events tab, scroll through the list of events.

If you are familiar with the access_combined format of Apache logs, you
recognize some of the information in each event, such as:
IP addresses for the users accessing the website.
URIs and URLs for the pages requested and referring pages.
HTTP status codes for each page request.
GET or POST page request methods.

15

These are events for the Buttercup Games online store, so you might recognize
other information and keywords, such as Arcade, Simulation, productId,
categoryId, purchase, addtocart, and so on.
To the left of the events list is the Fields sidebar. As Splunk Enterprise retrieves
the events that match your search, the Fields sidebar updates with Selected
fields and Interesting fields. These are the fields that Splunk Enterprise
extracted from your data.

16

Selected Fields are the fields that appear in your search results. The default
fields host, source, and sourcetype are selected. These fields appear in all the
events.
You can hide and show the fields sidebar by clicking Hide Fields and Show
Fields, respectively.

3. Click All Fields.

The Select Fields dialog box opens, where you can select the fields to show in
the events list.

17

You see more default fields, which includes fields based on each event's
timestamp (everything beginning with date_*), punctuation (punct), and
location (index).

Other field names apply to the web access logs. For example, there are
clientip, method, and status. These are not default fields. They are
extracted at search time.
Other extracted fields are related to the Buttercup Games online store. For
example, there are action, categoryId, and productId.
4. Select action, categoryId, and productId and close the Select Fields
window.
The three fields appear under Selected Fields in the sidebar. The selected fields
appear under the events in your search results if they exist in that particular
event. Every event might not have the same fields.

18

The fields sidebar displays the number of values that exist for each field. These
are the values that Splunk Enterprise indentifies from the results of your search.

5. Under Selected Fields, click the action field.

This opens the field summary for the action field.

In this set of search results, Splunk Enterprise found five values for action, and
that the action field appears in 49.9% of your search results.
6. Close this window and look at the other two fields you selected, categoryId
(what types of products the shop sells) and productId (specific catalog number
for products).

7. Scroll through the events list.

If you click on the arrow next to an event, it opens up the list of all fields in that
event.

19

Use this panel to view all the fields in a particular event and select or deselect
individual fields for an individual event.

Run more targeted searches

The following are search examples using fields.
Example1: Search for successful purchases from the Buttercup Games store.

20

sourcetype=access_* status=200 action=purchase

This search uses the HTTP status field, status, to specify successful requests
and the action field to search only for purchase events.
You can search for failed purchases in a similar manner using
status!=200, which looks for all events where the HTTP status code is
not equal to 200.
sourcetype=access_* status!=200 action=purchase

Example 2: Search for general errors.

(error OR fail* OR severe) OR (status=404 OR status=500 OR

status=503) This doesn't specify a source type. The search retrieves
events in both the secure and web access logs.
Example 3: Search for how many simulation games were bought yesterday.
Select the Preset time range, Yesterday, from the time range picker and run:
sourcetype=access_*

status=200

action=purchase

categoryId=simulation
The count of events returned are the number of simulation games purchased.
To find the number purchases for each type of product sold at the shop, run this
search for each unique categoryId. For the number of purchases made each day
of the previous week, run the search again for each time range.

21

Use the search language

The searches you have run to this point have retrieved events from your Splunk
index. You were limited to asking questions that could only be answered by the
number of events returned.
For example, in the last topic, you ran this search to see how many simulation
games were purchased:
sourcetype=access_* status=200 action=purchase categoryId=simulation

To find this number for the days of the previous week, you have to run it against
the data for each day of that week. To see which products are more popular than
the other, you have to run the search for each of the eight categoryId values and
compare the results.

Learn with search assistant

This section explains in more detail one of the ways you can use the search
assistant to learn about the Splunk search processing language and construct
searches.
1. Return to the search dashboard and restrict your search to Yesterday:

sourcetype=access_* status=200 action=purchase

As you type in the search bar, search assistant opens with syntax and usage
information for the search command (on the right side). If search assistant
doesn't open, click the down arrow under the left side of the search bar.

22

You've seen before that search assistant displays type ahead for keywords
that you type into the search bar. It also explains briefly how to search.
2. Type a pipe character, " | ", into the search bar.
The pipe indicates to Splunk that you're about to use a command, and that you
want to use the results of the search to the left of the pipe as the input to this
command. You can pass the results of one command into another command in a
series, or pipeline, of search commands.

23

You want Splunk to give you the most popular items bought at the online store.
3. Under common next commands, click top.
Splunk Enterprise appends the top command to your search string.

According to search assistant's description and usage examples, the top

command "displays the most common values of a field."

24

4. Type the categoryId field into the search bar to complete your search.
sourcetype=access_* status=200 action=purchase | top categoryId

5. Run the search.

The count of events under the search bar indicates the number of events
retrieved that match the search for sourcetype=access_* status=200
action=purchase. The results of the top command appear in the Statistics
tab.

25

View reports in the Statistics tab

The results of a search are reports. The top command is a transforming
command and returns a tabulated report for the most common values of
categoryId. You can view the results of transforming searches in the
Statistics tab.

In this search for successful (status=200) purchases (action=purchase),

Splunk Enterprise found seven different category IDs. This report lists the
category ID values in order from highest to lowest.
The top command also returns two new fields: count is the number of times each
value of the field occurs, and percent is how large that count is compared to the
total count.

View and format reports in the Visualization tab

You can also view the results of transforming searches in the Visualizations tab
where you can format the chart type. For example, a search using the top
command can be illustrated with a pie chart.
1. Click the Visualization tab.

26

By default, the Visualizations tab opens with a Column Chart.

2. Click Column to open the visualization type selector.

Column, Bar, and Pie charts are recommended for this data set.

3. Select Pie.
Now, your report looks like this:

27

You can turn on drill down to delve deeper into the details of the information
presented to you in the tables and charts that result from your search.
4. Mouse over each slice of the pie to see the count and percentage values for
each categoryId.

28

5. Click on a slice, such as "Strategy".

Splunk Enterprise appends categoryId=strategy to your original search for
successful purchases and runs this new search.

29

30

31

Lab 5 Subsearchs
Use a subsearch
This topic walks you through examples of correlating events with subsearches.
A subsearch is a search with a search pipeline as an argument. Subsearches are
contained in square brackets and evaluated first. The result of the subsearch is
then used as an argument to the primary, or outer, search.

Example 1: Without a subsearch

Let's try to find the single most frequent shopper on the Buttercup Games online
store and what this customer has purchased.
To do this, search for the customer who accessed the online shop the most.
1. Use the top command:
sourcetype=access_* status=200 action=purchase | top limit=1 clientip

Limit the top command to return only one result for the clientip.

32

This search returns one clientip value, which we'll use to identify our
VIP customer.
2. Use the stats command to count this VIP customer's purchases:
sourcetype=access_* status=200 action=purchase clientip=87.194.216.51 |
stats count, dc(productId) by clientip

This search used the count() function which only returns the total count of
purchases for the customer. The dc() function is used to count how many
different products he buys.
The drawback to this approach is that you have to run two searches each time
you want to build this table. The top purchaser is not likely to be the same person
at any given time range.
Example 2: With a subsearch

1. Type or copy/paste the following into the search bar.

sourcetype=access_* status=200 action=purchase [search
sourcetype=access_* status=200 action=purchase | top limit=1 clientip |
table clientip] | stats count, dc(productId), values(productId) by
clientip

Here, the subsearch is the segment that is enclosed in square brackets, [ ]. This
search, search sourcetype=access_* status=200 action=purchase | top

33

limit=1 clientip | table clientip is the same

for the last piped command, | table clientip

as Example 1 Step 1, except

Because the top command returns count and percent fields as well, the table
command is used to keep only the clientip value.

34

These results should match the previous result, if you run it on the same time range.
But, if you change the time range, you might see different results because the top
purchasing customer will be different.

2. Rename the columns to make the information more understandable.

sourcetype=access_* status=200 action=purchase [search sourcetype=access_*
status=200 action=purchase | top limit=1 clientip | table clientip] | stats
count AS "Total Purchased", dc(productId) AS "Total Products",
values(productId) AS "Products ID" by clientip |
rename clientip AS "VIP Customer"

35

Lab 3 Add the sample data into Splunk Enterprise

1. Log

into Splunk.

If you are not in Splunk Home, click the Splunk logo on the
Splunk bar to go to Splunk Home.
Get file tutorialdata.zip
Under Explore Splunk Enterprise, click Add data.

The Add Data view opens. The Add Data displays three options for
adding data, lists of common data types, and add-ons you can use
to extend Splunk Enterprise's capabilities to add data.
Under "How do you want to add data?", click Upload.

36

2. Under Select Source, click Select File

to browse for the tutorial

data or Drop the data file into the outlined box.

Because the tutorial data file is an archived data file, the next
step in the Add Data workflow changes from Set Sourcetype
to Input Settings.
3. Click Next to

continue to Input Settings.

Under Input Settings, you can override the default settings for
Host, Source type, and Index.

37

4. Modify the host

the path name:

4.1. Select

settings to assign host names using a portion of

Segment in path from the menu.

4.2. Type in

1 for the segment number.

5. Click Review to

review your input settings.

38

6. Click Submit
7. .( PLEASE

DO NOT SUBMIT BECAUSE INSTRUCTOR ALREADY

SUBMITTED FOR ENTIRE CLASS)

the data added successfully, click Start

Searching. This opens the Search view and runs a search for the
tutorial data source.

8. To confirm that

39

40

Lab 6 Field Lookup

Use field lookups
This topic takes you through using field lookups to add new fields to your events.
Field lookups let you reference fields in an external CSV file that match fields in
your event data. Using this match, you can enrich your event data by adding
more meaningful information and searchable fields to each event.
Uncompress the following file:
Prices.csv.zip

Find the Lookups manager

1. In the Splunk bar, on the upper right, click Settings.
2. Under Knowledge, click Lookups.

41

This opens the Lookups editor where you can create new lookups or edit existing
ones.

Upload the lookup table file

1. In the Lookups manager under "Actions" for Lookup table files, click Add
new.

42

This takes you to the Add new' lookup table files view where you upload CSV
files to use in your definitions for field lookups.

2. To save your lookup table file in the Search app, leave the Destination app as
search.
3. Under Upload a lookup file, browse for the CSV file (prices.csv) to upload.
4. Under Destination filename, name the file prices.csv.
This is the name you use to refer to the file in a lookup definition.
5. Click Save.
This uploads your lookup file to the Search app and returns to the lookup table
files list.

43

Note: If Splunk does not recognize or cannot upload the file, check that it was
uncompressed before you attempt to upload it again.

Share the lookup table file globally

If the lookup file is not shared, you can not select it when you define the lookup.
1. Go to the Lookup table files list.

44

2. Under Sharing for the prices.csv lookup table's Path, click Permissions.
This opens the Permission dialog box for the prices.csv lookup file.

3. Under Object should appear in, select All apps.

4. Click Save.

Now, the lookup table should be shared with Global permissions.

Add the field lookup definition

1. Return to the Lookups manager.

2. Under Actions for Lookup definitions, click Add New.
This takes you to the Add new lookups definitions view where you define your
field lookup.

45

3. Leave the Destination app as search.

4. Name your lookup prices_lookup.
5. Under Type, select File-based.

46

File-based lookups add fields from a static table, usually a CSV file.
6. Under Lookup file, select prices.csv (the name of your lookup table).
7. Leave Configure time-based lookup and Advanced options unselected.
8. Click Save.
This defines prices_lookup as a file-based lookup.

Share the lookup definition with all apps

1. Return to the Lookup definitions list.

2. Under Sharing for prices_lookup, click Permissions.
The Permission dialog box for the prices.lookup opens.

3. Under Object should appear in, select All apps.

47

4. Click Save.
Now, prices_lookup should be shared with Global permissions.
Make the lookup automatic
1. In the Lookups manager, under Actions for Automatic lookups, click Add
New.

48

This takes you to the Add New automatic lookups view where you configure the
lookup to run automatically.

2. Leave the Destination app as search.

3. Name your automatic lookup price_lookup.
4. Under Lookup table, select prices_lookup.

49

5. Under Apply to and named, select sourcetype and type in

access_combined_wcookie.

6. Under Lookup input fields type in productId in both text areas under
Lookup input fields .

Splunk Enterprise matches the field in the lookup table (which is the one
specified on the left) with the field on the right (which is the field in your events).

50

In this case the field names match.

7. Under Lookup output fields, type in the name of the fields that you want to
add to your event data based on the input field matching and rename the fields.

7.1 In the first text area, type product_name, which contains the descriptive name
for each productId.

7.2. In the second text area, after the equal sign, type productName. This
renames the field to productName.
7.3. Click Add another field to add more fields after the first one.
7.4. Add the field price, which contains the price for each productId. Do not
rename this field.

8. Leave Overwrite field values unchecked.

9. Click Save.
This returns you to the list of automatic lookups and you should see your
configured lookup.

51

Show the new fields in your search results

1. Return to Search.
2. Run the search for web access activity.
sourcetype=access_*

3. Scroll through the list of Interesting Fields in the fields sidebar, and find the
price field.

4. Click price to open its field summary dialog box.

52

5. Next to Selected, click Yes.

6. Close the dialog box.
The price field appears under Selected Fields in the fields sidebar.

6. Repeat Steps 3 to 5 for the productName field.

Search with the new lookup fields

1. Copy and paste or type in the previous subsearch example to see what the
VIP customer bought. This time, replace the productId field with productName.
sourcetype=access_* status=200 action=purchase [search
sourcetype=access_* status=200 action=purchase | top limit=1 clientip |

53

table clientip] | stats count AS "Total Purchased", dc(productId) AS

"Total Products", values(productName) AS "Product Names" by clientip |
rename clientip AS "VIP Customer"

The result is the same as in the previous subsearch example, except that the VIP
customer's purchases are more meaningful with the added descriptive product names.

54

Lab 7 Saving and sharing Reports

About saving and sharing reports
This lab takes you through saving searches and more search examples.
Save as a report

1. Select the time range Yesterday and run the following search
2.
sourcetype=access_* status=200 action=purchase [search
sourcetype=access_* status=200 action=purchase | top limit=1 clientip |
table clientip] | stats count AS "Total Purchased", dc(productId) AS
"Total Products", values(productName) AS "Product Names" by clientip |
rename clientip AS "VIP Customer"

3. Click Save as above the search bar and select Report.

This opens the Save as Report dialog.

4. Enter a Title VIP Customer.

55

5. (Optional) Enter a Description Buttercup Games most frequent shopper.

6. Because the report is a table, for Visualization, click None.

7. To include a Time Range Picker, click Yes.
8. Click Save.
The Your report has been created dialog box opens.

56

There are other options in this window.

Continue Editing lets you refine the search and report format.
Add to dashboard lets you add the report to a new or existing dashboard.
View lets you view the report.
9. Click View.

View and edit saved reports

You can view and edit the saved report from its report view.
1. In the report view for "VIP Customer", click Edit.

57

You can open the report in the search view and edit the saved search's
description, permissions, schedule, and acceleration. You can also clone,
embed, and delete the report from this menu.
2. Click More Info.

You can view and edit different properties of the report, including its schedule,
acceleration, permissions, and embedding.
3. Look at the time range picker, located to the top left.
You saved this report with a time range picker. The time range picker lets you
change the time period to run this search. For example, you can use this time
range picker to run this search for the VIP Customer Week to date, Last 60
minutes, Last 24 hours just by selecting the Preset time range or defining a
custom time range.

58

Find and share saved reports

You can access your saved reports using the app navigation bar.
1. Click Reports to open the Reports listing page.

When you save a new report, its Permissions are set to Private. This means
that only you can view and edit the report. You can allow other apps to view, or

59

edit, or view and edit the reports by changing its Permissions.

1. Under Actions for the VIP Customer report, click Edit and select Edit
Permissions.

This opens the Edit Permissions dialog box.

2. In the Edit Permissions dialog box, set Display For to App and check the
box under Read for Everyone.

60

This action gives everyone who has access to this app the permission to view it.
3. Click Save.
Back at the Reports listing page, you see that the Sharing for VIP Customer now
reads App.

61

About report acceleration

If your search has a large number of events and is slow to finish, you might be able
to accelerate the resulting report so it finishes faster when you run it again. This
option is available when the report produced by your search qualifies for
acceleration. The "VIP Customer" report does not qualify for acceleration, because it
is based on a transforming search.

62

Lab 8 More Searches and Reports

Example 1: Compare counts of user actions

In this example, calculate the number of views, purchases, and adds to cart for
each type of product.
This report requires the productName field from the fields lookup example. If
you did not add the lookup, refer to that example and follow the procedure.
1. Run this search:
sourcetype=access_* status=200 | chart count AS views
count(eval(action="addtocart")) AS addtocart
count(eval(action="purchase")) AS purchases by productName | rename
productName AS "Product Name", views AS "Views", addtocart AS "Adds to
Cart", purchases AS "Purchases"

63

This search uses the chart command to count the number of events that are
action=purchase and action=addtocart.

2. Use the Visualization view options to format the results as a column chart.

Example 2: Overlay Actions and Conversion Rates on one chart

1. Run this search:
sourcetype=access_* status=200 | stats count AS views
count(eval(action="addtocart")) AS addtocart
count(eval(action="purchase")) AS purchases by productName | eval
viewsToPurchase=(purchases/views)*100 | eval
cartToPurchase=(purchases/addtocart)*100 | table productName views
addtocart purchases viewsToPurchase cartToPurchase | rename productName
AS "Product Name" views AS "Views", addtocart as "Adds To Cart",
purchases AS "Purchases"

Instead of the chart command, this search uses the stats command to count the
user actions. Then, it uses the eval command to define two new fields which

64

calculate conversion rates for "Product Views to Purchases" and "Adds to cart to
Purchases".

Steps 2 to 6 reformat the visualization to overlay the Conversion series onto the
Actions series.

2. Click Visualization.
This is the same chart as in Example 1, with two additional series,
"viewsToPurchase" and "cartToPurchase".

65

3. Click Format and X-Axis.

4.

4.1 Rotate the label -45 degrees and do not truncate the label.
4.2 Click Apply.

66

5. Click Format and Y-Axis.

5.1 For Title, choose Custom and type in "Actions".

5.2 Set the Max Value to 2500 and the Interval to 500.
5.3 Click Apply.

6. Click Format and Chart Overlay.

67

6.1 Type in or select the fields, "viewsToPurchase" and "cartToPurchase".

6.2 For View as Axis, click On.
6.3 For Title, choose Custom and type in Conversion Rates.
6.4 For Scale, choose Linear.
6.5 Set the Max Value to 100 and the Interval to 20.
6.6 Click Apply.

68

7. Click Save As and select Report.

7.1 In the Save Report As dialog box, enter a Title, "Comparison of Actions and
Conversion Rates by Product".
7.2 (Optional) Enter a Description, "The number of times a product is viewed,
added to cart, and purchased and the rates of purchases from these actions."
8. Click Save.

69

Example 3: Products purchased over time

For this report, chart the number of purchases that were completed for each item.
This report requires the productName field from the fields lookup example. If you
didn't add the lookup, refer to that example and follow the procedure.
1. Search for:
sourcetype=access_* | timechart count(eval(action="purchase")) by
productName usenull="f" useother="f"

Use the count() function to count the number of events that have the field
action=purchase. Use the usenull and useother arguments to make sure the
chart counts events that have a value for productName.

This produces the following statistics table.

70

2. Click the Visualization tab and Format the X-Axis, Y-Axis, and Legend to
produce the following line chart.

3. Click Save As and select Report.

71

3.1 In the Save Report As dialog box, enter a Title, "Product Purchases over
Time".
3.2 (Optional) Enter a Description, "The number of purchases for each product."

4. Click Save and View the report.

72

Example 4: Purchasing trends

This example uses sparklines to trend the count of purchases made over time.
For stats and chart searches, you can add sparklines to their results tables.
Sparklines are inline charts that appear within the search results table and are
designed to display time-based trends associated with the primary key of each row.
See "Add sparklines to your search results" in the Search Manual.
This example requires the productName field from the fields lookup example. If
you did not add the lookup, refer to that example and follow the procedure.
1. Run the following search:
sourcetype=access_* status=200 action=purchase| chart sparkline(count) AS
"Purchases Trend" count AS Total by categoryId | rename categoryId AS
"Category"

This search uses the chart command to count the number of purchases,
action="purchase", made for each product, productName. The difference is that the
count of purchases is now an argument of the sparkline() function.

3. Click Save As and select Report.

73

4. In the Save Report As dialog box, enter a Title, "Purchasing trends".

5. (Optional) Enter a Description, "Count of purchases with trending."
6. Click Save and View the report.

74

Lab 9 Dashboards
About dashboards
Dashboards are views that are made up of panels that can contain modules
such as search boxes, fields, charts, tables, and lists. Dashboard panels are
usually hooked up to saved searches.
After you create a visualization or report, you can add it to a new or existing
dashboard using the Save as report dialog box. You can also use the
Dashboard Editor to create dashboards and edit existing dashboards. Using the
Dashboard editor is useful when you have a set of saved reports that you want to
quickly add to a dashboard.
Change dashboard permissions
You can specify access to a dashboard from the Dashboard Editor. However,
your user role (and capabilities defined for that role) might limit the type of access
you can define.
If your Splunk user role is admin (with the default set of capabilities), then you
can create dashboards that are private, visible in a specific app, or visible in all
apps. You can also provide access to other Splunk user roles, such as user,
admin, and other roles with specific capabilities.
Change dashboard panel visualizations

After you create a panel with the Dashboard Editor, use the Visualization Editor
to change the visualization type in the panel, and to determine how that
visualization displays and behaves. The Visualization Editor lets you choose from
visualization types that have their data structure requirements matched by the
search that has been specified for the panel.

75

Edit the XML configuration of a dashboard

Although you are not required to use XML to build dashboards, you can edit a
dashboard's panels by editing the XML configuration for the dashboard. This
provides editing access to features not available from the Dashboard Editor. For
example, edit the XML configuration to change the name of dashboard or specify
a custom number of rows in a table.

Create dashboards and dashboard panels

This topic walks you through saving a search as a dashboard panel and adding
an input element to the dashboard.
Save a search as a dashboard panel
1. Run the following search:
sourcetype=access_* status=200 action=purchase | top categoryId

76

2. Click the Visualization tab and select the Pie chart type.

3. In the Search view, click Save as and select Dashboard Panel.

The Save as Dashboard Panel dialog box opens.

4. Define a new dashboard and dashboard panel.

77

4.1. For Dashboard, click New.

4.2. Enter the Dashboard Title, "Buttercup Games Purchases", The Dashboard
ID updates with "Buttercup_games_purchases".
4.3. (Optional) Add a Dashboard Description, "Reports on Buttercup Games
purchases data".
4.4. Type in the Panel Title, "Top Purchases by Category"
4.5. Leave the Panel Powered By as Inline search.
5. Click Save.

78

6. Click View Dashboard.

This creates a dashboard with one report panel. To add more report panels, you
can run new searches and save them to this dashboard, or you can add saved
reports.
View and edit dashboard panels
1. Click Dashboards in the app navigation bar.
This takes you to the Dashboards listing page.

79

You can Create a new dashboard and edit existing dashboards. You see the
Buttercup Games Purchases dashboard that you created.

2. Under the i column, click the arrow next to Buttercup Games Purchases to
see more information about the dashboard: What app context it is in, whether or
not it is scheduled, and its permissions.

You can use the quick links that are inline with the information to edit the
dashboard's Schedule and Permissions.
Add an input to the dashboard
1. In the Dashboards list, click Buttercup Games Purchases to return to that
dashboard.

80

2. Click Edit and select Edit Panels.

The Edit: Buttercup Games Purchases view opens.

In this view, you have edit buttons: Add Input, Add Panel, and Edit Source.

3. Click Add Input and select Time.

81

This adds a time range picker input to the dashboard editor.

4. Click the Edit Input icon for the time range picker. It looks like a pencil.
This opens a set of input controls. The Time input type should be preselected.

5. Change the Token value to Buttercup_Games_Time_Range and click

Apply.

This optional step redefines the name of the input token for the time range picker.
Because the default names of input tokens are not very descriptive (field1, field2,

82

field3, and so on), you may want to do this when you give your dashboard
multiple inputs. It makes it easier to understand which input you are working with.
You can also optionally change the default time range for the picker by changing
the value of Default. Right now it defaults to All time.
In the next two steps you connect your dashboard panel to this time range picker.
6. In the new dashboard panel, click the Inline Search icon and select
Edit Search String.
The Edit Search dialog opens.

7. Click Time Range Scope and select Shared Time Picker

(Buttercup_Games_Time_Range).
8. Click Save.

83

The panel is now hooked up to the shared time range picker input. The inline
search that powers the panel now uses the time range selected for the shared
time range picker.
As you add panels to this dashboard, repeat steps 6 through 8 to hook the new
panels up to the shared time range picker input.
You can have dashboards that offer a mix of panels that work with the shared
time range picker and panels that show data for fixed time ranges.
9. Click Done to save your changes to the dashboard.

Add more panels to the dashboard

In the previous section, you ran searches and saved them as reports. In this
topic, you add the saved reports to an existing dashboard.
Add saved reports to the dashboard
1. Return to the Buttercup Games Purchases dashboard.

84

2. Click Edit and select Edit Panels.

85

3. In the Buttercup Games Purchases dashboard editor, click Add Panel.

4.

The Add Panel sidebar menu slides open.

5. To add a new panel from a report, click New from Report.

This opens the list of saved reports.

86

6. Select Purchasing Trends.

This opens a preview of the saved Report.

7. Click Add to Dashboard.

87

The new panel is placed in the dashboard editor. You can click anywhere to
close the Add Panel sidebar menu or choose another report to add to the
dashboard.
Note: If you want the new panels to work with the shared time range picker input,
repeat steps 6 through 8 from the "Add an input to the dashboard" procedure to
connect them to that input.

8. Select the report Comparison of Actions and Conversion Rates by

Product and add it to the dashboard.

88

9. Close the Add Panel sidebar and rearrange the panels on the dashboard.
While in the dashboard editor, you can drag and drop a panel to rearrange it on
the dashboard.

10.

Click Done.

Your finished dashboard should look like this:

89

More dashboard actions

After you complete the dashboard, you can Export to PDF and Print the
dashboards using the buttons to the upper right. You can also share the
dashboard with other users by changing its permissions.

90

Lab 10 Create a new data model

This topic shows you how to create new data models based on the tutorial data. Data
models are created within Pivot and you need to have admin or power role to create
a data model.

Enable roles to create a data model

By default only users with the admin or power role can create data models. For
other users, the ability to create a data model is tied to whether their roles have
"write" access to an app. Since this is a first time install, you have admin privileges
by default and should be able to continue.
If you are not able to create or edit a data model, you may need to check your
permissions. For more information, read "About data model permissions" in the
Knowledge Manager Manual.

Navigate to the Data Models management page

1. In the Splunk bar, click Settings.

91

2. Under Knowledge, click Data Models.

This takes you to the Data Models management page. The Data Models
management page is a listing page of data models. If you have existing data models
in this Splunk Enterprise instance, this page lists them. Use this page to manage the
permissions, acceleration, cloning, and removal of existing data models. You can
also use this page to upload a data model or create new data models, using the
Upload Data Model and New Data Model buttons on the top right.

Create a new data model

1. In the Data Models management page, click New Data Model.
This opens the New Data Model dialog box.

92

2. Enter the Title, "Buttercup Games"

The Title field accepts any character, as well as spaces. The value you enter here
is what appears on the data model listing pages.
3. (Optional) Enter the ID, "Tutorial"
If you don't change the ID, it automatically reads "Buttercup_Games".
The ID must be a unique identifier for the data model. It cannot contain spaces or any
characters that aren't alphanumeric, underscores, or hyphens (a-z, A-Z, 0-9,
_, or -). Spaces between characters are also not allowed. Once you define the data
model ID, you can't change it.
4. Next to App, select "Search & Reporting" from the menu.
5. (Optional) Enter the Description, "Enables data analysis and reporting for
tutorial data."
6. Click Create.
This opens the Buttercup Games Edit Objects page.

Use this page to create objects for the new data model, define their constraints and
attributes, arrange the objects in logical hierarchies, and manage them.

93

Lab 11 Define a root object for the data model

In the last lab, you created the data model "Buttercup Games".
This lab walks you through adding a root object for Buttercup Games
purchases.

Edit data model objects

1. From the Data Models list, click Buttercup Games.

This opens the Buttercup Games object editor view.

Use the Edit Objects page to design a new data model or redesign an existing
data model. On the Edit Objects page, you can create objects for your data
model, define their constraints and attributes, arrange them in logical object
hierarchies, and maintain them.

Add a root object

Data models are typically composed of object hierarchies built on root event
objects. Each root event object represents a set of data that is defined by a

94

constraint, which is a simple search that filters out events that are not relevant to
the object.
Let's create an object to track purchase requests on the Buttercup Games
website.

1. To define the data model's first event base object, click Add Object.

Your first root object can be either a Root event or Root search.

2. Select Root event.

This takes you to the Add Event Object editor.

95

3. Enter the Object Name: Purchase Requests

The Object Name field can accept any character, as well as spaces. It's what
you'll see on the Choose an Object page and other places where data model
objects are listed.
4. Enter the Object ID: Purchase_Requests
This should automatically populate when you type in the Object Name. You can
edit it if you want to change it.
The Object ID must be a unique identifier for the object. It cannot contain spaces
or any characters that aren't alphanumeric, underscores, or hyphens (a-z, A-Z,
0-9, _, or -). Spaces between characters are also not allowed. Once you save the
Object ID value, you can't edit it.
5. Enter the following search Constraints: sourcetype=access_*
action=purchase

This defines the web access page requests that are purchase events.
After you provide Constraints for the event base object you can click Preview to
test whether the constraints you've supplied return the kinds of events you want.

96

6. Click Save.

97

The list of attributes for the root object include: host, source, sourcetype, and
_time. If you want to add child objects to client and server errors, you need to edit the
attributes list to include additional attributes.

98

Lab 12 Designing a Pivot report

About Pivot
The Splunk Enterprise Pivot tool lets you quickly design reports with tables and
data visualizations that present different aspects of a selected Data Model. Pivot
lets you generate these reports with a UI interface instead of having to use the
search processing language.

Pivot views
Pivot is part of the Search & Reporting app.

1. On the Search & Reporting app's navigation bar, click Pivot.

Entering Pivot takes you to the Select a Data Model page, where you should
see a list of the data models if any have been created. For example, this list

99

includes the Buttercup Games data model that you created earlier in this
tutorial. It also includes two sample data models that track Splunk Enterprise
internal and audit logs.
If you view Pivot in smaller browser windows, the Search & Reporting app's
navigation bar is hidden. To use the navigation bar, click the menu icon on the
upper right. The navigation bar slides down.

100

2. Use the arrows under the i column to view information for Buttercup Games.

Clicking Edit objects takes you to the object editor for the Buttercup Games data
model.

3. Click Buttercup Games.

This takes you to the Select a Data Object view. This view lists all the objects
that have been created for this data model. The Buttercup Games data model
consists of the Purchase Requests parent object and the Successful Purchases
and Failed Purchases child objects.

4. Use the arrows under the i column to view the information for the objects.

101

5. Click Purchase Requests.

102

Selecting an object from the Select a Data Object view takes you to the New
Pivot editor for that data model.

Components of Pivot

The following illustrates the Pivot editor components.

Visualization types: The left-hand vertical bar contains icons that represent
different visualization types. Selecting a different icon controls which Pivot builder
and report interfaces display. Visualization types are: Statistics Table (default),
Column Chart, Bar Chart, Scatter Chart, Bubble Chart, Area Chart, Line Chart,
Pie Chart, Single Value Display, Radial Gauge, Marker Gauge, and Filler Gauge.
Document Actions: The upper horizontal bar displays document-related
actions. These actions include:
Save as...: Save the current report as a new one (Report) or as a

103

dashboard panel (Dashboard Panel).

Clear: Reset the interface to its initial state, which will dismiss the saved
report (if applicable), change the visualization type to Statistics Table, and
populate the report with a single Column Value for the count of the object
and a time filter for all time (if _time is an applicable field).
Data model object: This is the right-most button. It takes its label from the
data model object that was selected. For example, in the screenshot it is
"Purchase Requests". Use this menu to navigate back to the list of data
models (Select another Data Model), navigate back to the list of data
model objects (Select another Object), or edit the selected data model
object (Edit Object). Additionally, you can rebuild acceleration and inspect
the acceleration job.

104

Job Actions: The Pause and Stop buttons control the progress of the Pivot job.
Other actions include: Share, Export, Print, and Open in Search. Clicking
Open in Search opens the Search view and runs the current search string.

105

Create and save a Pivot

This topic shows you how to use pivot to create and save a simple report.
This is a very simple example. More complicated examples are shown in later
topics of this tutorial.

Create a new Pivot

When you set out to design a report, you first need to select a data model that
represents the broad category of event data that you want to work with. For this
tutorial, that data model is the "Buttercup Games".

106

1. From the app navigation bar, select Pivot to enter the Select a Data Model
page.
2. In the data models list, click Buttercup Games.
This takes you to the Select an Object page.

The Buttercup Games data model has a root object to track Purchase Requests
from the game website. The Purchases object breaks down into Successful and
Failed purchases.
3. Select "Purchase Requests".
This opens a New Pivot editor for the Purchase Requests object.

107

By default, the Pivot Editor interface displays elements to define a pivot table.
There are four basic pivot element categories: Filters, Split Rows, Split Columns,
and Column Values. When you first open the Pivot Editor for a specific object,
only two elements will be defined:
A time range Filter element (set to All time).
A Column Values element (set to "Count of <object_name>".
This gives you the single value, which is the total count of events returned by the
object over all time. In this case, this count is the "Count of Purchase Requests".
4. Select the Single Value Display element from the visualization bar.

108

4.a Next to Under Label, type Purchase Requests.

By default, the time range filter element is set to All time.

Single value visualizations (single value, the three gauge types) use the
first column value element to get their single value. Here, the field is
"Count of Purchase Requests".
Single value visualizations do not use Split Row or Split Column elements.

Save the Pivot as a report

After you define a pivot, you can save it as either a report or a dashboard panel.
In this example, you save the single value display as a report. Dashboards and
dashboard panels are discussed in a later chapter.
1. Click Save As... and select Report.

109

The Save as Report dialog box opens.

2. Enter a Title "Total Purchase Requests" and Description (optional).

110

3. Select Yes to include the time range picker. (This should be the default.)
4. Click Save.
After the report saves, a window displays that "Your report has been created". You
can continue editing the current Pivot, add the pivot to a dashboard, change
additional settings for the saved report, or view the report.
5. Click View to view the report.

111

View saved reports

A report that is created from Pivot will always be saved under the current app and
owner namespace.
1. Click Reports in the app navigation bar to view the list of all saved reports.

2. Use the arrow in the i column to view information about Total Purchase
Requests report.

3. Click the report name to view the report.

112

Lab 13 Pivots and Visualizations

Create a pivot table
In the previous topic you used pivot to find the total number of purchase requests
and saved the single value display as a report. In this topic, you will use the pivot
visualization editor to create a pivot table of the Buttercup Games Successful
Purchases object.
The Successful Purchases object has attributes for the products purchased from
the Buttercup Games website. This includes the automatically extracted
attributes (categoryId and productId) as well as the lookup attributes (price and
product_name).
The Buttercup Games online store offers hundreds of products, of a variety of
categories, and you want to know more about the items that were purchased
over the past week. You can create a pivot report that breaks down the total
number of purchase events by product name, and through that quickly see which
of your products were the top sellers for that period.

Define a new Pivot

1. From the app navigation bar, select Pivot to enter the "Select a Data Model"
page.

2. Choose the Buttercup Games data model and select the Successful
Purchases child object.

113

The New Pivot editor for Successful Purchases opens.

Add pivot elements

You can add multiple elements from each pivot element category to define your
pivot table. It's easy to add, define, and remove pivot elements in the process of
determining what information your table should provide.
To add a pivot element: Click the + icon. This opens up the element
dialog, where you choose an attribute and then define how the element

114

uses that attribute.

To inspect or edit an element: Click the "pencil" icon on the element.
This opens the element dialog.
To reorder and transfer pivot elements: Drag and drop an element
within its pivot element category to reorder it. Drag and drop elements
between element categories to transfer them.
To remove pivot elements from the Pivot Editor: Open its element
dialog and click the Remove button, or drag the element up or down until
it turns red and drop it.
Under Filters, the time filter is always present when you build a pivot; you cannot
remove it. It defines the time range for which the pivot returns results. It operates
exactly like the time range menu that is in use throughout Splunk Web.

Change the time range filter

Currently your Pivot table shows a single value, the total count of Successful
Purchases over All time.
Change the time filter to view the Successful Purchases over a different time
range:

1. Under Filter, click the pencil next to All time to open the time range picker.

115

2. Under Presets and Relative, click "Last 7 days".

(If this shows no events, you can select "All time" and continue.)

Add a Split Row element

Add Pivot elements to see the Count of Successful Purchases for each product
by name:
1. Under Split Rows, click + and select productName, the lookup field that
contains the name of each product, based on the productId.

116

This opens a dialog box that lets you format the field.

2. Rename the field, Product Name and Click Add To Table.

117

Add a Column Value element

Add a Column Value to see total earned for each product that was successfully
purchased:
1. Under Column Values, click + and select price.

2. In the dialog box, format the field:

118

2.a Enter the label Total Revenue.

2.b Select the Value Sum.
This creates a field called Total Revenue, which is the summation of the price
for each successful purchase of the product. (You can add the price values as
another Split Row, if you want to see the cost of each individual product in this
table.)
3. Click Add To Table.

119

Save the Pivottable

Save the Pivot table as a report named Purchases by Product.

1. Click Save as and select Report.

2. In the Save as Report dialog box:

2.a Enter the Title "Purchases by Product".

2.b (Optional) Add the Description "Table of Product Purchases".
2.c Include a Time Range Picker.
3. Click Save.

4. In the Your Report Has Been Created dialog box, click View.

120

Create a pivot chart

In the previous topic you used Pivot visualization editor to build a table. In this
topic, you will use the same object to create chart visualizations.

Define a new Pivot

1. From the app navigation bar, select Pivot to enter the "Select a Data Model"
page.

2. Choose the Buttercup Games data model and select the Successful
Purchases child object.

121

The New Pivot editor for Successful Purchases opens.

Visualization types are listed in the black sidebar that runs down the left-hand
side of the Pivot editor. By default, the statistics table visualization is selected
when you enter Pivot.
It can be helpful to begin building your pivot as a table and then switch over to
the visualization of your choice. When you switch between pivot visualization
types, Pivot will find the elements it needs to create the visualization, discard the
elements it does not need, and notify you when needed elements need to be
defined. This happens when you switch between tables and charts as well as
between chart types.

Add Pivot elements

In the last topic, we looked at purchases by product ID and name. Now, let's
report on the count of successful purchases by category.

122

Add a Split Row for the categoryId field.

1. Under Split Rows, click + and select categoryId from the list.

2. Enter the label Category and click Add to table.

This returns the following Pivot table.

123

Change the visualization type

1. Click the Column Chart icon from the visualization bar.

The New Pivot editor for the Column chart displays.

124

 Column charts use the first split row element in pivot table definitions to
provide their X-axis values. In this case, that Split Row is Category.
Column charts use the first column value element in pivot table definitions to
provide their Y-axis values. Here, that Column Value is Count of Successful
Purchases.

This data can also be visualized as a pie chart.

2. Click the Pie Chart icon from the visualization bar:

125

The New Pivot editor for the Column chart displays.

Pie charts use the values from the first Split Row element (Category) to
determine the number and colors of their slices.
Pie charts use the first Column Value element (Count of Successful
Purchases) to determine the relative sizes of their slices.

126

Mouseover a slice of the pie chart to view the metrics: Category, Count of
Successful Purchases, and percentage of the total Count of Successful
Purchases.

127

Lab 14 Pivots and Dashboards

Creating a dashboard

About dashboards
Splunk Enterprise makes it easy to interactively build and edit dashboards
without writing a single line of XML code.
Add a pivot you have just created to a new or existing dashboard:
You can jump right into dashboard creation after creating a pivot
visualization you like with the Create Dashboard Panel feature. It guides
you through the process of creating a dashboard panel based on the
search and adding it to a new or preexisting dashboard. When you finish,
you are still in the Pivot view.
Use the Dashboard Editor to create dashboards and populate them
with dashboard panels: You can also use the Dashboard Editor to edit
existing dashboards. This method of dashboard creation is useful if you
have a set of pivot reports that you want to quickly base a set of
dashboard panels upon.

Change dashboard permissions

You can specify access to a dashboard from the Dashboard Editor. However,
your user role (and capabilities defined for that role) may limit the type of access
you can define.
If your user role is admin (with the default set of capabilities), then you can create
dashboards that are private, visible in a specific app, or visible in all apps. You
can also provide access to other user roles, such as user, admin, and other roles
with specific capabilities.
For additional information on setting up permissions for dashboards and other
knowledge objects refer to "Manage knowledge object permissions" in the Admin
Manual.

128

Change dashboard panel visualizations

After you create a panel with the Dashboard Editor, use the Visualization Editor
to change the visualization type displayed in the panel, and to determine how
that visualization displays and behaves. The Visualization Editor only allows you
to choose from visualization types that have their data structure requirements.

Save a Pivot as a dashboard panel

You just created a pie chart in the last lab, now let's save it to a dashboard
panel.
1. Click Save as and select Dashboard panel.

This opens the Save as Dashboard Panel dialogue.

129

2. Define a new dashboard to save the panel to:

For Dashboard, click New.
Enter the Dashboard Title: Buttercup Games. The Dashboard ID will
update with Buttercup_games.
(Optional) Add a Dashboard Description: Reports on Buttercup Games
online shop data.
3. Define the dashboard panel:
Enter the Panel Title: Successful Purchases by Category
Leave the Panel Powered BY as Inline search.
4. Click Save.
The dashboard was successfully created.
5. To continue, click View Dashboard.

130

View and edit dashboard panels

After you save a dashboard, you can access it by clicking Dashboards in the
app navigation bar.
1. Click Dashboards in the app navigation bar.
This takes you to the Dashboards listing page.

You can Create a new dashboard and edit existing dashboards. You see the
Buttercup Games dashboard you just created.

2. Under the i column, click the arrow next to Buttercup Games to see more
information about the dashboard: What app context it is in, whether or not it is
scheduled, and its permissions.

131

There are also quick links to edit the dashboard's Schedule and Permissions
inline with the information.
To view the dashboard, click the dashboard's Title or select the Edit option
under Actions.
Note: If you click to view a dashboard and you cannot view it (or it displays
blank), check that you have read access to the data model. To do this, go to the
Manage Data Models view and edit the Permissions for the Buttercup Games
data model to share in the App.

Add an input to the dashboard

1. In the Dashboards list, click Buttercup Games to return to that dashboard.
2. Click Edit and select Edit Panels.
The Edit: Buttercup Games view opens.

132

In this view, you have edit buttons: Add Input, Add Panel, and Edit Source.

3. Click Add Input and select Time.

This adds a shared time range picker input to the dashboard editor.

133

4. Click the Edit Input icon for the time range picker. It looks like a pencil.
This opens a set of input controls. The Time input type should be preselected.

5. Change the Token value to Buttercup_Games_Time_Range and click

Apply.

This optional step redefines the name of the input token for the time range picker.
Because the default names of input tokens are not very descriptive (field1, field2,

134

field3, and so on), you may want to do this when you give your dashboard
multiple inputs. It makes it easier to understand which input you are working with.
You can also optionally change the default time range for the picker by changing
the value of Default. Right now it defaults to All time.
In the next two steps you connect your dashboard panel to this time range picker.
6. In the new dashboard panel, click the Inline Pivot icon and select Edit
Search String.
The Edit Search dialog opens.

7. Click Time Range Scope and select Shared Time Picker

(Buttercup_Games_Time_Range).
8. Click Save.

135

The panel is now hooked up to the shared time range picker input. The inline
search that powers the panel now uses the time range selected for the shared
time range picker.
As you add panels to this dashboard, repeat steps 6 through 8 to hook the new
panels up to the shared time range picker input.
You can have dashboards that offer a mix of panels that work with the shared
time range picker and panels that show data for fixed time ranges.
9. Click Done to save your changes to the dashboard.

Add saved reports to the dashboard

Add another panel using one of the saved reports you created earlier.
1. In the Buttercup Games dashboard, click Edit and select Edit Panels.

136

2. In the Edit: Buttercup Games view, click Add Panel.

The Add Panel sidebar menu slides open.

3. To add a new panel from a report, click New from Report.

137

4. Click Total Purchase Requests.

This slides open a preview panel with information about the saved report.

4.a Click Add to Dashboard.

The new panel is placed in the dashboard editor. You can click anywhere to
close the Add Panel sidebar menu or choose another report to add to the
dashboard.

138

Before you close the Add Panel sidebar menu, add a second report.
5. Click Purchases by Product.

5.a Click Add to Dashboard.

6. Close the sidebar menu.
While in the dashboard editor view, drag and drop the panels to rearrange them
on the dashboard.
7. Click Done.
Your dashboard should look like this:

139

140

Lab 15 Alerts
Create alerts

Create scheduled alerts

A scheduled alert evaluates the results of a historical search that runs over a
specified time range on a regular schedule. The alert fires when it encounters the
trigger condition.
For example, you can create a scheduled alert to monitor online sales. The
search runs daily at midnight and triggers when the sum of the sales of a specific
item is below 500 for the previous day. When the alert triggers, it sends an email
to the appropriate administrators monitoring sales.
1. From the Search Page, create the following search. Select Last 24 Hours
for the time range:
index=_internal (log_level=ERROR OR log_level=WARN* OR
log_level=FATAL OR log_level=CRITICAL) | stats count as
log_events

2. Select Save As > Alert

The Save As Alert dialog box opens.
3. Specify Settings:
Title: Server Errors Last 24 hours
Alert Type: Scheduled
Time Range: Run Every Day
Schedule At: 0:00
Trigger Condition: Number of Results
Trigger if number of results: is Greater than 5
4. Specify Trigger Conditions:
Trigger alert when: Number of Results is Greater than 5
Trigger it: Once
5. Specify Trigger Actions:

141

 Add Actions: List in Triggered Alerts

See Set up alert actions for information on other actions.
6. Click Save.

Use cron notation for scheduled alerts

When scheduling an alert, you can use cron notation for customized schedules.
When specifying a cron schedule, only five cron parameters are available, not
six. The sixth parameter for year, common in other forms of cron notation, is not
available.
The following cron parameters:
* * * * *

correspond to:
minute hour day month day-of-week

Following are some cron examples:

*/5 * * * *

Every 5 minutes.

*/30 * * * *

Every 30 minutes.

0 */12 * * *

Every 12 hours, on the hour.

*/20

Every 20 minutes, Monday through Friday.

* * * 1-5

0 9 1-7 * 1

First Monday of each month, at 9am.

When you select Run on Cron Schedule for the time range of a scheduled alert,
enter the earliest and latest parameters for a search. What you enter overrides
the time range you set when you first ran the search.
To avoid overlaps or gaps, the execution schedule should match the search time
range. For example, to run a search every 20 minutes the search's time range
should also be 20 minutes (-20m).

142

Manage the priority of concurrently scheduledsearches

Depending on your Splunk Enterprise deployment, you might be able to run only
one scheduled search at a time. In this case, when you schedule multiple
searches to run at approximately the same time, the search scheduler ensures
that all scheduled searches run consecutively for the period of time over which
they gather data.
However, you might have cases where you need certain searches to run ahead
of others. This is to ensure that the searches obtain current data or to ensure that
there are no gaps in data collection.
You can configure the priority of scheduled searches in the savedsearches.conf
configuration file.

143

Best practices for scheduled alerts

This section discusses some best practices for scheduled alerts.

Coordinate an alert's schedule with the search time range

Coordinating the alert's schedule with the search time range prevents situations
where event data is evaluated twice by the search. This can happen if the search
time range exceeds the search schedule, resulting in overlapping event data
sets.
In cases where the search time range is shorter than the time range for the
scheduled alert, an event might never be evaluated.

Schedule alerts with at least 60 seconds of delay

This practice is important in distributed search deployments where event data
might not reach the indexer precisely at the moment when it is generated. A
delay ensures that you are counting all events, not just the events that were
quickest to get indexed.

Best practices example

This example shows how to configure an alert that builds 30 minutes of delay into
the alert schedule. Both the search time range and the alert schedule span one
hour, so there are no event data overlaps or gaps.

144

The alert runs every hour at the half hour. It collects an hour's worth of event
data, beginning an hour and a half before the search runs. When the scheduled
search kicks off at a designated time, such as 3:30 pm, it collects the event data
that was indexed from 2:00 pm to 3:00 pm.
1. From the Search Page, create a search and select Save As > Alert.
2. In the Save As Alert dialog, specify the following to schedule the alert:
Title: Alert Example (30 Minute Delay)
Alert Type: Scheduled
Time Range: Run on Cron Schedule
Earliest: -90m
Latest: -30m
Earliest and Latest values set the time that the search covers to a
period that begins 90 minutes before the search launch time,
ending 30 minutes before the search launch time.
Cron Expression: 30 * * * *
The alert runs every hour on the half hour

3. Continue defining actions for the alert.

Set up triggering conditions for a scheduled alert

Trigger conditions apply to two types of conditional alerts:
Basic conditional alert
Advanced conditional alert

145

Set the triggering conditions when you set values for the Trigger condition field
in the Save As Alert dialog box, as described in the following subtopics.

Basic conditional alert

A basic conditional alert triggers when the number of results of a scheduled
search meet, exceed, or are less than a specified numerical value. When you
create the alert, you can specify the following conditions:
Number of results
Number of hosts
Number of sources
The alert triggers when the number of hosts in the results rises by a count of
more than 12.
1. From the Search Page, create a search and select Save As > Alert.
2. In the Save As Alert dialog box, specify the following fields to schedule
the alert:
Title: Alert Example (Basic Conditional)
Alert Type: Scheduled
You can also select Real Time for a basic conditional search.
Time Range and Schedule: Select any time range and schedule.
Trigger Condition: Number of Hosts
You can also select Number of Results or Number of Sources
Trigger if number of results: Select a comparison operator and
trigger value.

3. Continue defining actions for the alert.

146

Basic conditional alert for rolling-window alerts

The behavior for basic conditional alerts differs slightly for a rolling-window alert.
The alert triggers when the set condition occurs within the rolling time window of
the search.
For example, a rolling-window alert that triggers when a time window of 60
seconds has five or more results. If the real-time search returns one result and
then four more results five minutes later, the alert does not trigger. The alert does
trigger If the search returns five results within a single 60-second span.

Advanced conditional alert

An advanced conditional alert uses a secondary, custom conditional search to
evaluate the results of a scheduled or real-time search. The alert triggers when
the custom search returns any number of results. If the alerting conditions are not
met, then the custom conditional search returns zero results.
A secondary conditional search can help reduce the incidence of false positive
alerts.
In the following example, the alert triggers when there are 10 or more log level
events that are not INFO. When the alert triggers, it sends an email with the
results of the search. The search results detail the count for each log level.
1. From the Search Page, create the following search. Specify Last 7 days
for the time period.
index=_internal (log_level=ERROR OR log_level=FATAL OR
log_level=CRITICAL) | stats count by log_level

2. Select Save As > Alert.

3. In the Save As Alert dialog box, specify the following fields to
schedule the alert:
Title: Alert Example (Advanced Conditional)
Alert Type: Scheduled
You can also select Real Time for an advanced conditional search.
Time Range and Schedule: Select any time range and schedule.
Trigger Condition: Custom

147

 Custom condition: search count > 10

4. Define an action that sends an email that includes the results of the
search.
When you configure a Send Email action that includes search results, the
email contains the results of the original base search. It does not include
the results of the custom search.
It might appear that you can get the same results if you specify instead, the
following search for the base search of a basic conditional search:
log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count
by log_level | search count > 10

However, a basic conditional alert based on this search provides different results.
The search results contain only log level values that are greater than 10. The
results from the advanced conditional search details the count for all log levels,
but triggers only when the log levels are greater than 10.

Advanced conditional alert for rolling-window alerts

The behavior for advanced conditional alerts differs slightly for a rolling-window
alert, which runs in real-time. For a rolling-window alert, the alert triggers when
the set condition occurs within the rolling time window of the search.

148

For the previous example, you can design a rolling-window alert with the same
base search and get similar results with the custom condition search. Set the
rolling window to a 10 minutes time span. When the real-time search returns 10
log level entries within the 10 minute time span, the alert triggers.

Create per-result alerts

The per-result alert is the most basic type of alert. It runs in real-time over an
"all-time" time span. The alert triggers whenever the search returns a result.
You can create a search to retrieve events from an index. You can also use
transforming commands to return results based on processing the retrieved
events. A per-result alert triggers in both cases, when the search returns an
event or when a transforming command returns results.

Create a per-result alert

The following procedure shows how to create a per-result alert.
1. From the Search Page, enter the following search:
index=_internal (log_level=ERROR OR log_level=WARN* OR
log_level=FATAL OR log_level=CRITICAL) | stats count as
log_events

2. Select Save As > Alert

3. In the Save As Alert dialog box, enter a Title for the alert.
4. For Alert Type, select Real Time.
A per-result alert is always a real-time alert type.
5. For trigger condition, select Per-Result.
6. Select the actions you want to enable.
For this example, select List in Triggered Alert.

See Set up alert actions for information on other actions.

7. Click Save.

149

Create rolling-window alerts

Use a rolling-window alert to monitor and evaluate events in real time within a
rolling window. The alert triggers only when it meets the trigger condition within a
specified time period.
The rolling-window alert type is in some ways a hybrid of a per-result alert and a
scheduled alert. A rolling-window alert and a per result alert both run in real-time.
But unlike the per result alert, a rolling-window alert does not trigger each time
the search returns a result. A rolling-window alert fires only when it meets
specified trigger conditions within the specified time window. This makes the alert
similar to a scheduled alert.

1. From the Search Page, create the following search. Select Last 24 Hours
for the time range:
index=_internal (log_level=ERROR OR log_level=WARN* OR
log_level=FATAL OR log_level=CRITICAL) | stats count as
log_events

2. Select Save As > Alert

3. In the Save As Alert dialog box, specify the following:
Title: Alert Example (Rolling-Window)
Alert Type: Real Time
Trigger alert when: Number of Results is Greater than 5
in: 30 minutes

150

4. Continue defining actions for the alert.

Set the width of the rolling window

When you create a rolling-window alert, you specify a time span for a real-time
search window. Real-time search windows can be any number of minutes, hours,
or days. The alert monitors events as they pass through the window in real-time.
For example, you can create an alert that triggers when a login for a user fails
four times in a 10 minute period. When the alert runs, various login failure events
pass through this window. The alert triggers only when four login failures for the
same user occur within the span of the 10 minute window.
This example might appear to fail in the following scenario. A user experiences
three login failures in quick succession. After 11 minutes pass, the user has
another login failure. The alert does not trigger because the first three failures
and the fourth failure are in different time windows.

Set up triggering conditions for a rolling-window alert

Trigger conditions apply to two types of rolling-window alerts:
Basic conditional alert
Advanced conditional alert
You set the triggering conditions when you set values for the Trigger condition
field in the Save As Alert dialog, as described in the following subtopics.

151

Basic conditional alert

A basic conditional alert triggers when the number of results from a search,
within a specified time window, meet, exceed, or are less than a specified
numerical value. When you create the alert, you can specify the following
conditions:
Number of results
Number of hosts
Number of sources
You create a basic conditional alert for a rolling-window similarly to how you
create one for a scheduled alert.

Advanced conditional alert

An advanced conditional alert uses a secondary, custom conditional search to
evaluate the results of a scheduled or real-time search. For a rolling-window
alert, the alert triggers when the custom search returns any number of results
within the specified time window. If the alerting conditions are not met, then the
custom conditional search should return zero results.
A secondary conditional search can help reduce the incidence of false positive
alerts.

Scheduled alert example

A scheduled alert searches for events on a regular basis. It triggers an alert
action if results meet the conditions that you specify.
Alert example summary
Use case
Send an email notification if there are more than five errors in a
twenty-four hour period.

152

Alert type
Scheduled
Search
Look for error events in the last twenty-four hours.
Schedule
Run the search every day at the same time. In this case, the search runs
at 10:00 A.M.
Trigger conditions
Trigger the alert action if the search has more than five results.
Alert action
Send an email notification with search result details.
Set up the alert
1. From the Search Page, create the following search. index=_internal "
error " NOT debug source=*splunkd.log* earliest=-24h latest=now

2. Select Save As > Alert.

153

3. Specify the following values for the fields in the Save As Alert dialog box.
Title: Errors in the last 24 hours
Alert type: Scheduled
Time Range: Run every day
Schedule: At 10:00
Trigger condition: Number of Results
Trigger when number of results: is greater than 5.
4. Select the Send Email alert action.
5. Set the following email settings, using tokens in the Subject and Message
fields.
To: email recipient
Priority: Normal
Subject: Too many errors alert: $name$
Message: There were $job.resultCount$ errors reported on
$trigger_date$.
Include: Link to Alert and Link to Results
Accept defaults for all other options.
6. Click Save.

Real-time alert example

A real-time alert searches continuously for results in real time. You can configure realtime alerts to trigger every time there is a result or if results match the trigger conditions
within a particular time window.
Alert example summary
Use case
Monitor for errors as they occur on a Splunk platform instance. Send an
email notification if more than five errors occur within one minute.
Alert type
Real-time
Search
Look continuously for errors on the instance.

154

Trigger conditions
Trigger the alert if there are more than five search results in one minute.
Alert action
Send an email notification.
Set up the alert
1. From the Search Page, create the following search. index=_internal "
error " NOT debug source=*splunkd.log*

2. Select Save As > Alert.

3. Specify the following values for the alert fields.
Title: Errors reported (Real-time)
Alert type: Real-time
Trigger condition: Number of Results
Trigger if number of results: is greater than 5 in 1 minute.
4. Select the Send email alert action.
5. Specify the following email settings, using tokens in the Subject and
Message fields.
To: email recipient
Priority: Normal
Subject: Real-time Alert: $name$
Message: There were $job.resultCount$ errors.
Include: Link to Alert, Link to Results, Trigger Condition, and
Trigger Time.
Accept defaults for all other options.
6. Click Save.

Throttle the real-time alert

Throttle an alert to reduce its triggering frequency and limit alert action behavior. For
example, you can throttle an alert that generates more email notifications than you
need.
Throttle the example real-time alert. The following settings change the alert triggering
behavior so that email notifications only occur once every ten minutes.
1. From the Alerts page in the Search and Reporting app, select the alert. The
alert details page opens.
2. Next to the alert Trigger conditions, select Edit.

155

3. Select the Throttle option. Specify a 10 minute period.

4. Click Save.

Custom trigger condition example

When you create an alert you can use one of the available result or field count
trigger condition options. You can also specify a custom trigger condition. The
custom condition works as a secondary search on the initial results set.
Alert example summary
Use case
Use the Triggered Alerts list to record WARNING error instances.

Alert type
Real-time
Search
Look for all errors in real-time.
Triggering condition
Check the alert search results for errors of type WARNING. Trigger the alert
action if results include any WARNING errors.
Alert action
List the alert in the Triggered Alerts page.

Set up the alert

1. From the Search and Reporting home page, create the following search.
index=_internal source="*splunkd.log" ( log_level=ERROR OR
log_level=WARN* OR
log_level=FATAL OR log_level=CRITICAL)

2. Select Save As > Alert.

3. Specify the following alert field values.
Title: Warning Errors
Alert type: Real-time
Trigger condition: Custom

156

 Custom Condition: search log_level=WARN* in 1 minute

4. Select the List in Triggered Alerts alert action.
5. Click Save.

157

Lab 16 Macros
Define and use search macros
Search macros are chunks of a search that you can reuse in multiple places, including
saved and ad hoc searches. Search macros can be any part of a search, such as an
eval statement or search term, and do not need to be a complete command. You can
also specify whether or not the macro field takes
any arguments.

Create search macros in Splunk Web

In Settings > Advanced Search > Search macros, click "New" to create a new search
macro.

Define the search macro and its arguments

Your search macro can be any chunk of your search string or search command pipeline
that you want to re-use as part of another search.
Destination app is the name of the app you want to restrict your search macro to; by
default, your search macros are restricted to the Search app.
Name is the name of your search macro, such as mymacro.
If your search macro takes an argument, you need to indicate this by appending the
number of arguments to the name; for example, if mymacro required two arguments, it
should be named mymacro(2). You can create multiple search macros that have the
same name but require different numbers of arguments: foo, foo(1), foo(2), etc.

Definition is the string that your search macro expands to when referenced in another
search.
If the search macro requires the user to input arguments, they are tokenized and
indicated by wrapping dollar signs around the arguments; for example, $arg1$. The
arguments values are then specified when the search macro is invoked.

158

If Eval Generated Definition? is checked, then the 'Definition' is expected to be an eval

expression that returns a string that represents the expansion of this macro.
If a macro definition includes a leading pipe character ("|"), you may not use it as
the first term in searches from the UI.
Example: "| metadata type=sources".
The UI does not do the macro expansion and cannot correctly identify the initial pipe to
differentiate it from a regular search term. The UI constructs the search as if the macro
name were a search term, which after expansion would cause the metadata command
to be incorrectly formed and therefore invalid.
Arguments are a comma-delimited string of argument names. Argument names may
only contain the characters: alphanumeric 'a-Z, A-Z, 0-9'; underscore '_'; and dash '-'.
This list should not contain any repeated elements.
If a macro argument includes quotes, you need to escape the quotes
when you call the macro in your search. For example, if you wanted to pass a quoted
string as your macro's argument, you would use:
`my-macro("He said \"hello!\"")`.
Validate your argument values
You can verify that the argument values used to invoke the search macro are
acceptable. How to invoke search macros are discussed in the following section,
"Apply macros to saved and ad hoc searches".
Validation Expression is a string that is an 'eval' expression that evaluates to a
boolean or a string.
If the validation expression is a boolean expression, validation succeeds when it returns
true. If it returns false or is null, validation fails, and the Validation Error Message is
returned.
If the validation expression is not a boolean expression, it is expected to return a string
or NULL. If it returns null, validation is considered a success. Otherwise, the string
returned is rendered as the error string.

Apply macros to saved and ad hoc searches

To include a search macro in your saved or ad hoc searches, use the left quote
(also known as a grave accent) character; on most English-language keyboards,
this character is located on the same key as the tilde (~). You can also reference
a search macro within other search macros using this same syntax.

159

Example - Combine search macros and transactions

Transactions and macro searches are a powerful combination that you can use to
simplify your transaction searches and reports. This example demonstrates how you
can use search macros to build reports based on a defined transaction.
Here, a search macro, named "makesessions", defines a transaction session from
events that share the same clientip value that occurred within 30 minutes of each other:
transaction clientip maxpause=30m

This search takes web traffic events and breaks them into sessions, using the
"makesessions" search macro:
sourcetype=access_* | `makesessions`

This search returns a report of the number of pageviews per session for each
day:
sourcetype=access_* | `makesessions` | timechart span=1d
sum(eventcount) as pageviews count as sessions

If you wanted to build the same report, but with varying span lengths, just save it
as a search macro with an argument for the span length. Let's call this search
macro, "pageviews_per_second(1)":
sourcetype=access_* | `makesessions` | timechart $spanarg$
sum(eventcount) as pageviews count as sessions

Now, you can specify a span length when you run this search from the Search
app or add it to a saved search:
`pageviews_per_second(span=1h)`

160

Lab 17 Lookups
Example of HTTP status lookup
This examples walks through defining a static lookup that adds two informational
fields, status_description and status_type, into your Web access events. This
lets you search for the events you want when you might not know the specific
error code. For example, instead of searching for all the server error codes, you
can use status="Server Error".
Upload the lookup table to Splunk Enterprise
1. Download the http_status.csv file:
http_status.csv
Here's a sampling of the file:
status,status_description,status_type
100,Continue,Informational
101,Switching Protocols,Informational
200,OK,Successful
201,Created,Successful
202,Accepted,Successful
203,Non-Authoritative Information,Successful
...

2. Go back to the Search app, then select Settings > Lookups.

3. In the Lookups page, select Add new for Lookup table files.
4. In the Add new page,
Select search for the destination app.
Browse for the CSV file that you downloaded earlier.
Name the lookup table http_status.
Click Save.
Now, let's go back to the Settings > Lookups view. To do this, click on the
Lookups link in the page's breadcrumb. You can always use this to navigate
back to a previous view.
Define the lookup
1. From Settings > Lookups, select Add new for Lookup definitions.
In the Add new page:
2. Select search for the Destination app.
3. Name your lookup definition http_status.
4. Select File-based under Type.
5. Click Save.
After Splunk Enterprise saves your lookup definition, it takes you to the following
page:
Notice there are some actions you can take on your lookup definition.
Permissions lets you change the accessibility of the lookup table. You can
Disable, Clone, and Move the lookup definition to a different app. Or, you can

161

Delete the definition.

Once you define the lookup, you can use the lookup command to invoke it in a
search or you can configure the lookup to run automatically.
Set the lookup to run automatically
1. Return to the Settings > Lookups view and select Add new for Automatic
lookups.
In the Add new page:
2. Select search for the Destination app.
3. Name the lookup http_status.
4. Select http_status from the Lookup table drop down.
5. Apply the lookup to the sourcetype named access_combined.
6. Lookup input fields are the fields in our events that you want to match with the
lookup table. Here, both are named status (the CSV column name goes on the
left and the field that you want to match goes on the right):
7. Lookup output fields are the fields from the lookup table that you want to add
to your events: status_description and status_type. The CSV column name goes
on the left and the field that you want to match goes on the right.
8. Click Save.

162

Lab 18 Workflows
Example - Google search from field values
Here's an example of the setup for a GET link workflow action that sets off a Google search on
values of the topic field in search results:

163

In this example, we set the Label value to Google $topic$ because we have a field called
topic in our events and we want the value of topic to be included in the label for this workflow
action. For example, if the value for topic in an event is CreatefieldactionsinSplunkWeb the
field action displays as Google CreatefieldactionsinSplunkWeb in the topic field menu.
The Google $topic$ action applies to all events.

164

The Google $topic$ action URI uses the GET method to submit the topic value to Google for
a search.

Example - Provide an external IP lookup

You have configured your Splunk Enterprise app to extract domain names in web services logs
and specify them as a field named domain. You want to be able to search an external WHOIS
database for more information about the domains that appear.
Here's how you would set up the GET workflow action that helps you with this.
In the Workflow actions details page, set Action type to link and set Link method to get.
You then use the Label and URI fields to identify the field involved. Set a Label value of
WHOIS: $domain$. Set a URI value of http://whois.net/whois/$domain$.
After that, you can determine:

whether the link shows up in the field menu, the event menu, or both.
whether the link opens the WHOIS search in the same window or a new one.
restrictions for the events that display the workflow action link. You can target the
workflow action to events that have specific fields, that belong to specific event types, or
some combination of the two.

Set up a POST workflow action

You set up POST workflow actions in a manner similar to that of GET link actions. However,
POST requests are typically defined by a form element in HTML along with some inputs that are
converted into POST arguments. This means that you have to identify POST arguments to send
to the identified URI.
1. Navigate to Settings > Fields > Workflow Actions.
2. Click New to open up a new workflow action form.
3. Define a Label for the action.
The Label field enables you to define the text that is displayed in either the field or event
workflow menu. Labels can be static or include the value of relevant fields.
4. Determine whether the workflow action applies to specific fields or event types in your data.
Use Apply only to the following fields to identify one or more fields. When you identify
fields, the workflow action only appears events that have those fields, either in their event
menu or field menus. If you leave it blank or enter an asterisk the action appears in menus
for all fields.

165

Use Apply only to the following event types to identify one or more event types. If you
identify an event type, the workflow action only appears in the event menus for events
that belong to the event type.
5. For Show action in determine whether you want the action to appear in the Event menu, the
Fields menus, or Both.
6. Set Action type to Link.
7. Under URI provide the URI for a web resource that responds to POST requests.
8. Under Open link in, determine whether the workflow action displays in the current window
or if it opens the link in a new window.
9. Set Link method to Post.
10. Under Post arguments define arguments that should be sent to web resource at the identified
URI.
These arguments are key and value combinations. On both the key and value sides of the
argument, you can use field names enclosed in dollar signs to identify the field value
from your events that should be sent over to the resource. You can define multiple
key/value arguments in one POST workflow action.
Enter the key in the first field, and the value in the second field. Click Add another field
to create an additional POST argument.
11. Click Save to save your workflow action definition.
Splunk Enterprise automatically HTTP-form encodes variables that it passes in POST link
actions via URIs. This means you can include values that have spaces between words or
punctuation characters.

Example - Allow an http error to create an entry in an issue tracking application

You have configured your Splunk Enterprise app to extract HTTP status codes from a web
service log as a field called http_status. Along with the http_status field the events typically
contain either a normal single-line description request, or a multiline python stacktrace
originating from the python process that produced an error.
You want to design a workflow action that only appears for error events where http_status is
in the 500 range. You want the workflow action to send the associated python stacktrace and the
HTTP status code to an external issue management system to generate a new bug report.
However, the issue management system only accepts POST requests to a specific endpoint.
Here's how you might set up the POST workflow action that fits your requirements:

166

Note that the first POST argument sends server error $http_status$ to a title field in the
external issue tracking system. If you select this workflow action for an event with an

167

http_staus

of 500, then it opens an issue with the title server error 500 in the issue tracking

system.
The second POST argument uses the _raw field to include the multiline python stacktrace in the
description field of the new issue.
Finally, note that the workflow action has been set up so that it only applies to events belonging
to the errors_in_500_range event type. This is an event type that is only applied to events
carrying http_error values in the typical HTTP error range of 500 or greater. Events with
HTTP error codes below 500 do not display the submit error report workflow action in their
event or field menus.

168

Lab 19 Tagging
Tagging
Tags are used to label specific values of a ffield. For example, many names of servers
may not be immediately recognized, and using a tag format can help them be more
easily recognizable or distinguishable from each other.
To tag the value of a ffield, use the following steps:
1. Go to Settings | Tags. A window will open, as shown in the
following screenshot:

Adding Tags

2. Under List by tag name, click Add new.

3. Here we want to tag an item as ITEM14 whenever the value of
itemId=EST-14, as shown in the following screenshot:

169

Naming Tags and Specifying Ffield Value Pairs

4. You will now see your tag listed as shown in the following screenshot:

List by Tag Name

170

5. Go back to the event list and click the > sign next to an event. You will see
details of the event open up in a way similar to that presented in the following
screenshot. You can see here that itemid=EST-14 has been tagged as ITEM14.
Now everywhere that EST-14 occurs, it will be tagged as ITEM14.

Note that itemid=EST-14 has been tagged as ITEM14

Tags enable you to search more easily and to convey meaning about the field
values. When you search tag=ITEM14, all the cases where itemid=EST-14
show up. By using tags in this manner, you can facilitate your analysis.

Setting event types

Another way of preparing data to be reported is to set event types, which let you put
events into categories. When setting event types, you can use wildcards, field values,
and Boolean expressions. This capability makes event types more versatile and
powerful than tags, for which you can only use field values. As with tags, you can
choose the categories you like.
When setting event types, be aware of the following:
1. You can't do a sub-search to create an Event type.
2. You can't use pipes in a search that create an Event type.

171

As an example of how to create an Event type, take the following steps using
the buttercupgames ile:

Enter this into the search bar:

sourcetype="access_*" status=200 action=purchase

This creates a search for events where the sourcetype is an accessed

web page, the access was successful (status=200), and it ended in
a purchase:

Search that will be saved as an Event Type

172

3. Click Save As | Event Type in the upper-right corner of the screen and
create a name for the event type. In this case, we have used the name success.
4. In this screenshot, when we enter buttercupgames | stats count by
eventtype, we get a count of each event type. In this case, we have only one
event type, so we get only one count in our table, but we could easily put
other event types in:

Shows Count by Eventtype

173

5. If you want to remove an event type, go to Settings | Event types, and you will
get a screen similar to what is shown in the following screenshot. Just ind the
event type you want to remove and click on Delete:

Event Types (Notice that you can Delete the one you just made.)

174