Finding Order(s) in the Chaos

How Splunk Software is Used to Troubleshoot Transactions

C U S T O M E R p ro f i l e

Splunk customer profiles are a collection of innovative, in-depth use cases that highlight
the value that Splunk customers gain from collecting, analyzing and visualizing the massive
streams of machine data generated by their IT systems and technology infrastructures.

Each “real world” customer profile introduces a unique business challenge and shows how
leveraging machine data and Splunk software in new and interesting ways has helped drive
powerful business and operational outcomes.
 C U S T O M E R p ro f i l e

Executive Summary • Seeing beyond the complexity of the software stack. The
retailer’s multiple, custom-built internal- and external-facing
ERP applications do not use a single unique identifier for each
From the simple storefronts and shopping carts of
transaction to correlate events from one system to another.
the 1990s, e-commerce has grown into complicated Consequently, debugging issues that impeded transactions
supply chains and partner networks. The capturing, was extremely challenging. The support team learned of
problems only when customers called to report unfulfilled
processing and fulfilling of orders has also grown, orders. Splunk Enterprise now provides comprehensive optics
presenting a unique set of challenges. Losing orders across all systems to track transactions and identify issues.
within a complex network of internal and/or partner • Simplify the troubleshooting process. Once an issue was
systems is one of the biggest nightmares for online reported, multiple members of the support and operations
staff were frequently needed to identify and resolve the
retailers. Orders that fall between the cracks not only
problem. This involved tracking, tracing and correlation of
directly impact the revenue stream, they also damage data manually through millions of lines of inconsistently
customer satisfaction, the company’s reputation and formatted log entries, and trying to link unique identifiers
from each individual system and silo. Tickets were passed
long-term relationships with consumers. from group to group while customers awaited resolution.
With a complex stack of dozens of different software Now that Splunk software coheres log events to display both
components handling up to four million online orders per hour, holistic and granular views of the transaction chain, support
one Splunk customer, a large online retailer, found that Splunk staff can address issues rapidly and cost-effectively.
software provided the visibility into its environment that it • Improve the customer experience. Company executives
required to catch issues with orders before they could affect had been concerned about customer satisfaction issues.
customers. Founded in the 1980s, this rapidly expanding retailer Support staff was always in triage mode, fighting fires and
has $25B in annual revenue and more than 90,000 employees in never getting ahead of the problem. Now with greater insight
over 25 countries. into its transaction processing, the retailer has improved the
Upon deploying Splunk Enterprise, the customer could visualize efficiencies of its online business to better meet customer
its online processes in dashboards for end-to-end visibility into expectations.
order transactions across the enterprise. Specifically, the retailer
defines transactions in Splunk software to monitor customer The Nightmare of Online Retailing—Orders
orders as they travel through the IT data integration backbone, Lost in the System
a complex environment which consists of multiple, disparate
applications. The retailer also gained visibility into its systems to To support a business that has grown over the years to millions of
better understand usage and trends and to enable its support transactions per hour, this online retailer built a multi-step, loosely
and operation teams to detect problems before they impact decoupled ordering process and a corresponding IT architecture
customers. As a result, the retailer met its business-critical needs: to underpin it (see Figures 1 & 2).

Business Benefits at a Glance

Splunk Value How Value Is Measured Business Impact

• Decrease in lost orders • $900,000 annually in recovered revenue
End-to-end visibility on by preventing lost orders
customer orders • Faster resolution when
customers inquire about orders • Higher customer satisfaction and loyalty

• Reduced Mean Time to Repair • More efficient order fulfillment

Integrated, yet granular (MTTR)
• Issue analysis and resolution accelerated
views across siloed • Reduced SLA violations by 25%
systems for order processing and
fulfillment • Reduction of lost revenue

• Rapid ROI via operational efficiencies,

High business value, low •N o IT systems require re-
recovered revenue and a renewed focus
IT pain architecting
on high value projects

2
 C U S T O M E R p ro f i l e

Inventory Systems Check Pricing

Customer Enters Order Order Captured Determine the Check Inventory Process Order Send Confirmation Ship Product
(Order Management (Order Management Supply Warehouse (Rules Database)
Capture) System)

Figure 1. Multi-step business process and systems.

East Region West Region

Order Entry EAI EAI Proxy EAI iSeries

XML/JMS iSeries/JMS iSeries/MQ

iSeries/JMS
Propriety

XML to iSeries

Operational Data

MQ

Figure 2. The high-level IT architecture supporting the ordering process.

Whenever orders went missing, troubleshooting issues through build. Starting over to create a more tightly integrated system that
such a complex stack was a laborious task that required extensive could easily provide an end-to-end view of business transactions
IT resources over long periods of time. Since the IT systems simply was not feasible. Instead, Splunk Enterprise allowed the
involved—SOA, EAI-connected and platform-based services— retailer to leave the existing architecture unchanged while attacking
were loosely integrated and the underlying data generated was the problem “bottom up” by aggregating, indexing and analyzing
siloed, tracing a particular customer’s transaction end-to-end was machine data across the entire environment (see Figure 3).
nearly impossible. Multiple applications with common fields but
different identifiers made it extremely difficult to correlate data, For a view of what this data can look like, Figure 4 below is a
which complicated matters further. As a result, issue tracking screenshot from the retailer’s key process orchestration system,
and resolution often took weeks or longer, which undermined the webMethods Service Bus. The top and bottom left are
productivity and customer satisfaction. sample webMethods data and fields. The bottom right shows
how the Splunk REGEX command pulls out special fields such as
ServiceName and OrderNum, which are particularly useful for
Enter Splunk tracking orders.
One reason the retailer sought out Splunk Enterprise as a solution Another key source of data is Tuxedo, a high-performance
was its desire to minimally impact its loosely coupled IT architecture transaction processing system the retailer uses to commit order
and its complex, siloed data systems, which had taken years to data into its ERP order systems. In Figure 5, log messages from

3
 C U S T O M E R p ro f i l e

East Region West Region

Order Entry EAI EAI Proxy EAI iSeries

Log File A Log File B Log File C Log File D MQ

Forwarder A Forwarder B Forwarder C Forwarder D Generic JMS

(10 Lines Code) (10 Lines Code) (10 Lines Code) (10 Lines Code) (200 Lines Java Code)

Data Store

Indexer

Source Type Config Dashboard Source Type Config

(Regular Expressions) (1 Day) (Regular Expressions)

Search Head

West End User East End User

Figure 3. Collecting log data from all systems into Splunk Enterprise.

Tuxedo are in the left column, while the right column shows how This particular set of search commands looks through extracted
the REGEX command extracts important data fields such as the webMethods and Tuxedo logs and informs the system that
application transaction number (appl_tran_no) and transaction orderNum from webMethods and appl_tran_no from Tuxedo
error description (xtrn_error_desc). are the same data fields that uniquely identify each order. Splunk
Enterprise then coalesces these two fields into a single new field
After pulling relevant data from such separate sources as order_id and, lastly, uses order_id to group and trace events in
webMethods and Tuxedo logs, the retailer connects the webMethods and Tuxedo.
information using Splunk’s Search Processing Language (SPL™),
specifically the transaction command. This step groups events
from webMethods and Tuxedo systems together and builds an The Splunk Solution: Complete Order
end-to-end trace for purchase orders. Visualization
> (sourcetype=webMethods_log OR Using Splunk Enterprise to extract and group meaningful data
sourcetype=tux_log) AND action=purchase from disparate systems, the retailer then created a number of
| eval order_id = coalesce(orderNum, dashboards that enable its IT and business users to visualize
appl_tran_no) | transaction order_id orders as they travel across the production environment.

4
 C U S T O M E R p ro f i l e

webMethods Logs, Props and Transforms

Figure 4. Pulling useful data from webMethods logs.

A dashboard was created for IT operations to monitor Orders Found, Time and Revenue Regained
transactions going through the retailer’s integration backbone
system (see Figure 6). As retailers grow, they inevitably reach a stage where there are
multiple in-house and partner systems that integrate to fulfill
This dashboard shows a pie chart of the count of services that customer orders. One by-product of this growth and these
have sent any events in six single value metrics, breaking down multiple systems is order loss due to complex data integration.
the number of transactions versus failures versus SLA violations, Order loss can significantly impact a retailer in terms of reduced
as well as a table of these results. The retailer’s IT operations team revenue and customer satisfaction as well as expensive technical
can then drill down into any particular service name in the chart troubleshooting involving numerous resources and man-hours.
for granular details. From the dashboard above, users can track
a specific transaction via a unique transaction ID across all the Rather than go through the costly, lengthy and unpredictable
disparate systems. process of tightening systems integration or data warehousing,
this particular retailer deployed Splunk Enterprise to trace orders
Figure 7 represents the transaction path dashboard, which allows throughout their life cycle. As with most Splunk customers, the
a user to view details about a particular transaction. A successful retailer saw immediate value from Splunk software. By using the
transaction log shows the hops it went through, as well as the raw solution to track customer orders, the retailer has prevented up
events. If there was a failure, the dashboard also provides a link to 100 lost orders per week, with a corresponding $900,000 in
to the integration page. This was a huge win for the IT developers annual revenue recovery. In addition, the retailer has enhanced
because it enabled them to trace any problem to the error origin
and even the error cause.

The retailer also uses Splunk Enterprise to aggregate transactional

data for real-time optics into the volume of orders going through
the system. Figure 8, for example, is an executive dashboard
featuring a stack chart broken out by orders over a 30-day period
by geography and business unit.

In addition, Splunk Enterprise has helped IT monitor the overall

trends and health of the entire stack. The trends dashboard (see
Figure 9) shows comparisons of services between days of the week,
for entire weeks, and quarterly, as well as a stack chart of all the
services during the previous month. When executives request this
type of information, it can be made available instantly, enabling
them to holistically view service levels and anticipate future needs.

Figure 5. Pulling useful data from Tuxedo logs.

5
 C U S T O M E R p ro f i l e

Figure 6. Integration dashboard monitors transactions and SLAs.

Figure 8. Real-time order dashboard.

esbd1as12.host.com

esbd1as12.host.com

esbd1as12.host.com
esbd1as12.host.com

esbd1as12.host.com
esbd1as12.host.com

Figure 9. Transactions trends.

Figure 7. Detailed order transaction tracing.

operational efficiency, saving numerous man-hours for the • Correlations drive analytics. Because Splunk software
IT Support and Development Operations teams involved in correlates different types of data, the customer links error
troubleshooting. Since deploying Splunk software, these teams messages in one system’s logs to useful evidence in other
have found and addressed issues up to 25% faster, freeing up time logs to resolve issues.
for higher value projects.
• Flexible analytics powered by a read-time schema.
Splunk software not only allows this retailer to extract critical Because Splunk software keeps the data unstructured, the
business insights out of its application and systems data, the customer views the data through the lens that matters at
solution also enables the customer to improve overall operational the time it queries the data, not the lens that mattered to
efficiency and increase customer satisfaction. the system at the time it generated the event.
• Significantly reduced manual intervention and labor cost.
With Splunk, They Found Order(s) in Chaos Because Splunk Enterprise provides real-time visibility of end-
to-end data flow through the customer’s business process,
In this use case, we explored how Splunk software can enable the retailer tracks orders across disparate systems and
operational intelligence for a classic transaction/workflow reduces MTTR.
challenge. Despite a daunting stack of dozens of different
software components handling four million orders per hour, an
online retailer has the optics to figure out when something has One Splunk. Many Uses.
gone wrong and how to fix it. This use case demonstrated: While the business problems discussed in this case were specific
• Elimination of data silos. Because Splunk software indexes to this customer and their industry, and the solution made creative
all kinds of machine data, the customer’s reams of siloed use of Splunk Enterprise’s features to solve these particular
data no longer hinder its ability to quickly find lost orders problems, the underlying theories are applicable to a variety of
and rectify underlying issues. single-path workflows.

6
 C U S T O M E R p ro f i l e

With the right sets of data, Splunk can trace the time it takes for a
specific file to go through a series of compression processes, which
steps in an online purchase stymie the most customers, or which
coupon-downloaders actually go into the store to buy the product.

While the possibilities are endless, the process is simple.

Next Steps
To learn more about Splunk customer success, customer
snapshots, ROI stories, customer profiles and more, please
visit: http://www.splunk.com/view/customer-case-studies/SP-
CAAABB2

Splunk software is also available as a free download. Download

Splunk Enterprise and get started today: http://www.splunk.
com/download

If you would like to speak to a salesperson, please use our online

form: http://www.splunk.com/index.php/ask_expert/2468/3117

Free Download
Download Splunk for free. You’ll get a Splunk Enterprise 6
license for 60 days and you can index up to 500 megabytes
of data per day. After 60 days, or anytime before then,
you can convert to a perpetual Free license or purchase an
Enterprise license by contacting [email protected].

250 Brannan St., San Francisco, CA 94107 [email protected] | [email protected] 866-438-7758 | 415-848-8400 www.apps.splunk.com

www.splunk.com
© 2014 Splunk Inc. All rights reserved. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Hunk, Splunk Cloud, Splunk Storm and SPL are trademarks and
registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners.. Item # CS-X-Splunk-Retail-104