Architecture

 
Overview  
©  Copyright  2011  –  Company  Confiden6al   Visibility  when  you  need  IT  most.  
 What  does  Splunk  collect?  

Machine  Generated  Data,  not  Human  Generated  Data  

 
•  Machine  Data  contains  categorical  record  of  all  ac6vity  and  
behavior  –  customer  behavior,  user  transac6ons,  machine  
behavior  
 

©  Copyright  2011  –  Company  Confiden6al   2   Visibility  when  you  need  IT  most.  
 What  does  Splunk  do?  
Scripts   Database  
Server  Metrics   Router   Host  Config   Logs  
Card  Key  
Vulnerability  Data  
Virtual   Email  
RAS   Logs  
Custom   Host  IDS   VPN   Patch     Logs  
ApplicaDons   Server  Logs   Mgmt  
Physical  
Windows   DNS  Logs   Security   ApplicaDon  Logs  
registries  

©  Copyright  2011  –  Company  Confiden6al   3   Visibility  when  you  need  IT  most.  
 What  does  Splunk  Provide?  

A  common  interface  for  all  IT  Data  

search   alert   report   share  

That  provides  Opera/onal  Intelligence  

©  Copyright  2011  –  Company  Confiden6al   4   Visibility  when  you  need  IT  most.  
 The  Big  Data  Problem  

What  is  Big  Data?   “

“ …When  the  size  of  the  data  itself  becomes  part  of  the  problem  
Mike  Loukides  
   O’Reilly  Radar  
 Machine  data  has  its  own  “OSI  model”  

Complexity  has  increased  

logarithmically  with  the  addiDon  
of  each  of  these  layers  to  the  IT  
architecture.  

©  Copyright  2011  –  Company  Confiden6al   6   Visibility  when  you  need  IT  most.  
 Point  Solu6ons  are  Common  –  lots  of  consoles  

groan  
Problem  is  exasperated  since  
these  stovepipe  tools  do  not  
provide  the  ability  to  correlate  
events  between  them.    

©  Copyright  2011  –  Company  Confiden6al   7   Visibility  when  you  need  IT  most.  
 Point  Solu6ons  are  Common  –  lots  of  consoles  

Point  solu6on  tools:  

  No  end  to  end  situa6onal  

Awareness  across  technologies  
 
  High  cost  of  ownership  
 
  Slower  resolu6on  /  Incident  
groan   Response  

Problem  is  exasperated  since  

these  stovepipe  tools  do  not  
provide  the  ability  to  correlate  
events  between  them.    

©  Copyright  2011  –  Company  Confiden6al   8   Visibility  when  you  need  IT  most.  
 What  is  needed  is  a  single  method  to  access  IT  informa6on.  .  .  

Phew  

©  Copyright  2011  –  Company  Confiden6al   9   Visibility  when  you  need  IT  most.  
 Across  the  en6re  IT  Architecture  

Ah-­‐Ha!  

©  Copyright  2011  –  Company  Confiden6al   10   Visibility  when  you  need  IT  most.  
 Across  the  en6re  IT  Architecture  

Correla6on  of  disparate  data  in  a  

single  tool  across  all  technologies:  
 
  Quicker  root  cause  analysis  
 
  Increases  situa6onal  awareness  

  Lowers  business  risk  

Ah-­‐Ha!  

©  Copyright  2011  –  Company  Confiden6al   11   Visibility  when  you  need  IT  most.  
 See  all  IT  and  make  IT  useful  
Finding  your  faults,  just  like  Mom  
Because  ninjas  are  too  busy  
All  batbelt  no  6ghts  
Needle.  Haystack.  Found  
It’s  like  grep  on  steroids  

Only  cavemen  use  event  viewer  

Take  the  SH  out  of  IT  

Log  management  at  the  speed  of  thought  

©  Copyright  2011  –  Company  Confiden6al   12   Visibility  when  you  need  IT  most.  
 What  can  you  do?  

Over  2,900  enterprise  customers  use  Splunk  to  

gain  beDer  insight  and  visibility  from  their  
machine  data.  Why?  
 Index  all  data  without  parsers  or  connectors  

Customer     Outside  the  

Facing  Data   Datacenter  
•  Click-­‐stream  data   •  Manufacturing,  
•  Shopping  cart  data   logis6cs…  
•  Online  transac6on   •  CDRs  &  IPDRs  
data   •  Power  consump6on  
Logfiles   Configs   Messages   Traps     Metrics   Scripts   Changes   Tickets   •  RFID  data  
 Alerts   •  GPS  data  

VirtualizaDon    
Windows   Linux/Unix   &  Cloud   ApplicaDons   Databases   Networking  
•  Registry   •  Configura6ons   •  Hypervisor   •  Web  logs   •  Configura6ons   •  Configura6ons  
•  Event  logs   •  syslog   •  Guest  OS,  Apps   •  Log4J,  JMS,  JMX   •  Audit/query  logs   •  syslog  
•  File  system   •  File  system   •  Cloud   •  .NET  events   •  Tables   •  SNMP  
•  sysinternals   •  ps,  iostat,  top   •  Code  and  scripts   •  Schemas   •  nedlow  

©  Copyright  2011  –  Company  Confiden6al   Visibility  when  you  need  IT  most.  
 Search  using  a  powerful  search  language  

sourcetype="access_combined"  |  transac6on  JSESSIONID  |  where  mvcount(clien6p)  >  1  

©  Copyright  2011  –  Company  Confiden6al   Visibility  when  you  need  IT  most.  
 Automa6c  Chronology  

20:23:49  –  20:28:16      Firewall  events  

20:28:16      Events  detected  moving  across  network  from  
                                     SQL  server  to  other  nodes  

20:24:10      Reboot  in  Sever  Logs  /  Odd  Audit  Events  

20:23:54      Web  Service  Unavailable  

20:23:50      CPU  Spike  on  OS  of  SQL  Server    

20:23:49      Medium  priority  IDS  events  to  SQL  Server  

©  Copyright  2011  –  Company  Confiden6al   16   Visibility  when  you  need  IT  most.  
 Alert  in  Real  Time  

Define  the  Alert   Alert  Op6ons   Who  gets  Alerted  

©  Copyright  2011  –  Company  Confiden6al   Visibility  when  you  need  IT  most.  
 Enrich  data  from  external  sources  

LDAP,  AD   Watch    

Lists  

CMDB   CRM/ERP  

Beter  Understanding  
 

©  Copyright  2011  –  Company  Confiden6al   Visibility  when  you  need  IT  most.  
 Report  to  any  level  
Support  Mul6ple  Use  Cases   IT,  Line  of  Business  or  
Management  

VPs  of  Infrastructure  

Website  Managers   Mash  up  Web  Apps  

©  Copyright  2011  –  Company  Confiden6al   19   Visibility  when  you  need  IT  most.  
 Delivering  Opera6onal  Intelligence  

Three  Primary  Capabili6es  

Search/Navigate  
  Real-­‐Dme  Visibility   Historical  AnalyDcs  
•  Data  drilldown   •  Live  dashboards   •  Baseline  and  thresholds  
•  “Needle  in  a  haystack”   •  Event  correla6on   •  Trending  
•  Root  cause  analysis/ •  Monitoring  and  aler6ng   •  Opera6onal  insights  
 troubleshoo6ng   •  Performance  issues   •  Historical  paterns  
•  Incident  inves6ga6ons   •  Transac6on  levels   •  Compliance  reports  
•  SLA  tracking  

©  Copyright  2011  –  Company  Confiden6al   20   Visibility  when  you  need  IT  most.  
 Why  Splunk  scales  

“
“
Splunk  has  been  tackling  [big  data]  with  a  unique  solu/on  that  is  genera/ng  a  
significant  amount  of  commercial  success  
David  Menninger    
VP  &  Research  Director  

©  Copyright  2011  –  Company  Confiden6al   21   Visibility  when  you  need  IT  most.  
 Databases  are  not  suited  for  unstructured  data  

Rela6onal  Databases   Mul6dimensional  Databases   Machine  Data  Engine  

" Financial  records,   " Mul6dimensional  data  for   " Time  series  
manufacturing  and   business  management  and   unstructured  data,  with  
logis6cal  informa6on,   sta6s6cs   no  predefined  schema  
personnel  data   " Math  computa6on  strength  –   " Generated  by  all  IT  
" Data  highly  structured  —   dense  data   systems,  non-­‐standard  
database  highly   " Pivots  data  for  flexible  financial   data,  unpredictable  
structured   analysis   formats  
" Inflexible  schema,     " Monthly  repor6ng,  not  for   " Massive  volume;  fast  
long  deployment  cycle   real-­‐6me  events   naviga6on  and  
correla6on  paramount  
©  Copyright  2011  –  Company  Confiden6al   22   Visibility  when  you  need  IT  most.  
 Distributed  Search  using  Map  Reduce  
MapReduce  is  a  soyware  
framework  introduced  by  Google  
in  2004  to  support  distributed  
compu6ng  on  large  data  sets  on  
clusters  of  computers.  -­‐  Wikipedia  

1   1   1  
⁄  n   ⁄  n   ⁄  n  
A  ‘search’  (ques6on  to  be   Each  Indexer  processes  a  
answered)  is  distributed   subset  of  the  en6re  
amongst  mul6ple  cores.   dataset  and  produces  
  Appliance   part  of  the  overall  answer  
back  to  the  search  head  
for  “reduce”  
 
Data  is  load  balanced  into    
commodity  computers  
(indexers)  where  it  is  
‘mapped’.  
 

©  Copyright  2011  –  Company  Confiden6al   23   Visibility  when  you  need  IT  most.  
 Ques6ons?  Talk  to  a  Splunk  representa6ve  
Library  of  Congress  
 
Anna  Tant  
Civilian  Account  Execu/ve  
Federal  
[email protected]  

Free  Download  
Limited  to  500mb/day  
No  aler6ng  
 
www.splunk.com  

©  Copyright  2011  –  Company  Confiden6al   24   Visibility  when  you  need  IT  most.