DevOps  

Testing
Module  1  – Introduction  to  Continuous  Testing

IBM  Agile  Academy  – DevOps

 Becoming  Agile Agile  P ractices DevOps  Courses Testing  Modules
Leadership Development Intro  to  Continuous   Testing
Lesson   Focus Collaboration Testing Test  Coverage  Strategy

Delivery  /  Engineering  /  DevOps Deployment Functional  Test  Automation

Infrastructure Non-­Functional  Test  Automation

Automation  Strategy

Acceptance  Testing  &  BDD

Test  Environment  Management

Embedding  Testing  into  Continuous  Integration  

Incremental  Integration  &  Test  Virtualization

Why  is  this  topic  important? Test  Metrics

§ Continuous  Testing  is  a  crucial  capability  for  achieving  Continuous  Delivery

§ Numerous  strategies,  techniques,  and  tools  are  available  to  help  you  implement  Continuous  Testing

Out  of  this  course  you  will:

• Explore  “shift-­left”  thinking  for  testing

• Understand  test  categories,  levels,  test  quadrants,  and  other  testing  frameworks
• Be  able  to  create  and  implement  a  test  strategy  leveraging  automation,  BDD,  test  virtualization,  and  more

2
Imagine…tests are executed automatically at the push of a
button… …whenever you are ready
Create  test  cases  at  the  beginning  of  the  development  process  as  an  integral  part  of  user  story  definition.  
They  are  updated  throughout  the  cycle…and  executed automatically and frequently…
…to assure production readiness at all times.

Test Early Test Often

…in  addition  to  your  suite  of  manual  test  cases,  “Static Analysis” technologies  can  be  used  to  
enhance  test  coverage  and  automatically  verify  compliance,  security,  and  technical  debt.
Testing  Categories

Functional Tests Non-Functional Tests

Performance
Usability  
Compatibility
Security
Compliance
Resiliency

Non-­functional  test  cases  target  performance,  

Functional  test  cases  target  business  goals.   resource  utilization,  usability,  compatibility  etc.  

4
Levels  of  Functional  Testing  (before  release)

Acceptance Client  UAT  tester  verifies  system  function  is  satisfied

System Dynamic and holistic testing for all system components

Integration Testing  full  range  of  functionality  with  other  systems

Component Similar to Unit level testing but using stubs and drivers

Internal  logic  &  design,  condition  &  exception  coverage,  error  

Unit handling,  and  test  path

5
Testing  Quadrants

Integration Acceptance
System Usability
Combinatorial   /  Orthogonal Exploratory

Quadrant 2 Quadrant 3

Quadrant 1 Quadrant 4

Performance
Unit   Security
Component Compatibility
Static  Analysis Compliance
Resiliency

6
Course  Outline

7
 Thanks  for  watching!

8
DevOps  Testing
Video  Education  Series
Module  2  – Test  Coverage  Strategy  

IBM  Agile  Academy  -­ DevOps

 Becoming  Agile Agile  P ractices DevOps  Courses Testing  Modules
Leadership Development Intro  to  Continuous  Testing

Lesson   Focus Collaboration Testing Testing   Coverage  Strategy  by  Quadrant

Delivery  /  Engineering  /  DevOps Deployment Functional  Test  Automation

Infrastructure Non-­Functional  Test  Automation

Automation  Strategy

Acceptance  Testing  &  BDD

Test  Environment  Management

Embedding  Testing  into  Continuous  Integration  

Incremental  Integration  &  Test  Virtualization

Test  Metrics

Why  is  this  topic  important?

• Continuous testing re-positions the question from “are you done testing?” to “is the level of risk understood and accepted?”
• A test coverage strategy is essential to test the minimal amount, eliminate waste and manage risk
Out of this module, you will be able to
• Know what you need to consider when developing a test coverage strategy
• Leverage a template to build a testing strategy for your project
• Use resources to get started and to learn more

2
Seven  Software  Testing  Principles
1. Testing  shows  presence  of  defects
2. Exhaustive  testing  is  impossible  
3. Early  testing
4. Defect  clustering
5. Pesticide  Paradox
6. Testing  is  context  dependent
7. Absence-­of-­errors  fallacy

http://www.guru99.com/software-­testing-­seven-­principles.html

3
Testing  Pillars  

DevOps  Testing

Embrace  Testing  Early   Utilize  proven   Apply  appropriate  

and  Ensure   technologies   and   rigor
Production   Readiness   patterns
at  all  Times
Automation  strategy  b ased   Test  e nvironments  a re  
Leverage  TDD  a nd  B DD under  change  control
on  testing  p yramid  
Deliver  real  b usiness  value   Planning  for  complete  test  
Utilize  Test  V irtualization,  
every  iteration  to  p roduction coverage  (Functional  a nd  
as  required
Design  A pplication  to  b e   Non-­Functional)
Test  S uites  e xecuted  a s  
testable
part  o f  Continuous  
Integration

4
Test  Coverage  
§ Definition  – measure  of  how  much  of  executable  code  was  tested
Myth:  Test  coverage  =  Quality  Target
Reality:  Test  coverage  helps  find  untested  
code  

Myth:  A  test  suite  that  passes  without  any  

failures  is  indicative  of  high  quality  code
Reality:  Cannot  guarantee  that  all  of  the  
code  is  tested  by  test  suite  

Myth:  A  good  measure  for  test  suite  quality  is  

code  coverage  achieved  by  tests
Reality:  Even  with  100%  code  coverage  and  
all  tests  passing,  there  can  be  undiscovered  
bugs

5
Test  Coverage:  Black  Box  and  White  Box  Testing  
§ Black  Box  Testing  – Testing  method  where  the  internal  structure  /  design  is  NOT  known  to  the  tester  
§ White  Box  Testing  – Testing  method  where  the  internal  structure  /  design  is  known  to  the  tester  

Black Box White Box

Applicable to  higher  levels  of  testing   Applicable to  lower  levels  of  testing  
(e.g.,  acceptance,  integration  &  system)   (e.g.,    mainly  unit,  component,  some  integration  &  system)

Programming  knowledge  not  required Programming knowledge  required

User stories  /  specifications  used  as  basis  for  test  cases   Detail  design  /  code used  as  basis  for  test  cases  (inputs,  outputs)

• All-pairs Testing • Statement

• Orthogonal Array / Combinatorial Testing • Branch (Decision)
• Path
• Full Regression

6
Testing  Quadrants

Integration Acceptance
System Usability
Combinatorial   /  Orthogonal Exploratory

Quadrant 2 Quadrant 3

Quadrant 1 Quadrant 4

Performance
Unit   Security
Component Compatibility
Static  Analysis Compliance
Resiliency

7
Test  Coverage  Strategy  using  Test  Quadrant

Current Where we need to be

Quadrant 2 Quadrant 3
Quadrant 2 Quadrant 3

Quadrant 1 Quadrant 4
Quadrant 1 Quadrant 4

8
How  do  we  know  when  to  start  testing?
§ Test  begins  w hen  the  project  begins
For  example:  TDD,  BDD  

How  do  we  know  when  to  stop  testing?  

§ Don’t  stop  testing  -­ Assess  production  readiness  for  iteration  

§ Risk  assessment  for  any  deviations   from  plan  /  standard  process

§ Thoroughness   measures  – code/risk  coverage

§ Cost  &  iteration   boundary
§ Reached  an  explicit  level   of  testing
9
Strategy  Template  – Test  Coverage  

Test  Level   Recommended  %   Target  %  for  Project  

Test  Cases  by  Level
Unit 60-75%
Component 10-25%
Integration 5-10%
System 1-5%
Acceptance 0-2%

10
References
• More  Agile  Testing  by  Janet  Gregory  &  Lisa  Crispin  
• Seven  principles  of  testing:  http://www.guru99.com/software-­testing-­seven-­principles.html
• Black  Box  and  White  Box  Testing
• http://softwaretestingfundamentals.com/black-­box-­testing
• http://softwaretestingfundamentals.com/white-­box-­testing
• http://tutorialspoint.com/software_testing_dictionary/black_box_testing.htm
• http://tutorialspoint.com/software_testing_dictionary/white_box_testing.htm
• Orthogonal  Array  Testing
• https://en.wikipedia.org/wiki/Orthogonal_array_testing
• http://www.softwaretestinghelp.com/combinational-­test-­technique/
• www.51testing.com/ddimg/uploadsoft/20090113/OATSEN.pdf

11
DevOps  Testing
Video  Education  Series
Module  3  – Functional  Test  Automation

IBM  Agile  Academy  -­ DevOps

 Becoming  Agile Agile  P ractices DevOps  Courses Testing  Modules
Leadership Development Intro  to  Continuous  Testing

Lesson   Focus Collaboration Testing Test  Coverage  Strategy

Delivery  /  Engineering  /  DevOps Deployment Functional   Test  Automation

Infrastructure Non-­Functional  Test  Automation

Automation  Strategy

Acceptance  Testing  &  BDD

Test  Environment  Management

Embedding  Testing  into  Continuous  Integration  

Incremental  Integration  &  Test  Virtualization

Why  is  this  topic  important? Test  Metrics

• Automated testing is a core capability required for continuous delivery

• Often teams find it difficult to adopt automated testing
Out of this module, you will be able to:
• Learn the concept of a Testing Framework for Automated Testing
• Concentrate on different granularities of verification found within these Functional Tests:
• Unit tests, Component tests, Integration tests, System tests, and Acceptance tests
• Explore various techniques and tools which leverage these levels of test and enable highly effective automation

2
Automated  Testing  Framework
Types of Test Frameworks
Main goal: Facilitate  automated  testing

§ The  testing  framework:

– Defines  the  format  for  the  test  cases
– Creates  a  mechanism  for  running  tests  
– Performs  execution  of  tests
– Reports  test  results

§ Also,  a  testing  framework  should:

– Be  easy  to  maintain  and  expand
– Incorporate  assumptions,  concepts,  practices  to  
help  drive  adoption  of  the  testing

3
Modular  Testing  Framework
§ Independent  scripts  aligned  to  module  structure  of  application  being  tested

§ Modules  used  hierarchically  to  build  l arger  test  cases

Advantage:  Quick  startup

Enables  changes  at  lowest  levels  as  not  to  impact  other  test  cases
Disadvantage:  Data  is  embedded  in  the  test  script,  maintenance  difficult

4
Data  Driven  Testing  Framework
§ Test  input  and  expected  results  are  stored  in  a  separate  file  usually  in  tabular  format
§ A  single  script  can  execute  with  multiple  sets  of  data
§ Driver  script  navigates  through  program,  reads  data  input  and  logs  test  status

Advantage:  Reduces  the  number  of  test  scripts  required  (over  Modular)
Disadvantage:  T ight  coupling  between  scripts  and  data  may  exist.

5
Keyword  Driven  Testing  Framework
§ Utilizes  data  tables  and  self  explanatory  keywords  that  describe  actions

§ Test  data  stored  separately  just  like  the  Keywords/Actions  (Directives)

§ Keyword  Driven  Testing  separates  test  creation  process  into  two  distinct  stages:  
1. Design  &  development  stage
2. Execution  stage

Advantage:  Both  data  and  keywords  can  be  reused  across  scripts  providing  flexibility
Disadvantage:  Increased  flexibility  can  drive  complexity

6
Hybrid  Testing  Framework
§ Combination  of  modular,  data  driven  and  keyword  driven  frameworks.

§ Data  driven  scripts  can  access  information  provided  by  keyword  driven  approach

Advantage:  Incorporates  all testing  framework  approaches

Disadvantage:  Most  complex  approach
7
Open  Source  Testing  Frameworks
STAF Omelet

A  multi-­platform,  multi-­language  framework  

designed  around  the  idea  of  reusable   Simple, yet powerful automation library for Selenium
components,  called  services  (such  as  process   WebDriver which uses TestNG as the test runner for
invocation,  resource  management,  logging,  and   UI Automation
monitoring).  
http://staf.sourceforge.net/ http://springer-opensource.github.io/omelet/

Other Frameworks based on Selenium

Create  an  automated  testing  framework  with  Selenium  

http://www.pluralsight.com/courses/automated-­testing-­framework-­selenium

8
Levels  of  Functional  Testing

Acceptance Client  UAT  tester  verifies  system  function  is  satisfied

System Dynamic and holistic testing for all system components

Integration Testing  full  range  of  functionality  with  other  systems

Component Similar to Unit level testing but using Stubs and Drivers

Internal  logic  &  design,  condition  &  exception  coverage,  error  

Unit handling,  and  test  path

9
Testing  Tools:  Unit  &  Component
Tools Website
http://abbot.sourceforge.net/doc/api/overview-summary.html

http://junit.org/
Unit  Tests
http://www.clarkware.com/software/JUnitPerf.html

Component  
http://mockito.org/
Tests
Test drivers & stubs

http://testng.org/doc/index.html

Unit  T est  Example Component  T est  Example

JUnit  Testing  in  Eclipse   Test Drivers and Stubs
https://www.youtube.com/watch?v=QEyxgtCEWMw&list=PL0951947FC3CB5BB3 https://www.youtube.com/watch?v=NXUNLJ1O_AU

10
Testing  Tools:  Integration  

Tools Website

http://www-03.ibm.com/software/products/en/rtw

http://www.soapui.org/

Integration  testing  –Examples

IBM  Rational  Integration  T ester:  Creating  a  test  from  WSDL  
https://www.youtube.com/watch?v=3j6uK6TkD34
Rational  Integration  T ester:  HTTP  Virtualization  
https://www.youtube.com/watch?v=jN75fl3HnxM

11
System  &  Acceptance  Testing  Tools
Tools Website

http://docs.seleniumhq.org/

System  Tests
http://appium.io/

Acceptance  Tests
http://cukes.info/

System  Test  Examples Acceptance  Test  Examples

31  Best  Selenium  tutorials Strategies for Effective Acceptance testing
http://blog.xebialabs.com/2014/10/23/strategies-effective-acceptance-testing-part/
http://www.softwaretestinghelp.com/selenium-­tutorial-­1/
Cucumber Tutorial
Appium  Tutorial http://www.guru99.com/cucumber-tutorials.html

https://docs.saucelabs.com/tutorials/appium/ Dave Farley on Agile, DevOps and Acceptance testing

http://www.infoq.com/interviews/dave-farley-devops

12
References
Testing  Frameworks:  http://www.cs.colorado.edu/~kena/classes/5828/s12/presentations/testing-­frameworks-­by-­
gayat.html

Acceptance  Testing
§http://blog.xebialabs.com/2014/10/23/strategies-­effective-­acceptance-­testing-­part/

13
DevOps  Testing
Video  Education  Series
Module  4  – Non-­Functional   Test  Automation

IBM  Agile  Academy  -­ DevOps

 Becoming  Agile Agile  P ractices DevOps  Courses Testing  Modules
Leadership Development Intro  to  Continuous  Testing

Lesson   Focus Collaboration Testing Test  Coverage  Strategy

Functional  Test  Automation

Delivery  /  Engineering  /  DevOps Deployment

Infrastructure Non-­Functional   Test  Automatio

n
Test  Automation  Strategy

Acceptance  Testing  &  BDD

Test  Environment  Management

Embedding  Testing  into  Continuous  Integration  

Incremental  Integration  &  Test  Virtualization

Why  is  this  topic  important? Test  Metrics

• Automated testing is a core capability of Continuous Delivery

• Supports “Fail Early, Fail Often”
Out of this module, you will be able to:
• Understand the different types of non-functional tests
« Performance, Resiliency, Compatibility, Usability, Security/Penetration, Compliance
• Learn techniques and tools applicable to non-functional testing
• Leverage useful resources to help you get started and continue learning

2
 Performance Testing

3
Performance  Test
DevOps  Techniques  
Whole  team  Ownership  of:  
– Setting  performance  test  priorities
– Developing  of  user  stories  to  cover  performance  test
– Creating  performance  test  cases  
– Sharing  performance  findings
– Identifying  &  resolving  performance  problems  

Performance  acceptance  criteria   defined  by  stakeholders  and  covered  by  test  cases

TDD  practices are  applied

– Performance  testing  should  be  done  at  the  earliest  test  level  (including  unit  test)  
– Any  performance  problem  found  is  considered  a  test  case  not  yet  written  – so,  write  it!

4
Performance  Test  Tools
Apache  JMeter
Rational  Performance  Tester
Server  load  simulation
End-­to-­end  system  performance
http://jmeter.apache.org/
https://www.youtube.com/watch?v=cv7KqxaLZd8 http://www-­03.ibm.com/software/products/en/performance
https://www.youtube.com/watch?v=ZtISOlCQzIw
https://www.youtube.com/watch?v=jhaYEMpKtM4
https://www.youtube.com/watch?v=aenKpQDjjYQ
https://www.youtube.com/watch?v=n_rVR9o-­GrY

LoadRunner
End-­to-­end  system  performance  
http://www8.hp.com/us/en/software-­solutions/loadrunner-­load-­testing/index.html
https://www.youtube.com/watch?v=8L3xagVuTMQ
https://www.youtube.com/watch?v=AicAqUC0K6Q&list=PLFB7DB42D42A7C31C
https://www.youtube.com/watch?v=-­kDf9dZzE94&index=2&list=PLFB7DB42D42A7C31C

5
Resiliency  Test

Ensures  system  recovery  without  loss  of  data  or  functionality

DevOps  Techniques
– Force  failure  of  application  components  to  simulate  real  world  errors  
§ (e.g.,  Chaos  Monkey)
– Ensure  ability  to  remain  operational  in  the  event  of  a  failure
§ Consider  resiliency  options  (e.g.,  static  content  page,  auto-­rollback)
§ Instantiate  new  middleware  components  seamlessly
– Apply  resiliency  testing  to  include  monitoring  effectiveness  and  auto-­provisioning  for  recovery

Reference:  Design  for  Cloud  and  Feature  Decoupling  modules  in  the  DevOps  Development  c ourse  

6
Resiliency  Test  Tools

Chaos  Monkey
Randomly  disables  parts  of  the  system  to  simulate  production  failures
–https://github.com/Netflix/SimianArmy/wiki/Quick-­Start-­Guide
–https://github.com/Netflix/SimianArmy/wiki/The-­Chaos-­Monkey-­Army

Hystrix
Latency  and  fault  tolerance  library  designed  to  isolate  points  of  access  to  
remote  systems,  services  and  3rd  party  libraries
– https://github.com/Netflix/Hystrix
– https://www.youtube.com/watch?v=RzlluokGi1w

7
Compatibility  Test
Ensure  user  interfaces  are  compatible  with  commonly  used  browsers  and  devices

DevOps  Techniques
§Leverage  automation  for  cross  platform  &  device  testing

§Leverage  methodologies  to  minimize  what  is  tested  

(i.e.,  combinatorial,  sampling,  UI  sub-­components)  

§Consider  crowdsourcing  compatibility  test  (e.g.,  uTest)

http://www.utest.com/getting-­started

8
Compatibility  Tools
Browsershots  
Test  rendering  of  the  same  URL  across  operating  systems  and  browsers
http://browsershots.org/
https://www.youtube.com/watch?v=HQdFXwxOivI

BrowserStack
Test  rendering  of  the  same  URL  across  browsers  and  mobile  devices
– iOS,  Android  and  Opera  Mobile  and  Tablet  Emulators  
https://www.browserstack.com/
https://www.youtube.com/watch?v=yg_5hXSHbVw

Browsera  
Locates  cross-­browser  layout  problems  automatically  by  comparing  each  browser's  output
– Tests  all  pages  on  your  site  
http://www.browsera.com/

9
Compatibility  Test
MobiReady
§Evaluate  a  website's  mobile-­readiness  using  industry  practices  and  standards
http://ready.mobi/
https://www.youtube.com/watch?v=u5VIu-­AELS4

SauceLabs Telerik
§ Provides  infrastructure  to  run  tests  in  the   § Cross-­Platform  Mobile  Development  for  iOS,  
cloud  on  more  than  260  different  browser   Android  and  Windows  Phone  
platforms  and  devices
§ Delivers  robust  mobile  analytics  and  mobile  app  
https://saucelabs.com/home testing  

http://www.telerik.com
https://www.youtube.com/watch?v=OBKU_1OP87U
https://www.youtube.com/watch?v=EpW6QfmBZng
https://www.youtube.com/watch?v=Uz3VDWdFNh0

10
Usability  Test
Observe  user  interactions  with  a  product  under  controlled  conditions  to  create  the  best  user  experience

DevOps  Techniques
– Leverage  Design  T hinking  methods:  
§ Understand  (empathy  mapping)
§ Explore  (ideation)    
§ Prototype  (low-­fidelity  “mock  ups”)
§ Evaluate
– Iteration  managers  align  the  whole  team  around  user  value  to  include  sponsor  users  
§ Create  user  stories  and  determine  acceptance  criteria
– Conduct  usability  testing  earlier  in  the  lifecycle  to  generate  feedback  sooner
– Look  for  repeatable patterns  to  automate  and  maximize  automation  where  possible

11
Usability  Tools
CAMTASIA OVO  LOGGER

ScreenShare Tool
SameTime,
GoToMeeting,WebEx Extensible  observational  logging  platform  that  is  intended  for  the  
rigors  of  user-­centered  design  activities  in  an  enterprise
Facilitates  moderated  remote  tests
http://www.ovostudios.com/ovologger.asp
https://www.techsmith.com/camtasia.html

USER  TESTING

An  online  service  network  of  pre-­screened  testers  meeting  several  specified  demographics,  or  
practitioners  can  invite  their  own  testers  to  use  the  tool,  which  includes  video  editing  tools.

http://www.usertesting.com/

12
Security  Test

Checking  for  known  security  vulnerabilities  within  code  components,  infrastructure,  systems  and  services.

DevOps  Techniques
– Whole  team  is  responsible  for  security
– Build  security  acceptance  criteria  into  user  stories  
– Security  tests  can  be  applied  at  all  l evels  of  testing  
– Automate  security  testing  where  possible
– Employ  static  analysis  to  identify  security  issues

http://devops.com/2015/04/06/automated-­security-­testing-­continuous-­delivery-­pipeline/

13
Penetration  (PEN)  Test
Uses  ethical  hacking  techniques  to  penetrate  an  application  for  the  purpose  of  finding  security  vulnerabilities  
that  a  malicious  hacker  could  potentially  exploit.

DevOps  Techniques
– Use  monitoring  tools  and  data  analytics  to  identify  penetration  attacks
§ e.g.  SQL  injections  i dentified  by  pre-­programmed  error  messages
§ e.g.  Monitor  for  strange  l og  outputs  automated  to  alert  team  members
– Consider  crowdsourcing  for  penetration  testing
– Leverage  automation  using  vulnerability  scanning  tools  early  in  the  lifecycle
– Utilize  development  platforms  and  frameworks  that  employ  current  security  capabilities

http://devops.com/2015/01/20/devops-­need-­pay-­attention-­security/

14
 Security  &  PEN  Test  Tools
IBM Provides  web  &  mobile  application  security,  application  security  program  management  and  regulatory  compliance

http://www-­03.ibm.com/software/products/en/appscan/

ZAP  by  OWASP

Integrated  penetration  testing  tool  for  web  applications  

https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project

METASPLOIT   by   Provides  penetration  testing  in  addition  to  other  security  &  compliance  automation  solutions
http://www.rapid7.com/products/metasploit/

BURP  by
Provides  penetration  testing  in  addition  to  other  security  testing  
https://portswigger.net/burp/

BDD-­Security  by
Extensible  framework  for  security  testing  

http://www.continuumsecurity.net/bdd-intro.html

15
Compliance   Test
Testing  for  compliance  against  prescribed  IT  standards

DevOps  Techniques
– Compliance  items  should  be  acceptance  criteria  for  user  stories
– Automate  the  validation  of  acceptance  criteria,  if  possible
§ Question  any  compliance  item  that  cannot  be  automated
– Express  compliance  requirements  in  a  form  that  is  code-­able:
§ Compliance  as  code
– Automated  tests  results  become  the  documented  proof  of  compliance

http://devops.com/2014/11/04/automating-­away-­regulatory-­compliance-­myth/
http://www.datical.com/can-­regulatory-­compliance-­automated-­enterprise-­devops/

16
Compliance   Tools

Provides  web  &  mobile  application  security,  application  security  program  management  and  regulatory  compliance

http://www-­03.ibm.com/software/products/en/appscan/

An  automated,  enterprise-­level  compliance  management  tool  can  reduce  the  cost  of  compliance

http://www.mastercontrol.com/21_cfr_regulations/compliance_management/tools.html

Requirements  included  as  infrastructure  code  to  embed  compliance  within  automated  auditing  capabilities.

https://www.chef.io/solutions/audit-compliance/

Creates  compliance  modules  tailored  for  environment  regulations  and  ensures  mandates  are  regularly  
checked,  variances  reported,  and  when  appropriate,  reverts  non-­compliant  systems  back  to  known-­good-­state.  
Entire  configuration  policy  is  also  readable  and  version  controlled  to  track  changing  policies.

https://puppetlabs.com/solutions/it-compliance

17
References
§ Performance  Test
– https://msdn.microsoft.com/en-­us/library/hh917308.aspx

§ Resiliency  Test
– Embracing  Failure:  Fault  Injection  and  Service  R esilience  at  N etflix:  https://www.youtube.com/watch?v=9R710ry-­Cbo
– Injecting  Failure  at  Netflix:  https://www.youtube.com/watch?v=ioXV28GtXeo

§ Compatibility  Test
– https://www.youtube.com/watch?v=zIgmkllfT-­w

§ Usability  Test
– https://design.ibm.com/thinking.html
– http://www.usefulusability.com/14-­usability-­testing-­tools-­matrix-­and-­comprehensive-­reviews/

§ Security  Test
– http://devops.com/2015/04/06/automated-­security-­testing-­continuous-­delivery-­pipeline/
– http://devops.com/2015/01/20/devops-­need-­pay-­attention-­security/
– https://www.novacoast.com/devop-­secops-­part-­3-­continuous-­security-­in-­
action/?utm_content=10777161&utm_medium=social&utm_source=twitter
§ Compliance  Test
– http://devops.com/2014/11/04/automating-­away-­regulatory-­compliance-­myth/
– http://www.datical.com/can-­regulatory-­compliance-­automated-­enterprise-­devops/

§ Consulting  Firms
– https://www.threatintelligence.com/

18
DevOps  Testing
Video  Education  Series
Module  5  – Test  Automation  Strategy

IBM  Agile  Academy  -­ DevOps

 Becoming  Agile Agile  P ractices DevOps  Courses Testing  Modules
Leadership Development Intro  to  Continuous  Testing

Lesson   Focus Collaboration Testing Test  Coverage  Strategy

Functional  Test  Automation

Delivery  /  Engineering  /  DevOps Deployment
Non-­Functional  Test  Automation
Infrastructure
Test  Automation   Strategy
Acceptance  Testing  &  BDD

Test  Environment  Management

Embedding  Testing  into  Continuous  Integration  

Incremental  Integration  &  Test  Virtualization

Test  Metrics

Why  is  this  topic  important?

• Automated testing is a core capability of Continuous Delivery & supports “Fail Early, Fail Often”
• One automation strategy will not meet the needs of all projects. Each project team will need to build an automation
strategy (consider where automation is appropriate and what tools are right for them).
Out of this module, you will be able to
• Learn the steps for developing an automation strategy
• Understand what can be automated and what shouldn’t be automated
• Leverage a template to build a test automation strategy for your project
• Use resources to get started and to learn more

2
Where  do  we  start?

Identify  areas   • Understand  the  area  of  greatest  need  for  testing
for  automation   • Start  there  with  automation

• Don’t  go  for  more  sophistication  than  you  need

Choose  the   • Plan  time  to  evaluate  tools  
right  tools   • Experiment
• Master  one  automation  tool  at  a  time

• Automate  at  lowest  level  possible  

Test  Design  
• Limit  scope  of  each  test  c ase  to  one  test  condition  
&   • Understand  purpose  of  test
Maintenance • Avoid  dependencies  between  tests  

3
Where  should  we  automate  first?    

Test Pyramid Test Quadrant

Acceptance Integration
System Acceptance
System Combinatorial  /  Orthogonal Usability
Exploratory
Automation
Integration Quadrant 2 Quadrant 3

Quadrant 1 Quadrant 4

Component Performance
Unit   Security
Component Compatibility
Unit Static  A nalysis Compliance
Resiliency

Note:  Focus  o n  Unit  – easy  to  a utomate,  q uick  feedback,  h igh  ROI  

4
Test  Automation  Myths

Myth:  Every  manual  test  can/should  be  automated

Reality:  Consider  cost  savings  and  make  tradeoff

Myth:  Test  automation  is  just  a  matter  of  purchasing  the  "right"  tool

Reality:  Rare  that  an  off  the  shelf  tool  will  meet  all  requirements

Myth:  Test  automation  always  leads  to  cost  savings

Reality:  Time  to  train  test  teams,  documenting  test  cases,  learning  test  tools  are  sometimes  not  considered
Testing  Pillars  

DevOps  Testing

Embrace  Testing  Early   Utilize  proven   Apply  appropriate  

and  Ensure   technologies   and   rigor
Production   Readiness   patterns
at  all  Times
Automation  strategy  b ased   Test  e nvironments  a re  
Leverage  TDD  a nd  B DD under  change  control
on  testing  p yramid  
Deliver  real  b usiness  value   Planning  for  complete  test  
Utilize  Test  V irtualization,  
every  iteration  to  p roduction coverage  (Functional  a nd  
as  required
Design  A pplication  to  b e   Non-­Functional)
Test  S uites  e xecuted  a s  
testable
part  o f  Continuous  
Integration

6
Strategy  Template  – Test  Automation    
Test Recommended  Automation Target  Automation  for  
Project  
Unit   80-­100%
Component 80-­100%
System 40-­50%
Integration 40-­50%
Acceptance 30-­50%
Static   80-­100%
Combinatorial  /  Orthogonal 40-­50%
Usability   0-­10%
Exploratory   0%
Performance   80-­100%
Resiliency 80-­100%
Compatibility 80-­100%
Security/Penetration 80-­100%
Compliance 80-­100%

7
References
§ Agile  Testing  by  Janet  Gregory  &  Lisa  C rispin
§ Test  Automation  myths:  http://www.ibm.com/developerworks/rational/library/10/build-­test-­automation-­around-­
rational-­quality-­manager/

8
DevOps  Testing
Module  6  – Acceptance  Testing  and  Behavior-­Driven  Development

IBM  Agile  Academy  – DevOps

 Becoming  Agile Agile  P ractices DevOps  Courses Testing  Modules
Leadership Development Intro  to  Continuous  Testing
Lesson   Focus Collaboration Testing Test  Coverage  Strategy

Delivery  /  Engineering  /  DevOps Deployment Functional  Test  Automation

Infrastructure Non-­Functional  Test  Automation

Test  Automation  Strategy

Acceptance   Testing   and  BDD

Test  Environment  Management

Embedding  Testing  into  Continuous  Integration  

Incremental  Integration  &  Test  Virtualization

Why  is  this  topic  important? Test  Metrics

• Behavior Driven Development (BDD) is the ability to code and test knowing the acceptance criteria are in the form of an executable test
• This allows you to execute acceptance testing continually with every new coding change
• Facilitates much more in-depth conversation with stakeholders

Out of this module you will be able to:

• Define BDD and its relationship to Acceptance Testing
• Explain the concept of Domain-Specific Language as it relates to BDD
• Understand the benefits of BDD as viewed through a DevOps lens
• Learn from examples and find resources to learn more about BDD

2
Behavior  Driven  Development  (BDD)
Before the first line of code is created…
…stakeholders, analysts, developers and testers work to translate feature behaviors into a common language. The resulting
acceptance criteria are then defined in the form of executable tests to run against the code.

BDD uses an “Outside-In” Methodology:

Stories describe features
from stakeholder POV

Whole team
delivers

Runnable, tested
features

3
 BDD:  Origins  &  Aliases

2nd Generation Approach Built On:

BDD also known as:
User Stories (Agile) Acceptance Test-Driven Development (ATDD)
• Kent Beck (TDD) / Martin Fowler
eXtreme Programming (XP)
Specification by Example
• Continuous Integration • Gojko Adzic
• Test-Driven Development (TDD)

• Acceptance Testing: Uses acceptance criteria to objectively judge if a feature has been correctly implemented

Domain-Driven Design
• Stakeholders represent one or more domains
• Each has a particular jargon or domain language

4
Domain  Specific  Language:   Applied  to  BDD
A  business  readable  or  “common”  language  facilitating  communication  across  the  whole  team  (roles,  individuals)  

Example
Gherkin:  Describes  software’s  behavior  without  detailing  
how  that  behavior  is  implemented.

Uses  plain  language  (typically  English)  as  key  words  to  

setup  test  scenarios  to  accomplish  the  business  outcome  
desired  from  the  features  being  designed  and  tested.

Developed  for  the                                    BDD  Tool

*  It’s  possible  to  create  your  own  domain-­specific  languages  based  on  whatever  unique  criteria:

-­ Finance,  Applied  Sciences,  etc.

5
Gherkin  Example  within  Cucumber

Acceptance criteria in domain-specific language

Scenario: Successful booking of a hotel reservation

Given a user navigates to the hotels reservation page

And the user searches for a hotel
Key Words

When the user selects the hotel

Then the user fills out all the required information
When the user gets to the confirmation page
Then the user views his complete booking information
And the user clicks on the Submit button

6
BDD  &  DevOps  Practices

Collaborative Development - Continuous Feedback – Centralization

•Capture stakeholder requirements into common language
•Translate acceptance criteria into user stories and prioritize
•Whole team collaborates in discussions about application behavior as
stated in domain-specific language scenarios

Test Early!
• Conduct user acceptance tests from the very start

Automation – Continuous Testing

•Use tools framework to write and automate stories
•Update test scenarios as the “living” and integrated documentation
• In order to properly test new or changed features
• Write every test to pass only when acceptance criteria is met

7
BDD  Benefits
Living documentation has Reduces communication
Enables more efficient user
acceptance criteria built into barriers between domain
story prioritization
test scenarios experts and team
Domain Experts

Better understanding of the Software development meets

Increased code quality user need
software being built

8
BDD  Tools

Selenium
Browser Automation

https://cucumber.io/ http://www.fitnesse.org/ http://www.seleniumhq.org/

Gherkin
Cucumber’s BDD language
https://github.com/cucumber/cucumber/wiki/Gherkin http://jbehave.org/ http://docs.behat.org/en/v2.5/quick_intro.html

9
References
Books
Test-Driven Development ATDD By Example
BDD In Action By Example
by Kent Beck by Markus Gärtner
by John Ferguson Smart

Amazon Web Link Amazon Web Link Amazon Web Link

BDD Philosophy
http://pythonhosted.org/behave/philosophy.html

Use Behavior-Driven Development To Make Your Software Development Project More Successful
belatrixsf.com/index.php/whitepapers-successful-project-with-behavior-driven-development

Blog Series on BDD: Value Through Collaboration By Viktor Farcic

http://technologyconversations.com/2013/11/14/behavior-driven-development-bdd-value-through-collaboration-part-1-introduction/
http://technologyconversations.com/2013/11/17/behavior-driven-development-bdd-value-through-collaboration-part-2-narrative/
http://technologyconversations.com/2013/11/25/behavior-driven-development-bdd-value-through-collaboration-part-3-s cenarios/
http://technologyconversations.com/2013/12/08/behavior-driven-development-bdd-value-through-collaboration-part-4-automation/

IBM Articles and Links

TDD content in this presentation culled from “Junit and Test Driven Development…..” presentation by Justin Gordon.

http://www.ibm.com/developerworks/java/library/j-cq09187/index.html … In pursuit of code quality: Adventures in BDD

10
 Thanks  for  watching!

11
DevOps  Testing
Video  Education  Series
Module  7  – Test  Environment  Management

IBM  Agile  Academy  – DevOps

Becoming  Agile Agile  P ractices DevOps  Courses Testing  Modules
Leadership Development Intro  to  Testing
Lesson   Focus Collaboration Testing Test  Coverage  Strategy

Delivery  /  Engineering  /  DevOps Deployment Functional  Test  Automation

Infrastructure Non-­Functional  Test  Automation

Automation  Strategy

Acceptance  Testing  &  BDD

Test  Environment   Management

Embedding  Testing  into  Continuous  Integration

Incremental  Integration  &  Test  Virtualization

Test  Metrics

Why  is  this  topic  important?

• Automated test environment creation and teardown is core in enabling Continuous Testing
• Separate Capacity Test Environment is unique for performance sensitive applications
• Managing test data enhances the quality of testing efforts
• Out of this module you will be able to:
• Learn key principles for test environment management
• Get additional resources to further enhance your understanding on this topic

2
Test  Environment  and  DevOps

Everyone  is  involved  in  testing!

q Test  environment  configuration  should  be  as  close  to  
production  as  possible

q Readily  available  environments  enable  the  ability  to  shift  

testing  to  the  left

q Test environments provide capability they do not define process

q Automated  tools  reduce  the  cost  of  testing:

• By  making  work  proceed  faster  

• By  making  work  more  traceable  and  shareable

3
Transient  Test  Environment
Definition:  An  environment  created  for  short  durations

9:00AM 9:30AM 11:10AM 12:15PM

Run Ser ver Build Scripts Ser ver Ready For Use Testing Complete / Ser ver Terminated
Run Terminate Scripts

Features  that  make  up  a  Transient  environment

Scripted  Environments:  F ully  scripted,  versioned,  and  tested

Self-­service  Environments:  Any  authorized  person  on  the  team  can  launch  a  new  environment

Automatic  Termination:  Environments  automatically  terminated  based  on  team  policy  

-­ Team  members  have  no  option  to  override  the  policy

4
Transient  Test  Environment:  Tasks  &  Benefits
Key  tasks  to  achieving  benefits

1.  Create  a  team   2.  Automate  environment   3.  Schedule  environment  

policy termination termination

Determine  a  policy  that  fits   Terminate  all  environments   Create  a  schedule  to  run  the  
your  project  requirements   that  exceed  the  team  lease   environment  termination  script
policies

Key  benefits
Reduce  environment  d ependency

Better  u se  o f  resources

Reduce  n eed  for  knowledge  transfer

5
Capacity  Test  Environment
Is  your  application  performance  sensitive?
§ Use  an  environment  as  close  to  production  as  possible
§ Aim  for  stable,  reproducible  results
§ Isolate  environment  from  other  influences  

Alternative  Techniques

Canary release
Roll  out  to  a  subset  of  servers

Scaling
Run  tests  on  small  configurations  of  servers  but  be  
cautious  of  just  extrapolating  results

6
Test  Data  Management  

Practices

1. Coverage of  end  to  end  test  data

2. Automate creation  of  realistic  “right-­sized”  test  data

3. Mask sensitive  information  for  compliance  and  protection

4. Refresh test  data  for  continuous  integration  and  delivery

5. Analyze test  data  results

7
Test  Data  Management  Considerations
Commit  Tests Acceptance  Tests
– Avoid  elaborate  data  set  up – Data  is  more  complex  due  to  system  and  integration  needs

– Simulate   dependencies – Minimize   dependency  on  large  data  structures  as  much  as  possible

– Use  the  minimum   amount  of  test  data  to   – Use  the  minimum   amount  of  test  data  to  test  the  expected  behavior  

assert  the  unit  being  tested  exhibits  the   of  the  system

expected  behavior – Test  isolation   strategies  are  k ey  to  ensure  y our  test  data  is  
unaffected  by  other  tests
– Use  tests  that  create  the  data  needed  as  part  of  the  set  up  process
– Do  not  use  full  production  data  dumps,  instead  carefully  select  the  
data  

Capacity  Tests Other  Tests

– High  volume  of  “transactional”  data  with   – For  manual  tests,  use  customized  data  sets
“reference”  data  is  needed
– Data  must  support  simultaneous  tests
– Automation   of  data  generation  is  the  best  
practice
8
References

§ Test  Environments  and  DevOps

– https://blog.newrelic.com/2014/06/12/divingintodevops/

§ Transient  Environments
– https://www.ibm.com/developerworks/library/a-­devops3/

§ Capacity  Test  Environments

– Continuous  Delivery,  Jez  Humble  &  David  Farley,  Chapter  9  
Testing  Non-­Functional  Requirements
§ Test  Data  Management
– https://www.ibm.com/developerworks/community/blogs/invisibl
ethread/entry/test_data_management_in_the_devops_lifecycle
?lang=en
– Continuous  Delivery,  Jez  Humble  &  David  Farley,  Chapter  12  
Managing  Data
9
DevOps  Testing
Video  Education  Series
Module  8  – Embedding  Test  into  Continuous  Integration

IBM  Agile  Academy  – DevOps

Becoming  Agile Agile  P ractices DevOps  Courses Testing  Modules
Leadership Development Intro  to  Testing
Lesson   Focus Collaboration Testing Test  Coverage  Strategy

Delivery  /  Engineering  /  DevOps Deployment Functional  Test  Automation

Infrastructure Non-­Functional  Test  Automation

Automation  Strategy

Acceptance  Testing  &  BDD

Test  Environment  Management

Embedding   Test  into  Continuous   Integration

Incremental  Integration  &  Test  Virtualization

Test  Metrics

Why  is  this  topic  important?

• Without  Continuous  Test,  there  is  no  Continuous  Integration.    
• Beyond  test  automation,  there  are  other  practices  that  are  aimed  to  integrate  those  automated  test  
steps  into  your  CI  process  and  tools,  creating  an  effective  Continuous  Integration  pipeline.
• Out  of  this  module  you  will  be  able  to:
• Learn  key  principles  to  integrate  your  automated  tests  into  your  CI  Pipeline.
• Get  additional  resources  to  further  enhance  your  understanding  on  this  topic.

2
Continuous  Integration  Test:  DevOps  Principles

1. Automation:  Create  a  comprehensive  

Automated  T est  suite
– As  many  test  levels  as  you  can

2. Process:  Keep  the  test  process  short

– Easy  to  follow  and  execute
– Takes  less  time  to  run
– Fosters  multiple  integrations  per  day  mindset

3. Technical  Capability:  Use  a  powerful  CI  

system
– Integration  of  test  execution  and  results  in  
one  place
– Optimize  speed  of  execution
– Accelerate  feedback

3
Continuous  Test  Techniques  Applied  to  CI:  Automation

1. Automate Unit Tests

2. Automate Component Tests

3. Automate System Tests

4. Automate Functional/Acceptance Tests

4
Continuous  Test  Techniques  Applied  to  CI:  Process

To allow to run slow tests at different

1. Categorize developer tests intervals than faster tests. Optimize
velocity.

2. Run faster tests first Unit, then component then system

Increase your code coverage “naturally”

3. Write tests for defects and ensure the defect does not surface
again
Use Db testing frameworks to make
4. Make component tests certain the data is a “”known state”,
repeatable facilitating repeatability of component
tests
Spend less time tracking down the cause
5. Limit test cases to one assert of a test failure by limiting your automated
tests to one assertion per test

5
Continuous  Test  Techniques  Applied  to  CI:  Technical  Capabilities

Use  an  automated  CI  Tool  that  can  easily  integrate  all  types/  
levels  and  frameworks  of  test  and  can  aggregate  test  results

Once  a  test  fails,  you  can  easily  isolate  the  error  and  then  
trace  it  back  to  the  offending  commit

Ability  to  trigger  defects  from  your  CI  tool  for  ease  of  integration  
with  the  software  delivery  lifecycle  management  process

6
Continuous  Test  Techniques  Applied  to  CI:  Technical  Capabilities

Encapsulation  and  orchestration  by  test  stages  or  other  categories

Execute  tests  based  on  pre-­set  schedules

Provide  whole  team  visibility/dashboards  of  

the  Continuous  Integration  steps

7
CI  Tools  and  Automated  Test  Tools:  Integration

Rational
CI Tool/Test Tool Ant xUnit TestNG
Integration Test
Selenium Cucumber

RTC Yes Yes Yes* Yes Yes No

Jenkins Yes Yes Yes Yes Yes Yes

Bamboo Yes No No No Yes No

CruiseControl Yes Yes Yes No Yes Yes

Gradle Yes Yes Yes No Yes Yes

Yes No No No Yes No

*may require 3rd party plug-in for test results report

References
§ Continuous  Integration  by  Paul  M.  Duvall

§ Continuous  D elivery  by  Jez  Humble  &  David  Farley

§ Agile  Testing  by  Lisa  Crispin  &  Janet  Gregory

§ Continuous  Integration  Tools

– IBM  RTC:  https://www-­01.ibm.com/support/knowledgecenter/SSYMRC_5.0.2/com.ibm.team.build.doc/topics/t_build_overview.html
– Jenkins:  h ttps://www.cloudbees.com/jenkins/about
– Bamboo:  
https://www.atlassian.com/software/bamboo?_mid=a7cb976d9753e14607bebbc137801f1b&gclid=COGeoomms8cCFVcvgQodI_EAl
g
– Cruisecontrol:  h ttp://cruisecontrol.sourceforge.net/
– Gradle:  https://gradle.org/
– Codeship:  h ttps://codeship.com/pricing-­organizations

9
DevOps  Testing
Video  Education  Series
Module  9  – Incremental  Integration   &  Test  Virtualization

IBM  Agile  Academy  -­ DevOps

 Becoming  Agile Agile  P ractices DevOps  Courses Testing  Modules
Leadership Development Intro  to  Continuous  Testing

Lesson   Focus Collaboration Testing Test  Coverage  Strategy

Functional  Test  Automation

Delivery  /  Engineering  /  DevOps Deployment
Non-­Functional  Test  Automation
Infrastructure
Test  Automation  Strategy

Acceptance  Testing  &  BDD

Test  Environment  Management

Embedding  Testing  into  Continuous  Integration  

Incremental   Integration  &  Test  Virtualization

Why  is  this  topic  important? Test  Metrics

– An  optimal  test  strategy  finds  defects  at  the  earliest  point  possible,  w here  they  are  least  expensive  to  find  &  fix
– By  definition,  “incremental  testing”  i s  the  partial  testing  of  an  i ncomplete  product
– “Incremental  Integration  testing” allows  you  to  find  additional  defects  after  unit  testing  but  before  a  full-­blown  i ntegration  
test  where  all  components  are  required  to  be  available
– “Test  Virtualization” enables  you  to  simulate  data  &  transactions  that  cannot  be  made  part  of  the  test  environment  but  are  
necessary  for  pre-­production  testing

Out of this module, you will be able to:

– Understand  w hen  this  type  of  testing  should  be  applied
– Leverage  tools  &  techniques
– Leverage  references  for  getting  started  &  continued  l earning

2
Integration  Test  approaches

Top  Down  
Start  with  the  highest  level  component  and  integrate  additional  components  (working  
downwards)  replacing  not-­yet-­available  components  with  “stubs”,  which  mimic  their  behavior

Bottom  Up
Starting  from  the  bottom,  integrated  components  until  the  entire  system  is  complete,  not-­yet-­
available  components  are  replaced  by  “drivers”  

Hybrid
Combinations  of  available  modules  are  tested.    Stubs  and  drivers  can  be  applied  to  accelerate  
testing  until  a  full  integration  test  can  be  performed

3
Integration  Testing:  Top  Down
Start  with  the  highest  level  component  and  integrate  additional  components  (working  
downwards)  replacing  not-­yet-­available  components  with  “stubs”,  which  mimic  their  behavior

Test 1 Test 2 Test 3 Test 4

Test Framework Test Framework Test Framework Test Framework

A A A A

B C B C B C B C

D D D D
Under test
Already tested
Stub or driver

Pros Cons
-­ Early  demonstration  o f  function  can  h elp  validate  stakeholder  intent -­ Complex  stubs  can  introduce  more  e rrors

-­ Preferred  if  d efects  a re  more  likely  in  the  top  level  modules   -­ Cost  to  create  stub  -­vs-­ actual  module

4
Integration  Testing:  Bottom  Up
Starting  from  the  bottom,  integrated  components  until  the  entire  system  is  complete,  not-­
yet-­available  components  are  replaced  by  “drivers”    

Test 1 Test 2 Test 3 Test 4

Test Framework Test Framework Test Framework Test Framework

A A A A

B C B C B C B C

D D D D Under test
Already tested
Stub or driver

Pros Cons
-­ Preferred  if  d efects  a re  more  likely  in  the  b ottom  level  modules   -­ Need  to  create  d rivers  (usually  e asier  than  stubs)

-­ Test  conditions  a re  e asier  to  create -­ The  e ntire  system  is  subject  to  the  smallest  a mount  o f  test  b ecause  the  top  
modules  a re  included  in  the  tests  a t  the  final  stages
-­ Reduced  e ffort  in  creating  stubs
-­ Deferred  a bility  to  validate  stakeholder  requirements  a t  the  “UI”  level

5
Integration  Testing:  hybrid  
Combinations  of  available  modules  are  tested.    Stubs  and  drivers  can  be  applied  to  accelerate  
testing  until  a  full  integration  test  can  be  performed

Testing can occur concurrently – when modules, stubs and drivers are available
Test Framework Test Framework Test Framework Test Framework

A A A A

B C B C B C B C

D D D D Under test
Already tested
Stub or driver

Pros Cons
-­ It  a llows  the  u se  o f  top  d own  a nd  b ottom  u p  a pproaches  concurrently   -­ Regression  testing  is  time  consuming  u nless  tests  a re  a utomated

-­ It  a llows  a  focus  o n  the  middle  tier  where  the  most  complex  functionality  may  b e   -­ Cost  to  create  stubs  a nd  d rivers

6
 Considerations   for  when  to  apply  
Incremental  Integration  testing

l Does  the  cost  &  time  to  create  stubs  &  drivers  outweigh  cost  of  developing  the  real  
components?
l How  accurate  can  /  should  you  be  in  mimicking  real  behavior?
l Can  you  afford  to  wait  for  a  full  system  integration  test?
l How  valuable  is  early  exposure  of  system  functionality  to  your  stakeholders?

l Do  you  know  the  value  of  your  test  in  terms  of  possible  test  coverage  (based  on  analysis  of  
module  cohesion  &  coupling)?
l Can  you  avoid  the  “Big  Bang”?

7
Why  Test  Virtualization?

In complex projects, components necessary for

full integration testing might not be yet available
due to:

• Not complete, or still evolving

• Controlled by a third-party
• Limited availability
• Difficult to provision
• Restricted or costly to use

Such components can be “virtualized” and managed / accessed in a

virtualized test environment

8
Test  Virtualization  solutions

Application  emulation
Virtual components can simulate the behavior of an entire
application or a specific component.

Multiple  test  environments  

Test environments can be created by using virtual components
configured for different needs

Same  testing  tools

Existing testing can be used, the tools can’t tell the difference
between a real system and a virtual service.

9
Tools  &  References
IBM Rational Test Workbench
http://www-­03.ibm.com/software/products/en/rtwFifth

IBM Rational Test Virtualization Server

http://www.ibm.com/developerworks/rational/products/testvirtualizationserver/

Test Virtualization IBM Rational

https://www.youtube.com/watch?v=_wl7cfV2xWE

When to use Service Virtualization

https://www.youtube.com/watch?v=j1f5vP3gCIM

Getting better results with Virtualization testing

https://www.youtube.com/watch?v=j1f5vP3gCIM

10
DevOps  Testing
Video  Education  Series
Module  10  – Test  Metrics

IBM  Agile  Academy  -­ DevOps

 Becoming  Agile Agile  P ractices DevOps  Courses Testing  Modules
Leadership Development Intro  to  Continuous  Testing

Lesson   Focus Collaboration Testing Test  Coverage  Strategy

Functional  Test  Automation

Delivery  /  Engineering  /  DevOps Deployment
Non-­Functional  Test  Automation
Infrastructure
Test  Automation  Strategy

Acceptance  Testing  &  BDD

Test  Environment  Management

Embedding  Testing  into  Continuous  Integration  

Incremental  Integration  &  Test  Virtualization

Why  is  this  topic  important? Test  Metrics

• Metrics show progress towards objectives and drive course correction

• Metrics are vital for understanding and improving process and team performance
Out of this module, you will be able to:
• Understand the DevOps view of test metrics
• Identify types of test metrics and examples of each type
• Gain useful resources to help you get started and continue learning

2
Test  Metrics

Results  are  driven  by  what  you  measure  and  how  you  measure  it

DevOps  View  of  Test  Metrics:

– Automate  measurement  and  reporting  

– Implement  continuous  measurement
– Whole  Team  – Make  test  results  visible  to  entire  team
– Focus  on  trends  to  drive  continuous  improvement

3
Types  of  Test  Metrics

Test  Metrics  must  look  at  both  product  and  process

Multiple  dimensions:

– Test  process  effectiveness  and  efficiency

– Functional  Product  Quality  
– Non-­functional  Product  Quality

4
Test Process Effectiveness and Efficiency Metrics
Test Automation

• Percent  of  testing  that  is  automated  within  each  

Test Automation
quadrant
• Percent  of  testing  that  is  automated  within  each  
testing  level

Percent test cases based on TDD and

BDD

Test Confidence Metric Defect Identification Efficiency

• Measure  of  the  testing  team’s   • Compares  the  number  of  defects  
confidence  level  in  the  testing   identified  in  different  stages  of  software  
efforts  executed  on  a  release,   lifecycle  versus  which  stage  the  defects  
based  on  test  execution  coverage,   were  created  in,  with  target  to  be  more  
expertise  of  testers,  and   efficient  by  finding  defects  earlier
quantitative  results  of  defects
Production Defects
Functional Product Quality Metrics
Pass / Fail

• Percent  of  planned  functional  tests  that  have  been  completed  

Test Automation
successfully
Structural Testing Coverage

• Code  Coverage  – what  percent  of  code  and  code  paths  have  been  tested
• Story  Coverage  – what  percent  of  user  stories  relevant  to  the  system  
under  test  been  tested
• Menu  Coverage  – what  percent  of  menu  options  and  combinations  /  
sequences  of  menu  options  have  been  exercised
Other functional metrics

• Velocity  Variance  – Predictability  of  velocity  is  indicator  of  ability  to  meet  
acceptance  criteria  consistently  
Non-Functional Product Quality Metrics
Performance

• Response  time  – actual,  trend,  

variance    

Static Analysis Coverage

• Percentage  of  code  that  has  gone  

through  static  analysis

Unit Test Non-functional Coverage

• Percentage  of  non-­functional  test  

cases  executed  at  unit  test  level
Other DevOps Measurements:

Technical Debt

Code Complexity and

Cohesiveness

Frequency of Deployments Also…

Deployment Failures
Rate of change among all
indicators
Customer Feedback

Repair Time

Volume of Repair Requests

References
Testing Metrics
Continuous Delivery by Jez Humble

DevOps & Performance Metrics

https://blog.sei.cmu.edu/post.cfm/missing-metrics-devops-149

https://blog.newrelic.com/2015/04/28/devops-jez-humble/

https://decoupledlogic.wordpress.com/2014/05/31/optimizing-the-software-delivery-pipeline/

http://www.slideshare.net/JAXLondon2014/jax-london-slideshare