Inverting the Pyramid

Naresh Jain
[email protected]
@nashjain
http://nareshjain.com

Copyright © 2013, AgileFAQs. All Rights Reserved.

 Plan
Back in the Stone-age

Design
Happiness/Excitement

Distribute

Work in
Isolation

Integrate

Time/Money/Opportunity Cost
Last Minute Integration Surprises

Copyright © 2013, AgileFAQs. All Rights Reserved.

BAD things were visible too Late...
Birth of CI
 CI Helped Us Learn That...
Life can Suck a lot Less!
Collaboration Feedback Quality

Delivery Time Wastage

Copyright © 2013, AgileFAQs. All Rights Reserved.
Lean-Start-up Community
Tried something quite disruptive...
Continuous Deployment

Copyright © 2013, AgileFAQs. All Rights Reserved.

But...
Software Testing Ice-cream Cone

Manual Checking

End-to-End
GUI Tests 10-20%

Integration
5-10%
Tests

Unit
Tests 1-5%

Copyright © 2013, AgileFAQs. All Rights Reserved.

Commercial Break!
Copyright © 2013, AgileFAQs. All Rights Reserved.
Mumbai
Tech Talks!
 Testing VS. Checking
Testing is explorative, probing and learning oriented.
Checking is confirmative (verification and validation of what we already
know). The outcome of a check is simply a pass or fail result; the outcome
doesn’t require human interpretation. Hence checking should be the first
target for automation.
James Bach points out that checking does require some element of testing; to create
a check requires an act of test design, and to act upon the result of a check requires
test result interpretation and learning. But its important to distinguish between the
two because when people refer to Testing they really mean Checking.

Why is this distinction important?

Michael Bolton explains it really well: “A development strategy that emphasizes checking at the
expense of testing is one in which we’ll emphasize confirmation of existing knowledge over
discovery of new knowledge. That might be okay. A few checks might constitute sufficient testing
for some purpose; no testing and no checking at all might even be sufficient for some purposes.
But the more we emphasize checking over testing, and the less testing we do generally, the more
we leave ourselves vulnerable to the Black Swan.”

Copyright © 2013, AgileFAQs. All Rights Reserved.

Inverting the Testing Pyramid
Typical testing strategies lead to an inverted testing pyramid...

Manual Checking

End-to-End
GUI Tests 80-90%

Integration
5-15%
Tests

Unit
Tests 1-5%
Copyright © 2013, AgileFAQs. All Rights Reserved.
Inverting the Testing Pyramid
GUI 1%
Tests Performance
Tests
End to End 4% Security
Flow Tests Tests
One Layer Below GUI

Workflow Tests 6%

Integration Test 9%

Biz Logic Acceptance Tests 10%

Unit Tests 70%

This is the need of the hours...

Copyright © 2013, AgileFAQs. All Rights Reserved.
Number of End-to-End Tests
T1 T2 T3

DB

Classes
Method Calls

8 9 11

70 Complex, Fragile End-to-End Tests

vs.
61 Unit tests + 8 End-to-End Tests

Copyright © 2013, AgileFAQs. All Rights Reserved.

 Best ROI for Testing
Impact Lines of Test Code

End to End 10% 90%

Layers

API
65% 35%

Modules 70% 30%

Unit 90% 10%

0 25 50 75 100

Value
Copyright © 2013, AgileFAQs. All Rights Reserved.
Inverting the Pyramid
How to…?
 Types of Tests in the Testing Pyramid
GUI
Tests GUI 1%
Tests Performance
Acceptance Tests
Tests End to End 4% Security
Flow Tests Tests
One Layer Below GUI

Unit Tests Workflow Tests 6%

Small Agile Projects with Single Team Integration Test 9%

Biz Logic Acceptance Tests 10%

Unit Tests 70%

Large Scale Agile Projects with Global Teams

Copyright © 2013, AgileFAQs. All Rights Reserved.
 Types of Tests for Large Agile Project
Type Tool Who When Level Objective
xUnit, Mockito/ Each class functions correctly in isolation. The
Unit Tests Moq, Jasmine, Developers During the sprint, while coding Class UI components (including JS) can also be unit
tested in isolation.
HSqlDB, Jumble

Biz Logic Cucumber (some TechBAs + Acceptance Criteria are Work-steam specific
From a business logic point of view, the
functionality is implemented correctly. Works
defined before sprint starts.
Acceptance may need Lisa) Developers + Acceptance Tests are written business functionality in isolation with other work-streams/modules.
(At component level) Also good point to do some basic
Tests JMeter/LoadRunner Testers (QA) during sprint, before coding performance testing.

Between dependencies
Integration Developers + Contracts are defined before 2 Components can communicate with each
(work-streams, other correctly. Best suited for negative path
xUnit + Lisa Testers (QA) + sprint, tests are implemented
Test TechBAs during the sprint upstream/down-stream testing. Should not test any functional aspects
of the application.
systems, 3rd Party)

SMEs + TechBAs + UX Across a few work- Inside a trade work flow, a particular step
Workflow When a theme is picked (issuance or acceptance workflow) works
Cucumber + Lisa + Developers + Arch (functional group of sprints) streams (Issuance/
Tests + Testers (QA + UAT) Acceptance)
correctly from the business point of view.
Upstream/Downstream dependencies stubbed.

When an Internal Milestone is

SMEs + TechBAs + UX decided, the E2E flow tests are Ensure a specific type of trade can go thru all
End to End Cucumber defined & authored. As & when End to end Trade flow, the steps successfully.
+ Developers + Arch + the complete workflow is
Flow Tests JMeter/LoadRunner
Testers (QA + UAT) implemented these tests are
one layer below the UI Very important to do detailed performance
testing at this stage.
enabled.

Usually its takes 3-4 sprints to

Selenium/Sahi/ complete one whole workflow End to end Trade flow
TechBAs + UX + Ensure few basic types of trades can go thru
GUI Tests Watir/White/ including the UI. After that the
SWTBot Testers (QA) UI tests will be authored and with GUI
all the steps successfully via a UI.
executed.

Copyright © 2013, AgileFAQs. All Rights Reserved.

“Testing” Touch Points during Project Lifecycle
Work- Internal
Product Release 1st Release Sprint Sprint
Themes Stream Sprint Milestone Release
Discovery Planning Planning Review
Planning Release

➡Collaborate to ➡Ensure themes ➡Validate Acceptance ➡Show demo (parts) ➡Sanity Testing in
create the story map are coherent criteria at story level ➡Confirm that the biz Live environment
➡Help SMEs to ➡Create theme ➡Estimate Tasks requirements are met ➡Participate in
define Acceptance level epics with stakeholders Release retros
Criteria at each level ➡Author End-to- ➡Participate in Sprint
in story map end flow tests retrospectives

➡Define goals & ➡Author Work-flow tests ➡Collaborate with Dev ➡Sanity Testing in ER cycles
readiness criteria ➡Collaborate on identifying & Biz to write Automated ➡Confirm user goals are
for each Release. dependency and Acceptance Test met with operations team
documenting them ➡Exploratory Testing ➡Participate in Internal
➡Create stream level epics ➡Buddy testing Release retrospective
➡Create Integration Test ➡UI automation of
previous sprint tasks
➡Performance &
Security Tests

➡ The Continuous Integration and Build Pipeline process will execute all tests as part
of the regression pack and report various quality parameters to all stakeholders.
➡ On going Governance & Test Data management
Copyright © 2013, AgileFAQs. All Rights Reserved.
Collaborative Product Discovery Steps
Personas
Elevator Business Pragmatic User A
Pitch Goals Personas Goals c
Chartering c
Day in Life
e
Scenarios & p
of each
Narratives
Persona t
a
Participants n
Product Owners Story Mapping
Activity Map c
SMEs
Operation/Support Reps e
Tech BAs UI Sketch
Interaction
Task Map
User Experience Designer Design C
App Developers r
Architects Planning i
Testers (QA+UAT) Reiterating
Grouping by Prioritization t
Themes e
Project Managers
r
User Story Authoring
i
Acceptance User
Criteria Stories a
Typical Continuous Integration Process
Reports

Checkstyle

Test Test

Copyright © 2013, AgileFAQs. All Rights Reserved.

 Dev Box Continuous Integration Build Pipeline
Update
Local Version Checkout
Dev Build Checkin
Control
~5 Min
Team CI Server ut
Checkout
Project/Task

Checkout
ko
ec Each build will update
Management

Ch
Smoke Ch Task statues

eck
Build
Tool

out
~10 Min
Team CI Server
Retrieve Functional
Artifacts Build
~30 Min
Retrieve Dev Env
Artifacts
Cross
ts

Publish
ifac

Artifacts Stream
Art

(jars) Build
d

~60 Min
ts
she

c SIT Env
ti fa cts
it fa
ubli

Ar r
e d
nt
A Theme
eP

lish nd
e Build
ub
riev

e P e pe ~60 Min
v eD acts
e Artif
Ret

tri ot shed
QA Env
Re om u bl i
h /Pr e t r i eve P Product
Dependency u bli
s R
ed Artifacts Build
Management
P Retrieve Publish ~60 Min
UAT Env
UAT
Build
~30 Min
Staging Env
SCM Staging
Each build will broadcast build
stats and other reports to all Build
~30 Min
relevant stakeholders at each Prod Env
stage of the build pipeline
Live
Copyright © 2013, AgileFAQs. All Rights Reserved.
Build
~30 Min
 Local Cross
Smoke Func Theme Prod UAT Staging Live
Dev Stream
Build Build Build Build Build Build Build
Build Build

Ensure the Ensure each Ensure each

Ensure each story
developer has Ensure each story’s implemented, implemented end-to- Ensure the product
integrates with other Ensure
checked in all files, business logic is complete trade end feature works as as as a whole can
Ensure basic sanity streams and business Operations can
the code compiles, implemented workflow works expected including work with other Deploy the product
before the logic continues to execute an end to
Objective developer checks
all unit tests work correctly and
work as expected.
as expected. various performance
end workflow and
systems in the to the live
and basic code satisfies the Interact with as and security related enterprise. Make environment.
in code. Promote new it meets their
quality related doneness criteria at many sub-systems SLAs are met. Also test Go/No-Go
produced jars to goals.
stats are reported the story level & interfaces as out the migration decision.
dependent teams.
to the devs. possible. scripts

Integrate promoted
Clean environment, code with other Clean Migrate existing system Migrate existing
Retains the Migrate existing
Compiles, Unit test setup database, team modules, runs cross- environment, to the latest version, system to the
system to the latest
environment. Only
and static/dynamic specific biz team workflow & setup dependent setup dependent latest version,
version, test the Migrate existing
compiles and tests acceptance tests. integration tests in systems, execute systems, execute setup dependent
Job locally changed
code analysis on an
Stubs/Mocks out an incremental env. theme-level product-level systems, let the
system with other system to the latest
incremental systems during the version and party!
code other functional Stubs/Mocks other acceptance tests. acceptance and UI Operations team
environment Enterprise Release
(incremental). modules or sub- sub-systems. As few stubs/ tests. Almost no stubs/ use the system
cycle.
systems. Promote jars to mocks as possible. mocks. thoroughly.
dependent stream.
Duration ~ 5 mins. ~ 10 mins. ~ 30 mins ~ 60 mins ~ 60 mins ~ 60 mins ~ 30 mins ~ 30 mins ~ 30 mins

Stake- Developer writing Developers of a

Team members of a
given team + Teach
All functionally
affected Teams + All teams + SMEs All teams + SMEs
SCM + SMEs +
Enterprise Wide World-wide
holders the code given team
BA SMEs
Operations Team

On successful On Successful
Trigger/ Before checking in
code. At least
Every checkin. At
least every 30
On successful On successful
Smoke build. At least Functional build. At Cross-Stream
On successful Theme
build. At least once Each Sprint
Each Enterprise Staging Build
Frequency every 30 mins. mins. every 2 hours least twice each day build. At least
once each day
each week
Release Cycle (Manually triggered
for now)
On developer’s Stating
Where workstation
Team CI Server Team CI Server Dev Environment SIT Environment QA Environment UAT Environment
Environment
Production Env

Team produced Jars/ Fully functional Configured & packaged Configured & Configured &
Team produced Integrated jars
Code+Tests Wars. Static+ app with app (war) with packaged app packaged app (war)
Artifacts +Config
Jars, static code
Dynamic Code
pushed to
workflow level migration scripts & (war) with with migration
$$
analysis reports dependent teams
Analysis reports reports traceability reports. migration scripts scripts

➡ DevOps Roles (currently missing) & Test Data strategy is very important to get this working.
➡ Overall governance model will also be required to make sure build failures are addressed immediately.
Copyright © 2013, AgileFAQs. All Rights Reserved.
Example Jenkins Build Pipeline Plugin’s View

Copyright © 2013, AgileFAQs. All Rights Reserved.

 Test Driven Development

Add a Test

Pass
Run the Test

Fail
TDD Rhythm - Test, Code, Refactor Make a little
change

Fail
Run the Test
Pass
Refactor

Copyright © 2013, AgileFAQs. All Rights Reserved.

 Acceptance Test Driven Development
Iteration

Automated
Acceptance Acceptance Tests P
E
Criteria R
F
O
R
Story Automated M
Unit Test E
T
Automated UI N
E
C
Tests S
E
T
S

Automated
Acceptance Tests

Acceptance
Exploratory Criteria
Testing

Copyright © 2013, AgileFAQs. All Rights Reserved.

Thank you
Naresh Jain
[email protected]

Copyright © 2013, AgileFAQs. All Rights Reserved.