Bringing collaborative test to life:

An example of community effort

Fred Blesser
Technology Manager – Symbian Foundation

Mariusz Lasek
Business Development Director – Comarch

1
Collaborative test
 Collaborative Test consists of having the community contributing to the
validation and testing of the Symbian Foundation asset, with specific
focus on:
 Feature development
 Hardening
 These tests
 Shall be run on vanilla kits delivered by the Symbian Foundation to the
community
 No differentiation
 Shall be easily reproducible, without the need of specific equipment/tools,
except the ones available through the Symbian Foundation or open source
community
 No discrimination
 Test results shall be available for all to see and compare

2
Test working group update
 Key current responsibilities:
 Define the Symbian^3 test plan
 Define the Symbian^4 test plan
 Help the Release Council define the milestone acceptance criteria
(proposals, etc)
 Feature Complete
 Stable
 Participate in the validation and testing of Symbian^3
 New members
 Atelier
 Symbian DevCo
 Comarch

3
Comarch Introduction

4
Comarch Intro

 Comarch established in 1993

 Founder & CEO Prof. Janusz Filipiak
 Comarch is the largest Polish Software House
 Quoted on the Warsaw Stock Exchange (Public company)
since 1999
 Headquartered in Kraków, Poland
 Well diversified in order to embed business stability and security
 Developing products and rendering services for
Telecommunications, Banking & Finance, Industry &Utilities, Public
Sector, Trade and Small & Medium Enterprises

5
Comarch Today
 Independent software vendor selling
proprietary products, solutions and services
on the global market
 Strong direction towards matured markets of
Western Europe and the USA
 Offices located in Europe, Americas and
APAC
 Market Capitalization – 225 M€

Customers on 5 continents in more than 30 countries,

with over 3000 successfully completed projects

6
 Customer Mobile Services
 Application Development
 Platform Development
 Customization and Variant Creation
 User Experience Design

Customers
 OEM
 ODM
 Telco operators
 Banking
 Public Sector
 Retailers

Customer Mobile Applications

 Comarch Drafter
 Comarch VOM (Targeted Adds)
 Comarch Mobile Insurance
 Comarch Mobile Banking

7
Comarch Contribution
 Events and Marketing
 World Mobile Congress in Barcelona

 Towel Day 2010 in Turku, Helsinki

 Educating
 Training Contribution

 Bug Fixes

 Code Contribution
 Headset Simulator

 Collaborative Test
Database

8
 Bluetooth Headsets Simulator
 Business case
 Existing client – Mobile Services for OEMs
 Code contribution
 Services for new clients based on contributed
solution
 Consulting
 Customization

 Simulator controlled remotely by API and UI

 Test automation support
 Modular architecture
 New profiles support
 New platforms support

9
Bluetooth Headsets
Simulator

Plug-in
Test Module

Control Control
client
server

10
Bluetooth Headsets
Simulator

 Headsets Simulator
 Package: /btservices/bluetoothengine/

/*
* Component Name: Headset Simulator
* Author: Comarch S.A.
* Version: 1.0
* Copyright (c) 2010 Comarch S.A.
*/
 Code size: 30 kloc
 Effort: 9 Man-Months
 Current Status: Architectural Review

11
SYMBIAN^3 TEST PLAN

12
Symbian^3 system test plan
 System test cases have been contributed by:

 These test cases have now been prioritized:

 Priority 1: 32 test cases
 Non-concurrent test cases; to be run with each kit
 Priority 2: 152 test cases
 All priority 1 test cases, plus some more, providing sufficient feature coverage
 Will be part of FC criteria
 Priority 3: 468 test cases
 All non-concurrent test cases, plus some key concurrent ones
 Will be part of Stable criteria
 Priority 4: remaining test cases
 To be run ad-hoc

13
Proposal for Symbian^3 stable milestone
 Take the latest Symbian^3 PDK (3.0.3)
 The community will run the tests on this vanilla PDK according
to the test plan defined in previous slide
 All priority 2 system tests cases
 These results will constitute an intermediate milestone
between the Feature Complete milestone and the Stable
milestone
 The Test Working Group will then have to analyze these
results, and propose to the Release Council one of the
acceptance criteria for the Stable milestone
 All test results will be stored on the new tool contributed by
Comarch: the Collaborative Test Database (CTD)

14
Collaborative Test Database
 Collaborative Test Database (CTD)
goes live during SEE’2010 event
 New Symbian Package under
Tools domain
 Grzegorz Wachocki – Package
Owner from Comarch
 Project is hosted at Symbian.org
 Source Code available
 Contribution welcome
 Defects managed by Symbian’s Bug
Tracking
 Manual available at Symbian’s Wiki
 http://developer.symbian.org/wiki/CTD_User_Guide

15
Collaborative Test Database
 CTD has been established to record
and present the Symbian quality
 Various PDK releases
 Various test targets
 CTD is to be used by community
members
 Community verifies and correct test
results
 Check-in / Check-out mechanism in
order to improve community
efficiency
 Integration with Bug Tracking
system

16
Collaborative Test Database
 Every test result is confirmed by at least
two members (users)
 Single authority is not possible – cross-
checking improves reliability of results

 Once the test is check-out for the testing,

the whole community is aware of that fact
 Avoid overlapping in testing effort

 Check-in of the final result doesn’t mean

that it’s fixed for ever
 Any user can change the verdict at any
time – results always up to date & reflect
the actual platform quality

 Results and summaries available for

everybody

17
Collaborative Test Database
 Similarities to Test Management
Tools
 Recording Test Specifications
– list of Test Cases
 Grouping Test Cases based on the
Domain & Sub-domain
 Multimedia
 Telephony
 Kernel…

 Verifying status of the each Test Case

 Draft
 Published
 Un-published

18
Collaborative Test Database
 What makes the CTD Different?
 Designed for the purpose of
collaboration -> use by the
ecosystem in contradiction to regular
test management tools
 Build from scratches to reflect certain
workflow typical for the Symbian
community
 Ad-hoc testing instead of test plans
and sessions – test when you need
and when you can
 Open for everybody
 The tool is open source
 Results are freely available

19
Collaborative Test Database Demo

20
 THE FUTURE OF
COLLABORATIVE TEST

21
“1 person-day per month”
 Each test working group member should if possible allow one
person to run tests for one day, each month, in order to gather
results
 Not mandatory
 ... but each member can offer to contribute more
 Effort project-managed through the Foundation, then
transition to CTD for:
 Organize the test case list
 Dispatch test cases to avoid redundant results
 Follow-up on execution and bugs raised
 Gather and collate test results
 Follow-up with the bug squad
 This is an effort that is being asked jointly and regardless of
company size
 But this effort is small when compared to the benefit it can
bring to the Symbian platform and the Symbian Community

22
WHAT NEXT???

 CTD will continue to evolve with new features in the

pipeline
 Collaborative test is a concept that can be applied to any
open source community
 Open information sharing
 Provides standard test specifications and tools
 CTD offers a centralized collection point for results
 Hopefully this is bound to continue in the new world
organization as announced this week

23
 Thank You
Fred Blesser Mariusz Lasek
[email protected] [email protected]

24