Software Quality

and Testing in MySQL

Trim Pershad
Omer BarNir

MySQL – User Conference and Expo 2009

 Topics

• Overview of QA
• Process
• Testing
• Bug Analysis and Metrics

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Overview of QA
• MySQL products
– Server, Cluster, ET, Developer Tools, Connectors
– We will talk mostly about server
• QA over the years
• QA group structure
• Continuously improving quality

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Overview of QA (cont)

• Internal Processes
– Well defined release criteria
– High quality is the goal
• All releases are tested by QA
• New Tests added all the time
• Build most test tools internally

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Topics

• QA Background in MySQL
• Process
• Testing
• Bug Analysis and Metrics

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Process ..Development Cycle
• Development
– Specifications – Worklogs
– Design reviews
– Code reviews
– Unit tests
– Team Trees and Main Trees
• QA
– Test Plans
– Testing – functional, performance, platforms
– Code coverage
– Platforms

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Process ..Release Cycle

• Build
– Clone off
– Build binaries
– Packaging
• QA
– Run Install Tests
– Run Functional Tests
– System and Performance Testing
• Development
– Bug Fixes

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Process ..Release Cycle
Build Steps QA Steps Development Steps

Code
Clone Install
off Merge
Tests Process

Release Functional Bugs Bug Fix

Binaries Tests Analyzed
Built

Package System
/Stress
Binaries
Tests

Binaries Performance
Released Tests

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Topics
• QA Background in MySQL
• Process
• Testing
• Bug Analysis and Metrics

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Testing
• Testing frameworks
• Testing suites
• Cross Product Integration Testing

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Testing - Frameworks
• MTR (mysql-test-run)
– Single user test driver (based on ‘test’ files)
– Used for Functional tests
– Distributed with the MySQL Server

• mysql-stress-test
– Multi connection scenarios (based on ‘test’ files)
– Uses for system/concurrency tests
– Distributed with the MySQL Server (framework only)

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Testing – Frameworks (cont.)
• Systems Test Framework
– Multi connection scenarios (Based on mysql-stress-test)
– Concurrency and Longevity Testing
– Not Distributed

• Random Query Generator

– Test Generator
– Single/multi-user test driver (based on ‘grammar’ files)
– Used for Functional, Compatibility and System tests
– Separate Tree

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Testing – Frameworks (cont.)

• Upgrade-downgrade
– Functional scenarios
– Uses for upgrade, downgrade
– Not Distributed

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Testing – Frameworks, Performance
• Sysbench
• C Based application, Multi threaded connections
• Built in scenarios: OLTP, Memory, CPU, threads
• Supports Lua extensions (v0.5.0)
• Public Tree

• MySQLBench
• Single user ‘atomic’ test scenarios
• Perl Based
• Included in the mysql distribution

• DBT2 Tests
– Based on TPC-C benchmark
MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Testing
• Testing frameworks
• Testing suites
• Cross Product Integration Testing

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Test Suites
• Functional
– Functional Test suite
– NIST compliance tests
– Large table tests
– Random/Complex query testing
– Upgrade/Downgrade Compatibility

• System/Longevity
– Concurrent IUDS scenarios
– Replications
– Crash/Recovery tests

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Upgrade/Downgrade Compatibility
– Start Old Server
– Run test script (init_xxx.test)
– Shut down old server start new one
– Run upgrade process (*)
– Run test script (upgrade_xxx.test)
– Shut down new server, start old one
– Run test script (downgrade_xxx.test)
– Shutdown server

– (*) Different upgrade path

• Live (same database)
• Dump/restore
• Replication (old master new slave)

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Concurrent IUDS scenarios
• Start Server
• Run script initializing tables/users/Procedures etc.
• Launch multiple clients,
– Continuously run until the end of the test
– Each running a random scenario and exists
• Runs tasks
– ‘management’ activities (truncate log tables etc)

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Concurrent IUDS scenarios - Example
#
# This test is deleting rows based on a random value
#
--source suite/systems/include/system_1_init.inc
#
let $NUM_VAL=`SELECT @NUM_VAL`;
USE systest1; # This test is updating rows based on a random value
#
# Setting parameter intormation --source suite/systems/include/system_1_init.inc
--replace_result $NUM_VAL <NUM_VAL>
eval set @f1_nums=$NUM_VAL; let $NUM_VAL=`SELECT @NUM_VAL`;
set @tmp_num=int_rand(@f1_nums); use systest1;

delete from tb0_eng1 where f1=@tmp_num; # Setting parameter intormation

--replace_result $NUM_VAL <NUM_VAL>
#
eval set @f1_nums=$NUM_VAL;
# This test is inserts rows based on a random value
set @tmp_num=int_rand(@f1_nums);
# using a stored procedure
set @tmp_word=str_rand(4);
--source suite/systems/include/system_1_init.inc
let $NUM_VAL=`SELECT @NUM_VAL`;
update tb0_eng1
USE systest1;
set f2=concat('U:',@tmp_word,'-',@tmp_num), f3=f3+1
where f1=@tmp_num-1;
# Setting parameter intormation
--replace_result $NUM_VAL <NUM_VAL>
eval SET @f1_nums=$NUM_VAL;
SET @tmp_num=int_rand(@f1_nums);
SET @tmp_word=str_rand(4);

CALL ins_tb0_eng1 (@tmp_num, @tmp_word);

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Concurrent IUDS scenarios – Example (cont.)
# This procedure scans 'tb0_master' table for rows where f1 = 'num_pr'
# and for each row INSERTs a row in 'tb0_eng1'
#---------------------------------------------------------------------
CREATE PROCEDURE ins_tb0_eng1 (num_pr INT, str_pr CHAR(15))
BEGIN
DECLARE done INT DEFAULT 0;
DECLARE v3 DECIMAL(5,3);
DECLARE cur1 CURSOR FOR
SELECT f3 FROM tb0_master WHERE f1 = num_pr;
DECLARE CONTINUE HANDLER FOR SQLSTATE '01000' SET done = 1;
DECLARE CONTINUE HANDLER FOR SQLSTATE '02000' SET done = 1;
OPEN cur1; #
FETCH cur1 INTO v3; # This test is inserts rows based on a random value
wl_loop: WHILE NOT done DO # using a stored procedure
INSERT INTO tb0_eng1 (f1, f2, f3, f4) VALUES CREATE TRIGGER tb0_eng1_ins AFTER INSERT ON tb0_eng1 FOR EACH ROW
(int_rand(@f1_nums), INSERT INTO tb0_logs (dt1, entry_dsc, f4)
CONCAT('I:',str_pr,'-',num_pr), VALUES (NOW(), CONCAT('Insert row ', NEW.f1,' ',
v3, NOW()); NEW.f2, ' ', NEW.f3, ' (tb0_eng1)'), NEW.f1);
FETCH cur1 INTO v3;
END WHILE wl_loop; CREATE TRIGGER tb0_eng1_upd AFTER UPDATE ON tb0_eng1 FOR EACH
ROW
CLOSE cur1; INSERT INTO tb0_logs (dt1, entry_dsc, f4)
END// VALUES (NOW(), CONCAT('Update row ', OLD.f1,' ', OLD.f2, '->',
NEW.f2, ' ', OLD.f3, '->', NEW.f3, ' (tb0_eng1)'), NEW.f1);

CREATE TRIGGER tb0_eng1_del AFTER DELETE ON tb0_eng1 FOR EACH ROW

INSERT INTO tb0_logs (dt1, entry_dsc, f4)
VALUES (NOW(), CONCAT('Delete row ', OLD.f1,' ', OLD.f2, ' ',
OLD.f3, ' (tb0_eng1)'), OLD.f1);

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Test Suites (cont.)
• Install Testing

• Load/Stress
– Testing the system under high user load with connect/disconnect
and IUDS operations

• Performance
– Performance testing of SQL operations
– Performance trend and changes with changing number of
threads

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Test Suites – Performance Charts

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Test Statistics
Test Statistics - Release 5.0 Vs 5.1

8000
7000 7,273
6000
# of Tests

5000
4000
3000 3,604

2000
1000
0
5.0 5.1
Server Release

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Testing
• Testing frameworks
• Testing suites
• Cross Product Integration Testing

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Cross-Product Integration Testing
• Server Testing
– Based on using client tools based on the C-API Interface

• Connector Testing
– Systematic testing of connectors
– Testing using sample applications built with common tools
• Microsoft Access
• Crystal Reports

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Topics
• QA Background in MySQL
• Process
• Testing
• Bug Analysis and Metrics

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Metrics and Bug Analysis

• Bug KPI Tracking

– Used in making Quality
decisions
v5.1 P1 Inflow/Outflow

• Bug Analysis 20
Inflow
Outflw
Series3

16

– Bug Analysis 15

– Prioritization of bugs 10

• (cross team activity)

5 4 4 4
3 3 3 3 3
2 22
1 1 1 1 1 1 1 1
0 0 0 00 0 0
0

9-Jan

16-Jan

23-Jan

30-Jan

6-Feb

13-Feb

20-Feb

27-Feb

1-May
6-Mar

13-Mar

20-Mar

27-Mar

3-Apr

10-Apr

17-Apr

24-Apr
MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Bug LifeCycle
Open

Verification Triage
Approved QA Testing

Dev & QA
In Progress Review
Testing

Documenting

Rejected
Closed
Before
Development QA Resolution
Engineering

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Metrics and Charts

5.1 Targets
Server
400
375
350
325 329 328
311 310 306
300
288 283 284
275 276 274
250 242
234 240 241
225 226
200
175
150
125
100
75
50
25
0
2-Jan

9-Jan

16-Jan

23-Jan

30-Jan

6-Feb

13-Feb

20-Feb

27-Feb

6-Mar

13-Mar

20-Mar

27-Mar

1-May
3-Apr

10-Apr

17-Apr

24-Apr
MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Metrics and Charts

v5.1 P1 Inflow/Outflow
Inflow Outflw

20

16

15

10

5 4 4 4
3 3 3 3 3
2 22
1 1 1 1 1 1 1 1
0 0 0 00 0 0
0
9-Jan

16-Jan

23-Jan

30-Jan

6-Feb

13-Feb

20-Feb

27-Feb

6-Mar

13-Mar

20-Mar

27-Mar

1-May
3-Apr

10-Apr

17-Apr

24-Apr
MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL
 Questions?

Contact Information
• Trim Pershad [email protected]
• Omer BarNir [email protected]

MySQL – User Conference and Expo 2009 – Software Quality and Testing in MySQL