Proceedings of the 2nd International Conference on Computer Science and Electronics Engineering (ICCSEE 2013)

Application of Orthogonal Experimental Design for the Automatic Software Testing

Hao Wu
Department of Computer Science
Zhuhai College of Jilin University, JLUZH
Zhuhai, China
e-mail: [email protected]

Abstract—According to inputting different combination of in the lower cost and the lower risk. The ultimate aim is to
conditions so as to produce different impacts, software testing effectively improve testing results and the efficiency of
designs a large number of test cases. If the implementation of software testing.
an overall test, due to the limit of the combination of conditions,
it is difficult to carry out. In order to generate high quality test II. MAIN METHODOLOGY
cases as early as possible to improve the efficiency of software
testing, it is designed a generation tool of the automatic A. Test Case Design
software testing case on orthogonal experimental design. For
The design of tests for software and other engineered
the test data, the use of that tool design test cases. The practice
products can be as challenging as the initial design of the
shows that a small number of test cases are generated, the
error detection ability is strong, and it greatly improves the product itself. A rich variety of test case design methods
efficiency of software testing. have evolved for software. These methods provide the
developer with a systematic approach to testing. More
Keywords- orthogonal experimental design; test case; important, methods provide a mechanism that can help to
software testing ensure the completeness of tests and provide the highest
likelihood for uncovering errors in software.
I. INTRODUCTION B. Orthogonal Experimental Design
Along with the more powerful software function and the 1) O
improvement of software complexity, software development rthogonal Experimental Design Concepts
process is not easy to be controlled. Software quality Orthogonal experimental design is an important branch
problem has become an important factor to influence the of statistical mathematics, based on the probability theory,
development of computer application technology. Software the mathematical statistics, and the standardized orthogonal
testing is a critical element of software quality assurance and table to arrange the test plan. According to the modern
represents the ultimate review of specification, design, and algebra of Galois theory, Orthogonal experimental design is
code generation. The design of tests for software and other a scientific test design method, which selects the right
engineered products can be as challenging as the initial amount of representative points or uses cases from a large
design of the product itself. A rich variety of test case design number of experimental data, so as to arrange experiments or
methods have evolved for software. tests reasonably.
There are many applications in which the input domain is Orthogonal experimental design usually determines the
relatively limited. That is, the number of input parameters is standard of test results pros and cons, which is called
small and the values that each of the parameters may take are indicators that may affect tests as factors, and the impact
clearly bounded. When these numbers are very small, it is factor as factor levels. In designing the optimal test, it must
possible to consider every input permutation and have a reasonable indicator and a reasonable reference to
exhaustively text processing of the input domain. However, selecting factor and the corresponding level, in order to
as the number of input values grows and the number of complete the specific test purpose.
discrete values for each data item increases, exhaustive 2) O
testing becomes impractical or impossible. rthogonal Experimental Design Characteristics
Orthogonal experimental design can be applied to Orthogonal experimental design is the study of multi-
problems in which the input domain is relatively small but factor and level of design method, through the part of the test
too large to accommodate exhaustive testing. The orthogonal to find out the optimal level combination. To complete test
experimental design is particularly useful in finding errors requirements needed less number of experiment. The greater
associated with region faults-an error category associated the number of factors and levels, the more obvious the
with faulty logic within a software component. advantage of this method is. The distribution of data points is
Based on the research of orthogonal experimental design uniform.
method, it considers that to be applied to design test cases. It is available to use corresponding range analysis method,
Through designing an automatically generated tool to variance analysis method and regression analysis method to
generate test cases, it can be effective in reducing the number analyze the test results, and can lead to many valuable
of test cases, so as to ensure accomplishing software testing

Published by Atlantis Press, Paris, France.

© the authors
2298
 Proceedings of the 2nd International Conference on Computer Science and Electronics Engineering (ICCSEE 2013)

conclusions, better reliability. Orthogonal experimental many orthogonal table structure, and so far that have not
design, base on the orthogonal table, is efficiency, rapid and been resolved.
economic test design method. 2) O
C. Orthogonal Table rthogonal Table Nature
• The frequency of different digital in each column is
1) O equal, such as L4(23) that there are two different
rthogonal Table Concepts figures in each column, and which appears twice.
Orthogonal table is a kind of two-dimensional digital • If two figures are seen as the ordered pairs in any
form with the neat comparability and the balanced two lines, and the occurrence number of each
collocation. The neat comparability means that each factor of ordered pairs appears equal. For example, there are
each level is completely the same in the same piece of four ordered numbers of L4(23) and each of them
orthogonal table. Since the test of each factor of each level appear once.
and other factors involved in the probability of each level test Due to the orthogonal nature of the table to arrange the
is exactly the same, it ensures that each factor level is ruled experiment, the various levels of each factor are collocation
out other factors level of interference in the greatest degree. balanced, and the experimental number is small, but there is
Therefore, it can be most effective in comparison and a strong representation. The intuitive analysis, the variance
prospect, and be easy to find a good test condition. The analysis and effect analysis can all analyze the size of the
balanced collocation means that any two columns (two impact of various factors on the experiment results, and to
factors) level collocation (horizontal formation of digital for) determine the best experiment plan. Effect analysis can also
is exactly the same in the same piece of orthogonal table. estimate the trend to the index values, and it can guide the
This helps to ensure that the test condition is balanced experiment process conversely.
dispersion in the level of combination of factors completely, 3) O
thus which has a strong representation, and is easy to get rthogonal Experimental Design for Test Cases Strategy
good test conditions. By orthogonal experimental design for test cases strategy,
The orthogonal table is the foundation of the orthogonal there are simple designs and also complex ones. In the real
experimental design, which forms as follow: work, it often encounters very complicated situations,
because of the increase of the system scale.
Ln（mk） (1) There are two key strategies on designing the orthogonal
experiment:
The L is the symbol of the orthogonal table, and its a) To Determine Factors
symbol is expressed as follows: That all may affect the program result should be taken
• n: The number of trials arranged by the orthogonal into account. That factors are not interactive with others can
table. That is the number of rows in the orthogonal be added after the formation of use cases.
table in directly corresponding to the number of b) To Determine Levels
production test cases. The limited value level is taken all. The infinite value
• m: The most number of factors arranged in the level is determined by the boundary value analysis, the
orthogonal table. That is the number of columns in equivalence class analysis and the error determination
the orthogonal table in directly corresponding to the method.
input parameters in the software module.
• k: The each factor level number is to get the largest III. ORTHOGONAL EXPERIMENTAL TESTING CASE DESIGN
number in each single factor. The orthogonal table To design software test cases by orthogonal experimental
contains values from 0 to the level number - 1 or design can greatly reduce the number of test cases, so as to
from 1 to the level number, and each factor level improve the efficiency of software testing. That is the most
values is corresponding to the input parameter. suitable for the software module that a number of input
In the process of the orthogonal experimental design, it parameters are together to determine the output results. As
can make use of the existing orthogonal table, and also use a the software structure illustrated in Figure 1, three input
mathematical model to generate an orthogonal table. The parameters like 1, 2 and 3 together determine the result of the
structure of the orthogonal table mainly uses the theory of output parameter 1, and three input parameters like 3, 4 and
the abstract algebra, and there are many existing 5 together determine the result of the output parameter 2,
standardized orthogonal tables, such as L4(23) 、 L8(27) 、 then to choose the orthogonal table should be respectively
L9(34) 、 L16(45) , etc. Using these orthogonal tables, the suitable for three factors as 1, 2, 3 and 3, 4, 5.
general experimental design can be completed. But for some
experiments, due to the factor and the level of the testing
modules are quite complex, it may not be able to select the
proper one from these orthogonal tables. Then it should use a
mathematical model to construct a new orthogonal table,
involving more abstract algebra knowledge and as well as

Published by Atlantis Press, Paris, France.

© the authors
2299
 Proceedings of the 2nd International Conference on Computer Science and Electronics Engineering (ICCSEE 2013)

Figure 1. Software modules of Orthogonal Array to Design Test Cases.

A. Test Data Description Figure 2. L9(34) orthogonal array

It illustrates the orthogonal experimental design to design To illustrate the use of the L9(34) orthogonal array,
test cases as example that it sets input interface on the consider the send function for a fax application. Four
software query module of report issued in the financial parameters, P1, P2, P3, P4, are passed to the send function.
management and control project. For example, adding a Each takes on three discrete values.
query can filter out the corresponding budget statement in Given the relatively small number of input parameters
the units of the issued report, through setting the input and discrete values, exhaustive testing is possible. The
parameters on this interface, which together determine the number of tests required is 34=81, large, but manageable. All
budget statement. faults associated with data item permutation would be found,
The input parameters are as follows: but the effort required is relatively high.
• Budget cycle: Annual budget, Month budget The orthogonal array testing approach enables us to
• Budget time: 2010、2011、201008、201108 provide good test coverage with far fewer test cases than the
• Budget classification: Sale power budget, Purchase exhaustive strategy. An L9(34) orthogonal array for the fax
power budget, Transportation power budget. send function is illustrated in Table Ⅰ.
• Unit: A provincial grid company, a city, State grid.
• Report/Issue: Reported query, Issued query. TABLE I. L9(34) ORTHOGONAL ARRAY
• Reported/Issued number: Reported number, Issued Test parameters
number. Test case (m)
(n)
B. Argumentation of Selection Orthogonal Experimental P1 P2 P3 P4
Design 1 1 1 1 1
1) C
2 1 2 2 2
omprehensive Test
Comprehensive testing is to put all the parameters and 3 1 3 3 3
domains into one test collocation. For example, the test of 4
factors and 3 levels is needed to do testing use cases of 3 of 4 4 2 1 2 3
times square equal to 81. 5 2 2 3 1
The Advantage of the comprehensive test is that it can
achieve the comprehensiveness of the test, analyze the effect 6 2 3 1 2
of various factors and the interaction, and also choose the
7 3 1 3 2
optimal level combination. However, the disadvantage of
this approach is to require the test too many times and the 8 3 2 1 3
workload is relatively large. For the test of m factors and n
levels, the total number of test cases is n of m Times Square, 9 3 3 2 1
which is difficult to fully achieve in the actual test work in Because of the balanced collocation, a factor of any
some cases. levels and other ones are combined with once, and the only
2) O
once. It can be conductive to analyze the testing results on
rthogonal Eerimental Dsign account of that.
When orthogonal experimental testing occurs, an L9(34)
orthogonal array of test cases is created. The L9(34) C. Designing Test Cases on Orthogonal Experimental
orthogonal array has a balancing property. That is, test cases Method
are dispersed uniformly throughout the test domain, as According to the above analysis, it selects the orthogonal
illustrated in Figure 2. Test coverage across the input domain experiment scheme. It designs the test as long as each of the
is more complete. investigated factors are optionally corresponding to the
orthogonal table column, and then the number of each

Published by Atlantis Press, Paris, France.

© the authors
2300
 Proceedings of the 2nd International Conference on Computer Science and Electronics Engineering (ICCSEE 2013)

column is corresponding to that of one-to-one factor levels. Monthly

Transmission T grid
Issued
In this way, the combination of each line of each level 6 A2B3C1D2 electricity provincial
budget number
budget company
constitutes a test condition.
Sale
According to the above analysis, the first step is to 7 A3B1C3D2
Annual
electricity State grid
Issued
determine the factor. Given that there are total 6 factors in budget number
budget
this module, there are 2, 3, 4 kinds of level. Based on the Purchase T grid
Monthly Reported
principle to select the orthogonal table, to select an 8 A3B2C1D3
budget
electricity provincial
number
orthogonal table of 6-factor and 5-level, the number of test budget company
Transmission
cases generated is 25. It is clear that the number of test cases Annual t Grid Reported
9 A3B3C2D1 electricity
generated by this table is very large, and then it should cut budget
budget
city number
factors and corresponding levels. The analysis of the above process can get 9 test cases,
According to the product requirement specification, it can which can cover the critical points of the demand
analysis the correlation between various factors. Because the substantially.
relevance of the budget time and the budget cycle is close, it
can cut out the factor of the budget time. Similarly the IV. AUTOMATIC GENERATION TOOL ON THE
correlation of the reported / issued factor and that of number ORTHOGONAL TABLE TEST CASE
is also very close, and then can also be cut out. Thus the
There are many advantages in use of the orthogonal
input parameters remain four, which are as following:
experiment to design test cases. For example, that can save
• Budget cycle: Annual budget, Monthly budget
test time and control the number of test cases generated, and
• Budget classification: Sale electricity budget, which coverage is good. But in process of designing
Purchase electricity budget, Transmission electricity orthogonal experimental test cases, to find the orthogonal
budget table is very troublesome, so it leads to selecting the
• Unit: T grid provincial company, t Grid city, State orthogonal table with less factor levels. That problem can not
grid be resolved in the artificial stage. In the actual testing work,
• Reported / issued number: Reported number, Issued it is to use an easier orthogonal table. If it is lack of an
number automatic method, the application of the orthogonal
According to the above analysis conditions, to determine experimental design on test cases is a very complicated and
four factors, that is the budget cycle, the budget classification, difficult work, so it is necessary to study and improve the
the unit and the reported / issued number, as well as its method in using process carefully.
corresponding level of 2, 3, 3 and 2. The number of
experiments is not less than 3 × (3-1) +1 = 7 (s), therefore, it A. Code Framework
can be considered to use the orthogonal table L9(34) with 4 This is the part of code framework which is designed
factors and 3 levels. The factor A, B or C can be orthogonal table test cases strategy to the automatic
corresponding to the orthogonal table L9(34) in any three generated tool.
columns arbitrarily, for example, A, B and C are placed in / / To define a class of orthogonal table:
1,2 and 3 column, and then test them line by line and in public class UniFormTable
unlimited order. Mapping the determined factors and levels {
with the orthogonal table L9(34), and filling null value, it can private int m_Runs;
draw a set of test case table, such as shown in table 3. / / Rows number is that of rows of the orthogonal table,
i.e. the one of tests.
TABLE II. THE ORTHOGONAL TABLE CORRESPONDING TO TEST private int m_FactorLevelCount;
PROGRAM - THE TEST CASE TABLE
// If it equals 1, that is the the level orthogonal table
Test condition
private int[] m_Factors;
Test // The factor number is that of columns in the orthogonal
Horizontal Budget Reported
case
combination
Budget
classification Unit / issued
table.
cycle private int[] m_Levels;
number
Sale T grid // The level number is that any single factor can get the
Annual Reported
1 A1B1C1D1
budget
electricity provincial
number maximum value.
budget company private string[] m_TableMatrix;
Purchase // The data of the orthogonal matrix table.
Annual t Grid Issued
2 A1B2C2D2 electricity
budget city number private string[] m_TableMatrixString;
budget
Transmission // The block of orthogonal matrix table data.
Annual Reported ……
3 A1B3C3D3 electricity State grid
budget number //Case1: The factors number and the levels number are
budget
Monthly
Sale
t Grid Issued
the same.
4 A2B1C2D3 electricity for (int i = 0; i < UniFormTableList.Count; i++)
budget city number
budget {
Purchase if (UniFormTableList[i].FactorLevelCount == 1)
Monthly Reported
5 A2B2C3D1 electricity State grid
budget number
budget

Published by Atlantis Press, Paris, France.

© the authors
2301
 Proceedings of the 2nd International Conference on Computer Science and Electronics Engineering (ICCSEE 2013)

{ //To find the levels and factors number in the topic in software testing. The Orthogonal experimental
orthogonal table is equals to the specified one. design is a good solution to such problems.
if (UniFormTableList[i].Factors[0] == MyFactors.Count) Using the Orthogonal experimental design to design test
…… cases is the effective means to reduce test cases, thereby
// Case2: The levels number is equal, but factors number improving the efficiency of software testing. This method is
is not equal. most suitable for the software module that contains a
List<UniFormTable> FitTableList = new List<UniFormTable>(); plurality of inputting parameters to determine one outputting
for (int i = 0; i < UniFormTableList.Count; i++) results together. It studies the designing scheme of the
{ orthogonal experimental design to design test cases, and puts
if (UniFormTableList[i].FactorLevelCount == 1) forward the solving strategies on multi-factor levels. To
{ select the right and strong representative points from a large
if (UniFormTableList[i].Levels[0] == number of test cases, which are more comprehensive and
MyFactors[0].Levels.Count) more objective understanding comprehensive tests, and to
{ // To find that is more than or equal to the specified select the most optimal level combinations. This method can
factors number and is the most close to the standard avoid one-sidedness and blindness testing, thereby can
orthogonal table. improve the efficiency of the software testing and can reduce
…… the cost of it. Practice has proved that such orthogonal
// Case3: Each factors number is not the same. experimental design which is “equilibrium dispersion and
for (int i = 0; i < UniFormTableList.Count; i++) neat comparable” is a kind of multi-factor testing and
{ effective method. If it will put together a number of test
if (UniFormTableList[i].FactorLevelCount ==
cases designing techniques, the effect would be better.
FactorLevelPair.Count)
With the advancement of technology, the software scale
……
is much larger and the complexity increases more and much
// Outputting test cases:
for (int aa = 0; aa < TestCaseTable.Runs; aa++)
higher, which bring the new challenges to testers that must
{ master the scientific test cases designing method to apply to
for (int bb = 0; bb < MyFactors.Count; bb++) the software testing in the actual work, and the orthogonal
{ test method is the best choice by testers undoubtedly.
MapOutPut.AppendText(TestCaseTable.TableMatrix[aa,
bb].Trim() + " "); ACKNOWLEDGMENT
} Thank Zhuhai College of Jilin University for supporting
MapOutPut.AppendText("\n"); the Focus Professional Development Funds.
}
REFERENCES
B. Automation Tools Generation
[1] Dashan SUI and Zhenshan CUI, “Application of orthogonal
It realizes a software, develops a tool of the research and experimental design and Tikhonov regularization method for the
analyzes on the orthogonal experimental to design test cases. identification of parameters in the casting solidification process”,
The software provides a solution strategy of three Acta Metall. Sin.(Engl. Lett..)Vol.22 No.1 pp13-21 Feb..2009.
orthogonal experimental design designing test cases that is [2] Fu, Jianping; Lu Minyan; Ruan, Lian; Huang, Baiqiao; “Application
of experimental design in software reliability test ” , Beijing
the same as the factor number and the level number, not the Hangkong Hangtian Daxue Xuebao, vol.34, pp. 1379-1383,
same as one. That realization principle is to read a file December 2008.
contains a number of orthogonal tables. When the program [3] Lan, B; Feng, P.F. ; Wu, Z. J. ; Yu, D. W. “Determination of
starts, it loads the orthogonal table file and analyses each constitutive equation parameters for orthogonal cutting through
orthogonal tables, and then finds out factors number and pressure bar tests and FEA method”, Ken Engineering Materials,
levels number and stores to the orthogonal table object. To v499, p56-61, 2012, Anti-Fatigue Deaign and Manufacturing
Technologies.
explain the inputting contents and to analyses factors and
levels, it can find the matching orthogonal table and product [4] Chan, W.K.; Cheng, M.Y.; Cheung, S.C. ;Tse, T.H. “Automatic goal-
oriented classification of failure behaviors for testing XML-based
the corresponding test cases. multimedia software applications: An experimental case study”,
The main function of the software is to input the factor Journal of Systems and Software, vol 79, pp.602-612, May 2008.
and the corresponding level, and can automatically analyze [5] Qiao, Wen-Feng; Zhou, Yong-Min; “Optimizing technological
the suitable orthogonal table to product the corresponding parameters of preparation of Bi nanoparticles by orthogonal
test cases. experimental method”, Lubrication Engineering, September 2008..
[6] Zhou, De-Jian; Huang, Hong-Yan; Peng, Kai-Qiang; “Study on
V. CONCLUSION evaluation method for the heat characteristics of IC devices based on
the combination of orthogonal experimental method and response
Software testing need analysis and design test cases from surface method”, Proceedings of International Symposium on High
different angles, and thus test the system effectively and Density Packaging and Microsystem Integration 2007, HDP'07.
scientifically. However, due to the limitation of the testing [7] Zhang, Qingpia; Zhou, Kan; Gui, Yingchun; Yu, Chengda; Sun, Ning;
time and resources, it is impossible to test the system Li, Shizhong; “Simulation research of laminated solid armature based
completely that is limited to test. That how to distribute on orthogonal experimental method”, 2011 2nd International
limited resources to the system scientifically is an important

Published by Atlantis Press, Paris, France.

© the authors
2302
 Proceedings of the 2nd International Conference on Computer Science and Electronics Engineering (ICCSEE 2013)

Conference on Mechanic Automation and Control Engineering,

MACE 2011 – Proceedings, 1616-1618.

Published by Atlantis Press, Paris, France.

© the authors
2303