See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/324133319

ABC Algorithm for Combinatorial Testing Problem

Article · October 2017

CITATIONS READS

7 134

8 authors, including:

AbdulRahman A. Alsewari Ammar K. Alazzawi

Universiti Malaysia Pahang Universiti Teknologi PETRONAS
62 PUBLICATIONS   499 CITATIONS    14 PUBLICATIONS   31 CITATIONS   

SEE PROFILE SEE PROFILE

Taha Rassem Muhammad Nomani Kabir

Universiti Malaysia Pahang Universiti Malaysia Pahang
48 PUBLICATIONS   281 CITATIONS    92 PUBLICATIONS   605 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

A NEW MASKING MODEL IN IMAGE AND VIDEO PROCESSING View project

ABCVS: An Artificial Bee Colony for Generating Variable T-Way Test Sets View project

All content following this page was uploaded by Yazan A. Alsariera on 31 March 2018.

The user has requested enhancement of the downloaded file.

ABC Algorithm for Combinatorial Testing Problem
AbdulRahman A. Alsewari1,2, Amaar K. Alazzawi1, Taha H. Rassem1, Muhammad N. Kabir1,
Ameen A. Ba Homaid1, Yazan A. Alsariera1, Nasser M. Tairan3, and Kamal Z. Zamli1
Software Engineering Research Group,
Faculty of Computer Systems and Software Engineering,
1
2
IBM Centre of Excellence, Universiti Malaysia Pahang, Pahang, Malaysia.
3
College of Computer Science, King Khaled University.
[email protected]

Abstract—Computer software is in high demand everywhere case requires five minutes, and it will take a whole 15 months
in the world. The high dependence on software makes software to examine only the display tab completely which is not
requirements more complicated. As a result, software testing probable practically according to the testing standards.
tasks get costlier and challenging due to a large number of test
cases, coupled with the vast number of the system requirements.
This challenge presents the need for reduction of the system
redundant test cases. A combinatorial testing approach gives an
intended result from the optimization of the system test cases.
Hence, this study implements a combinatorial testing strategy
called Artificial Bee Colony Test Generation (ABC-TG) that
helps to get rid of some of the current combinatorial testing
strategies. Results obtained from the ABC-TG were
benchmarked with the results obtained from existing strategies
in order to determine the efficiency of the ABC-TG. Finally,
ABC-TG shows the efficiency and effectiveness in terms of
generating optimum test cases size of some of the case studies
and a comparable result with the existing combinatorial testing
strategies.

Index Terms—Computational Intelligence; Combinatorial

Optimization Problem; Software Testing; Test Data
Generation.

I. INTRODUCTION
Figure. 1: Microsoft Word, Options, Customize Ribbon
Software systems continue to develop progressively in
complexity and size in this era. Software has become Exhaustive testing is impossible. Thus, there are a lot of
gradually ubiquitous in tools and methods used for science, strategies have been designed and developed to minimize the
engineering, medicine and human interactions. A software test cases based on optimization algorithms Genetic
fault is a mistake in the programmed code which, when faced Algorithm (GA) [8], Simulated Annulling (SA) [5, 9], and
may be the reason for the software’s failure. The software Harmony Search Strategy (HSS) [1, 2]. The first approach
behaves in an unexpected way when it encounters faults in it. that used to minimize the test cases is the Pairwise Testing
Different methods are implemented to avoid, notice and approach. Pairwise testing approach is that used to generate
rectify the errors throughout the software design life cycle test cases based on all possible pairs of system’s input values.
phases. Thus, software testing is a fundamental activity in Pairwise testing is the basic level of combinatorial testing
securing the quality assurance of most software products [1- approach. Combinatorial testing approach is the test suites
3]. It is the utmost key in guaranteeing a reliable software generator that covers all combinations of the system
product. In software development life cycle, software testing parameters based on the combination degree. Thus, it is much
acts as an integral and tedious activity [4]. smaller than exhaustive ones yet still very effective in finding
The presence of faults in a software system can result into defects [10, 11]. However, one of the main complications of
unprecedented cost or even life losing [4]. Software testing Combinatorial is finding a minimal test suite.
takes a vital part in inspecting detects via probable test data To address this issue, many algorithms have been
to make sure it's quality. Most of the software systems of implemented such as: Automatic Efficient Test Generator
nowadays are produced using components. Most times, the (AETG) [10], GA [8], In Parameter Order (IPO) and its
system bugs or errors are as a result of the unexpected fusion family (IPOG-D) [12, 13], Test Configuration (TConfig)
between the components used [5-7]. For instance, if the [14], Pairwise Independent Combinatorial Testing
testing of Microsoft's words displays tab is considered in the (PICT)[15], Classification-Tree Editor eXtended Logics
dialog as seen in Figure 1. (CTE-XL)[16], Jenny [17], ITCH[18], Test Vactor Generator
It has seventeen feasible options (parameters P=17) that (TVG) [19], SA [5, 9], and HSS [1, 2]. It is observed that most
can take two probable values (V=2) 217= 131,072 are to be of them are not efficient and the existing strategies are based
analyzed. These are virtually ineffective. Analyzing a test on the optimization algorithms. The current strategies cannot

e-ISSN: 2289-8131 Vol. 9 No. 3-3 85

Journal of Telecommunication, Electronic and Computer Engineering

achieve the optimal balance between exploration and onlooker or employed bees [20]. Assuming D is the
exploitation. The aims of this paper is to present, design and optimization parameter number (System configuration
implement a new combinatorial testing strategy based on parameters), then every single solution test case ( ) (i =1, 2...
Artificial Bee Colony, called Artificial Bee Colony Test SN) basically will exist as a D-dimensional vector. By using
Generation (ABC-TG) strategy. Then benchmark the ABC- the Equation (1) produces all the initial test cases for
TG’s results with the existing combinatorial testing employed bees.
strategies’ results to evaluate the efficiency of the ABC-TG
strategy. 𝑥𝑖𝑗 = 𝑥𝑚𝑖𝑛,𝑗 + rand(0,1)( 𝑥𝑚𝑎𝑥,𝑗 - 𝑥𝑚𝑖𝑛,𝑗 ) (1)
II. TEST CASES GENERATION ALGORITHM BASED ON ABC- where the value of xmin and xmax are sequentially upper &
TG STRATEGY lower limits for the test case variable xi in dimension j (j=1,
2… D), and rand is a random digit number scaling factor
ABC-TG strategy has been applied to solve numerous test which is between the number [0, 1]. D-dimensional test cases
cases optimization problems. The position of the food (food provenance positions) created during the initialization
provenance discovered, represents a possible test case of the stage (C=0) are subject to the cycles of Iterative (C=1, 2…,
optimization problem, and the quality (fitness function) of the MCN), till a termination condition is satisfied and are
related test case, corresponds with a nectar amount (represent implemented locally and also as a global probabilistic
the combination pairs). selection/search in a one cycle ABC-TG. Each cycle depicts
The ABC-TG has three sets of bees; each of them is a total number of tasks made by the different types of bee.
working to accomplish a certain task. These sets are Thus, these whole methods are principally independent which
employed, onlookers, and scouts’ bees. In making the can be explicated in a separate form as follows, to have a
decision of selecting a food provenance, a set of bees must be better understanding of the ABC-TG methodology.
waiting on the dance area, this are called onlooker. The other
group which go to visit a food provenance are dubbed B. Employed Bee Phase
employed bee. The last set of bees are the scout bee, they Firstly, the phase of employed bees where creates a new
execute random work of discovering new province of food. candidate solutions (test cases) by evaluating the capability
The discovery of a food provenance represents a possible test of the test cases and interchanges the data with the onlooker
case to the optimization problem, and the nectar amount of a bees’ stage. And the employed bee creates (food position) a
food provenance corresponds to the quality [11]. candidate test case by the removal of the former (xij) test case
The first part of the algorithm consists of a few numbers of solution in its memory, by utilizing Equation (2) test case
employed artificial bees, while the second part of the only is updated [21].
algorithm has the onlooker bees. In each part of the algorithm,
the employed bees and the onlooker bees will represent the 𝑣𝑖𝑗 = 𝑥𝑖,𝑗 + 𝑟𝑎𝑛𝑑[−1,1](𝑥𝑖𝑗 − 𝑥𝑘𝑗 ) (2)
test cases in the population. Pass the ABC-TG strategy in four
phases respectively; initialization, employed, onlooker and
scout bee’s phases (see Figure 2). Here j{1,2,...,D} and k{1,2,..., SN} (k≠i) are randomly-
selected indexes, & rand exists as a random number of [-1,
1], which works as a scaling factor. It is clear that the
1: Generate the initial population test cases (xi) using
optimum test case is reached in the search area, this gets
Eq(1) , i= 1…SN
reduced because of the disorder in the solution. It evaluates
2: Evaluate the xi fitness (fi) of the population
3: Set cycle to 1 , and MCN=number
the fitness of the new solution by the employed bee, and it
4: Repeat updated the fitness values that is found, and replaces with the
5: for each employ bee { new test case instead of the former one in the employed bee’s
Produce new solution vi using Eq(2) memory (a greedy-selection).
Evaluate vi fitness (fi)
Select the best test case } C. Onlooker Bee Phase
6: Calculate the probability (pi) for test case (xi) using Eq(3) In ABC-TG algorithm, the principal task of every single
7: for each on looker bee{ onlooker bee is to indicate a food provenance (test case value)
Select a test case xi based on pi according to probability value, and depending on the fitness
Produce new solution vi using Eq(2)
value related to the food provenance Pi. This can be
Evaluate vi fitness (fi)
calculated using Equation (3).
Select the best test case }
8: If there is an abandoned test case for the scout
𝑓𝑖𝑡𝑛𝑒𝑠𝑠𝑖
Then replace it with a new test case produced randomly using Pi= ∑𝑠𝑛 (3)
Eq(1) 𝑛=1 𝑓𝑖𝑡𝑛𝑒𝑠𝑠𝑛
9: Memorize the best test case
10: cycle=cycle+1; Here, fit represents the value of fitness of a certain test case.
11 until cycle=MCN The probabilistic selection is implemented by making a
comparison of Pi against a randomly selected number which
Figure 2.ABC-TG Pseudo-Code
is between [0, 1]. The selection is approved if the created
random value is equal or less than Pi, and if otherwise, it will
A. Initialization Phase be rejected. Thus, the assignment of an onlooker bee to that
Initially, the ABC-TG algorithm starts producing randomly particular test case will be approved if the conforming
distributed population of Solutions size (SN) of test cases probabilistic selection is sanctioned. When evaluating the
(food provenance positions). Where SN shows the size of an new solution fitness, the new food provenance (test case) will

86 e-ISSN: 2289-8131 Vol. 9 No. 3-3

 Manuscript Title

be selected by an onlooker bee in the area of the former one times of running the experiments in the environment that is
which is in her memory, utilizing Equation 2. In case the new composed of PC with Windows 10 Pro 64-bit, Intel Core i7
test case has a fitness value that is better, then an onlooker 3.40 GHz, 4.00 GB of RAM. The time of implementation is
bee will make an update on the new test case that exists in her determined in seconds. The ABC-TG strategies are
memory and let go of the old one. This is related to the implemented in Java (JDK 1.8.0.31). The results are
employed bee case. presented in Tables, Table 2 and Table 3 compared with the
best results obtained by other existing strategies. The
D. Scout Bee Phase darkened cells with bold numbers represent the best results
In this phase, the scout bees work randomly to discover all obtained for the test configuration. The results for some
the search spaces to be able to get a new test case (enhanced) strategies are not available through the literature
to the problem of the global optimization. On the other hand, (publications), these cells are marked by NP (not published).
unlike the onlooker bees and employed bees (that have a limit Some strategies do not support certain interaction strength,
to produce a trial test case around the former test case), scout these cells are marked by NS (not supported). The
bees are indefinite in this sense. They adopt their samples configuration systems used in this evaluation details as:
from a wide range of D-dimensional vectors; so far remains Covering Array (CA), number of systems parameters (P), the
in the search space limits. Otherwise, cannot improve the values for each parameter represent by (V), and N is the
solution (test case) after a determined number of cycles. minimum size of test list. For example: CA(16, 2, 4 5). The
Formerly, if it is non-global, then it will abandon this test case minimum test list that can be produced N=16, interaction
and the employed bee will be employed to that particular testing degree t=2, system parameters P=5, each parameter
situation which will then be converted to a scout bee with has value V=4.
principally scout-type functionality. The process will To clarify the performance in terms of the support
continue until the exit criteria has been meet interaction strength (i.e. 2 ≤ t ≤ 6), the following experiments
adopted three configurations system from the published
III. BENCHMARKING results in the works by [1, 2, 5, 9, 12, 13].
In cases with high interaction strength within a
To evaluate the ABC-TG strategy, there will be several configuration system CA (N; t, 37) as shown in Table 1, ABC-
experiments collected from the publications [1, 2, 5, 9, 12, TG mostly provides an optimal result. However, the ABC-
13]. ABC-TG strategy has initialized its parameters such as: TG does not generate the most optimal results in all cases, but
the number of the improvisation = 80, number of bees = 50, it generates satisfactory results. TConfig, obtained best result
and the value of limit = 100 as suggested by the researchers when interaction strength is 6, ITCH obtained only one
[21]. The results obtained by ABC-TG strategy based on 20 optimum result when the interaction strength is 3.
Table 1.
CA (N; t, 37), t is variable from 2 to 6.

CTE- ABC-TG
T Jenny TConfig ITCH PICT TVG IPOG-D IPOG
XL Best B. time A. size A. time
2 16 15 15 16 15 16 18 17 15 19.3 15.5 19.5
3 51 55 45 51 55 54 63 57 49 266.2 50.8 270.6
4 169 166 216 168 167 NS NP 185 158 1918.2 161.3 1930.7
5 458 477 NS 452 464 NS 735 608 443 5741.0 451.3 5761.2
6 1087 921 NS 1015 1016 NS 1548 1281 945 4950.1 977.5 5119.06

Constrain wise, IPOG-D does not support for only one configuration is CA (N; 3, 36). While ITCH, and ABC-TG
configuration CA (N; 4, 37). However, Jenny, PICT, IPOG-D usually produce the near minimum and comparable results of
and IPOG commonly produce the worst results overall. On the final test cases. However, Jenny, Tconfig, PICT, TVG,
the other hand, when interaction strength is equal to 2, ABC- CTE-XL, IPOG-D and IPOG commonly produce the worst
TG generates the most optimal only once. results overall. On the other hand, ABC-TG generated
As shown in Table 2 with a configuration system CA (N; comparable results for two cases namely CA (N; 3, 3 9) CA
3, 3P), ABC-TG generates the most optimal minimum result. (N; 3, 310), Contrariwise other strategies that produce the
Comparing with the other strategies for only one worst results.
Table 2.
CA (N; 3, 3P), P is variable from 4 to 10.

CTE- ABC-TG
P Jenny TConfig ITCH PICT TVG IPOG-D IPOG
XL B b.time A.size A.time
4 34 32 27 34 34 34 27 39 33 6.82 34.5 7.16
5 40 40 45 43 41 43 49 43 40 28.33 41.9 28.9
6 51 48 45 48 49 52 49 53 43 90.8 46.8 94.9
7 51 55 45 51 55 54 63 57 50 264.2 52.0 268.2
8 58 58 45 59 60 63 63 63 54 654.2 55.8 663.2
9 62 64 75 63 64 66 71 65 58 1452.9 59.8 1470.1
10 65 68 75 65 68 71 71 68 62 3022.3 63.6 3041.5

Regarding to Table 3, ABC-TG generates the minimum test outperforms comparing with other strategies. As well as,
cases size and the satisfactory results for two configuration TConfig, ABC-TG, IPOG-D and ITCH generate the near
systems namely; CA (N; 3, 27) and CA (N; 3, 67) and optimal results in some cases and the competitive results in

e-ISSN: 2289-8131 Vol. 9 No. 3-3 87

Journal of Telecommunication, Electronic and Computer Engineering

the others. Although Jenny, PICT, TVG and CTE-XL does optimal results only once, and the acceptable results for other
not generate any most minimum result. But usually produces strategies.
comparable results, while IPOG-D generates the most
Table 3.
CA (N; 3, V7), V is variable from 2 to 6.

ABC-TG
V Jenny TConfig ITCH PICT TVG CTE-XL IPOG-D IPOG
B B.time A.size A. time
2 14 16 13 15 15 15 14 19 12 36.4 14.3 39.6
3 51 55 45 51 55 54 63 57 49 264.07 52 268.7
4 124 112 112 124 134 136 112 208 116 1218.6 120.9 1230.4
5 236 239 225 241 260 267 292 275 228 4315.3 231.4 4360.1
6 400 423 1177 413 464 467 532 455 391 12522.6 394 12588.2

IV. CONCLUSION Engineering and Computer Systems (ICSECS), 2015 4th International
Conference on, 2015, pp. 154-159.
[7] K. Z. Zamli, A. R. Alsewari, and B. Al-Kazemi, “Comparative
This paper proposes a new combinatorial testing strategy benchmarking of constraints t-way test generation strategy based on
called ABC-TG strategies, based on the ABC algorithm late acceptance hill climbing algorithm,” International Journal of
which generate test list. The main motivation of the ABC-TG Software Engineering & Computer Sciences (IJSECS), vol. 1, pp. 14-
26, 2015.
is to reduce the size of final test list. According to different
[8] T. Shiba, T. Tsuchiya, and T. Kikuno, “Using artificial life techniques
sets of experiments that have been conducted, ABC-TG has to generate test cases for combinatorial testing,” in the 28th Annual
shown performance and efficiency in term of generating a International on Computer Software and Applications Conference,
near optimal final test list size. As part of our future work, we 2004. COMPSAC 2004. , 2004, pp. 72-77.
[9] J. Stardom, Metaheuristics and the Search for Covering and Packing
are planning to enhance the ABC-TG, introduce a variable
Arrays. Simon Fraser University, 2001.
strength and seeding into the current implementation. [10] D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The
AETG system: an approach to testing based on combinatorial design,”
ACKNOWLEDGMENT Software Engineering, IEEE Transactions on, vol. 23, no. 7, pp. 437-
444, 1997.
[11] D. V. Reddy and A. R. M. Reddy, “An approach for fault detection in
This research is funded by UMP RDU150369: A new software testing through optimized test case prioritization,”
Hybrid Variable Interaction Strength Test Data Generation International Journal of Applied Engineering Research, vol. 11, no. 1,
Strategy Based on Harmony Search Algorithm and Cuckoo pp. 57-63, 2016.
[12] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence,
Search Algorithm, UMP RDU1603119 Grant: Modified
“IPOG/IPOG‐D: efficient test generation for multi‐way combinatorial
Greedy Algorithm Strategy for Combinatorial Testing testing,” Software Testing, Verification and Reliability, vol. 18, no. 3,
Problem with Constraints Supports. Also it is partially funded pp. 125-148, 2008.
by FRGS RDU160102: A New Global Optimization [13] Y. Lei and K.-C. Tai, “In-parameter-order: a test generation strategy
Algorithm based on Stochastic Approach to Minimize for pairwise testing,” in the Third IEEE International on High-
Assurance Systems Engineering Symposium, 1998, 1998, pp. 254-261.
Software Testing Redundancy [14] A. Williams, J. Lo, and A. Lareau, “TConfig,” ed, 2010.
[15] J. Czerwonka, D. Butt, and C. Gens, “Pairwise testing in real word:
REFERENCES practical extensions to test case generators,” in Proc. of the 24th pacific
northwest software quality conf. 2006, 2006.
[16] E. Lehmann and J. Wegener, “Test case design by means of the CTE
[1] A. A. Alsewari and K. Z. Zamli, “Interaction test data generation using
XL,” in Proceedings of the 8th European International Conference on
harmony search algorithm,” in Proceeding of IEEE Symposium on
Software Testing, Analysis & Review (EuroSTAR 2000), Kopenhagen,
Industrial Electronics & Applications, Langkawi, Malaysia, 2011, pp.
Denmark, 2000.
559-564.
[17] Jenkins, “Test Tool,” 2003. Available at
[2] A. R. A. Alsewari and K. Z. Zamli, “Design and implementation of a
http://www.burtleburtle.net/bob/math/jenny.html.
harmony-search-based variable-strength t-way testing strategy with
[18] A. Hartman, T. Klinger, and L. Raskin, “IBM intelligent test case
constraints support,” Information and Software Technology, vol. 54,
handler,” Discrete Mathematics, vol. 284, pp. 149-156, 2010.
no. 6, pp. 553-568, 2012.
[19] P. J. Schroeder, E. Kim, J. Arshem, and P. Bolaki, “Combining
[3] A. A. Ahmed and C. Xue Li, “Analyzing Data Remnant Remains on
behavior and data modeling in automated test case generation,” in the
User Devices to Determine Probative Artifacts in Cloud Environment,”
Third International Conference on Quality Software, 2003., 2003, pp.
Journal of Forensic Sciences, 2017.
247-254.
[4] B. Hambling, P. Morgan, A. Samaroo, and P. Williams, Software
[20] B. Nozohour-leilabady and B. Fazelabdolabadi, “On the Application of
Testing: An ISTQB-ISEB Foundation Guide: BCS, The Chartered
Artificial Bee Colony (ABC) Algorithm for Optimization of Well
Institute, 2010.
Placements in Fractured Reservoirs; Efficiency Comparison with the
[5] M. B. Cohen, M. B. Dwyer, and J. Shi, “Interaction testing of highly-
Particle Swarm Optimization (PSO) methodology,” Petroleum, 2015.
configurable systems in the presence of constraints,” in the 2007
[21] D. Karaboga and B. Akay, “A comparative study of artificial bee
international symposium on Software testing and analysis, 2007, pp.
colony algorithm,” Applied mathematics and computation, vol. 214,
129-139.
no. 1, pp. 108-132, 2009.
[6] A. A. B. Homaid and A. A. Alsewari, “A variable combinatorial test
suite strategy based on modified greedy algorithm,” in Software

88 e-ISSN: 2289-8131 Vol. 9 No. 3-3

View publication stats