UNIVERSITY TEKNOLOGI MARA

ANT COLONY ALGORITHM FOR

TEXT CLASSIFICATION IN
MULTICORE-MULTITHREAD
ENVIRONMENT

AHMAD NAZMI BIN FADZAL

Thesis submitted in fulfilment

of the requirements for the degree of
Master of Science

Faculty of Computer and Mathematical Sciences

January 2017

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 CONFIRMATION BY PANEL OF EXAMINERS

I certify that a Panel of Examiners has met on 10th August 2016 to conduct the final
examination of Ahmad Nazmi Bin Fadzal on his Master of Computer Science thesis
entitled "Ant Colony Algorithm on Text Classification in Multicore-Multithread
Environment" in accordance with Universiti Teknologi MARA Act 1976 (Akta 173).
The Panel of Examiners recommends that the student be awarded the relevant degree.
The panel of Examiners was as follows:

Seripah Awang Kechil, PhD

Associate Professor
Fakulti Sains Komputer dan Matematik
University Teknologi MARA
(Chairman)

Shuzlina Binti Abdul Rahman, PhD

Associate Professor
Faculty of Computer and Mathematical Sciences
Universiti Teknologi MARA
(Internal Examiner)

Mohd Zakree Bin Ahmad Nazri, PhD

Associate Professor
Faculty of Information Science and Technology
Universiti Kebangsaan Malaysia
(External Examiner)

DR MOHAMMAD NAWAWI
DATO' HAJI SEROJI, PhD
Dean
Institute of Graduate Studies
Universiti Teknologi MARA
Date: 19th January 2017

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 AUTHOR'S DECLARATION

I declare that the work in this thesis was carried out in accordance with the regulations
of Universiti Teknologi MARA. It is original and is the results of my own work,
unless otherwise indicated or acknowledged as referenced work. This thesis has not
been submitted to any other academic institution or non-academic institution for any
degree or qualification.

I, hereby, acknowledge that I have been supplied with the Academic Rules and
Regulations for Post Graduate, Universiti Teknologi MARA, regulating the conduct of
my study and research.

Name of Student Ahmad Nazmi Bin Fadzal

Student I.D. No. 2012308179

Programme Master of Computer Science - CS750

Faculty Computer and Mathematical Science

Thesis Title Ant Colony Algorithm for Text Classification in

Multicore-Multithread Environment

Signature of Student

Date January 2017

in

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 ABSTRACT

In the age of wide digital usage, text classification is one of the significant prominent
attribute required in order to automatically arrange emails, articles, and other textual
data in an organization. Unclassified data can lead to slower data retrieval thus a
reliable method is required to effectively retrieve data efficiently and in systematic
manner. Ant Colony Optimization (ACO) is a bio-inspired technique that was
introduced to solve Non-Polynomial hard problem of high text data dimension that is
similar to Traveling Salesman Problem (TSP) using probabilistic way. Pheromone
concept is the main criterion that distinguish ACO to other algorithms. Based on the
concept, pheromone saturation is used to combine stackable solution pattern that is
discovered while straying to different term node to build a path. ACO classification
accuracy is compared to Genetic Algorithm classifier which also a wrapper method.
On integration of the technique, ACO is proposed to work in a multicore-multithread
environment to gain additional execution time advantage. In multicore-multithread
environment, the adjustment aims to make artificial ants communicate across the
physical core of processor. As a trade to the investment for more computing power,
the execution time reduction is expected to show an improvement without
compromising the original classification accuracy. The unthreaded and multicore-
multithreaded version of ACO was experimented and compared in term of accuracy
and execution time. It was found that the result show a positive improvement.

iv

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 ACKNOWLEDGEMENT

Firstly, I wish to thank God for giving me the opportunity to embark on my master
programme and for completing this long and challenging journey successfully. My
gratitude and thanks go to my supervisor Prof Madya Dr. Mazidah Puteh, with co-
supervisor, Assoc. Prof. Dr. Adnan Ahmad and Dr. Norlela Samsudin. Thank you for
the support, patience and ideas in assisting me with this project.
Finally, this thesis is dedicated to the loving memory of my very dear mother and late
father for the vision and determination to educate me. This piece of victory is
dedicated to both of you. Alhamdulillah.

v
COPYRIGHT UNIVERSITI TEKNOLOGI MARA
 TABLE OF CONTENTS

Page

CONFIRMATION BY PANEL OF EXAMINERS ii

AUTHOR'S DECLARATION iii
ABSTRACT iv
ACKNOWLEDGEMENT v
TABLE OF CONTENTS vi
LIST OF TABLES ix
LIST OF FIGURES x

CHAPTER ONE: INTRODUCTION 1

1.1 Background of Study 2
1.2 Problem Statement 3
1.3 Objectives 6
1.4 Scope 6
1.5 Research Methodology 7
1.6 Significance 8
1.7 Summary and Contribution 9
1.8 Term Definition 9
1.9 Thesis Organization 11

CHAPTER TWO: LITERATURE REVIEW 12

2.1 Introduction 12
2.2 Text Classification 12
2.3 Genetic Algorithm 14
2.4 Support Vector Machine 20
2.5 Ant Colony Algorithm 24
2.5.1 Pheromone Reading 29
2.5.2 Evaporation 30
2.5.3 Stagnation 31

vi

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 2.5.4 Factors in Choosing ACO over GA and other Techniques 32
2.6 Multicore-Multithread Environment 33
2.6.1 Threading Over Parallelism for Multicore-Multithread Representation 35
2.6.2 Staying Idle 36
2.6.3 Code Development of Multitasking 38
2.6.4 Multithreading in Software Level 39
2.6.5 Multiplexing Time Division 40
2.6.6 Schedule of Multiplexing 41
2.6.7 Multithreading in Hardware Level 43
2.6.8 Advantages and Disadvantages Multicore Processor 44
2.6.9 Backup Slower Threads 46
2.7 Summary 49

CHAPTER THREE: RESEARCH METHODOLOGY 50

3.1 Introduction 50
3.2 Phase 1: Knowledge Acquisition 51
3.3 Phase 2: Text Data Collection and Identification 53
3.3.1 Data Pre-processing 54
3.3.2 GA Experiment Setting 58
3.3.3 Development of ACO 60
3.3.3.1 Simulation of Artificial Ant Visiting Term Nodes 63
3.3.3.2 Stopping Criterion 65
3.3.4 ACO Text Classification 66
3.3.4.1 Skip Common Term Calculation with Same Frequency 66
3.3.4.2 Obtaining Classification Rule Every Cycle 68
3.3.4.3 Choosing Next State 69
3.3.4.4 Pheromone Usage 72
3.3.5 Set Up Multicore-Multithread Environment 73
3.3.6 Multicore-Multithread ACO Model 75
3.3.6.1 Overcome Unstable Execution Speed 78
3.3.6.2 Signalling and Stopping Multiple Threads 79
3.4 Phase 3: Evaluation on Multicore-Multithread ACO Model 82
3.5 Summary 84

vn

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

CHAPTER FOUR: RESULTS AND DATA ANALYSIS 85
4.1 Introduction 85
4.2 ACO Text Classification Accuracy 85
4.2.1 Experiment 85
4.2.2 Discussion 86
4.3 GA Text Classification Accuracy 87
4.3.1 Experiment 87
4.3.2 Discussion 88
4.4 Comparison between ACO and GA Results 88
4.4.1 Discussion on ACO and GA 89
4.5 Results of Optimal Number of Thread Usage 90
4.5.1 Discussion 93
4.6 Accuracy Results of ACO with Multicore-Multithreading 94
4.6.1 Experiment 94
4.6.2 Discussion on Single and Multicore-Multithreaded 96
4.7 Results of Time Reduction Test for ACO with Multicore-Multithreading 97
4.7.1 Experiment 97
4.7.2 Discussion 98
4.8 Summary 100

CHAPTER FIVE: CONCLUSIONS AND FUTURE WORKS 101

5.1 Introduction 101
5.2 Research Outcome 101
5.3 Research Contribution 103
5.4 Future Suggestion 104

REFERENCES 106
AUTHOR'S PROFILE 114

Vlll

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 LIST OF TABLES

Tables Title Page

Table 2.1 Percentage Improvement over SVM 21
Table 2.2 Different Scheduling Algorithm Categories 42
Table 3.1 Summarization of Relevant Issue in Literature Review 52
Table 3.2 Example of Term-Frequency Table 57
Table 3.3 Relationship between Frequencies and Visited Category 70
Table 3.4 Probability Selection by Random Generated Value 72
Table 3.5 Machine and Operating System Specs 74
Table 3.6 Windows Experience Index Details 74
Table 3.7 Variables to Calculate Accuracy Performance 83
Table 4.1 Average Accuracy from Every 10 Runs of Every 50 Files
Addition 85
Table 4.2 Accuracy Difference between File Addition Increment 86
Table 4.3 Average Accuracy from 10 Runs of Every 50 Files Addition 87
Table 4.4 Accuracy Difference between File Addition Increment on GA 87
Table 4.5 Accuracy Comparison between ACO and GA Results 89
Table 4.6 Result of Optimal Thread Number Test 91
Table 4.7 Average Accuracy from Every 10 Runs of Every 50 Files
Addition 94
Table 4.8 Accuracy Difference between File Addition Increment 95
Table 4.9 Accuracy Difference between Single Threaded and Multicore-
Multithreading Version of ACO 95
Table 4.10 Time Taken of Text Classification Test for every 50 File
Addition Without Multicore-Multithreading 97
Table 4.11 Time Taken to Classify Text using ACO Featuring Multicore-
Multithreading 97
Table 4.12 Time Difference between Single Threaded and Multicore-
Multithreaded Version of ACO 98

IX

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 LIST OF FIGURES

Figures Title Page

Figure 1.1 Summary of Main Research Methodology 7

Figure 2.1 Flowchart for Text Classification 13
Figure 2.2 Flowchart of GA 15
Figure 2.3 One-Way Traffic Network with 20 Signals 16
Figure 2.4 Revolutionary Genetic Algorithm for Cluster 18
Figure 2.5 Sample Result of 5-Fold between GAMoN and other
Techniques 19
Figure 2.6 Optimal Separating Hyper-Plane 22
Figure 2.7 Mapping Data Points into High Dimensional Feature Space
using Kernel Functions 22
Figure 2.8 Vector Space of the Conventional SVM Classifier with Optimal
Separating Hyper-Plane 23
Figure 2.9 Euclidean-SVM Classifier with the Euclidean Distance Function
inside Vector Space as the Classification Decision Making
Algorithm 24
Figure 2.10 Two Paths with Different Length between Nest and Food 25
Figure 2.11 MMAS Pseudo-code with Local Search 27
Figure 2.12 Mechanism of Multicore-Multithread to Reduce Execution Time 33
Figure 2.13 Difference between Concurrency and Parallelism 36
Figure 2.14 Task Dependency Graph 39
Figure 2.15 Pheromone Deposit with Atomic Instructions 46
Figure 2.16 Scatter to Gather Transformation for Pheromone Deposit 48
Figure 3.1 Research Framework 51
Figure 3.2 Text Classification with Classifier (ACO) Framework 55
Figure 3.3 Main Tasks in Pre-processing Phase 56
Figure 3.4 Sample Content of Graphic Category Document 56
Figure 3.5 Sample Content of Microsoft Miscellaneous Category

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Document 57
Figure 3.6 Decision Making on Term Nodes 58
Figure 3.7 Pseudo-code of ACO Searching Method 60
Figure 3.8 Pseudo-code of Artificial Ants' Creation and Assignment 61
Figure 3.9 Pseudo-code of Artificial Ant Task 62
Figure 3.10 9 Randomly Generated Nodes before Simulation Run 64
Figure 3.11 The First Tour Path 64
Figure 3.12 The Path after Third Tour 65
Figure 3.13 Different Path that is Chosen between each Cycle 68
Figure 3.14 Sample of Pheromone Value 69
Figure 3.15 Steps to Calculate Heuristic Value 70
Figure 3.16 Difference between Sequential and Multithreaded Search 75
Figure 3.17 Main Part of Multicore-Multithread ACO Model 77
Figure 3.18 Task with Different Length Scheduled and Execution Sequence 79
Figure 3.19 Implementation on How a Thread Send Stopping Signal to other
Threads 82
Figure 4.1 CPU Utilization for Unthreaded ACO Classification 92
Figure 4.2 CPU Utilization for 9 Threads ACO Classification 93

XI

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 CHAPTER ONE

INTRODUCTION

In the digital age, text classification is one of the important features required to
organize articles, emails and other text data of an organization effectively. Larger
companies usually have larger databases with different data added constantly. These
companies require a systematic method that is not only efficient but also fast in
analysing and compiling their data according to categorical manner. However,
unclassified data can lead to slower data retrieval. Text data can accumulate and
become unmanageable over the years. To overcome the problem, the data is
categorized manually by using human resource. However, it is not a good solution
because it can be costly, slow and susceptible to human error. Therefore, fully
automatic classification by computer science research is required. Numerous
classification techniques have been developed to handle massive unstructured data.
The area of this research has also been explored from statistical approach to bio-
inspired classification algorithm.
Ant Colony Optimization (ACO) has been used in text classification (Dorigo,
1991). In contrast to statistical classification method, the whole ACO algorithm is
based on the behaviour of real-life ants that lived as an organized community. The
imitation of their communication method is based on liquid substances known as
pheromone rather than direct communication through speech. Based on this model,
large database is scanned continuously by several artificial ants to search for the text
classification criterion. Text classification criterion will conduct a search from specific
to repetitive appearance of words in text as well as making comparison between each
article and other form of text data. Repetition and built up in generating solution are
several features that differentiate ACO algorithm from traditional approach. The
technique is chosen as text classification approach to further explore the significance
of natural algorithm and inspiration of biological mechanism (Bonasio et al., 2010;
Dressier & Akan, 2010; Emily, 2014).
This research develops the use of ACO in multicore-multithread environment
to achieve text classification. Its implementation can be used in modern search
engines, online libraries and data centres. The approach is anticipated to improve

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

classification accuracy. Using the combination with multicore-multithreaded
computing, user will be able to experience faster running time by increasing the
number of concurrent search.

1.1 Background of Study

Text classification is important in organizing growing data over the years.

However, text data organization can be tricky because the terms used inside the text
may have ambiguous meaning and available in high dimension relationship. Text
classification method must be accurate and fast to handle larger data set, while the
algorithm used should be simple to be implemented. Methods to classify text are
available as statistical type or bio-inspired algorithm. Bio-inspired algorithm consists
of evolutionary algorithm such as Genetic Algorithm (GA) and ACO that are usually
known by its wrapper attribute. In this research, ACO is chosen to be compared with
GA to discover its capabilities to accurately classify text and to work in multicore-
multithread environment. Since its discovery by Marco Dorigo in 1991, ACO has
been actively researched and applied in text classification as it is able to solve non-
polynomial hard problems.
In real life, ants wander randomly to search for food before gradually finding
the optimal route. This happens because an ant leaves pheromone in its trail and
adjusted the strength of its smells as a mean for effective communication (Rekaya et
al., 2013). Ant Colony Optimization (ACO) is a wrapper method which uses iteration
to build up solution. The probabilistic and meta-heuristic features are established to
solve difficult optimization problems (Rekaya et al., 2013). The quality of the solution
is expected to increase as the number of iteration increase.
Based on ACO technique, pheromone is deposited to mark selection criterion
when assigning a document into one or more classes. Using this method, the
researcher tries to use the relationship between each term as the length from a node to
the next one. The ACO algorithm used in this research is taken from Dorigo (1991)
which had been used to find the shortest path in Traveling Salesman Problem (TSP).
The length need to be converted to the inversed value of itself so that the shortest path
will be equal to the strongest bond between terms. To obtain precise result, the
researcher will keep the standard setting as the experiment set up will remain the

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

same. The term-frequency will be used in representing bond between terms as the
equal path searching of TSP.
In order to cater a big data problem processing such as documents residing in
data centre, multicore-multithreaded computing provides a great feature of
concurrency. Throughout the research, ants have been found to communicate through
pheromone regardless of which ants have deposited them. Such ability could be taken
as a great advantage for effective communication in concurrent situation. ACO
approach in multicore-multithreaded environment helps to find solution faster than a
single computer could do. Other classification methods rarely have standard
information to be adapted in concurrent processing as compared to the ants'
pheromone.

1.2 Problem Statement

In a short period, articles and other form of text data like emails can
accumulate massively. This phenomenon creates various problems in text
classification when data grows larger over time. This situation happens especially
when using supervised and semi-supervised classification algorithm. Larger database
can have complex text pattern and classification criterion.
The following are the problems found in text classification:
1) The main disadvantage of wrapper methods is the computational cost
(Saeys, Inza, & Larranaga, 2007) that relate from invoking induction
algorithm of every relevant feature set (Langley, 1994). In choosing
ACO algorithm, using multiple artificial ants could cost equivalent
amount of pheromone copy. Therefore, it will cost a large amount of
memory to be used at one time. From similar situation, calculating
heuristic and cross validation process also has high computation cost.
One of the modern technology that is suitable to handle large data
processing is multicore-multithread technology. Particularly, an
effective design to address computational cost is needed for ACO
based text classification system on a multicore-multithread
environment that will overcome data inconsistency issues.

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

2) Pheromone sharing could be used to reduce pheromone copy to be only
a single copy. All artificial ants will use central pheromone access
during solution searching. However, sequential access to update
pheromone must be done one by one thus requires affected artificial
ants to wait in a queue. Pheromone update consists of atomic processes
thus prevent the update to be done simultaneously by multiple artificial
ants. As a wrapper method, ACO could go through for this same
pheromone update of every cycle continuously up to several hundred or
even thousand time. Compared to filter method, wrapper method has to
bear this computational cost in a single classification which makes a
large gap in execution time. This cost has alternated some researchers
to design ingenious techniques for speeding the evaluation process
(Langley, 1994). One of widely available technology that provide
better performance in time-consuming computation is multicore-
multithread. Therefore, a design for ACO based text classification
system on a multicore-multithread environment that reduces
computation time is required.
3) Multicore-multithread technology provides general solution in handling
computation problem. Different usage of hardware and software with
distinctive setting returns different efficiency benefit. In ACO for text
classification in multicore-multithread environment, a set of test to
determine ideal number of multiple thread of specified setting is
needed. We also need to learn the reason for efficiency value that relate
to number of multiple thread. Discussion of effect on using pheromone
sharing to multicore-multithread implementation is also required.
4) In order to handle computation cost, accuracy trade off become a
concern (John, Kohavi, & Pfleger, 1994; Caruana & Freitag, 1994). For
a more complex adjustment to have ACO run concurrently, accuracy of
the text classification could deteriorate when pheromone update
serialization become random in sequence.
5) Lacks of integration module to simulate classifier algorithm on
multicore-multithread could be problematic when current technology
provide sufficient resource on today's machine. Since not all classifiers
are compatible to be executed concurrently, algorithm like ACO and

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 other wrapper methods should be further explored for running on
multiple instances adaptation. Other similar researches has been done
but differ at parallelism concept against concurrency such as A Parallel
ACO Algorithm to Select Terms to Categorise Longer Documents
(Janaki Meena, Chandran, Karthik, & Vijay Samuel, 2011) and An
Enhanced ACO Algorithm to Select Features for Text Categorization
and its Parallelization by Janaki Meena et al., (2012). Cecilia, Garcia,
Nisbet, Amos and Ujaldon (2013) work is also similar but differ when
it is used on Graphical Processing Unit (GPU) with parallelism
implementation. Compared to work done by Umbarkar, Joshi and
Hong in 2014, the algorithm consists of dual population which is a
major distinguish criterion in population base algorithm. ACO for this
research is limited to be used on single colony and multi colony ACO
is valued as future improvement.
Use of pheromone in ACO can reduce the computational cost when dealing
with high dimensional data. Number of dimension of data for supervised learning
refers to the number of document in training set. It is different than unsupervised
learning where the number of dimension refers to the number of terms (Gasanova,
Sergienko, Semenkin, & Minker, 2014). The pheromone is stored in larger arrays as
the number of terms required to process it increased. However, the calculation of its
heuristic remains simple and pheromone value can be read immediately to compensate
larger memory used for the array. Furthermore, random selection in the heuristic is
also straightforward. The problem to differentiate similar categories can be closely
related to ACO classification accuracy. In the real situation where a group of ants is
searching for food, the strength of pheromone concentrations is measured up to
determine which path is shorter. By applying this in ACO, the pheromone
concentration can help in measuring which categories is closer for the given test
article based on the information from the training set. Automatic classification is also
run by sophisticated method that always produces constant result thus is less prone to
do unintended things from the classification definition. To solve the time-consuming
problem when using wrapper method, the researcher has further extends the ACO to
work concurrently. This is done by creating multiple instances of artificial ants that
moved independently from each other through the application of multithreading

5
COPYRIGHT UNIVERSITI TEKNOLOGI MARA
technologies. Lacks of integration module problem is dealt by using the simulation of
multiple artificial ant instances to observe the time reduction result.
The following are several research questions that are solved in this research:
1) Could an algorithm shorten the execution time by eliminating sharing
variable usage?
2) Could we get the most time reduction if we use a large number of threads?
If not, what is the optimal number of thread to be specified in the
parameter setting?
3) Is pheromone sharing suitable to be adapted to reduce cost of multiple
copy of pheromone array value?

1.3 Objectives

This research aims to develop an effective method for text classification to

classify text document with supervised learning mode. The objectives of this research
are:
i. To develop ant colony optimization algorithm for text classification,
ii. To investigate the optimal setting for ACO for text classification in
multicore-multithread environment,
iii. To design architecture of ant colony optimization in multicore-multithread
environment for text classification in order to reduce the processing time of
text classification,
iv. To evaluate ant colony algorithm in multicore-multithread environment
with single-thread and multicore-multithread environment.

1.4 Scope

The following are the scopes which reflect the objectives of this research:
i. Non-text such as numeric and symbol are omitted.
ii. Algorithm used on single colony of artificial ant.
iii. Covers on English text file,
iv. Use of multicore-multithreading environment for concurrent processing.

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 v. Text classification for two categories which are Computer Graphics and
Computer Operating System: Microsoft Miscellaneous,
vi. Classify text type of newsgroup collection.

1.5 Research Methodology

This chapter explained and summarized the processes involved in solving the
research problem and in fulfilling the research objectives. They are divided into seven
phases as stated in the next page:

Text Data Collection and Identification

explains how data is collected and its sources

Data Preprocessing
explains how data are filtered

Development of Ant Colony Algorithm

explains the development process and pseudo-codes

ACO Text Classification

explains the mechanism of ACO

Set Up Multicore-Multithread Environment

provides specification and settings

Multicore-Multithread ACO Model

explains the mechanism of the model

Evaluation on Multicore-Multithread ACO Model

explains how the evaluation is done

Figure 1.1: Summary of Main Research Methodology

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Figure 1.1 describes the processes and the result for every phase. The
methodology starts with data collection which explains the validity of data for the
experiment. This is followed by the next phase which is known as data filtering. This
process clean the original data in order for the main algorithm to recognize the data
and then to process and analyze the data. The next phase involves the way the
development is carried out. This phase focuses on the code development and the code
utilization feature provided by the chosen programming language. The process
continues with the mechanism of ACO. A summary of machine specification and
parameter setting in multicore-multithread ACO are listed. The summary points out
relevant explanations on the works of multicore-multithread mechanism works in this
phase. Lastly, the software evaluation phase which provides the formula used to
evaluate the experiment.

1.6 Significance

One of the benefits from the suggested improvement in this research would be
the increments of running efficiency. This impact is less noticeable at the beginning
but in a long run or for big data problem, it will add up and become significantly
useful. Such improvement could give better response time in searching database
query. This can be achieved when multiple cores are working together where low
overhead is needed to distribute the workload. The nature of pheromone presented
also allows it to be updated at any time when other computer demands it. The benefits
could affect various mediums such as blogs, searching engines, newspaper reviews
and bibliography categorization.
Another running efficiency that could be improved from the result of the
research is time reduction. For an evolutionary algorithm, processing time becomes
important factor for its usability. The time reduction becomes noticeable when dealing
with a larger data set as data dimensionality also increased proportionally. Depending
on the method chosen and data amount, classification process could take from a few
minutes to even a whole year. Some techniques even involved data crunching problem
where there is no lead for an algorithm to get close to a better result. Besides that
looping condition could also be reached infinitely when the problem occurred.

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Therefore, improving time reduction for classification process is more important in
wrapper classification type than filter and embedded.
In order for concurrency development to achieve shorter execution time in this
research, some penetration in multicore-multithread improvement is given. The
benefit is also to help in other related areas such as integration of AI and parallelism.
Multicore-multithread environment can provide better service to a bigger data set and
challenging evolutionary algorithm especially text classification service.

1.7 Summary and Contribution

Research result and contributions are listed below:

1. Model for classifying texts that appear in database or website.
2. Nature-based algorithm that is accurate, fast, closer representation of
real phenomenon and can be applied in a wide diversity of
classification at user definition and customization.
3. An algorithm that is compatible to work in multiple instances which
utilize both multicore and multithread to increase computation
efficiency.
4. Corpus from online collection has wide variety of documents and is
meant to have random and fair length to depict a real situation to
observe the algorithm effectiveness when dealing with extreme
irregular data pattern.
5. Provide extended capability to systematic classification task with
concurrency as in the multicore-multithread environment.

1.8 Term Definition

Understanding the meaning for each main terminology could help towards
better understanding of this thesis. This chapter explains the definition for the
frequently used terms which are critical to the main concept of this research such as
pheromone, concurrency and parallelism.

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Term
Term used in this thesis refer to the obtain output of an input document which
comprise of words in the textual writing of newsgroup collection. The terms output
are obtained in pre-processing phase that filter out number and symbols. They are
separated by white space.
Pheromone
Pheromone is a term used to refer to secreted or excreted liquid chemical that
signal meaningful message to other individual by some species of insects. It is used as
a distinguished and unique communication medium. There are many types of
pheromone with different applications but the term pheromone used in this research
refers to the pheromone released by an ant. The researcher tries to use its application
in real world where real ants frequently release a trail of pheromone as they move to
other place. This behaviour could be observed when ant worker leaves its nest to
forage for food. Different types of pheromone keep different information. However
pheromone used for foraging food has constant amount of evaporation. It can be used
to interpret distance between two points when it is compared with different route or
path. Similarly, this research keeps an array of variable to keep track of status for each
term to find the optimal solution for text classification.
Parallelism
Parallelism aims to achieve concurrency to be applied on a synchronize task.
In code development, a software engineer needs to change from applying
straightforward task to a more focused, breakable operation of a task. It is required to
do so for a physical multicore processor to run the code in parallel. The software
engineer needs to put the code in synchronized block which will give signal to the
operating system to call for the multicore processing. Note that parallelism is applied
on the processor itself rather than having abstract implementation at user level.
Therefore, codes for parallelism are not effective on single core processor. This
restriction is different with multithreading that has wider option and tweak which will
be explained further later.
Multithreading
While parallelism is much more abstract and mainly to describe the theory
itself, multithreading refers to a subset for the implementation of parallelism. It
specifically has its implementation to the physical core of processor and wider
customization at software level development. Compared to the pure implementation of

10

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

parallelism, multithreading has more dynamic construct. In multicore-multithread
representation, the multithreading is used to simulate multiple instances of artificial
ants. For each artificial ant instance created to each thread, memory for global variable
can be shared directly as compared to multi-processing. This feature is important for
artificial ant to work collectively and efficiently.

1.9 Thesis Organization

Chapter 2 discusses previous researches related to ACO, text classification,

GA and multicore-multithread environment. The discussion focuses on how problems
in text classification can be solved, their advantages and disadvantages. Besides that,
various researches that are related to text classification processing, ACO and
multicore-multithread environment to classify text are also discussed here.
Chapter 3 presents the research methodology that is used to achieve the stated
research goals. Discussion starts with the main framework and continues with the data
collection for data identification and text pre-processing. ACO text classification
processes are explained after a detail discussion of ACO code development. Then,
multicore-multithread mechanism is thoroughly explained together with the set up to
create an environment of multicore-multithread for ACO text classification.
Chapter 4 discusses all the experiment that had been carried out by the
researcher in details especially regarding the nature of the experiments' result. The
experiments that had been conducted can be divided into three major parts. The first
part constitutes the single threaded version of ACO to test out its classification
accuracy. GA for text classification is tested for its accuracy in the second part. The
result for GA is a complementary to make an accurate comparison for ACO. The last
part is for testing ACO adaptation to multicore-multithread environment which
measures time reduction improvement. Every experiment is discussed with relevant
details.
Chapter 5 concludes the conducted research in general and lists out a number
of advantages from using the selected research method in text classification. The
chapter also explains the limitation of the method, followed by some suggestions for
future improvements.

11

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 CHAPTER TWO

LITERATURE REVIEW

2.1 Introduction

This chapter explains the previous researches that are related and relevant to
ACO text classification. The discussion starts by describing text classification in
general from past researches. The discussion continues with GA which algorithm is
comparable to the ACO. The content explains its implementation that had been
discovered from the previous researches in a simplistic fashion. Then, the developed
method of ACO is discussed in the subsequent chapter. The research suggests a
repetitive or cyclic algorithm that builds up its solution quality over increasing
iteration. Therefore, explanation on current ACO version that is used widely for text
classification is given in order to overview its main process when applied in text
classification. Finally, an overview of multicore-multithread environment is discussed.
Multicore-multithread environment is intended to be run together with ACO
implementation in this research. For that purpose, discussion on previous researches
of multicore-multithread computing is included.

2.2 Text Classification

Text mining is roughly equivalent to text analytics and it can be broadly

defined as a knowledge intensive process, in which a user interacts with a document
collection over time by using a suite of analysis tools (Feldman & Sanger, 2007). Text
documents in their original form cannot be directly processed by common classifiers
and available learning algorithms. Therefore, during a pre-processing step, the
documents are represented by feature vectors (Zhang & Pan, 2011). Pre-processing
step is important to filter out unnecessary data and text documents are turned into
smaller parts in order for it to be suitable input to the selected classification methods.
A number of classifiers have been used to classify documents. An example of these
classifiers are neural networks (Aggarwal & Zhai, 2012), Support Vector Machines

12

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

(SVM) (Wang & Manning, 2012), genetic programming (Aggarwal & Zhai, 2012),
Kohonen type self-organizing maps (Hyotyniemi, 1996), Bayesian network classifier
(Aggarwal & Zhai, 2012), and Naive Bayes classifier (Vaidya, Shafiq, Basu, & Hong,
2013). Figure 2.1 shows a flowchart for text mining architecture:

Documents

I
Preprocess
Document

JZ
Information
Extraction Management
Information
^3X
Clustering Summarization
P=?i System

Figure 2.1: Flowchart for Text Mining Architecture (Nalini & Sheela, 2014)

Most of the related works on text classification focus on eliminating the

problem of data sparseness (Sriram, Fuhry, Demir, Ferhatosmanoglu, & Demirbas,
2010). Text mining can be broadly defined as a knowledge intensive process, which a
user interacts with a document collection over time by using a suitable analysis tools
(Feldman & Sanger, 2007). Text documents in their original form cannot be directly
processed by common classifiers and available learning algorithms. However, ACO
could use their vector form to represent TSP version of terms relationship. To cope
with different text pattern and main terms, ACO extract the classification rule using IF
(termi) AND (term2) THEN {class) method to induce the texts pattern.
Computer scientists are required to develop new algorithm or use existing one
to mine the scattered and unstructured data. There are some existing algorithms
applied in data mining; such as C4.5, Apriori, Naive Bayes, kNN, SVM and Ant
Colony algorithms (Wu et al., 2008). Ant Colony Algorithms, proposed by Dorigo
(1992), is a probabilistic technique which constitutes some meta-heuristic
optimizations and can be used to solve difficult optimization problems (Rekaya et al.,

13

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

2013). Ants are social insect that live in organized colonies which consists of huge
number of individuals for dominating larger territories.

2.3 Genetic Algorithm

In text classification problem, GA (Wang & Cao, 2002) can search the optimal
characteristic variable given in a set of document. Selection of candidate is done
naturally by survival of the fittest and genetic evolution mechanisms where it is
possible to isolate inaccurate discernment in the algorithm and produce better text
classification result (Baharudin, Lee, & Khan, 2010). GA as a type of evolutionary
algorithms that has been used widely for its performance in classifying text
documents. Last year, its usability had been extended to the induction of classification
rules (Pietramala, Policicchio, Rullo, & Sidhu, 2008). In Olex-GA implementation,
the developed method relies on an efficient several-rules-per-individual binary
representation. For solution domain evaluation, the fitness function uses F-measure to
keep the algorithm accurate. In this research, the ACO which makes selection
randomly with minimal heuristic will be compared to Olex-GA in term of
classification accuracy. Figure 2.2 the flowchart for basic GA:

14

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Start

Generate initial
population

*
Evaluate fitness function
Rank individual fitness

Generate new population

Selection
Crossover
Mutation

Figure 2.2: Flowchart of GA (Kachitvichyanukul, 2012)

In text classification, the speed to access weighting evaluation and calculate

classification algorithm heuristic is one of the important factors in choosing
techniques usability. For continuous evaluation of the classification problem, the
relevant selection of term categorization accuracies will influence the amount of
resources needed. It will continue until the whole classification process is finished. In
other words, a high quality heuristic creation can lead to better accessibility and
shorter running time. GA is one of the known techniques that can repeatedly assess
the text classification process with clear selection mechanism. Selection process can
be naturally formed and other mechanism suggestions include mutation and traits
variation research. GA has been honed for better accuracy which is derived as its
predecessor such as Adaptive GA (AGA). For a problem with many constraints such
as oversaturated traffic, GA has been chosen in the research in comparison with ACO
for their operability in efficient manner.

15

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Figure 2.3: One-Way Traffic Network with 20 Signals (Putha, Quadrifoglio, & Zechman,
2012)

One of the studies includes further discussion with smoother switching

planning (Putha et al., 2012) that can be closely related to category selection in text
classification. Figure 2.3 shows the example of how one-way traffic network with 20
signals problem is mapped (Putha et al., 2012). This research has elaborated traffic
problems that require optimization over green light timing especially when the traffic
is oversaturated. In discovering the duration of all three phases, GA mutation and
natural selection in serial succession are suitable to analyse individual delays. This is
done when input variable of queue length frequently changed. GA is also required to
plan for signal timing that is capable to minimize traffic jam, handle constraints on
maximum queue lengths, and be transparent for time-varying demands. GA is used to
analyse critical operational problem on signalized intersections and eliminate separate
lane groups of various oversaturated situation (Liu & Chang, 2011). In this research,
the performance between GA and ACO are also compared. The statistic is further
accumulated with a test of two-stage timing method and it is included as reference that
features an integrated approach. In the same area of studies, research of ACO has been
successfully applied to plan for transportation schedule (Vitins & Axhausen, 2009;
Yang, Yu, & Cheng, 2007).
GA has been widely used in text classification as the preferred option in using
wrapper method. It allows representation of effective relationship between each term.
Multiple terms with semantic information can be effectively analysed by GA as it
features weighting to the term in a form of genetic representation. Even though there
are many options to transform text, majority of text classification studies has chosen to

16

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

use Latent Semantic Indexing (LSI). The preference is based on its proven
performance (Meng, Lin, & Yu, 2011; Thorleuchter, & Van den Poel, 2013; Wang &
Yu, 2009; Wang, Xu, Li, & Craswell, 2013; Yang, Sun, Sun, Cao, & Zheng, 2009;
Yu, Xu, & Li, 2008; Zhang, Yoshida, & Tang, 2011) which can be integrated with
GA. A key feature of LSI is its ability to extract the conceptual content of a body of
text by establishing associations between those terms that occur in similar contexts
(Deerwester et al., 1988).
Since GA is mainly an algorithm related to systematic structure of DNA
information, GA can perform well with LSI feature to overcome synonym words. The
feature implementation is important to address overall classification performance.
This is because it tackles the usage limitation of binary representation such as Boolean
keyword queries. On recent research (Uysal & Gunal, 2014), LSI technique that is
oriented by GA known as Genetic Algorithm Latent Semantic Feature (GALSF)
shows statistically better result than LSI and filter-based feature selection for various
feature dimensions. For other text related issue, LSI can be integrated well with GA to
run dynamic clustering based on the theoretical content of documents. For that
purpose, GA can work as self-contained to adapt with automatic classification feature
from text classification principle. This also means that GA has the capabilities to work
without using example or training document, based on the similarities between the
concepts that they possessed. The capabilities also cover the concepts that are
contained in the example documents.
GA is further explored with its performance to work with highly
computational expensive techniques, which is one of LSI limitations. LSI requires
relatively high computational performance and memory in comparison to other
information retrieval techniques based on researches (Karypis & Han, 2000). The
problem can be partially solved with parallel processing on software engineering sides
and also using modern processor technologies that are designed to work on massive
processing task.
GA has been used in dimension reduction research and has been compared
with other techniques for their performance on reduced dimensionality in text
classification (Gasanova et al., 2014). While text classification algorithm can be
unaffected by reduced dimensionality, GA is found to work well with the improved
suggestion to build cluster vocabulary. The terms in training documents are assessed
with goal to find different term with similar or close meaning by analysing the context

17

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

of how they are used. The dimensional reduction is an important factor to reduce
computational cost and running time in further classification algorithm and is parallel
to our research objective. The underlying process involves the use of cluster weights
optimization which features GA. There is a hierarchy tree for each term class and an
assigned value to all clusters. GA is required to find global optimum value. To take
advantage of all of weights improvement, cooperative revolutionary GA is applied
for local search. Figure 2.4 is an overview on how coevolutionary algorithm works for
cluster classification:

GA optimizes GA optimizes J
the weights of the weights of
5 generations —J 1st category th
n category
15 times - < j a
" ^ ^

! Update all weights with values of

• the best obtained individuals from
j last 5 generations

Figure 2.4: Coevolutionary Genetic Algorithm for Cluster (Gasanova et al., 2014)

One of the factors GA works better in coevolutionary algorithm is due to the

fact that all individual GA work separately in first phase which yield the result of the
length of the phase. The result used to determine on how many generations or
tournaments that individual algorithms can communicate without information
interchange. The process is similar to the ACO algorithm developed in this research in
term of updating the resulting pheromone where it stops all separate algorithms. The
process also includes updating all weights that have been obtained by the best
individual among all algorithms. The process of the two stages shown is repeated until
maximum number of generation is obtained. The result was concluded in term of
classification efficiency which increases or unnoticeable decreases with the use of k-
nearest neighbours with weight. Kernel Bayes classification with Laplace correction,
neural network, Rocchio and linear SVM in RapidMiner application are also
concluded.

18

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Recently, a number of wrapped types for text classification algorithms has
been developed and enhanced for better performance to compete with statistical
methods. One of the techniques introduced is Genetic Algorithm: M-of-N (GAMoN).
GAMoN refer to a technique that features the use of GA which constitute the lattice-
based classification structure of hypothesis dimension. It is effective to induce
accurate inclusive negation of M-of-N hypothesis. In the test mentioned for the
research (Policicchio, Pietramala, & Rullo, 2012), more than 13 real-world text data
sets was benchmarked. Four classification algorithm were used which comprise two
evolutionary algorithm, namely Olex-GA and Bioinformatics-oriented Hierarchical
Evolutionary Learning (BioHEL). The other two are non-evolutionary algorithm,
listed as C4.5 and Ripper. In term of the best text classification comparison, linear
SVM is mentioned in the studies. GAMoN was discovered to provide state-of-the-art
classification performance that is balanced between accuracy and complexity. Another
concern for GAMoN feature in this research is its ability to work in massive
workload. GAMoN is found to be able to handle large and realistic real-world domain
better than C4.5 and Ripper. Figure 2.5 shows the result comparison between GAMoN
and many other selected techniques such as Ripper, Sequential Minimal Optimization
(SMO), C4.5, Olex-GA and BioHEL.

1 Micro-averaged PRavg results obtained by 5-fold cross-validationonOhO. Oh5, OhlO, Oh15. BlogsGender, Ohscale and RIO (80/20 split) and by holdout on
the remaining data sets (70/30 split).

Dataset Ripper SMO C4.5 OlexGA BioHEL GAMoN

Ohl5 79.55 79.95 76.75 74.33 66.03 80.46
Oh5 83.74 84.29 82.30 80.76 76.72 84.85
OhO 84.37 84.80 79.24 81.29 73.20 84.27
OhlO 78.82 74.70 74.78 74.46 67.39 78.62
BkjgsCender 60.96 60.95 58.33 66.75 62.82 68.82
Ohscale 72.96 69.52 70.77 74.36 68.26 75.03
RIO 85.21 88.94 84.67 84.07 83.59 86.50 |
20NC 72.66 83.64 74.86 72.97 70.65 75.72 |
Ohsumed 60.35 66.94 63.25 65.58 63.79 67.25 J
cade 44.31 54.06 48.10 44.10 42.96 48.42
SRAA 81.04 90.06 86.85 79.60 81.34 85.25
ODP 66.73 80.65 74.76 69.46 71.21 71.14
Maricet 94.63 95.56 95.37 88.95 74.75 92.42
avg microPRavg 74.26 78.00 74.62 73.59 69.44 76.83
avg rank 3.96 2.08 3.62 4.31 5.23 2.08

Figure 2.5: Sample Result of 5-Fold Between GAMoN and other Techniques (Policicchio et
al., 2012)

One of the factors that cause GA to work better in GAMoN induction is the
use of M-of-N relationship creates better representation with GA. The ability
represents the accommodation of complex DNA relationship map. The M-of-N
allocation can structure the classical concept by having negation and disjunction in
term to term relationship. The relationship is designed as a binary categorization

19

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

appliance. The test outcomes show that GAMoN performs well as featuring GA
because it builds simple and compact model in classification process. Therefore, it
helps human in understanding the learning process (Policicchio et al., 2012).

2.4 Support Vector Machine

Researches has been done thoroughly to create effective algorithm to classify

textual documents. One of them is SVM. SVM is a method which search for the ideal
hyper-plane among two categories of term. The process is done by dividing opposing
values negative and positive examples using substantial line at the center that is
known as decision line (Hassan, Rafi, & Shaikh, 2011). One of researches done
recently by using SVM classifier has been tested on Wikitology. Wikitology is a
composed knowledge based on organized and unorganized data derived from
Wikipedia. The structured composition of Wikitology could improve prediction ability
of SVM by organized format of the data. The Wikipedia requirement and quality
excerpt on publishing particular topic helps in completing data for a class. The SVM
classifier could have better ability to distinguish a class characteristic from just the
definition. The process happens when a proper definition has more precise term used
to describe things and author tends to give more attention with choice of word used.
SVM classifier methodology use more than frequency of term thus this will favor its
accuracy among other frequency based classifier such as NB. The definition
availability in most of topic in Wikipedia followed by formal choice of word and
history format would be more advantageous to dimensional region in SVM
calculation. In term of performance, the access of correct data through sequential
manner would help the execution run faster especially when the data need to be
analyzed for the first time. The effectiveness of SVM is further proven in this research
when it was compared with NB in term of accuracy. However, in term of complexity,
NB is more simple and straightforward thus run much faster. NB simply calculate
probability based on frequency of words occurrences. SVM calculation task on the
other hand involved in normalizing dimensionality and find hyper plane.
Particular to Wikitology, information retrieval is also done and kept to
unstructured or unorganized form. Knowledge retrieved from Wikitology added to
document using different text document representation and enrichment technique
(Hassan et al., 2011). Unorganized data of Wikipedia could have been slower in

20

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

aspect of data retrieval and processing. Experimental results proved that there was a
better text clustering method and classification by enrichment of textual documents
supported by the knowledge of Wikitology, Wikipedia, Open Project Directory and
Word Net when evaluated against to their defined baseline on converge dataset
(Hassan et al.).
Based on the experiments result in Table 2.1, an increment of +6.36% (as
compared to baseline) was observed by implementing machine learning technique
SVM. The research continues by comparing the result with Naive Bayes classification
that use The 20 Newsgroups dataset. From the analysis, Different Experimental result
showed that SVM performance exceed than NB classification accuracy in general
classification tasks. Better result is achieved by integrating semantic background to
document. This improvement is implemented by researchers with various knowledge
repositories like OPD, WorldNet, Wikitology, Wikipedia and etcetera.
Table 2.1:
Percentage Improvement over SVM (Hassan et al, 2011)
SVM SVM -Improvement
Enrichment Micro Macro Micro Macro
Baseline 0.868 0.865 - -

Al 0.784 0.768 -9.68% -11.21%

A2 0.770 0.757 -11.29% -12.49%
A3 0.843 0.830 -2.88% -4.05%
A4 0.919 0.920 5.88% 6J6*/o
A5 0.851 0.839 -1.96% -3.01%

Since its discovery, there are enhancements made to the algorithm. One of
them include integration with Euclidean distance (Euclidean-SVM) in order to test
classification of text (Lee, Wan, Rajkumar, & Isa, 2012). Also known as Euclidean
metric, Euclidean distance is a normal straight-line gap between two spots in
Euclidean space. The SVM approach is applied in training phase and implement a
classifier by drawing decision surface, which is the optimal divided hyper-plane as
shown in Figure 2.6. It is to separate distinctive classes of data points in the resulting
vector space.

21

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Figure 2.6: Optimal Separating Hyper-Plane (Lee et al., 2012)

The idea of the optimal divided hyper-plane can be normalized for the non-
linear separating event by kernel operation to match the data points from the input
space (Figure 2.7) into a high dimensional attribute space so that they are divided by a
linear hyper-plane. This attribute results in the application of distinctive kernel
operations to lead to a high implication on the categorization accuracy of the SVM.

c>

Input space Feature space

Figure 2.7: Mapping Data Points into High Dimensional Feature Space using Kernel
Functions (Lee et al., 2012)

Beside the kernel operations, the value of soft margin region, C is the other
important element in ascertain the ability of SVM classifier. Therefore, one of main

22

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

problems of the conventional SVM categorization process is the requirements of
identifying the suitable kernel operation and the suitable value of variable C. It is for
distinctive datasets of different characteristics, for the propose of providing high
accuracy of the classifier.

Figure 2.8: Vector Space of the Conventional SVM Classifier with Optimal
Separating Hyper-Plane (Lee et al., 2012)

In the proposed method (Figure 2.9), the author introduced a distance

calculation method, which use the Euclidean distance method to change the optimal
line of separating hyper-plane as the process for categorization decision selecting
function in the SVM (Lee et al., 2012). Based on the approach taken, the supporting
vector for every class are searched from training data characteristic throughout
training stage using the SVM. In the classification stage, when a new data node is
synchronized into the native vector space, the average distances between the new data
node and the support vectors from distinctive classes are calculated using the
Euclidean metric method. The selection of category is chosen based on the
classification of support vectors which has the lowest value of average distance with
the new data node. This condition affects categorization decision regardless of the
ability to produce result from hyper-plane constructed by implementing the specific
kernel operation and soft margin variable. The experiment run with multiple textual
datasets. The experimental results with appropriate analysis show that the proposed
approach affect the accuracy of the Euclidean-S VM text categorize to have a minimal
implication on the integration of kernel operations and soft margin variable C.

23

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Figure 2.9: Euclidean-SVM Classifier with the Euclidean Distance Function inside
Vector Space as the Classification Decision Making Algorithm (Lee et al., 2012)

2.5 Ant Colony Algorithm

Researches have been done thoroughly to understand how small animal with
limited vision is able to find food far from its nest. The route used by the ant changes
to optimal as the time goes by. This happens because a real ant leaves pheromone in
its trail and adjusted the strength of its smells as a mean of effective communication
(Rekaya et al., 2013). This is one of the abilities gifted from God to His creation and it
is a miracle for each of the ant to be able to learn such phenomena in order to adapt
themselves with the challenging surrounding. The mechanism of ant finding the
shortest route to the food had been studied and Ant Colony Optimization algorithm
was formed as the result. Based on the analogy, the algorithm is suitable to be applied
in text classification.
ACO algorithms have been inspired by an experiment which was carried by
Goss et al., in 1989 that use a real colony of Argentine ants (Iridomyrmex Humilis)
(Dorigo & Thomas, 2004). By placing a food source to a separate area linked by a
bridge with two branches of different length, most of the ants use the shortest branch
to their nest after a few minute. The ants used pheromone as an indirect form of
communication known as stigmergy (Grasse, 1959).

24

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Nest 1 <f > r Food

Figure 2.10: Two Paths with Different Length between Nest and Food (Wu et al., 2008)

Figure 2.10 illustrated two paths with different length and the number of ant
which is divided into two equal groups. The ants that follow the shorter path will
make more trips to the destination, thus leading to a higher intensity of pheromone.
The other group eventually will change their route to the shorter path after pheromone
intensity value is calculated at each branch node.
According to Kugu and Sahingoz (2013), ACO is the imitation of the actual
ant colonies that imitate the population behaviours in an electronic simulation. It is
used to deal with some compounded optimization problems by manipulating artificial
ants. It has been used over the decades in applications domain that aim to operate in
real world challenge. However, there are subtle distinctions between artificial ants and
actual ants. An artificial ant:
• Explore in a discrete and electronic instance.
• Has its own memory and possess a knowledge base.
• Can sense the surrounding environment aside from depositing
pheromones.
• Able to lay pheromones not only while on-the-go but also after
completing a travel.
• Able to communicate synchronization for tactical movements with
other ants.
• Able to apply additional abilities.
ACO algorithm has been researched that result in many extensions. To name a
few, the five main ACO algorithms are Ant System, Elitist Ant System, Rank-Based
Ant System, Max-Min Ant System and Ant Colony System. The algorithms have
different mechanism which are used for different extension especially pheromone as it
is the main part of ACO algorithm.

25

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Other type of ACO that is mainly used is Rank-Based Ant System (Kugu &
Sahingoz, 2013). This ACO extension is the enhanced version of Ant System in which
every artificial ant calculates the amount of pheromone value based on ant ranking
system. According to the concept, currently best-so-far artificial ants will deposit the
highest value of pheromone concentration on all occasions. At the end of every
iteration, artificial ants are recorded by measuring their travel distances. In every
iteration, only the best ranked artificial ants and the ant that generated the currently
best-so-far travel path can update pheromone value. The selections are made
according to their ranks.
The first example of ACO algorithm is Ant System (AS) (Dorigo, 1991;
Dorigo et al., 1991; Dorigo, 1992; Dorigo et al., 1996), which was proposed to solve
the well-known TSP (Applegate, Bixby, Chvatal, & Cook, 2006; Lawler, Lenstra,
Rinnooy-Kan, & Shmoys, 1985; Reinelt, 1994). Despite encouraging initial results,
AS could not compete with the state-of-the-art algorithms for the TSP (Liao, Dorigo
& Stiitzle, 2011). Other than AS, ACO is also used to solve numerous text
classification problems with different implementation. The research problems are very
identical but the methods contribute in different discipline (Tian, Yu, & Xie, 2008).
Ant Colony System (Dorigo & Gambardella, 1997) has been contrary exploited to AS
by using Nezamabadi-Pour, Saryazdi, and Rashedi (2006) methods, as explained by
Tian et al. (2008). The exploits are contrary to cAnt-Miner that is coping with
continuous attributes (Otero, Freitas, & Johnson, 2008). It has been shown that the
above fundamental difference is crucial to the respective designed ACO-based
algorithms (Dorigo et al., 2006).
In other research for Ant-Miner, a research named cAnt-Miner (Ant-Miner
coping with continuous attributes), incorporates an entropy-based discretization
method. This is done to cope with continuous attributes during the rule construction
process (Otero et al., 2008). cAnt-Miner has the ability to create discrete intervals for
continuous attributes "on-the-fly". Therefore it can take advantage of all continuous
attributes information, rather than requiring a discretization method to be used in a
pre-processing step. Although many Ant-Miner variations have been proposed - as
reviewed in (Freitas, Parpinelli, & Lopes, 2008) - none of them can discretize
attributes "on-the-fly" (during the rule construction process). This was proposed by
Otero et al. (2008). Other research also has started to combine SVM to ACO as
mentioned by Ressom et al. (2007).

26

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Initialize colony parameters |
Initialize pheromone matrix r with rmii for each pair of cities
for / = 1 to ni do
for k = 1 to m do (TOUR CONSTRUCTION) |
Place anti on a randomly chosen city
while the n cities are not all visited do
| Move anti to next city according to state transition rule
Compute length L* of the tour F* produced by ant^
for A - l i >mdo{l N -•! SI \RA Mr
while 7* is improved do |
for all 0 < a < n and 0 < h < n and 0 < c < n do j
Delete arcs {a, a + 1), (b. b + 1 >, (c» c + I)
Paxluce F' by reconnecting partial tours with other arcs
Compute L'
If V < Lk then
I Update Tk with F'
for A = 1 to m do |
if Z^ < L„ then
1 Update Ttl with Tk
it Lt, < LKi then 1
! Update TKi with Tts |
Evaporate r and update r for each pair of cities of Tlt or Tgi |
Control !„»,„ < r < rmat and update r with trail smo hin - mechanism |

Figure 2.11: MMAS Pseudo-code with Local Search (DeleVacq, Delisle, Gravel, &
Krajecki, 2013)

ACO algorithm has been used to create and tackle complex computational
problem such as TSP and text classification. Max-Min Ant System (MMAS) (Stutzle
& Hoos, 2000) is one of the versions of the algorithm that has been used widely as it
is one of the most effective ACO algorithms at present time. Its metaheuristic use
general structure of memory and main processes that integrated many of the
algorithms. Figure 2.11 demonstrates a short pseudo-code of the MMAS. In the
algorithm design concept, number of cities n is matched up with the number of ants m.
The ants visitation creation mechanism is done in each consecutive ni iterations. To
that event, every of ant antk is initially assigned on a sparse and undetermined chosen
city. After that, on every searching move, antk plans its tour Tk that rapidly
implementing a status transition rule to select the cities that will be assigned to its
tour. The selection is done on the unvisited city list. When all artificial ants have
finished building their tactical tour, pheromone is renewed based on certain rule that
agree with two objectives. The first one is to expand the influence of arcs assigned to
the foremost solution searched at current time Tgi or to the best quality solution of the
present iteration Tu. Also, the influence value is used to minimize number of the

27

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

instances that have not been operated. In order to avoid pheromone stagnation, the
value is retained between lower and upper limit value for each arc. The limit value is
also predetermined at maximum value in order to support tour exploration of the
possible search space in the initial part of the algorithm. On the other hand, MMAS
incorporates a trail-smoothing process which also advocates tour exploration by
incrementing the probability of arcs selection with minimal pheromone strength.
There were a number of studies on parallel ants construct that tried to perform
the artificial ants travel creation phase on most of operating aspects which were
carried out by Bullnheimer, Kotsis, and StrauB (1998). The proposals consist of two
parallelization mechanisms for the AS particular to distributed-memory architecture
and message passing. The aim of the first mechanism is to reduce time and
communication overhead. This is done by allowing the algorithm to execute a
predetermined number of iterations without transmitting communication message. The
second mechanism is a synchronous and low-level mechanism. It is designed to speed
up computational operations by assigning artificial ants to processor of a master-slave
relationship. In each iteration, the master propagates the pheromone status to its slave
that will perform operations on their tours in parallel fashion. They will also inform
the result back to the master. A certain amount of time is spent for these global
interactions and synchronizations which can be assumed as noticeable overhead. The
authors mentioned that the partially asynchronous design is favourable due to better
reduction of communication overhead. A parallelization implementation with ACO
algorithm has been done to select term. The work was done in order to analysing
longer documents (Janaki Meena et al., 2011).
ACO algorithms are based on population optimal searching algorithms.
Therefore, they should rapidly travel for solutions until stopping criteria are found.
Normally, there are four distinctive stopping criteria (Kugu & Sahingoz, 2013) that
can be applied (the criterion can be used individually or combined):
• Time limit: the time bound is set up for maximum searching time
• Quality limit: searching is continue until lower bound solution is found
• Stagnation limit: searching is continue until the pheromone reach a
stagnation state
• Iteration limit: searching is set up to run until maximal solution or
lifecycle of the ant is reached

28

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

2.5.1 Pheromone Reading

Pheromone plays a significant role to represent the strength of relationship

between each term in the node. Other comparable wrapper method such as GA has
different way to calculate node to node or terms relationship. In every iteration, the
algorithm will select a default number of the best candidate from the population after
evaluating every individual that runs a mutation. In this case, the mutation values are
the way to deal with terms relationship. However, the method is more complex to be
implemented and additional steps to select mutation candidate may be required. ACO,
on the other hand, uses randomized selection with optional heuristic and it is simpler
to be implemented.
In ant colony method, pheromone concept is the main part of the algorithm
(Pahuja, Chhabra, & Khokhar, 2012) to find quality solution. Whether the ACO is
modified to solve different research problem, the pheromone part is included as the
basic component to implement ACO. Referring to real ants, pheromones are deposited
along the way as the ants travel to forage foods. It is important to note that the
pheromone must be linked to the nest as the initial starting point. However, the
pheromone implementation can have random starting point in ACO concept as the
goal is to find the shortest path at any point of solution findings. Two dimensional
arrays are used to store pheromone information to represent the pheromone
concentration at any time. As the real pheromone is exposed to open air, it evaporates
constantly (Ivan, Svetozar, & Jerzy, 2009; Mavrovouniotis & Yang, 2013) over time.
Similarly, a default value of 1% (could be adjusted as per different research) of
pheromone concentration is deducted for every cycle or iteration, every time artificial
ants 'die'.
In a process to construct classification rule, artificial ants need to lay a new
pheromone value in every category that is chosen for each term nodes. Adding a new
pheromone concentration requires artificial ants to check and compare the next term
node pheromone of the term node category. A comparison is needed to calculate
heuristic value that has unique relationship between the current visiting term node
category and the next term node category. The values are then summed up to get the
total heuristic value and each heuristic category is converted to a percentage value. A
random floating point value is generated that has a floating point value from 0.0

29

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

(inclusive) to 1.0 (exclusive). This is where the percentage of pheromone influences
the value to be picked by random generated value.
There is a subtle different on how different the size of area will affect majority
on the decision outcome. In regards to a blank A4 paper, converted percentages can be
described as a region that is divided on an A4 paper. By randomly dropping a pencil
tips on the paper, a larger region will get the most drops over a fixed amount of tries.
The random drop does not necessarily happen at the first time or the second time as
other predetermined number but certainly a fair random will guarantee the ratio region
drawn on the paper. This is similar to an artificial ant which does repetitive cycle
while the ratio of the region drawn is based on the percentage of the pheromone across
the two categories. Because the algorithm has an evolutionary development, the
pheromone ratio keeps on changing and adapting to affiliate with solution searching
pattern.

2.5.2 Evaporation

Pheromone that is used by a real foraging ant evaporates in open air. Other
than to avoid the path from being all sticky and wet, pheromone evaporates as a
strategic mean to inform other ants that found the laid pheromone. Given that the
pheromone is laid in the same area and time, pheromone evaporation happens at a
constant rate (Ivan et al., 2009; Mavrovouniotis & Yang, 2013). How high the
evaporation rate happen is not important but the relative attribute matters to represent
meaningful information. Since pheromone evaporates continuously, longer path
suffers less frequent refresh thus giving an impression to other ants that the path is less
efficient to use. When they arrive at a diverge point on a path, a smell concentration
comparison is done to evaluate which path has less distance to a food source.
Evaporation also plays an important role to reset the less optimal solution so that the
path that had been visited in the previous cycle could gradually decrease. Note that
beside all of the term nodes that artificial ants visit, all of other nodes are affected by
evaporation calculation. Pheromone evaporation could continuously happen until it
depletes to a certain preset minimum cap which is manually set by user (Ivan et al.,
2009; Mavrovouniotis & Yang, 2013). The same situation could also be observed to
real pheromone. In the searching algorithm, all of nodes that reach minimum cap will

30

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

have equal chance to be chosen again by other random decision of artificial ants. It
works as a newly or refresh starting point in which all nodes that reach minimum cap
have a potential to lead to a path that has better solution than the highest current path
score.

2.5.3 Stagnation

Compare to GA, ACO algorithm which includes random base heuristic and
pheromone concentration has a chance to overcome two peaks with different height
solution. The algorithms can have different result when overcoming two peaks with
different height. In the case of ACO, there is a chance for the algorithm to reset from
lower peak to higher peak when stopping criterions are not fulfilled yet. However,
when lower peak is found by majority of individuals, they will stay at that point
because mutation mechanism cannot influence the majority to take different
exploration. More of the details are explained below.
Both algorithms are able to choose the better iteration if there are more than
one solution which exist in the search space. ACO algorithm can have a random
chance to make different path searching. The heuristic bases of maker allow the ants
to always choose a path that diverge from the highest pheromone concentration. The
pheromone value is having an indifferent concentration between other pheromone on
different path. The divergence usually happens at the earlier stage of searching when
all pheromones are having the same initial value. The other situation happens when it
is in a stagnation state where pheromone values meet its maximum cap value in a
continuous iteration. In stagnation state, solution quality has hardly increased because
the artificial ants will look for other possible way that is different than the current
path. Stagnation could happen at a peak because pheromone concentration has
bounced back to maximum value. The artificial ant that continuously laid more
pheromone will lead to a saturation that stalls pheromone concentration. Pheromone
evaporation happens at all time and this makes the artificial ants laid pheromone on
different path (Ivan et al., 2009; Mavrovouniotis & Yang, 2013). If they found a path
shorter than the current optimal one, the new path will get a stronger pheromone
concentration than the previous one over time. Therefore, a new peak could be
discovered over the cost of excessive cycle after reaching stagnation.

31

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

2.5.4 Factors in Choosing ACO over GA and other Techniques

Based on previous researches that have been explained in chapter 2.3, GA is

found to have ample performance in solving text classification problem. GA has been
compared with Ripper, SMO, C4.5 and BioHEL. The performance is used to proof
GA effectiveness in text classification. In choosing bio-inspired algorithm, ACO can
be fairly compared with GA because both of these algorithm are wrapper methods. In
choosing ACO as the research classification method, the following criteria have been
considered:
ACO heuristic is simpler than GA and SVM because the use of random value
in heuristic calculation is simple and costs a minimal overhead.
ACO has an algorithmic criterion to be adapted to concurrent execution (Putha
et al., 2012). SVM is a statistical algorithm thus is not comparable to bio-
inspired algorithm.
ACO stagnation mechanism has potential to overcome multiple peaks solution.
Real ant colony has an organized and systematic community. For concurrency
objective, multicore-multithread environment can adapt with ant organization
in which the queen, worker and soldier ants are assigned with respective tasks.
Adaptation of multicore-multithread environment can increase ACO text
classification capacity and take advantage of processor technologies such as
additional L3 cache.
In order to handle computational cost, parallelism has been researched recently
by Cecilia et al. in 2013. Other similar research is done by Umbarkar et al. in
2014 by using dual population of GA. The works distinguished GA over ACO
in the aspect of concurrency, single population and the concurrency is
implemented on processor.
This research used ACO for classifying document content only. Therefore,
work done by Sara9 and Ozel in 2014 for experimenting html tags that include
title, tagged terms and bag of terms to observe more accurate classification is
the distinctive factors for comparison consideration.

32

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

2.6 Multicore-Multithread Environment

Multicore-multithread feature is the use of multiple thread on a modern

processor with more than one core. It has advantage in application running time and
benefits from effective multicore-multithreads implementation with higher scalability.
The scalability to integrate with other compatible feature such as concurrency, allows
it to be used across multiple computers in a network. This is different to statistical
classification approach, especially evolutionary algorithm or wrapper approach often
requires high computational cost. This is due to its repetitive heuristic calculation. The
concept of concurrency is a suitable feature to handle continuous solution searching.
Figure 2.12 shows how multicore-multithread reduce processing time by having
concurrent execution and memory access.
When only one thread executable at a time (assumed case)

Thread 1 Process 1 Memory access

Thread2 Process2 Memory access
Thread3 Process3 Memory access
Thread4 Process4 Memory access
Thread5 ProcessS Memory access
Thread6 ProcessB Memory access
Thread7 Process7 Memory access
Threads ProcessS Memory access

^: Elapsed time
^••••••••••••••••••••^^
Processing time

When two threads executable at a time (SPARC T4)

Thread 1 Process 1 Memory access

Thread2 Process2 Memory access
Thread3 Process3 Memory access
Thread4 Process4 Memory access
Threads Process 1 Memory access
Thread6 Process2 Memory access
Thread7 Process3 Memory access
Threads Process4 | Memory access

- . J Elapsed time
Processing time

Figure 2.12: Mechanism of Multicore-Multithread to Reduce Execution Time (Fujitsu, n.d.)

33

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 In real ant situation, the colony of ant is showing a sign of cooperative manner.
Therefore, the ACO algorithm can be further explored for breakable classification
task. There are a number of researches that explore ACO for concurrency or
parallelism (Cecilia et al., 2013; DeleVacq et al., 2013; Kugu & Sahingoz, 2013;
Sharma, Elmiligi, & Gebali, 2014). A research regarding numerical optimization had
used multicore implementation to differential ant-stigmergy algorithm (Korosec,
Vajtersic, Silc, & Kutil, 2013). In graphics processing units, parallel ACO had been
tested to find effective parallelization strategies (DeleVacq et al.). For continuous-
attribute classification rule discovery, a research had been done to utilize multiple
pheromones in ant-based algorithm (Salama, Abdelbar, Otero, & Freitas, 2013). To
test real time performance of multicore systems, a simulation for evaluation is created
(Sharma et al., 2014).
As the ACO algorithm is a bio-inspired ant community that runs their living in
organized life cycle (Bonasio et al., 2010; Emily, 2014), the algorithm could be
explored for further potential of concurrency concept in multicore-multithread
environment. Over a pure parallelism, multithreading that is available in the
multicore-multithread environment can be adjusted with multiple instances of
artificial ant. The major goal for the developed assignment to be adapted to the
indigenous system is to improve ACO execution time. In order to carry out the task,
artificial ant can be distributed to one particular thread at a time for specific instance
of artificial ants. The suggested idea is developed in order to try to replicate the real
situation of ants' imitation for which an ant can move at independent speed,
interruptible and deposit pheromone as a mean for communication (Ratnieks, 2015;
Wyatt, 2015). These situations can address the problem faced in running multiple
instances of artificial ant which a thread that process the ant could have different
priority, different status and need a potent, real time and repeatedly exchange state
information to another child threads. Distinguishable from parallel computing,
multithreading is a potential option that offers extended capability and flexible to cater
the multiple instances of artificial ant. Imitating real pheromone mechanism that is
effective for any other ants that track it, pheromone in multithreading is designed to be
volatile to a single copy that can be accessed simultaneously between multiple threads
of artificial ants. A test between single threaded and multithreaded version of ACO
show high potential results that decreases the classification execution time.

34

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

2.6.1 Threading Over Parallelism for Multicore-Multithread Representation

Focusing on multicore-multithread implementation mechanism, equating

multithreading to parallelism approach is not accurate (Moth, 2008). Therefore, one of
the two options must be chosen for multicore-multithread environment representation.
Both multithreading and parallelism can sometimes produce identical results. But
underlying execution procedure differences demand further discussion to make clear
of their effects on this research. This chapter explains the related conclusive
architecture on execution speed, concurrency, background tasking and compatibility
of multithreading. Explanation on why multithreading is more suitable to be adapted
over parallelism instances is given. In further principle, multithreading can have
different discussion where it elevates hardware implementation and also software
implementation.
Before discussing further on multithreading and parallelism, let's assume that
single core and multiple core machine can be treated the same for this discussion. This
is because both parallelism and concurrency of multithread can be implemented in
those situations. One might say that the resource management is different between
single and multiple cores. But the point of this discussion focused on the processing
part that has the same principle of resource management between single and multiple
cores.
For a single core machine, multithreading can be implemented on it but not
parallelism (Moth, 2008). Parallelism needs another real processer core to work.
However, thread switching can be done at software level combined with minimal
instruction set to use multithreading on a single machine. Figure 2.13 illustrate how
the task distribution difference between concurrency and parallelism implemented.

35

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Taskl r> Sub-Task 1 1

Task 2 Taskl -• Sub-Task 2

Task 3 -• Sub-Task 3

Concurrency - multiple Parallelism - a task is divided

task progress at the same into sub-tasks which can be

time processed in parallel

Figure 2.13: Difference between Concurrency and Parallelism

There are advantages that can be gained when using multithreading on

multicore machine. When the job needed to be executed with a single core,
parallelism can consume much more unnecessary resources compare to
multithreading. This is because multithreading offer concurrency option on a single
core that will save another activation of the next processor score. Contrary to
parallelism, multiple core must be activated on the request of task scheduling even
though the job is simple and can be achieved by a single core execution. Energy
conservation can be higher just to activate a core and to warm it up before used. In
term of efficiency, the overhead will cost more. Multithreading that operates on the
correct number of core in response to task request is more flexible to be compared
with unmanaged multicore activation.

2.6.2 Staying Idle

In response to the task scheduling request from an operating system, both

multithreading and core activation need to be prepared. Common operation or
operating system always have irregular request as to when multitasking is needed. The
pattern that we can observe sometime can be unexpected or just random. Therefore,
the least we can do to save CPU resource is one core that needs to be alive at all time.
While the request of multiple tasking needs to be run at the same time for parallelism
case, the remaining core will be activated. Operating system can enable or disable
permanently the remaining core activation (Aaron, 2011) for power consumption and

36

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

performance balancing. As mentioned before, activating a core can take time as an
overhead compared to multithreading but can be overcome with operating system
running mode. Most operating system will have a maximum number of core to be
activated when a user requested the operating system to operate at maximum
performance. Even when there is no task request, the core continues to stay alive
while consuming additional electric power resource. The benefit of this situation is
responsiveness of the system. There is minimal overhead for the CPU to communicate
the current status of core. For a more complex task with many cores in one machine,
the waste of electric resource to power up multiple cores can be multiplied to even
greater value. In terms of gaining further concurrency benefit, Graphic Processing
Units (GPU) can be used as additional processing hardware. GPU and CPU can be
combined to work collectively (Kugu & Sahingoz, 2013) as to cater heavy
computational workload before the needs to integrate a new machine to the network.
ACO model for multicore-multithread also has random pattern request. It is
designed to copy the real foraging situation where ants can be late, slower,
experienced sudden death or change the searching route. Therefore, we can assume
and implement similar situation with multithreading in which a thread is created
unequally (Scali, 2012). It may run faster or slower, waiting in queue or in sleep mode
and then die. There is a concern that the number of node is determined before
execution therefore all artificial ants will visit the same number of term node.
Therefore they will have the same execution speed and work in parallel. This is
inaccurate because all artificial ant need to do is to centralize their pheromone value in
which a thread will lock the pheromone access while updating new pheromone
concentration. When an artificial ant locks pheromone access, the other artificial ant
needs to wait. Then the artificial ant will release the resource and continue its travel.
The situation will leave uneven execution speed and also random. Any artificial ants
that come to lock the resource will be randomized and will be put in a queue.
Parallelism that can operate strictly at higher execution level will have more
complexity to deal with queuing and need to consent uneven artificial ant running
time. Sequential access demand on the same resource in which the pheromone would
rise further to deadlock issue (Nazeem, Reveliotis, Wang, & Lafortune, 2011).

37

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

2.6.3 Code Development of Multitasking

Another concern to the difference of concurrency is the code to generate

parallelism scenario. Code used on a single core machine that use multithreading is
ran normally. In writing parallelism code, programmer must identify which part that is
truly safe to run simultaneously and put the code into a block, then tag it for parallel
execution. As for multithreading, software developer needs to consider the overall
system and identify which part can be improved and have threads handle tasks
simultaneously. Usually, input, output and numerous array operations can be slow,
making it suitable to dispatch the few task running using thread.
As for multicore-multithread adaptation, using multithreading opens more
rooms for customization as it is more flexible to use. In ACO, there are a lot of
artificial ants to be run in the model, so they need to communicate to each other as
frequent as possible to update their pheromone values. At the same time, they also
need to apply systematic queuing to do pheromone calculation as they share the same
copy of pheromone array. As for parallelism that runs on multiple cores or multiple
computers, they need to use processes and manage task dependencies. For software,
task can be independent, breakable or need other task output that can make
multitasking challenging. Figure 2.14 shows an example of task dependencies that is
the essential design for multitasking. Inter process communication is much more
complex and heavier to use frequently because of the complex piping or pipelining
that introduce further overhead. But it is different when using thread over process
where it is far lighter and can appear by another call of threads, inside a process or a
much lighter proto thread.

38

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 ~

i 3
I1
5 \ !

Figure 2.14: Task Dependency Graph (Sharma et al., 2014)

For all of the above explanation, multithreading is considered more suitable to

represent multicore-multithread environment simulation as compared to parallelism.
The comparisons of these differences are done as they are significant to clear out
many debates on how to simulate multithreading. Multithreading mechanism seems to
be similar to a complete parallelism concept. The content stated above also describes
multithreading advantage over parallelism in regard to relate it to multicore-
multithread implementation. Many real ant situations could be simulated correctly by
using the multithreading flexibility and design concept.

2.6.4 Multithreading in Software Level

The first part of thread mechanism that is used in multicore-multithread

computing phase is multithreading at software level. Thread execution is a smaller
sequence or a minimal version of programmed instruction set that are available for
independent management. The management is handled by task scheduler which is a
part of an operating system notion. The implementation of processes and thread usage
could be unique from an operating system to another. Threads are usually created in
processes as a subset component that is assigned upon request with some exceptional
cases to some operating system. One common distinguishable criterion between

39

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

processes and threads is the memory usage or the communication between their
multiple instances. Multiple threads can be created within the same processes instance
and share the same hardware signals while a process is designed to be unique. It works
independently to another processes and consume different resource allocation. To be
specific, one or more threads access the same instructions or programming code which
represent the values of all global variables at all time of threads lifecycle.

2.6.5 Multiplexing Time Division

Multithreading on a machine with single processor is done by dividing time as

multiplexing (Scali, 2012). It can be implemented on multitasking system and the
processor will be given instructions to do threads swapping from the software
provided threads. Usually context switching can happen very fast under one second
which is fast enough (Scali, 2012) for a human to perceive that his or her single core
machine has multiple core instead. On the other hand, machine that operates
multiprocessor or multi-core system can create multiple processes that run at a true
concurrency concept. It is also possible for a parent process to create a new process
that run the same code with separate memory allocation and having unique process ID
which is known as forking (Napster, 2010). Similar to multithreading, an operating
system can manage threads behaviour to run concurrently using multiple cores with
every processor core running the thread simultaneously. It can be more efficient for
created threads to be dedicated for each CPU core. To achieve such situation, the
operating system can use hardware threads which have been developed over a decade.
The hardware threads however are usually not open for software engineer's
manipulation under normal circumstance and have multiple layer of abstraction. We
can safely assume that hardware threads are different than software threads as we had
mentioned earlier. Software threads are solely logical to software construct when the
abstraction is managed by the operating system itself. Some processor designs provide
more than one thread usable to OS per CPU core such as Intel Hyper-threading
technology (Hoffman, 2014). Even the processor has no consciousness of software
threads and does not realize about their creation.

40

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

2.6.6 Schedule of Multiplexing

Nowadays, process scheduler used by major operating systems could support

time-division or time slice (Scali, 2012) multiplexing and multicore processor
threading. The internal decision made by core layer of operating system allows
programmers to instantiate threading by supplying required methods or functions
(term use varies by programming language). The methods can be channelled through
system exposed part called interface. A part of threading usage is discovered as kernel
thread that operates through kernel layer of operating system. Light Weight Processes
(LWP) is a subtype of process kernel that does threading which shares the same
memory spaces and communication (Napster, 2010). For flexibility, programmers can
design their threading program by having user-space threads that can make use of
timers, signals, pools and other user define functions. Those operations can be used to
interrupt or change the flow of program execution, leveraging a sort of fast time-
slicing.
Multicore-multithread on the other hand can be managed independently by
time-division multiplexing and multicore processor threading. Situation of using
single core and multiple core processor machines that communicate through the same
memory could be designed as multithreading. For that situation, multithreading offer
enough setting and multi-layer abstraction. As an additional advantage from threading
implementations, LWP can also be substituted by readjusting the thread creation
setting. LWP is to be created mostly at minimal overhead than forking (Napster,
2010). This is also a proof that multitasking demand is favourable to smaller and basic
division concept offered in multithreading.
Normal written code need to be adjusted first before executing single tasking
code into multiple parts that are free from each other. Then, scheduling algorithm can
be applied. Any task in which its input is dependent to other task output cannot be
executed concurrently. Therefore, these tasks are required to be placed consecutively
in task schedule. After this step has been applied, many other factors could be taken
into account for scheduling mechanism.
As explained by Runger (2009), scheduling algorithm can be categorized into
three classes, which are:
• Layer-based scheduling algorithms, which scheduling is implement in layers
right before distribute the task to processor cores.

41

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 • Allocation and scheduling based algorithms, which first distribution is
implemented and then scheduling is applied.
• Configuration based scheduling algorithms, which the schedules are created
composed on an initial set of feasible settings of every task.
Below is a Table 2.1 that explains the scheduling algorithm in more details:
Table 2.2:
Different Scheduling Algorithm Categories (Sharma etal, 2014)
Category Phase Description
Layered-based Shrinking Phase Tasks are merged to
Scheduling Algorithm minimize solution
searching through
algorithm
Layering Phase Demultiplex the task into
layer of independent task
Layer Scheduling Phase Map the schedule for each
layer that have
independent tasks
Allocation and Allocation step Assign core number to
Scheduling Based each tasks
Algorithms Scheduling step Identify the start time and
processor category of tasks
Configuration based One step Task schedule is created
Scheduling Algorithms related to the initial
configuration of every task
for example running time

There are some mapping algorithms to schedule task based on type of

multicore processor architecture that had been explored by researchers. Particular to
System on Chip (SoC) multicore processor, Chen, Liao, and Tsai (2013) had explored
scheduling algorithm that use heterogeneous system. The algorithm is designed to be
implemented to a CPU that has master-slave feature. Known as Heterogeneous Dual-
core Scheduling Algorithm (HDS), the distribution job to map task of a schedule can
be implemented on processor and coprocessor. It also uses pre-emption points to
render void unsuitable total time of task blocking status.

42

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Below are several criteria that point out the differences between threads and
traditional multitasking used in processes:
Processes are created mostly independent in major cases while threads are
created and instantiated by the process itself (Napster, 2010).
Processes typically have higher amount of state values that keep signals from
kernel than threads but threads could select the already available process state
other than their own.
Processes must have dedicated or concrete address space while threads can
extend process spaces shared among themselves (Napster, 2010).
Threading can communicate directly to each other while process must initiate
and maintain inter-process communication (Napster, 2010) that similar to
buffering within pipe.
Context switching which involves saving and loading state from memory
among threads (or known as inter-thread communication) is often faster than
processes switching (Napster, 2010).

2.6.7 Multithreading in Hardware Level

Structure of multicore processor with multithreading depends on its

architecture. Aside from its implementation in software level, there is also mechanism
that exists at hardware level in multithreading. It is mostly developed by processor
developer to offer support for an operating system in order to utilize its usage. The
technology allows one or more users to interact to the machine by exposing the
required status to an operating system. Different from straightforward parallelism,
multithreading can avoid multiple copies of software instances from running when
multiple requests are issued. Several developments of hardware support are installed
with factory setting to have threading run efficiently. Note that this is not to be
confused to multiprocessing or multi-core system where they are more towards actual
parallelism. In software level, threads are created faster as a subset that share the same
resource as its creator such as process or parent thread (Napster, 2010). But at
software level, the processor is designed to share its resource for every core to threads
in hardware level. The most common concern of sharable resources is CPU caches
and Translation Lookaside Buffer (TLB). TLB can cache virtual memory translation

43

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

into physical address that is used recently for faster data retrieval from memory block
(Haughn, 2014). For utilizing processing resource, multiprocessing system is designed
to spread execution into more cores. However multithreading in this discussion is
designed to focus execution sequence into a single core which gives distinguished
mechanism. Both implementations are independent from each other thus they can be
combined and integrated into the same processor. For example, multi-core processor
with Hyper-threading from Intel, Intel i7 5960X Extreme Octa Core CPU has eight
cores and 16 logical threads released in 2014. In the end, we can gain a processor that
has multiple cores and every core is capable to execute concurrent multithreading.
Term throughput computing has been used by developers and computer scientists who
refer to the maximum amount of execution done over a time unit. Many developers
use multithreading on a single process to execute user define software in parallel to a
multiprocessing system.

2.6.8 Advantages and Disadvantages Multicore Processor

One of the multicore-multithread main designs is to allow multiple threads to

communicate and share resource on its own global space which is dedicated to handle
their requests. The technologies offered in multithreading is able to handle these tasks
much cheaper in term of cost compare to the same function offered in parallelism.
This is especially true in multicore or multi-processor system (Napster, 2010). When
there are more than one user who instantiate the same process, multithreading can
create child thread to serve every user. This can save another multithreading over
process creation with minimal instantiation overhead and resources (Napster, 2010).
Since multithreading provides the option to apply concurrency concept on single core,
multithreading concept becomes more appealing with multicore machine. This is
because the machine concurrency ability increased by double or triple fold. As a
result, overall multicore-multithread system responsiveness has increased
dramatically.
Another concern to processor operation is a cache misses. Since there is a large
difference of access speed between processor cache, main memory and other storage
medium, cache miss is a significant problem in execution sequence. Processor cache
offers the fastest access to the processor because it is mounted on the processor itself

44

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

as compared to other memory medium such as Read Access Memory (RAM) and
Solid State Drive (SSD). A cache miss can be dire as the processor will stall from
execution up to 300 cycles. The number of cycle stall varies from different processor
architecture. Processor needs to flush or dismiss all of subsequent instructions and
needs to find the missing variable such as indirect address or input and output
operation. After flushing is done, processor then needs to start again with a new
branch of execution which consumes a lot of cycle. CPU architecture that adapts
longer pipeline such as Prescott processor has 31-stage pipeline (Gennadiy, 2015)
could suffer more drawbacks when a miss happens. This happen because it takes more
time to fill up its pipeline for all stages. To solve the stall issue, processor that has
been pre-installed built in threading technology can pass the execution of multiplexing
to other threads. This is important so that the processor can still continue its execution
to execute different code while waiting a paused thread to finish from fetching and
decoding missing code.
In respond to cache miss, multicore-multithread can be faster on overall
execution by applying the solution offered in multithreading. We would experience a
waste of whole core throughput when a cache miss happen as parallelism would wait
for each core to run concurrently, most of the time. A developer needs to consider
thread switching to execute other task while waiting for another thread to finish an
atomic operation (see example update of atomic operation in Figure 2.15). The thread
needs to finish executing it in a single step relative to other threads (Preshing, 2013).
From a broader point of view, multithreading is fulfilling its aim for concurrency but
the underlying mechanism uses manual adjustment by developer. Pure parallelism
would have searched on how the whole code can be synchronized. The search requires
pheromone sharing and the updates need to be updated manually thus delaying the
other core execution.

45

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Atomic Instructions
Atomic approach
instructions
required Phcromone Matrix
City 0 . . . City j . . . Cityn
CityO
1 i
\
•

• \ \ 1 /
• /
Cityi
•
• •
#
J* 3J-
1

i
• * *

S
••

Cityn / i
-i— J 1
Thread 0 Thread i Thread m

i j \ j h_ j
Tour

Ant0's tour Ant's tour Ant's tour

Figure 2.15: Pheromone Deposit with Atomic Instructions (Cecilia et al., 2013)

2.6.9 Backup Slower Threads

In a race condition (Napster, 2010) that occur in multithreading, it is inevitable

to assume that there will be threads that are slower or the slowest among other threads
on the same channel. The same thing happens to real ant in natural environment, each
of the ant does not necessarily have the same physical condition thus their efficiency
vary according to their type. Because ACO uses multiple instances of artificial ants,
the algorithm used must be designed to have the artificial ants to cooperate with each
other. The faster one can support the slower one to create a balanced overall time
completion. This is achieved due to the use of centralized and single copy pheromone.
Theoretically, the faster thread will complete more cycle than the slower threads at
same amount of execution time. Given an artificial ant will update pheromone value in
every iteration, slower artificial ant will use the latest value of pheromone that is
updated by other faster thread. Using this way, all slower thread will not have
deprecated pheromone reading and will not be left behind in calculation part. Every
thread will contribute to additional pheromone concentration according to its update
frequency. If pheromone variable is not sharable and is localized for every artificial
ant, it will not be a built up solution for cooperative approach. Another concern with

46

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

speed difference between multiple threads is the queue waiting time efficiency may
increase. As long as multiple threads are using and updating thread at different time,
queue can be shorter and congestion is avoidable compared to parallelism. Faster
threads could be accessing pheromone variable earlier than the slower threads and
release the variable lock earlier too. Before slower threads are placed in a queue, the
chance of the faster thread to finish updating pheromone value is high. Therefore a
slower thread is no longer needed to wait for the pheromone resource to be available
again. Therefore, a mix of slower and faster threads could have closer total execution
time gap compare to a set of faster parallelism execution.
Identically, the situation in thread race condition is similar to real ant in natural
environment. Any child thread might request status at any time that can lead to data
integrity issue as what could happen in thread race condition. It can happen while
updating global variable by many requests from multiple thread (Napster, 2010). In
fact, multicore-multithread feature has been offered to open business world and
organizations which comprise of different operating system and machine
specification. From the cross platform that is widely implemented in practical,
multicore-multithread environment has the flexibility to handle problems in program
execution. Multithreading over parallelism has suitable feature as explained in the
previous paragraph to achieve the same flexibility over multicore parallelism. In
multicore-multithread management model, main process often provides services that
are specific to inter-thread communication. It is effectively encapsulating confidential
information by every thread created. The equivalent usage for single thread
implementation in ACO multicore-multithread model is the rule of variable set and
pheromone value. While pheromone is centralized and shared among multiple threads
during execution, rule discovery of each artificial ant is not reachable for other child
thread access. This restriction is important to make sure that the data is integrated
properly during execution. Also, main process can schedule up multitasking partial
execution code to spread sharable global variable between child threads. This can be
done to savage another portion of efficiency as achievable in inter-thread
communication for sharing memory address space (Napster, 2010). Sharing resources
is an important feature which makes multicore-multithread environment friendly and
secure to use.

47

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Scatter to Gather <£ Tilling
City 0 . . . C i t y j . . . City n
Thread
f Bl0ck

Ante's tour Ant/s tour Ant's tour

Figure 2.16: Scatter to Gather Transformation for Pheromone Deposit (Cecilia et al., 2013)

The length of an artificial ant's travel path could be higher than the maximal
number of total thread instances that the thread block can support (NVIDIA, 2010).
The algorithm proposed by Cecilia et al. (2013) is designed to avoid this problem by
having the empirically illustrated optimal thread block presentation. Then, the length
of the travel path is divided into tiles. The problem for this calculation happen because
the length of the travel path cannot be divided when tiling is used. The issue has been
solved by leaving padding into ant travelling array so that warp divergence is properly
avoided (see Figure 2.16).
According to Cecilia et al., (2013) redundant operations to device memory can
be prevented by exploiting the advantage of symmetric structure of TSP. Therefore,
the number of thread instances can be distributed into two parts, thus effectively cut
the number of memory operation of the device by half. The particular reduction
edition of the algorithm minimize the total time of access operations to both shared or
device memory and also implements tiling. The number of access operations per
thread stays the same.

48

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

2.7 Summary

This subchapter highlights the conclusion of literature review. Many of recent

research has been done recently but they are differ to this research in aspects of
concurrency concept, hardware used to move multiple artificial ants and the use of
dual or multiple population. Recent research by Sara9 and Ozel in 2014 includes an
implement of ACO to web page URL. From the web tag such as title, tagged terms
and bag of terms, the information effectiveness are investigated over classification
performance. Therefore, ACO has been compare to single population GA in
unparalleled algorithm. ACO in this research also differ to the work done by Janaki
Meena et al. in 2011 and 2012 in term of parallelism while work by Cecilia et al. in
2013 run its parallelization on GPU instead of CPU. Work done by Umbarkar et al. in
2014 use dual population instead of one that is not in scope of this research.

49

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 CHAPTER THREE

RESEARCH METHODOLOGY

3.1 Introduction

The research framework as shown in Figure 3.1 summarizes the methodology

used in this research. Phase 1 begins with acquiring knowledge for the core process.
Various readings on ACO were done in order to understand and review the previous
researches and the fundamental content of ACO and multicore-multithread computing
as well as text mining. Phase 2 begins by outlining the main concept of ACO. The
second objective of this research is expected to be achieved during this phase. The
whole phase covers the development of the main idea and produces a working
software that depicts ACO implementation which integrated with multicore-
multithread model for text classification. The last phase is the integration of
multicore-multithread with ACO model testing. It is an important research procedure
that will evaluate the model and determine the accuracy of the main idea as well as
producing the result to support the idea and objectives.

50

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Q START J
Phase 1
| Objective 1

Knowledge acquisition
Phase 2 Objective 2 j
•
Text data collection and identification

•
Data pre-processing

T
Development of Ant Colony Algorithm

•
ACO text classification

T
Set up multicore-multithread environment

i
Multicore-multithread ACO model

ir

Phase 3 Evaluation on Multicore-multithread ACO model

Objective 3
lr
Documentation

(^ END ^)

Figure 3.1: Research Framework

3.2 Phase 1: Knowledge Acquisition

This phase involved the acquisition of literature review and information which
are related to text classification, GA, ACO and multicore-multithread environment
issues. The information is important to make sure that this research is up-to-date and
is heading towards a valid direction. Table 3.1 shows the summary of the collected
information:

51

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Table 3.1:
Summarization of Relevant Issue in Literature Review
Subchapter Issues
Text classification Text classification in raw form cannot be processed
directly by common classifier, therefore pre-processing
step is required. During the pre-processing step, the
documents are represented by feature vectors (Zhang &
Pan, 2011). On other issues, most of the related works on
text classification focus on trying to eliminate the problem
of data sparseness (Sriram et al., 2010). In respond to the
problem, computer science experts are required to develop
new algorithm or use existing one to mine scattered and
unstructured data.
GA GA can search the optimal characteristic variable by giving
a set of document. GA is run to analyse critical operational
problem on signalized intersections and eliminate separate
lane groups of various oversaturated situation (Liu &
Chang, 2011). GA can be integrated with effective LSI-
based text classification which is a highly computational
expensive technique. Dimension reduction is another text
classification problem. GA has been used in dimension
reduction research and compared with other techniques for
their performance on reduced curse of dimensionality in
text classification (Gasanova et al., 2014).
ACO In ant colony method, pheromone concept is the main part
of the algorithm to find quality solution (Pahuja et al.,
2012). ACO uses randomized selection with optional
heuristic and it is simpler to be implemented. Pheromone
evaporation that happens at all time will evaporate ongoing
saturated pheromone and makes the artificial ants laid
pheromone on different path (Ivan et al., 2009;
Mavrovouniotis & Yang, 2013). If they found a shorter
path than the current optimal, the new path will get a
stronger pheromone concentration than the previous one

52

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 over time. Compared to GA natural selection method, ACO
algorithm which include random base heuristic and
pheromone concentration has a chance to overcome two
peaks with different height solution
Multicore- As the ACO algorithm is a bio-inspired ant community that
Multithread runs their living in organized life cycle (Bonasio et al.,
Environment 2010; Emily, 2014), the algorithm could be explored for
further potential on concurrency concept in multicore-
multithread environment. The suggested idea is developed
to replicate the real situation of ants' behaviour for which
an ant can move at independent speed, interruptible and
deposit pheromone as a mean for communication
(Ratnieks, 2015; Wyatt, 2015). For a single core machine,
multithreading can be implemented on it but not in
parallelism (Moth, 2008).

From the table we can summarize that GA has valuable method for
comparison with ACO. The resulting classification accuracy can be useful in its
relation to the chosen algorithm in aspect of the given advantage. GA has organized
and clear approach which adapt in expanding population generation and tournament
concept. For ACO, its primary attribute is the use of pheromone which rated as
stigmergy that help the artificial agents to operate in simplistic manner. From the
literature review also shows that concurrency concept has potential for further
improvement to speed up execution process. Parallelism has been explored by many
previous researches and use of different platform and hardware utilization make the
research unique from each other.

3.3 Phase 2: Text Data Collection and Identification

After acquiring knowledge for the whole process overview, the next step is to
collect and identify data for text classification. Some of the criterions of the desired
documents as data set are publicly available with easy online access and the category
could be general and simple to understand. The 20 Newsgroups data set (Jrennie,

53

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

2008) is identified to exceed the requirement and the data is available in a raw or
compiled (removed redundancy) nature.
The collection of 20 Newsgroups data sets has a total of approximately of 20
thousand newsgroup text documents. It is redistributed to nearly (fair composition) 20
different newsgroups. This is one of the rare features needed as the classification in
this research require the same number of document run for each category to be tested.
Other collection can have large margin for the total number of file in which the first
category could have 100 documents and the other category contains of 1000
documents. The collection has become a popular data set for machine learning test
input in text classification and text clustering (Jrennie, 2008). The following are some
of the categories:
1. Computer graphics
2. Computer operating system: Microsoft Windows miscellaneous
3. Talks: politic miscellaneous
4. Talks: politic mid-east
5. Sciences: space
As stated in the scope of research, the experiments for this research will only
run on two categories. They are the computer graphics and computer operating
system: Microsoft Windows miscellaneous. Many of the document contents are taken
from emails. Thus, there is a need for an effective pre-processing to filter out
redundant fields and terms.
The data can be found as a subgroup of a directory to a broader category when
classification on a subsequent topic is required. For the feature, computer graphic and
computer operating system can be closely related in the aspect that they are under
computer subjects, but they discuss different contents. Therefore, there are some
challenges for the classification algorithm to effectively distinguish the given
categories and can be very useful in real life problems.

3.3.1 Data Pre-processing

There are two main phases in text classification. The first step is the pre-
processing phase where it produces the working input for the next phase. The ACO
will use training set document to analyse text classification rules and produce ACO

54

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

classification model. The model will be used to classify the first seen uncategorized
documents. The process is stated in Figure 3.2:

(a) Training

Category •w
r
ACO
feature algorithm
text -> selection —>C 1 1 1 1 1 1 r — •

terms

input

0)) Testin g
>f
feature ACO two
text *w
r selection —»cM i l l 1 h~> —>
model category
terms
input output

Figure 3.2: Text Classification with Classifier (ACO) Framework (adapted from Bird, Klein
and Loper (2009))

Figure 3.3 illustrates the main tasks in pre-processing phase. From a set of
document, the text files from The 20 Newsgroup documents are accessed one by one
in pre-processing phase. Based on fold iteration number of current execution, the
content of text files are split into training and testing parts. Training document cover
90% of the whole content and testing set cover 10%. This means that 10-fold set cover
complete rotation of testing set. Tokenization is done when converting text to the
input of frequency table. It is a process of dividing a stream of text into a list of
symbol, words or other meaningful units recognized as tokens. In this research, the
token is called as terms and artificial ants will visit them and decide their affinity
towards a category. The token is also act as a key in frequency table therefore its
existence is unique to the other tokens. Originally, they are separated by a white space
in the text documents. By using JAVA native split method, the white space are
removed automatically and the method returns a list in form of array with String type.
Comparisons to distinguish their uniqueness are done using JAVA provided local
method. In split method, numbers only and words with number combinations are also
dropped as the scope of this research is to exclude numeric tokens. In the end of the
phase, frequency table is created.

55

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Read The 20 Newsgroup documents

Split into training and test array based on nth fold

Text tokenization and filter

Create frequency table

Figure 3.3: Main Tasks in Pre-processing Phase

The file content can have a number of irrelevant terms that does not give
meaningful information to the intended category or influence for the searched
criterion. For example, Figure 3.4 is one of the text files from 20 news group
collection of email, with graphic category:
SPONSOR: NESS (Navy Engineering Software System) is sponsoring a
one-day Navy Scientific Visualization and Virtual Reality Seminar.
The purpose of the seminar is to present and exchange information for
Navy-related scientific visualization and virtual reality programs,
research, developments, and applications.

PRESENTATIONS: Presentations are solicited on all aspects of

Navy-related scientific visualization and virtual reality. All
current work, works-in-progress, and proposed work by Navy
organizations will be considered. Four types of presentations are
available.

Figure 3.4: Sample Content of Graphic Category Document (Jrennie, 2008)

Consecutively, Figure 3.5 shows one of the text file from Microsoft
miscellaneous category:

56

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Well, someone at Microsoft told you wrong. Windows will use a swap file larger than the
recommended size. Last summer I went to a Microsoft Windows 3.1 Technical Workshop and they
distributed A LOT of information that the general public might not be aware of. Anyway in the
main book they handed out they have a section on "Creating larger than recommended swap file".
I'll quote the information

If you create a permanent swapfile larger than the recommended size, you will receive a message
telling you that Windows will not use anything larger than the recommended size. THIS ERROR
MESSAGE IS INCORRECT, we will allow the use of the larger swapfile, up to four times the
amount of RAM on your machine. - (file 9482)

Figure 3.5: Sample Content of Microsoft Miscellaneous Category Document (Jrennie, 2008)

During the scanning, the words that appeared more than once will be counted
up and indexed in a frequency arrays. For simplicity, any word with length less than
or equal to three characters is removed such as c is', 'the', cor' and 'to'. Example of the
frequency is shown in Table 3.2:
Table 3.2:
Example of Term-Frequency Table
Terms Frequency
SPONSOR
NESS
Navy
Engineering
Sponsoring

The process of counting frequency is repeated for each file in the categories to
get the total view of frequency. Since tokenization method is chosen, letter in
capitalization is left unchanged. Comparing done between tokens as a unit and the
count of occurrences is recorded as frequency.
Text classification starts by arranging the list of terms into nodes in a graph.
Each node connects with each other by bidirectional relationship. This situation is
adapted from solving the Traveling Salesman Problem (TSP). In TSP, nodes represent
cities where a businessman needs to travel to each city for exactly one time. Similarly,
in this adaptation we create artificial ant to simulate the agent travelling to all cities
and measure the total distances. The main problem of this scenario is Non-Polynomial

57

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Hard (NP-Hard) problem. NP-Hardness or non-deterministic polynomial-time refers
to a complexity in computational theory. To be specific, a problem H is NP-hard when
all of the problem L in NP can be shrunk in polynomial time to H. The research uses
training data that is divided by positive or negative classes. This research implements
the category selection as the options during the visitation time for each node. This
means that ant will choose one of the categories for each visit to the nodes. The nodes
will be tagged by pheromone values to the category that have higher chance to be
chosen. Figure 3.6 describes the mechanism that is used by ACO for each term.

positive class

negative class

Figure 3.6: Decision Making on Term Nodes

Travelling by ants to all different paths, one by one will consume too much
time even for supercomputer. A technique named Brute Force algorithm can also be
used to travel to all possible paths. For example, if a businessman needs to travel to 60
cities, there are 6060 = 4.89e+106 different possible ways. The ant colony algorithm
can find the optimal path in less than 500 attempts.
In this research, instead of taking one word at a time, category scanning is
done by using all words at once. It can be achieved by using ACO that is adapted from
ACO for TSP to map all terms. The adjustment is made by leaving the artificial ants to
choose term category at each node while depositing pheromone on the chosen
category. The process is repeated until the ants reach close to or perfect, desired
training score.

3.3.2 GA Experiment Setting

Experiments on Olex-GA were run by using Weka 3.6 extension. Every

experiment started with 50 text documents from two distinctive categories that were

58

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

obtained from the existing test set from the internet. For each subsequent series of test,
50 texts of the same category were added to each category as a constant addition up to
200 text documents. Classification accuracy for each addition was recorded. GA
classification accuracy was recorded for further analysis and will be discussed later.
Besides that, the result of classification accuracy could be compared between ACO
and GA only in the single threaded classification test. The test for multicore-
multithreading version of GA was not prepared because integration of GA technique
into Weka software is currently unavailable. However, the tests of ACO with
multicore-multithreading were still conducted to record time reduction between,
before and after multithreading mode. Each experiment was repeated ten times since it
involved the use of 10-fold cross validation techniques. GA test run the 10-fold cross
validation that is readily available in Weka but ACO test used its own 10-fold cross
validation that was developed manually. All the tests used the same ratio of 10% for
testing and 90% for training dataset. The average value of classification accuracy was
used across 10-fold cross validation techniques for each conducted experiment. All of
the data obtained was presented in tabular format.
The objectives of conducting the experiments were to test text classification
effectiveness process such as communication effectiveness between multiple solution
instances. Therefore, times to build up solution were recorded in addition to the
constant data increment between experiment series.
Parameter setting of the test is done on third party software which is available
for free download at the time of the test was conducted. The default setting also is
suggested and stated in the website. The summary of the setting run with with 500
populations, 200 generations, 0.2 elitism rate, uniform cross-over at 1.0, use of
tournament selection, Chi Square scoring function, 0.001 mutation rate and 50
features.
Mutation rate is implemented to help the heuristic influence on the randomize
trait done to individuals. Once applied, the next generation individual could have
better adaptations to counter classification problem. It helps in the genetic diversity
and effects on elite individuals. Chi-square is statistical hypothesis that is integrated
by the Olex-GA as a of type feature selection function used to select candidate
features. For elitism rate, the setting value determine how many individuals selected
that has the best adaptation from every tournament. This function is not directly affect
heuristic calculation but it can save time by skipping incompetent individual

59

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

analyzation. Similar to ACO implementation in this research, rather than by
percentage, the algorithm only selects one path for every cycle. 0.2 elitism rate imply
top 20% of the population will be forwarded to next generation. Crossover is used as a
convergence to influence the population to approach local minimum and maximum as
evolutionary take place. The generation is the process similar to ACO cycle where all
of its main function repeatedly executed.

3.3.3 Development of ACO

Ant Colony Optimization code is developed by using JAVA language. The

language is chosen because it is widely used and is suitable in various platforms in
delivering software product. The process in writing the code is simpler since the code
heavily used the modular task and kept organized.

1 procedure ACO_search();
2 while (termination_condition_not_occur)
! 3 scheduletasks
4 create_and_assign_ants();
5 update_path_selection();
6 update_measurement();
7 update_pheromone();
| 8 end scheduletasks
9 end while 1
10 end procedure

Figure 3.7: Pseudo-code of ACO Searching Method

Figure 3.7 is the pseudo-code of ACO searching method. The method starts
with sub-method to create and assign new artificial ants in its task scheduler. In
sequential search, this method is regularly invoked. Next is the procedure that contains
update_path_selection where this function will be invoked after the artificial ants'
creation. The function requires term-frequency table that is prepared during pre-
processing phase to initialize searching space for the artificial ant.
update measurement method is responsible to measure a newly generated route after
all of the term nodes are visited. The method will provide the total length or score to

60

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

be compared in discovering the shortest path. Lastly, update_pheromone are executed
as it can produce the length of the entire visited route.

1 procedure create.and_assign_ants();
2 while(not_reach _max_number();
3 set_ant_and_pheromone();
4 assign_ant_to _starting_route(); |
5 end while
6 end procedure

Figure 3.8: Pseudo-code of Artificial Ants' Creation and Assignment

Figure 3.8 is the code development procedure to create more than one artificial
ant to its search space. The number of ant usually is set more than one so that the best
path or route could be calculated by comparing multiple results from two or more
artificial ants. The methods provided are responsible to initialize an artificial ant to its
starting point at random location. Location placement is randomly generated to imitate
the real situation of real ant which is able to find foods regardless of the location. Each
of the newly created ants will have centralized pheromone and share the same array
table.

61

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 1 procedure newly_created_ant()
2 initialize_ant(); j
3 rg = update_ant_memory();
4 while (currentstate ^ targetstate)
5 £ = read_term_frequency_table();
6 p = calculate jrobabilities_for_each_remain_term(£,iTj,Q);
7 nextterm = apply_selection_with_pheromone(p,£2);
8 move_to_next_term(next_term); j
9 if (notendofterm)
10 update_routing_table();
1 11 end if
12 irj = update_ant_state(); |

! 13 end while
14 while (notendofterms) j
15 if (is_visited)
16 calculate__pheromone_addition();
I 17 update_pheromone_on_route();
18 else
19 deduct_evaporation();
20 end if
21 end while
22 die();
22 end procedure

Figure 3.9: Pseudo-code of Artificial Ant Task

Pseudo-code in Figure 3.9 shows the main activities of an artificial ant, starting
from its creation until the end of its lifecycle. The procedure starts with the
initialization of the artificial ant's properties which has been mentioned in Figure 3.8.
With the availability of the artificial ant memory, initial random selection of category
for the first term in the cycle is recorded to the memory of the artificial ant. The next
step in the loop is the category selection for the remaining terms. The loop starts by
fetching the value of term frequency from the term-frequency table of the next term.
The value is used to calculate the probabilities for the next term and generate random
value 0.0 (inclusive) to 1.0 (exclusive) as an input to the probabilities calculations.

62

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Next term selection is done by updating the memory of artificial ant. In this phase,
artificial ant is ready to run the next term selection loop. Just before the end of the
loop, a check to determine whether the loop has actually reached the last term is
executed. The status of the artificial ant is changed in the term index list to keep track
of the current selection. The tracking step is important for the search loop to prevent
the visited terms from being selected twice. The next part of the procedure is to
calculate pheromone value over the entire term nodes by using separate loop. The loop
will check on both categories of each term on whether they have been visited in the
current cycle. If it has been visited, an amount of pheromone concentration based on
solution quality is deposited on the category of the term node by using pheromone
table. Otherwise, a 1 % pheromone value is deducted as evaporation occurred on the
route.

3.3.3.1 Simulation of Artificial Ant Visiting Term Nodes

Figure 3.10, 3.11 and 3.12 in this subchapter are describing the simulation to
demonstrate the artificial ant which had discovered the shortest path after completing
a few tours. This simulation is generated using web application, TSPAntSim. The
simulation is setup by using 10 artificial ants and nine random generated nodes
pheromone with the trail updated intensity mode is ON. Figure 3.10 shows nine
different locations of node distributed randomly before the simulation run. All the
following figures (Figure 3.10, 3.11 and 3.12) contain nodes that are tagged in the
number where they represent words or term during the process of rule discovery.

63

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Figure 3.10: 9 Randomly Generated Nodes before Simulation Run

Figure 3.11: The First Tour Path

64

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

1

1 J !—!—! ©
o i 1 In _J
m
*** i r >$9
Wt
1*
1
&? T

^ ~ ^

^i » — « , , • -

o
1
Figure 3.12: The Path after Third Tour

Figure 3.11 and 3.12 shows the different paths discovered by artificial ants. As
the intensity of pheromone trail at node 1 increase, the path changed from node 0-7 to
1-7. From the observation, we can observe that pheromone intensity increased on
vertices 1-0-2 in Figure 3.12 when compared to Figure 3.11. We can imply that their
distances are shorter than other places and the heuristic calculation respond to this
condition by analyzing the vertices routing. Finally the artificial ant reroute those
vertices and analyze the total distance again. Meanwhile, other vertices has
pheromone concentration increased reflect their length. By the same rule, the rest
vertices 1 -0-2 shorter length has condenser pheromone display than longer one.

3.3.3.2 Stopping Criterion

Stopping criterion is important in scoring calculation. It is used to determine

whether ACO algorithm needs to be executed for another cycle. It is defined to
achieve determined solution quality that reflect characteristic of a category. For this
text classification research, one of the stopping criterions is in calculating the score
function. Artificial ants need to keep repeating what they are doing in order to build
different path of node selection in which the path is rated to a score value. The aim for
the searching is to achieve maximum score which is determined by total number of
distinguished term that can be assigned from term-frequency table. Common term is

65

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

also tagged and is given the same score that will make the artificial ants to
automatically skip it and is counted as a normal score in the score calculating
function. This is done because a common term could accidently pass through feature
selection filter. Besides achieving maximum score, maximum iteration can also be a
stopping criterion. The maximum number of iteration is determined before running the
whole application and is set large enough for artificial ants to make thorough search.
Usually, maximum number of iterations is set between 500 to 1000 cycles but if the
corpus consists of more than 1000 terms, the maximum cycle can be set to 2000. The
number is just a random assumption but it needs to be adjusted again with a few
running programs if the user chooses to use different random generator. Maximum
cycle set in this research is 1000.

3.3.4 ACO Text Classification

3.3.4.1 Skip Common Term Calculation with Same Frequency

In text feature selection, it is not guaranteed that a provided term frequency list
has completely free from the common terms. Terms that exist in both categories can
have random frequency and possibly have the same frequency. Since they do not give
any useful information that affects computation, it is best for the algorithm to include
built in common term filter. Scoring function will automatically skip the rest of its
calculation when it found a term that has common terms criterion. In order not to
disrupt the flow of the whole computation, common term that is detected will still be
giving a score for the sake of striving stopping criterion. If there is no score given to
the common term upon discovery, artificial ant will be forever staying in a loop to get
full score that consist of the number of the total term. Common term that is validated
when the scoring function is applied to the term will also ensure that the artificial ants
have less chance to guess on different category. This is because common terms
usually do not influence the category selection. The artificial ants can keep skipping
heuristic calculation and generate random value. Therefore, ACO algorithm can gain
additional execution speed according to the number of common term found in the
term-frequency list. The function can be useful when an effective common term filter
is done before the text classification process take place. In term of pheromone

66

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

application, pheromone value will be used on the term that is compared to the other
category. In ongoing iteration, a pheromone value will be adjusted to determine
whether it is relevant for the term to be treated as a common term. The assumption is
included in the checking process in case the common term meets the criterion in
coincidental event. The status is also checked in order to see if it is different from
actual suggested pattern of training case. If that is the case, the artificial ants need to
include a check on the node selection along with the other path pattern. In ACO
algorithm, the case can be achieved by exploiting the nature of random selection.
While a pheromone value has a stronger concentration to a particular category of a
term, there is a slight chance that an artificial ant will choose weaker pheromone
value. The situation is the same with actual ants in which the ants do not always have
the obligation to follow a fixed path that is taken by majority of the other real ants.

67

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

3.3.4.2 Obtaining Classification Rule Every Cycle

1 First cycle Second cycle

category A category B category A category B

term I
'sponsor'

\ choose B choose A /

term
'ness'
6 t 1

term
'navy'
l/ \
s 1 ^ ^ ^ ' F

\ 7
1 denotes term node

Figure 3.13: Different Path that is Chosen between each Cycle

Figure 3.13 shows how an artificial ant visits from one term to another. An
artificial ant will complete a full visit of all term nodes for every cycle until it has
reached the stopping criterion. At the same time, the latest classification rule list is
also produced. If the model is set to run at x number of artificial ants, therefore we can
obtain x number of classification rule per cycle. The process in the cycle is repeated to
search for more suitable classification. Unless stopping criterions are met, the
classification rules obtained are considered inferior than the latest cycle version. This
is because ACO is an algorithm that produces solution to build up over time. For
multiple artificial ants setting, rule classification need to be selected as the best quality
for the current cycle. This is because different artificial ant will produce its own
classification rule. Unlike pheromone array that is centralized and sharable between

68

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

artificial ants, classification rule array is set locally to a new child threads that
required new initialization. Classification rule is an array that has the same size as the
total number of term in term-frequency table. Besides the recording of its frequency, a
term will be assigned to a category as a classification rule. Training data set is referred
as a main evaluation that influences the category that is suitable to the term. It is the
ACO algorithm task to find different pattern and to infer the training data set that
produce the classification rule. After visiting all term nodes, an artificial ant needs to
invoke scoring function and send the list of classification rule. The rule is evaluated
by the number of terms that are tagged correctly and matches the training document
set. The score number shows the term number that is correctly chosen by artificial ant.
That means, the number that is close to the total number of terms indicate that the
whole process is reaching its end and will fulfil the stopping criterion soon. A higher
score too will have a stronger scent of pheromone which will alert the current artificial
ants and all other artificial ants alert not to search all paths.

3.3.4.3 Choosing Next State

In this research, the state here refers to the possible category for each term.
Upon visiting each term, the ant will inspect the pheromone value from previous
iteration. The formula to calculate pheromone will be described in the next section.
Ant will use some heuristic rules that influence the pheromone usage. Figure 3.14
shows some of the array value:

uycie u
term 0, category 0 [0 to 0] = 99.5, [0 to 1] = 99.5, category 1 [1 to 0] = 99.5, [1 to 1] 99.5
term 1, category 0 [0 to 0] = 99.5, [0 to 1] = 99.5, category 1 [1 to 0] 99.5, [1 to 1] = 99.5
term 2, category 0 [0 to 0] = 102.83333333333334, [0 to 1] = 99.5, category 1 [1 to 0] = 99.5,
[1 to 1] = 99.5
term 3, category 0 [0 to 0] = 99.5, [0 to 1] = 99.5, category 1 [1 to 0] = 99.5, [1 to 1] = 99.5
term 4, category 0 [0 to 0] = 99.5, [0 to 1] = 99.5, category 1 [1 to 0] = 99.5, [1 to 1] = 99.5
term 5, category 0 [0 to 0] = 99.5, [0 to 1] = 99.5, category 1 [1 to 0] = 99.5, [1 to 1] = 99.5
term 6, category 0 [0 to 0] = 102.83333333333334, [0 to 1] = 99.5, category 1 [1 to 0] = 99.5,
[1 to 1] = 99.5
term 7, category 0 [0 to 0] = 99.5, [0 to 1] = 99.5, category 1 [1 to 0] = 99.5, [1 to 1] = 99.5
term 8, category 0 [0 to 0] = 99.5, [0 to 1] = 99.5, category 1 [1 to 0] = 99.5, [1 to 1] = 99.5
Figure 3.14: Sample of Pheromone Value

69

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Table 3.3 shows the condition that will favour the pheromone influence:

Table 3.3:
Relationship between Frequencies and Visited Category
Frequency Class Current Visited
Frequency class 1 more than class 2 Category 1
Frequency class 2 more than class 1 Category 2
Frequency class 1 more than 0 while class 2 is 0 Category 0
Frequency class 2 more than 0 while class 1 is 0 Category 1

As shown in table 3.3, the ant will attempt more tries on the most likely
category that will be the solution. However, there is no guarantee that the foreseen
solution is the best one. In TSP, there are times when the longer path leads to the
shortest overall route. If that is the case, the pheromone will become stronger on the
newly found best routes during the iteration process. Note that category 2 indicates
common term and not relevant to heuristic influence or other part of calculation. Its
purpose is there only to be skipped so that the whole classification is more efficient
and not to execute unnecessary calculation.

Compare frequency of each term

Read previous pheromone and calculate likelihood

0
o
value
* - - . . . . . • • • . . . . . . . . .. . . . . .• - - . - . • . . . . . . . . . . . . . . * . . . . .

Check new pheromone value against lower and upper

0
limit

Normalize likelihood value into percentage value

Cast a randomize value and check it to cumulative

array
o
Figure 3.15: Steps to Calculate Heuristic Value

Figure 3.15 shows step by step of how heuristic value is calculated to produce
a category prediction. Its calculation requires a table that contain all of the terms

70

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

found in the training corpus with the summary of its frequency. A table for each
category is also needed to store the term frequency. For each term in the training data
set, it needs to be checked in order for the pheromone to determine the current affinity
of the term. The checking process requires frequency of each term in each category to
be fetched and compare. The comparisons are then followed by different action that
determines whether further calculation is required or not. Firstly, frequency for each
category is compared to determine whether they have value that is greater than 0 or
not. If the case is true, another comparison is done to determine which frequency is
higher between the two categories. A full mark of one hundred percent is given if the
category passes all of the condition. By the end of the checking series, only one
category that has the highest mark will be selected. If all of the above conditions are
not met, the percentage value will be divided from one hundred percent with the
number of categories. This is done to represent when the frequency pattern shows
irregular value for current term.
The next step is to calculate the heuristic of the likelihood for an artificial ant
to decide a visit for the next term categories. A pheromone value for the current visit
of category will be powered up by an alpha value. The alpha value is taken up from
original heuristic calculation from ACO that is used for TSP event. The resultant value
is multiplied again with the previous stored percentage. To make sure that the ant has
a chance to divert or try other solution route, the percentage is divided by the number
of categories and then power up by the number of categories again.
After the heuristic calculation is done, the end value needs to be checked for
its lower and upper limit. The check is necessary because it is possible for the stored
value will overflow the next pheromone value usage calculation. The higher the
iteration, the higher the risk of overflow case. For this model, the lower limit is set to
0.0001 and the upper limit is the maximum value which is divided by the total number
of all terms in term frequency table and multiplies by 100.
The next part is to convert the heuristic value to normalized percentage. A
variable to total up the heuristic is needed before hand. Getting a normalized
probability value for each category enables us to get cumulative value which is stored
in an array. Since a category is representing a resizable value of probability, the total
value in cumulative array would be 1.0 or a hundred percent.
Finally, an artificial ant will make the choice based on cumulative value array.
Cumulative array has lower bound value at zero and has upper bound value capped at

71

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

1.0. Suppose there are two or more categories involve, there will be an intermediate
value between each category to split up each category affinity before a random value
is taken.
Table 3.4 shows an example of cumulative array and its value. Values included
in the array are random value generated by the artificial ants, each time they need to
make a choice. A random value generated by major programming language is always
a floating point between 0.0 (inclusive) and 1.0 (exclusive).
Table 3.4:
Probability Selection by Random Generated Value
Array Index 0 1 2
Array Value 0 0.7 1
Random value 0.4
Category selection 0

The table above shows that the random value generated is 0.4, which falls
between 0.0 (inclusive) and its upper cap is at 0.7 (exclusive). In this case, the
artificial ant will make a move or mark its travel to first category. Suppose that the
random value falls between 0.7 (inclusive) and 1.0 (exclusive), the second category is
selected.

3.3.4.4 Pheromone Usage

There are many types of pheromone for different usage that are used in
situational event such as to attack other ants and to make them go frenzy in dangerous
situation. This research refers to the pheromone used by ants while they were
searching for food.
In this research, ant that visits all nodes will examine the amount of
pheromone value that was left on the trail in the previous iteration. The decision will
be influenced by how much concentration value the term node has during the current
time. Different from real world ants, the artificial ants do not automatically update the
new pheromone value as they leave the node. This is because the score for the whole
path has yet to be available. Artificial ants can update the value only after all of the
terms have been visited. Pheromone evaporates in constant amount as time passed by

72

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

and pheromone trail can be calculated by using the following formula as shown in
(3.1):

TiJ^[(l-p)TU+&T?fSt]TmaX , (3.1)
J
imin

where ij refer to path between term i and term j , Arf^5t refer to best path, p refer to
current pheromone value, rmm is lower bound and Tmax is upper bound. For each
solution found, artificial ants need to determine the correct amount of pheromone
concentration for each node. The setup for the research has decided that each correct
term node chosen by ants will increase 1% of pheromone concentration from its
original values. To be fair with the incorrect decision chosen by artificial ants, 1%
pheromone concentration will be deducted from the previous iteration per incorrect
training score.

3.3.5 Set Up Multicore-Multithread Environment

The model is further modified to operate in multicore-multithread environment

specifications. The software that had been developed and explained in subchapter
3.3.3 is adjusted to operate in more than one thread. The adjustment involves
assigning the artificial ant creation procedure to a new child thread instead of the
parent thread. For that purpose, parent thread task is changed to synchronization
process. Upon running the model, artificial ants' instance is set from one to four.
Multithreading setting can be significantly influenced by the operated machine
as explained in chapter 2. Therefore, the machine used in this experiment is checked
and found to be suitable to run the software. Table 3.5 is the summary for the machine
specifications:

73

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Table 3.5:
Machine and Operating System Specs
Machine and Operating System Specs Details
Windows Experience Index 5.9 (lowest sub-score)
Processor Intel(R) Core(TM) i7-3630QM
Processor clock 2.40GHz
Number of core 4
Number of factory thread 8
Installed memory 8.00GB
System type 64-bit operating system

Table 3.6 is Windows Experience Index information in greater details that

measure extended performance of the machine:
Table 3.6:
Windows Experience Index Details
Component What is rated Subscore
Processor Calculation per seconds 7.6
Memory (RAM) Memory operations per second 7.6
Graphics Desktop performance for Windows Aero 7.3
Gaming graphics 3D business and gaming graphics 7.3
performance
Primary hard disk Disk data transfer rate 5.9

74

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Sequential search Multithread search
c
eg' o
3 * creation overhead
• lli.IL.-ll- LUI.ll.-i -II I -..••.•"U IU...I.L-III1IIIUI1

ON

r—|
* i Main thread
•1

o
rp K7 i *
<J7~
*
\^
*
\J/
*
>|/
*
j\^
*
\^
*
\|/
*
vj/
*
~~^
ft:
a
•
n
o
CD

i 1 w w w w V • v Y • o
a

5]
&3
\ &
-%
\
OOOOOOOOO
thread I thread 2 thread 3 thread 4 thread 5 thread 6 thread 7 thread 8 thread 9

R
i i
CD Legend: ""^ Artificial ant
s-
Q-

CD
4 y i Term node routing

o Til ne

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Figure 3.16 is the comparison between sequential and 9 concurrent searches
that represent the multicore-multithread environment. It is found that the execution
time for 9 multithreaded artificial ants are lesser and more efficient. Sequential search
on the other hand need to finish computation one by one. However, multithreaded
search has additional overhead to consecutively create artificial ants that create
differences on time gap compared to the theory illustration. The overhead that is
discussed here requires expensive resources. Having 8 additional thread instances at a
time would fairly need an equal number of processor cores and threads. Other factor
that can cause this experiment to have lesser time of execution reduction is that the
pheromone has only one single copy of array. The situation would cause the artificial
ants to waste a portion of time in accessing the pheromone table.

76

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 ->
create 9 artificial ant

-» * * * one by one through

looping
* * *

Main thread sleep safely with

synchronization function

record time-stamp here

5
<? C o o o
o ooo
9 concurrent
route searching

record time-stamp here one ant finish (any)

Main thread woke up by first

ant's death

-> -* -* * Save best route and then

kill all 9 ant consecutively

Main thread is free after all bindings

from synchronization are released

Figure 3.17: Main Part of Multicore-Multithread ACO Model

Figure 3.17 illustrates the main part of the ACO model. It shows the overall
procedure from the pre-processing step to the end of concurrent execution for this
research. The explanation reveals that the mechanism for 9 concurrent executions
greatly differ from the single threaded version of ACO.

77

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

3.3.6.1 Overcome Unstable Execution Speed

Upon dispatching artificial ants from threads, they are released by iterating a
loop. Although the released is done in sequential manner, it is not guaranteed that they
will keep the sequence during its life cycle. These happen because sometimes the
speed can be uncertain as there are certain events that can interrupt a thread especially
the child thread. Figure 3.18 shows how tasks are divided and scheduled to a linear
sequence by operating system. Between multiple tasks that an operating system needs
to handle, a task can take a short time or a normal time length to finish. However
some of the tasks can take too much time that can cause other task to be waiting in
queue and would not be executed in timely manner. This problem happens even
though the task just needs a very short time to finish. The problem can become even
significant when the task that is waiting in the queue has high priority and need to be
completed in real time.
An example of this type of task is time recording function. If the function is
not being executed immediately, it will have problem with data inconsistency because
the recording is done at later time from the long queue. This can give a large time
difference as it required high accuracy recording, in other word to record in nano
second in order to record the operating time for each artificial ant. To overcome this
problem, task scheduling repeatedly apply context switching between the tasks given
to the threads. Every task has small portion of time depending on its priority and time
for it to complete. Therefore, shorter and simpler task get executed in small time
compared to the heavier and longer task execution. However, the decision taken to
obtain such solution makes context switching overhead accumulated as the number of
task increase in the queue. A thread can pause or get interrupted if the task takes up
even slightly longer than the readily allocated time slice. Context switching that is
done by the scheduler makes it as if the threads execute random part of code even
though they have the same released time. To overcome uneven execution speed while
preserving responsiveness from context switching, implementation of how signaling
and stopping multiple thread is adjusted. The implementation detail is presented in
chapter 3.3.6.2.

78

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 tl
>

break a task into

smaller parts

A
time
length

task(T)
Tl T2 T3 T4

Tl T2 T3 T4 Tl T2 T3 T2
1

>
execution sequence

Figure 3.18: Task with Different Length Scheduled and Execution Sequence

3.3.6.2 Signalling and Stopping Multiple Threads

In order to overcome uneven speed of artificial ants from the effect of thread
race condition (chapter 3.3.6.1), this research implement and adjust different setting.
Instead of main thread remain active to wait for one of artificial ants found the

79

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

solution, the main thread is put to sleep state. The research implementations also
execute artificial ant termination in sequential manner. Besides that, the application of
queueing and pheromone sharing make some adjustment necessary in order to achieve
higher efficiency, accuracy and performance. Further explanations on the
implementation details are discussed.
When artificial ants are busy visiting nodes, they need to communicate to each
other to update their progress status. This is because a thread that is created from the
process allows its variable space to be shared by its child thread. They are simpler to
be used as an immediate communication compare to the Inter-Process Communication
(IPC). This feature is important in order to have high accuracy of calculating total
time spend for other thread as stopping signals are sent directly to the main thread.
Recording stopping time is more complex than to record starting time because when
all threads are released, the main thread is put to sleep. Aside from the operating
system, the main thread also issues an interrupt to its child thread that is represented as
artificial ant. Child thread is created to manage one artificial ant per instances.
Interrupt in the developed multicore-multithread model is used with the synchronized
method. Beside interrupt, thread also can be put to enter a sleep mode. The difference
between the mentioned methods is that sleep thread can be used for the next operation
in less conditionally. The sleep method can be fast as it skips on creating and
destroying a new variable in memory allocation compared to interrupt. Therefore, the
developed model used interrupt on child thread method because the software
development can be secured when starting a new cycle. Only the main thread is using
sleep mode as it carries more state variable. Interrupt is more secured to be used
because each time the child thread is created, the variable is also initialized using
preset default value. If we design it to use sleep mode, an amount of additional code is
needed to manually return the variable to its default value. This is done to overwrite
the old data; otherwise we will face data integrity issue. Interrupt mode is more
suitable as part of the design because destroying child thread at interrupt will leave
global variable of parent thread to be untouched. Besides that, interrupting has a wide
and standard support by JAVA language which is used to develop the model. As the
child thread is repeatedly created and destroyed, an operating system can acquired
execution time resource temporarily. This situation happens when the main thread
executes different part of code such as examining classification rule and calculating
pheromone increment. Operating system can schedule a different task or opt to

80

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

conserve memory after a child thread is destroyed until a new thread creation is
requested.
Figure 3.19 shows how a child thread issues stopping signal to the main thread
after it becomes the first one to meet the stopping criterions. The stopping criterion for
the illustration consists of the event when the artificial ant found a solution or reaches
the maximum cycle. The main thread is also bound with synchronized mechanism to
ensure that all threads are properly started and properly muted before calculating their
total time and solution value. It is possible for a child thread to be alive in a loop and
the thread will be using uninitialized data in the next cycle if synchronize mechanism
is not enforced. When an artificial ant finally reaches a stopping condition, it tells the
other artificial ant to stop searching too. Firstly, it accesses main thread global
variable to contact the child thread list. Then the artificial ant set local variable for
each artificial ant about the premature stopping condition. When the artificial ants
read the signal, they need to update timestamp immediately which later can be used by
the main threads. Each of the artificial ants will end its lifecycle so that the main
thread will be released by the synchronized binding which had been created
previously. When all the bindings are released, the main thread is now free to continue
its execution. The tasks done consist of totalling up execution time, reading
classification rule and updating pheromone values for each artificial ants.

81

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Artificial ant 2 found stopping condition

ant 1

ant 2
(ant 3 to 8)
ant 9

->

x time time

the whole artificial ant responds to the event

issue
stopping
signal

xtime time

Figure 3.19: Implementation on How a Thread Send Stopping Signal to other Threads

3.4 Phase 3: Evaluation on Multicore-Multithread ACO Model

A few experiments had been conducted in order to measure the effectiveness

of the developed method. First, the experiments were conducted to determine the
accuracy of ACO without multithreading adaptation. The experiments utilized 200
documents (discussed in Chapter 4). All of the data is used and the pre-processing part
will filter unnecessary terms prior to the main process. The experiments were run
using the software that had been developed by using JAVA language. For Genetic
Algorithm, experiments were conducted by using Olex-GA binary that had been
integrated manually into WEKA 3.6 (stable released) by using the provided plugin

82

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

which is available online. The other experiment was conducted to evaluate ACO in
multicore-multithread environment. The main aspect which becomes the focus is time
reduction compared to the original or single threaded ACO model. The last
experiments also revealed that there is an acceptable difference between ACO single
threaded and multithreaded ACO.
Most of the previous studies use accuracy performance measurement to
evaluate the effectiveness of classification prediction. These values are calculated by
using the variables as shown in Table 3.7:
Table 3.7:
Variables to Calculate Accuracy Performance
Categories Positive Prediction Negative Prediction
Computer graphic tp fit
Microsoft miscellaneous fp tn

Where

• tp represent total computer graphic article correctly predicted

• fp represent total Microsoft miscellaneous article correctly predicted
• fn represent total computer graphic article falsely predicted
• tn represent total Microsoft miscellaneous article falsely predicted
Accuracy calculation clarifies the percentage of article count that has been
predicted correctly according to the categories of the articles. Calculations of the
accuracy values are done in formula 3.2 and 3.3. The accuracy used in past researches
to evaluate the effectiveness of classification algorithms also correspond to this
calculation.

total documents correctly predicted (3.2)

Accuracy = —
total documents

tp + tn v(3.3)
7
Accuracy7 =
tp+fp + fn + tn

83

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

3.5 Summary
This chapter started with text data collection and identification after
completing knowledge acquisition phase. Two categories from online source, The 20
Newsgroup have been found suitable for experiment in the research. In the pre-
processing phase frequency table has been produced. Pseudo-code has been presented
in ACO development and a simulation to observe pheromone deposition is done.
Following to next phase, multicore-multithread model has been presented. The main
part highlight the setting for multicore-multithread with hardware specification and
calibration to satisfies testing requirement in next chapter. Accuracy calculation and
formulae used in order to analyse ACO and GA classification are specified.

84

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 CHAPTER FOUR

RESULTS AND DATA ANALYSIS

4.1 Introduction

Test to analyze text classification effectiveness by using ACO algorithm are

discussed in this chapter. The test continues with ACO classification effectiveness test
from locally stored text file after using the developed method (chapter 4.2). Chapter
4.3 aims to find classification accuracy of GA for the same data used in the previous
test. For both conducted tests in the previous chapters (chapter 4.2 and 4.3),
comparison between ACO and GA is done and explained in chapter 4.4. Chapter 4.5
describes the effectiveness test of ACO text classification with integrated multicore-
multithreading as extension functionality. Time reduction test for ACO multicore-
multithreading is explained in chapter 4.6.

4.2 ACO Text Classification Accuracy

4.2.1 Experiment

This experiment was created to measure ACO text classification accuracy. The
results from the test are recorded in Table 4.1. The accuracy differences between file
numbers added was analysed.
Table 4.1:
Average Accuracy from Every 10 Runs of Every 50 Files Addition
Text Files Average Accuracy (%)
50 68.66
100 75.16
150 82.33
200 82.25

85

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 The experiment results shows that ACO text classification has different
average accuracy as the number of files increased. This is clearly illustrated in Table
4.2. The experiment was done to analyse ACO classification algorithm effectiveness
in handling bigger data set.
Table 4.2:
Accuracy Difference between File Addition Increment
Text Files Percentage difference
50-100 Increase by 6.50%
100 - 150 Increase by 7.17%
150 - 200 Decrease by 0.08%
Average Increase by 4.53%

Table 4.2 shows that the ACO text classification accuracy has increased. The
accuracy increment from file in which the number ranges between 50 to 100 is 6.5%.
The accuracy increased further in the experiment involving 150 files with 7.17%.
However, the result for the next experiment that included 200 files showed a small
decrement of 0.08%. The overall experiment result shows an acceptable beneficial
improvement that outweighed its decreased accuracy values.

4.2.2 Discussion

The objective of the text classification process in this research is to achieve

comparable accuracy level. This is done by having the artificial ant to visit each of the
given term that exists in term frequency table and lays its initial pheromone value.
Pheromone strength is calculated when all of the term has been visited for the current
cycle. Each term can have different frequency which will affect the heuristic artificial
ant decision to search different pattern of classification solution. On the other hand,
the high frequencies of a term found sometimes could not correctly present the
importance of the term to the category described in the text. This situation is
especially true when some part of both categories contain identical topic and share
common terms. The frequencies could also be identical for both categories.

86

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

4.3 GA Text Classification Accuracy

4.3.1 Experiment

This experiment was created to measure GA text classification accuracy. Olex-

GA run with default setting with 500 populations, 200 generations, 0.2 elitism rate,
uniform cross-over at 1.0, use of tournament selection, Chi Square scoring function,
0.001 mutation rate and 50 features. The results from the test were recorded in Table
4.3. Accuracy differences between file numbers were analysed.
Table 4.3:
Average Accuracy from 10 Runs of Every 50 Files Addition
Text Files Average Accuracy (%)
10 71.00
100 67.50
150 67.70
200 73.00

Table 4.3 showed that the average accuracy of GA text classification varies as
the number of files increased. The experiment is done to analyse GA classification
algorithm in handling bigger data set.

Table 4.4:
Accuracy Difference between File Addition Increment on GA
Text Files Percentage difference
50-100 Decrease by 3.50%
100 - 150 Increase by 0.20%
150 - 200 Increase by 5.30%
Average Increase by 0.67%

Table 4.4 shows that GA text classification accuracy increased over the
increasing data that had been tested. The accuracy decrement from file in which the
number ranges between 50 to 100 is 3.5%. The accuracy had slightly increase in the
experiment which consist of 150 files by 0.2%. Result from the next experiment that
includes 200 files shows an acceptable increment of 5.3%. The overall experiment

87

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

result shows a marginal beneficial improvement that outweighed its decrement
accuracy values.

4.3.2 Discussion

GA text classification process is typically intended to achieve comparable

accuracy level. This is done by having every term to be treated as individual to take
part in the natural selection or the survival of the fittest concept of an organism. The
main process consists of the fitness function, which is done to breed a new generation.
In the heuristic method, randomly generated number is also used to calculate and
influence the selection done. This is the part in which the use of random generator
concept coincides. The number generated is treated as a chance to select the best
solution in text classification. The fitness function is used to measure the quality of the
overall solution in every cycle that occurred. GA algorithm is used widely in global
discipline while fitness function is adapted to the problem used.
One of the factors that contribute to GA text classification accuracy is the
elitism concept. Elitism is applied to individual term that has grown up and survives
many tournaments. Such individual is valuable because it has traits that could adapt
over various measure of text classification problem that is treated as environmental
factor. Over the passing generations, the elite individual trait will bring forward to an
evolution of a new generation. This is done to preserve the individual traits and can
hardly be affected by mutations during traits exchange.

4.4 Comparison between ACO and GA Results

Both ACO and GA experiments were compared to measure the differences in

text classification accuracy. The results from the test were recorded in Table 4.5.
Accuracy differences between file numbers added is analysed.

88

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Table 4.5:
Accuracy Comparison between ACQ and GA Results
Text Files ACO Average GA Average Difference (%)
Accuracy (%) Accuracy (%)
50 68.60 71.00 GA surpass by 2.40
100 75.16 67.50 ACO surpass by 7.66
150 82.33 67.70 ACO surpass by 14.63
200 82.25 73.00 ACO surpass by 9.25
Average 77.08 69.80 7.28 difference

Table 4.5 shows the ACO and GA text classification accuracy result from
previous experiments. For 50 files used in classification, GA methods have better
accuracy. The GA accuracy is calculated to be 2.4 percent higher than the ACO
method. The accuracy for 100 files shows that ACO surpasses GA by 7.66 percent.
When conducting the experiment with 150 files, it is revealed that ACO has the
highest difference of 14.63 percent over GA average accuracy. The experiment with
200 files revealed that ACO accuracy is 9.25 percent higher than GA accuracy.

4.4.1 Discussion on ACO and GA

Take note that both ACO and GA are wrapper methods that requires a set of
solution over time in order to solve text classification problem. They are
fundamentally the same which iterate and include random generation in their heuristic
decision making. However, they differ in how they perceive text classification criteria
problem by using natural and biological methods respectively. ACO was inspired by
the analogy of colony of ants using pheromone and food foraging mechanism. On the
other hand, GA uses general tournament and living cell activities comparison.
First of all, the test should not be taken as absolute reference in which the
result has shown that ACO surpass GA algorithm in text classification most of the
time. This is because there are many factors that contribute to or affect classification
accuracy such as ambiguous meaning from ambiguous term and abbreviation. Other
factors also consist of misspelled words and words with number. It is difficult to
possibly test all of the factors into a single set of experiment and research. Therefore

89

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

this discussion can only be discussed based on general assumptions of text
classification as the result had suggested. On overall average, ACO is 7.28% more
accurate than GA from the experiments.
One of the factors that increases the ACO classification accuracy over GA is
the use of pheromone concept. The main idea used here is the artificial ants release
pheromone value on each term that forms a recognizable pattern to build up or to
improve solution searching. The pheromone could also experience stagnation state in
which the pheromone value reaches upper or lower limit, leading the artificial ants to
find a new solution if it exists.
However, GA is a type of evolutionary algorithm that is affected by the
majority or dominant traits. The influences of dominant trait triggered the other
majority to convert to dominant traits when it reaches a solution. Such circumstance
disallows the minority group to explore different solution that might lead to a better
solution than the one explored by major elitism individuals. One might argue that
mutation could be the solution that can affect new solution discovery. However,
mutation in GA can happen very slow with one traits is exchanged at a time. The ideal
rate by heuristic calculation is lesser as compared to ACO pheromone usage.

4.5 Results of Optimal Number of Thread Usage

A small test to determine the most suitable number of artificial ants to run
concurrently is done manually. The test used 100 documents as the input data for
classification and it is run multiple times with the same setting except for the number
of concurrent artificial ants. The test is run repeatedly by increasing the number of ant
from one to sixteen which double the factory thread's max number (eight). All of the
test runtime are recorded to determine the shortest time. For the current test machine
specification, using nine threads concurrently are found to give optimal performance.
Table 4.6 is the information for the test result by recording the average for 10 times.
From the table we found that the best time reduction happened when 9 threads was
used, with the result of 35.53% of improvement.

90

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Table 4.6:
Result of Optimal Thread Number Test
Number of threads Average time(Seconds) Difference from Unthreaded (%)
Unthreaded 0.76 N/A
2 0.51 + 32.89
3 0.52 + 31.58
4 0.67 + 11.84
5 0.69 + 9.21
6 0.71 + 6.58
7 0.75 + 1.32
8 0.68 + 10.53
9 0.49 + 35.53
10 0.55 + 27.63
11 0.63 + 17.11
12 0.62 + 18.42
13 0.70 + 7.89
14 0.68 + 10.53
15 0.67 + 11.84
16 0.68 + 10.53

From the same test, CPU utilization is also recorded. Task manager by
Windows 7 is used. Figure 4.1 shows the print screen version of unthreaded version of
ACO. Red region indicate the processing part from the start to the end of execution.
Since there are 10 run (10-fold), the graph shows a sudden drop for a short time in a
while, which represent the file reading. File reading consume less processing power
thus the phenomenon occurred. As shown in the graph, only one core of the processor
has higher utilization, while other cores show passive usage.

91

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

CPU Usage History

Figure 4.1: CPU Utilization for Unthreaded ACO Classification

For comparison, CPU utilization for 9 threaded version of ACO test was also
recorded. Figure 4.2 shows the print screen version of 9 thread of ACO run. Observe
that the average processing power for current version shows a higher value.

92

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 Figure 4.2: CPU Utilization for 9 Threads ACO Classification

From the graph, we can assume that using multithreading in ACO is beneficial
in terms of processing time and efficiency. It also uses the multicore-multithread
processor to a higher potential and fulfil its purpose well.

4.5.1 Discussion

By using context switching, it is possible to create more than one artificial ant
in order to handle a single thread. But we cannot gain additional time cut as time is an
immutable resource. Continuous thread switch required scheduler to work even more
while the scheduler itself has execution cost such as method overloading and
algorithm calculation. In writing code, there are no specific rules to limit a number of
a thread that can be created by a single process. All of the threads that are requested

93

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

are created at software level in which the processor is unaware of their existence. This
is because an operating system will hide the thread first and manage it to thread pool.
Threads from thread pool are fetched to a number of processor that is currently
activated. If four core processor has hardware level multithreading, for example two
threads per core, then the maximum number of physical thread the processor can
handle at a time are eight. Operating system always measures the amount of a divided
task that could incur a cost to a thread. On lighter task, operating system will reduce
the number of activated core and will have the remaining core to do context switching
repeatedly. The context switching could result in processor to work at higher clock
speed. Operating system will manage a context switching scheduler in case a user
wants to conserve operating power source. However, if the user wants the computer to
dedicate for maximum performance, all of the core will be activated in order to
prepare heavier and higher concurrency task. In case the number of concurrency that
are requested are higher than the processor capability, operating system will go back
to resorting for context switching again.

4.6 Accuracy Results of ACO with Multicore-Multithreading

4.6.1 Experiment

This experiment was conducted to measure ACO text classification accuracy

with the use of multicore-multithreading feature. The results from the test were
recorded in Table 4.7. The accuracy differences between file numbers were analysed.
Table 4.7:
Average Accuracy from Every 10 Runs of Every 50 Files Addition
Text Files Average Accuracy Multicore-Multithreading (%)
50 68.66
100 75.83
150 82.11
200 82.22

Experiment shows that the average accuracy of ACO text classification with
multicore-multithreading. It has different average accuracy as the number of files

94

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

increased which can be seen in Table 4.7. The experiment was done to analyse
multithreading version of ACO classification algorithm in handling bigger data set.
Table 4.8:
Accuracy Difference between File Addition Increment
Text Files Percentage difference
50 - 100 Increase by 7.17%
100 - 150 Increase by 6.28%
150 - 200 Increase by 0.11 %
Average Increase by 4.52%

Table 4.8 shows that ACO text classification accuracy increased for the overall
file addition. The accuracy increment from file in which the number ranges between
50 to 100 is 7.17 percent. The accuracy also shows an increment in experiment
involving 150 files with 6.28 percent. However, the result for the next experiment that
includes 200 files shows a small increment of 0.11 percent. The overall experiment
result shows a beneficial improvement that increases every file addition.
Take note that the accuracy differences for the same data set in the single
threaded version have very similar accuracy to the multicore-multithreading version.
The accuracy differences between single threaded and multicore-multithreaded
version are recorded in Table 4.9:
Table 4.9:
Accuracy Difference between Single Threaded and Multicore-Multithreading Version ofACO
Text Files Single threaded Multicore-Multithread Percentage
ACO Accuracy (%) ACO Accuracy (%) Difference (%)
50 68.66 68.66 0.0
100 75.16 75.83 0.67
150 82.33 82.11 0.22
200 82.25 82.22 0.03

Table 4.9 shows that there are only slight differences of classification accuracy
between single threaded and multicore-multithreaded version of ACO. For the
classification involving 50 files, there is no accuracy difference at all. The
classification accuracy difference is 0.67% for 100 files. The difference is recorded as
the highest accuracy difference of all the test series. The next file addition which uses

95
COPYRIGHT UNIVERSITI TEKNOLOGI MARA
150 files shows 0.22 percent difference. The last file addition in the test which
includes 200 files has produced ACO accuracy difference by 0.03 percent.

4.6.2 Discussion on Single and Multicore-Multithreaded

Adjusting the ACO model from single threaded to multicore-multithreaded

ACO does not impact the accuracy result at a high rate. From the experiments,
multicore-multithreading ACO has less than one percent accuracy difference on all
tested file additions. One of the concerns is why the accuracy more or less unaffected
even though the mechanism has encountered major change by having multicore-
multithreading integrated into the model.
The experiments aim to test the competency of multicore-multithreaded
feature. By having multitasking, the model is able to keep the classification accuracies
that are identical to the previous level with minimal trade off. Based on the results, we
can conclude that ACO algorithm is able to work simultaneously to preserve
classification accuracy and efficient resource management.
The objective for the test of text classification to be carried out is to increase
classification accuracy in order to increase text classification. It is examined especially
for the processes involving large data set and inaccurate category definition. One of
the factors that affects text classification accuracy in the experiment outcome is how
well a category is constructed and defined with specific criteria. Category or class is
usually constructed manually in which an individual uses his or her best opinion,
common sense and knowledge to describe the category. However, the method has a
slight deficiency that prevents classification accuracy to hit one hundred percent rate.
This situation could happen because different person could have different opinion and
perception on a category. Surely a testing set could have high quality and acceptable
by many experts but that does not make it 100% correct. This is due to the nature of
the text itself that is different from mathematical measurement. Similar with ACO
algorithm, the classification rule could suggest an optimal pattern that covers major
criteria of a category, but it can produce unexpected results that are out of its solution
searching scope. Instead, other text classification algorithm could cover the scope but
may come with different weaknesses.

96

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

4.7 Results of Time Reduction Test for ACO with Multicore-Multithreading

4.7.1 Experiment

This experiment was conducted to measure the time taken for ACO to classify
text. The results from the test are recorded in Table 4.10. The time differences
between the file numbers added is analysed.
Table 4.10:
Time Taken of Text Classification Test for every 50 File Addition Without Multicore-
Multithreading
Text Files Time Taken (seconds)
~50 021
100 0.76
150 1.71
200 5.68

Experiments show that ACO text classification has recorded different average
running time as the number of files increased as listed in Table 4.11. The experiment
was done to analyse ACO classification running time as the data set becomes larger.
Table 4.11:
Time Taken to Classify Text using ACO Featuring Multicore-Multithreading
Text Files Time Taken in Multicore-
Multithreading (seconds)
~50 018
100 0.49
150 1.27
200 2.28

Table 4.11 shows the time taken to run classification test using ACO with
multicore-multithreading. For the test that used 50 files, the time taken to classify the
entire test file was 0.18 seconds. The time recorded for 100 files to be classified is
0.49 seconds. The next file addition involving 150 files shows that 1.27 seconds was
needed to run the test. ACO which uses multicore-multithreading feature requires 2.28
seconds to classify 200 files as compare to the other file addition.

97

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Table 4.12:
Time Difference between Single Threaded and Multicore-Multithreaded Version ofACO
Text Files Time Taken Time Taken (Multicore- Time Difference
Multithreading)
50 021 (U8 Faster by 14.28%
100 0.76 0.49 Faster by 35.52%
150 1.71 1.27 Faster by 25.73%
200 5.68 2.28 Faster by 59.85%

Table 4.12 shows the differences in time required to run ACO text
classification between single and multicore-multithreaded version of ACO. For 50
files used in the test, there is 14.28% time difference in which the multicore-
multithreaded ACO is faster by 0.03 seconds from the single threaded ACO. The next
test involving 100 files shows that multicore-multithreaded ACO lead the time
reduction race by a gap of 35.52%. The result is ahead of the single threaded ACO by
0.27 seconds. The current amount of file test is the second highest time reduction
improvement compare to all other file addition tested. The next test involving the use
of 150 files have reduced the time by 25.73% for multicore-multithreaded ACO time
improvement which shows a total of 0.44 second time saving. The highest file
addition of the test which involved the use of 200 file additions produced the highest
amount of time reduction by 59.85% with 3.4 seconds of gap.

4.7.2 Discussion

One of the advantages of multitasking is that the resource management could

be used more efficiently. The computer technology has been evolving rapidly since
the last few decades. Instead of the single core processor, multicore processor
becomes the normal products sold in the market. Manufacturing multicore machine
could be really cheap and has unnoticeable cost in comparison to single core machine.
This research attempt to utilize the feature as multitasking is managed with the current
flow of computer technologies.
Pheromone sharing is one the focused problems that this research is trying to
solve in multitasking development. Sharing a resource can actually slow the

98

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

classification process because multiple instances need to access the pheromone one by
one from a queue. Roughly, the more artificial ants participate in solution searching,
the longer the queue. One of the reasons the test had to be conducted is to see how
much the waiting time overhead could affect the whole running time. Pheromone
sharing time overhead needs the most attention because pheromone usage is one of the
most frequent features used in ACO. It is also the core for processing calculation that
carries sensitive information.
For sequence file addition from 50 files to 200 files, the test result shows that
ACO with multicore-multithreading integrations has successfully reduced
classification time. Similar to the previous file addition, the communication overhead
does not completely disappear but has become less noticeable. The result shown has
fulfilled the objective to integrate ACO in multicore-multithread environment.
One of research question mentioned is could we get the most time reduction if
we use a large number of threads? From the test that we had analysed, the answer is
no. Even though we run the same exact test over and over again, there is no guarantee
that the running time average remains the same. The time could show identical time
reduction in the range of the file addition test is used but the time recorded is
inconsistent to a margin. The reason for this situation could be caused by the operating
system schedule that always changes from time to time. The threaded created by ACO
model can sometime be paused frequently and are instructed to execute other tasks by
the operating system. The details on how the mechanism works has been discussed in
Chapter 3.
Research questions raised for multicore-multithreading is could a software
shorten the execution time by eliminating sharing variable usage. This is done
especially on the variable that needs the most update at all time. It is theoretically true
that eliminating variable sharing could reduce execution time. However, it is difficult
to test the assumption without going through a very detailed discussion. The factors
involve specific hardware issue for testing in hardware level and manually setting up
the operating system scheduling. On top of that, processor manufacturer mostly has
different and unique hardware design on how the multicore processor handle
multithreading request from software level. Therefore, it gives inconsistent software
execution time.

99

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

4.8 Summary

This chapter presented the tests and discussion done of this research. One of
the tests compared between ACO and GA in the aspect of classification accuracy.
From the result, GA has higher accuracy when run on lower number of text files.
ACO on the other hand has better classification accuracy on medium to high
dimension of data than GA. According to the method used, the discussion relates the
effectiveness of tournament concept of GA which occur on increasing individual
elitism in scaled population. Multiple peak solution that has higher chance to happen
at high dimensional data could deteriorate GA classification accuracy as training
dataset increased in the tests. To counter the problem, ACO has stagnation mechanism
that react when it reaches a peak and will be bound or reset its pheromone so that a
new exploration is done. Further test is done to test time reduction for ACO in
multicore-multithread environment. Based on the result, the overall speed gain has
meet the objective requirement that is to reduce time consumption of ACO
classification time. The reason discussed for ACO to achieve positive result is the use
of pheromone sharing and use of global variable to eliminate unnecessary sharing
variables. This feat has directly make the classification efficient as memory resource
operate in shorter time and require no more than single copy.

100

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 CHAPTER FIVE

CONCLUSIONS AND FUTURE WORKS

5.1 Introduction

This chapter concludes the research contributions as well as a few suggestions

that can be done in the future. The suggestions are included to increase the activities
of text classification using ACO in the context of multicore-multithread environment.
This chapter is organized as follow: Research Outcome; Research Contribution and
lastly Future Suggestion.

5.2 Research Outcome

The first objective of this research is to figure out optimization algorithm for
text classification. As the scope of bio-inspired algorithm, ACO is chosen to classify
text from a general collection which consists of two distinctive classes. Regardless
what the topics are, the classes could at least be distinguished normally by humanoid
evaluation. ACO that is adjusted from TSP problem solving is needed to find the
systematic criterion in order to run a classification process. The objective is achieved
when the adjustment brings satisfying result and is used for further research expansion
by using it in multicore-multithread environment. ACO is found to be suitable in text
classification when classification rule can be done cooperatively. There are many
instances of artificial ant that can be set out to work together using pheromone
naturally as a mean of communication. The traveling artificial ant behaviour closely
resemble the real ant behaviour in savaging for food where more route are created
until the food source is found and choose shorter path comparatively. The situation of
visiting the path randomly is drawn back as a text classification algorithm. With
collective classification rule over repeating cycle, the rule is adjusted and evaluated
constantly in building up a general classification solution. Since ACO is a wrapper
type for classification method, the comparison is made with other techniques in the
same category. GA is chosen because it is also a bio-inspired type that produces
solution in an evolutionary way. The ACO model is run against the third party

101

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

software, Olex-GA that is integrated with WEKA. From the result, ACO model shows
comparable classification accuracy.
The second objective of this research is to integrate the ACO searching method
into a multicore-multithread environment. The integration tasks demand that the
artificial ant must be working simultaneously over multiple cores that are
manufactured in a set of pipeline. There has been some arguments that address the
multithreading as just a copy of parallelism. Discussion and explanation of deeper
structure were done to point out the distinctive criteria and to favour the
multithreading selection over parallelism. The multicore-multithread environment
design can affect how the pheromone sharing is done collaboratively. Race conditions
happens when two or more thread run on same object without correct synchronization
and the operation overlap on each other. Data consistency issue arises on pheromone
sharing when it is updated individually for every child thread. The issue is overcome
by the process of repetition and centralization of the pheromone copy that is held by
the master node or the main thread. The implementation is extended to design the
system to handle unstable or varying execution capability of child thread. The model
is adjusted to back up slower threads. For single copy of pheromone problem, a queue
was added to facilitate pheromone update and prevent data inconsistency in addition
to cater the update of slower threads. Child thread interrupt mechanism is also
introduced to aid stopping signal among other child threads that are used together with
the main or the parent thread synchronization. The features used are available with
easier implementation of multithreading.
The third objective of this research is to evaluate ACO in multicore-
multithread environment. The main reason of implementing multicore-multithread
environment is to reduce time taken for executing path searching. The time reduction
is one of the important features to achieve high efficiency in order to reduce operating
cost. Test had been done to observe on how much time decrease along all of the tests
that were executed with ACO without multithreading. Thread switching to update
shorter execution is not prioritized for a single threading usage. There is no real
benefit to gain from running unbreakable code part in different way than a
straightforward method. However, other tests that comprise bigger number of file set
showed positive result. The pattern observed showed an irregular time reduction
within a stable margin. The irregular benefit gain from using multithreading can be
explained by unstable execution speed between child threads. The unstable thread

102

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

capability to execute a task is a welcomed feature that depicts the different capability
of real ants in real world. While slower threads could reduce overall performance,
centralized pheromone sharing that requires a queue has hide partial devaluation of
execution speed. Therefore, all objective are met and achieve the main aim of the
research.

5.3 Research Contribution

The main contribution of this research is the ACO for mining text
classification rules in order to categorize text document. The developed framework
has integrated a few of the following activities:
1) Text classification using ACO algorithm in multicore-multithread
environment.
2) Theory replication that depicts natural problem solving in real life
traveling salesman. The problem criterion similarity that is addressed in
theoretical bio-inspired could simplify the learning and understanding
progression. The statistical method could introduce straightforward
method but bio-inspired could be more suitably natural and human
friendly.
3) Method conversion from ACO algorithm from TSP solution is adjusted
to term node relationship.
Besides that, this research is contributing a few values in multicore-
multithread environment implementation. The contributions are listed as follows:
1) Multicore-multithread environment design that needs to be adapted in
ACO model need to comply with its specific requirement. One of them
is the multiple core need to handle a number of threads processing. In
contrast to parallelism, multithreading does not wait for the slower or
interrupted child thread. Therefore the pheromone value must have
dynamic update to keep its integrity at all time.
2) Improvement over single threaded version of ACO in term of execution
efficiency and time reduction. The bottleneck has also been found in
which increasing the number of thread does not give stable rate of time
reduction. This is true when the nine threads used in the experiment

103

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

 produce maximal time reduction over the other number of thread used.
Otherwise, the model gives an acceptable time reduction.
Lastly, this research gives additional information to text classification
processing in artificial intelligent method by comparing ACO to GA as equivalent
algorithm. The wrapper method that explore bio-inspired course is the less popular
method since its overall performance has less accuracy and efficiency as compare to
statistical method. It is hoped that the research will improve bio-inspired potential
closer to the higher performance method. All of the contributions mentioned should be
a good reference to other researchers to continue with their development and
improvement.

5.4 Future Suggestion

The research has achieved its initial objectives. However, there are a lot of
improvements that can be done for future research on text classification in multicore-
multithread environment scope. Some of the main suggestions that cover both ACO
for text classification and its multicore-multithread implementation are listed as
follows:
a) In the future, the ACO in this research is hoped to be adapted in cloud
environment. Cloud environment use of ACO in multi colony method
as oppose to single colony method is known as cellular method. The
testing case could be adjusted to suit multi colony structure. The
performance in term of classification speed can be compared with
cellular method and the result could suggest an important overall
performance as a good ACO algorithm wide application. Multi colony
can be broken into different partition such as course, medium and fine
grain of granularity.
b) The developed technique in this research also could be used to mine
different field of data such as opinion mining. Opinion mining has
more abstract definition thus require a deeper care to differentiate the
classification rule. Opinion mining is useful to evaluate one's
behaviour and favourite subjects. It has marketing value and requires
more exploration from classification research.

104

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

c) Another desired improvement that can be added to the model in the
future is to increase the number of classes to more than two. As of now,
only two classes can be processed at a time. The time reduction also
plays important feature because a wrapper method tends to take a
higher processing time when more instances are added as opposed to
non-wrapper methods.
d) One of the desired features that can be added to the model in the future
is to allow the model to process different kind of data other than text
such as video, sound and images classification. The communication of
such medium has been widely used over the world especially internet
and teleconferences.
e) Another feature that can be included later is to allow the software to
process different languages. This can be an important feature as
international communication is rapidly growing and expanding. There
are a lot of important document in different languages that are needed
for text classification.
f) One of the desired features in multicore-multithread environment is a
preprocessing mechanism that evaluates and solves optimization
setting of a workload that will be run subsequently in a queue. There is
limitation for a task or job that is breakable to its smallest
decomposition including text classification. Machine that has multiple
core and multithreading technologies usually costs one physical core to
manage other core job distribution while other machine architecture
takes half of the amount. To figure out the most optimized setting in
assigning the number of artificial ants, the suggested pre-processing
mechanism should be run prior to the actual execution. It measures the
most beneficial number of artificial ants and slave machine required for
a text classification to run.

105

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

REFERENCES

Aaron, C. (2011, September 12). Enable Multiple Cores on Windows 7. Retrieved

from http://www.tech-recipes.com/rx/16720/enable-multiple-cores-on-
windows-7/
Aggarwal, C. C , & Zhai, C. (2012). A survey of text classification algorithms. In
Mining text data (pp. 163-222). Springer US.
Applegate, D., Bixby, R., Chvatal, V., & Cook, W. (2006). Concorde TSP solver.
Baharudin, B., Lee, L. H., & Khan, K. (2010). A review of machine learning
algorithms for text-documents classification. Journal of advances in
information technology, 7(1), 4-20.
Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python. "
O'Reilly Media, Inc.".
Bonasio, R., Zhang, G., Ye, C , Mutti, N. S., Fang, X., Qin, N., ... & Liebig, J. (2010).
Genomic comparison of the ants Camponotus floridanus and Harpegnathos
saltator. science, 329(5995), 1068-1071.
Bullnheimer, B., Kotsis, G., & StrauB, C. (1998). Parallelization strategies for the ant
system. In High Performance Algorithms and Software in Nonlinear
Optimization (pp. 87-100). Springer US.
Caruana, R. A., & Freitag, D. (1994). Greedy attribute selection. Proceedings of the
Eleventh International Conference on Machine Learning (pp. 28- 36). New
Brunswick, N J: Morgan Kaufmann.
Cecilia, J. M., Garcia, J. M., Nisbet, A., Amos, M., & Ujaldon, M. (2013). Enhancing
data parallelism for ant colony optimization on GPUs. Journal of Parallel and
Distributed Computing, 73(1), 42-51.
Chen, Y. S., Liao, H. C , & Tsai, T. H. (2013). Online real-time task scheduling in
heterogeneous multicore system-on-a-chip. IEEE Transactions on Parallel and
Distributed Systems, 24(1), 118-130.
Deerwester, S. et al. (1988). Improving Information Retrieval with Latent Semantic
Indexing, Proceedings of the 51st Annual Meeting of the American Society for
Information Science 25, 1988, pp. 36-40.

106

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

DeleVacq, A., Delisle, P., Gravel, M., & Krajecki, M. (2013). Parallel ant colony
optimization on graphics processing units. Journal of Parallel and Distributed
Computing, 73(1), 52-61.
Dorigo, M., Maniezzo, V., & Colorni, A., (1991) The Ant System: An autocatalytic
optimizing process. Dipartimento di Elettronica, Politecnico di Milano, Italy
on Technical Report 91-016 Revised.
Dorigo, M. (1992). Optimization, learning and natural algorithms. Ph. D. Thesis,
Politecnico di Milano, Italy.
Dorigo, M., Maniezzo, V., & Colorni, A. (1996). Ant system: optimization by a
colony of cooperating agents. IEEE Transactions on Systems, Man, and
Cybernetics, Part B (Cybernetics), 26(1), 29-41.
Dorigo, M., & Gambardella, L. M. (1997). Ant colonies for the travelling salesman
problem. BioSystems, 43(2), 73-81
Dorigo, M., & Thomas, S. (2004). Ant Colony Optimization. Cambridge, vol. 9, Dec.
2002.
Dressier, F., & Akan, O. B. (2010). A survey on bio-inspired networking. Computer
Networks, 54(6), 881-900.
Emily, S. (2014, April 9). The Remarkable Self-Organization of Ants. Retrieved from
https://www.quantamagazine.org/20140409-the-remarkable-self-organization-
of-ants/
Freitas, A. A., Parpinelli, R. S., & Lopes, H. (2008). Ant colony algorithms for data
classification. Encyclopedia of Information Science and Technology, I, 154-
159.
Gasanova, T., Sergienko, R., Semenkin, E., & Minker, W. (2014, September).
Dimension reduction with revolutionary genetic algorithm for text
classification. In Informatics in Control, Automation and Robotics (ICINCO),
2014 11th International Conference on (Vol. 1, pp. 215-222). IEEE.
Gennadiy, S. (2015, September 23). Intel Pentium 4 Processor Families. Retrieved
from http://www.cpu-world.com/CPUs/Pentium_4/
Grasse, P. P. (1959). The reconstruction of the nest and coordination inter chez
Bellicositermes natalensis etCubitermes sp. theory stigmergy: Interpretive
Essay behavior of termites manufacturers. Soc insects, 6, 41-80.
Goss, S., Aron, S., Deneubourg, J. L., & Pasteels, J. M. (1989). Self-organized
shortcuts in the Argentine ant. Naturwissenschaften, 76(12), 579-581.

107

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Hassan, S., Rafi, M., & Shaikh, M. S. (2011, December). Comparing SVM and naive
bayes classifiers for text categorization with Wikitology as knowledge
enrichment. InMultitopic Conference (INMIC), 2011 IEEE 14th
International (pp. 31-34). IEEE.
Haughn, M. (2014, September). Translation Lookaside Buffer (TLB). Retrieved from
http://whatis.techtarget.com/definition/translation-look-aside-buffer-TLB
Hoffman, C. (2014, August 18). CPU Basics: Multiple CPUs, Cores, and Hyper-
Threading Explained. Retrieved from
http://www.howtogeek.com/194756/cpu-basics-multiple-cpus-cores-and-
hyper-threading-explained/
Hyotyniemi, H. (1996). Text document classification with self-organizing maps.
STeP'96, Genes, Nets and Symbols, 64-72.
Ivan, L., Svetozar, M., & Jerzy, W. (2009). Large-Scale Scientific Computing. 7th
International Conference, LSSC 2009, Sozopol, Bulgaria, June 4-8, 2009
Revised Papers
Janaki Meena, M., Chandran, K. R., Karthik, A., & Vijay Samuel, A. (2011). A
parallel ACO algorithm to select terms to categorise longer documents.
International Journal of Computational Science and Engineering, 6(4), 238-
248.
Jrennie. (2008, January 14). 20 Newsgroups. Retrieved from
http://qwone.com/~jason/20Newsgroups/
John, G. H., Kohavi, R., & Pfleger, K. (1994). Irrelevant features and the subset
selection problem. Proceedings of the Eleventh International Conference on
Machine Learning (pp. 121-129). New Brunswick, N J: Morgan Kaufmann.
Kachitvichyanukul, V. (2012). Comparison of three evolutionary algorithms: GA,
PSO, and DE. Industrial Engineering and Management Systems, 77(3), 215-
223.
Karypis, G., & Han, E. H. S. (2000, November). Fast supervised dimensionality
reduction algorithm with applications to document categorization & retrieval.
In Proceedings of the ninth international conference on Information and
knowledge management (pp. 12-19). ACM.
Kugu, E., & Sahingoz, O. K. (2013, October). ACO algorithms with multi-core
implementation. In Application of Information and Communication
Technologies (AICT), 2013 7th International Conference on (pp. 1-5). IEEE.

108

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Korosec, P., Vajtersic, M , Silc, J., & Kutil, R. (2013). Multi-core implementation of
the differential ant-stigmergy algorithm for numerical optimization. The
Journal of Supercomputing, 63(3), 757-772.
Langley, P. (1994, November). Selection of relevant features in machine learning.
In Proceedings of the AAAI Fall symposium on relevance (Vol. 184, pp. 245-
271).
Lawler, E. L., Lenstra, J. K., Rinnooy-Kan, A. G., & Shmoys, D. B. (1985). traveling
salesman problem.[The].
Lee, L. H., Wan, C. H., Rajkumar, R., & Isa, D. (2012). An enhanced support vector
machine classification framework by using Euclidean distance function for
text document categorization. Applied Intelligence, 37(1), 80-99.
Liao, T., Dorigo, M., & Stutzle, T. (2011). Improved ant colony optimization
algorithms for continuous and mixed discrete-continuous optimization
problems (Doctoral dissertation, Masters thesis, Universite Libre de Bruxelles,
Belgium).
Liu, Y. & Chang, G.-L. (2011), An arterial signal optimization model for intersections
experiencing queue spillback and lane blockage. Transportation Research Part
C: Emerging Technologies, 19(1), 130-44.
Mavrovouniotis, M., & Yang, S. (2013, April). Adapting the pheromone evaporation
rate in dynamic routing problems. In European Conference on the
Applications of Evolutionary Computation^^. 606-615). Springer Berlin
Heidelberg.
Meena, M. J., Chandran, K. R., Karthik, A., & Samuel, A. V. (2012). An enhanced
ACO algorithm to select features for text categorization and its
parallelization. Expert Systems with Applications, 39(5), 5861-5871.
Meng, J., Lin, H., & Yu, Y. (2011). A two-stage feature selection method for text
categorization. Computers & Mathematics with Applications, 62(1), 2793-
2800.
Moth, D. (2008, November 3). Threading/Concurrency vs Parallelism. Retrieved from
http://www.danielmoth.com/Blog/threadingconcurrency-vs-parallelism.aspx
Nalini, K., & Sheela, L. J. (2014). Survey on Text Classification.
Napster. (2010, January 11). Forking vs Threading. Retrieved from
http://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel/

109

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Nazeem, A., Reveliotis, S., Wang, Y., & Lafortune, S. (2011). Designing compact and
maximally permissive deadlock avoidance policies for complex resource
allocation systems through classification theory: The linear case.IEEE
Transactions on Automatic Control, 5(5(8), 1818-1833.
Nezamabadi-pour, H., Saryazdi, S., & Rashedi, E. (2006). Edge detection using ant
algorithms. Soft Computing, 70(7), 623-628.
NVIDIA, NVIDIA CUD A. (2010) C Programming Guide 3.1.1
Otero, F. E., Freitas, A. A., & Johnson, C. G. (2008, September). cAnt-Miner: an ant
colony classification algorithm to cope with continuous attributes.
In International Conference on Ant Colony Optimization and Swarm
Intelligence (pp. 48-59). Springer Berlin Heidelberg.
Pahuja, H., Chhabra, J., & Khokhar, A. (2012). Ant Colony Optimization-based
Selected Features for Text-Independent Speaker Verification. International
Journal of Engineering Research and Applications (IJERA), pp. 1469
Pietramala, A., Policicchio, V. L., Rullo, P., & Sidhu, I. (2008, September). A genetic
algorithm for text classification rule induction. In Joint European Conference
on Machine Learning and Knowledge Discovery in Databases(pp. 188-203).
Springer Berlin Heidelberg.
Policicchio, V. L., Pietramala, A., & Rullo, P. (2012). GAMoN: Discovering M-of-N
{^,V} hypotheses for text classification by a lattice-based Genetic
Algorithm. Artificial Intelligence, 191, 61-95.
Preshing, J. (2013, June 18). Atomic vs. Non-Atomic Operations. Retrieved from
http ://preshing. com/20130618/atomic-vs-non-atomic-operations/
Putha, R., Quadrifoglio, L., & Zechman, E. (2012). Comparing ant colony
optimization and genetic algorithm approaches for solving traffic signal
coordination under oversaturation conditions. Computer-Aided Civil and
Infrastructure Engineering, 27(1), 14-2 8.
Ratnieks, F. L. (2015). Trail Pheromones: An Integrative View of Their Role in Social
Insect Colony Organization. Annu. Rev. Entomol, 60, 581-99.
Reinelt, G. (1994). The traveling salesman: computational solutions for TSP
applications. Springer-Verlag.
Rekaya, R., Robbins, K., Spangler, M., Smith, S., Hay, E. H., & Bertrand, K. (2013).
Ant Colony Algorithm with Applications in the Field of Genomics. Ant Colony
Optimization-Techniques and Applications, 177.

110

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Ressom, H. W., Varghese, R. S., Drake, S. K., Hortin, G. L., Abdel-Hamid, M.,
Loffredo, C. A., & Goldman, R. (2007). Peak selection from MALDI-TOF
mass spectra using ant colony optimization. Bioinformatics, 23(5), 619-626.
Feldman, R., & Sanger, J. (2007). The text mining handbook: advanced approaches in
analyzing unstructured data. Cambridge University Press.
Fujitsu, (n.d.) . In High Throughput SPARC T4 Processors. Retrieved from
http://www.fujitsu.com/global/products/computing/servers/unix/sparc-
enterprise/technology/performance/processor3.html
Runger, G. (2009) Mapping and Scheduling of Parallel Task for Multi-core Systems.
Retrieved from http://complexhpc.org/events/lisbon/PDF/Runger.pdf
Saeys, Y., Inza, I., & Larranaga, P. (2007). A review of feature selection techniques in
bioinformatics. bioinformatics, 23(19), 2507-2517.
Salama, K. M., Abdelbar, A. M., Otero, F. E., & Freitas, A. A. (2013). Utilizing
multiple pheromones in an ant-based algorithm for continuous-attribute
classification rule discovery. Applied Soft Computing, 73(1), 667-675.
Sara9, E., & Ozel, S. A. (2014). An ant colony optimization based feature selection for
web page classification. The Scientific World Journal, 2014.
Sharma, M., Elmiligi, H., & Gebali, F. (2014). SMARTs: A Tool to Simulate and
Analyze the Performance of Real-time Multi-core Systems. Procedia
Computer Science, 34, 544-551.
Scali. (2012). Multi-core and multi-threading performance (the multi-core myth?).
Retrieved from https ://scalibq. wordpress.com/2012/06/01 /multi-core-and-
multi-threading/
Sriram, B., Fuhry, D., Demir, E., Ferhatosmanoglu, H., & Demirbas, M. (2010, July).
Short text classification in twitter to improve information filtering.
In Proceedings of the 33rd international ACM SIGIR conference on Research
and development in information retrieval (pp. 841-842). ACM.
Stiitzle, T., & Hoos, H. H. (2000). MAX-MIN ant system. Future generation
computer systems, 16(8), 889-914.
Tian, J., Yu, W., & Xie, S. (2008, June). An ant colony optimization algorithm for
image edge detection. In 2008 IEEE Congress on Evolutionary Computation
(IEEE World Congress on Computational Intelligence) (pp. 751-756). IEEE.
Thorleuchter, D., & Van den Poel, D. (2013). Technology classification with latent
semantic indexing. Expert Systems with Applications, 40(5), 1786-1795.

Ill

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Umbarkar, A. J., Joshi, M. S., & Hong, W. C. (2014). Multithreaded Parallel Dual
Population Genetic Algorithm (MPDPGA) for unconstrained function
optimizations on multi-core system. Applied Mathematics and
Computational, 936-949.
Uysal, A. K., & Gunal, S. (2014). Text classification using genetic algorithm oriented
latent semantic features. Expert Systems with Applications, 4/(13), 5938-5947.
Vaidya, J., Shafiq, B., Basu, A., & Hong, Y. (2013, November). Differentially private
naive Bayes classification. In Proceedings of the 2013 IEEE/WIC/ACM
International Joint Conferences on Web Intelligence (WI) and Intelligent
Agent Technologies (lAT)-Volume 01 (pp. 571-576). IEEE Computer Society.
Vitins, B. J., & Axhausen, K. W. (2009). Optimization of large transport networks
using the ant colony heuristic. Computer-Aided Civil and Infrastructure
Engineering, 24(1), 1-14.
Wang, S., & Manning, C. D. (2012, July). Baselines and bigrams: Simple, good
sentiment and topic classification. In Proceedings of the 50th Annual Meeting
of the Association for Computational Linguistics: Short Papers-Volume 2 (pp.
90-94). Association for Computational Linguistics.
Wang, Q., Xu, J., Li, H., & Craswell, N. (2013). Regularized latent semantic indexing:
A new approach to large-scale topic modeling. ACM Transactions on
Information Systems (TOIS), 37(1), 5.
Wang, W., & Yu, B. (2009). Text categorization based on combination of modified
back propagation neural network and latent semantic analysis. Neural
Computing and Applications, 75(8), 875-881.
Wang, X. P., & Cao, L. M. (2002). Genetic algorithm—theory, application and
software implementation. Xi 'an Jiaotong University Press, Xi 'an.
Wu, X., Kumar, V., Quinlan, J. R., Ghosh, J., Yang, Q., Motoda, H., ... & Zhou, Z. H.
(2008). Top 10 algorithms in data mining. Knowledge and information
systems, 14(\), 1-37.
Wyatt, T. D. (2015). How animals communicate via pheromones. American
Scientist, 103(2), 114.
Yang, X. Q., Sun, N., Sun, T. L., Cao, X. Y., & Zheng, X. J. (2009). The application
of latent semantic indexing and ontology in text classificationJnternational
Journal of Innovative Computing, Information and Control, 5(12), 4491-4499.

112

COPYRIGHT UNIVERSITI TEKNOLOGI MARA

Yang, Z., Yu, B., & Cheng, C. (2007). A parallel ant colony algorithm for bus
network optimization. Computer-Aided Civil and Infrastructure
Engineering^ 2(1), 44-55.
Yu, B., Xu, Z. B., & Li, C. H. (2008). Latent semantic analysis for text categorization
using neural network. Knowledge-Based Systems, 27(8), 900-904.
Zhang, S., & Pan, X. (2011, March). A novel text classification based on Mahalanobis
distance. In Computer Research and Development (ICCRD), 2011 3rd
International Conference on (Vol. 3, pp. 156-158). IEEE.
Zhang, W., Yoshida, T., & Tang, X. (2011). A comparative study of TF* IDF, LSI
and multi-words for text classification. Expert Systems with
Applications, 38(3), 2758-2765.

113
COPYRIGHT UNIVERSITI TEKNOLOGI MARA
 AUTHOR'S PROFILE

Ahmad Nazmi bin Fadzal completed his Master in Computer Science at the
Faculty of Computer and Mathematical Science, Universiti Teknologi MARA. He
received his Bachelor in Computer Science at Kuliyyah of Information and
Communication Technology, International Islamic University Malaysia.

LIST OF PUBLICATIONS

o Conference on title of Ant Colony Algorithm for Text Classification

114

COPYRIGHT UNIVERSITI TEKNOLOGI MARA