Ayon Dey / (IJCSIT) International Journal of Computer Science and Information Technologies, Vol.

7 (3) , 2016, 1174-1179

Machine Learning Algorithms: A Review

Ayon Dey
Department of CSE, Gautam Buddha
University, Greater Noida, Uttar Pradesh, India

Abstract – In this paper, various machine learning algorithms

have been discussed. These algorithms are used for various II. TYPES OF LEARNING
purposes like data mining, image processing, predictive
analytics, etc. to name a few. The main advantage of using A. Supervised Learning
machine learning is that, once an algorithm learns what to do The supervised machine learning algorithms are those
with data, it can do its work automatically. algorithms which needs external assistance. The input
dataset is divided into train and test dataset. The train
Keywords – Machine learning, algorithms, pseudo code dataset has output variable which needs to be predicted or
classified. All algorithms learn some kind of patterns from
I. INTRODUCTION
the training dataset and apply them to the test dataset for
Machine learning is used to teach machines how to prediction or classification [4]. The workflow of supervised
handle the data more efficiently. Sometimes after viewing machine learning algorithms is given in Fig. 2. Three most
the data, we cannot interpret the pattern or extract famous supervised machine learning algorithms have been
information from the data. In that case, we apply machine discussed here.
learning [1]. With the abundance of datasets available, the
demand for machine learning is in rise. Many industries 1) Decision Tree: Decision trees are those type of trees
from medicine to military apply machine learning to extract which groups attributes by sorting them based on their
relevant information. values. Decision tree is used mainly for classification
The purpose of machine learning is to learn from the purpose. Each tree consists of nodes and branches. Each
data. Many studies have been done on how to make nodes represents attributes in a group that is to be classified
machines learn by themselves [2] [3]. Many and each branch represents a value that the node can take
mathematicians and programmers apply several approaches [4]. An example of decision tree is given in Fig. 3.
to find the solution of this problem. Some of them are The pseudo code for Decision tree is described in Fig. 4;
demonstrated in Fig. 1. where S, A and y are training set, input attribute and target
All the techniques of machine learning are explained in attribute respectively.
Section 2. Section 3 concludes this paper.

Fig. 1. Types of Learning [2] [3]

www.ijcsit.com 1
 Fig. 2. Workflow of supervised machine learning algorithm [4]

Fig. 4. Pseudo code for Decision Tree [5]

Fig. 3. Decision Tree [5]

2) Naïve Bayes: Naïve Bayes mainly targets the text

classification industry. It is mainly used for clustering and
classification purpose [6]. The underlying architecture of
Naïve Bayes depends on the conditional probability. It
creates trees based on their probability of happening. These
trees are also known as Bayesian Network. An example of
the network is given in Fig. 5. The pseudo code is given in
Fig. 6.

Fig. 6. Pseudo code for Naïve Bayes [6]

3) Support Vector Machine: Another most widely used

state-of-the-art machine learning technique is Support
Vector Machine (SVM). It is mainly used for classification.
Fig. 5. An Example of Bayesian Network [7] SVM works on the principle of margin calculation. It
basically, draw margins between the classes. The margins
are drawn in such a fashion that the distance between the
margin and the classes is maximum and hence, minimizing
the classification error. An example of working and pseudo
code of SVM is given in Fig. 7 and Fig. 8, respectively.

Fig. 10. K-Means Clustering [12]

Fig. 7. Working of Support Vector Machine [8]

Fig. 8. Pseudo code for Support Vector machine [9]

B. Unsupervised Learning
The unsupervised learning algorithms learns few features
from the data. When new data is introduced, it uses the
previously learned features to recognize the class of the
data. It is mainly used for clustering and feature reduction.
An example of workflow of unsupervised learning is given
in Fig. 9.
Fig. 11. Pseudo code for k-means clustering [13]

2) Principal Component Analysis

In Principal Component Analysis or PCA, the dimension of
the data is reduced to make the computations faster and
easier. To understand how PCA works, let’s take an
example of 2D data. When the data is being plot in a graph,
it will take up two axes. PCA is applied on the data, the
data then will be 1D. This is explained in Fig. 12. The
pseudo code for PCA is discussed in Fig. 13.
Fig. 9. Example of Unsupervised Learning [10]

The two main algorithms for clustering and

dimensionality reduction techniques are discussed below.
1) K-Means Clustering: Clustering or grouping is a type
of unsupervised learning technique that when initiates,
creates groups automatically. The items which possesses
similar characteristics are put in the same cluster. This
algorithm is called k-means because it creates k distinct
clusters. The mean of the values in a particular cluster is the
center of that cluster [9]. A clustered data is represented in Fig. 12. Visualization of data before and after applying PCA [11]
Fig. 10. The algorithm for k-means is given in Fig. 11.
 and their actions in the future. Reinforcement learning
solely depends on two criteria: trial and error search and
delayed outcome [17]. The general model [18] for
reinforcement learning is depicted in Fig. 14.

Fig. 14. The Reinforcement Learning Model [18]

Fig. 13. Pseudo code for PCA [14]
In the figure, the agent receives an input i, current state s,
C. Semi - Supervised Learning state transition r and input function I from the environment.
Semi – supervised learning algorithms is a technique Based on these inputs, the agent generates a behavior B and
which combines the power of both supervised and takes an action a which generates an outcome.
unsupervised learning. It can be fruit-full in those areas of E. Multitask Learning
machine learning and data mining where the unlabeled data
Multitask learning has a simple goal of helping other
is already present and getting the labeled data is a tedious
learners to perform better. When multitask learning
process [15]. There are many categories of semi-supervised
algorithms are applied on a task, it remembers the
learning [16]. Some of which are discussed below:
procedure how it solved the problem or how it reaches to
1) Generative Models: Generative models are one of the the particular conclusion. The algorithm then uses these
oldest semi-supervised learning method assumes a structure steps to find the solution of other similar problem or task.
like p(x,y) = p(y)p(x|y) where p(x|y) is a mixed distribution This helping of one algorithm to another can also be termed
e.g. Gaussian mixture models. Within the unlabeled data, as inductive transfer mechanism. If the learners share their
the mixed components can be identifiable. One labeled experience with each other, the learners can learn
example per component is enough to confirm the mixture concurrently rather than individually and can be much
distribution. faster [19].
2) Self-Training: In self-training, a classifier is trained F. Ensemble Learning
with a portion of labeled data. The classifier is then fed with
When various individual learners are combined to form
unlabeled data. The unlabeled points and the predicted
only one learner then that particular type of learning is
labels are added together in the training set. This procedure
called ensemble learning. The individual learner may be
is then repeated further. Since the classifier is learning
Naïve Bayes, decision tree, neural network, etc. Ensemble
itself, hence the name self-training.
learning is a hot topic since 1990s. It has been observed
3) Transductive SVM: Transductive support vector that, a collection of learners is almost always better at doing
machine or TSVM is an extension of SVM. In TSVM, the a particular job rather than individual learners [20]. Two
labeled and unlabeled data both are considered. It is used to popular Ensemble learning techniques are given below
label the unlabeled data in such a way that the margin is [21]:
maximum between the labeled and unlabeled data. Finding
1) Boosting: Boosting is a technique in ensemble
an exact solution by TSVM is a NP-hard problem.
learning which is used to decrease bias and variance.
Boosting creates a collection of weak learners and convert
them to one strong learner. A weak learner is a classifier
D. Reinforcement Learning which is barely correlated with true classification. On the
Reinforcement learning is a type of learning which other hand, a strong learner is a type of classifier which is
makes decisions based on which actions to take such that strongly correlated with true classification [21]. The pseudo
the outcome is more positive. The learner has no knowledge code for AdaBoost (which is most popular example of
which actions to take until it’s been given a situation. The boosting) is give in Fig. 15.
action which is taken by the learner may affect situations
 Fig. 18. Structure of an Artificial Neural Network [24]

An artificial neural network behaves the same way. It

works on three layers. The input layer takes input (much
like dendrites). The hidden layer processes the input (like
soma and axon). Finally, the output layer sends the
Fig. 15. Pseudo code for AdaBoost [21]
calculated output (like dendrite terminals) [24]. There are
basically three types of artificial neural network:
2) Bagging: Bagging or bootstrap aggregating is applied
supervised, unsupervised and reinforcement [25].
where the accuracy and stability of a machine learning
algorithm needs to be increased. It is applicable in 1) Supervised Neural Network: In the supervised neural
classification and regression. Bagging also decreases network, the output of the input is already known. The
variance and helps in handling overfitting [23]. The pseudo predicted output of the neural network is compared with the
code for bagging in given in Fig. 16. actual output. Based on the error, the parameters are
changed, and then fed into the neural network again. Fig. 19
will summarize the process. Supervised neural network is
used in feed forward neural network.

Fig. 16. Pseudo code for Bagging [21]

G. Neural Network Learning Fig. 19. Supervised Neural Network [25]

The neural network (or artificial neural network or ANN)
is derived from the biological concept of neurons. A neuron 2) Unsupervised Neural Network: Here, the neural
is a cell like structure in a brain. To understand neural network has no prior clue about the output the input. The
network, one must understand how a neuron works. A main job of the network is to categorize the data according
neuron has mainly four parts (see Fig. 17). They are to some similarities. The neural network checks the
dendrites, nucleus, soma and axon. correlation between various inputs and groups them. The
schematic diagram is shown in Fig. 20.

Fig. 17. A Neuron [24]

Fig. 20. Unsupervised Neural Network [25]
The dendrites receive electrical signals. Soma processes
the electrical signal. The output of the process is carried by 3) Reinforced Neural Network: In reinforced neural
the axon to the dendrite terminals where the output is sent network, the network behaves as if a human communicates
to next neuron. The nucleus is the heart of the neuron. The with the environment. From the environment, a feedback
inter-connection of neuron is called neural network where has been provided to the network acknowledging the fact
electrical impulses travel around the brain. that whether the decision taken by the network is right or
wrong. If the decision is right, the connections which points III. CONCLUSION
to that particular output is strengthened. The connections
are weakened otherwise. The network has no previous This paper surveys various machine learning algorithms.
information about the output. Reinforced neural network is Today each and every person is using machine learning
represented in Fig. 21. knowingly or unknowingly. From getting a recommended
product in online shopping to updating photos in social
4) networking sites. This paper gives an introduction to most
of the popular machine learning algorithms.

REFERENCES
[1] W. Richert, L. P. Coelho, “Building Machine Learning Systems with
Python”, Packt Publishing Ltd., ISBN 978-1-78216-140-0
[2] M. Welling, “A First Encounter with Machine Learning”
[3] M. Bowles, “Machine Learning in Python: Essential Techniques for
Predictive Analytics”, John Wiley & Sons Inc., ISBN: 978-1-118-
96174-2
[4] S.B. Kotsiantis, “Supervised Machine Learning: A Review of
Classification Techniques”, Informatica 31 (2007) 249-268
Fig. 21. Reinforced Neural Network [25]
[5] L. Rokach, O. Maimon, “Top – Down Induction of Decision Trees
Classifiers – A Survey”, IEEE Transactions on Systems,
H. Instance-Based Learning [6] D. Lowd, P. Domingos, “Naïve Bayes Models for Probability
In instance-based learning, the learner learns a particular Estimation”
[7] https://webdocs.cs.ualberta.ca/~greiner/C-
type of pattern. It tries to apply the same pattern to the 651/Homework2_Fall2008.html
newly fed data. Hence the name instance-based. It is a type [8] D. Meyer, “Support Vector Machines – The Interface to libsvm in
of lazy learner which waits for the test data to arrive and package e1071”, August 2015
then act on it together with training data. The complexity of [9] S. S. Shwartz, Y. Singer, N. Srebro, “Pegasos: Primal Estimated sub -
Gradient Solver for SVM”, Proceedings of the 24th International
the learning algorithm increases with the size of the data. Conference on Machine Learning, Corvallis, OR, 2007
Given below is a well-known example of instance-based [10] http://www.simplilearn.com/what-is-machine-learning-and-why-it-
learning which is k-nearest neighbor [26]. matters-article
[11] P. Harrington, “Machine Learning in action”, Manning Publications
1) K-Nearest Neighbor: In k-nearest neighbor (or KNN), Co., Shelter Island, New York, 2012
the training data (which is well-labeled) is fed into the [12] http://pypr.sourceforge.net/kmeans.html
[13] K. Alsabati, S. Ranaka, V. Singh, “An efficient k-means clustering
learner. When the test data is introduced to the learner, it algorithm”, Electrical Engineering and Computer Science, 1997
compares both the data. k most correlated data is taken from [14] M. Andrecut, “Parallel GPU Implementation of Iterative PCA
training set. The majority of k is taken which serves as the Algorithms”, Institute of Biocomplexity and Informatics, University
new class for the test data [27]. The pseudo code for KNN of Calgary, Canada, 2008
[15] X. Zhu, A. B. Goldberg, “Introduction to Semi – Supervised
is given in Fig. 22. Learning”, Synthesis Lectures on Artificial Intelligence and Machine
Learning, 2009, Vol. 3, No. 1, Pages 1-130
[16] X. Zhu, “Semi-Supervised Learning Literature Survey”, Computer
Sciences, University of Wisconsin-Madison, No. 1530, 2005
[17] R. S. Sutton, “Introduction: The Challenge of Reinforcement
Learning”, Machine Learning, 8, Page 225-227, Kluwer Academic
Publishers, Boston, 1992
[18] L. P. Kaelbing, M. L. Littman, A. W. Moore, “Reinforcement
Learning: A Survey”, Journal of Artificial Intelligence Research, 4,
Page 237-285, 1996
[19] R. Caruana, “Multitask Learning”, Machine Learning, 28, 41-75,
Kluwer Academic Publishers, 1997
[20] D. Opitz, R. Maclin, “Popular Ensemble Methods: An Empirical
Study”, Journal of Artificial Intelligence Research, 11, Pages 169-
198, 1999
[21] Z. H. Zhou, “Ensemble Learning”, National Key Laboratory for
Novel Software Technology, Nanjing University, Nanjing, China
[22] https://en.wikipedia.org/wiki/Boosting_(machine_learning)
[23] https://en.wikipedia.org/wiki/Bootstrap_aggregating
[24] V. Sharma, S. Rai, A. Dev, “A Comprehensive Study of Artificial
Neural Networks”, International Journal of Advanced Research in
Computer Science and Software Engineering, ISSN 2277128X,
Volume 2, Issue 10, October 2012
[25] S. B. Hiregoudar, K. Manjunath, K. S. Patil, “A Survey: Research
Summary on Neural Networks”, International Journal of Research in
Engineering and Technology, ISSN: 2319 1163, Volume 03, Special
Issue 03, pages 385-389, May, 2014
[26] https://en.wikipedia.org/wiki/Instance-based_learning
[27] P. Harrington, “Machine Learning in Action”, Manning Publications
Co., Shelter Island, New York, ISBN 9781617290183, 2012
Fig. 22. Pseudo code for K-Nearest Neighbor [28] [28] J. M. Keller, M. R. Gray, J. A. Givens Jr., “A Fuzzy K-Nearest
Neighbor Algorithm”, IEEE Transactions on Systems, Man and
Cybernetics, Vol. SMC-15, No. 4, August 1985