Naive Bayes

Classifiers
COLLECTION OF CLASSIFICATION ALGORITHMS
Principle of Naive Bayes
Classifier:
 A Naive Bayes classifier is a probabilistic machine learning model
that’s used for classification task. The crux of the classifier is
based on the Bayes theorem.

 Bayes theorem can be rewritten as:

 It is not a single algorithm but a family of algorithms where all of

them share a common principle, i.e. every pair of features being
classified is independent of each other.
Example:
Let us take an example to get some better intuition. Consider the problem of playing golf. The dataset is represented as below.
 We classify whether the day is suitable for playing golf, given the
features of the day. The columns represent these features and the
rows represent individual entries. If we take the first row of the
dataset, we can observe that is not suitable for playing golf if the
outlook is rainy, temperature is hot, humidity is high and it is not
windy. We make two assumptions here, one as stated above we
consider that these predictors are independent. That is, if the
temperature is hot, it does not necessarily mean that the humidity
is high. Another assumption made here is that all the predictors
have an equal effect on the outcome. That is, the day being windy
does not have more importance in deciding to play golf or not.
 According to this example, Bayes theorem can be rewritten as:

 The variable y is the class variable(play golf), which represents if it

is suitable to play golf or not given the conditions.
Variable X represent the parameters/features.
 X is given as,
 Here x_1,x_2….x_n represent the features, i.e they can be mapped
to outlook, temperature, humidity and windy. By substituting
for X and expanding using the chain rule we get,

 Now, you can obtain the values for each by looking at the dataset
and substitute them into the equation. For all entries in the dataset,
the denominator does not change, it remain static. Therefore, the
denominator can be removed and a proportionality can be
introduced.

 In our case, the class variable(y) has only two outcomes, yes or no.
There could be cases where the classification could be multivariate.
Therefore, we need to find the class y with maximum probability.

 Using the above function, we can obtain the class, given the
predictors.
 We need to find P(xi | yj) for each xi in X and yj in y. All these
calculations have been demonstrated in the tables below:

 So, in the figure above, we have calculated P(x i | yj) for each xi in X
and yj in y manually in the tables 1-4. For example, probability of
playing golf given that the temperature is cool, i.e P(temp. = cool |
play golf = Yes) = 3/9.
 Also, we need to find class probabilities (P(y)) which has been calculated in the
table 5. For example, P(play golf = Yes) = 9/14.
 So now, we are done with our pre-computations and the classifier is ready!
 Let us test it on a new set of features (let us call it today):
Types of Naive Bayes Classifier:

 Multinomial Naive Bayes: This is mostly used for document

classification problem, i.e whether a document belongs to the
category of sports, politics, technology etc. The
features/predictors used by the classifier are the frequency of the
words present.

 Bernoulli Naive Bayes: This is similar to the multinomial naive

bayes but the predictors are boolean variables. The parameters
that we use to predict the class variable take up only values yes
or no, for example if a word occurs in the text or not.

 Gaussian Naive Bayes: When the predictors take up a

continuous value and are not discrete, we assume that these
values are sampled from a gaussian distribution.


Gaussian Distribution(Normal Distribution)

Conclusion:
Naive Bayes algorithms are mostly used in sentiment analysis, spam
filtering, recommendation systems etc. They are fast and easy to
implement but their biggest disadvantage is that the requirement of
predictors to be independent. In most of the real life cases, the predictors
are dependent, this hinders the performance of the classifier.
Decision Tree
CLASSIFICATION ALGORITHM
 Decision tree algorithm falls under the category of supervised learning. They can
be used to solve both regression and classification problems..
 Decision tree builds classification or regression models in the form of a tree
structure. It breaks down a dataset into smaller and smaller subsets while at the
same time an associated decision tree is incrementally developed. The final
result is a tree with decision nodes and leaf nodes. A decision node (e.g.,
Outlook) has two or more branches (e.g., Sunny, Overcast and Rainy). Leaf node
(e.g., Play) represents a classification or decision. The topmost decision node in a
tree which corresponds to the best predictor called root node. Decision trees
can handle both categorical and numerical data.
 We can represent any boolean function on discrete attributes using the decision
tree.
 Types of decision trees
 Categorical Variable Decision Tree: Decision Tree which has categorical target
variable then it called as categorical variable decision tree.
 Continuous Variable Decision Tree: Decision Tree which has continuous target
variable then it is called as Continuous Variable Decision Tree.
 Root Node: It represents entire population or sample and this further gets
divided into two or more homogeneous sets.
 Splitting: It is a process of dividing a node into two or more sub-nodes.
 Decision Node: When a sub-node splits into further sub-nodes, then it is called
decision node.
 Leaf/ Terminal Node: Nodes with no children (no further split) is called Leaf or
Terminal node.
 Pruning: When we reduce the size of decision
trees by removing nodes (opposite of Splitting),
the process is called pruning.
 Branch / Sub-Tree: A sub section of decision
tree is called branch or sub-tree.
 Parent and Child Node: A node, which is divided
into sub-nodes is called parent node of sub-nodes
where as sub-nodes are the child of parent node.
Algorithm
 Algorithms used in decision trees:
 ID3
 Gini Index
 Chi-Square
 Reduction in Variance
 The core algorithm for building decision trees is called ID3. Developed by J.
R. Quinlan and it uses Entropy and Information Gain to construct a decision
tree.
 The ID3 algorithm begins with the original set S as the root node. On each
iteration of the algorithm, it iterates through every unused attribute of the
set S and calculates the entropy H(S)or information gain IG(S) of that
attribute. It then selects the attribute which has the smallest entropy (or
largest information gain) value. The set S is then split or partitioned by the
selected attribute to produce subsets of the data
Entropy
 Entropy is a measure of the randomness in the information being
processed. The higher the entropy, the harder it is to draw any
conclusions from that information. Decision tree algorithm uses
entropy to calculate the homogeneity of a sample. If the sample
is completely homogeneous the entropy is zero and if the sample
is an equally divided it has entropy of one.
Example:
 To build a decision tree, we need to calculate two types of
entropy using frequency tables as follows:
 a) Entropy using the frequency table of one attribute:
 b) Entropy using the frequency table of two attributes:
Information gain

The information gain is based on the decrease in entropy after a dataset is

split on an attribute. Constructing a decision tree is all about finding attribute
that returns the highest information gain (i.e., the most homogeneous
branches).

Step 1: Calculate entropy of the target.

 Step 2: The dataset is then split on the different attributes. The
entropy for each branch is calculated. Then it is added
proportionally, to get total entropy for the split. The resulting
entropy is subtracted from the entropy before the split. The
result is the Information Gain, or decrease in entropy.
 Step 3: Choose attribute with the largest information gain as the
decision node, divide the dataset by its branches and repeat the
same process on every branch.
 Step 4a: A branch with entropy of 0 is a leaf node

 Step 4b: A branch with entropy more than 0 needs further

splitting
 Step 5: The ID3 algorithm is run recursively on the non-leaf
branches, until all data is classified.
 Decision Tree to Decision Rules
 A decision tree can easily be transformed to a set of rules by
mapping from the root node to the leaf nodes one by one.
 Limitations to Decision Trees
 Decision trees tend to have high variance when they utilize
different training and test sets of the same data, since they tend
to overfit on training data. This leads to poor performance on
unseen data. Unfortunately, this limits the usage of decision
trees in predictive modeling.
 To overcome these problems we use ensemble methods, we can
create models that utilize underlying(weak) decision trees as a
foundation for producing powerful results and this is done in
Random Forest Algorithm