Decision Tree

Example of a Decision Tree - Tax evasion

Splitting Attributes
Tid Refund Marital Taxable
Status Income Cheat

1 Yes Single 125K No

2 No Married 100K No Refund
Yes No
3 No Single 70K No
4 Yes Married 120K No NO MarSt
5 No Divorced 95K Yes Single, Divorced Married
6 No Married 60K No
7 Yes Divorced 220K No TaxInc NO
8 No Single 85K Yes < 80K > 80K
9 No Married 75K No
NO YES
10 No Single 90K Yes
10

Training Data Model: Decision Tree

 Decision Tree Induction
• Many Algorithms:
– Hunt’s Algorithm (one of the earliest)
– CART
– ID3, C4.5
– SLIQ,SPRINT
ID-3 Algorithm for Decision Tree
MAKE A DECISION TREE THAT PREDICTS IF TENNIS WILL BE PLAYED
ON A PARTICULAR DAY
Understanding the dataset
This data set consists of 4 attributes
• Outlook
• Temperature
• Humidity
• Wind
The Target Variable is to Play Tennis

The ID3 algorithm for the Decision Tree

• If you want to draw a decision tree using ID3 Algorithm, we need to understand
which attribute gives the maximum available information.

• Out of 4 attributes, we have to calculate the information gain of each attribute.

• Find the attribute which has the maximum information gain.

• The one which has the maximum information gain will be treated as the root
node.
Steps in ID3
1. Compute the entropy for the entire Data-set [ Entropy (S)]
2. For Every attribute / Feature
1. Calculate the Entropy [ Entroty (A)]
2. Calculate Average Information current attribute
3. Calculate Gain for the current attribute

3. Pick the highest Gain Attribute.

4. Repeat until we get the desired tree.

• Calculating the Information Gain:
• Total Instances = 14
• Number of Positive Instances = 09
• Number of Negative Instances = 05
Calculating the Entropy (S) for the overall data set

• Total Instances = 14
• Number of Positive Instances (p)= 09
• Number of Negative Instances (n) = 05
Calculate the Entropy of each attribute followed by
the information gain
• Let us start with Attribute ( Outlook)

• Outlook has three values ‘sunny’, ‘rainy’ , ‘overcast’

Outlook Play Tennis Outlook Play Tennis Outlook Play Tennis
Sunny No Rainy Yes Overcast Yes
Sunny No Rainy Yes Overcast Yes
Sunny No Rainy No Overcast Yes
Sunny Yes Rainy Yes Overcast Yes
Sunny Yes Rainy No

Total Instances = 5 Total Instances = 5 Total Instances = 4

Positive (p) = 2 Positive (p) = 03 Positive (p) = 04
Negative (n) = 3 Negative (n) = 02 Negative (n) = 00
 2 2 3 3
S 𝑶𝒖𝒕𝒍𝒐𝒐𝒌𝑺𝒖𝒏𝒏𝒚 = − 𝐿𝑜𝑔2 − 𝐿𝑜𝑔2
5 5 5 5
2 2 3 3
S 𝑶𝒖𝒕𝒍𝒐𝒐𝒌𝒓𝒂𝒊𝒏𝒚 = − 𝐿𝑜𝑔2 − 𝐿𝑜𝑔2
5 5 5 5
4 4 4 4
S 𝑶𝒖𝒕𝒍𝒐𝒐𝒌𝑹𝒂𝒊𝒏𝒚 = − 𝐿𝑜𝑔2 − 𝐿𝑜𝑔2
4 4 4 4

Outlook Entropy
Sunny 0.971
Rainy 0.971
Overcast 0
Calculate Average Information Entropy
𝑃𝑠𝑢𝑛𝑛𝑦 + 𝑛𝑠𝑢𝑛𝑛𝑦
∗ 𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑜𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑠𝑢𝑛𝑛𝑦 +
𝑃𝑡𝑜𝑡𝑎𝑙 + 𝑛 𝑡𝑜𝑡𝑎𝑙

𝑃𝑟𝑎𝑖𝑛𝑦 + 𝑛𝑟𝑎𝑖𝑛𝑦
𝑰(𝑶𝒖𝒕𝒍𝒐𝒐𝒌) = ∗ 𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑜𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑟𝑎𝑖𝑛𝑦 +
𝑃𝑡𝑜𝑡𝑎𝑙 + 𝑛 𝑡𝑜𝑡𝑎𝑙

𝑃𝑜𝑣𝑒𝑟𝑐𝑎𝑠𝑡 + 𝑛𝑜𝑣𝑒𝑟𝑐𝑎𝑠𝑡
∗ 𝐸𝑛𝑡𝑟𝑜𝑝𝑦(𝑜𝑢𝑡𝑙𝑜𝑜𝑘 = 𝑜𝑣𝑒𝑟𝑐𝑎𝑠𝑡
𝑃𝑡𝑜𝑡𝑎𝑙 + 𝑛 𝑡𝑜𝑡𝑎𝑙

3+2 3+2 4+0

𝑰(𝑶𝒖𝒕𝒍𝒐𝒐𝒌) = ∗ 0.971 + ∗ 0.971 + ∗ 0
9+5 9+5 9+5
𝑰(𝑶𝒖𝒕𝒍𝒐𝒐𝒌) = 𝟎. 𝟔𝟗𝟑

Calculate Gain : Outlook

Gain = Entropy(S) – I (Attribute)
Gain = 0.940 -0.693
Gain Outlook = 0.247
Calculate the Entropy of Temperature followed by
the information gain
• Attribute ( Temperature)

• Outlook has three values ‘hot ’, ‘mild’ , ‘cool’

Temp Play Tennis
Temp Play Tennis Temp Play Tennis
Mild Yes
Hot No Cool Yes
Mild No
Hot No Cool No
Mild Yes
Hot Yes
Mild Yes Cool Yes
Hot Yes
Mild Yes Cool Yes
Mild No
Total Instances = 4 Total Instances = 6 Total Instances = 4
Positive (p) = 2 Positive (p) = 04 Positive (p) = 03
Negative (n) = 2 Negative (n) = 02 Negative (n) = 01
Calculate Entropy (S) of Attribute Temperature

2 2 2 2
S 𝑻𝒆𝒎𝒑𝒆𝒓𝒂𝒕𝒖𝒓𝒆𝑯𝒐𝒕 = − 𝐿𝑜𝑔2 − 𝐿𝑜𝑔2
2 2 2 2
4 4 2 2
S 𝑻𝒆𝒎𝒑𝒆𝒓𝒂𝒕𝒖𝒓𝒆𝒎𝒊𝒍𝒅 = − 𝐿𝑜𝑔2 − 𝐿𝑜𝑔2
6 6 6 6
3 3 1 1
S 𝑻𝒆𝒎𝒑𝒆𝒓𝒂𝒕𝒖𝒓𝒆𝑪𝒐𝒍𝒅 = − 𝐿𝑜𝑔2 − 𝐿𝑜𝑔2
4 4 4 4

𝑻𝒆𝒎𝒑𝒆𝒓𝒂𝒕𝒖𝒓𝒆 Entropy
Hot 1
Mild 0.918
Cool 0.811
Calculate Average Information Entropy

2+2 4+2 3+1

𝑰(𝑻𝒆𝒎𝒑) = ∗ 1 + ∗ 0.918 + ∗ 0.811
9+5 9+5 9+5

𝑰(𝑻𝒆𝒎𝒑) = 0.911

Calculate Gain : Temp

Gain = Entropy(S) – I (Attribute)
Gain = 0.940 -0.911
Gain Outlook = 0.029
Calculate the Entropy of Humidity followed by the
information gain
• Attribute ( Humidity)

• Outlook has two values ‘Normal ’, ‘High’’

Humidity Play Tennis Humidity Play Tennis
Normal Yes High No
Normal No High No
Normal Yes High Yes
Normal Yes High Yes
Normal Yes High No
Normal Yes High Yes
Normal Yes High No

Total Instances = 7
Total Instances = 7 Positive (p) = 06
Positive (p) = 3 Negative (n) = 01
Negative (n) = 4
Calculate Entropy (S) of Attribute Humidity

2 2 2 2
S 𝑯𝒖𝒎𝒊𝒅𝒊𝒕𝒚𝑵𝒐𝒓𝒎𝒂𝒍 = − 𝐿𝑜𝑔2 − 𝐿𝑜𝑔2
2 2 2 2
4 4 2 2
S 𝑯𝒖𝒎𝒊𝒅𝒊𝒕𝒚 𝒉𝒊𝒈𝒉 = − 𝐿𝑜𝑔2 − 𝐿𝑜𝑔2
6 6 6 6

𝑻𝒆𝒎𝒑𝒆𝒓𝒂𝒕𝒖𝒓𝒆 Entropy
Normal 0.985
High 0.591
Calculate Average Information Entropy

3+4 6+1
𝑰(𝑯𝒖𝒎𝒊𝒅𝒊𝒕𝒚) = ∗ 0.985 + ∗ 0.591
9+5 9+5

𝑰(𝑯𝒖𝒎𝒊𝒅𝒊𝒕𝒚) = 0.788

Calculate Gain : Humidity

Gain = Entropy(S) – I (Attribute)
Gain = 0.940 -0.788
Gain Outlook = 0.152
Calculate the Entropy of Wind followed by the
information gain
• Attribute ( Wind)

• Wind has two values ‘Weak ’, ‘Strong’’

Humidity Play Tennis Humidity Play Tennis
Weak No Strong No
Weak Yes Strong Yes
Weak Yes Strong Yes
Weak Yes Strong Yes
Weak No Strong Yes
Weak Yes Strong No
Weak Yes
Weak Yes
Total Instances = 6
Total Instances = 8 Positive (p) = 04
Positive (p) = 6 Negative (n) = 02
Negative (n) = 2
 INFORMATION GAIN

Attributes Gain
Outlook 0.247
Temp 0.029
Humidity 0.152
Windy 0.048
Naïve Bayes Classifier
Predict the outcome for the following Scenarios using Naïve Bayes
Classifier

Outlook= Sunny Outlook= Rain

Temperature = Cool Temperature = Cool
Humidity= High Humidity= High
Wind= Strong Wind= Strong
Step1
Calculating the overall probability of the outcomes
Probability of the outcome Play Tennis (Yes) = 9/14 = 0.64

Probability of the outcome Play Tennis (No) = 5/14 = 0.36

 Outcome(Play Tennis)

Outlook Yes No

Sunny 2/9 3/5

overcast 4/9 0

Rain 3/9 2/5

 Outcome(Play
Tennis)

Temp Yes No

Hot 2/9 3/5

Mild 4/9 0

Cool 3/9 1/5

 Outcome(Play Tennis) Outcome(Play Tennis)
Outlook Yes No Humidity Yes No
Sunny 2/9 3/5 High 3/9 4/5
overcast 4/9 0 Normal 6/9 1/5
Rain 3/9 2/5

Outcome(Play Tennis) Outcome(Play Tennis)

Temp Yes No Windy Yes No
Hot 2/9 3/5 Strong 3/9 3/5
Mild 4/9 0 Normal 6/9 2/5
Cool 3/9 1/5
Predicting the outcome for the scenario-1 using Naïve Bayes
Classifier

Outlook= Sunny
Temperature = Cool
Humidity= High
Wind= Strong
Predicting the outcome for the scenario-2 using Naïve Bayes
Classifier
Outlook= Rain
Temperature = Cool
Humidity= High
Wind= Strong