Scribd
Upload
49 views9 pages

Decision Tree Tutorial

The document discusses decision trees and the ID3 algorithm. It provides an example of using ID3 to build a decision tree to predict whether to play tennis based on attributes like outlook, temperature, and humidity. It calculates the information gain of each attribute and selects the attribute with the highest gain, outlook, as the root node. It then recursively calculates the information gain of the remaining attributes to build out the decision tree branches.

Uploaded by

Muhammad Usman
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
49 views9 pages

Decision Tree Tutorial

The document discusses decision trees and the ID3 algorithm. It provides an example of using ID3 to build a decision tree to predict whether to play tennis based on attributes like outlook, temperature, and humidity. It calculates the information gain of each attribute and selects the attribute with the highest gain, outlook, as the root node. It then recursively calculates the information gain of the remaining attributes to build out the decision tree branches.

Uploaded by

Muhammad Usman
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 9

Part 3: Decision Trees

Decision tree representation


ID3 learning algorithm

Entropy, information gain


Decision Tree for PlayTennis

 Attributes and their values:


 Outlook: Sunny, Overcast, Rain

 Humidity: High, Normal

 Wind: Strong, Weak

 Temperature: Hot, Mild, Cool

 Target concept - Play Tennis: Yes, No

ICS320 2
Training Examples
Day Outlook Temp. Humidity Wind Play Tennis
D1 Sunny Hot High Weak No
D2 Sunny Hot High Strong No
D3 Overcast Hot High Weak Yes
D4 Rain Mild High Weak Yes
D5 Rain Cool Normal Weak Yes
D6 Rain Cool Normal Strong No
D7 Overcast Cool Normal Weak Yes
D8 Sunny Mild High Weak No
D9 Sunny Cool Normal Weak Yes
D10 Rain Mild Normal Strong No
D11 Sunny Mild Normal Strong Yes
D12 Overcast Mild High Strong Yes
D13 Overcast Hot Normal Weak Yes
D14 Rain Mild High Strong No
ICS320 3
Selecting the Next Attribute
The information gain values for the 4 attributes
are:
• Gain(S,Outlook) =0.247
• Gain(S,Humidity) =0.151
• Gain(S,Wind) =0.048
• Gain(S,Temperature) =0.029

where S denotes the collection of training examples

Note: 0Log20 =0

ICS320 4
Selecting the Next Attribute
S=[9+,5-]
E=0.940
Outlook

Over
Sunny Rain
cast
[2+, 3-] [4+, 0] [3+, 2-]
E=0.971 E=0.0 E=0.971
Gain(S,Outlook)
=0.940-(5/14)*0.971
-(4/14)*0.0 – (5/14)*0.0971
=0.247
ICS320 5
Selecting the Next Attribute
S=[9+,5-] S=[9+,5-]
E=0.940 E=0.940
Humidity Wind

High Normal Weak Strong

[3+, 4-] [6+, 1-] [6+, 2-] [3+, 3-]


E=0.985 E=0.592 E=0.811 E=1.0
Gain(S,Humidity) Gain(S,Wind)
=0.940-(7/14)*0.985 =0.940-(8/14)*0.811
– (7/14)*0.592 – (6/14)*1.0
=0.151 =0.048
Humidity provides greater info. gainICS320
than Wind, w.r.t target classification. 6
ID3 Algorithm Note: 0Log20 =0

[D1,D2,…,D14] Outlook
[9+,5-]

Sunny Overcast Rain

Ssunny=[D1,D2,D8,D9,D11] [D3,D7,D12,D13] [D4,D5,D6,D10,D14]


[2+,3-] [4+,0-] [3+,2-]
Test for this node
? Yes ?
Gain(Ssunny , Humidity)=0.970-(3/5)0.0 – 2/5(0.0) = 0.970
Gain(Ssunny , Temp.)=0.970-(2/5)0.0 –2/5(1.0)-(1/5)0.0 = 0.570
Gain(Ssunny , Wind)=0.970= -(2/5)1.0 – 3/5(0.918) = 0.019
ICS320 7
 Rain table: p=3, n=2 Entropy: 0.970
 Entropy for wind: Entropy of Temperature

 Weak : p=3, n=0 E:0 Hot: p-0; n=0 E:0


 Strong: p=0, n=2 E:0 Mild: p=2 n=1
E:0.918
 Cold: p=1 n=1
E:1
 E( wind)=0 E(Temp) =
0+3/5(0.918)+2/5(1)

ID3 Algorithm
Outlook

Sunny Overcast Rain

Humidity Yes Wind


[D3,D7,D12,D13]

High Normal Strong Weak

No Yes No Yes

[D1,D2] [D8,D9,D11] [D6,D14] [D4,D5,D10]


ICS320 9

You might also like