LDA … Two Classes - Example

• Compute the Linear Discriminant projection for the following two-

dimensional dataset.
– Samples for class ω1 : X 1 = (x1,x2)={(4,2),(2,4),(2,3),(3,6),(4,4)}

– Sample for class ω 2 :X 2 = (x1,x2)={(9,10),(6,8),(9,5),(8,7),(10,8)}

10

6
x2

1
00
1 2 3 4 5 6 7 8 9 10
x1
 LDA … Two Classes - Example
• The classes mean are :

1 1  4   2   2   3   4    3 
 =
1 
N1 x 1
x =   +   +   +   +   =  
5  2   4   3   6   4    3.8 

 = 1 1  9   6   9   8  10    8.4 
2 
N 2 x 2
x =   +   +   +   +   =  
5 10   8   5   7   8    7.6 
 LDA … Two Classes - Example
• Covariance matrix of the first class:

 4   3   2   3 
2 2

S 1 S=1𝒏=− 𝟏𝒏
𝟏
𝟏
−𝟏
(x −(1x)(−x−1)(1x)−=𝒏1𝟏)− 𝟏=  2  −   +  4  −  
T 𝟏
𝟏
x 1 x 1   3.8    3.8 
 2   3   3   3   4   3 
2 2 2

+   −   +   −   +   −  
 3   3.8   6   3.8   4   3.8 
 1 − 0.25
= 
 − 0.25 2.2 
 LDA … Two Classes - Example
• Covariance matrix of the first class:
𝟐 𝟐
𝟏 −𝟏
 4   3   2   3 
2 2

S 1 S=1𝒏=− 𝟏𝒏
𝟏
𝟏
−𝟏
(x −(1x)(−x−1)(1x)−=𝒏1𝟏)− 𝟏=  2 −𝟏.
T 𝟏 − 𝟖  +  𝟎.−𝟐 
𝟏
x 1 x 1   3.8   4  3.8 
𝟐 𝟐 𝟐

 2   3   3   3   4   3 
−𝟏 𝟎 𝟏
2 2 2

+  −𝟎.
− 𝟖  +  𝟐.−𝟐  +   𝟎.− 𝟐 
 3   3.8   6   3.8   4   3.8 
 1 − 0.25
= 
 − 0.25 2.2 
 LDA … Two Classes - Example
• Covariance matrix of the first class:

    3−𝟏 𝟎. 𝟐
2 2
 4𝟏
  3   2  −𝟏

S 1 S=1𝒏=− 𝟏𝒏
𝟏
𝟏
−𝟏
(x 
−  (1
1)(1x)−=1 ) = −𝟏. 𝟖−   + + 𝟎.−𝟐 
x)(−x− T 𝟏 𝟏 −𝟏. 𝟖
𝟏
x 1 x 1
𝒏𝟏 − 𝟏
 2  3.8   4  3.8 
𝟎 𝟏
 2  −𝟏 −𝟎.
3 𝟖  3  𝟎 3𝟐.𝟐  4  𝟏 3𝟎. 𝟐
−𝟏 2 2 2
+ +−𝟎.𝟖  −   + 𝟐. 𝟐 −   + 𝟎. 𝟐 −  
 3   3.8   6   3.8   4   3.8 
 1 − 0.25
= 
 − 0.25 2.2 
 LDA … Two Classes - Example
• Covariance matrix of the first class:

−𝟎. 𝟐2
 4 𝟏  −𝟏.
3 𝟖  2   3 
𝟏
2

S1 =
𝟏
 (x −  )(x−  ) =
1 1
T 𝟏
 2−𝟏.
 𝟖 3.8  +  −𝟎.
𝟑. 𝟒𝟐 − 𝟐 𝟎.𝟎𝟒
     4  3.8 
𝒏𝟏 − 𝟏 𝟓−𝟏
x 1

 2𝟏 𝟎. 𝟖3   3𝟎 𝟎 3   4𝟏 𝟎. 𝟐3 

2 2 2

+  𝟎.𝟖 − 𝟎. 𝟔𝟒  +  𝟎 −𝟒.𝟖𝟒  +  𝟎. 𝟐 −𝟎. 𝟎𝟒 

 3   3.8   6   3.8   4   3.8 
 = 1𝟏 𝟒 − 0.25 −𝟏 𝟏 −𝟎. 𝟐𝟓
=  𝟒 −𝟏 𝟖. 𝟖 = −𝟎. 𝟐𝟓 𝟐. 𝟐
 − 0.25 2.2 
 LDA … Two Classes - Example
• Covariance matrix of the second class:

 9  8.4  6  8.4
2 2

S2 =
𝟏
 (x −  2 )(x−  2 )T
= 𝟏
10  −   +  8  −  
   7.6     7.6 
𝒏𝟐 − 𝟏 𝒏 −𝟏
𝟐
x 2

 9  8.4  8  8.4 10  8.4

2 2 2

+   −   +   −   +   −  
 5   7.6   7   7.6   8   7.6 
 2.3 − 0.05
= 
 − 0.05 3.3 
 LDA … Two Classes - Example
• Covariance matrix of the second class:

 9𝟔  𝟎.8.4
𝟏𝟎.  −𝟐. 8.4
2 2
6  𝟒

S2 =
𝟏
 (x −  2 )(x−  2 )T
= 𝟏
𝟐.10  −
𝟒  
𝟔 𝟐.

𝟒
 +
+ 
 𝟎.
𝟒− 
 
−𝟐. 𝟒 𝟎. 𝟒

   7.6   8   7.6 
𝒏𝟐 − 𝟏 𝒏 −𝟏
𝟐
x 2

 9  8.4  8  8.4 10

𝟏. 𝟔 
8.4
2−𝟎. 𝟒 2 2
𝟎. 𝟔
+ + 𝟎. 𝟔 −−𝟐.
 𝟔 + −𝟎.
+ 𝟔  −   + 𝟎. 𝟒  − 𝟔 𝟎.𝟒
−𝟎. 𝟒 −𝟎. 𝟔 𝟏.
−𝟐. 𝟔
 5   7.6   7   7.6   8   7.6 
 2.3 − 0.05
= 
 − 0.05 3.3 
 LDA … Two Classes - Example
• Covariance matrix of the second class:

 𝟎.  8.4  6𝟓.𝟕𝟔 8.4

2 −𝟎. 𝟗𝟔 2
9 𝟑𝟔 𝟏. 𝟒𝟒
S2 =
𝟏
 (x −  2 )(x−  2 )T
= 𝟏
10  − 𝟓. 𝟕𝟔  + −𝟎.𝟗𝟔
𝟏. 𝟒𝟒 −  𝟎. 𝟏𝟔


   7.6   8   7.6 
𝒏𝟐 − 𝟏 𝟓−𝟏
x 2

 9𝟎 8.4  8𝟎.𝟏𝟔 8.4 10  8.4

2 2 2
−𝟏. 𝟓𝟔 𝟎. 𝟐𝟒 𝟐. 𝟓𝟔 𝟎. 𝟔𝟒
+ −𝟏.𝟓𝟔−  𝟔. 𝟕𝟔 +  𝟎.𝟐𝟒− 𝟎. 𝟑𝟔 +  𝟎. 𝟔𝟒  − 𝟎. 𝟏𝟔 
 5   7.6   7   7.6   8   7.6 
= 2.3
𝟏 𝟗. 𝟐− 0.05
−𝟎. 𝟐 𝟐. 𝟑 −𝟎. 𝟎𝟓
=  𝟒 −𝟎. 𝟐 𝟏𝟑. 𝟐 = −𝟎. 𝟎𝟓 𝟑. 𝟑
 − 0.05 3.3 
 LDA … Two Classes - Example
• Within-class scatter matrix:

 1 − 0.25   2.3 − 0.05 

S w = S1 + S 2 =  + 
 − 0.25 2.2   − 0.05 3.3 
 3.3 − 0.3 
= 
 − 0.3 5.5 
 LDA … Two Classes - Example
• Between-class scatter matrix:

S B = (1 − 2 )(1 − 2 )
T

=  3  −  8.4  3  −  8.4 

T

3.8 7.6  3.8 7.6 

      
 − 5.4 
= (− 5.4 − 3.8 )
 − 3.8 
 29.16 20.52 
=
 20.52 14.44 
 LDA … Two Classes - Example
• The LDA projection is then obtained as the solution of the generalized eigen
value problem −1
SW S B w =  w
 SW−1 S B −  I = 0
−1
 3.3 − 0.3  29.16 20.52   1 0
    −    =0
 − 0.3 5.5   20.52 14.44   0 1 
 0.3045 0.0166  29.16 20.52   1 0
   −   =0
 0.0166 0.1827  20.52 14.44   0 1
 9.2213 −  6.489 
 
 4.2339 2.9794 −  
= (9.2213 −  )(2.9794 −  )− 6.489 4.2339 = 0
 2 −12.2007 = 0   ( −12.2007) = 0
 1 = 0, 2 = 12.2007
 LDA … Two Classes - Example

• Hence
 9.2213 6.489   w1 
 w1 = 0  
 4.2339 2.9794  1  w2 

and
 9.2213 6.489   w1 
  w2 = 12.2007  
 4.2339 2.9794  2  w2 
Thus;
 − 0.5755   0.9088  = w *
w1 =   and w2 = 
 0.8178   0.4173 
• The optimal projection is the one that given maximum λ= J(w)
 LDA … Two Classes - Example
Or directly; 2
−1
2  3.3 − 0.3   3   8.4 
w = S (1 −  2 ) = 
* −1
   −  

 − 0.3 5.5   3.8   7.6 
W

2

 0.3045 0.0166 
 − 5.4 
=
 0.0166 0.1827  − 3.8 

= 
9.219 6.488 

 4.233 2.9788
After Normalization For example:

= 
0.9088  by dividing the result 9.219
at each column by: 9.2192 + 4.2332
 0.4173  = 0.9088
𝑤12 + 𝑤22