Scribd
Upload
166 views48 pages

Multiclass Classification: Kashif Murtaza: PUCIT 1

The document discusses multiclass classification and support vector machines (SVM). It covers SVM concepts like hard margin vs soft margin, maximum margin discriminant, confidence, and the SVM optimization problem. It also discusses SVM mathematics, including the generalized linear model, binary SVM formulation, and SVM dual formulation using kernels. Key algorithms like minimizing the margin width and Lagrangian dual are presented.

Uploaded by

ماہم مبشر
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
166 views48 pages

Multiclass Classification: Kashif Murtaza: PUCIT 1

The document discusses multiclass classification and support vector machines (SVM). It covers SVM concepts like hard margin vs soft margin, maximum margin discriminant, confidence, and the SVM optimization problem. It also discusses SVM mathematics, including the generalized linear model, binary SVM formulation, and SVM dual formulation using kernels. Key algorithms like minimizing the margin width and Lagrangian dual are presented.

Uploaded by

ماہم مبشر
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 48

Multiclass Classification

Kashif Murtaza: PUCIT 1


Multiclass Classification
Multiclass Classification
Multitarget Linear Regression

Kashif Murtaza: PUCIT 4


SVM
Linear Discriminants

Kashif Murtaza: PUCIT 5


SVM
Polynomial Fitting: Linearity?
 
𝑦 𝑖 = 𝑤 0 + 𝑤 1 𝑥 𝑖 1+ 𝑤 2 𝑥𝑖 2 +𝑤 3 𝑥 2𝑖 1+ 𝑤 4 𝑥 2𝑖 2+ 𝑤 5 𝑥 𝑖1 𝑥𝑖 2

Kashif Murtaza: PUCIT 6


SVM
Maximum Margin Discriminant

Kashif Murtaza: PUCIT 7


SVM
Hard Margin vs Soft Margin

Kashif Murtaza: PUCIT 8


SVM
Confidence

Kashif Murtaza: PUCIT 9


SVM
SVM Optimization Problem
Find out a separating surface (Decision
Surface/boundary) such that even the closest
point to the surface is as far away as possible.

Kashif Murtaza: PUCIT 10


SVM
SVM vs Logistic Regression

Sparsity

Kashif Murtaza: PUCIT 11


SVM Mathematics
Generalized Linear Model

𝑇
𝑦 𝑖 = 𝑓 (𝑤 𝑥 𝑖 + 𝑤 0 )
 

Kashif Murtaza: PUCIT 12


SVM Mathematics
 

For Binary SVM }

𝑇
𝑦 𝑖 = 𝑠𝑖𝑔𝑛 (𝑤 𝑥𝑖 +𝑤 0 )
 

Kashif Murtaza: PUCIT 13


SVM Mathematics
 

For Binary SVM }

𝑦 𝑖 = 𝑠𝑖𝑔𝑛 ¿
 

Kashif Murtaza: PUCIT 14


SVM Mathematics
Confidence

¿ ¿
 

𝑦𝑖 ¿ ¿ ¿
 

Kashif Murtaza: PUCIT 15


SVM Mathematics

Kashif Murtaza: PUCIT 16


SVM Mathematics

Kashif Murtaza: PUCIT 17


SVM Mathematics

Kashif Murtaza: PUCIT 18


SVM Mathematics

Kashif Murtaza: PUCIT 19


SVM Mathematics

Kashif Murtaza: PUCIT 20


SVM Mathematics
Confidence

𝑦𝑖 ¿ ¿ ¿
 

Algebraic Distance vs Geometric Distance


Kashif Murtaza: PUCIT 21
SVM Mathematics
SVM Optimization Problem

  𝑎𝑟𝑔𝑚𝑎𝑥
1 𝑚𝑖𝑛 𝑇

𝑤 ,𝑤0
{
‖𝑤 ‖ 𝑖 [
𝑦𝑖 ( 𝑤 ∅ ( 𝑥 𝑖 )+ 𝑤 0 ) } ]

Kashif Murtaza: PUCIT 22


SVM Mathematics
SVM Optimization Problem

  𝑎𝑟𝑔𝑚𝑎𝑥
1 𝑚𝑖𝑛 𝑇

𝑤 ,𝑤0
{
‖𝑤 ‖ 𝑖 [
𝑦𝑖 ( 𝑤 ∅ ( 𝑥 𝑖 )+ 𝑤 0 ) } ]

Kashif Murtaza: PUCIT 23


SVM Mathematics
SVM Optimization Problem (Hard Margin)
  𝑚𝑖𝑛 𝑇
𝑖 [ 𝑦 (𝑤
𝑖 ∅ ( 𝑥 𝑖 ) + 𝑤 0 ) =1 ]
We can always scale algebraic distance
 
1 2
Minimize ‖𝑤‖
2
𝑇
𝑦 𝑖 ( 𝑤 ∅ ( 𝑥 𝑖 ) +𝑤 0 ) ≥ 1 , ∀ 𝑖
 
Subject to
Kashif Murtaza: PUCIT 24
SVM Mathematics
SVM Optimization Problem (Soft Margin)
 
𝑁
1 2
Minimize ‖𝑤‖ + 𝐶 ∑ 𝑒 𝑖
2 𝑖=1
𝑇
𝑦 𝑖 ( 𝑤 ∅ ( 𝑥 𝑖 ) +𝑤 0 ) ≥ 1 − 𝑒𝑖 , ∀ 𝑖
 
Subject to
 
𝑒𝑖 ≥ 0 , ∀ 𝑖
Kashif Murtaza: PUCIT 25
SVM Mathematics
Langragian Dual and the Kernels
  𝑁 𝑁 𝑁
1
(
max ⁡ ∑ 𝑎𝑖 − ∑ ∑ 𝑎𝑖 𝑎 𝑗 𝑦 𝑖 𝑦 𝑗 𝑘(𝑥 𝑖 , 𝑥 𝑗 )
𝑖=1 2 𝑖=1 𝑗=1 )
0 ≤ 𝑎𝑖 ≤ 𝐶 𝑁
   

  𝑁

∑ 𝑎𝑖 𝑦 𝑖=0 𝑤=∑ 𝑎𝑖 𝑦 𝑖 𝑥 𝑖
𝑖=1
𝑖=1
Kashif Murtaza: PUCIT 26
  𝑎𝑟𝑔𝑚𝑎𝑥
1 𝑚𝑖𝑛 𝑇

𝑤 ,𝑤
{
‖𝑤 ‖ 𝑖
𝑦[
𝑖 ( 𝑤 ∅ ( 𝑥 𝑖 )+ 𝑤 0 )]}
0

Courtesy: Andrew Ng (Stanford)


Courtesy: Andrew Ng (Stanford)
Courtesy: Andrew Ng (Stanford)
Courtesy: Andrew Ng (Stanford)
Courtesy: Andrew Ng (Stanford)
 
1 2
Minimize ‖𝑤‖
2
𝑇
𝑦 𝑖 ( 𝑤 ∅ ( 𝑥 𝑖 ) +𝑤 0 ) ≥ 1 , ∀ 𝑖
 
Subject to
Kashif Murtaza: PUCIT 33
Courtesy: Andrew Ng (Stanford)
Courtesy: Andrew Ng (Stanford)
Home work
Soft margin
  𝑁 𝑁 𝑁
1
(
max ⁡ ∑ 𝑎𝑖 − ∑ ∑ 𝑎𝑖 𝑎 𝑗 𝑦 𝑖 𝑦 𝑗 𝑘(𝑥 𝑖 , 𝑥 𝑗 )
𝑖=1 2 𝑖=1 𝑗=1 )
0 ≤ 𝑎𝑖 ≤ 𝐶 𝑁
   

  𝑁

∑ 𝑎𝑖 𝑦 𝑖=0 𝑤=∑ 𝑎𝑖 𝑦 𝑖 𝑥 𝑖
𝑖=1
𝑖=1
Kashif Murtaza: PUCIT 37

You might also like