I

DSA3102 Cheatsheet.
Univanate Problem

General gaussianrandemretA
Coordinate Descent KKT point Theorem 3 2- & Bisection & Golden
.
.

Notations Ratio

Is
Method
.

:
2
Definition A point * satisfies KKT1st order conditions if Con Rh
- f(x) =
0 <= &xif(x) = 0
Uz E(z) E(Ax
: x
Suppose that f(x) is open set D in Complexity
. Use log -
C = 11-d)2
Am +b.
.

If * = = +b) AE(x) +b
is optimal =
& reg point
= *

#ite -
/ is

&0
x

= is Conve only HiPs

.

5)
mi FEATAxbbxAbTSUzAmmATEXTbEA
+
E((z Mz((z Mz]T] =
ElzTz 7
-
uzUT E(Ax +b) (Ax+ bi] -
UzUzT (a0-bo) (0
&
=
.
-

Jf(x * 1
[Thy( **
-

) =
47,8
·
+
Exitgi(x* ) + 0 , .

③ I isa Lanblo-
-
-

knpr As Algo
doncave
-

y KKT condition & x)y3 + 2xx +

AmmiAT-Ambi-pmiAT-pbl necessary Xi
Klogtlog
.

A & AT = 0 618ait 38 i s
:
.

0 .

& Cauchy-Schwarz inequality xi +

.

off
<X ]y
X , (2
11XIIIIylly
<1 sub

1Gauss
> = -

< 11 Unconst =
2x , Xa + X2
11y11
=
stat =
-

Method O(n2) Pt
.

Jacobi :
Ui
. -

0 382 bi
.
=
ai + 0 618
Linear
.

General Formulation
- .

points Half space x*

->
const
Jacobi
: KKT "f(x) convergence to
convergency to
, =

compared
.

(Constrained)
flu
Feasible set memory intensive Thm If f
I s
min
f(x) 12-1
f(x)
.
-

(or max
efficient· log
+ 530Hf(x)
(6x2) decreases
* error
memory c linearly
=
(1 +
3X13-Jx-E
-
=

figi hy Suppose
&
ES is If local
suppose are
regular x* is

) Cmpexity dennoble
x

sty=, RP(g(x) h(x)0] SGD-

. .

Well suited for I escale

,

x ce S =
(x = = 0
-

X3y
mm *iSK condition
, If
can handle non-convex
=
(175 a
O(n)-
I

Local & Global Optimum ai + 1 = Xi

memory efficient·
-
.
-

Min &
f(x) +Xt
#Initial
*
s 3 S R + f(x) > -(x interval length)
.

=
& It 210
Converg
*
A point * Min s
,
is
encenotguaranteedecal
local
-

KkT
-

x -

is x
Convex Set minima. & Hul ** ) Hf(x
*
) + Exitgi( ** 7+ [MjHn](**)
myAnts 19/00 bot
-

S =
Hal x* ) is P D all *
) + 2
Bz(x
-

x =S1
.

optimal
#
-
Ifs is stat ↑

Pt , If sc is convex = > X is - sufficient

KKT condition
618]"
.
-

Global (8

I
- local min
min To
- -
.
=

Paints
.

:fi.Suppose minimiser
Definition
.
Active point
-

constraints 8 addle .
-

DAIRh
-

A set is If for Ex , off S ⑮

convex
y
ED , line segment on
I
KKT & Dual f(x) = xi + x 2 hi(x) = 1 -

y(x -2y (x2 12 20 ha(x] Xi

Get*Sh
, : -
at X* neither
- +

active x*
.

local
is
nice
.

min max
If
-
.

hand
and
joining X
y is x ,
y tD =>
(x + 11 x)y
-
= D . Vxt(0 ,
1]
1kT order
necessary conditions -

*
at feasible i

(i)independent
j(x point Constrained rly
essible
KKT
+
& x is
NLP L
Thi e
) 2jE() pz hj(x
*
- = :
)
.
= X 2]
-))
-

ha(x)
=
0
(j) ha(x)
- -
* T =
) be : -
& Let T
tangent space at * X

[0gi(x 2020hy(x ) )]
x
2(xa
·
*
ind
hj(x] =8
-

IHalf spaccein
) je ](x early min f(x >
*
i
gi(x]
: a
(7 (00
: = 1. 2
.
=
... =

R(ax
)
(yFR : Ogi*
*

*
(x
↑(

=Y*
=

b] pt/Regulanty holds at
*
H + = is
=>> is
regular x* ** b) KKT points
ber
-

y is global

Famiuse hista)
=
, , = Loca m min
feasible ?
↑
-
If I const X
cheek -g, Thm
,
2

ban
12 :
/
If
I
-

active check
,
jg ,
soga not
multiple of each other -f 21 ** is If *
local min Convex NLP
Regular KKT
figh ?
*
h - .

x is , is If

nat
- .

g
.
,

Checklog gm) constraa ent ?

-

D nu==
have an a
Bla =
Extrale as

Endedenecessaryon
, .
↑
-
> def(A) #0 < > non
singular
:
s
14
& Check LICQ
*
Newton rgSGD iii) Dual Problem
Example all order cen- - is
Determine reg points ** minf(x) x
KKT

knowledge-xuUX-UsUx
: =
X1x2X3 Sty , (x) =
saD x require

s on a
.

has sol ?
-

(3 -
x,
j3 -
x22 = 0
↓
about .
data
As
* andRegpts Example)
·
g((x)
: all feasible Pts exe i
/-3(X3)]
S
-

=
# comp lexity Q(u)
- and
&0
=
# ki order If n 10 1]
local 1
=
is min , ,

require Hessian
X
-

-
A Else
Feasible is P

x3351 Ex
D
x22 (3-x1)3 w r t iv) Solve dual
.

: =
. . -
SGD - Newton same
Sub into Ha( x* ) Show P D mathx * is local
.

T(x* 3

D
- accuracy
x2 =(13 337 3 . Dual gap & (4 * ) f (
X) =
* -

=
33
=
0
-

x x
For
-
-

=C
, ,
⑯ Projected Gradient
, ,
C
.

is
,
yEC , convex
.
any E /1 , m)
. optimum
. DescenI >
-

* 196/y : (2)
fIxs-g
=
& & Logistic · 6(a) : 1/Ite - ·

((W) = wixi) Constraind opt

=
= #(3) : = xx On

* Polyhedron f(x)

'mumofcoordinates f =maxN-N x x] FormulateSGD Agea

① min
g(x) hj(x) -
02
,
.
m aX
- f(x) + x(i) = 0
set
vector-b polyhedral flz) :

:
For mathx dim S
PlyiX
,
A ,
Find
=

mxn s m It
dual f(x)
Lagrangian
6. maxy,a ② the KKT *
Ax b. Zi a for all i Does hold at X 1

GER"1Ax*b &
=
is
-

xg(x) + uh(x)
-
convex
minf(x)
,

G(x , u) +
.

* =

Convex function
=
I
a =
max(zi - .
zr] check h(x) which =0 . Then check KKT and : ff(x Stua -
. =
.
0 -

F(x a, b) =
t(aTx b) 2.
↓
, -

Find
is corrector c tradicto
-
a
be f f(z) x f(x) -f is ⑮1 xixx ](x ) + 11)
-x)f(y)
f(x] 1 xx)
DER" f(x) = E(F
=> **
D ( convex NLP min XX)
[11(x
*
< + (x) +
- -

R )
:
Let convex + = = ,
-
-
> 1 = -
x

Solve Dual
-
.
.

a) snow all feasible pts are regular (x*, U* ] is lower bound [xi Exa2 + x2 + x(x2)
Non-convex f(x) (X 1)(X 2)(x + 1) - = *1 + +

x)f(y)
x(x )
f(xx Xf(x)
=
ED
.

+ (1 Ex
-

x)y) =
-

(1
&

+
-

hi(x) =
y
- -
,
-
scases
: 0 ha(x) < 0 Check +nix]
,

2 + 8 . max 2(x , u) -
Si fa
strictly convex .

& Linear Combination

↓
-
.

bsolveNLPhixha
=
0 Check a
an
2 & If X >0 af) X1 -1
Rh
.

Program ?
, = +x

x)y) > xf(x) 1)

x)f(y) D &
,
fk R be functions set Cenvex
f(xx + (1 + Let fi fa D convex
-
: -
> convex on
(Possible )
, ...
>1
-

- X = 1 -
. If
x

isa
faxt E Gap
cent
Duality
Examples
f(x)
&
& If X ,< 0 a fi X -1 X
fj(x)
= -

Does regulantyno
,
where dj is
every feasible
:
>, 0 also
,

, convex Pt ?

comtor otbeinside1
global optimal
,
* & *
] a.
*
-
/
12 is
strictly f is
strictly Prove parallel find Eg , (x) psgald
-

-
If 31 to is convex , convex [ (x 92
( )] ·
not =
S S u
② If 10 ,
Consider
EzF(x z)
-

f(x) =
,
+2 = + or X3 =8 ·

Global
-

-7 min for
22 population loss :
Choose lowest f(x ] * x - -
Fab
=

(-
. -
x
((g(x a) b)
Nonlinear T e
, , -

& N10 13. If an N10 2) Po

R3
bat a Lagrangian
Coordinate Descent Jacobig Gauss-Siedel
. ,
a) dual where X
↑
.

&xXxxs
=

Eg
. -
-
+ x2 /
-
X3
11 xTa
*
Ta E(1 11(x ** ]a -
21
Gauss-Siedel CD SGD Example F( z] X = 5
+g(x) = 2X//X2 2-1xz
-
- = -

X -

4 =8
- Elas]
aix*-X* 3 2/x ** [Ta 2 + 3272
E La =]
- -
.

Elar2)
.
·

Ef(x-x* >Ta &

Rigorous
:

Pop Conditions

·
U b aTx * N(0 1) loss Ez (F(z x))
.

+ 2 :
- . -

as
E 2
-
=
(Var(ao) + E(ao)]
-
.
,
-
, = ,
& =
X8 *
- *
(i] 1 (x x ]TE(aaT)(x x ) (x x ]G(x ** )
*
= 8
, z E((x (aaT(x x* )] * *
then (of(x* 0)
-

* If
=
,
= -
x =
-
-
= - - X EX is local min , is stat pt =

0j
-

2x3 +
.

z2 [ 12
=
(1 +
1
=

o)'b
, n = 0 1 -

& E(2(x *
Saz) 2(X *
(T 0 0
x = 0 =

sollax-a
-

= x =
f (2
.
x
If is Hf(X
*
P
-

, aTX & )
-

is
- .

jf(x) 25(x * , S D
) f(x * ) +
1 y
. .

Dates los ** -
311GE(22) = 1 F(x , a- b) = (x -
x
* - x
. = -
x
-
.

*
Fix 2)
=
& If f is <" *
is stat - Hf(x* ] is P D
Example X Matrix Differentiation
.

**
, ,
:
=
x x -

FIXZ =GTX
-

=>
2(x) = - 0 (f x )
** is local min Ax AT
-
.

/1) Convex Set Funct convex

b) Solve Dual Problem Find
subg A
[ & A
= &

/111(9) 1
0 1 2

Is
-

Jacobi For
. -

& f(x
.

parallel
.

da
> can
"/4 subf at 10 , 0 subgrad ?
.

J
Find Which

Det:
+ 1x1
-

(x1 not
:

f 3 =
x2) = + X2) -

3x2) 2x
If D-R
, .
·
:
is
& .

[0
42 2x2x}af cor]]]]] **
2
]
= one ,
f(x , a
.

1
.

g , (x) ) - (x)
-

= [1 12)
.
max(2xSi 2x2 4x ATX
8 2 -

* For =- Ax +

Ex
=
x2 Ax
-

for 1 , to
-

= ,

ExER/f(x)
X
xi + Sc
=
6x +5 set met =
-

sublevel
,
5 CER
- any > is
, =
,
<< be feas ible
&
-

*i
]S & convex
g(x) f Duality 1
8
g((x))
=
) 100 =Eat o
-

18 x1 + 6 Primal
,
= = 0
Problem--aT
=
= -
·

2(x* )

Rans ranandez)dz
.

Me (14 43] ecof- is hard to find

angmins ) x22 is so
-

Method B intersection If WGX2 = 83 XIE

100.24
of

=2x+ :
( convergence)
4 (1 8)
g(x)
= =
= 0 - -
.
Check ,

(pminf
=

&
Algorithm xp(xx
.

SGD
· y
.
X
= ⑪ subgrad f(xix2] 133 2) 1x2)
subgrad,
(x1 + 1x2 + (x , 1 +

:-y gen
7 -
-
-
-

C
x means not
5x2
-

i
multipliers hard
-
is 132 + (xz 2) i to find
=

g2(x2) (x2) 0
felx 7
G

ban
uz] E(z Uz](z
f , (x,y 1
-

x + 3x2 + (x1) + =
(x) m
Var(z)
= =
/
=
-
= -
..
= E(z =
uz) mince i

helal
:
-
- -

ge(x2)
&B
3 + 18x2 · IN)
⑫ Projection Ex kj(x)xj 1..
=

X E21] min f(x)

brahs S27xX
, : fa p
X
EX10
.

S : = X1 -
/ =
+ 2 x 2, =
: x22,0
x2 & & , ,

3/10 a Lagrangian Penalty

42(f)
310
= -

C xkT
(1) for
1)
n(j)
inequality const
-
1, -
= +
+
, -4
·

x I ( 3 1]x = 2 1
30
.
=
-

x
- ,
-

(5 5)
& (x)
=x2
= 7 [0 , 8]

=minX
,

tpyF)x x, zP
+
] Linear
x120 xzX10012
7
x(k + 1 x" X f(xj large h , (x)

So do
n mini(x)
-

x230 x42 : + u1 =
= -

prog
.
, , , , 01 min < 0

caus-Siedsuma
↑

g2 100 (2 3)

essa
*
/Verticess &
x ,
min
.
hix
Whenhix
(x2) + 3 xN when 8
Output
-

=
10x2 6
Unconstrained
-

&

opt
1 , 20)
-

xa 1000)
= - 0 f(x) (x) f(x)
Logic max +
36x 36) Flow uin
-
= +
1 0-6 0 . ,
-
- .
=
f(x)
,

↓
min
& D3 &

· mindpcu &Duality3123]
= Cu , minimise f(x) & Combine
↓
X +
] · =
/XI + X21 + 31X-X2) Find (C-1 , -1) Th

maxx
3,
:
& when x- 1 g /xi)
=
-a
xER X
=
, ,

(x)
of(x)
-1 ·* f
Is convex min <= > min max f(x) + un
is global ,

9ae
-

11 <(xu) /12 Find

- imp ⑪ hi(x) <

hix
=

& When 1
g1(X) 2X + [ 1] 3x +2xa-6 hzx =
-Xt a x
X :
X 4138
+ 2x1 4 Increasing >
directly
< =
= = > on +
1 , , -
1 Check
optimal
·
,
*
Jf(x* ) =
Penalty
=
(

+
.

+>
=420 ,
.
2
He is P S D of
equality const

·
. .

& 3x (10) A(x)

I
When 1 = X (x) on -
S-L
g
=
Increasing
↓
- . -

>* local

mini A
man
is N
(u)=
minimiser &

Ab
- is f(x minf(x)
>
-

is u) Y s + g, (x)
.
=
N , = 0 min
-

- f(x) + xg(x)
= >
2 *
KkT 7
op Is Hf(x ) PD
- -

92 sixt & g , (x)

x ↑
↓
=

,
↑ p,
P2/n] ↑
g
>0
S is dr lus ,

?[
convex
any
unis make
↓ f ind &
.
8
Is coersive
Gb IfPLU Pls
*
Jacob PD) is saddle g i (x)o
]

piluphs-biu
g2(x2) [ 1+
( not Glus = 1st part suggests should finite local
-
=
Both Jacobis Sie del
-
we

&
u)- bin
mins
ill-posed

minmagLagrangianfunction
Gaus Go is
through
-

only search in
xly direction m
&> p , (u)
-
bin

Minemm
-
x ga(x = - = Lagrangian Dual Problem
.

= max G(x u) inf(f(x) + XTg(x) + min(x) Generalisation

-17
xex]
,
X = [ - 1 ,
= X =
:

Proposition
+Exigi
(xn) 8) *

minmax
>
As 0 -
10 6
d) 1
u - , ,
=

is
.

X
↑ (x , u) =
min(h(X , x, u) := f(x) +
xig(x) + un(x)(XER]

f(xh
finite for all Then Q is
max min
(U , X) , 1 8. concave
maxa(X , U) Find
stationary
-
pts ·
 ·
29 .

-
& ye R3 find its (y) 41X11 * 1]
-

1Example Of Solve Lagrangian Dual Problem Convex Optimisation in constraints

- Unit ball) For , for S : Theorem
:
Properties .

supposeconstrained poblems SNeu)

veag(x)]
:z] (x)3[u + v f(x)
11 x-y112
min f
f(x) x22 is f(x) subdiff is (f + g) = u =
min hi(x) f(x) fe Sconvex
fig
-
constrained problem If of a ,

min +x E s
-

= :
Xi + ·
= -

X-X2 + 450 ·
X , op s .
convex

& f hij are all functions s - + h(x) ·

: Iff is diff at then 2(f
g)(x) Gy f(x) + V ag(x)]
..
-

Gigi
& 11x1
+ =
Lagrangian convex x = v
< /x ,
U) fIxUh
,

XX230 S
: ,
:
: = -

150
Es has optimal sol Xu E 5: Then limit of
Exu]
an of ↓
: linear
equality const :
gi(x ai bi
K KT 1st order cand : ,
any point.
-nx2 + 44
-
=
X = - of Convex Hull- single
a
Proposition
= Chiy
-f + juihiex) = 0 11y1) isoptimal so
Def of If

&mix
Given a set s conv(s] is the convex hull 5 < is

) =
: C=
= S : ,
,

Feasible IP) is ①If the smallest contains 5.

20
set convex
(lfy(s)
x + 2ux u
y
=
8 convex set that
. -
= 0 x=
↑ sublevel y
.

,
<s + - + 110 Find PlX US & its minimiser
hyperplane min
=> .
conv -
,

+stai
X =

Ev
hemma

:
Is If h(x) P(x(u) f(x)
:
If S = ·
Then cour(s)

a
cour
ng) h(x))
a
is use vis ,
u) ?
a
11 yl
=
1 x l 1 1 +2n
& & maxa(u)
-

0 = -
= =

CE)
24. , = =
-
,
=
1
(x 1)
- g
expixiuc-
-

x =
(convex)
-
=
+ 44 n
+
=
430 y I 2 P(Xu) =230 Proposition
Theorem B
-

2
214) 8 ↑
[f
=

convex c & f(x) (x) fr (x] ] fi

.

aix-bi If * Find Y2) Suppose max where all

gi(x) <05 convex s
: ,
Isly) for ,
S
..

ESiIT
=

0 EX2f
are
-
= 2
c. .

MinimiserWhen u
Xu 1 +4

Gf(:coulfis
is
.

, = *
f,
S
f(x )
·

X2 1
Ci ·
S
abpose
=
(x* ) =.
f(x* ) Then
regula as
·
=
** of f be Double (nested
aOptimal
is
global min on s sol must f(x)
② min f(x) 2
x23s +hi(x) 450
=
((x y))2
① ...
jfj)x* ]])
x) x2
is
=

I
-

+
- -

+
-2 *

-]
= x , -

*= -2] Es
Eg) Show ,

h , (x) xha(x) 1hz(x)

3 . 98 .
n -
For need to optimise P(X UK). &
(g(x)) max( g(x) g(x)]
I every UK
=
x1 =
x2 /
-

((y182X41
*

3
=
d -
9 , =
-
,
13x255
.

h(x , x) = x , + x2
-
u , x -
x =X = , 7
f(x)
min
= x22
ja
XI Inner loop
-
-

-2x , Xa +
-

hy(x) find x * angmin P(X UK) If

renseiscone
-
25g(x7]
S

g(x)
=

a(g(x))
,

4)
= x2 - > 0 =
-

uxa (x , (x2 - y
=2)
+ 4u + ,
+ 44
3)"
UEA
=
hi(x) X
-

5 + (Xi + + as
-

(no] (ay] (8)

=
KKT If
cond
( ) (4) g(x) 2(g(x)) 2 -g(x 7]
-

(
: > 8
E
+ +
= (Need *]
+ : =
X Xa + , -
=
to check
feasibility) ha(x) -Uk
,
= x1 + x2 + 350 -

. - -
If
g(x) =
0
2(g(x)) conv( Jg(x)
stoppingInten :Duality g
hz(x) hz(x) X
- y)
+g(x)]
, =
41 + 42
&
Ev 2x1 + 350 x p
3 Hf(x)
03a
If
-

[Os2] not Use . 8

-

HE allowed = :

- is P
=
*
)
X1 =
x2 xa = , -

with
! given
, .

(x )
,
-

16
.

u
= *
cases (x) If f(x) is c's
Conry u= 2 then r & Check KKT cond for x
* -
Th , (x) Hu , (x 30 (Convex>
x2 -

yz
-
43 + ux = = + ,
g is not ,
just need to
try find
2 lu) : min < (x -
u) , ...
&
& first then add oflo] to
2g.
-
,

use j
:
O
(((E
=
, 1) , u) : But
- f(x ) * =
(8) = of(x* ) + n 1 fh , (x) + - -
hix] 0 h2(x] cannot = o
simultaneously . H3SH4 also 8
Eg (x -
1) , g(x) = X -
1

2(x
xj khj(x)
U170 & Haza #true xk)
conv --
-

check hi ha his all =** is global optimiser

-

& If UE (2 , 12] <O KKT 1st cond ·

Ig(0)1 591033
=

EX means v g(0)
.
,
x
=
, = ,
, =
Xa Use 1
local
.

X
is
.

v
Xi a
.

of
=

4) ((x , xk]
. -

US sU4 min
> # both true =
conv2
E 2) Solve the optimisation it
Duality
-1
13 =
[ -1 1]
.

If is ,
-

assume

pobamxi
G (u)
.
,

(2/1 , ( U) g gap Stop when

1stcaseSeu
min
.

Proposition
= / >
-
=
+ 3u +
(n (1 + x ) s f(x)
min f(x) =
x2
-

+ hi(x) hz(x) & (xk) f(xk) + convL41 3 sah( ** ]

pur pu Suppose dg(
=
x50 ** 1
-

I conv]
-
=
Xa = 0
-
= = .
& um
-

be be
Take
-12 by 42>8 =
.
=
-

cannot cannot 4)
,
U-10 , X = 0
-

,x2
-
,
hz(x) &
a) If
Penalty
2x) + x2 350

29th him
U17 8

n
= &
-
42 V ....
=
method
equality ·
x2 0
2 Then
5 o constraints
.
=

[But un123
QM) NECO , ]
. .
.

:
①

in 211) mingii
vensy fin convey a = - hi(x) = 0 = y X = 0
&
/
x*
!
Eg) f(x) = +
f(x)
[IX0)-c Y=H (only happens when y e
-

+)
26 24
( Hf PSD
-

Convex
hj
=
linearo
:

Ex=gtx
= are
(k) gi c' h(x)
·

=
3 case
.
= (x -
1)
& when Plu) is
G'Cus : EU + 3 convex -

yz + 43 44 When
heay
+
concave. ,
x2 u
10) penalty
u E
g(x)
-
=
(2
(0 very
xiU he
1]
&Venty KKT Point hi
,
,
ul 42 X1 y fix
y)
= = , ,
* = 0 = :
u 6 &(u] 18 a)
&
=
y2/0
.

&
=
When
.

f(u) 22x
E
130 x2 0
decreasing
diy
u >10 =
G because
when a
,
0 =
(n) <
,
2x +
U2 Uz = 44 17

&
=
,
= 1+ 43 >1 =
0 xz
yz
=
Yz [8 1] I
-

x >1
&(10)
,
= .
6 = >

4ax
x ul + 243 = 0 D combinations . he is active h2 = 22x 2 x 41
Suppose
]
X" +
-

8 0 is the exact
-

WeakQualityTheorem
each
global optimiser
-

of &K UK)8 ,
- is convex (-1 , 1] x = 1
(2x
,

-natuo - UK 60
Exit
1 2x +
1yx 1
ausu Then point of
=
-

① 0 limit
global 2.
*
af(x * ,
mal prop(P) be feasibleo a
= e + b =
every is to OE >
-

Let min is optimal If

.
.
x &
Ex
(P)e(X , n) be sol (D) min
x
ample
.
to Then f(x) f(x)
-y(1s
If
=** [2x + 1 ]
-

Of
> / x, 11x <= )x =
4) active 8 h3
= ,

h(x) X71
·

X2 = 8
he + aix + by centradicts

inXStX
=
> is = o
. =
.
.

Collary If *
feas to (P) s * jf My 0 If
xx1 8 =
(2x - 7
-

:
is ) to hi(x)0 17kT +
(x *, 4 (B) s-t 2
=

Projected gradient x .
x :
= >
descent a x
,
= >

=
X. +
=
3/2
↑ (x* )
.

* X of 0f(2x y
2/x*, ) ua
15
=
y + .
0 2x
- -

u Then +
optimal
* x
kno
to (P) s
0f[1
= = 0
is a - Q (u , x)
,
=
& vk of(x* ) Not true

=
,

3)

14th]
.

=
#c
-
,
* & =
zoEne
(x *, u ) optmal IDC feasible 2 : 1 2 Case 1. : U = 0
& e(t) + + (k)
a
** as
to is f(x* Find "12
-

(1 0) = -
-
5 = .
,
a
S(h(y) = 0]
,
x
y y =
Eg
=

th min
=

I
(Th2(x) Th3/x) is
ha ha
linearly independent
active const -
minimise f(x)
24-5x 3x)
.

above
=
max
Example
.

: - I x) 8

=
/
.

x,
& case d
. -

170

E=
: .

Strong Duality
-
u* tiuk + 5x38
G(u *
-

6 3 10 , x( 434x(17
- -
= =
-
= )
-

aix + b= aly-u - %3x

-
-

-
.

=
* = 21 3] 4(1 3) 0 V f(13) = 10 : Glu * ) =
xisopt
Thm 13 -
2 ·

Suppose in (P) :X is convex ,

finj convex ,
gi affine
subin
any naia + b
& xx + 1
=

is(XkT find
xi = X2 ?
2
4 -
Sx 33x7 =
74785 =
73X
2
+
n 2u
- -
= x1
j
·
.
X -
30 =
- =
, = X2
& There I

I
Duality Gap s-th(x) aTy + b Ulla)k
"T fixed step sizes & 8 - x = 3x(

of(x) [lX-
I
is

I
X 09
g(x0
-
.

xo 0 > 23x

2-
=
-
=
0
:
.
=

=S]- max[2(x & < g(x) & Let vk

If xs
1a(x) gx
-

(f(x) 130] 4

i
min O is of
interor xex]
=
u) 28(x) 5
g(x)
- -

: = = =
, : : : -
.

I x +,
& tnVk
Then and
=
x* &(x)
45x=
new x = + =
3

couvy
G is useful to good is current
Duality x (a(x)
-

know

earning
sol gap
-

=
o
13
=
5
&
·
-

( 5,
)
Sum
=
-

X+ 1 Is (X']
,

a(x
-

stopping crotena
-

yas -
+ yi)xi , a > 1 = x =x =
El
Linear Optimisation Proposition f(x] Dist btun <x(a) =
13 (X , ) : - is 2/ll a l l sum :
Example min X2

=couv3
= - Ix
-

: - -

1
)
-billai
need
(RYxx230)
+
dist(A
Set ASB B) min
G11x-y1) By
-
=

2x22
-

m
,
X A
cix feasible 2x ye
.

min f(x) Suppose set is bounded ni(x) x2 + 440 ,

-

x = X
-
=
Y
= ,
-
= X

Let
.

5 - y bem(x 1 - yTa 1

x maxhbixo
sol to the If Thm 12 3 a) = = =

@jx-bj<0
,

hj(x) Polygon one linear .

satisfy - can use n2(x) 2

. Find f(x) 10 1) 1)
subgradient At /1 , 03
=
(1 ,
,
= -
x -z hy(x) x40
=
/ xi + x2) (x , 1 -

dist1ySly-T +b
, ,
, = - + ,

O Solve
. .

o ppb is in set V= VEhjlo for Kit of primal problem

jy 11 x11 + 10 , 03 is global min
?

⑳ simun
e
+ 632 Which
[
.. -. .

hijd = 0
z of all combet
of
cond

uni
-

a) ·, f(x)
(x,
modeltraining, b
c union
[+]]
* A
= = max

( ()
+ x2 + X1 x x2 + x , ,
]
-

S x + x) xz x
x2
-

+ : , - -
x1 - -

,
:

is in external F(x - a() > minf(x) El F(x z))

&Ex , , ,
=
R(x)
- = = max
+
/vertices
, ,

pts
-

2tauix-n = Ul = 7 => > U2 = /impossible) X

+ x2 , - 2X1 x2] -

3
2 =
F(x , zi)
mix) XP/11X11
-

&
m +
↑24
↑LagrangeDualFunctions Probleamix
So #
u1z h , (x] =
0 im g a Eg
. Linear Regression Es
1 + 4u , X2 43 reg
ajx
.
-
, -

S
0
- . .
=

hj(x) = b = 0 If A + (1 , a)
f(x)
42
(2↑ 23 f , (x)
-

ha(x) 11x-Ts(y)() (c) Max

=
8 :
x))
8
.

0
y
=

/y
-

= 0 = X =
042 =
2(i r) For S , = , , ,

(
any =
-
-
,
=
cTx (a ,+x b um/amx-bm S i
M
↓
4, ) + 42 0
f (x)
(9) f
+ = 2
-

[ff , (x)]

S
, --

If XP + x21 (x)
a 1(x) h3 Eg. min f(x) Find Xin = ,
+b S +
X , + x2
Proj 1 =
-
= Xo
(
=
If : 0 GD to
am)x- 1(i
. .

=
-

quibi AbAx-b
=
+ U1a+ . + um us 38 a r)
.
.

, = = - x2 =
0 , uz = -
-

Litt:]s
- At
a and a to
- :
xxi 10
- 13
41
=
(C + A + 2TX * uTb #
U3 0 = + ,
f(x) = max , + , 1 , 13 = f
i (x) = fz(x)
2 f(x)
Then min ((x) a+ b) of in [LATAx-2ATb] + 2xx *
in /in &17 "A Tb of ,
ofsy
((X u) 2 ATA
corh(5]
:

(i]]
c = = conv
min
=
=
-

+ 2UIX1 +
(If c +
= =
-

Ai + 0) 0

4
= =

]X : ] sub into
,
a

he
-
,
,

consider - +=A s
< 11x14 =13
=
2
Im
-

S
x
[i] feasible
ATA
= []/5 Is (by f(x)?
:

l( ac)
! x2 = 0
X2
-
=
πs(x) = =
-

Types of
= 2 =
( x(3] + (1 -
x)[9] xE(0 13] =
regu
-
, ,

uTb 1 If <+ An 0]
sparsity
452
-
=
R(X) 11x112 f(x)
② Lagrange 4x2 = =t On E
Dual Smooth

(2)
=
convex
-
.

x
,Droblem
=
30 feasible -
-

[Ixi) favours

Let c-
R(x) 11x1))
Eg 2)
-
-
= =

&
max 2(u) s .
+ 1 3 8. -
ob
KKT- min f(x) = E(x b) A(x - -
b) s - + 0 = X , x22 2 -

A + (1) -
1]
(x)
[E* 2]
,
=

&OndXPDA=]
-
a
subgradient
c
E
max-uib/ s + u26 , Ad to - Ta find min ,
=
o global
-
~
,

min uTb S + n > 0 c+ An & Dual Problem the T Definition :

0
Complete
-

, = - .

#
In
R as] be A rector is
min f(x) =
Xax2 . st X , x230 ,
x xe ↓ (x , u) = f(x) + uh(X) Let f :
-
> 1-c , convex .
v a

a subgradient of fly) f(x) + <V X)

FyRn
minu
at if -
ni(x] =
-
X,
&
hi(x) h2(x) 0 = - x, = [0 , 0]
x ,
y -
,

S
= =

M
n2(x] = X2 a Suc
- -Subdifferential f(x) f(xlyx-f(x) A 100
La
·
max
ff
&

2
-
:
fot
:

I
= im
.
:
(v x) =
y
- :
,

43(x) = x +x 2 iff :
&
X-X -

E
X, Proposition :
2 f(x) = conv LaS 1, efe
-
of s 2
ful 1)
,
(2 .

X* =
*
In min f(x)

at ifOf
then is attained 10
f(x , X Iff is can s convex
conv2[ ? ] 1
= , =
-

[, ]
,

[S]]
(optimal
[i]
*

sol is 10 8)
&
x = 1
Glu) ,

3
,
- (x2) Dual prob min of is convex .
set If f is diff at
,
=
↑ : max 2(u) = -
X ,
then

[(-X] 13
=

Xo
h 7
f(Xz)
fJfxsinglepaint a b
I
2
ina m Penalty Methods
differ
=
210 11 ,
A Exi
.

min +
xin 0
14 =
= , ,

y112 timal ,
11x sol
& Formulate
-

diffu
. .

su 7
Dual Problem D
-

solve
Inequality
Constraints

So
& .

sub-diff : of(x) =
Ev :
fly) - f(x) + [V
,
y
-
x) e

2 11
= f(x) 1x- When f(x] diff
2) 1 in K is
=
+4 -1
f(x) = =
=
0
I
X 71
·

ush)(x) + u2bz(x) xdi

X,
Uzhz(X) Dual
.

u) 31kT
hj(x)o
...
same
S
f(x)
.

+
...p
+ l
min st
N)]}}
-

= =
- -
- f(x) = 1 => I
- 2f(x) = & *f (x]] 21 = -

X, + 2x2 + ( <E f(x) caf

projectionanconvex set
= When
a x 1 ] + ( 42xx) + 43x + + 43x2 24z] x , = 1 x 0f(x) 1
=
h
feasible
-
,
non-active
- - , -
=
=

Strictly region
-

xkT)
.

.
=
(1 4) closed For
set there is = f(1) holds for all
= x" -
trbf(x * )
+ (3)x) + (2 convex When + >V ,
p2 + 13)X2
every y fly]
a 1 <v , y -17
-

1
243 -- x=
y =
y
z -
-
, ,

fj)
-

↑
-

ofi elements of subgrad

-

((y Replace
-

unique minimiser Its (g) of the problem min

&Ex-y113/xes]. 5 4x RV hj(x) + 0 113(v y 1

y
-

x(f1 = :
a(u)
-
,
us + uy
22 0)
,
n2 + 13 =
-

+
= -

0 = Just Choose
DIY VISFV
-

1
~

Strategy "O Solve the

:

yre(
:
constrained apt (Use KITS
penaltyB
-

243
-

(If of 5
.

1 1 + Hz & 2 12 + 43 2) penalises sol outside 11) f(x * + 1) f(x k)

0
&
-

No =
-

guarantee
=
-

graphical interpretation I check ansatzi n

-
=
Use KKT nB(x) -

Lagrange Dual +
·

&
Problem & If 7 :8 030 - 2f(c) :
1-1 . 1) Need book
.
keeping
, -

Barrier Function
2(u) ocaf(x]
P
max 243

Bxp(x)
E S t
=

Us 42 : l
a
.

, , 43 30
. =
>X = 1
*
43 0 ,* *= 2 aux) 0
satisfy
= u = 1 ne =

P(X Uk]
, ,
min ,

I
As
Duality .
0 If finite ,
Iasible
Up b central path
-

gap
=
-
u X
-
,

-
XK is always
,