Scribd
Upload
51 views2 pages

Conjugate Gradient - 1 1 Linear Conjugate Gradient Method

The document summarizes the conjugate gradient method and coordinate descent method for optimization problems. It discusses: 1) The conjugate gradient method is an iterative method to solve systems of equations by minimizing an objective function with respect to conjugate directions that are orthogonal to each other. 2) The coordinate descent method minimizes an objective function along one coordinate axis at a time, which can take multiple iterations to converge if the objective function is not aligned with the coordinate axes. 3) In conjugate gradient, the search directions must be orthogonal to satisfy the condition of being conjugate directions, and there exists a set of orthogonal eigenvectors of the Hessian matrix that can serve as conjugate directions.

Uploaded by

Arindam Dutta
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
51 views2 pages

Conjugate Gradient - 1 1 Linear Conjugate Gradient Method

The document summarizes the conjugate gradient method and coordinate descent method for optimization problems. It discusses: 1) The conjugate gradient method is an iterative method to solve systems of equations by minimizing an objective function with respect to conjugate directions that are orthogonal to each other. 2) The coordinate descent method minimizes an objective function along one coordinate axis at a time, which can take multiple iterations to converge if the objective function is not aligned with the coordinate axes. 3) In conjugate gradient, the search directions must be orthogonal to satisfy the condition of being conjugate directions, and there exists a set of orthogonal eigenvectors of the Hessian matrix that can serve as conjugate directions.

Uploaded by

Arindam Dutta
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

DS 211

Instructor: Deepak Subramani Lecture 6 Scribe: Arindam Dutta

Conjugate Gradient - 1
1 Linear Conjugate Gradient Method
The Linear Conjugate Gradient Method is an iterative method to solve the system of equa-
tions Ax = b, which can be equivalently formulated as a minimization problem of the type:
1
φ(x) = xT Ax − bT x
2
Hence, the method can well be extended to solve optimization problems.

2 Co-ordinate Descent Method


This method attempts to minimize the objective function, by minimizing the function along
one (axis) direction at a time. For a function of the form
f (X) = 4X12 + X22
this method tries to reduce the function along X1 first and then along X2 , eventually reaching
it’s minima (0,0) in exactly two iterations. However for a function of the type
g(X) = 4X12 + X22 − 2X1 X2
the same method takes multiple iterations to converge the minimum.

• The reason for multiple iterations for g(X) to converge can be attributed to the fact
that g(X) doesn’t have it’s axes lined up along co-ordinate axes, X1 and X2 .
• Whenever the Hessian(H) would be a diagonal matrix (as in case of f (X), not for g(X)),
co-ordinate descent method would converge to the solution in exactly N iterations (N
being the dimension of the Hessian).

3 Conjugate Gradient Method


Let us consider a minimization problem of the type
1
φ(x) = xT Hx + cT x, (1)
2
where H is assumed to be symmetric and positive definite. Let d0 , d1 , ....., dn−1 be n such
linearly independent vectors(directions). We have,
n−1
X
x = x0 + αi di , (2)
i=0

1
DS 211
Instructor: Deepak Subramani Lecture 6 Scribe: Arindam Dutta

upon back substitution to the original problem, we get


1
ψ(α) = αT DT HDα + cT Dα, (3)
2
where D = [d0 d1 .....dn−1 ] and α = [α0 α1 .....αn−1 ].
0 0
Let H = DT HD, then H is a matrix with elements as diT Hdj ∀ i,j.
0
• For H to be a diagonal matrix, assuming H = I, diT dj = 0 ∀i6= j.

This reduces to condition of orthogonality, i.e. the vectors(di ) must be orthogonal to each
other. Such vectors(di ) are referred to as H-conjugate vectors.
Proof of Existence:Let H be a symmetric matrix with n orthogonal eigen-vectors, if v1 v2
eigen vectors, then v1T v2 = 0 and Hv1 = λ1 v1 , a multiplication by v2T results in v2T Hv1 = 0.
This suffices the proof of existence of H-conjugate vectors.

• Residual(rk ) is defined as ∇φ(x) = -c -Hx (from 1).

• Error(ek ) = xk − x∗ , x∗ is the true solution.

• eTk+1 dk =0, i.e. error in (k + 1)th iteration is orthogonal to dk ∀ k ∈ [0,n-1].

You might also like