Scribd
Upload
9 views11 pages

Math 1013366

Uploaded by

rarehighrise94
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
9 views11 pages

Math 1013366

Uploaded by

rarehighrise94
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

Certainly!

Here's the content rewritten as plain text without LaTeX symbols:

---

Optimization Theory: Important Theorems and Examples

1. Fermat's Theorem (First-order Necessary Condition)

Formula:

For a function f(x), if x* is a local extremum (maximum or minimum), then the first derivative of
f(x) at x* must be zero:

f'(x*) = 0

Example:

Let's say we want to find the maximum or minimum of the function f(x) = x^2 - 4x + 3.

First, compute the derivative of the function: f'(x) = 2x - 4

Set the derivative equal to zero to find the critical points: 2x - 4 = 0 → x = 2

Now, check whether this is a maximum or minimum by examining the second derivative: f''(x) =
2 Since f''(x) > 0, x = 2 is a minimum.
---

2. Second-order Necessary Conditions

Formula:

If f(x) has a critical point x* where f'(x*) = 0, then:

If f''(x*) > 0, x* is a local minimum.

If f''(x*) < 0, x* is a local maximum.

If f''(x*) = 0, further tests are needed.

Example:

Consider the function f(x) = x^3 - 3x^2 + 2.

The first derivative is: f'(x) = 3x^2 - 6x

Setting it to zero to find critical points: 3x^2 - 6x = 0 → x(x - 2) = 0 So, x = 0 or x = 2.


Now, compute the second derivative: f''(x) = 6x - 6

At x = 0, f''(0) = -6, so x = 0 is a local maximum.

At x = 2, f''(2) = 6, so x = 2 is a local minimum.

---

3. Lagrange Multiplier Theorem (Constrained Optimization)

Formula:

For an optimization problem with a constraint, the method of Lagrange multipliers states that
we need to find the stationary points of the Lagrangian function:

L(x, λ) = f(x) + λ g(x)

where:

f(x) is the objective function to be maximized or minimized.


g(x) = 0 is the constraint.

λ is the Lagrange multiplier.

The optimal point occurs when the gradient of the Lagrangian is zero:

∇L(x, λ) = 0

Example:

Maximize f(x, y) = x + y subject to the constraint x^2 + y^2 = 1.

The Lagrangian is: L(x, y, λ) = x + y + λ(x^2 + y^2 - 1)

Taking partial derivatives and setting them equal to zero: ∂L/∂x = 1 + 2λx = 0 ∂L/∂y = 1 + 2λy = 0
∂L/∂λ = x^2 + y^2 - 1 = 0

Solving this system of equations gives the solution x = 1/√2, y = 1/√2.

---

4. Karush-Kuhn-Tucker (KKT) Conditions (Nonlinear Constrained Optimization)


Formula:

For a constrained optimization problem with inequality constraints, the KKT conditions include:

1. Primal feasibility: g(x) ≤ 0

2. Dual feasibility: λ ≥ 0

3. Stationarity: ∇f(x) + ∑ λi ∇gi(x) = 0

4. Complementary slackness: λi gi(x) = 0

Example:

Minimize f(x) = x^2 subject to g(x) = x - 1 ≥ 0.

The KKT conditions give:

Primal feasibility: x - 1 ≥ 0

Dual feasibility: λ ≥ 0
Stationarity: 2x + λ = 0

Complementary slackness: λ(x - 1) = 0

Solving these conditions gives the optimal solution x = 1 with λ = 0.

---

5. Convex Optimization Theorem

Formula:

If f(x) is a convex function and X is a convex feasible region, then any local minimum is also a
global minimum.

Example:

Minimize f(x) = x^2 subject to x ∈ R.

Since f(x) = x^2 is convex (the second derivative f''(x) = 2 > 0), and the feasible region is R, the
global minimum occurs at x = 0.
---

6. Duality Theorem (Strong Duality)

Formula:

For a convex optimization problem, the dual problem provides a lower bound to the primal
problem. Strong duality means that the optimal value of the dual problem is equal to the optimal
value of the primal problem.

If the primal problem is:

min f(x) subject to g(x) ≤ 0

then the dual problem is:

max min L(x, λ) subject to λ ≥ 0

Example:

For a linear programming problem, the duality theorem states that solving the dual problem
gives the same optimal value as solving the primal problem.

---
7. Subdifferential Theorem (Subgradient)

Formula:

For a non-smooth function f(x), the subdifferential at a point x is the set of all vectors g such
that:

f(y) ≥ f(x) + gᵀ(y - x) for all y

where g is called the subgradient of f(x) at x.

Example:

For the function f(x) = |x|, the subdifferential at x = 0 is {-1, 1}, because the function has a sharp
corner at x = 0.

---

8. Frank-Wolfe Algorithm (Conditional Gradient Method)

Formula:

The Frank-Wolfe algorithm updates the current solution by solving a linear approximation of the
objective function at each step:
x(k+1) = argmin { x ∈ X } ∇f(x(k))ᵀ(x - x(k))

where X is the feasible region.

Example:

For a convex problem like minimizing f(x) = x^2 over the interval [0, 1], the Frank-Wolfe method
iteratively moves toward the optimal solution by solving linearized versions of the function.

---

9. Gradient Descent and Convergence Theorem

Formula:

The gradient descent algorithm updates the solution by moving in the direction of the negative
gradient:

x(k+1) = x(k) - α∇f(x(k))

where α is the step size.

Example:

Minimize f(x) = x^2 using gradient descent:


Start with x0 = 2, choose α = 0.1.

Iteratively update x(k) until convergence.

---

10. No-Free-Lunch Theorem (Optimization)

Formula:

The No-Free-Lunch Theorem states that no optimization algorithm can outperform others
across all possible problems. There’s no universal best algorithm.

Example:

For simple problems, methods like gradient descent work well, but for non-smooth or discrete
problems, algorithms like genetic algorithms or simulated annealing may perform better.

---

This content is now in plain text and can be easily used in any text editor or converted to a PDF
using online LaTeX tools.

You might also like