Computational Methods for Mechatronics (EMEng 6441)

Chapter 5 Notes: Optimal Control Theory

Remember from Chapter 4 that constrained optimization problem is expressed mathematically as:

From a general perspective, an optimal control problem is an optimization problem. The

difference between the two is that, in optimal control theory, the optimizer is a function, not just
a single value. This function that optimizes is called the optimal control. The technical
definition of an optimal control problem is the process of determining control and state
trajectories for a dynamic system over a period of time to minimize a performance index. The
state variable (or function) is the set of variables (functions) used to describe the mathematical
state of the system. The control or control function is an operation that controls the recording,
processing, or transmission of data. These two functions drive how the system works and how
the desired control is found. With these definitions, a basic optimal control problem can be
defined.

Let us consider a dynamic system, described by the state vector x(t) , and governed by the ODE
system

where u(t) is a set of tunable control inputs.

At time t0, we start at the initial state x(t0) = x[0], and wish to determine the trajectory of control
inputs u(t) within t ∈ [t0, t1] that minimize an integral cost functional

With these definitions, a basic optimal control problem can be defined.

One of the methods we use to solve this problem is using Maximum Principle for Optimality.

To solve our basic optimal control problem, a set of what is called necessary conditions
must be satisfied. Pontryagin introduced the adjoint function to affix the differential equation to
the objective functional. These functions serve a similar purpose as the Lagrange multipliers.

The necessary conditions needed to solve the basic problem are derived from what is referred to
as the Hamiltonian, H,

where denotes the adjoint function and is dependent on t, x and u.

The Maximum Principle says that there is an adjoint function (also called co-state variable) λ(t),
such that an optimal state x(t), and optimal control u(t) must necessarily:

(1) satisfy the state equation,

(2) satisfy the co-state equation

(3) Maximize the Hamiltonian, considered as a function of the control.

This is called Pontryagin’s Maximum Principle1.

1
The derivation of these results can be found in [S. Lenhart and J. T. Workman, Optimal
Control Applied to Biological Models, Boca Raton, FL, Florida: Taylor & Francis Group, LLC,
2007.]
These conditions can be solved explicitly sometimes; however, for most problems, the conditions
are too complicated to be solved explicitly. This is especially true for problems that also involve
additional constraints on the state or the control. Because of these, numerical approaches are used
to construct approximations to these difficult equations. For this, the Runge-Kutta algorithm will
be used to solve such problems.

The steps to solve optimal control problems based on Pontryagin’s Maximum Principle are
summarized as follows:

(1) Form the Hamiltonian for the problem.

(2) Write the adjoint equation and the optimality condition in terms of the three
unknowns

Activity 6

Given the following optimal control problem

Using Pontryagin’s Maximum Principle, the state equation, adjoint function and transversality
boundary condition and the optimality condition are derived as follows:

Step 1: Construct the Hamiltonian

The Hamiltonian for the problem is:

Step 2: Write the adjoint function and optimality condition

Adjoint function

Optimality condition

Step 3: Solve for the optimal control

Step 4: Solve the state and adjoint and state differential equations with their boundary
conditions

6.1. Show that the actual solutions for the state and adjoint are given by:

6.2. We will now solve the optimal control problem using numerical approximations.

One of the methods to solve this problem is the Forward Backward Sweep (FBS). This
iterative method is named based on how the algorithm solves the problem’s state and adjoint
ODE’s. Given an approximation of the control function, FBS first solves the state ‘forward’ in
time (from t0 to t1) then solves the adjoint backwards (from t1 to t0). Once the state and the
adjoint are determined, the control (u) is determined from the optimality condition.
Forward Runge-Kutta-4 algorithm for three inputs is given as:

Backward-Runge-Kutta4 algorithm for three inputs is given as:

The following two Matlab functions (forward_runge_kutta_4.m and

backward_runge_kutta_4) implement the forward RK-4 and backward RK-4, respectively.

function x = forward_runge_kutta_4(t,x,u,N,ode)
% 4th order classic Runge-Kutta Method for solving
% the state equation (optimal control):
% x' = f(t,x,u), x(0) = x0 (can be vector valued)

% vector x(1) = x0 (on input) (row vector or row by number of dim)

% vector u contains control values (row vector or row by dimension)
% vector t contains time values (constant step 1/N)

% constant step
h=1/N;
h2=h/2;

for i=1:N
K1 = ode(t(i),x(:,i),u(:,i));
 K2 = ode(t(i)+h2,x(:,i)+h2*K1,(u(:,i)+u(:,i+1))/2);
K3 = ode(t(i)+h2,x(:,i)+h2*K2,(u(:,i)+u(:,i+1))/2);
K4 = ode(t(i)+h,x(:,i)+h*K3,u(:,i+1));

x(i+1)=x(i)+(h/6)*(K1+2*K2+2*K3+K4);
end

function lambda = backward_runge_kutta_4(t,x,lambda,u,N,ode)

% 4th order classic Runge-Kutta Method for solving
% the adjoint equation (from optimal control):
% lam' = f(t,lam,x,u), lam(T) = lamT (can be vector valued)
% like regular RK, but solved backward in time

% vector lam(N+1) = lamT (on input) (row vector or row by number of dim)
% vector x contains state function values (row vector or row by dim)
% vector u contains control values (row vector or row by dimension)
% vector t contains time values (constant step 1/N)

% constant step
h=1/N;
h2=h/2;

for j=1:N
i = N+2-j;

K1 = ode(t(i),x(:,i),lambda(:,i),u(:,i));
K2 = ode(t(i)-h2,(x(:,i)+x(:,i-1))/2,lambda(:,i)-h2*K1,(u(:,i)+u(:,i-
1))/2);
K3 = ode(t(i)-h2,(x(:,i)+x(:,i-1))/2,lambda(:,i)-h2*K2,(u(:,i)+u(:,i-
1))/2);
K4 = ode(t(i)-h,x(:,i-1),lambda(:,i)-h*K3,u(:,i-1));

lambda(i-1)=lambda(i)-(h/6)*(K1+2*K2+2*K3+K4);
end

The following Matlab function F_B_Sweep.m uses the above two functions to find the state (x), the
adjoint (lambda) and the control (u). The stopping criteria is determined by finding the relative errors for
the state, the co-state, and the control and requiring that all three be less than a specified value.

function [x,lambda,u,k]=F_B_Sweep(x0,t,h,h2,u_func,state,adjoint,control,N)
%Forward_Backward_Sweep(x0,t,h,h2,u_func,state,adjoint,control,N) solves an
%optimal control problem using the forward backward sweep. x0 is the
%initial value for the state ODE. t,h,h2 are the time interval broken into
%intervals with length h. h2 is half the mesh interval size. u_func is how
%the FBS computes the control for each step. state,adjoitn, and control are
%the equations related to the optimal control problem. N is the number of
%mesh points in the interval.

%intial error for while loop

test = -1;

%error bound
delta = 0.001;

%set up for control vector

u = zeros(1,N+1);

%set up for the state vector

x = zeros(1,N+1);

%intial state at time 0

x(1,1) = x0;

%intial for the ajoint equation

lambda = zeros(1,N+1);

%forward backward sweep

k=1;
while(test < 0)

%Saved old data for error computation to see if the stop criteria have
%been met.
oldu = u;
oldx = x;
oldlambda = lambda;

%forward loop for the state

[x] = forward_runge_kutta_4(t,x,u,N,state);

%backward loop for adjoint

[ lambda] = backward_runge_kutta_4(t,x,lambda,u,N,adjoint);

%construction of the control

u1 = control(t,x,lambda);
u= u_func(u1,oldu);

%contrcution of the error for while loop

tempu= delta*sum(abs(u))-sum(abs(oldu-u));
tempx = delta*sum(abs(x))-sum(abs(oldx-x));
templambda = delta*sum(abs(lambda))-sum(abs(oldlambda-lambda));
test = min(tempu,min(tempx,templambda));

%Iteration set
k=k+1;
end
end

Finally, the following Matlab Script uses F_B_Sweep.m to calculate the state, adjoint and control
and plot the results.
%Mesh size
N=1000;
%The integrand for the control problem
f = @(X,U) -.5*(X.^2+U.^2);
%initial state at time 0
x0 = 1;

%the time interval [0,1] broke into N+1 pieces

initial_time=0;
end_time=1;

%The State, Adjoint, and Control functions needed for both processes
state = @(T,X,U)U-X;
adjoint = @(T,X,L,U)L-X;
control = @(T,X,L)-(L);
state_adjoint = @(T,X,U)[state(T,X(1),U);adjoint(T,X(1),X(2),U)];

%Function that choose how the new u compares to the old u

u_func = @(x,y)(x+y)/2;

%Time interval broken into N+1 mesh points

t=linspace(initial_time,end_time,N+1);

%Step size of mesh

h=(end_time-initial_time)/N;
%Have the mesh size
h2=h/2;

[x_gen,lambda_gen,u_gen,k_gen] =
F_B_Sweep(x0,t,h,h2,u_func,state,adjoint,control,N);

%graph of data
% State
figure
subplot(3,1,1)
plot(t,x_gen)
xlabel('Time')
ylabel('State')

% Control
subplot(3,1,2)
plot(t,u_gen)
xlabel('Time')
ylabel('Control')

%Adjoint
subplot(3,1,3)
plot(t,lambda_gen)
xlabel('Time')
ylabel('Adjoint')