Lecture 1: Introduction

The setting. Let x(t) n , u(t) m , and y(t) p represent vector functions which are parameterized by time t. The general form of a (nite dimensional) nonlinear control system is x = f (x, u, t), y = h(x, u, t). x(t0 ) = x0 , (1) (2)

The vector x(t), called the state history, describes the system state at a given time t. The vector u(t), called the input history or control input, is chosen to inuence the dynamics in some desired way. The vector of functions f describes the systems dynamics and the vector of functions h provides a set of output measurements. If f and h do not depend explicitly on time, then we call the system autonomous or time-invariant. Otherwise, the system is nonautonomous or time-varying. Remark: Any system of ordinary dierential equations can be written in the rst-order form (1). We call any pair (x(t), u(t)) satisfying (1) over some time interval including t = t0 a solution or trajectory. The dening problem of control theory is to nd a control input u, a function of t and possibly of y, which drives the system state from some initial value x0 at time t0 to a desired nal value at some future time tf . If this can be done for any initial state, any nal state, and any future time, we call the system controllable. If u depends only on time, then the control input is called an open-loop control law. If u depends on the output y then the control input is called a closed-loop or feedback control law. (In the case y = x, we call the control law a full state feedback control law.)

x0

t0

Figure 1: Illustration of the (free endpoint) optimal control problem. In optimal control, the problem is to drive the system state to a desired nal value while minimizing some function which measures the cost of state and control excursions. For example, if one wishes to drive the state to zero and to do so rather quickly (regardless how much control eort is required), one would weight x very heavily in the cost function, relative to the weight on u. Alternatively, if one is less concerned that the state converge quickly and more concerned that the actuator not be over-taxed, one would weight the input u more heavily in the cost function. The Optimal Control Problem: Dynamic Programming Approach. Consider the nonlinear dynamics (1) x = f (x, u, t), x(t0 ) = x0

tf

where, for simplicity, f is assumed to be smooth in its arguments. Suppose also that there is a suitably dened performance index (or cost function)
T

V (x(t0 ), u(), t0 ) =
t0

l(x( ), u( ), )d + m(x(T ))

(3)

where the integrand l(, , ) is a smooth, non-negative loss function and the function m() is a smooth, non-negative terminal cost function. One formal statement of the optimal control problem is: Determine the optimal control u (t), for t [t0 , T ], which minimizes the performance index (3). If the problem can be solved, one should expect that increasing the terminal cost relative to the loss function, would drive the state closer and closer to the desired nal value (say zero) at the nal time t = T . Thus, one may think of this optimal control problem as an optimal regulation problem, in the sense that the state is regulated to a set-point (zero). In this problem statement, we do not require that the nal state be zero, although we do penalize the nal error. We therefore call this a free endpoint problem. As outlined in [2] and shown explicitly in [1], the problem stated above can be transformed into the problem of solving the nonlinear PDE V V V V = l x(t), u x(t), ,t ,t + f x(t), u x(t), ,t ,t t x x x for V (x(t), t) subject to the boundary condition V (x(T ), T ) = m(x(T )). The functional dependence of u on its arguments is determined directly from the problem statement. (One simply chooses u to minimize the right-hand side of (4) at each instant in time.) Once the PDE is solved for V , one may compute its gradient and substitute into the formal expression for the optimal control u x(t), V , t . x Equation (4) is known as the Hamilton-Jacobi-Bellman equation. As we will see later in the course, the equation provides an elegant and intuitive approach to optimal control, an approach which Bellman termed dynamic programming. Unfortunately, the approach is not very useful for most problems. In general, the PDE (4) is impossible to solve. The Optimal Control Problem: Variational Approach. In a slightly dierent approach, one may seek to minimize the cost function ( or performance index)
T

(4)

J=
t0

l(x( ), u( ), )d + m(x(T ))

(5)

subject to the constraint x = f (x, u, t), x(t0 ) = x0 by appending the constraint using a Lagrange multiplier as follows: J=
T

(l(x( ), u( ), ) + (f (x( ), u( ), ) x( ))) d + m(x(T )).

Those familiar with calculus of variations will recognize the Lagrange multiplier as a familiar concept. Those familiar with Lagrangian mechanics may recognize it as a technique for incorporating nonholonomic constraints into the Lagrangian formulation. 2

 The minimizing control law u , if it exists, must be such that J is insensitive to small perturbations in u , (t), and (t). (The latter two terms are the optimal state and co-state histories corresponding to the x optimal input u (t).) That is, the augmented cost function J must have a stationary point corresponding (t), (t), u (t)} for each time in the interval [t , T ]. to {x 0 Let H(x, , u, t) = l(x, u, t) + f (x, u, t). (6) A bit of variational calculus leads to the following dierential equations which, in the optimal control literature, are referred to as the Euler-Lagrange equations: H = x and x = H = f (x, u, t) with (T ) = m x (7)
T

with

x(t0 ) = x0 .

(8)

Boundary conditions for this system of rst order ordinary dierential equations are given at opposite ends of the time interval [t0 , T ], forming a two-point boundary value problem. In addition to these dierential equations, the function H must satisfy a stationarity condition in the input: 0= H . u (9)

Equation (9) is akin to the critical point condition, from freshman calculus, in problems where one wishes to maximize or minimize a smooth function. Together, equations (7) through (9) provide necessary conditions for optimality. An additional condition, which is sucient for optimality, is that 2H > 0. u2 {x (t), (t),u (t)} (10)

Linear control systems. Linear control systems are a special case of nonlinear control systems in which f (x, u, t) = A(t)x + B(t)u h(x, u, t) = C(t)x + D(t)u. (11) (12)

The matrix A(t) is called the state matrix, B(t) is called the input matrix, C(t) is called the output matrix, and D(t) is called the throughput matrix. If A, B, C and D do not depend explicitly on time, then we call the system linear time-invariant (LTI). Otherwise, the system is linear time-varying (LTV). We know quite a lot about linear systems. For example: 1. Superposition Principle: The sum of any two solutions for a linear system is also a solution. Among other advantages, this observation simplies the solution of linear ODEs by allowing one to break a given problems solution into homogeneous and particular parts. 2. Unique Equilibrium: An equilibrium is a solution (x(t), u(t)) for which both x and u are constant. (If u = 0, the equilibrium is called a controlled equilibrium.) A generic linear system has a unique equilibrium at the origin x(t) = 0 and its stability may be readily determined. For a LTI system, for example, stability can be determined by computing the eigenvalues of A. In the non-generic case that x(t) = 0 is not a unique equilibrium, there is a continuous family of equilibria (including x(t) = 0) which corresponds to the null space of A(t). A linear system never has multiple, isolated equilibria. 3

3. Controllability: There are easily computed necessary and sucient conditions under which a control exists to drive the state of a linear system from any initial value to any desired nal value in a specied amount of time. Furthermore, a successful check of these necessary and sucient conditions yields one such control law. 4. Observability: There are easily computed necessary and sucient conditions under which the systems state history can be unambiguously determined from its input and output history. This property can be used to construct a provably convergent state estimator. 5. Optimal Control with a Quadratic Cost Function: Given controllable linear dynamics and quadratic loss and terminal cost functions, 1 and m(x(T )) = x(T )T M x(T ) 0, l(x(t), u(t), t) = x(t)T Q(t)x(t) + u(t)T R(t)u(t) 0 2 the solution of the optimal control problem stated earlier can always be obtained, subject to certain conditions on the choice of Q and R. Moreover, the solution is such that the optimal control law u (x, t) is linear in the system state x(t). The problem of nding an optimal controller for a linear system under a quadratic performance index is called the linear-quadratic (LQ) control problem and the resulting controller is called a linear-quadratic controller. State Estimation. As noted in the nal item above, LQ control design leads to a linear feedback control law, u = K(t)x, (13) where K is a time-varying gain matrix. The control law requires that the full state be available for feedback. In general, this will not be feasible. If the full state x is not measured, but the measurements are suciently rich that one may construct a reasonable (i.e., convergent) estimate x, then one may substitute this estimate into the control law (13) and expect the system to perform essentially as it would with full state feedback. Moreover, a separation principle allows one to design the state estimator and the controller independently of one another. Essentially, one denes a ctitious system x = A(t) + B(t)u + G(t) (y y ) x y = C(t) + D(t)u x (14) (15)

where the observer (or output injection) gain G(t) is chosen to ensure that x converges to x. As we will see, the problem of designing G(t) is mathematically dual to the problem of designing the gain K(t) for a linear controller. In fact, there are very good reasons to use a state estimator in any case. If there are unknown random plant disturbances and measurement noise, such an estimator acts as a lter to give a maximum likelihood estimate of the true system state. The Kalman lter is one of the most widely used tools available for guidance and navigation and we will discuss it in fair detail. We will also discuss a commonly used extension, the extended Kalman lter, which is useful for nonlinear systems and we will consider its application for parameter identication problems.

References
[1] B. D. O. Anderson and J. B. Moore. Linear Optimal Control. Prentice Hall, 1971. [2] P. Dorato, C. T Abdallah, and V. Cerone. Linear Quadratic Control: An Introduction. Krieger Publishing, 1995.