Chapter: 1

Introduction to Engineering
Optimization
 Introduction
• Optimization is derived from the Latin word “optimus”, the best. Thus
optimization focuses on
– “Making things better”
– “Generating more profit”
– “Determining the best”
– “Do more with less ”
• The determination of values for design variables which minimize
(maximize) the objective, while satisfying all constraints.
• Optimization is defined as a mathematical process of obtaining the
set of conditions to produce the maximum or the minimum value
of a function.
• It is ideal to obtain the perfect solution to a design situation.
• Usually all of us must always work within the constraints of the time
and funds available, we can only hope for the best solution
possible.
• Optimization is simply a technique that aids in decision making
but does not replace sound judgment and technical know‐how.
 Historical Development
• Ancient Greek philosophers: geometrical optimization problems,
Zenodorus, 200 B.C.: “A sphere encloses the greatest volume for a
given surface area”.
• People have been “optimizing” forever, but the roots for the modern
day optimization can be traced to World War II.
• Newton, Lagrange, Cauchy, Bernoulli, Euler, and Weirstrass (1697) laid
the foundations of differential calculus methods and calculus of
variations, which deals with the minimization of functionals.
• Cauchy (1847) made the first application of the steepest descent
method to solve unconstrained minimization problems.
• von Neumann (1928) laid the foundations of game theory then the
technique has been applied to solve several mathematical economics
and military problems.
• Dantzig (1947) developed the simplex method for linear programming
problems.
 Continued …
• Kuhn and Tucker (1951) on the necessary and sufficiency conditions
for the optimal solution of programming problems laid the
foundations for a great deal of later research in nonlinear
programming.
• Bellman (1957) developed the principle of optimality for dynamic
programming problems.
• Duffin, Zener, and Peterson (1960s) developed Geometric
programming.
• Gomory (1960s) did pioneering work in integer programming.
• Charnes and Cooper (1961) first proposed goal programming (multi-
objective programming methods).
Modern Methods of Optimization (non-traditional optimization
methods)
• John Holland (1975) originally proposed genetic algorithms.
• Kirkpatrick, Gelatt, and Vecchi (1980s) originally developed
simulated annealing method is based on the mechanics of the
cooling process of molten metals through annealing.
 Continued …
• Hopfield and Tank (1985) originally used the method for optimization
which is based on the immense computational power of the nervous
system to solve perceptional problems in the presence of massive
amount of sensory data through its parallel processing capability.
• Rao (1986) first presented the fuzzy optimization methods to solve
optimization problems for single and multiobjective optimization in
engineering involving design data, objective function, and constraints
stated in imprecise form involving vague and linguistic descriptions.
• Marco Dorigo (1992) first developed ant colony optimization based on
the cooperative behavior of ant colonies, which are able to find the
shortest path from their nest to a food source.
• Kennedy and Eberhart (1995) originally proposed particle swarm
optimization algorithm that mimics the behavior of social organisms
such as a colony or swarm of insects (for example, ants, termites,
bees, and wasps), a flock of birds, and a school of fish.
 Engineering Application of Optimization
• Optimization techniques can be used for:
– Getting a design / system to work
– Reaching the optimal performance
– Making a design / system reliable and robust
• Also provide insight in:
– Design problem
– Underlying physics
– Model weaknesses
• Achieving common goals of Engineering design so as to perform desired
functions under given constraints:
– Functionality
• Better performance: More efficient or effective ways to execute tasks
• Multiple functions: Capabilities to execute two or more tasks
simultaneously
– Value
• Higher perceived value: More features with less price
• Lower total cost: Same or better ownership and sustainability with lower
cost
 Continued …
• Optimization, in its broadest sense, can be applied to solve any
engineering problem.
• Some typical applications from different engineering disciplines
indicate the wide scope of the subject:
1. Design of aircraft and aerospace structures for minimum weight
2. Finding the optimal trajectories of space vehicles
3. Design of civil engineering structures such as frames, foundations,
bridges, towers, chimneys, and dams for minimum cost
4. Optimum design of linkages, cams, gears, machine tools, and other
mechanical components
5. Selection of machining conditions in metal-cutting processes for
minimum production cost
6. Design of material handling equipment, such as conveyors, trucks,
and cranes, for minimum cost
7. Design of pumps, turbines, and heat transfer equipment for maximum
efficiency
 Continued …
8. Optimum design of electrical machinery such as motors,
generators, and transformers
9. Shortest route taken by a salesperson visiting various cities during
one tour
10. Optimal production planning, controlling, and scheduling
11. Optimum design of chemical processing equipment and plants
12. Design of optimum pipeline networks for process industries
13. Selection of a site for an industry
14. Planning of maintenance and replacement of equipment to reduce
operating costs
15. Inventory control
16. Allocation of resources or services among several activities to
maximize the benefit
17. Planning the best strategy to obtain maximum profit in the presence
of a competitor
18. Optimum design of control systems
 Basic Terminology and notations

C2 Set of continues and twice differentiable functions

 Continued …
Design variables
• All the quantities that are treated as variables in the design process and are
called design or decision variables xi, i = 1, 2, . . . , n and are collectively
represented as a design vector X = {x1, x2, . . . , xn}T.
• They are parameters whose numerical values are to be determined to
achieve the optimum design.
• They include values such as size or weight, or the number of teeth in a
gear, coils in a spring, or tubes in a heat exchanger, or etc.
• Design parameters represent any number of variables that may be required
to quantify or completely describe an engineering system.
• The number of variables depends upon the type of design involved.
• As this number increases, so does the complexity of the solution to the
design problems.
Norm/Length of a vector
• If we let x and y be two n‐dimensional vectors, then their dot product is
defined as:

• Thus, the dot product is a sum of the product of corresponding elements of

the vectors x and y.
 Continued …
• Two vectors are said to be orthogonal (normal) if their dot product is
zero, i.e., x and y are orthogonal if x ∙ y =0.
• If the vectors are not orthogonal, the angle between them can be
calculated from the definition of the dot product:

• Where θ is the angle between vectors x and y, and ||x|| represents the
length of the vector x.
• This is also called the norm of the vector.
• The length of a vector x is defined as the square root of the sum of
squares of the components, i.e.,

• The double sum can be written in the matrix form as follows:

• Since Ax represents a vector, the triple product of the above equation

can be also written as a dot product:
 Continued …
Constraints
• Numerical values of identified conditions that must be satisfied to
achieve a feasible solution to a given problem.
• External constraints
– Uncontrolled restrictions or specifications imposed on a system by
an outside agency.
– Ex.: Laws and regulations set by governmental agencies,
allowable materials for house construction
• Internal constraints
– Restrictions imposed by the designer with a keen understanding
of the physical system.
– Ex.: Fundamental laws of conservation of mass, momentum, and
energy
 Mathematical formulation of engineering
optimization
• General mathematical form of a constrained optimization
problem:

• Where f(x), gj(x) and hj(x) are scalar functions of the real
column vector
• The continuous components of xi of x =[x1,x2,…, xn]T are called
the (design) variables
– f(x) is the objective function,
– gj(x) denotes the respective inequality constraints, and
– hj(x) the equality constraint function
 Continued …
• The optimum vector x that solves the formerly defined problem is denoted
by x* with the corresponding optimum function value f(x*).
• If no constraints are specified, the problem is called an unconstrained
minimization problem.
• Other names of Mathematical Optimization:
– Mathematical programming
– Numerical optimization
• The values of the functions f(x), gj(x), hj(x) at any point x = [x1,x2,…, xn] T,
gj(x), may in practise be obtained in different ways.
i. From analytically known formulae, e.g., f(x)= x12 +2x22+Sin x3
ii. As the outcome of some complicated computational process e.g.,
g1(x) = a(x) –amax, where a(x) is the stress, computed by means of a
finite element analysis, at some point in structure, the design of
which is specified by x; or
iii. From measurement taken of a physical process, e.g., h1(x)= T(x)‐To,
where T(x) is the temperature measured at some specified point in a
reactor, and x is the vector of operational settings.
 Continued …
Design space:
• The total region or domain defined by the design variables in the
objective functions–Usually limited by constraints.
• The use of constraints is especially important in restricting the region
where optimal values of the design variables can be searched.
– Unbounded design space
– Not limited by constraints
– No acceptable solutions
• The following operations on the objective function will not change the
optimum solution x*:
• Minimum of f(x) is same as maximum of −f(x), thus without
loss of generality, optimization can be taken to mean
minimization.
• Multiplication (or division) of f(x) by a positive constant c.
• Addition (or subtraction) of a positive constant c to (or from)
f(x).
 Continued …

Minimum of f(x) is same as maximum of −f(x).

Optimum solution of cf(x) or c + f(x) same as that of f (x).

 Classification of Optimization Problems
Optimization pitfalls!
– Proper problem formulation is critical!
– Choosing the right algorithm for a given problem
– Many algorithms contain lots of control parameters
– Optimization tends to exploit weaknesses in models
– Optimization can result in very sensitive designs
– Some problems are simply too hard / large / expensive
• Thus the first task of a designer would be to investigate the class of
problem encountered.
• This will, in many cases, dictate the types of solution procedures to
be adopted in solving the problem.
• Optimization problems can be classified in several ways, as
described below.
 Continued …
 Classification Based on the Existence of Constraints: Constrained
and non-constrained.
 Classification Based on the Nature of the Design Variables:
Parametric and non-parametric.
• Example: problem of minimum-weight design of a prismatic beam
(Cantilever beam under concentrated load): parameter or static
optimization problems and a trajectory or dynamic optimization
problem.
 Continued …
 Classification Based on the Physical Structure of the Problem:
optimal control and nonoptimal control problems, two types of variables:
the control (design) and the state variables
• Control variables define the system and govern the evolution of the
system from one stage to the next, and the state variables describe the
behavior or status of the system in any stage.
• Total objective function (also known as the performance index, PI):

 Classification Based on the Nature of the Equations Involved:

linear, nonlinear, geometric, and quadratic programming problems.
• Nonlinear Programming Problem: If any of the functions among the
objective and constraint functions is nonlinear, the problem is called a
non-linear programming (NLP) problem.
 Continued …
• Geometric Programming Problem. A geometric programming (GMP)
problem is one in which the objective function and constraints are
expressed as posynomials in X.

• A function h(X) is called a posynomial if h can be expressed as the sum

of power terms each of the form,

• Quadratic Programming Problem. A quadratic programming problem is

a nonlinear programming problem with a quadratic objective function and
linear constraints. It is usually formulated as follows:
 Continued …
• Linear Programming Problem. If the objective function and all the
constraints are linear functions of the design variables, the mathematical
programming problem is called a linear programming (LP) problem. A
linear programming problem is often stated in the following standard form:

 Classification Based on the Permissible Values of the Design Variables:

Integer Programming Problem or real-valued programming problem.
 Classification Based on the Deterministic Nature of the Variables:
deterministic and stochastic/probabilistic programming problems
 Classification Based on the Separability of the Functions: separable and
nonseparable programming problems.
• Definition: A function f(X) is said to be separable if it can be expressed as the
sum of n-single-variable functions, f1(x1), f2(x2), . . . , fn(xn), that is,

 Classification Based on the Number of Objective Functions: single- and

multiobjective programming problems.
 Optimization in the design process
Comparison of Conventional Design (CD) and Optimal Design (OD)

• The CD process involves the • The OD process forces the

use of information gathered designer to identify explicitly a
from one or more trial designs set of design variables, an
together with the designer’s objective function to be
experience and intuition. optimized, and the constraint
• Its advantage is that the functions for the system.
designer’s experience and • This rigorous formulation of
intuition can be used in making
the design problem helps the
conceptual changes in the
designer gain a better
system or to make additional
understanding of the problem.
specifications in the procedure.
• The CD process can lead to • Proper mathematical
uneconomical designs and can formulation of the design
involve a lot of calendar time. problem is a key to good
solutions.
 Continued …

• Typical Design Process

 Continued …

Specchio
 Continued …
• Is there one aircraft which is the fastest, most efficient, quietest (making
little or no noise), most inexpensive?

“You can
only
make
one thing
best at a
time.”
 Continued …

• Why Design Optimization ?

 Continued …
Automated Design Optimization:
• Basic Procedure:
Continued …
Continued …
Continued …
Continued …
 Selecting an Optimization Method
• There is no single method available for solving all optimization
problems efficiently.
• Hence a number of optimization methods have been developed for
solving different types of optimization problems.
 Continued …

• Optimization Methods:
 Summary
• Design variables: variables with which the design
problem is parameterized: x = (x1, x2, x3, . . . , xn)
• Objective: quantity that is to be minimized
(maximized), Usually denoted by: f(x) ( “cost function”)
• Constraint: condition that has to be satisfied
– Inequality constraint: g(x) ≤ 0
– Equality constraint: h(x) = 0
• General form of optimization problem:
• Optimization problems are typically
solved with an iterative algorithm: