Metaheuristic

Optimization
Dr. Alejandro Arbelaez
 Presentation
• Alejandro Arbelaez
• Assistant Lecturer at the Department of Computer
Science
• Email: [email protected]

• Qualification:
• Postdoctoral Researcher – University College Cork
(UCC), 2013 – 2017
• Postdoctoral Researcher – University of Tokyo, 2011 –
2013
• PhD. In Computer Science – Université Paris XI, 2007
– 2011
Dr. Alejandro Arbelaez
Dr. Alejandro Arbelaez

4
Dr. Alejandro Arbelaez

5
• 2017 – Present: Assistant Lecturer

• 2013 – 2017: Senior Postdoc at University College Cork

• Combinatorial Optimization
• Decision Analytics
• Machine Learning

• 2011 – 2013: Postdoc at University of Tokyo / Franco-Japanese Research

Lab
• Massively Parallel Algorithms (SAT)

• 2007 – 2011 : PhD Candidate at Université Paris XI

• Combinatorial Optimization & Machine learning (CP/SAT)
K Computer
(RIKEN AICS, Kobe)

No.1 @ Top500 Jun+Nov 2011

8 petaflops
88000 CPUs
each with 8 cores
i.e. total of 704,000 cores

from 04/2012
@ University of Tokyo
1 petaflop
4800 nodes each with 16 cores
Fujitsu PRIMEHPC FX10 i.e. total of 76,800 cores
The DIStributed Core for unlimited
bandwidth supply for all Users
and Services
Co-ordinating partner: Trinity College Dublin, Ireland
Project Coordinator: Prof. David Payne
Tel.+44 (0)1728 747019
Mob. +44(0)7918 658723
e-mail: [email protected]
Project Manager: Dr. Marco Ruffini
Tel: +353 1 896 4336
e-mail: [email protected]

Duration 1-11-2012 to 31-10-2015

Funding Scheme: IP
Total Cost: € 11,652,198
EC contribution € 8,112,824
Proposal No. 318137:

Universities
Telecoms vendors
Telecoms components fabrication
facilities
National operators
Coriant. National research centres
SME
Learning Outcomes
• Module Descriptor
https://courses.cit.ie/index.cfm/page/module/moduleId/13443
Machine Learning

Complexity
Combinatorial Optimization
Learning Outcomes
• Module Descriptor
https://courses.cit.ie/index.cfm/page/module/moduleId/13443

• Categorize a real-life problem with respect to its computational

complexity
• Assess the benefits and limitations of meta-heuristics to solve NP-hard
problems
• Solve an NP-hard problem with meta-heuristics to find a satisfactory lower-
bound solution
• Analyze the average performance of a randomised algorithm to solve an
NP-hard problem
• Apply nature-inspired and local search meta-heuristics to solve real-life
problems
Main Components
• Complexity
• Combinatorial Problems
• Local Search algorithms
• Nature inspired algorithms
Module workload
• 2 hours lecture every week
• 2 hours lab
• 3 hours Independent studies
Assessment Breakdown
• Week 6 (50%)
• Week 12 (50%)
Programming Language
• This module is designed for students with prior programming
experience (and basic probability & statistics knowledge)
• Programming Language à Java / Eclipse
Content
• Introduction
• NP-completeness
• Population-based Meta-heuristics
• Genetic Algorithms
• Swarm Particle optimization
• Ant-Colony Optimization
• Single-solution based Meta-heuristics
• Local Search
• Simulated Annealing
• Etc..
Combinatorial Problems
• Example:
• Given: a set of of points in Euclidean plane
• Objective: Find the shortest round trip

• Note:
• A round trip corresponds to a sequence of two points that are visited one
directly after the other
• Candidate solution: round trip
• Solution: round trip with minimal length
Combinatorial Problems
• Combinatorial Problems arise in many areas of computer science and
applications domains:
• Finding shortest / cheapest round trips (TSP)
• Finding models of propositional formulae (SAT)
• Planning, scheduling, time-tabling
• Vehicle routing
• Location and clustering
• Internet data packet routing
• Protein structure prediction
Optimization
• Optimization refers to choosing the best element from some set of
available alternatives
• Optimization problems:
• Arise in wide variety of applications
• Arise in many different forms, e.g., continuous, combinatorial, multi-
objective, stochastic, etc.
• Here we focus mainly on combinatorial problems
• Range from quite easy to hard ones
• Here we focus on the hard ones
Easy example
Find the best (most valuable) element from the set of alternatives
A more difficult (but still “easy”) one
Find best (shortest) route from A to B in an edge-weighted graph
A harder one
Find best (shortest) round trip through some cities, aka Traveling
Salesman Problem (TSP)
A real-life like problem
TSP arises as sub-problem, e.g., in vehicle routing problems (VRPs)
Combinatorial Problems
• Realistic problems can involve many complicating details
• Examples in VRP case are:
• Time windows, access restrictions, priorities, split delivery, …
• Capacity constraints, different cost of vehicles, …
• Working time constraints, breaks, …
• Stochastic travel times or demands, incoming new request, …
• We will focus on simplified models of (real-life) problems
• Useful for illustrating algorithm principles
• They are “hard” to capture essence of more complex problems
• Are treated in research to yield more general insights
Optimization problems arise everywhere!

Most such problems are very hard (NP-hard!)

How to solve Combinatorial Problems?
• Many possible approaches
• Systematic enumeration is probably not realistic
• Some people may eliminate certain assignment or partial tours
through careful reasoning
• Other intuitive approach: start with some good guess and then try to
improve it iteratively

The latter is an example of heuristic approach to optimization

Solving (combinatorial) optimization
problems
• Systematic enumeration
• Problem specific, dedicated algorithms
• Generic methods for exact optimization
• Heuristic methods
What is a heuristic?
• A heuristic is an alternative optimization methods able to determine
not a perfectly accurate solution, but a set of good quality
approximations to exact solutions, but a set of quality approximations
to exact solution.
• Heuristics, were initially based essentially on experts’ knowledge and
experience and aimed to explore the search space in a particularly
convenient way
Heuristic methods
• Heuristic methods intend to compute efficiently, good solutions to a
problem with no guarantee of optimality
• Range from rather simple to quite sophisticated approaches
• Inspiration often from
• human problem solving
• rules of thumb, common sense rules
• design of techniques based on problem-solving experience
• natural processes
• evolution, swarm behaviors, annealing, ...
• usually used when there is no other method to solve the problem under
given time or space constraints
• Often simpler to implement / develop than other methods
Main characteristics
• A heuristic is designed to provide better computational performance
as compared to conventional optimization techniques, at the expense
of lower accuracy
• The rules of thumb underlying a heuristic are often very specific to
the problem under consideration
• Heuristics use domain-specific representations
What are Metaheuristics?
• The term metaheuristics was proposed by Glover at mid-80s as a
family of searching algorithms able to define a high level heuristic
used to guide other heuristics for a better evolution in the search
space
• The most attractive feature of a metaheuristic is that its application
requires no special knowledge on the optimization problem to solve

Fred W. Glover
This module
• Provide Answers to these questions:
• Which metaheuristic methods are available and what are their
features?
• How can meta-heurists be used to solve computationally hard
problems?
• How should heuristics methods be studied and analyzed empirically?
• How can heuristic algorithms be designed, developed, and
implemented?
Motivations
• Different Metaheuristics
• Difficult selection of the appropriate metaheuristic for a given
problem
• Synergies among different metaheuristics
Different Metaheuristics
• Genetic Algorithms à GAs
• Differential Evolution à DE
• Ant Colony Optimization à ACO
• Evolutionary Computation à ECs
• Local Search à LS
• Tabu Search à TS
• Particle Swarm Optimization à PSO
• More …
Difficult Selection of a Metaheuristic
Difficult Selection of a Metaheuristic
Difficult Selection of a Metaheuristic
Algorithm # of Publications % Total

Local Search 61,700 39.81%

Evolutionary Strategies 19,400 12.51%
Genetic Algorithms 18,900 12.19% 155,000 results
Simulated Annealing 13,300 8.58%
Tabu Search 8,960 5.78%
for the TSP in
Ant Colony Optimization 8,830 5.70% Google Scholar
Scatter Search 5,620 3.63%
Differential Evolution 2,300 2.88%
Particle Swarm Optimization 11,530 1.48%
Others 11,530 7.43%
Difficult Selection of a Metaheuristic
• You may be confused by the different methods, but there are many
more out there!
• I’ll only cover a selection of the metaheuristics which I consider
essential
• There is no such thing as a best metaheuristic!. This has been
actually proven mathematically. No Free Lunch Theorem (FLT), select
the best metaheuristic for the problem at hand
Difficult Selection of a Metaheuristic
• There is no such thing as a best metaheuristic!. This has been
actually proven mathematically. No Free Lunch Theorem (FLT), select
the best metaheuristic for the problem at hand
Taxonomy of Metaheuristics

• Population-based metaheuristics: Evolutionary Algorithms, Scatter

Search, Ant Systems, etc.
• Solution-based metaheuristics: Hill Climbing, Simulated Annealing,
Tabu Search, etc…
Complexity
Time and space
• To analyze an algorithm means:
• developing a formula for predicting how fast an algorithm is,
based on the size of the input (time complexity), and/or
• developing a formula for predicting how much memory an
algorithm requires, based on the size of the input (space
complexity)
• Usually time is our biggest concern
• Most algorithms require a fixed amount of space
What does “size of the input” mean?
• If we are searching an array, the “size” of the input could be the size
of the array
• If we are merging two arrays, the “size” could be the sum of the two
array sizes
• If we are computing the nth Fibonacci number, or the nth factorial,
the “size” is n
• We choose the “size” to be the parameter that most influences the
actual time/space required
• It is usually obvious what this parameter is
• Sometimes we need two or more parameters
Average, best, and worst cases
• Usually we would like to find the average time to perform an algorithm
• However,
• Sometimes the “average” isn’t well defined
• Example: Sorting an “average” array
• Time typically depends on how out of order the array is
• How out of order is the “average” unsorted array?
• Sometimes finding the average is too difficult
• Often we have to be satisfied with finding the worst (longest) time required
• Sometimes this is even what we want (say, for time-critical operations)
• The best (fastest) case is seldom of interest
Average, best, and worst cases -- Sorting

Method Worst Case Average Case

Selection sort N2 n2
Inserstion sort n2 n2
Merge sort n log n n log n
Quick sort n2 n log n
Constant time
• Constant time means there is some constant k such that this
operation always takes k nanoseconds
• A Java statement takes constant time if:
• It does not include a loop
• It does not include calling a method whose time is unknown or is
not a constant
• If a statement involves a choice (if or switch) among operations,
each of which takes constant time, we consider the statement to take
constant time
• This is consistent with worst-case analysis
What is the runtime of g(n)?
void g(int n) {
for (int i = 0; i < n; ++i) f();
}
What is the runtime of g(n)?
void g(int n) {
for (int i = 0; i < n; ++i) f();
}

void g(int n) {
for (int i = 0; i < n; ++i)
for (int j = 0; j < n; ++j) f();
}
What is the runtime of g(n)?
void g(int n) {
for (int i = 0; i < n; ++i)
for (int j = 0; j <= i; ++j) f();
}
Constant time is (usually) better than linear time
• Suppose we have two algorithms to solve a task:
• Algorithm A takes 5000 time units
• Algorithm B takes 100*n time units
• Which is better?
• Clearly, algorithm B is better if our problem size is small, that is, if n < 50
• Algorithm A is better for larger problems, with n > 50
• So B is better on small problems that are quick anyway
• But A is better for large problems, where it matters more
• We usually care most about very large problems
• But not always!
Complexity analysis
• A technique to characterize the execution time of an algorithm
independently from the machine, the language and the compiler.

• Useful for:
• evaluating the variations of execution time with regard to the
input data
• comparing algorithms

• We are typically interested in the execution time of large instances of

a problem, e.g., when n à ¥, (asymptotic complexity).
Big O
• A method to characterize the execution time of an algorithm:
• Adding two square matrices is O(n2)
• Searching in a dictionary is O(log n)
• Sorting a vector is O(n log n)
• Solving Towers of Hanoi is O(2n)
• Multiplying two square matrices is O(n3)
• …
• The O notation only uses the dominating terms of the execution time.
Constants are disregarded.
Big O: formal definition
• Let T(n) be the execution time of an algorithm when the size of input
data is n.
• T(n) is O(f(n)) if there are positive constants c and n0 such that T(n) £
c×f(n) when n ³ n0.
c×f(n)

T(n)

n0 n
Simplifying the formulae
• Throwing out the constants is one of two things we do in
analysis of algorithms
• By throwing out constants, we simplify 12n2 + 35 to just n2
• Our timing formula is a polynomial, and may have terms of
various orders (constant, linear, quadratic, cubic, etc.)
• We usually discard all but the highest-order term
• We simplify n2 + 3n + 5 to just n2
Big O notation
• When we have a polynomial that describes the time requirements of
an algorithm, we simplify it by:
• Throwing out all but the highest-order term
• Throwing out all the constants
• If an algorithm takes 12n3+4n2+8n+35 time, we simplify this formula
to just n3
• We say the algorithm requires O(n3) time
• We call this Big O notation
• (More accurately, it’s Big W)
Can we justify Big O notation?
• Big O notation is a huge simplification; can we justify it?
• It only makes sense for large problem sizes
• For sufficiently large problem sizes, the highest-order term
swamps all the rest!
• Consider R = x2 + 3x + 5 as x varies:
x=0 x2 = 0 3x = 0 5=5 R=5
x = 10 x2 = 100 3x = 30 5=5 R = 135
x = 100 x2 = 10000 3x = 300 5 = 5 R = 10,305
x = 1000 x2 = 1000000 3x = 3000 5 = 5 R = 1,003,005
x = 10,000 x2 = 108 3x = 3*104 5 = 5 R = 100,030,005
x = 100,000 x2 = 1010 3x = 3*105 5 = 5 R = 10,000,300,005
Big O: example
• Let T(n) = 3n2 + 100n + 5, then T(n) = O(n2)
• Proof:
• Let c = 4 and n0 = 100.05
• For n ³ 100.05, we have that 4n2 ³ 3n2 + 100n + 5

• T(n) is also O(n3), O(n4), etc.

Typically, the smallest complexity is used.
y = x2 + 3x + 5, for x=1..10
y = x2 + 3x + 5, for x=1..20
Common time complexities
BETTER • O(1) constant time
• O(log n) log time
• O(n) linear time
• O(n log n) log linear time
• O(n2) quadratic time
• O(n3) cubic time
• O(2n) exponential time
WORSE
Big O: examples
Complexity ranking
 Complexity analysis: examples
for (int i = 0; i < n; ++i) f();

Let us assume that f() has complexity O(1)

 Complexity analysis: examples
for (int i = 0; i < n; ++i) f();

for (int i = 0; i < n; ++i)

for (int j = 0; j < n; ++j) f();

Let us assume that f() has complexity O(1)

 Complexity analysis: examples
for (int i = 0; i < n; ++i) f();

for (int i = 0; i < n; ++i)

for (int j = 0; j < n; ++j) f();

for (int i = 0; i < n; ++i)

for (int j = 0; j <= i; ++j) f();

Let us assume that f() has complexity O(1)

 Complexity analysis: examples
for (int i = 0; i < n; ++i) f();

for (int i = 0; i < n; ++i)

for (int j = 0; j < n; ++j) f();

for (int i = 0; i < n; ++i)

for (int j = 0; j <= i; ++j) f();

for (int i = 0; i < n; ++i)

for (int j = 0; j < n; ++j)
for (int k = 0; k < n; ++k) f();

Let us assume that f() has complexity O(1)

 Complexity analysis: examples
for (int i = 0; i < n; ++i) f();

for (int i = 0; i < n; ++i)

for (int j = 0; j < n; ++j) f();

for (int i = 0; i < n; ++i)

for (int j = 0; j <= i; ++j) f();

for (int i = 0; i < n; ++i)

for (int j = 0; j < n; ++j)
for (int k = 0; k < n; ++k) f();

for (int i = 0; i < m; ++i)

for (int j = 0; j < n; ++j)
for (int k = 0; k < p; ++k) f();
Let us assume that f() has complexity O(1)
Inappropriate Expressions

f (n)X
£ O( g (n))

f (n)X
³ O( g (n))
Properties of Big-Oh
• If f(n) is O(g(n)) then af(n) is O(g(n)) for any a.
• If f(n) is O(g(n)) and h(n) is O(g’(n)) then f(n)+h(n) is O(g(n)+g’(n))
• If f(n) is O(g(n)) and h(n) is O(g’(n)) then f(n)h(n) is O(g(n)g’(n))
• If f(n) is O(g(n)) and g(n) is O(h(n)) then f(n) is O(h(n))
• If f(n) is a polynomial of degree d , then f(n) is O(nd)
• nx = O(an), for any fixed x > 0 and a > 1
• An algorithm of order n to a certain power is better than an algorithm of order a ( > 1) to the
power of n
• log nx is O(log n), for x > 0 – how?
• log x n is O(ny) for x > 0 and y > 0
• An algorithm of order log n (to a certain power) is better than an algorithm of n raised to a
power y.
Asymptotic analysis - terminology
• Special classes of algorithms:
logarithmic: O(log n)
linear: O(n)
quadratic: O(n2)
polynomial: O(nk), k ≥ 1
exponential: O(an), n > 1
• Polynomial vs. exponential ?
• Logarithmic vs. polynomial ?
Warnings about O-Notation
• Big-O notation cannot compare algorithms in the same complexity
class.
• Big-O notation only gives sensible comparisons of algorithms in
different complexity classes when n is large .
• Consider two algorithms for same task:
Linear: f(n) = 1000 n
Quadratic: f'(n) = n2/1000
The quadratic one is faster for n < 1000000.
Complexity analysis: recursion
void f(int n) {
if (n > 0) {
DoSomething(n); // O(n)
f(n/2);
}
}
Complexity analysis: recursion
n
void f(int n) {
if (n > 0) { n/2 n/2
DoSomething(n); // O(n)
f(n/2); f(n/2);
} n/4 n/4 n/4 n/4
}
… … … … … … … …

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Complexity analysis: recursion
void f(int n) {
if (n > 0) {
DoSomething(n); // O(n)
f(n-1);
}
}
Complexity analysis: recursion

void f(int n) {
if (n > 0) {
DoSomething(); // O(1)
f(n-1); f(n-1);
}
}
Asymptotic complexity (small values)
Asymptotic complexity (larger values)
Execution time: example
Let us consider that every operation can be executed in 1 ns (10-9 s)
“Relatives” of Big-Oh
• “Relatives” of the Big-Oh
• W (f(n)): Big Omega – asymptotic lower bound
• Q (f(n)): Big Theta – asymptotic tight bound
• Big-Omega – think of it as the inverse of O(n)
• g(n) is W (f(n)) if f(n) is O(g(n))
• Big-Theta – combine both Big-Oh and Big-Omega
• f(n) is Q (g(n)) if f(n) is O(g(n)) and g(n) is W (f(n))
• Make the difference:
• 3n+3 is O(n) and is Q (n)
• 3n+3 is O(n2) but is not Q (n2)
How about “big data”?
Source: Jon Kleinberg and Éva Tardos, Algorithm Design, Addison Wesley 2006.
Polynomial vs. Exponential growth
Polynomial vs. exponential
Summary
• Complexity analysis is a technique to analyze and compare algorithms
(not programs).
• It helps to have preliminary back-of-the-envelope estimations of runtime
(milliseconds, seconds, minutes, days, years?).
• Worst-case analysis is sometimes overly pessimistic. Average case is also
interesting (not covered in this course).
• In many application domains (e.g., big data) quadratic complexity, 𝑂 𝑛# ,
is not acceptable.
• Recommendation: avoid last-minute surprises by doing complexity
analysis before writing code.
Two key problems in computer science
• The Boolean Satisfiability Problem (SAT)
• Traveling Salesman Problem
The Boolean Satisfiability
Problem
SAT
SAT
• Definition of the satisfiability problem: Given a Boolean
formula, determine whether this formula is satisfiable or not.

• A literal: xi or -xi
• A clause: x1 v x2 v -x3 º Ci
• A formula: conjunctive normal form
C1& C2 & … & Cm
The Satisfiability Problem
The satisfiability problem
A logical formula:
x1 v x2 v x3
& - x1
& - x2
the assignment :
x1 ← F , x2 ← F , x3 ← T
will make the above formula true.
(-x1, -x2 , x3) represents x1 ← F , x2 ← F , x3 ← T
SAT
• If there is at least one assignment which satisfies a formula, then we
say that this formula is satisfiable; otherwise, it is unsatisfiable.
• An unsatisfiable formula:
x1 v x2
& x1 v -x2
& -x1 v x2
& -x1 v -x2
Traveling Salesman Person
TSP
The Mission: A Tour Around the World

Given a list of cities and

the distances between
each pair of cities, what
is the shortest possible
route that visits each
city and returns to the
origin city?
The Problem: Traveling Costs Money

1795$
If(questions != null && questions.count == 0) {
Thanks for your attention
}