ARTIFICIAL

INTELLIGENCE &
EXPERT SYSTEMS

Heuristic & Meta-

Heuristic Search
By
Engr. Dr. Jawwad Ahmad
&
Dr. Nasir Uddin
1
 Today’s Goal
 Heuristic

 Meta-Heuristic

 Single Vs Multi-objective Optimization

 Rational / Motivation

 Types of Classification Algorithm

 Trajectory Methods

 Population-Based Methods
Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 2
 Optimization Techniques
 Optimization
o The action of making the best or most effective use of a
situation or resource.
 Heuristic Search is used to find solutions to
problems.
 Heuristic
 “The art of discovering new strategies (rules) to solve
problems”
 Heuristic Optimization is used to find the best
possible solution to an optimization problem.
 Meta
 “Upper level methodology”

 Metaheuristic
 “Upper level general methodologies that can be used as
guiding strategies in designing underlying heuristics to
solve specific optimization problems”
Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 3
Single Vs Multi-objective Optimization
Single Objective Optimization:

 When an optimization problem involves only one objective

function, the task of finding the optimal solution is called Single
Objective Optimization.

 Example: Find out a CAR for me with Minimum cost.

Multi-objective Optimization:

 When an optimization problem involves more than one objective

function, the task of finding one or more optimal solutions is known
as Multi-objective Optimization.

 Example: Find out a CAR for me with Minimum cost and

Maximum comfort.

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 4

Heuristic Optimization Techniques
 In mathematical optimization and computer science, heuristic (from
Greek ευρίσκειν [evrískein] “I find, discover”) is a technique
designed for solving a problem more quickly when classic methods
are too slow or for finding an approximate solution when classic
methods fail to find any exact solution.

 This is achieved by trading optimality, completeness, accuracy,

or precision for speed. In a way, it can be considered a shortcut.

 A heuristic function, also simply called a heuristic, is

a function that ranks different in search algorithms at each
branching step based on available information to decide which
branch to follow.

 For example, it may approximate the exact solution.

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 5

Heuristic Optimization Techniques
 The objective of a heuristic is to produce a solution in a
reasonable time frame that is good enough for solving the
problem at hand.

 This solution may not be the best of all the solutions to this
problem, or it may simply approximate the exact solution.

 But it is still valuable because finding it does not require a

prohibitively long time.

 Heuristics may produce results by themselves, or they may be

used in conjunction with optimization algorithms to improve
their efficiency (e.g., they may be used to generate good seed
values).

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 6

Meta-Heuristic Optimization Techniques
 Similarly, a metaheuristic is a higher level procedure or heuristic
designed to find, generate, or select a heuristic (partial search
algorithm) that may provide a sufficiently good solution to
an optimization problem, especially with incomplete or imperfect
information or limited computation capacity.

 Metaheuristics sample a subset of solutions which is otherwise too

large to be completely enumerated or otherwise explored.

 Metaheuristics may make relatively few assumptions about the

optimization problem being solved and so may be usable for a
variety of problems.

 Compared to optimization algorithms and iterative methods,

metaheuristics do not guarantee that a globally optimal
solution can be found on some class of problems.

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 7

Meta-Heuristic Optimization Techniques
 Many metaheuristics implement some form of stochastic
optimization, so that the solution found is dependent on the set
of random variables generated.

 In combinatorial optimization, by searching over a large set

of feasible solutions, metaheuristics can often find good solutions
with less computational effort than optimization algorithms, iterative
methods, or simple heuristics. As such, they are useful approaches
for optimization problems.

 Several books and survey papers have been published on the

subject. Most literature on metaheuristics is experimental in
nature, describing empirical results based on computer
experiments with the algorithms.

 But some formal theoretical results are also available, often

on convergence and the possibility of finding the global optimum.

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 8

Matrix of Major Meta-Heuristic
Optimization Algorithms

Reference:

C. Blum and A. Roli, “Metaheuristics in

combinatorial optimization: Overview and
conceptual comparison”, ACM Computing
Surveys (CSUR), vol. 35, no. 3, pp. 268-
308, 2003.

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 9

 Rational / Motivation
 Let’s consider the problem of find the best visiting
sequence (route) to serve 14 customers for any
Salesman known as Traveling Salesman Problem
(TSP).

 How many possible routes exists?

billion solutions

 If a PC can check about 1 million solution per second,

then we need
routes / routes/ sec = sec
hours
to check them all and find the best!
Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 10
 Combinatorial Explosion

400 Cities
796,095 Km2

What type of algorithm would you pick to solve a problem

with feasible solutions?
http://www.maps-of-the-world.net/maps/maps-of-asia/maps-of-pakistan/large-scale-detailed-road-map-of-pakistan-with-all-cities.jpg
https://worldpopulationreview.com/countries/cities/pakistan
Engr. Dr. Jawwad Ahmad & Dr. Nasir
11
Uddin
 Two Basic Definitions
 A neighborhood C around a solution is a function that
returns a subset of solutions that are somehow close or
related to
 implies a “direction of search”
 multiple neighborhoods can be defined for a
solution
 could be exhaustive or not (if not, it could
actually be random)

 A move constitutes a choice of a solution shows

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 12

Classification Algorithm Type 1
Based on algorithm “inspiration”

Nature Non-nature

Iterated
Genetic Ant Colony Tabu
Local
Algorithms Optimization Search Search

A classification may not be mutually exclusive

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 13

Classification Algorithm Type 2
Based on whether objective changes along algorithm
execution

Static objective Dynamic objective

Iterated Guided
Tabu
Local Local …
Search
Search Search

Why would we ever “modify” the objective?

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 14
Classification Algorithm Type 3
Based on whether they remember (and utilize for future) their
search history
Memory-less
Using memory
(“Markov” property)

Most NLP
Ant Colony
Simulated local
Tabu Search Optimizatio GRASP
Annealing optimization
n
algorithms

Memory can be “short-term” or “long-term”

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 15
Classification Algorithm Type 4
Based on whether they use multiple types of neighborhoods
(“directions of search”)

Multiple
One neighborhood
neighborhoods

Variable
Most algorithms Neighborhood
Search

Depends on what you classify as neighborhood

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 16
Classification Algorithm Type 5
Based on whether the search evolves one or multiple solutions

(single) Trajectory Population-based

methods methods

Adaptive
Simulated Iterated Local Genetic Ant Colony
Tabu Search Memory
Annealing Search Algorithms Optimization
Programming

Trend towards population-based methods, with integration of

proven ideas from trajectory methods
Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 17
Classification Algorithm Type 5
Trajectory Population-based
(S-metaheuristics) (P-metaheuristics)

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 18

 Trajectory Methods
Trajectory
(S-metaheuristics)
C(s0)

s0

s1

C(s1)

s2

C(s2)

Turns out two global optima in this problem,

Stop because of no
but none was identified
improvement
• One was missed in region
during search C(s2)C(s )
of region 1
• One was far away from searched space
Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 19
Population-Based Methods
Population-based 0Drop
21Get
All
th
st
nd
somenewold points
population
population
points
some points
sampled
(P-metaheuristics)

Again, optimum may or may not have been

sampled
• Typically, the incumbent always remains in the
population, so need only focus on last generation

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 20

Exploration vs. Exploitation
Exploration of the search space (Diversification) and Exploitation of the
best solutions found (Intensification) - Good solutions are clue for
promising regions

In intensification, the promising regions are explored more thoroughly in

the hope to find better solutions

In diversification, non-explored regions must be visited to be sure that all

regions of the search space are evenly explored and to avoid from local
optima traps

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 21

Intensify vs. Diversify

Engr. Dr. Jawwad Ahmad & Dr. Nasir Uddin 22

Thank you

23