Engineering Systems Design and Analysis

(ENG 504)
504)
Lecture 2
/


Email: [email protected] / [email protected]
Webpage: http://www.staff.zu.edu.eg/amhm/

Agenda

AI Paradigms.
Solving Problems by Search
Tree Search Algorithms
Local Search Algorithms

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

AI Paradigms

AI Paradigms

The term intelligence is always defined as the ability to learn effectively, to

react adaptively, to make proper decisions, to communicate in language or
images in a sophisticated way, and to understand.

Main objectives of AI are to develop methods and systems for solving

problems, usually solved by the intellectual activity of humans.

Signs of AI:
Learn or understand from experience.
Make sense out of ambiguous or contradictory messages.
Respond quickly and successfully to new situations.
Think and reason.
Use reasoning to solve problems.
Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

AI Paradigms

Signs of AI:
Deal with perplexing situations.
Understand and infer in ordinary, rational ways.
Apply knowledge to manipulate the environment.
Recognize the relative importance of different elements in a situation.
Turing Test for Intelligence:
A computer can be considered to be smart only
when a human interviewer, conversing with
both an unseen human being and an unseen
computer, can not determine which is which

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

AI Paradigms

AI represents knowledge as sets of symbols.

AI Programs manipulate symbols to solve problems.
Symbols and symbol structures form knowledge representation.
AI deals primarily with symbolic, non-algorithmic problem-solving methods.
Heuristic methods for processing information:

Search.
Inference .

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Solving Problems by
Searching

Search

Many of the tasks underlying AI can be phrased in terms of a search for the
solution to the problem .

For example: in production systems, the need to search for a sequence of rule
applications that lead to the required action.

A search algorithm takes a problem as an input and returns a solution in the

form of an action sequence.

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Search

Problem Space:
Initial state(s).
Goal state(s). (Intermediate states)
Actions or operations.
A solution: is a sequence of actions (path) that map an initial state to
a goal state.
Best solution: the shortest path from the initial state to the goal state.
All available (feasible) paths form a search tree.

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

Search

2013/2014

Example: 8-Puzzle
Goal state

Initial state

Actions: moves up, down, left, right.

Solution: a set of movements that reach the goal state.

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Types of Search Algorithms

UnInformed Search
(brute force)

Informed Search

Breadth-first (BFS)

Greedy
Best first

Depth-first (DFS)

A*
Hill climbing

Uniform-cost (UCS)

(Greedy Local Search)

(Dijkstra shortest path)

Genetic algorithms
Iterative deepening (IDS)

Local Search

Simulated annealing

Bidirectional

Tabu search

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Types of Search Algorithms

UnInformed Search

Blind search

Informed Search
Heuristic search
(a guess about which is best, not exact)

Search through the space of possible Use a cost (heuristic) function to

solutions

evaluate each solution.

No knowledge about which path is

Backtrack if it realizes that the

likely to be best

present path proves unworthy.

Heuristic (it is of Greek origin) means discovery

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Properties of search algorithms

I.

10

Completeness. find one solution at least?

II. Optimality. find the best solution?

III. Time Complexity. Upper bound of consumed time?
IV. Space Complexity. Upper bound of required memory storage?

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

Local Search

2013/2014

Local Search Algorithms paradigm

11

State space = set of "complete" configurations.

Find configuration satisfying constraints.
Define a cost (heuristic) function to evaluate solutions.
Start with some random solution, then keep a single "current" state
and try to improve it.
They are iterative improvement algorithms.
The goal is to find the state with the best cost.

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

Local Search Algorithms paradigm

2013/2014

12

Generation mechanism, generating a transition from one configuration

to another.
Neighborhood is a function which defines a simple transition
from a solution to another solution by doing an action.
Selection of neighborhood:
choose the first lower cost neighbor found;
select the best neighbor in the entire neighborhood;
choose the best of a sample of neighbors.

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Local Search Algorithms paradigm

13

Local search is visualized as state-space landscape.

Location: state.
Elevation: value of heuristic cost function.
The search aims to find the best solution
(global minimum).

Local minimum

(using heuristic cost function)

cost

global minimum

*
State space
Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

Local Search Algorithms

2013/2014

14

Example #1: n-queens problem

Put n queens on an n n board with no two queens on the same row,
column, or diagonal.

h=4

n=4

h= 3

h= 2

Cost function: how many queens attack each other?

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Types of Search Algorithms

UnInformed Search
(brute force)

15
Informed Search

Breadth-first (BFS)

Greedy
Best first

Depth-first (DFS)

A*
Hill climbing

Uniform-cost (UCS)

(Greedy Local Search)

(Dijkstra shortest path)

Genetic algorithms
Iterative deepening (IDS)

Local Search

Simulated annealing

Bidirectional

Tabu search

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Hill climbing algorithm

16

Problem: depending on initial state, can get stuck in local minima.

One solution: start from different initial solutions.

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Hill climbing algorithm

17

Example #1: 4-queens problem solution

(one possible solution path by a computer program)

Initial state

Solution: h= 0
Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Hill climbing algorithm

18

Example #1: 4-queens problem solution

(another possible solution path by a computer program)

Initial state

h=3

Solution: h= 0

h=4

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Hill climbing algorithm

19

What about 8-queens problem?

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Types of Search Algorithms

UnInformed Search
(brute force)

20
Informed Search

Breadth-first (BFS)

Greedy
Best first

Depth-first (DFS)

A*
Hill climbing

Uniform-cost (UCS)

(Greedy Local Search)

(Dijkstra shortest path)

Genetic algorithms
Iterative deepening (IDS)

Local Search

Bidirectional

Tabu search

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Simulated annealing

Zagazig Univ-Eng. Faculty

2013/2014

Genetic Algorithms

Genetic Algorithms

21

The most important terms:

Gene: a basic unit, which controls a property of an individual.
Chromosome: a string of genes; it is used to represent an individual,
or a possible solution of a problem in the solution space.
Population: a collection of individuals.
Crossover (mating) operation: substrings of different individuals are taken
and new strings (offsprings) are produced.
Mutation: random change of a gene in a chromosome.

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Genetic Algorithms

22

The most important terms:

Fitness (goodness) function: a criterion which evaluates each individual.
Selection: a procedure for choosing a part of the population that will
continue the process of searching for the best solution, while the other
part of the population "dies".

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Genetic Algorithms (GA)

23

A simple pseudo code:

1.

Initialize population of possible solutions.

2.

WHILE criterion of termination is not reached DO

{
2.a Crossover two individuals (mother and father) and generate new
individuals (offspring).
2.b Select the most promising ones according to fitness function.
2.c Development. (Terminate if a solution is found)
2.d Mutation (rare).
} (Terminate if time is up)

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Flow chart:

24

If necessary

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

Genetic Algorithms (GA)

2013/2014

25

Graphical representation for the pseudo code:

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Genetic Algorithms

26

Example #2: Guess a number game

One of two players (Pep) thinks up a number (e.g., 001010) and the other
(Alice) has to find it out with a minimal number of questions.
Alice guesses a solution and Pep replies another number indicating the
number of correctly guessed bits.
The number of correct bits is the criterion (for Alice) for the selection
of promising solutions.
If there is no improvement after a certain number of steps, then Alice
should make a change (mutation: randomly flip one bit).

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

Genetic Algorithms

2013/2014

27

Example #2: Guess a number game

(one possible solution)

First iteration:

Alice produces four solutions and Peb replies :

individuals

fitness

A 0 1 0 1 0 1

B 1 1 1 1 0 1

C 0 1 1 0 1 1

D 1 0 1 1 0 0

001010
in mind of Peb

Promising father & mother

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Genetic Algorithms

28

Example #2: Guess a number game

(one possible solution)

001010
in mind of Peb

Second iteration:
mating

New individuals

fitness

C 0 1 1 0 1 1

E 0 1 1 1 0 0 3

D 1 0 1 1 0 0

F 1 0 1 0 1 1 4
Promising father & mother

C 0 1 1 0 1 1

G 0 1 1 0 0 0 4

D 1 0 1 1 0 0

H 1 0 1 1 1 1 3

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Genetic Algorithms

29

Example #2: Guess a number game

(one possible solution)

001010
in mind of Peb

third iteration:
mating

New individuals

fitness

F 1 0 1 0 1 1

H 1 1 1 0 0 0 3

G 0 1 1 0 0 0

0 0 1 0 1 1 5
Promising father & mother

F 1 0 1 0 1 1

G 0 1 1 0 0 0

K 0 1 1 0 1 1 4

1 0 1 0 0 0 4

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Genetic Algorithms

30

Example #2: Guess a number game

(one possible solution)

001010
in mind of Peb

fourth iteration:
mating

New individuals

fitness

I 0 0 1 0 1 1

L 0 0 1 0 0 0 5

J 1 0 1 0 0 0

M 1 0 1 0 1 1 4

I 0 0 1 0 1 1

N 0 0 1 0 1 0 6

J 1 0 1 0 0 0

O 1 0 1 0 0 1 3

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

solution

End

Zagazig Univ-Eng. Faculty

Genetic Algorithms

2013/2014

31

In Example #2:
There was no need for mutation:
Mutation takes evolution out of a "dead end.
When the fitness or matching becomes worse.
Success is achieved after 16 questions, which is four times faster than
checking all the possible combinations (64 possible variants).

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Genetic Algorithms

32

Termination of a GA:
A solution is found that satisfies minimum criteria.
Fixed number of generations reached.
Allocated budget (computation time/money) reached.
The highest ranking solution's fitness is reaching or has reached a plateau
such that successive iterations no longer produce better results.
Combinations of the above.

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

Genetic Algorithms (GA)

2013/2014

33

More GA variations:
Parallel and Distributed Genetic Algorithm (PGA and DGA)

Hybrid Genetic Algorithm (HGA)

Adaptive Genetic Algorithm(AGA)
Fast Messy Genetic Algorithm (FMGA)
Independent Sampling Genetic Algorithm(ISGA)

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Genetic Algorithms (GA)

34

Advantages

Solve problems with multiple

solutions

Easy to avoid trapped in local

optima/minima

Limitations
No absolute assurance that a
genetic algorithm will find a global
optimum/minima
If it is hard to define a fitness
function then GA will not be
useful

Easy for implementation on

GA can not assure constant

parallel architectures

optimization response times

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

Genetic Algorithms (GA)

2013/2014

35

Problem domain of GA:

Timetabling and Scheduling problems.
Traveling Salesman Problem (TSP).
Bioinformatics.
Combinatorial Optimization Problem.
Network Design and Routing Problems.
In general for global optimization problems.

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Materials

36

Chapter 1, Nikola K. Kasabov

Kasabov,, pages 1-14
Chapter 3,, Stuart Russell & Peter Norvig
Norvig,, pages 64
64--106
Chapter 4,, Stuart Russell & Peter Norvig
Norvig,, pages 120
120--130
For more information: ask Google !

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Required

37

Have a look on the following search algorithms:

Breadth-first (BFS)
Depth-first (DFS)
Dijkstra shortest path
A* Best first search
For each one: to know the search strategy , applications, advantages,
limitations and solve few examples.

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

2013/2014

Assignment #1
#1

38

Can we solve an 8-queens problem using genetic algorithms?

Write a computer program in the language you prefer

(i.e., C++, Java, VB, Matlab, ) in order to solve this
problem. Then draw the correct solution on a chess
board either by hand or by a computer program.

Send a soft copy of your answer by email.

DeadLine: 15/12/2013 (two weeks later !)

Dr. A. Helmi Eng. Sys. Design and Analysis (ENG 504)

Zagazig Univ-Eng. Faculty

Thanks

2013/2014