Artificial Intelligence

For Hedspi Project

Lecture 1 Course presentation

Lecturer : Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST

General information
Course name
Artificial Intelligence

Volume
15 x 2h

Lecturer
Trn c Khnh Department of Information Systems, SOICT, HUT [email protected]

Goals
Total image of AI Basic concepts of AI Major techniques of AI Important applications of AI

Content
Introduction Agents Problem Solving
Search Algorithms Constraint Satisfaction Problems

Logic and Inference Planning Machine Learning

Reference
Book
Artificial Intelligence A Modern Approach. Stuart Russell and Peter Norvig. Prentice Hall

Evaluation
Final examination
70% Achievement test

Continuous assessment
30% Class attendance, midterm test, short tests or quizzes, project

Some educational recommendations

Attend classes Turn off your cell phone Read the reference book
Artificial Intelligence A Modern Approach. Stuart Russell and Peter Norvig. Prentice Hall

Do not hesitate to
Ask questions Give your opinions/feedbacks Discuss with your lecturer

Artificial Intelligence
For HEDSPI Project

Lecture 1 - Introduction
Lecturer: Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST
1

Outline
What is AI? Foundations of AI Short history of AI Philosophical discussions

What is AI?
Views of AI fall into four categories: Think like humans Thinking rationally Act like humans Acting rationally The textbook advocates "acting rationally"

Think like humans

1960s "cognitive revolution": information-processing psychology Scientific theories of internal activities of the brain What level of abstraction? Knowledge'' or circuits? Cognitive science: Predicting and testing behavior of human subjects (top-down) Cognitive neuroscience: Direct identification from neurological data (bottom-up) Both approaches now distinct from AI Both share with AI the following characteristic:
The available theories do not explain (or engender) anything resembling human-level general intelligence}

Act like humans

Turing (1950) "Computing machinery and intelligence": "Can machines think?" "Can machines behave intelligently?" Operational test for intelligent behavior: the Imitation Game

Predicted that by 2000, a machine might have a 30% chance of fooling a lay person for 5 minutes Anticipated all major arguments against AI in following 50 years Suggested major components of AI: knowledge, reasoning, language understanding, learning

Thinking rationally
The Laws of Thought approach What does it mean to think rationally? Normative / prescriptive rather than descriptive Logicist tradition: Logic: notation and rules of derivation for thoughts Aristotle: what are correct arguments/thought processes? Direct line through mathematics, philosophy, to modern AI Problems: Not all intelligent behavior is mediated by logical deliberation What is the purpose of thinking? What thoughts should I have? Logical systems tend to do the wrong thing in the presence of uncertainty
6

Acting rationally
Rational behavior: doing the right thing The right thing: that which is expected to maximize goal achievement, given the available information Doesn't necessarily involve thinking, e.g., blinking Thinking can be in the service of rational action Entirely dependent on goals! Irrational insane, irrationality is sub-optimal action Rational successful Our focus here: rational agents Systems which make the best possible decisions given goals, evidences, and constraints In the real world, usually lots of uncertainty and lots of complexity Usually, were just approximating rationality Computational rationality a better title for this course
7

Rational agents
An agent is an entity that perceives and acts An agent function maps from percept histories to actions:
Agent Sensors Percepts

Environment

?
Actuators Actions

P*

For any given class of environments and tasks, we seek the agent (or class of agents) with the best performance Computational limitations make perfect rationality unachievable So we want the best program for given machine resources

Foundations of AI
Philosophy logic, methods of reasoning, mind as physical system foundations of learning, language, rationality Mathematics formal representation and proof algorithms, computation, (un)decidability, (in)tractability, probability Economics utility, decision theory Neurosciencephysical substrate for mental activity Psychology phenomena of perception and motor control, experimental techniques Computer building fast computers engineering Control theory design systems that maximize an objective function over time Linguistics knowledge representation, grammar

Short history of AI
1940-1950: Early days
1943: McCulloch & Pitts: Boolean circuit model of brain 1950: Turing's ``Computing Machinery and Intelligence'

195070: Excitement: Look, Ma, no hands!

1950s: Early AI programs, including Samuel's checkers program, Newell & Simon's Logic Theorist, Gelernter's Geometry Engine 1956: Dartmouth meeting: ``Artificial Intelligence'' adopted 1964: ELIZA 1965: Robinson's complete algorithm for logical reasoning

197088: Knowledge-based approaches

196979: Early development of knowledge-based systems 198088: Expert systems industry booms

198893: Expert systems industry busts: AI Winter 1988: Statistical approaches

Resurgence of probability, focus on uncertainty General increase in technical depth Agents, agents, everywhere AI Spring?

2000: Where are we now?

10

Expert system
computer data input software hardware output input output knowledge

knowledge engineering

Expert system = Human Expertise + Inference/Reasoning

Some examples: DENDRAL, MYCIN, PROSPECTOR, MOLGEN, ICAD/ICAM

11

State of the art

May, '97: Deep Blue vs. Kasparov
First match won against world-champion Intelligent creative'' play 200 million board positions per second! Humans understood 99.9 of Deep Blue's moves Can do about the same now with a big PC cluster

Proved a mathematical conjecture (Robbins conjecture) unsolved for decades No hands across America (driving autonomously 98% of the time from Pittsburgh to San Diego) During the 1991 Gulf War, US forces deployed an AI logistics planning and scheduling program that involved up to 50,000 vehicles, cargo, and people NASA's on-board autonomous planning program controlled the scheduling of operations for a spacecraft Proverb solves crossword puzzles better than most humans

12

Philosophical discussions
What Can AI Do? Play a decent game of table tennis? Drive safely along a curving mountain road? Buy a week's worth of groceries on the web? Discover and prove a new mathematical theorem? Converse successfully with another person for an hour? Perform a complex surgical operation? Unload a dishwasher and put everything away? Translate spoken English into spoken Vietnamese in real time? Write an intentionally funny story? Can machine think?

13

Some problems with AI

People might lose their jobs to automation. People might have too much (or too little) leisure time. People might lose their sense of being unique. People might lose some of their privacy rights. The use of A1 systems might result in a loss of accountability. The success of A1 might mean the end of the human race.

14

Artificial Intelligence
For HEDSPI Project

Lecture 2 - Agent
Lecturer: Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST
1

Outline
1. 2.

3. 4.

Agents and environments PEAS (Performance measure, Environment, Actuators, Sensors) Environment types Agent types

Agents and environments

An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators Example 1: human agent
Sensors: eyes, ears, Actuators: hands, legs, mouth,

Example 2: robotic agent (e.g., Aishimo)

Sensors: camera, infrared range finders Actuators: various motors
3

Agents and environments (cont)

The agent function maps from percept histories to actions:
[ f: P* A]
Agent Sensors Percepts

Environment

?
Actuators Actions

The agent program runs on the physical architecture to produce the agent function agent = architecture + program

Agent function based on conditional table

Function TABLE-DRIVEN-AGENT(percept) returns an action static: percepts, a sequence, initially empty table, a table of actions, indexed by percept sequences, initially fully specified

Append percept to the end of percepts action LOOKUP(percepts, table) Return action

Drawback: huge table!

Vacuum-cleaner world

Percept sequence Action

Percepts: location (A or B), state (clean or dirty) Actions: Left, Right, Suck, NoOp [A, clean] [A, dirty] [B, clean] [B, dirty] [A, clean][A, clean] [A, clean][A, dirty] Right Suck Left Suck Right Suck

Vacuum-cleaner world
Funtion Reflex-Vacuum-Agent([position, state]) returns action If state = Dirty then return Suck Else if position = A then return Right Else if position = B then return Left End Function Does the agent act reasonably?

Rational agent
A rational agent is one that does the right thing the one that will cause the agent to be most successful Performance measure embodies the criterion for success of an agent's behavior.
E.g., performance measure of a vacuum-cleaner agent:
amount of dirt cleaned up amount of time taken amount of electricity consumed amount of noise generated
8

Rational agent
For each possible percept sequence, a rational agent should select an action that is expected to maximize its performance measure, given the evidence provided by the percept sequence and whatever built-in knowledge the agent has. An agent is autonomous if its behavior is determined by its own experience (with ability to learn and adapt)
9

PEAS
4 factors should be considered when design an automated agent:
Performance measure Environment Actuators Sensors

10

PEAS - automated taxi driver

Performance measure: Safe, fast, legal, comfortable trip, maximize profits, Environment: Roads, other traffic, pedestrians, weather, Actuators: Steering wheel, accelerator, brake, signal, horn, Sensors: Cameras, sonar, speedometer, GPS, odometer, engine sensors, keyboard,

11

PEAS - Medical diagnosis system

Performance measure: Healthy patient, minimize costs, lawsuits, Environment: Patient, hospital, staff Actuators: Screen display (questions, tests, diagnoses, treatments, referrals) Sensors: Keyboard (entry of symptoms, findings, patient's answers)
12

PEAS - Spam Filtering Agent

Performance measure: spam block, false positives, false negatives Environment: email client or server Actuators: mark as spam, transfer messages Sensors: emails (possibly across users), traffic, etc.

13

Environment types
Fully observable (vs. partially observable): An agent's sensors give it access to the complete state of the environment at each point in time. Deterministic (vs. stochastic): The next state of the environment is completely determined by the current state and the action executed by the agent. Episodic (vs. sequential): The agent's experience is divided into atomic "episodes" (each episode consists of the agent perceiving and then performing a single action.
14

Environment types
Static (vs. dynamic): The environment is unchanged while an agent is deliberating. Discrete (vs. continuous): A limited number of distinct, clearly defined percepts and actions. Single agent (vs. multiagent): An agent operating by itself in an environment.
15

Agent types
Four basic agent types:
Simple reflex agents Model-based reflex agents Goal-based agents Utility-based agents

16

Simple reflex agent

Agent
Sensors What the world is like now

These agents select actions on the basis of the current percept, ignoring the rest of the percept history
Conditionaction rules

Environment

What action I should do now

Effectors

Function SIMPLE-REFLEX- AGENT(percept) returns an action static: rules, a set of condition-action rules state INTERPRET-INPUT(percept) rule RULE-MATCH(state, rules) action RULE-ACTION[rule] return action
17

Model-based reflex agents

These agents maintain internal states that depends on the percept history and thereby reflects at least some of the unobserved aspects of the current state.
Sensors State How the world evolves What the world is like now

Environment

What my actions do

Conditionaction rules

What action I should do now

Agent

Effectors

function REFLEX-AGENT-WITH-STATE(percept) returns an action) static: state, a description of the current world state rules, a set of condition-action rules action, the most recent action, initially none state UPDATE-STATE(state, action, percept) rule RULE-MATCH(state, rules) action RULE-ACTION[rule] return action
18

Goal-based agents
Sensors State How the world evolves What the world is like now What it will be like if I do action A

Environment

What my actions do

Goals

Agents that take actions in the pursuit of a goal or goals.

What action I should do now

Agent

Effectors

Goals introduce the need to reason about the future or other hypothetical states. It may be the case that none of the actions an agent can currently perform will lead to a goal state.
19

Utility-based agents
Sensors State How the world evolves What the world is like now What it will be like if I do action A How happy I will be in such a state What action I should do now

Environment

What my actions do

Utility

Agents that take actions that make them the most happy in the long run.

Agent

Effectors

More formally agents that prefer actions that lead to states with higher utility. Utility-based agents can reason about multiple goals, conflicting goals, and uncertain situations.

20

Learning agents

Learning allows the agent to operate in initially unknown environments and to become more competent than its initial knowledge alone might allow. The most important question: What kind of performance element will my agent need to do this once it has learned how?
21

Knowledge bases
Knowledge base is a set of sentences in a formal language, telling an agent what it needs to know Agent can ASK itself what to do, the answer should follow from the KB Agents can be viewed at:
the knowledge level: what they know, what its goals are the implementation level: data structures in KB and algorithms that manipulate them

The agent must be able to:

Incorporate new percepts Update internal representations of the world Deduce hidden properties of the world Deduce appropriate actions
22

Knowledge-based agents
function KB-AGENT(percept) returns an action static: KB, a knowledge base t, a counter, initially 0, indicating time TELL(KB, MAKE-PERCEPT-SENTENCE(percept,t) action ASK(KB, MAKE-ACTION- QUERY(^)) TELL(KB, MAKE-ACTION-SENTENCE(action,t) ) t t+1 return action

23

Multi-agent planning
Environment: cooperative or competitive Issue: the environment is not static synchronization Require a model of the other agent's plans Cooperation: joint goals and plans, e.g., team planning in doubles tennis. Joint goal: returning the ball that has been hit to them and ensuring that at least one of them is covering the net Joint plan: multibody planning Coordination mechanisms: decompose and distribute tasks Competition: e.g., chess-playing An agent in a competitive environment must
recognize that there are other agents compute some of the other agent's possible plans compute how the other agent's plans interact with its own plans decide on the best action in view of these interactions.
24

Artificial Intelligence
For HEDSPI Project

Lecturer 3 - Search
Lecturer: Tran Duc Khanh Dept of Information Systems School of Information and CommunicationTechnology HUST

Outline
Problem-solving agents Problem types Problem formulation Example problems Basic search algorithms
breadth-first search depth-first search depth-limited search iterative deepening depth-first search
2
2

Problem-solving agents

3
3

Example: Route Planning

Performance: Get from Arad to Bucharest as quickly as possible Environment: The map, with cities, roads, and guaranteed travel times Actions: Travel a road between adjacent cities
4

Problem types
Deterministic, fully observable single-state problem Agent knows exactly which state it will be in; solution is a sequence Non-observable sensorless problem (conformant problem) Agent may have no idea where it is; solution is a sequence Nondeterministic and/or partially observable contingency problem percepts provide new information about current state often interleave search, execution Unknown state space exploration problem

Search Problem Definition

A problem is defined by four items: 1. initial state: e.g., Arad 2. actions or successor function S(x) = set of action-state pairs
e.g., S(Arad) = {<Arad
3.

Zerind, Zerind>, }

goal test, can be

explicit, e.g., x = Bucharest implicit, e.g., Checkmate(x)

4.

path cost (additive)

e.g., sum of distances, number of actions executed, etc. c(x,a,y) is the step cost, assumed to be 0

A solution is a sequence of actions leading from the initial state to a goal state
6

Example: The 8-puzzle

states? actions? goal test? path cost?

locations of tiles move blank left, right, up, down = goal state (given) 1 per move
7

Search tree

Search trees:
Represent the branching paths through a state graph. Usually much larger than the state graph. Can a finite state graph give an infinite search tree?

Search space of the game Tic-Tac-Toe

Tree and graph

B is parent of C C is child of B A is ancestor of C C is decendant of A

10

Convert from search graph to search tree

We can turn graph search problems into tree search problems by:
replacing undirected links by 2 directed links avoiding loops in path (or keeping trach of visited nodes globally)
11

Tree search algorithms

Basic idea:
offline, simulated exploration of state space by generating successors of already-explored states
fring e

n0

n successors(n) Goal

12

Implementation: general tree search

13

Implementation: states vs. nodes

A state is a (representation of) a physical configuration A node is a data structure constituting part of a search tree includes state, parent node, action, path cost g(x), depth

The Expand function creates new nodes, filling in the various fields and using the SuccessorFn of the problem to create the corresponding states.
14

Search strategies
A search strategy is defined by picking the order of node expansion Strategies are evaluated along the following dimensions: completeness: does it always find a solution if one exists? time complexity: number of nodes generated space complexity: maximum number of nodes in memory optimality: does it always find a least-cost solution? Time and space complexity are measured in terms of b: maximum branching factor of the search tree d: depth of the least-cost solution m: maximum depth of the state space (may be )

15

Uninformed search strategies

Uninformed search strategies use only the information available in the problem definition Breadth-first search Expand shallowest unexpanded node fringe = queue (FIFO) out n fringe successors(n) Uniform-cost search Expand cheapest unexpanded node fringe = queue ordered by path cost Depth-first search Expand deepest unexpanded node fringe = stack (LIFO) Iterative deepening search
16

in

successors(n)

in

out

n fringe

Depth-limited search: depth-first search with depth limit

Breadth-first search
Expand shallowest unexpanded node

17

Breadth-first search (cont)

Complete? Yes (if b is finite) Time? 1+b+b2+b3+ +bd + b(bd-1) = O(bd+1) Space? O(bd+1) (keeps every node in memory) Optimal? Yes (if cost = 1 per step)

18

Uniform-cost search
Expand cheapest unexpanded node n (cost so far to reach n) fringe = queue ordered by path cost Equivalent to breadth-first if step costs all equal Complete? Yes, if step cost Time? # of nodes with g cost of optimal solution, O(bceiling(C*/ )) where C* is the cost of the optimal solution Space? # of nodes with g cost of optimal solution, O(bceiling(C*/ )) Optimal? Yes nodes expanded in increasing order of g(n)

19

Priority Queue Refresher

A priority queue is a data structure in which you can insert and retrieve (key, value) pairs with the following operations:
pq.setPriority(key, value) pq.dequeue() inserts (key, value) into the queue. returns the key with the lowest value and removes it

You can promote or demote keys by resetting their priorities Unlike a regular queue, insertions into a priority queue are not constant time, usually O(log n) Well need priority queues for most cost-sensitive search methods.
20

Depth-first search
Expand deepest unexpanded node

21

Depth-first search (cont)

Complete? No: fails in infinite-depth spaces, spaces with loops
Modify to avoid repeated states along path complete in finite spaces

Time? O(bm): terrible if m is much larger than d

but if solutions are dense, may be much faster than breadth-first

Space? O(bm), i.e., linear space! Optimal? No

22

Depth-limited search
Depth-first search can get stuck on infinite path when a different choice would lead to a solution Depth-limited search = depth-first search with depth limit l, i.e., nodes at depth l have no successors

23

8-puzzle game with depth limit l = 5

Iterative deepening search

1. 2. 3. 4.

Problem with depth-limited search: if the shallowest goal is beyond the depth limit, no solution is found. Iterative deepening search:
Do a DFS which only searches for paths of length 1 or less. (DFS gives up on any path of length 2) If 1 failed, do a DFS which only searches paths of length 2 or less. If 2 failed, do a DFS which only searches paths of length 3 or less. .and so on.

25

Iterative deepening search (cont)

26

Iterative deepening search (cont)

27

Iterative deepening search (cont)

Number of nodes generated in a depth-limited search to depth d with branching factor b: NDLS = b0 + b1 + b2 + + bd-2 + bd-1 + bd Number of nodes generated in an iterative deepening search to depth d with branching factor b: NIDS = (d+1)b0 + d b^1 + (d-1)b^2 + + 3bd-2 +2bd-1 + 1bd For b = 10, d = 5,

NDLS = 1 + 10 + 100 + 1,000 + 10,000 + 100,000 = 111,111 NIDS = 6 + 50 + 400 + 3,000 + 20,000 + 100,000 = 123,456
Overhead = (123,456 - 111,111)/111,111 = 11%
28

Properties of iterative deepening search

Complete? Yes Time? (d+1)b0 + d b1 + (d-1)b2 + + bd = O(bd) Space? O(bd) Optimal? Yes, if step cost = 1

29

Summary of algorithms

30

Artificial Intelligence
For HEDSPI Project

Lecturer 4 - Search
Lecturer: Tran Duc Khanh Dept of Information Systems School of Information and Communicarion Technology HUST

Outline
Graph search Best-first search A* search

Graph search

Get from Arad to Bucharest as quickly as possible

3

Graph search
Failure to detect repeated states can turn a linear problem into an exponential one!

Very simple fix: never expand a node twice

4

Graph search

Never expand a node twice!

5

Review uniform-cost search

Task: Find the shorted/cheapest path from the start node to the goal. Pick cheapest unexpanded node (node with minimum cost from the start node to it) Problem Uniform-cost search concerns only with expanding short paths; it pays no attention to the goal
6

Straight Line Distances

Best-first search
Idea: use an evaluation function f(n) for each node
estimate of "desirability" Expand most desirable unexpanded node

Order the nodes in fringe in decreasing order of desirability Special cases:

greedy best-first search A* search
8

Greedy Best-First Search

Evaluation function f(n) = h(n) (heuristic) = estimate of cost from n to goal e.g., hSLD(n) = straight-line distance from n to Bucharest Greedy best-first search expands the node that appears to be closest to goal

10

Greedy best-first search example

Arad 366

11

Greedy best-first search example

Arad 366

Sibiu 253

Timisoara 329

Zerind 374

12

Greedy best-first search example

Arad 366

Sibiu 253

Timisoara 329 Rimnicu Vicea 193

Zerind 374

Arad 366

Fagaras 176

Oradea 380

13

Greedy best-first search example

Arad 366

Sibiu 253

Timisoara 329 Rimnicu Vicea

Zerind 374

Arad 366 Sibiu 253

Fagaras 176 Bucharest 0

Oradea 380

14

Greedy Best-First Search

Complete? No can get stuck in loops, e.g., Iasi Neamt Iasi Neamt Time? O(bm), but a good heuristic can give dramatic improvement Space? O(bm) -- keeps all nodes in memory Optimal? No What do we need to do to make it complete? A* search Can we make it optimal? No
15

A* search
Idea: Expand unexpanded node with lowest evaluation value Evaluation function f(n) = g(n) + h(n) g(n) = cost so far to reach n h(n) = estimated cost from n to goal f(n) = estimated total cost of path through n to goal Nodes are ordered according to f(n).

16

A* search example
Arad 366 = 0 + 366

17

A* search example
Arad 366 = 0 + 366

Sibiu 393= 40+253

Timisoara 447=118+329

Zerind 449=75+374

18

A* search example
Arad 366 = 0 + 366

Sibiu 393= 40+253

Timisoara 447=118+329 Rimricu Vicea 413=220+193

Zerind 449=75+374

Arad 646=280+366

Fagaras 415=239+176

Oradea 671=291+380

19

A* search example
Arad 366 = 0 + 366

Sibiu 393= 40+253

Timisoara 447=118+329 Rimricu Vicea 413=220+193

Zerind 449=75+374

Arad 646=280+366

Fagaras 415=239+176

Oradea 671=291+380 Craiova 526=366+160

Pitesti 417=317+100

Sibiu 553=300+253

20

A* search example
Arad 366 = 0 + 366

Sibiu 393= 40+253

Timisoara 447=118+329 Rimricu Vicea 413=220+193

Zerind 449=75+374

Arad 646=280+366 Sibiu 591=338+253

Fagaras 415=239+176 Bucharest 450=450+0

Oradea 671=291+380 Craiova 526=366+160

Pitesti 417=317+100

Sibiu 553=300+253

21

A* search example
Sibiu 393= 40+253

Arad 366 = 0 + 366

Timisoara 447=118+329 Rimricu Vicea 413=220+193

Zerind 449=75+374

Arad 646=280+366 Sibiu 591=338+253

Fagaras 415=239+176 Bucharest 450=450+0

Oradea 671=291+380 Craiova 526=366+160 Bucharest 418=418+0

Pitesti 417=317+100 Craiova 615=455+160

Sibiu 553=300+253 Rimricu Vicea 607=414+193

Can we Prove Anything?

If the state space is finite and we avoid repeated states, the search is complete, but in general is not optimal If the state space is finite and we do not avoid repeated states, the search is in general not complete If the state space is infinite, the search is in general not complete
23

Admissible heuristic
Let h*(N) be the true cost of the optimal path from N to a goal node Heuristic h(N) is admissible if: 0 h(N) h*(N) An admissible heuristic is always optimistic

24

Admissible heuristics
The 8-puzzle: h1(n) = number of misplaced tiles h2(n) = total Manhattan distance (i.e., no. of squares from desired location of each tile)

5 6 7
h1(S) = ? h2(S) = ?

4 1 3
Start State

5 1 8 8 2 2 6 8 7

4 2

3 8 4

6
Goal State

2 5

7 2+3+3+2+4+2+0+2 = 18
25

Heuristic quality
Effective branching factor b* Is the branching factor that a uniform tree of depth d would have in order to contain N+1 nodes.

N + 1 = 1+ b * +(b*) 2 + ...+ (b*) d

Measure is fairly constant for sufficiently hard problems.
Can thus provide a good guide to the heuristics overall usefulness. A good value of b* is 1.

26

Heuristic quality and dominance

1200 random problems with solution lengths from 2 to 24. If h2(n) >= h1(n) for all n (both admissible) then h2 dominates h1 and is better for search

27

Inventing admissible heuristics

Admissible heuristics can be derived from the exact solution cost of a relaxed version of the problem: Relaxed 8-puzzle for h1 : a tile can move anywhere
As a result, h1(n) gives the shortest solution

Relaxed 8-puzzle for h2 : a tile can move to any adjacent square.

As a result, h2(n) gives the shortest solution.

The optimal solution cost of a relaxed problem is no greater than the optimal solution cost of the real problem.

28

Optimality of A*(standard proof)

Suppose suboptimal goal G2 in the queue. Let n be an unexpanded node on a shortest to optimal goal G.
= g(G2 ) since h(G2 )=0 > g(G) since G2 is suboptimal >= f(n) since h is admissible Since f(G2) > f(n), A* will never select G2 for expansion
29

f(G2 )

Optimality for graphs?

Admissibility is not sufficient for graph search
In graph search, the optimal path to a repeated state could be discarded if it is not the first one generated Can fix problem by requiring consistency property for h(n)

A heuristic is consistent if for every successor n' of a node n generated by any action a, h(n) c(n,a,n') + h(n') (aka monotonic) admissible heuristics are generally consistent

30

A* is optimal with consistent heuristics

If h is consistent, we have f(n') = g(n') + h(n') = g(n) + c(n,a,n') + h(n') g(n) + h(n) = f(n) i.e., f(n) is non-decreasing along any path.

Thus, first goal-state selected for expansion must be optimal Theorem: If h(n) is consistent, A* using GRAPH-SEARCH is optimal

31

Contours of A* Search
A* expands nodes in order of increasing f value Gradually adds "f-contours" of nodes Contour i has all nodes with f=fi, where fi < fi+1

32

Contours of A* Search

With uniform-cost (h(n) = 0, contours will be circular With good heuristics, contours will be focused around optimal path A* will expand all nodes with cost f(n) < C*
33

A* search, evaluation
Completeness: YES
Since bands of increasing f are added Unless there are infinitely many nodes with f<f(G)

34

A* search, evaluation
Completeness: YES Time complexity:
Number of nodes expanded is still exponential in the length of the solution.

35

A* search, evaluation
Completeness: YES Time complexity: (exponential with path length) Space complexity:
It keeps all generated nodes in memory Hence space is the major problem not time

36

A* search, evaluation
Completeness: YES Time complexity: (exponential with path length) Space complexity:(all nodes are stored) Optimality: YES
Cannot expand fi+1 until fi is finished. A* expands all nodes with f(n)< C* A* expands some nodes with f(n)=C* A* expands no nodes with f(n)>C*

Also optimally efficient (not including ties)

37

Compare Uniform Cost and A*

Uniform-cost expanded in all directions
A* expands mainly toward the goal, but does hedge its bets to ensure optimality

38

Artificial Intelligence
For HEDSPI Project

Lecturer 5 - Advanced search methods

Lecturers: Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST

Outline
Memory-bounded heuristic search Hill-climbing search Simulated annealing search Local beam search

Memory-bounded heuristic search

Some solutions to A* space problems (maintain completeness and optimality)
Iterative-deepening A* (IDA*)
Here cutoff information is the f-cost (g+h) instead of depth

Recursive best-first search(RBFS)

Recursive algorithm that attempts to mimic standard best-first search with linear space.

(simple) Memory-bounded A* ((S)MA*)

Drop the worst-leaf node when memory is full

Iterative Deeping A*
Iterative Deeping version of A*
use threshold as depth bound
To find solution under the threshold of f(.)

increase threshold as minimum of f(.) of

previous cycle

Still admissible same order of node expansion Storage Efficient practical

but suffers for the real-valued f(.) large number of iterations
4

Iterative Deepening A* Search Algorithm ( for tree search) start set threshold as h(s) put s in OPEN, compute f(s) yes threshold = min( f(.) | f(.) > threshold)

OPEN empty ?

Remove the node of OPEN whose f(.) value is smallest and put it in CLOSE (call it n) yes

n = goal ?

Success

Expand n. calculate f(.) of successor if f(suc) < threshold then Put successors to OPEN if pointers back to n
5

Recursive best-first search

A variation of Depth-first search Keep track of f-value of the best alternative path Unwind if f-value of all children exceed its best alternative When unwind, store f-value of best child as its f-value When needed, the parent regenerate its children again.

Recursive best-first search

function RECURSIVE-BEST-FIRST-SEARCH(problem) return a solution or failure return RFBS(problem,MAKE-NODE(INITIAL-STATE[problem]),) function RFBS( problem, node, f_limit) return a solution or failure and a new f-cost limit if GOAL-TEST[problem](STATE[node]) then return node successors EXPAND(node, problem) if successors is empty then return failure, for each s in successors do f [s] max(g(s) + h(s), f [node]) repeat best the lowest f-value node in successors if f [best] > f_limit then return failure, f [best] alternative the second lowest f-value among successors result, f [best] RBFS(problem, best, min(f_limit, alternative)) if result failure then return result

Recursive best-first search

Keeps track of the f-value of the best-alternative path available.
If current f-values exceeds this alternative f-value than backtrack to alternative path. Upon backtracking change f-value to best f-value of its children. Re-expansion of this result is thus still possible.

RBFS evaluation
RBFS is a bit more efficient than IDA* Still excessive node generation (mind changes) Like A*, optimal if h(n) is admissible Space complexity is O(bd). IDA* retains only one single number (the current f-cost limit) Time complexity difficult to characterize Depends on accuracy if h(n) and how often best path changes. IDA* and RBFS suffer from too little memory.

(simplified) memory-bounded A*
Use all available memory.
I.e. expand best leafs until available memory is full When full, SMA* drops worst leaf node (highest f-value) Like RBFS, we remember the best descendant in the branch we delete

What if all leafs have the same f-value?

Same node could be selected for expansion and deletion. SMA* solves this by expanding newest best leaf and deleting oldest worst leaf.

The deleted node is regenerated when all other candidates look worse than the node. SMA* is complete if solution is reachable, optimal if optimal solution is reachable. Time can still be exponential.

10

Local search algorithms

In many optimization problems, the path to the goal is irrelevant; the goal state itself is the solution State space = set of "complete" configurations Find configuration satisfying constraints, e.g., n-queens In such cases, we can use local search algorithms Local search= use single current state and move to neighboring states. Advantages: Use very little memory Find often reasonable solutions in large or infinite state spaces. Are also useful for pure optimization problems. Find best state according to some objective function.
11

Local search and optimization

12

Example: n-queens
Put n queens on an n n board with no two queens on the same row, column, or diagonal

13

Hill-climbing search
Simple, general idea:
Start wherever Always choose the best neighbor If no neighbors have better scores than current, quit

Hill climbing does not look ahead of the immediate neighbors of the current state. Hill-climbing chooses randomly among the set of best successors, if there is more than one. Some problem spaces are great for hill climbing and others are terrible.
14

Hill-climbing search
function HILL-CLIMBING(problem) return a state that is a local maximum input: problem, a problem local variables: current, a node. neighbor, a node. current MAKE-NODE(INITIAL-STATE[problem]) loop do neighbor a highest valued successor of current if VALUE [neighbor] VALUE[current] then return STATE[current] current neighbor

15

Drawbacks of hill climbing

Problems:
Local Maxima: depending on initial state, can get stuck in local maxima Plateaus: the space has a broad flat region that gives the search algorithm no direction (random walk) Ridges: flat like a plateau, but with dropoffs to the sides; steps to the North, East, South and West may go down, but a combination of two steps (e.g. N, W) may go up

Introduce randomness

16

Hill-climbing variations
Stochastic hill-climbing
Random selection among the uphill moves. The selection probability can vary with the steepness of the uphill move.

First-choice hill-climbing
Stochastic hill climbing by generating successors randomly until a better one is found.

Random-restart hill-climbing
Tries to avoid getting stuck in local maxima. If at first you dont succeed, try, try again
17

Simulated Annealing
Simulates slow cooling of annealing process Applied for combinatorial optimization problem by S. Kirkpatric (83) What is annealing?
Process of slowly cooling down a compound or a substance Slow cooling let the substance flow around thermodynamic equilibrium Molecules get optimum conformation

18

Simulated annealing

gradually decrease shaking to make sure the ball escape from local minima and fall into the global minimum

19

Simulated annealing
Escape local maxima by allowing bad moves.
Idea: but gradually decrease their size and frequency.

Origin; metallurgical annealing Implement:

Randomly select a move instead of selecting best move Accept a bad move with probability less than 1 (p<1) p decreases by time

If T decreases slowly enough, best state is reached. Applied for VLSI layout, airline scheduling, etc.

20

Simulated annealing
function SIMULATED-ANNEALING( problem, schedule) return a solution state input: problem, a problem schedule, a mapping from time to temperature local variables: current, a node; next, a node. T, a temperature controlling the probability of downward steps current MAKE-NODE(INITIAL-STATE[problem]) Similar to hill climbing, for t 1 to do but a random move T schedule[t] instead of best move if T = 0 then return current next a randomly selected successor of current E VALUE[next] - VALUE[current] case of improvement, make the move if E > 0 then current next else current next only with probability eE /T
Whats the probability when: T inf? Whats the probability when: T 0? Whats the probability when: =0? Whats the probability when: -?

Otherwise, choose the move with probability that decreases exponentially with the badness of the move.

Simulated Annealing parameters

Temperature T
Used to determine the probability High T : large changes Low T : small changes

Cooling Schedule
Determines rate at which the temperature T is lowered Lowers T slowly enough, the algorithm will find a global optimum

In the beginning, aggressive for searching alternatives, become conservative when time goes by

22

Simulated Annealing Cooling Schedule

T0 T(t) Tf

if Ti is reduced too fast, poor quality if Tt >= T(0) / log(1+t) - Geman System will converge to minimun configuration Tt = k/1+t - Szu Tt = a T(t-1) where a is in between 0.8 and 0.99
23

Tips for Simulated Annealing

To avoid of entrainment in local minima
Annealing schedule : by trial and error
Choice of initial temperature How many iterations are performed at each temperature How much the temperature is decremented at each step as cooling proceeds

Difficulties
Determination of parameters If cooling is too slow Too much time to get solution If cooling is too rapid Solution may not be the global optimum

24

Properties of simulated annealing

Theoretical guarantee:
Stationary distribution:

p( x) e

E ( x) kT

If T decreased slowly enough, will converge to optimal state!

Is this an interesting guarantee? Sounds like magic, but :

The more downhill steps you need to escape, the less likely you are to every make them all in a row People think hard about ridge operators which let you jump around the space in better ways
25

Local beam search

Like greedy search, but keep K states at all times:
Initially: k random states Next: determine all successors of k states If any of successors is goal finished Else select k best from successors and repeat.

Greedy Search

Beam Search

26

Local beam search

Major difference with random-restart search
Information is shared among k search threads: If one state generated good successor, but others did not come here, the grass is greener!

Can suffer from lack of diversity.

Stochastic variant: choose k successors at proportionally to state success.

The best choice in MANY practical settings

27

Artificial Intelligence
For HEDSPI Project

Lecturer 6 - Advanced search methods

Lecturers : Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST

Outline
Game and search Alpha-beta pruning

Games and search

Why study games? Why is search a good idea? Majors assumptions about games:
Only an agents actions change the world World is deterministic and accessible

Why study games?

machines are better than humans in: othello humans are better than machines in: go here: perfect information zero-sum games
4

Why study games?

Games are a form of multi-agent environment What do other agents do and how do they affect our success? Cooperative vs. competitive multi-agent environments. Competitive multi-agent environments give rise to adversarial search a.k.a. games Why study games? Fun; historically entertaining Interesting subject of study because they are hard Easy to represent and agents restricted to small number of actions

Relation of Games to Search

Search no adversary
Solution is (heuristic) method for finding goal Heuristics and CSP techniques can find optimal solution Evaluation function: estimate of cost from start to goal through given node Examples: path planning, scheduling activities

Games adversary
Solution is strategy (strategy specifies move for every possible opponent reply). Time limits force an approximate solution Evaluation function: evaluate goodness of game position Examples: chess, checkers, Othello, backgammon

Ignoring computational complexity, games are a perfect application for a complete search. Of course, ignoring complexity is a bad idea, so games are a good place to study resource bounded searches.
6

Types of Games
deterministic perfect information imperfect information chance

chess, checkers, go, backgammon othello monopoly battleships, blind tictactoe bridge, poker, scrabble nuclear war

Minimax
Two players: MAX and MIN MAX moves first and they take turns until the game is over. Winner gets award, looser gets penalty. Games as search: Initial state: e.g. board configuration of chess Successor function: list of (move,state) pairs specifying legal moves. Terminal test: Is the game finished? Utility function: Gives numerical value of terminal states. E.g. win (+1), loose (-1) and draw (0) in tic-tac-toe MAX uses search tree to determine next move. Perfect play for deterministic games

Minimax

From among the moves available to you, take the best one The best one is determined by a search using the MiniMax strategy
9

Optimal strategies
Find the contingent strategy for MAX assuming an infallible MIN opponent. Assumption: Both players play optimally !! Given a game tree, the optimal strategy can be determined by using the minimax value of each node: MINIMAX-VALUE(n)= UTILITY(n) maxs successors(n) MINIMAX-VALUE(s) mins successors(n) MINIMAX-VALUE(s)

If n is a terminal If n is a max node If n is a min node

10

Minimax

11

Minimax algorithm

12

Properties of minimax
Complete? Yes (if tree is finite) Optimal? Yes (against an optimal opponent) Time complexity? O(bm) Space complexity? O(bm) (depth-first exploration) For chess, b 35, m 100 for "reasonable" games exact solution completely infeasible

13

Problem of minimax search

Number of games states is exponential to the number of moves.
Solution: Do not examine every node

Alpha-beta pruning:
Remove branches that do not influence final decision Revisit example

14

- pruning
Alpha values: the best values achievable for MAX, hence the max value so far Beta values: the best values achievable for MIN, hence the min value so far At MIN level: compare result V of node to alpha value. If V>alpha, pass value to parent node and BREAK At MAX level: compare result V of node to beta value. If V<beta, pass value to parent node and BREAK

15

- pruning example

16

- pruning example

17

- pruning example

18

- pruning example

19

- pruning example

20

Properties of -
Pruning does not affect final result Entire sub-trees can be pruned. Good move ordering improves effectiveness of pruning. With "perfect ordering"
time complexity = O(bm/2) doubles depth of search Branching factor of sqrt(b) !! Alpha-beta pruning can look twice as far as minimax in the same amount of time

Repeated states are again possible.

Store them in memory = transposition table

A simple example of the value of reasoning about which computations are relevant (a form of metareasoning)
21

Why is it called -?
is the value of the best (i.e., highestvalue) choice found so far at any choice point along the path for max If v is worse than , max will avoid it
prune that branch

Define similarly for min

22

The - algorithm

23

The - algorithm

24

Imperfect, real-time decisions

Minimax and alpha-beta pruning require too much leafnode evaluations. May be impractical within a reasonable amount of time. Suppose we have 100 secs, explore 104 nodes/sec 106 nodes per move Standard approach (SHANNON, 1950):
Cut off search earlier (replace TERMINAL-TEST by CUTOFFTEST) Apply heuristic evaluation function EVAL (replacing utility function of alpha-beta)

25

Cut-off search
Change:
if TERMINAL-TEST(state) then return UTILITY(state)

into:
if CUTOFF-TEST(state,depth) then return EVAL(state)

Introduces a fixed-depth limit depth

Is selected so that the amount of time will not exceed what the rules of the game allow.

When cut-off occurs, the evaluation is performed.

26

Heuristic evaluation (EVAL)

Idea: produce an estimate of the expected utility of the game from a given position. Performance depends on quality of EVAL. Requirements:
EVAL should order terminal-nodes in the same way as UTILITY. Computation may not take too long. For non-terminal states the EVAL should be strongly correlated with the actual chance of winning.

Only useful for quiescent (no wild swings in value in near future) states

27

Evaluation function example

For chess, typically linear weighted sum of features Eval(s) = w1 f1(s) + w2 f2(s) + + wn fn(s) e.g., w1 = 9 with f1(s) = (number of white queens) (number of black queens), etc.
28

Chess complexity
PC can search 200 millions nodes/3min. Branching factor: ~35
355 ~ 50 millions if use minimax, could look ahead 5 plies, defeated by average player, planning 6-8 plies.

Does it work in practice?

4-ply human novice hopeless chess player 8-ply typical PC, human master 12-ply Deep Blue, Kasparov

To reach grandmaster level, needs a better extensively tuned evaluation and a large database of optimal opening and ending of the game
29

Deterministic games in practice

Checkers: Chinook ended 40-year-reign of human world champion Marion Tinsley in 1994. Used a precomputed endgame database defining perfect play for all positions involving 8 or fewer pieces on the board, a total of 444 billion positions. Chess: Deep Blue defeated human world champion Garry Kasparov in a six-game match in 1997. Deep Blue searches 200 million positions per second, uses very sophisticated evaluation, and undisclosed methods for extending some lines of search up to 40 ply. Othello: human champions refuse to compete against computers, who are too good. Go: human champions refuse to compete against computers, who are too bad. In go, b > 300, so most programs use pattern knowledge bases to suggest plausible moves.
30

Nondeterministic games
Chance introduces by dice, card-shuffling, coin-flipping... Example with coin-flipping: change nodes

31

Backgammon

Possible moves: (5-10,5-11), (5-11,19-24),(5-10,10-16) and (5-11,11-16)

32

Expected minimax value

EXPECTED-MINIMAX-VALUE(n)= UTILITY(n) If n is a terminal maxssuccessors(n) EXPECTEDMINIMAX(s) If n is a max node minssuccessors(n) EXPECTEDMINIMAX(s) If n is a max node ssuccessors(n) P(s) .EXPECTEDMINIMAX(s) If n is a chance node P(s) is probability of s occurence
33

Games of imperfect information

E.g., card games, where opponent's initial cards are unknown Typically we can calculate a probability for each possible deal Seems just like having one big dice roll at the beginning of the game Idea: compute the minimax value of each action in each deal, then choose the action with highest expected value over all deals Special case: if an action is optimal for all deals, it's optimal. GIB, current best bridge program, approximates this idea by
generating 100 deals consistent with bidding information picking the action that wins most tricks on average

34

Artificial Intelligence
For HEDSPI Project

Lecturer 8 Constraint Satisfaction Problems

Lecturer: Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST

Constraints Satisfaction Problems (CSPs)

CSPs example Backtracking search Problem structure Local search for CSPs

CSP
Standard search problems
State is a black-box
Any data structure that implements initial states, goal states, successor function

CSPs
State is composed of variables Xi with value in domain Di Goal test is a set of constraints over variables

Example: Map Coloring

Variables
WA, NT, Q, NSW, V , SA

Domain
Di = {red, green, blue}

Constraint
Neighbor regions must have different colors
WA /= NT WA /= SA NT /= SA

Example: Map Coloring

Solution is an assignment of variables satisfying all constraints

WA=red, and NT=green, and Q=red, and NSW=green, and V=red, and SA=blue

Constraint Graph
Binary CSPs
Each constraint relates at most two variables

Constraint graph
Node is variable Edge is constraint

Varieties of CSPs
Discrete variables
Finite domain, e.g, SAT Solving Infinite domain, e.g., work scheduling
Variables is start/end of working day Constraint laguage, e.g., StartJob1 + 5 <= StartJob3 Linear constraints are decidable, non-linear constraints are undecidable

Continuous variables
e.g., start/end time of observing the universe using Hubble telescope Linear constraints are solvable using Linear Programming

Varieties of Constraints
Single-variable constraints
e.g., SA /= green

Binary constraints
e.g., SA /= WA

Multi-variable constraints
Relate at least 3 variables

Soft constraints
Priority, e.g., red better than green Cost function over variables

Example: Cryptarimetic
Variables
F,T,O,U,R,W, X1,X2,X3

TWO + TWO FOUR

Domain
{0,1,2,3,4,5,6,7, 8,9}

Constraints
Alldiff(F,T,O,U,R,W) O+O = R+10*X1 X1+W+W= U+10*X2 X2+T+T= O+10*X3 X3=F

Real World CSP

Assignment
E.g., who teach which class

Scheduling
E.g., when and where the class takes place

Hardware design Spreadsheets Transport scheduling Manufacture scheduling

CSPs by Standard Search

State
Defined by the values assigned so far

Initial state
The empty assignment

Successor function
Assign a value to a unassigned variable that does not conflict with current assignment
Fail if no legal assignment

Goal test
All variables are assigned and no conflict

CSP by Standard Search

Every solution appears at depth n with n variables
Use depth-first search

Path is irrelevant Number of leaves

n!dn
Two many

Backtracking Search
Variable assignments are commutative, e.g.,
{WA=red, NT =green} {NT =green, WA=red}

Single-variable assignment
Only consider one variable at each node dn leaves

Backtracking search
Depth-first search+ Single-variable assignment

Backtracking search is the basic uninformed algorithm for CSPs

Can solve n-Queen with n = 25

Backtracking Search Algorithm

Backtracking Search Algorithm

Improving Backtracking Search

Which variable should be assigned next? In what order should its values be tried? Can we detect inevitable failure early? Can we take advantage of problem structure?

Choosing Variables
Minimum remaining values (MRV)
Choose the variable with the fewest legal values

Degree heuristic
Choose the variable with the most constraints on remaining variables

Choosing Values
Least constraining value (LCV)
Choose the least constraining value
the one that rules out the fewest values in the remaining variables

Keep track of remaining legal values for unassigned variables

Terminate search when any variable has no legal values

Forward Checking
Constraint propagation

NT and SA cannot both be blue

Simplest form of propagation makes each arc consistent

X -> Y is consistent iff for each value x of X there is some allowed value y for Y

Problem Structure
Assume we have a new region T in addition T and the rest are two independent problems
Each problem is a connected component in the constraint graph

Problem Structure
Tree-structured problem

Theorem
If the constraint graph has no loop then CSPs can be solved in O(nd2) time

Problem Structure
Algorithm for tree-structured problems

Iterative Algorithms for CSPs

Hill-climbing, Simulated Annealing can be used for CSPs

Complete state, e.g., all variables are assigned at each node

Allow states with unsatisfiable constraints Operators reassign variables Variable selection
Random

Value selection by min-conflicts heuristic Choose value that violates the fewest constraints
i.e., hill climbing with h(n) = total number of violated constraints

Example: 4-Queens
State: 4 queens in four columns (4*4 = 256 states) Operators: move queen in column Goal test: no attacks Evaluation: h(n) = number of attacks

Summary
CSPs are a special kind of problem:
states defined by values of a fixed set of variables goal test defined by constraints on variable values

Backtracking = depth-first search with one variable assigned per node Variable ordering and value selection heuristics help significantly Forward checking prevents assignments that guarantee later failure Constraint propagation (e.g., arc consistency) does additional work to constrain values and detect inconsistencies The CSPs representation allows analysis of problem structure Tree-structured CSPs can be solved in linear time Iterative min-conflicts is usually effective in practice

Exercice
Solve the following cryptarithmetic problem by combining the heuristics
Constraint Propagation Minimum Remaining Values Least Constraining Values

TWO + TWO FOUR

Exercice
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.

Choose X3: domain {0,1} Choose X3=1: use constraint propagation F/=0 F=1 Choose X2: X1 and X2 c have the same remaing values Choose X2=0 Choose X1: X1 has MRV Choose X1=0 Choose O: O must be even, less than 5 and therefore has MRV (T+T=O d 1 v O+O=R+10*0) Choose O=4 R=8 T=7 Choose U: U must be even, less than 9 U=6: constraint propagation W=3

Artificial Intelligence
For HEDSPI Project

Lecturer 9 Propositional Logic

Lecturers : Tran Duc Khanh Dept of Information Systems School of Information and CommunicationTechnology HUST

Outline
What is Logic Propositional Logic
Syntax Semantic

Inference in Propositional Logic

Forward Chaining Backward Chaining

Knowledge-based Agents
Know about the world
They maintain a collection of facts (sentences) about the world, their Knowledge Base, expressed in some formal language.

Reason about the world

They are able to derive new facts from those in the KB using some inference mechanism.

Act upon the world

They map percepts to actions by querying and updating the KB.

What is Logic ?
A logic is a triplet <L,S,R>
L, the language of the logic, is a class of sentences described by a precise syntax, usually a formal grammar S, the logics semantic, describes the meaning of elements in L R, the logics inference system, consisting of derivation rules over L

Examples of logics:
Propositional, First Order, Higher Order, Temporal, Fuzzy, Modal, Linear,

Propositional Logic
Propositional Logic is about facts in the world that are either true or false, nothing else Propositional variables stand for basic facts Sentences are made of
propositional variables (A,B,), logical constants (TRUE, FALSE), and logical connectives (not,and,or,..)

The meaning of sentences ranges over the Boolean values {True, False}
Examples: Its sunny, John is married

Language of Propositional Logic

Symbols
Propositional variables: A,B,,P,Q, Logical constants: TRUE, FALSE Logical connectives:

, , , ,

Sentences
Each propositional variable is a sentence Each logical constant is a sentence If and are sentences then the following are sentences

( ), , , , ,

Formal Language of Propositional Logic

Symbols
Propositional variables: A,B,,P,Q, Logical constants: T, F Logical connectives: , , , ,

Formal Grammar
Sentence -> Asentence | Csentence Asentence -> TRUE | FALSE | A | B| Csentence -> (Sentence) | Sentence | Sentence Connective Sentence | | | | Connective ->

Semantic of Propositional Logic

The meaning of TRUE is always True, the meaning of FALSE is always False The meaning of a propositional variable is either True or False
depends on the interpretation
assignment of Boolean values to propositional variables

The meaning of a sentence is either True or False

depends on the interpretation

Semantic of Propositional Logic

True table
P Q Not P P and Q P or Q P implies Q True True False True P equiv Q True False False True

False False True True

False True False True

True True False False

False False False True

False True True True

Semantic of Propositional Logic

Satisfiability
A sentence is satisfiable if it is true under some interpretation Ex: P or H is satisfiable P and P is unsatisfiable (not satisfiable)

Validity
A sentence is valid if it is true in every interpretation Ex: ((P or H) and A ) => P is valid P or H is not valid

Semantic of Propositional Logic

Entailment
Given
A set of sentences A sentence

We write

We said that entails

if and only if every interpretation that makes all sentences in true also makes true

Semantic of Propositional Logic

Satisfiability vs. Validity vs. Entailment

is valid iff True (also written ) is unsatisfiable iff False iff { } is unsatisfiable

Inference in Propositional Logic

Forward Chaining Backward Chaining

Forward Chaining
Given a set of rules, i.e. formulae of the form p1 p2 ... pn q and a set of known facts, i.e., formulae of the form q, r, A new fact p is added Find all rules that have p as a premise If the other premises are already known to hold then
add the consequent to the set of know facts, and trigger further inferences

Forward Chaining
Example

Forward Chaining

Forward Chaining

Forward Chaining

Forward Chaining

Forward Chaining

Forward Chaining

Forward Chaining
Soundness
Yes

Completeness
Yes

Backward Chaining
Given a set of rules, and a set of known facts We ask whether a fact P is a consequence of the set of rules and the set of known facts The procedure check whether P is in the set of known facts Otherwise find all rules that have P as a consequent
If the premise is a conjunction, then process the conjunction conjunct by conjunct

Backward Chaining
Example

Backward Chaining

25

Backward Chaining

26

Backward Chaining

27

Backward Chaining

28

Backward Chaining
Soundness
Yes

Completeness
Yes

Artificial Intelligence
For HEDSPI Project

Lecturer 10 First Order Logic

Lecturer: Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST

First Order Logic

Syntax Semantic Inference
Resolution

First Order Logic (FOL)

First Order Logic is about
Objects Relations Facts

The world is made of objects

Objects are things with individual identities and properties to distinguish them Various relations hold among objects. Some of these relations are functional Every fact involving objects and their relations are either true or false

FOL
Syntax Semantic Inference
Resolution

FOL Syntax
Symbols
Variables: x, y, z, Constants: a, b, c, Function symbols (with arities): f, g, h, Relation symbols (with arities): p, r, r Logical connectives: ,,, , Quantifiers: ,

FOL Syntax
Variables, constants and function symbols are used to build terms
X, Bill, FatherOf(X),

Relations and terms are used to build predicates

Tall(FatherOf(Bill)), Odd(X), Married(Tom,Marry), Loves(Y,MotherOf(Y)),

Predicates and logical connective are used to build sentences

Even(4),

X. Even(X) Odd(X+1), X. X > 0

FOL Syntax
Terms
Variables are terms Constants are terms If t1,, tn are terms and f is a function symbol with arity n then f(t1,, tn ) is a term

FOL Syntax
Predicates
If t1,, tn are terms and p is a relation symbol with arity n then p(t1,, tn ) is a predicate

FOL Syntax
Sentences
True, False are sentences Predicates are sentences If , are sentences then the followings are sentences

x. , x. , ( ), , , , ,

FOL Formal grammar

FOL
Syntax Semantic Inference
Resolution

FOL Semantic
Variables
Objects

Constants
Entities

Function symbol
Function from objects to objects

Relation symbol
Relation between objects

Quantifiers

x.P true if P is true under every value of x

Logical connectives
Similar to Propositional Logic

x.P true if P is true under some value of x

FOL Semantic
Interpretation (D, )
D is a set of objects, called domain or universe is a mapping from variables to D CD is a member of D for each constant C FD is a mapping form Dn to D for each function symbol F with arity n RD is a relation over Dn for each relation symbol R with arity n

FOL Semantic
Given an interpretation (D, ), semantic of a term/sentence is denoted

[ ]

D a

Interpretation of terms

FOL Semantic
Interpretation of sentence

Example
Symbols
Variables: x,y,z, Constants: 0,1,2, Function symbols: +,* Relation symbols: >, =

Semantic
Universe: N (natural numbers) The meaning of symbols Constants: the meaning of 0 is the number zero, Function symbols: the meaning of + is the natural number addition, Relation symbols: the meaning of > is the relation greater than,

FOL Semantic
Satisfiability Model
A sentence is satisfiable if it is true under some interpretation (D, )

An interpretation (D, ) is a model of a sentence if true under (D, ) Then we write (D, ) |=

is

A sentence is valid if every interpretation is its mode A sentence is valid in D if (D, ) |= for all A sentence is unsatisfiable if it has no model

Example
Consider the universe N of natural numbers x.x + 1 > 5 is satisfiable
x.x + 1 > 0 is valid is N x.2 x + 1 = 6 is unsatisfiable

Artificial Intelligence
For HEDSPI Project

Lecturer 11 Inference in First Order Logic

Lecturers : Le Thanh Huong Tran Duc Khanh Dept of Information Systems Faculty of Information Technology - HUT

First Order Logic

Syntax Semantic Inference
Resolution

Inference in FOL
Difficulties
Quantifiers Infinite sets of terms Infinite sets of sentences

Examples: x.King ( x) Greedy( x) Evil ( x)

Infinite set of instances
King ( Bill ) Greedy( Bill ) Evil ( Bill ) King ( FatherOf ( Bill )) Greedy( FatherOf ( Bill )) Evil ( FatherOf ( Bill )) ...

Robinsons Resolution
Herbrands Theorem (~1930)
A set of sentences S is unsatisfiable if and only there exists a finite subset Sg of the set of all ground instances Gr(S), which is unsatisfiabe

Herbrand showed that there is a procedure to demonstrate the unsatisfiability of a unsatisfiable set of sentences Robinson propose the Resolution procedure (~1950)

Idea of Resolution
Refutation-based procedure S |= A if and only if S {A} is unsatisfible Resolution procedure
Transform S {A} into a set of clauses Apply Resolution rule to find a the empty calause (contradiction) If the empty clause is found
Conclude S |= A

Otherwise
No conclusion

Clause
A clause is a disjunction of literals, i.e., has the form

P P2 ... Pn 1
Example

Pi []Ri

P ( x) Q( x, a ) R(b) P ( y ) Q(b, y ) R( y )
The empty clause corresponds to a contradiction Any sentence can be transformed to an equi-satisfiable set of clauses

Elements of Resolution
Resolution rule Unification Transform a sentence to a set of clauses

Elements of Resolution
Resolution rule Unification Transform a sentence to a set of clauses

Resolution rule
Resolution rule

A B

C D ( A D)

= mgu ( B, C )

mgu: most general unifier The most general assignment of variables to terms in such a way that two terms are equal Syntactical unification algorithm

Example of Resolution rule

x, y are variables a, b are constants
Q(b, y ) R( y ) P ( x ) Q ( x, a ) P (b) R (a ) A P( x) B Q ( x, a ) C Q(b, y ) D R( y )

= {x = b, y = a}

Example of Resolution rule

Elements of Resolution
Resolution rule Unification Transform a sentence to a set of clauses

Unification
Input
Set of equalities between two terms

Output
Most general assignment of variables that satisfies all equalities Fail if no such assignment exists

Unification algorithm

Vars(U), Vars(t) are sets of variables in U and t v is a variable s and t are terms f and g are function symbols

Example of Unification

Elements of Resolution
Resolution rule Unification Transform a sentence to a set of clauses

Transform a sentence to a set of clauses

1. 2. 3. 4. 5. 6. 7. 8. 9.

Eliminate implication Move negation inward Standardize variable scope Move quantifiers outward Skolemize existential quantifiers Eliminate universal quantifiers Distribute and, or Flatten and, or Eliminate and

Eliminate implication

Move negation inward

Standardize variable scope

Each variable for each quantifier

Move quantifiers outward

Skolemize existential quantifiers

Eliminate universal quantifiers

Distribute and, or

Flatten and, or

Eliminate and

Example of proof by Resolution

Prove Criminal(West) Success

Summary of Resolution
Refutation-based procedure S |= A if and only if S {A} is unsatisfiable Resolution procedure
Transform S {A} into a set of clauses Apply Resolution rule to find a the empty clause (contradiction) If the empty clause is found
Conclude S |= A

Otherwise
No conclusion

Summary of Resolution
Theorem
A set of clauses S is unsatisfiable if and only if upon the input S, Resolution procedure finds the empty clause (after a finite time).

Exercice
The law says that it is a crime for an American to sell weapons to hostile nations The country Nono, an enemy of America, has some missiles, and all of its missiles were sold to it by Colonel West, who is American Is West a criminal?

Modeling

Transform the problem to set of clauses and Resolution

Prove Criminal(West) Successful

Exercice
Jack owns a dog Every dog owner is an animal lover No animal lover kills an animal Either Jack or Curiosity killed the cat, who is named Tuna Did Curiosity kill the cat?

Modeling
x.Dog ( x) Owns ( Jack , x) xy.( Dog ( y ) Owns ( x, y )) AnimalLover ( x) xy.( AnimalLover ( x) Animal ( y ) Kills ( x, y )) Kills ( Jack , Tuna ) Kill (Curiosity , Tuna ) Cat (Tuna ) x.Cat ( x) Animal ( x)

Transform the problem to set of clauses

Dog ( D) Owns ( Jack , D) Dog ( y ) Owns ( x, y ) AnimalLover ( x) AnimalLover ( x) Animal ( y ) Kills ( x, y ) Kills ( Jack , Tuna ) Kill (Curiosity , Tuna ) Cat (Tuna ) Cat ( x) Animal ( x)

Artificial Intelligence
For HEDSPI Project

Lecturer 12 - Planning
Lecturer : Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST

Outline
Planning problem State-space search Partial-order planning Planning graphs Planning with propositional logic

Search vs. planning

Consider the task get milk, bananas, and a cordless drill Standard search algorithms seem to fail miserably:

Planning problem
Planning is the task of determining a sequence of actions that will achieve a goal. Domain independent heuristics and strategies must be based on a domain independent representation
General planning algorithms require a way to represent states, actions and goals STRIPS, ADL, PDDL are languages based on propositional or first-order logic

Classical planning environment: fully observable, deterministic, finite, static and discrete.

Additional complexities
Because the world is
Dynamic Stochastic Partially observable

And because actions

take time have continuous effects

AI Planning background
Focus on classical planning; assume none of the above Deterministic, static, fully observable
Basic Most of the recent progress Ideas often also useful for more complex problems

Problem Representation
State What is true about the (hypothesized) world? Goal What must be true in the final state of the world? Actions What can be done to change the world? Preconditions and effects Well represent all these as logical predicates

STRIPS operators
Tidily arranged actions descriptions, restricted language Action: Buy(x) Precondition: At(p); Sells(p; x) Effect: Have(x) [Note: this abstracts away many important details!] Restricted language efficient algorithm Precondition: conjunction of positive literals Effect: conjunction of literals A complete set of STRIPS operators can be translated into a set of successor-state axioms

Example: blocks world

On(b, x ): block b is on x, x is another block or Table. MoveToTable(b, x): move block b from the top of x to Table Move(b, x, y): move block b from the top of x to the top of y Clear(x): nothing is on x Action (Move (b, x, y) , PRECOND: On(b, x) Clear(b) Clear(y), EFFECT: On(b, y) Clear(x) On(b, x) Clear(y)) . Action(MoveToTable(b, x) , PRECOND: On(b, x) Clear(b), EFFECT: On(b, Table) Clear(x) On(b, x )) . Initial state: On(A,Table), On(C,A), On(B,Table), Clear(B), Clear(C) Goal: On(A,B), On(B,C)
9

Initial C A B

Goal A B C

Planning with state-space search

Both forward and backward search possible Progression planners

forward state-space search consider the effect of all possible actions in a given state Regression planners backward state-space search Determine what must have been true in the previous state in order to achieve the current state

10

Progression and regression

initial state

goal

11

Progression algorithm
Formulation as state-space search problem: Initial state and goal test: obvious Successor function: generate from applicable actions Step cost = each action costs 1 Any complete graph search algorithm is a complete planning algorithm. E.g. A* Inherently inefficient: (1) irrelevant actions lead to very broad search tree (2) good heuristic required for efficient search

12

Forward Search Methods: can use A* with some h and g

13

Regression algorithm
How to determine predecessors?
What are the states from which applying a given action leads to the goal? Goal state = At(C1, B) At(C2, B) At(C20, B) Relevant action for first conjunct: Unload(C1,p,B) Works only if pre-conditions are satisfied. Previous state= In(C1, p) At(p, B) At(C2, B) At(C20, B) Subgoal At(C1,B) should not be present in this state.

Actions must not undo desired literals (consistent) Main advantage: only relevant actions are considered.
Often much lower branching factor than forward search.

14

Regression algorithm
General process for predecessor construction Give a goal description G Let A be an action that is relevant and consistent The predecessors are as follows: Any positive effects of A that appear in G are deleted. Each precondition literal of A is added , unless it already appears. Any standard search algorithm can be added to perform the search. Termination when predecessor satisfied by initial state. In FO case, satisfaction might require a substitution.

15

Backward search methods

Regressing a ground operator

16

Regressing an ungrounded operator

17

Example of Backward Search

Heuristics for state-space search

Use relax problem idea to get lower bounds on least number of actions to the goal.
Remove all or some preconditions

Subgoal independence: the cost of solving a set of subgoals equal the sum cost of solving each one independently.
Can be pesimistic (interacting subplans) Can be optimistic (negative effects)

Simple: number of unsatisfied subgoals. Various ideas related to removing negative effects or positive effects.
19

Partial order planning

Least commitment planning Nonlinear planning Search in the space of partial plans A state is a partial incomplete partially ordered plan Operators transform plans to other plans by:
Adding steps Reordering Grounding variables

SNLP: Systematic Nonlinear Planning (McAllester and Rosenblitt 1991) NONLIN (Tate 1977)
20

A partial order plan for putting shoes and socks

21

Partial-order planning
Partially ordered collection of steps with
Start step has the initial state description as its effect Finish step has the goal description as its precondition causal links from outcome of one step to precondition of another temporal ordering between pairs of steps

Open condition = precondition of a step not yet causally linked A plan is complete iff every precondition is achieved A precondition is achieved iff it is the effect of an earlier step and no possibly intervening step undoes it
22

Example

Example

Example

Planning process
Operators on partial plans:
add a link from an existing action to an open condition add a step to fulfill an open condition order one step wrt another to remove possible conflicts

Gradually move from incomplete/vague plans to complete, correct plans Backtrack if an open condition is unachievable or if a conflict is unresolvable
26

POP algorithm sketch

function POP(initial, goal, operators) returns plan plan Make-Minimal-Plan(initial, goal) loop do if Solution?( plan) then return plan Sneed , c Select-Subgoal( plan) Choose-Operator( plan, operators,Sneed, c) Resolve-Threats( plan) end function Select-Subgoal( plan) returns Sneed , c pick a plan step Sneed from Steps( plan) with a precondition c that has not been achieved return Sneed , c
27

POP algorithm (cont)

procedure Choose-Operator(plan, operators,Sneed, c) choose a step Sadd from operators or Steps( plan) that has c as as an effect if there is no such step then fail add the causal link Sadd c Sneed to Links( plan) add the ordering constraint Sadd < Sneed to Orderings( plan) if Sadd is a newly added step from operators then add Sadd to Steps( plan) add Start < Sadd < Finish to Orderings( plan) procedure Resolve-Threats(plan) for each Sthreat that threatens a link Si c Sj in Links( plan) do choose either Demotion: Add Sthreat < Si to Orderings( plan) Promotion: Add Sj < Sthreat to Orderings( plan) if not Consistent( plan) then fail end

Clobbering and promotion/demotion

A clobberer is a potentially intervening step that destroys the condition achieved by a causal link. E.g., Go(Home) clobbers At(Supemarket):

Demotion: put before Go(Supermarket)

Promotion: put after Buy(Milk)

29

Properties of POP
Nondeterministic algorithm: backtracks at choice points on failure choice of Sadd to achieve Sneed choice of demotion or promotion for clobberer selection of Sneed is irrevocable POP is sound, complete, and systematic (no repetition) Extensions for disjunction, universals, negation, conditionals Can be made effcient with good heuristics derived from problem description Particularly good for problems with many loosely related subgoals

30

Example: Blocks world

31

Example: Blocks world

32

Example: Blocks world

33

Example: Blocks world

34

Example: Blocks world

35

Planning Graphs
A planning graph consists of a sequence of levels that correspond to time-steps in the plan Level 0 is the initial state. Each level contains a set of literals and a set of actions Literals are those that could be true at the time step. Actions are those that their preconditions could be satisfied at the time step. Works only for propositional planning.

36

Example:Have cake and eat it too

37

The Planning graphs for have cake,

Persistence actions: Represent inactions by boxes: frame axiom Mutual exclusions (mutex) are represented between literals and actions. S1 represents multiple states Continue until two levels are identical. The graph levels off. The graph records the impossibility of certain choices using mutex links. Complexity of graph generation: polynomial in number of literals.

38

Defining Mutex relations

A mutex relation holds between two actions on the same level iff any of the following holds:
Inconsistency effect:one action negates the effect of another. Example eat cake and presistence of have cake Interference: One of the effect of one action is the negation of the precondition of the other. Example: eat cake and persistence of Have cake Competing needs: one of the preconditions of one action is mutually exclusive with a precondition of another. Example: Bake(cake) and Eat(Cake). A mutex relation holds between 2 literals at the same level iff one is the negation of the other or if each possible pair of actions that can achieve the 2 literals is mutually exclusive.
39

Planning graphs for heuristic estimation

Estimate the cost of achieving a goal by the level in the planning graph where it appears. To estimate the cost of a conjunction of goals use one of the following:
Max-level: take the maximum level of any goal (admissible) Sum-cost: Take the sum of levels (inadmissible) Set-level: find the level where they all appear without Mutex

Graph plans are relaxation of the problem. Representing more than pair-wise mutex is not costeffective

40

The graphplan algorithm

41

Planning graph for spare tire a S2

goal: at(spare,axle)
S2 has all goals and no mutex so we can try to extract solutions Use either CSP algorithm with actions as variables Or search backwards

Search planning-graph backwards with heuristics

How to choose an action during backwards search: Use greedy algorithm based on the level cost of the literals. For any set of goals: 1. Pick first the literal with the highest level cost. 2. To achieve the literal, choose the action with the easiest preconditions first (based on sum or max level of precond literals).

43

Properties of planning graphs; termination

Literals increase monotonically
Once a literal is in a level it will persist to the next level

Actions increase monotonically

Since the precondition of an action was satisfied at a level and literals persist the actions precond will be satisfied from now on

Mutexes decrease monotonically:

If two actions are mutex at level Si, they will be mutex at all previous levels at which they both appear

Because literals increase and mutex decrease it is guaranteed that we will have a level where all goals are non-mutex
44

Planning with propositional logic

Express propositional planning as a set of propositions. Index propositions with time steps: On(A,B)_0, ON(B,C)_0 Goal conditions: the goal conjuncts at time T, T is determined arbitrarily. Unknown propositions are not stated. Propositions known not to be true are stated negatively. Actions: a proposition for each action for each time slot. Succesor state axioms need to be expressed for each action (like in the situation calculus but it is propositional)
45

Planning with propositional logic (continued)

We write the formula:
Initial state and successor state axioms and goal

We search for a model to the formula. Those actions that are assigned true constitute a plan. To have a single plan we may have a mutual exclusion for all actions in the same time slot. We can also choose to allow partial order plans and only write exclusions between actions that interfere with each other. Planning: iteratively try to find longer and longer plans.

46

SATplan algorithm

47

Complexity of satplan
The total number of action symbols is: |T|x|Act|x|O|^p O = number of objects, p is scope of atoms. Number of clauses is higher. Example: 10 time steps, 12 planes, 30 airports, the complete action exclusion axiom has 583 million clauses.

48

Artificial Intelligence
For HEDSPI Project

Lecturer 13 Machine Learning

Lecturer : Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST

Introduction of Machine learning

Definitions of Machine learning
A process by which a system improves its performance [Simon, 1983] Any computer program that improves its performance at some task through experience [Mitchell, 1997] Programming computers to optimize a performance criterion using example data or past experience [Alpaydin, 2004]

Representation of the learning problem [Mitchell, 1997]

Learning = Improving with experience at some task
Improve over task T With respect to performance measure P Based on experience E
2

Application examples of ML (1)

Web pages filtering problem
T: to predict which Web pages a given user is interested in P: % of Web pages correctly predicted E: a set of Web pages identified as interested/uninterested for the user

Web pages categorization problem

T: to categorize Web pages in predefined categories P: % of Web pages correctly categorized E: a set of Web pages with specified categories

Which cat.? Interested?

Application examples of ML (2)

Handwriting recognition problem
T: to recognize and classify handwritten words within images P: % of words correctly classified E: a database of handwritten words with given classifications (i.e., labels)

Robot driving problem

T: to drive on public highways using vision sensors P: average distance traveled before an error (as judged by human overseer) E: a sequence of images and steering commands recorded while observing a human driver

Which word?

Which steering command? way Go straight

4

we

do

in

the

right

Move left

Move Slow Speed right down up

Key elements of a ML problem (1)

Selection of the training examples
Direct or indirect training feedback With teacher (i.e., with labels) or without The training examples set should be representative of the future test examples

Choosing the target function (a.k.a. hypothesis, concept, etc.)

F: X {0,1} F: X a set of labels F: X R+ (i.e., the positive real numbers domain)

Key elements of a ML problem (2)

Choosing a representation of the target function
A polynomial function A set of rules A decision tree A neural network

Choosing a learning algorithm that learns (approximately) the target function

Regression-based Rule induction ID3 or C4.5 Back-propagation
6

Issues in Machine Learning (1)

Learning algorithm
What algorithms can approximate the target function? Under which conditions does a selected algorithm converge (approximately) to the target function? For a certain problem domain and given a representation of examples which algorithm performs best?

Training examples
How many training examples are sufficient? How does the size of the training set influence the accuracy of the learned target function? How does noise and/or missing-value data influence the accuracy?

Issues in Machine Learning (2)

Learning process
What is the best strategy for selecting a next training example? How do selection strategies alter the complexity of the learning problem? How can prior knowledge (held by the system) help?

Learning capability
What target function should the system learn?
Representation of the target function: expressiveness vs. complexity

What are the theoretical limits of learnability? How can the system generalize from the training examples?
To avoid the overfitting problem

How can the system automatically alter its representation?

To improve its ability to represent and learn the target function

Artificial Intelligence
For HEDSPI Project

Lecturer 13 Decision Tree Learning

Lecturer : Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST

Decision tree Introduction

Decision tree (DT) learning
To approximate a discrete-valued target function The target function is represented by a decision tree

A DT can be represented (interpreted) as a set of IF-THEN rules (i.e., easy to read and understand) Capable of learning disjunctive expressions DT learning is robust to noisy data One of the most widely used methods for inductive inference Successfully applied to a range of real-world applications
2

Example of a DT: Which documents are of my interest?

sport?
is present is absent

player?
is present is absent

football?
is present is absent

Interested

Uninterested

Interested
is present

goal?
is absent

Interested

Uninterested

(,sport,,player,) (,goal,) (,sport,)

Interested Interested Uninterested

3

Example of a DT: Does a person play tennis?

Outlook=?
Sunny Overcast Rain

Humidity=?
High Normal

Yes
Strong

Wind=?
Weak

No

Yes

No

Yes

(Outlook=Overcast, Temperature=Hot, Humidity=High, Wind=Weak) Yes (Outlook=Rain, Temperature=Mild, Humidity=High, Wind=Strong) No (Outlook=Sunny, Temperature=Hot, Humidity=High, Wind=Strong) No
4

Decision tree Representation (1)

Each internal node represents an attribute to be tested by instances Each branch from a node corresponds to a possible value of the attribute associated with that node Each leaf node represents a classification (e.g., a class label) A learned DT classifies an instance by sorting it down the tree, from the root to some leaf node
The classification associated with the leaf node is used for the instance

Decision tree Representation (2)

A DT represents a disjunction of conjunctions of constraints on the attribute values of instances Each path from the root to a leaf corresponds to a conjunction of attribute tests The tree itself is a disjunction of these conjunctions Examples
Lets consider the two previous example DTs

Which documents are of my interest?

sport?
is present is absent

player?
is present is absent

football?
is present is absent

Interested

Uninterested

Interested
is present

goal?
is absent

Interested

Uninterested

[(sport is present) [(sport is absent) [(sport is absent)

(player is present)] (football is present)] (football is absent) (goal is present)]

7

Does a person play tennis?

Outlook=?
Sunny Overcast Rain

Humidity=?
High Normal

Yes
Strong

Wind=?
Weak

No

Yes

No

Yes

[(Outlook=Sunny) (Humidity=Normal)] (Outlook=Overcast) [(Outlook=Rain) (Wind=Weak)]

Decision tree learning ID3 algorithm

ID3_alg(Training_Set, Class_Labels, Attributes)
Create a node Root for the tree If all instances in Training_Set have the same class label c, Return the tree of the single-node Root associated with class label c If the set Attributes is empty, Return the tree of the single-node Root associated with class label Majority_Class_Label(Training_Set) A The attribute in Attributes that best classifies Training_Set The test attribute for node Root A For each possible value v of attribute A Add a new tree branch under Root, corresponding to the test: value of attribute A is v Compute Training_Setv = {instance x | x Training_Set, xA=v} If (Training_Setv is empty) Then Create a leaf node with class label Majority_Class_Label(Training_Set) Attach the leaf node to the new branch Else Attach to the new branch the sub-tree ID3_alg(Training_Setv, Class_Labels, {Attributes \ A}) Return Root
9

ID3 algorithm Intuitive idea

Perform a greedy search through the space of possible DTs Construct (i.e., learn) a DT in a top-down fashion, starting from its root node At each node, the test attribute is the one (of the candidate attributes) that best classifies the training instances associated with the node A descendant (sub-tree) of the node is created for each possible value of the test attribute, and the training instances are sorted to the appropriate descendant node Every attribute can appear at most once along any path of the tree The tree growing process continues
Until the (learned) DT perfectly classifies the training instances, or Until all the attributes have been used

10

Selection of the test attribute

A very important task in DT learning: at each node, how to choose the test attribute? To select the attribute that is most useful for classifying the training instances associated with the node How to measure an attributes capability of separating the training instances according to their target classification
Use a statistical measure Information Gain

Example: A two-class (c1, c2) classification problem

Which attribute, A1 or A2, should be chosen to be the test attribute? A1=? (c1: 35, c2: 25)
v11 c1: 21 c2: 9 v12 c1: 5 c2: 5 v13 c1: 9 c2: 11
11

A2=? (c1: 35, c2: 25)

v21 c1: 27 c2: 6 v22 c1: 8 c2: 19

Entropy
A commonly used measure in the Information Theory field To measure the impurity (inhomogeneity) of a set of instances The entropy of a set S relative to a c-class classification

Entropy ( S ) = pi . log 2 pi
i =1

where pi is the proportion of instances in S belonging to class i, and 0.log20=0

The entropy of a set S relative to a two-class classification Entropy(S) = -p1.log2p1 p2.log2p2 Interpretation of entropy (in the Information Theory field)
The entropy of S specifies the expected number of bits needed to encode class of a member randomly drawn out of S Optical length code assigns log2p bits to message having probability p The expected number of bits needed to encode a class: p.log2p
12

Entropy Two-class example

Entropy(S)

S contains 14 instances, where 9 belongs to class c1 and 5 to class c2 The entropy of S relative to the two-class classification:
Entropy(S) = -(9/14).log2(9/14)(5/14).log2(5/14) 0.94

0.5

Entropy =0, if all the instances belong to the same class (either c1 or c2)
Need 0 bit for encoding (no message need be sent)

0.5

p1

Entropy =1, if the set contains equal numbers of c1 and c2 instances

Need 1 bit per message for encoding (whether c1 or c2)

Entropy = some value in (0,1), if the set contains unequal numbers of c1 and c2 instances
Need on average <1 bit per message for encoding
13

Information gain
Information gain of an attribute relative to a set of instances is
the expected reduction in entropy caused by partitioning the instances according to the attribute

Information gain of attribute A relative to set S | Sv | Gain( S , A) = Entropy ( S ) Entropy ( S v ) vValues ( A ) | S |

where Values(A) is the set of possible values of attribute A, and Sv = {x | xS, xA=v}

In the above formula, the second term is the expected value of the entropy after S is partitioned by the values of attribute A Interpretation of Gain(S,A): The number of bits saved (reduced) for encoding class of a randomly drawn member of S, by knowing the value of attribute A
14

Training set - Example

Lets consider the following dataset (of a person) S:
Day D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 Outlook Sunny Sunny Overcast Rain Rain Rain Overcast Sunny Sunny Rain Sunny Overcast Overcast Rain Temperature Hot Hot Hot Mild Cool Cool Cool Mild Cool Mild Mild Mild Hot Mild
15

Humidity High High High High Normal Normal Normal High Normal Normal Normal High Normal High

Wind Weak Strong Weak Weak Weak Strong Strong Weak Weak Weak Strong Strong Weak Strong

Play Tennis No No Yes Yes Yes No Yes No Yes Yes Yes Yes Yes No

[Mitchell, 1997]

Information gain Example

What is the information gain of attribute Wind relative to the training set S Gain(S,Wind)? Attribute Wind have two possible values: Weak and Strong S = {9 positive and 5 negative instances} Sweak = {6 pos. and 2 neg. instances having Wind=Weak} Sstrong = { 3 pos. and 3 neg. instances having Wind=Strong}
Gain( S , Wind ) = Entropy ( S ) | Sv | Strong} | S | Entropy (Sv ) v{Weak ,

= Entropy ( S ) (8 / 14).Entropy ( SWeak ) (6 / 14).Entropy ( S Strong )

= 0.94 (8 / 14).(0.81) (6 / 14).(1) = 0.048

16

Decision tree learning Example (1)

At the root node, which attribute of {Outlook, Temperature, Humidity, Wind} should be the test attribute?
Gain(S, Gain(S, Gain(S, Gain(S, Outlook) = ... = 0.246 Temperature) = ... = 0.029 Humidity) = ... = 0.151 Wind) = ... = 0.048 The highest IG value

So, Outlook is chosen as the test attribute for the root node!
Outlook=?
Sunny Overcast S={9+, 5-} Rain

Node1
SSunny={2+, 3-}

Yes
SOvercast={4+, 0-}

Node2
SRain={3+, 2-}

17

Decision tree learning Example (2)

At Node1, which attribute of {Temperature, Humidity, Wind} should be the test attribute?
Note! Attribute Outlook is excluded, since it has been used by Node1s parent (i.e., the root node)
Gain(SSunny, Temperature) =...= 0.57 Gain(SSunny, Humidity) = ... = 0.97 Gain(SSunny, Wind) = ... = 0.019

Outlook=? S={9+, 5-}

Sunny SSunny= {2+, 3-} Overcast Rain

Humidity=?

Yes
SOvercast= {4+, 0-} Normal

Node2
SRain= {3+, 2-}

High

Node3
SHigh= {0+, 3-}

Node4
SNormal= {2+, 0-}

So, Humidity is chosen as the test attribute for Node1!

18

DT learning Hypothesis space search (1)

ID3 searches in a space of hypotheses (i.e., of possible DTs) for one that fits the training instances ID3 performs a simple-to-complex, hill-climbing search, beginning with the empty tree The hill-climbing search is guided by an evaluation metric the information gain measure ID3 searches only one (rather than all possible) DT consistent with the training instances

19

DT learning Hypothesis space search (2)

ID3 does not performs backtracking in its search
Guaranteed to converge to a locally (but not the globally) optimal solution Once an attribute is selected as the test for a node, ID3 never backtracks to reconsider this choice

At each step in the search, ID3 uses a statistical measure of all the instances (i.e., information gain) to refine its current hypothesis
The resulting search is much less sensitive to errors in individual training instances

20

Inductive bias in DT learning (1)

Both the two DTs below are consistent with the given training dataset So, which one is preferred (i.e., selected) by the ID3 algorithm?
Outlook=?
Sunny Overcast Rain Sunny

Outlook=?
Overcast Rain

Humidity=?
High Normal

Yes

Wind=?
Weak

Temperature=?
Hot Cool Mild

Yes

Wind=?
Strong Weak

Strong

No

Yes

No

Yes

No

Yes

Humidity=?
High Normal

No

Yes

No
21

Yes

Inductive bias in DT learning (2)

Given a set of training instances, there may be many DTs consistent with these training instances So, which of these candidate DTs should be chosen? ID3 chooses the first acceptable DT it encounters in its simple-to-complex, hill-climbing search
Recall that ID3 searches incompletely through the hypothesis space (i.e., without backtracking)

ID3s search strategy

Select in favor of shorter trees over longer ones Select trees that place the attributes with highest information gain closest to the root node

22

Issues in DT learning
Over-fitting the training data Handling continuous-valued (i.e., real-valued) attributes Choosing appropriate measures for attribute selection Handling training data with missing attribute values Handling attributes with differing costs An extension of the ID3 algorithm with the above mentioned issues resolved results in the C4.5 algorithm

23

Artificial Intelligence
For HEDSPI Project

Lecturer 14 Reinforcement Learning

Lecturer : Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST

Reinforcement Learning (RL)

RL is ML method that optimize the reward
A class of tasks A process of trial-and-error learning
Good actions are rewarded Bad actions are punished

Features of RL
Learning from numerical rewards Interaction with the task; sequences of states, actions and rewards Uncertainty and non-deterministic worlds Delayed consequences The explore/exploit dilemma The whole problem of goal-directed learning

Points of view
From the point of view of agents
RL is a process of trial-and-error learning How much reward will I get if I do this action?

From the point of view of trainers

RL is training by rewards and punishments Train computers like we train animals

Applications of RL
Robot Animal training Scheduling Games Control systems

Supervised Learning vs. Reinforcement Learning

Supervised learning Teacher: Is this an AI course or a Math course? Leaner: Math Teacher: No, AI Teacher: Is this an AI course or a Math course? Learner: AI Teacher: Yes Reinforcement learning World: You are in state 9. Choose action A or B Leaner: A World: Your reward is 100 World: You are in state 15. Choose action C or D Learner: D World : Your reward is 50

Examples
Chess
Win +1, loose -1

Elevator dispatching
reward based on mean squared time for elevator to arrive (optimization problem)

Channel allocation for cellular phones

Lower rewards the more calls are blocked

Policy, Reward and Goal

Policy
defines the agents behaviour at a given time maps from perceptions to actions can be defined by: look-up table, neural net, search algorithm... may be stochastic

Reward Function
defines the goal(s) in an RL problem maps from states, state-action pairs, or state-action-successor state, triplets to a numerical reward goal of the agent is to maximise the total reward in the long run the policy is altered to achieve this goal

Reward and Return

The reward function indicates how good things are right now But the agent wants to maximize reward in the long-term i.e. over many time steps We refer to long-term (multi-step) reward as return

Rt = rt +1 + rt + 2 + ... + rT
where T is the last time step of the world

Discounted Return
The geometrically discounted model of return

Rt = rt +1 + rt + 2 + ... + rT
T

0 1

is called discount rate, used to Bound the infinite sum Favor earlier rewards, in other words to give preference to shorter paths

Optimal Policies
An RL agent adapts its policy in order to increase return A policy p1 is at least as good as a policy p2 if its expected return is at least as great in each possible initial state An optimal policy p is at least as good as any other policy

Policy Adaptation Methods

Value function-based methods
Learn a value function for the policy Generate a new policy from the value function Q-learning, Dynamic Programming

Value Functions
A value function maps each state to an estimate of return under a policy An action-value function maps from stateaction pairs to estimates of return Learning a value function is referred to as the prediction problem or policy evaluation in the Dynamic Programming literature

Q-learning
Learns action-values Q(s,a) rather than statevalues V(s) Action-values learning

Q( s, a) = R( s, a ) + max a ' Q(T ( s, a ), a ' )

Q-learning improves action-values iteratively until it converges

Q-learning Algorithm
1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.

Algorithm Q { For each (s,a) initialize Q(s,a) at zero Choose current action s Iterate infinitely{ Choose and execute action a Get immediate reward r Choose new state s Update Q(s,a) as follows: Q(s,a) s s } }

r + max Q(s,a)

Example
Initially Initialization

G 0 0

0 0 0 0 0

100 G 0 100 0

Example
s1 Assume Go right: s2
Reward: 0

= 0,9

A 0 0

0 0 0 0 0

100 G 0 0 0 0 100 0

100 A 0 0 0 0 0 0 G 100

Example
Go right
Reward: 100

Update s2
Reward: 100

0 0 0 0

100 A 0 0 0 0 0 0 G 0 100 0

100 A 0 0 0 0 0 0 G 100

Example
Update s1
Reward: 90

s2

A 0 0

90 0 0 0 0

100 G 0 0 0 0 100 0

90

100 G 0 100 0

A 0 0 0

0 0

Example: result of Q-learning

90 72 81 81 81 81 72

100 G 90 100

90 81

Exercice
Agent is in room C of the building The goal is to get out of the building

Modeling the problem

A A B C D E F

F
100

100 100

Result
= 0,8
A A B C D E Divide all rewards by 5 F
320 400 400 255 320 400 320 320 400 500 500

E
400

F
500

Result: C => D => B => F C => D => E => F

Artificial Intelligence
For HEDSPI Project

Lecturer 15 Artificial Neuron Networks

Lecturer : Tran Duc Khanh Dept of Information Systems School of Information and Communication Technology HUST

Artificial neural networks

Artificial neural network (ANN)
Inspired by biological neural systems, i.e., human brains ANN is a network composed of a number of artificial neurons

Neuron
Has an input/output (I/O) characteristic Implements a local computation

The output of a unit is determined by

Its I/O characteristic Its interconnections to other units Possibly external inputs

Artificial neural networks

ANN can be seen as a parallel distributed information processing structure ANN has the ability to learn, recall, and generalize from training data by assigning and adjusting the interconnection weights The overall function is determined by
The network topology The individual neuron characteristic The learning/training strategy The training data
3

Applications of ANNs
Image processing and computer vision
E.g., image matching, preprocessing, segmentation and analysis, computer vision, image compression, stereo vision, and processing and understanding of time-varying images

Signal processing
E.g., seismic signal analysis and morphology

Pattern recognition
E.g., feature extraction, radar signal classification and analysis, speech recognition and understanding, fingerprint identification, character recognition, face recognition, and handwriting analysis

Medicine
E.g., electrocardiographic signal analysis and understanding, diagnosis of various diseases, and medical image processing
4

Applications of ANNs
Military systems
E.g., undersea mine detection, radar clutter classification, and tactical speaker recognition

Financial systems
E.g., stock market analysis, real estate appraisal, credit card authorization, and securities trading

Planning, control, and search

E.g., parallel implementation of constraint satisfaction problems, solutions to Traveling Salesman, and control and robotics

Power systems
E.g., system state estimation, transient detection and classification, fault detection and recovery, load forecasting, and security assessment

...
5

Structure and operation of a neuron

The input signals to the neuron (xi, i = 1..m)
Each input xi associates with a weight wi x0=1 x1 x2 xm w1 w2 wm w0

The bias w0 (with the input x0=1) Net input is an integration function of the inputs Net(w,x) Activation (transfer) function computes the output of the neuron f(Net(w,x)) Output of the neuron: Out=f(Net(w,x))

Output of the neuron (Out)

Inputs to the neuron (x)

Net input (Net)

Activation (transfer) function (f)

Net input and The bias

The net input is typically computed using a linear function

Net = w0 + w1 x1 + w2 x2 + ... + wm xm = w0 .1 + wi xi = wi xi
i =1 i =0

The importance of the bias (w0)

The family of separation functions Net=w1x1 cannot separate the instances into two classes The family of functions Net=w1x1+w0 can Net Net = w1x1 Net

Net = w1x1 + w0 x1
7

x1

Activation function Hard-limiter

Also called the threshold function The output of the hard-limiter is either of the two values is the threshold value Disadvantage: neither continuous nor continuously differentiable
Binary hard-limiter

1, if Net Out ( Net ) = hl1( Net , ) = 0, if otherwise

Out ( Net ) = hl 2( Net , ) = sign( Net , )

Out 1
Bipolar hard-limiter

Out 1

Net

0 -1

Net

Activation function Threshold logic

0, if Net < 1 Out ( Net ) = tl ( Net , , ) = ( Net + ), if Net 1 1, if Net >

= max(0, min(1, ( Net + )))

It is called also saturating linear function A combination of linear and hard-limiter activation functions decides the slope in the linear range Disadvantage: continuous but not continuously differentiable
9

Out

( >0)

1 - 0 1/ (1/)-

Net

Activation function Sigmoidal

Out ( Net ) = sf ( Net , , ) =
Most often used in ANNs The slope parameter is important The output value is always in (0,1) Advantage
Both continuous and continuously differentiable The derivative of a sigmoidal function can be expressed in terms of the function itself
- 0

1 1 + e ( Net + )
Out
1 0.5

Net

10

Activation function Hyperbolic tangent

1 e ( Net + ) 2 Out ( Net ) = tanh( Net , , ) = = 1 ( Net + ) ( Net + ) 1+ e 1+ e
Also often used in ANNs The slope parameter is important The output value is always in (-1,1) Advantage
Both continuous and continuously differentiable The derivative of a tanh function can be expressed in terms of the function itself
- 0 -1 1

Out

Net

11

Network structure
Topology of an ANN is composed by:
The number of input signals and output signals The number of layers The number of neurons in each layer The number of weights in each neuron The way the weights are linked together within or between the layer(s) Which neurons receive the (error) correction signals input hidden layer output layer output An ANN with one hidden layer Input space: 3-dimensional Output space: 2-dimensional In total, there are 6 neurons - 4 in the hidden layer - 2 in the output layer bias

Every ANN must have

exactly one input layer exactly one output layer zero, one, or more than one hidden layer(s)
12

Network structure
A layer is a group of neurons A hidden layer is any layer between the input and the output layers Hidden nodes do not directly interact with the external environment An ANN is said to be fully connected if every output from one layer is connected to every node in the next layer An ANN is called feed-forward network if no node output is an input to a node in the same layer or in a preceding layer When node outputs can be directed back as inputs to a node in the same (or a preceding) layer, it is a feedback network
If the feedback is directed back as input to the nodes in the same layer, then it is called lateral feedback

Feedback networks that have closed loops are called recurrent networks
13

Network structure Example

single layer feed-forward network single node with feedback to itself

single layer recurrent network multilayer feed-forward network

multilayer recurrent network

14

Learning rules
Two kinds of learning in neural networks
Parameter learning
Focus on the update of the connecting weights in an ANN

Structure learning
Focus on the change of the network structure, including the number of processing elements and their connection types

These two kinds of learning can be performed simultaneously or separately Most of the existing learning rules are the type of parameter learning We focus the parameter learning
15

General weight learning rule

At a learning step (t) the adjustment of the weight vector w is proportional to the product of the learning signal r(t) and the input x(t) w(t) ~ r(t).x(t) w(t) = .r(t).x(t)
where (>0) is the learning rate
x0= 1 x1 x
...

w0 w1 wj wm x a neuron w Learning signal generator d Out

xj
...

xm

The learning signal r is a function of w, x, and the desired output d

r = g(w,x,d)

Note that xj can be either:

an (external) input signal, or an output from another neuron

The general weight learning rule w(t) = .g(w(t),x(t),d(t)).x(t)

16

Perceptron
A perceptron is the simplest type of ANNs Use the hard-limit activation function
m wj x j Out = sign( Net ( w, x) ) = sign j =0
x0=1 x1 x2 xm w1 w2 wm w0

Out

For an instance x, the perceptron output is

1, if Net(w,x)>0 -1, otherwise

17

Perceptron Illustration
The decision hyperplane x1 w0+w1x1+w2x2=0

Output=1

Output=-1 x2

18

Perceptron Learning
Given a training set D= {(x,d)}
x is the input vector d is the desired output value (i.e., -1 or 1)

The perceptron learning is to determine a weight vector that makes the perceptron produce the correct output (-1 or 1) for every training instance If a training instance x is correctly classified, then no update is needed If d=1 but the perceptron outputs -1, then the weight w should be updated so that Net(w,x) is increased If d=-1 but the perceptron outputs 1, then the weight w should be updated so that Net(w,x) is decreased
19

Perceptron_incremental(D, ) Initialize w (wi an initial (small) random value) do for each training instance (x,d)D Compute the real output value Out if (Outd) w w + (d-Out)x end for until all the training instances in D are correctly classified return w

20

Perceptron_batch(D, ) Initialize w (wi an initial (small) random value) do w 0 for each training instance (x,d)D Compute the real output value Out if (Outd) w w + (d-Out)x end for w w + w until all the training instances in D are correctly classified return w
21

Perceptron - Limitation
The perceptron learning procedure is proven to converge if
The training instances are linearly separable With a sufficiently small used A perceptron cannot correctly classify this training set!

The perceptron may not converge if the training instances are not linearly separable We need to use the delta rule
Converges toward a best-fit approximation of the target function The delta rule uses gradient descent to search the hypothesis space (of possible weight vectors) to find the weight vector that best fits the training instances
22

Error (cost) function

Lets consider an ANN that has n output neurons Given a training instance (x,d), the training error made by the currently estimated weights vector w:

1 Ex (w ) = (d i Outi ) 2 i =1

The training error made by the currently estimated weights vector w over the entire training set D:

1 ED (w ) = D

xD

E (w )
x
23

Gradient descent
Gradient of E (denoted as E) is a vector
The direction points most uphill The length is proportional to steepness of hill

The gradient of E specifies the direction that produces the steepest increase in E E E E

E ( w ) = w , w ,..., w 2 N 1

where N is the number of the weights in the network (i.e., N is the length of w)

Hence, the direction that produces the steepest decrease is the negative of the gradient of E E

w = -.E(w);

wi =

wi

, i = 1..N

Requirement: The activation functions used in the network must be continuous functions of the weights, differentiable everywhere
24

Gradient descent Illustration

One-dimensional E(w) Two-dimensional E(w1,w2)

25

Gradient_descent_incremental (D, ) Initialize w (wi an initial (small) random value) do for each training instance (x,d)D Compute the network output for each weight component wi wi wi (Ex/wi) end for end for until (stopping criterion satisfied) return w
Stopping criterion: # of iterations (epochs), threshold error, etc.
26

Multi-layer NNs and Back-propagation alg.

As we have seen, a perceptron can only express a linear decision surface A multi-layer NN learned by the back-propagation (BP) algorithm can represent highly non-linear decision surfaces The BP learning algorithm is used to learn the weights of a multi-layer NN
Fixed structure (i.e., fixed set of neurons and interconnections) For every neuron the activation function must be continuously differentiable

The BP algorithm employs gradient descent in the weight update rule

To minimize the error between the actual output values and the desired output ones, given the training instances
27

Back-propagation algorithm (1)

Back-propagation algorithm searches for the weights vector that minimizes the total error made over the training set Back-propagation consists of the two phases
Signal forward phase. The input signals (i.e., the input vector) are propagated (forwards) from the input layer to the output layer (through the hidden layers) Error backward phase Since the desired output value for the current input vector is known, the error is computed Starting at the output layer, the error is propagated backwards through the network, layer by layer, to the input layer The error back-propagation is performed by recursively computing the local gradient of each neuron
28

Back-propagation algorithm (2)

Signal forward phase Network activation Error backward phase Output error computation Error propagation

29

Derivation of BP alg. Network structure

Lets use this 3-layer NN to illustrate the details of the BP learning algorithm m input signals xj (j=1..m) l hidden neurons zq (q=1..l) n output neurons yi (i=1..n) wqj is the weight of the interconnection from input signal xj to hidden neuron zq wiq is the weight of the interconnection from hidden neuron zq to output neuron yi Outq is the (local) output value of hidden neuron zq Outi is the network output w.r.t. the output neuron yi Input xj (j=1..m) Hidden neuron zq (q=1..l) Output neuron yi (i=1..n)
x1 ... xj ... xm

wqj
... Outq ...

wiq
... Outi ...

30

BP algorithm Forward phase (1)

For each training instance x
The input vector x is propagated from the input layer to the output layer The network produces an actual output Out (i.e., a vector of Outi, i=1..n)

Given an input vector x, a neuron zq in the hidden layer m receives a net input of Net q = wqj x j
j =1

and produces a (local) output of

m wqj x j Out q = f ( Net q ) = f j =1

where f(.) is the activation (transfer) function of neuron zq

31

BP algorithm Forward phase (2)

The net input for a neuron yi in the output layer is l l m wqj x j Net i = wiq Out q = wiq f q =1 q =1 j =1 Neuron yi produces the output value (i.e., an output of the network) l m l Out i = f ( Net i ) = f wiq Out q = f wiq f wqj x j q =1 j =1 q =1 The vector of output values Outi (i=1..n) is the actual network output, given the input vector x

32

BP algorithm Backward phase (1)

For each training instance x
The error signals resulting from the difference between the desired output d and the actual output Out are computed The error signals are back-propagated from the output layer to the previous layers to update the weights

Before discussing the error signals and their back propagation, we first define an error (cost) function
1 n 1 n 2 2 E (w ) = (d i Out i ) = [d i f ( Net i )] 2 i =1 2 i =1
1 = d i f wiq Out q 2 i =1 q =1
n l
33

BP algorithm Backward phase (2)

According to the gradient-descent method, the weights in the hidden-to-output connections are updated by E wiq = wiq
E Out i Net i wiq = = [d i Out i ][ f ' ( Net i )] Out q = i Out q Out i Net i wiq (note that the negative sign is incorporated in E/Outi)

Using the derivative chain rule for E/wiq, we have

i is the error signal of neuron yi in the output layer

i =
E Out i E = = [d i Out i ][ f ' ( Net i )] Net i Out i Net i

where Neti is the net input to neuron yi in the output layer, and f'(Neti)=f(Neti)/Neti
34

BP algorithm Backward phase (3)

To update the weights of the input-to-hidden connections, we also follow gradient-descent method and the derivative chain rule
E Out q Net q E = wqj = wqj Out q Net q wqj

From the equation of the error function E(w), it is clear that each error term (di-yi) (i=1..n) is a function of Outq
1 E (w ) = d i f wiq Out q 2 i =1 q =1
n l 2

35

BP algorithm Backward phase (4)

Evaluating the derivative chain rule, we have
wqj = (d i Out i ) f ' ( Net i ) wiq f ' (Net q ) x j
n

= i wiq f ' (Net q ) x j = q x j

i =1

i =1 n

q is the error signal of neuron zq in the hidden layer

n E Out q E = q = = f ' (Net q ) i wiq Net q i =1 Out q Net q

where Netq is the net input to neuron zq in the hidden layer, and f'(Netq)=f(Netq)/Netq

36

BP algorithm Backward phase (5)

According to the error equations i and q above, the error signal of a neuron in a hidden layer is different from the error signal of a neuron in the output layer Because of this difference, the derived weight update procedure is called the generalized delta learning rule The error signal q of a hidden neuron zq can be determined in terms of the error signals i of the neurons yi (i.e., that zq
connects to) in the output layer with the coefficients are just the weights wiq

The important feature of the BP algorithm: the weights update rule is local
To compute the weight change for a given connection, we need only the quantities available at both ends of that connection!
37

BP algorithm Backward phase (6)

The discussed derivation can be easily extended to the network with more than one hidden layer by using the chain rule continuously The general form of the BP update rule is wab = axb
b and a refer to the two ends of the (ba) connection (i.e., from neuron (or input signal) b to neuron a) xb is the output of the hidden neuron (or the input signal) b,

a is the error signal of neuron a

38

Back_propagation_incremental(D, )
A network with Q feed-forward layers, q = 1,2,...,Q
qNet i

and qOuti are the net input and output of the ith neuron in the qth layer

The network has m input signals and n output neurons

qw

is the weight of the connection from the jth neuron in the (q-1)th layer to the ith neuron in the qth layer
ij

Step 0 (Initialization) Choose Ethreshold (a tolerable error) Initialize the weights to small random values Set E=0 Step 1 (Training loop) Apply the input vector of the kth training instance to the input layer (q=1)
qOut i

= 1Outi = xi(k), I Step 2 (Forward propagation) Propagate the signal forward through the network, until the network outputs (in the output layer) QOuti have all been obtained
q

Out i = f

q q 1 wij Out j Net i = f j

39

Step 3 (Output error measure) Compute the error and error signals Qi for every neuron in the output layer

1 n E = E + ( d i( k ) Q Out i ) 2 2 i =1

Step 4 (Error back-propagation)

i = (d i(k) Q Out i )f '( QNet i )

Propagate the error backward to update the weights and compute the error signals q-1i for the preceding layers

qwij = .(qi).(q-1Outj);
q 1

qw ij

= qwij + qwij

i = f '( q 1Net i ) q w ji q j ; for all q = Q , Q 1,..., 2

j

Step 5 (One epoch check)

Check whether the entire training set has been exploited (i.e., one epoch) If the entire training set has been exploited, then go to step 6; otherwise, go to step 1

Step 6 (Total error check)

If the current total error is acceptable (E<Ethreshold) then the training process terminates and output the final weights; Otherwise, reset E=0, and initiate the new training epoch by going to step 1
40

BP illustration Forward phase (1)

f(Net1)

x1
f(Net2)

f(Net4) f(Net6) Out6

x2
f(Net3)

f(Net5)

41

BP illustration Forward phase (2)

x1

w1x1 x1

f(Net1)

w1x2 x2
f(Net2)

f(Net4) f(Net6) Out6

x2
f(Net3)

f(Net5)

Out1 = f ( w1x1 x1 + w1x2 x2 )

42

BP illustration Forward phase (3)

f(Net1)

x1
w2 x1 x1
f(Net2)

f(Net4) f(Net6) Out6

x2

w2 x2 x2
f(Net3)

f(Net5)

Out 2 = f ( w2 x1 x1 + w2 x2 x2 )
43

BP illustration Forward phase (4)

f(Net1)

x1
f(Net2)

f(Net4) f(Net6) Out6

x2

w3 x1 x1

f(Net5)

w3 x2 x2

f(Net3)

Out3 = f ( w3 x1 x1 + w3 x2 x2 )
44

BP illustration Forward phase (5)

f(Net1)

x1
f(Net2)

w41Out1

w42Out2

f(Net4)

w43Out3
f(Net5)

f(Net6)

Out6

x2
f(Net3)

Out4 = f ( w41Out1 + w42Out 2 + w43Out3 )

45

BP illustration Forward phase (6)

f(Net1)

x1
f(Net2)

w51Out1

f(Net4) f(Net6) Out6

w52Out2
f(Net5)

x2
f(Net3)

w53Out3
Out5 = f ( w51Out1 + w52Out 2 + w53Out3 )
46

BP illustration Forward phase (7)

f(Net1)

x1
f(Net2)

f(Net4)

w 64Out4
f(Net6) Out6

x2
f(Net3)

f(Net5)

w65Out5

Out6 = f ( w64Out 4 + w65Out5 )

47

BP illustration Compute the error

f(Net1)

x1
f(Net2)

f(Net4)

6
f(Net6) Out6

x2
f(Net3)
6 =

f(Net5) d is the desired output value

E Out 6 E = = [d Out 6 ] [ f ' ( Net 6 )] Net 6 Out 6 Net 6
48

BP illustration Backward phase (1)

f(Net1)

x1
f(Net2)

4
f(Net4)

w64

6
f(Net6)

Out6

x2
f(Net3)

f(Net5)

4 = f ' (Net 4 )(w64 6 )

49

BP illustration Backward phase (2)

f(Net1)

x1
f(Net2)

f(Net4)

6
f(Net6)

Out6

5
f(Net5)

x2
f(Net3)

w65

5 = f '(Net5 )(w656 )
50

BP illustration Backward phase (3)

1
f(Net1)

x1
f(Net2)

w41
w51

4
f(Net4) f(Net6) Out6

5
f(Net5)

x2
f(Net3)
51

1 = f '(Net1 )(w414 + w515 )

BP illustration Backward phase (4)

f(Net1)

x1

4
w42 w52
f(Net4) f(Net6) Out6

2
f(Net2)

5
f(Net5)

x2
f(Net3)

2 = f '(Net 2 )(w42 4 + w52 5 )

52

BP illustration Backward phase (5)

f(Net1)

x1
f(Net2)

4
f(Net4) f(Net6) Out6

w43
w53

5
f(Net5)

x2

3
f(Net3)

3 = f '(Net3 )(w434 + w535 )

53

BP illustration Weight update (1)

w1x1
f(Net1)

1
f(Net4) f(Net6) Out6

x1

w1x2
f(Net2)

x2
f(Net3)

f(Net5)

w1 x1 = w1 x1 + 1 x1 w1 x 2 = w1 x 2 + 1 x 2
54

BP illustration Weight update (2)

f(Net1)

x1

w2 x1 2
f(Net2)

f(Net4) f(Net6) Out6

x2

w2 x2
f(Net3)

f(Net5)

w 2 x1 = w 2 x1 + 2 x1 w 2 x 2 = w 2 x 2 + 2 x 2
55

BP illustration Weight update (3)

f(Net1)

x1
f(Net2)

f(Net4) f(Net6) Out6

x2

w3x1
w3x2

3
f(Net3)

f(Net5)

w3 x1 = w3 x1 + 3 x1 w3 x 2 = w3 x 2 + 3 x 2
56

BP illustration Weight update (4)

f(Net1)

x1
f(Net2)

w41

4
f(Net4) Out6

w42

w43
f(Net5)

f(Net6)

x2
f(Net3)

w41 = w41 + 4Out1 w42 = w42 + 4Out2 w43 = w43 + 4Out3

57

BP illustration Weight update (5)

f(Net1)

x1
f(Net2)

f(Net4) f(Net6) Out6

x2
f(Net3)

w52

w 51
w53

5
f(Net5)

w51 = w51 + 5Out1 w52 = w52 + 5Out2 w53 = w53 + 5Out3

58

BP illustration Weight update (6)

f(Net1)

x1
f(Net2)

f(Net4)

w64 w65

6
f(Net6)

Out6

x2
f(Net3)

f(Net5)

w64 = w64 + 6Out 4 w65 = w65 + 6Out 5

59

Advantages vs. Disadvantages

Advantages
Massively parallel in nature Fault (noise) tolerant because of parallelism Can be designed to be adaptive

Disadvantages
No clear rules or design guidelines for arbitrary applications No general way to assess the internal operation of the network (therefore, an ANN system is seen as a black-box) Difficult to predict future network performance (generalization)

60

When using ANNs?

Input is high-dimensional discrete or real-valued The target function is real-valued, discrete-valued or vector-valued Possibly noisy data The form of the target function is unknown Human readability of result is not (very) important Long training time is accepted Short classification/prediction time is required

61