10/7/2010

General Purpose Methods for Combinatorial Optimization

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

Maximum Contiguous Sum

31

-41

59

26

-53

58

97

-93

-23

84

= 187

Given: A1 ... AN Z, at least one Ai > 0

j

Find i, j such that Ak is maximal

k =i

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

10/7/2010

10/7/2010

MCS: Algorithm 2
31 -41 59 26 -53 58 97 -93 -23 84

MaxSoFar := 0 for L := 1 to N do sum := 0 for U := L to N do sum := sum + x[U] sum of x[ L ... U ] MaxSoFar := max (MaxSoFar, sum) Faster: one nested for-loop less

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

Analysis of Algorithm 2
MaxSoFar := 0 for L := 1 to N do sum := 0 for U := L to N do sum := sum + x[U] sum of x[ L ... U ] MaxSoFar := max (MaxSoFar, sum) Constant time per inner loop iteration Count number of times inner loop is executed
L =1U = L

O(1) = O(N2)

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

10/7/2010

10/7/2010

Challenge: Algorithm 3

Can you develop a linear time algorithm? Solution next lecture

this

31

-41

59

26

-53

58

97

-93

-23

84

= 187

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

Algorithm 3: Linear Time (solution)

31 31 31 -41 0 31 59 59 59 26 85 85 -53 32 85 58 90 90 97 187 -93 94 -23 71 84 155 MaxEndingHere

187 187 187 187 MaxSoFar

Empty vector gives maximum sum == 0 Scan the array just once Keep track of max sum in first I elements (-> MaxSoFar) And of maximum sum ending in position I (-> MaxEndingHere) MaxSoFar := 0 MaxEndingHere := 0 for I := 1 to N do MaxEndingHere := max (0, MaxEndingHere + x[I]) MaxSoFar := max (MaxSoFar, MaxEndingHere)
ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs 10/7/2010 6

10/7/2010

10/7/2010

Comparison

algorithm computation time (sec) computation time as a function of N

1 3 3.4 N 10 3 10 4 10 10 6 10
5 2

2 2 13 N 130 ms 13 sec 22 min 1.5 days 5 months

3 33 N 3.3 ms 33 ms 0.33 sec 3.3 sec 33 sec

3.4 sec 1 hour 39 days 108 years 1080 centuries

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

Traveling Salesman Problem

No polynomial algorithm known Exact solution only by exhaustive search ((N-1)!) possible routes

Stirling:

ln n! n log n n
Large n:

n! e n log n n = e log n = n n = o e n

( )
8

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

10/7/2010

10/7/2010

Result for 1000 cities Clearly sub-optimal

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

Simulated Annealing
important much applied method for various combinatorial optimalization problems. Idea (thermo dynamics, static mechanics) Growing of crystals by means of Annealing 1. start off with high temperature 2. cool down slowly Theoretical Model matter strives for state with lowest energy movement of atoms small random displacements E < 0 accept displacement E > 0 accept with probability exp(-E/kBT)

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

10

10/7/2010

10/7/2010

Simulated Annealing (2)

state of atoms displacement configuration move cost function optimal solution

energy perfect crystal

Algorithm T := T0 X := start configuration while (not satisfied stop criterium) { for (a few times) { Xnew := new configuration if (accept (cost (Xnew) cost (X), T)) X := Xnew } T := update (T) }

accept (cost,T) { if (cost < 0) return TRUE else { if (exp(- cost/cT) > random (0,1)) return TRUE else return FALSE } }

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

11

Traveling Salesman by means of Simulated Annealing

Randomly select two cities, interchange order of visiting: ABCD ACBD relatively long calculations be careful with detailed parameters of algorithm

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

12

10/7/2010

10/7/2010

Shortest Path
Given: Weighted, directed graph G=(V,E) 1. Single-source shortest path Find shortest path form source vertex sV to each v V 2. Single-destination shortest path equivalent by reducing (reverse direction of edges) 3. Single-pair shortest path route-planning no known algorithm being asymptotically better then 1 4. All-pairs shortest path

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

13

Dijkstras Shortest Path Algorithm

v1 6 v2 1 v3 2 5 v4 1 1 3 1 2 v6 1 v5

Q: What is the shortest path from v1 to v2? A: v1 v4 v5 v6 v2 of length 4

1. Find from v1 the shortest path to its neighbors 2. Say u is the closest to v1 3. See if any of the routes from v1 to the neighbors of u becomes shorter if passing through u 4. Continue with step 2, until reaching v2 (target)

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

14

10/7/2010

10/7/2010

Dijkstras Shortest Path Algorithm

v1 6 v2 1 v3 2 5 v4 1 1 3 1 2 v6 1 v5

v1 v4 v5 v6 v2

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

15

Dijkstras Shortest Path Algorithm

v1 6 v2 1 v3 2 5 v4 1 1 3 1 2 v6 1 v5

Shortest distance to these vertices is known

1. Find from v1 the shortest path to its neighbors 2. Say u is the closest to v1 3. See if any of the routes from v1 to the neighbors of u becomes shorter if passing through u 4. Continue with step 2, until reaching v2 (target)

This is vertex u. Shortest distance to these vertices has just become known
ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs 10/7/2010 16

10/7/2010

10/7/2010

Dijkstra in PYTHON language

def Dijkstra (G, start, end = None, animate=0): # G[v] is a dict of distances keyed by vertices adjacent to v # G[v][u] is the distance from v to u if u is adjacent to v T = set () # set of vertices for which min distance is known. D = dict () # dictionary of final distances, # with vertex as key and distance as value V = G.keys () T.add (start) V.remove (start) D[start] = 0 for v in V: D[v] = sys.maxint for v in G[start]: D [v] = G[start][v] # initialize all distances # set which is initialized to all vertices.

# loop over all vertices reachable from start

while end not in T and len (V) > 0: if animate: print "T", T, "\n", "D", D u = closest (D, V) # find closest vertex to start, among those in V T.add (u) V.remove (u) for v in G[u]: www.python.org if D [v] > G[u][v] + D[u]: D [v] = G[u][v] + D[u] return D
ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs 10/7/2010 17

Optimization
Public Transport: It brings you from a place where you are not, to a place where you dont want to be, at a time that you do not prefer EDA: It provides a solution to a problem that you dont want to solve Why? Because most EDA problems are just too hard to be solved exactly Many optimization problems are NP-hard Therefore, problems are usually replaced by simpler problems that are solved instead.

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

18

10/7/2010

10/7/2010

Typical EDA Optimization Setting

Chip optimization problem Transform problem into similar, known, solvable problem in mathematics world

Standard problem Standard Algorithm Mathematics / computer science discipline of science / engineering

bend back into real world

Problem (approximately) solved

Sub-optimal chip

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

19

Continuous vs Discrete Optimization

Continuous Optimization: Solution space forms a continuous domain Example: adjust transistor sizes in a network to optimize the speed of the network, under the assumption of continuously variable transistor dimensions Discrete Optimization: the number of solutions can be counted, from a discrete set (might be infinite) Example: traveling salesman in graph there are a finite number of distinct tours visiting all vertices Or: adjust transistor sizes in a network to optimize the speed of the network, under the assumption that transistor dimensions should be integral multiples of a step size.

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

20

10/7/2010

10

10/7/2010

Discrete Optimization
1 6 5 4 2 3

Optimization algorithms typically go through a sequence of steps Set of choices at each step (Result of) each step is called a configuration Above we see 6 configurations with different cost, 4 choices in config of cost 6.

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

21

Discrete Optimization (2)

1 6 5 p A Xp = X = {xk} c(xk) G = (X, E) step (move) E 2 3

4 given optimization problem instance of algorithm for set of all possible (legal) configurations (solutions) of p cost of configuration xk a directed graph, the configuration graph the transformation (by the algorithm) of configuration xi in xj set of all steps that algorithm A considers
10/7/2010 22

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

11

10/7/2010

Discrete Optimization (3)

1 5 3 4
local minimum xi is a local minimum in case no neighboring configurations exist with lower cost global minimum desired configuration with absolute lowest cost greedy algorithm algorithm that only takes steps that result in a lower cost convex configuration graph in case every local minimum also global minimum (and strongly connected) greedy algorithm will give exact solution. (Not necessarily in optimal time.) Uphill move step of algorithm that (temporarily) increases cost aimed at escaping local minima
ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs 10/7/2010 23

2 6

What is the global minimum? Is this example convex? If not: What is/are local minima? Can you change the graph to become convex?

Standard Optimization Techniques

Backtracking, branch and bound ( 5.2) Dynamic programming ( 5.3) Linear programming (LP), integer LP (ILP) ( 5.4) Local Search ( 5.5) Simulated Annealing ( 5.6) Tabu search ( 5.7) Genetic algorithms ( 5.8) Boolean Satisfiability (SAT) Neural networks Simulated evolution Matching, max flow, shortest path, Non-linear programming: Lagrange relaxation, LevenbergMarquardt,

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

24

10/7/2010

12

10/7/2010

A 3 5

9 5 4

B 5 3

9 5 4

B 5

Branch and Bound

Example: TSP in graph
C

F E 1 D
f1 f2 f3 f4 f5 f6 f7 E F A 27 D F E A 31 D E A 33 C

8 C E 2

5 F 7 1 D
B F D E E x D C x A 27 A 20 A 27 B F C F B

2
A E

The blue partial solution leads to a kill

F F B D D x C B A 31 C D E A 20 B x C D E x C B x D E x E D C B A 27

F E D x

C D E C x

F B C x A 33 C B

B C D x B x

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

25

Bound-step using MST

A 3 5 F E 1 D f1 f2 f3 f4 f5 f6 f7 E F A 27 D F C F C C x B F D E x E D X B F A 20
10/7/2010 26

9 5 4

B 5 8 C 2 E 3

A 5 5 F 7 1 D A E D C F 14+10 F 11+9 F 8+16 E 3

5 F 1 D

F 5+15

22+9 21+6

23+8

MST can be solved in polynomial time!

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

13

10/7/2010

Linear Programming (LP)

Maximize z = x1 + x2 + 3x3 - x 4 Such that xi > 0 for all i x1 + 2x3 740 2x2 7x4 0 x2 x3 + 2x4 x1 + x2 + x3 + x4 = 9 x2 Example with 2 independent variables and only inequalities Feasible region
ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

(objective function) (primary constraints) (additional constraints)

Lines (hyperplanes) of constant z Optimum is in cornerpoint LP solver searches for optimum corner point
27

x1
10/7/2010

LP Solvers
Ellipsoid algorithm runs in polynomial time In practice, Simplex algorithm is often faster, but it has exponential worst-case time complexity Simplex algorithm is clever way of enumerating boundary points, it steps from point to point along the outside of the feasible region Based on principles from linear algebra Many on-line sources (e.g. see wikipedia -> linear programming) Also see the book Numerical Recipes in {C/C++/Fortran} Press et al. On-line C version of the book (pdf) via www.nr.com See lp_solve originally by Michel Berkelaar Now at TUD/EWI/CAS

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

28

10/7/2010

14

10/7/2010

Integer Linear Programming (ILP)

Linear Programming with the xi restricted to integer numbers Surprise: ILP is NP-hard (LP has polynomial time complexity) Does not work: solving an equivalent LP and rounding to nearest integer; solution may not be optimal, or even feasible. Yet other variant is 0-1 LP, xi {0, 1}

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

29

Genetic Algorithms
First parent: ( f(k)) Second parent: (g(k)) 01011 001 10000 110 First child: 0 1 0 1 1 1 1 0 (f(k 1)) Second child: 1 0 0 0 0 0 0 1 ( g(k 1))

Inspired on biology: survival of the fittest Work with a population P(k) F, set of feasible solutions Each f P(k) is encoded as a chromosome, e.g. a bit-string Cross-over operations between members of P(k) to derive P(k+1) Prefer parents with lower cost for mating Many variations possible (mutation, cloning, )

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

30

10/7/2010

15

10/7/2010

Dynamic Programming
Fibonacci numbers: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, Recursive algorithm is a lot of work: e.g. fib(2), fib(3) computed more then once 5 4 3 2 11 0 1 1 0 2 3 2 1 0

function fib(n) Each number starting from if n 3rd = 0 is orsum n = of 1 return previous n two Write algorithm else return fib(n 1) + fib(n 2) var f := map(0 0, 1 1) function fib(n) if n not in keys(f) f[n] := fib(n 1) + fib(n 2) return f[n]
ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

Dynamic programming: Overlapping subproblems Optimal substructure Memoization

10/7/2010 31

Fibonacci Sequence Using Dynamic Programming in Python

f = {0: 0, 1: 1} def dynfib (n): if n not in f: f[n] = dynfib (n(n-1) + dynfib (n(n-2) return f[n]

Recursive version Time and space O(n) Runs into max recursion depth problems Iterative (bottom-up) version Time O(n), space O(1) Runs easily for n = 100000

def dynfib2 (n): if n == 0 or n prevprev = 0 prev = 1 for i in range fib = prev prevprev = prev = fib return fib

== 1: return n

(2, n+1): + prevprev prev

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

32

10/7/2010

16

10/7/2010

Dynamic Programming
Elegant approach Two variants: bottom-up and top-down Several applications in EDA E.g. in channel routing, later lecture. E.g. technology mapping, Dirk-Jan Jongeneel, Optimized boolean equations Library of std cells Timing constraints Gate netlist

[Jongeneel and Otten, Integration, 2000] ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs 10/7/2010 33

Library of Std Cells

[Jongeneel and Otten, Integration, 2000] ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs 10/7/2010 34

10/7/2010

17

10/7/2010

Matching of Library Patterns

Cost at all primary inputs Cost of library element = 1 + #inputs (for sake of example) Cost at node after partial matching 0 3 8 9 15 2 2 13 18 16 4 5 3 8 17 21 19

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

[Jongeneel and Otten, Integration, 2000] 35 10/7/2010

Partial Matching of Library Patterns with Optimal Cost

Covering by contracting saved matches at every node, from output to inputs

[Jongeneel and Otten, Integration, 2000] ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs 10/7/2010 36

10/7/2010

18

10/7/2010

Techn. Map. Result

[Jongeneel and Otten, Integration, 2000] ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs 10/7/2010 37

Summary
Graph algorithms and optimization very common in EDA Discrete vs continuous optimization
Chip optimization problem Transform problem into similar, known, solvable problem in mathematics world

Standard problem Standard Algorithm Mathematics / computer science discipline of science / engineering

bend back into real world

Problem (approximately) solved Sub-optimal chip

Many different algorithms, with different properties Finding right one depends on transformation from and back into real world
ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs 10/7/2010 38

10/7/2010

19

10/7/2010

Summary
Many different optimization algorithmic techniques
Backtracking, branch and bound ( 5.2) Dynamic programming ( 5.3) Linear programming (LP), integer LP (ILP) ( 5.4) Local Search ( 5.5) Simulated Annealing ( 5.6) Tabu search ( 5.7) Genetic algorithms ( 5.8) Boolean Satisfiability (SAT) Neural networks Simulated evolution Matching, max flow, shortest path, Non-linear programming: Lagrange relaxation, LevenbergMarquardt,

Many applications of optimization in EDA (e.g. techn. mapping) Asymptotic complexity important for effectivity of algorithm Implementation can also count (e.g. recursive vs iterative implementation)

ET 4255 - Electronic Design Automation 10/11 Nick van der Meijs

10/7/2010

39

10/7/2010

20