11-711 Algorithms for NLP

Introduction to Analysis of Algorithms

Reading:
Cormen, Leiserson, and Rivest,
Introduction to Algorithms
Chapters 1, 2, 3.1., 3.2.
Analysis of Algorithms

Analyzing an algorithm: predicting the computational resources that

the algorithm requires
– Most often interested in runtime of the algorithm
– Also memory, communication bandwidth, etc.

Computational model assumed: RAM

– Instructions are executed sequentially
– Each instruction takes constant time (different constants)

The behavior of an algorithm may be different for different inputs.

We want to summarize the behavior in general, easy-to-understand
formulas, such as worst-case, average-case, etc.

1 11-711 Algorithms for NLP

Analysis of Algorithms
Generally, we are interested in computation time as a function of

input size.
– Different measures of input size depending on the problem
– Examples:
Problem Input Size
Sorting List of numbers to sort # of numbers to sort
Multiplication Numbers to multiply # of bits
Graph Set of nodes and edges # of nodes + # of edges
“Running time” of an algorithm: number of primitive “steps”

executed
– Think of each “step” as a constant number of instructions, each of
which takes constant time.
Example: Insertion-Sort (CLR, p. 8)

2 11-711 Algorithms for NLP

Insertion Sort (A)

cost times
1. for j 2 to length[A] n



2. do key A[j] n-1



3.
4. i j-1 n-1



5. while i 0 and A[i] key




 


 

6. do A[i+1] A[i]







 
 
7. i i-1







 
 
8. A[i+1] key n-1




3 11-711 Algorithms for NLP

Insertion-Sort
: # of times the while loop in line 5 is executed for the value



+ *

+ 0

+ 1
'

') 

' / ,

' / ,

'2 
"!


(
# $
# %&

#
%#

%#

%# ,

%#
.-,

.-,

.-,
&

&

&
In the best case, when the array is already sorted, the running time is
3

a linear function of .
4

!"


5
6%
In the worst case, when the array is opposite of sorted order, the
3

running time is a quadratic function of .

4
& 6%
!"


5

#
%

4 11-711 Algorithms for NLP

Analysis of Algorithms

We are mostly interested in worst-case performance because

7

1. It is an upper bound on the running time for any input.

2. For some algorithms it occurs fairly often.
3. The average case is often roughly as bad as the worst case.

We are interested in the rate of growth of the runtime as the input size
7

increases. Thus, we only consider the leading term.

– Example: Insertion-Sort has worst case

<;:9
8
=
We will normally look for algorithms with the lowest order of growth
7

But not always:

7

for instance, a less efficient algorithm may be preferable if it’s an

>

anytime algorithm.

5 11-711 Algorithms for NLP

Analyzing Average Case of Expected Runtime

Non-experimental method:
?

1. Assume all inputs of length are equally likely

@
2. Analyze runtime for each particular input and calculate the
average over all the inputs

Experimental method:
?

1. Select a sample of test cases (large enough!)

2. Analyze runtime for each test case and calculate the average and
standard deviation over all test cases

6 11-711 Algorithms for NLP

Recursive Algorithms (Divide and Conquer)

General structure of recursive algorithms:

A

1. Divide the problem into smaller subproblems.

2. Solve (Conquer) each subproblem recursively.
3. Merge the solutions of the subproblems into a solution for the full
problem.

Analyzing a recursive algorithm usually involves solving a

A

recurrence equation.

Example: Merge-Sort (CLR, p. 13)

A

7 11-711 Algorithms for NLP

Recursive Algorithms

time to divide a problem of size

<GF
B
C
ED

E
time to combine solutions of size
GF
B
ED

E
H

Assume we divide a problem into subproblems, each the size of

J K
B

I
the original problem.

The resulting recurrence equation is:

B

OND VUD

P Q YZ[
E \]^
TSR
M LI
F XWF K
G<F
L
ED

C
ED

ED

P_
\
H
XWF

8 11-711 Algorithms for NLP

 11-711 Algorithms for NLP
ij ij
ij ij m i m i
times
1
1
1

1
1
1
1
pv
i oq
: Initialization

up t

9
w
and

pv pv
mx zv
h i oj o lt
p pu o nt
fgc
ce st ij y iq y { {
on l
cd xk v x k ztv pk v pk v

3. create arrays
x

to
to
ba` <ml rm st w p p
xk k z i oj i oq t kx
ik j ik q st kz

do

do
Merge
w

4. for

6. for

10.
11.
1.
2.

5.

7.
8.
9.
 11-711 Algorithms for NLP
(cont.): Actual Merge

– –
‘ ‘

10
• •
‘ ‘   ”  ”
times
n
n

‰ˆŠ ˆŠ
‚ ˆŠ ‡ Œ
g~ † Œ „ Š “’ ƒ„ ˆŠ 
“’
~€ ‹Š ƒ ˆ ‰
~ to
… ‰ˆ Ž ‰„
Ž
„

then
‡

else
b}| ƒ„
do if
12. for
Merge

13.
14.
15.
16.
17.
Merge Sort (A,p,r)

times
1. if p r
—

2. then q (p+r)/2
˜
™

š
3. Merge-Sort(A,p,q)

›
 Ÿž Ÿ ž ¡
œ œ œ
¡ ¡
4. Merge-Sort(A,q+1,r)

›
5. Merge(A,p,q,r)
¢

11 11-711 Algorithms for NLP

A note on Implementation

Note that in the pseudocode of the Merge operation, the for-loop is

£

always executed times, no matter how many items are in each of

¤
the subarrays.

This can be avoided in practical implementation.

£

Take-home message: don’t just blindly follow an algorithm.

£

12 11-711 Algorithms for NLP

Merge-Sort

<ª© ª ©
¥
¦ ­

¬O§ ¨§
¨§ ¨§
« «
© ©
¥

According to the Merge-Sort algorithm, we divide the problem into 2

¥

subproblems of size .

® ¯
The resulting recurrence equation is:
¥

ª ¨ ¨µ
O¬§ §

¬ ¬
±² ±²
« ³°
© ® X´© ¯
ª©
°
¨§

¨§
«
©
By solving the recurrence equation we get ,

ª©
¥

¶¨§
¨§

·¸
¨¯
«

©
much better than for Insertion-Sort.
¯ ©
¨§
«

There are several ways of solving this recurrence equation. The

¥

simplest one uses a recurrence tree, where each node in the tree
indicates what the cost of solving it is. Then inspecting the tree can
give us the solution.
13 11-711 Algorithms for NLP
In this case, we get , because we divide the problem in half in
¹

º»
¾ ½¼
each recursive step.

14 11-711 Algorithms for NLP

Growth of Functions

We need to define a precise notation and meaning for order of growth

¿

of functions.
We usually don’t look at exact running times because they are too
¿

complex to calculate and not interesting enough.

Instead, we look at inputs of large enough size to make only the order
¿

of growth relevant - asymptotic behavior.

Generally, algorithms that behave better asymptotically will be the

¿

best choice (except possibly for very small inputs).

15 11-711 Algorithms for NLP

-notation (“Theta” notation)
À
Asymptotically tight bound
Á

Idea: if both functions grow “equally” fast

<ÆÅ
Á

ÄÂÃ

ÄÃ
Ç
ÈÃ
ÅÅ
Formally:
Á

there exist positive constants , , and

Æ ÅÅ
ÄÃ

ËÌ
ËÍ
Ä
Ç

Î
ÈÃ

ÂÄÃÉ
ÅÊ
such that for

Ë ÌÈ
ÄÃ

ÄÂÃ
Ë ÍÈ
ÄÃ
Ï
Ð

ÐÅ

ÐÅ

Å
all
Ä
ÑÄ
ÎÒ
means a constant function or a constant
Á
OÓÃ
Ç
Å

16 11-711 Algorithms for NLP

-notation (“Big-O” notation)
Ô
Asymptotic upper bound
Õ

Idea: if grows faster than , possibly much

Ú<Ù
Õ

ØÖ×

Ø×

Ø×

ØÖ×
Û
Ü×

Ü
ÙÙ

Ù
faster
Formally:
Õ

there exist positive constants and

Ú ÙÙ
Ø×

ß
Ø
Û

à
Ü×

ÖØ×Ý
ÙÞ
such that for all

ØÖ× Û Ü×
ß Ü ÙÙ
Ø×

Ø
á
â ÚÙ

â Ù Ø×

ãØ
àä
Ù
Note:
ÚÙ

Ú ÙÙ
Õ

ØÖ×

Ø×

ØÖ×
å

æ
Ü×

“Big-O” notation often allows us to describe runtime by just

Õ

inspecting the general structure of the algorithm.

– Example: The double loop in Insertion-Sort leads us to conclude
that the runtime of the algorithm is .
èçØ×
Û
Ù
We use “Big-O” analysis for worst-case complexity since that
Õ

guarantees an upper bound on any case.

17 11-711 Algorithms for NLP
-notation (“Big-Omega” notation)
é

Asymptotic lower bound

ê

Idea: if grows slower than , possibly

ï<î
ê

ð
íëì

íì

íì

íëì
ñì

ñ
îî

î
much slower

Formally:
ê

there exist positive constants and

ï îî
ð
íì

ô
í
õ
ñì

ëíìò
îó
such that for all

ôñ
íì

íëì

í
ö
÷

øí
õù
î÷

î
Theorem:
ê

For any two functions and ,

íëì ï î

íì î
ñ íñìì
î

iff î î and
ïî

ïî
ð
íëì

íì


í

íëì

íì
ú

û
ñì

ëì

ñì
îî

îî
18 11-711 Algorithms for NLP
-notation (“Little-o” notation)
ü
Upper bound that is not asymptotically tight
ý

Idea: if grows much faster than


 
ý

þÿ

þÿ
ÿ
ÿ



ÿ



Formally:
ý

for any positive constant , there exists

 


þÿ
ÿ
ÿ






a constant such that





þÿ

ÿ

for all




We will normally use “Big-O” notation since we won’t want to make
ý

hard claims about tightness.

19 11-711 Algorithms for NLP


-notation (“Little-omega” notation)

Lower bound that is not asymptotically tight



Idea: if grows much slower than
























Note: if and only if


























Formally:


for any positive constant , there exists

 


 

 

!








a constant such that



 

 



!

$
#
"


for all



%

"
&

20 11-711 Algorithms for NLP

Comparison of Functions

The , , , , and relations are transitive.

'

,
(

– Example: )

1 0


1 0


1 0


1 0

/.

/.

/.

5.

/.

/.

5.

/.
)

430

)
-

2.

-
0

0
0
The , , and relations are reflexive.
'

*
(

– Example:
1 0

/ .

/ .


)
-

-
.

0
0
Note the analogy to the comparison of two real numbers.
'

e.g.
10

8
7 0
/
.

/ .

9
<;:
)
-

2.

e.g.
1 0


8
7 0
/
.

/ .

9
;=
-

2.

21 11-711 Algorithms for NLP

 11-711 Algorithms for NLP
B
B

A@
Comparison of Growth of Functions

?
@
B B F
B B B

nor
B
A @

A @


A @

22
? ? B
@ @ ? B
@
D F
I
A@
?
@
C B C B

C B
 E
A @

A @

are neither
A @

Transpose symmetry:
G G

Lack of Trichotomy:
G

iff
iff

iff
B B B B B
B B B B B B
Transitivity: all

B
A
@ A
@ A @

A @

A @


A @


? ? ?
Reflexivity:
@ @ @ G@ G@ B

Symmetry:
G@
D E F D E H
A@
?

Some
CB CB CB C B
 C B
 C B

A
@ A
@ A
@ A
@ A
@ A
@
? ? ? ? ? ?
> > > > >