1 Machines & Models

Sebastian Wild
27 January 2020

version 2020-01-28 13:04

Outline

1 Machines & Models

1.1 Algorithm analysis
1.2 The RAM Model
1.3 Asymptotics & Big-Oh
What is an algorithm?
An algorithm is a sequence of instructions.
think: recipe

e. g. Java program
More precisely:
1. mechanically executable
� no “common sense” needed
2. finite description ≠ finite computation!

3. solves a problem, i. e., a class of problem instances

𝑥 + 𝑦 , not only 17 + 4

typical example: bubblesort

not a specific program but underlying idea

1
What is a data structure?

A data structure is
1. a rule for encoding data
(in computer memory), plus
2. algorithms to work with it
(queries, updates, etc.)

typical example: binary search tree

2
1.1 Algorithm analysis
Good algorithms
Our goal: Find good (best?) algorithms and data structures for a task.

Good “usually” means can be complicated in distributed systems

� fast running time

� moderate memory space usage

Algorithm analysis is a way to

� compare different algorithms,

� predict their performance in an application

3
Running time experiment
Why not simply run and time it?

� results only apply to

� single test machine
� tested inputs
� tested implementation
� ...
≠ universal truths

� instead: consider and analyze algorithms on an abstract machine

� provable statements for model survives Pentium 4
� testable model hypotheses

� Need precise model of machine (costs), input data and algorithms.

4
Data Models
Algorithm analysis typically uses one of the following simple data models:
� worst-case performance:
consider the worst of all inputs as our cost metric
� best-case performance:
consider the best of all inputs as our cost metric
� average-case performance:
consider the average/expectation of a random input as our cost metric

Usually, we apply the above for inputs of same size 𝑛.

� performance is only a function of 𝑛.

5
1.2 The RAM Model
Machine models
The machine model decides
� what algorithms are possible

� how they are described (= programming language)

� what an execution costs

Goal: Machine model should be

detailed and powerful enough to reflect actual machines,
abstract enough to unify architectures,
simple enough to analyze.

6
Random Access Machines
Random access machine (RAM) more detail in §2.2 of Sequential and Parallel Algorithms and Data Structures
by Sanders, Mehlhorn, Dietzfelbinger, Dementiev

� unlimited memory MEM[0], MEM[1], MEM[2], . . .

� fixed number of registers 𝑅 1 , . . . , 𝑅 𝑟 (say 𝑟 = 100)

� every memory cell MEM[𝑖] and register 𝑅 𝑖 stores a 𝑤-bit integer, i. e., a number in
[0..2𝑤 − 1]
𝑤 is the word width; typically 2𝑤 ≈ 𝑛
� Instructions:
� load & store: 𝑅 𝑖 := MEM[𝑅 𝑗 ] MEM[𝑅 𝑗 ] := 𝑅 𝑖
� operations on registers: 𝑅 𝑘 := 𝑅 𝑖 + 𝑅 𝑗 (arithmetic is modulo 2𝑤 !)
also 𝑅 𝑖 − 𝑅 𝑗 , 𝑅 𝑖 · 𝑅 𝑗 , 𝑅 𝑖 div 𝑅 𝑗 , 𝑅 𝑖 mod 𝑅 𝑗
C-style operations (bitwise and/or/xor, left/right shift)
� conditional and unconditional jumps

� cost: number of executed instructions

we will see further models later

� The RAM is the standard model for sequential computation.

7
Pseudocode
Typical simplifications for convenience:
� more abstract pseudocode to specify algorithms
code that humans understand (easily)

� count dominant operations (e. g. array accesses) instead of all operations

In both cases: can go to full detail if needed.

8
1.3 Asymptotics & Big-Oh
Why asymptotics?
Algorithm analysis focuses on (the limiting behavior for infinitely) large inputs.
� abstracts from unnecessary detail

� simplifies analysis

� often necessary for sensible comparison

Asymptotics = approximation around ∞

Example: Consider a function 𝑓 (𝑛) given by

2𝑛 2 − 3𝑛 �log2 (𝑛 + 1)� + 7𝑛 − 3�log2 (𝑛 + 1)� + 120

·104
2

1.5

0.5

0
0 10 20 30 40 50 60 70 80 90 100
9
Why asymptotics?
Algorithm analysis focuses on (the limiting behavior for infinitely) large inputs.
� abstracts from unnecessary detail

� simplifies analysis

� often necessary for sensible comparison

Asymptotics = approximation around ∞

Example: Consider a function 𝑓 (𝑛) given by

2𝑛 2 − 3𝑛 �log2 (𝑛 + 1)� + 7𝑛 − 3�log2 (𝑛 + 1)� + 120 ∼ 2𝒏 2

·104
2

1.5

0.5

0
0 10 20 30 40 50 60 70 80 90 100
9
Asymptotic tools – Formal & definitive definition
if, and only if
𝑓 (𝑛)
� “Tilde Notation:” 𝑓 (𝑛) ∼ 𝑔(𝑛) iff = 1 lim
𝑔(𝑛) 𝑛→∞
„ 𝑓 and 𝑔 are asymptotically equivalent”

� �
�
also write ‘=’ instead
� � 𝑓 (𝑛) �
� �
� “Big-Oh Notation:” 𝑓 (𝑛) ∈ 𝑂 𝑔(𝑛) iff � � is bounded for 𝑛 ≥ 𝑛0
� 𝑔(𝑛) �
� �
� 𝑓 (𝑛) �
� �
need supremum since limit might not exist!

iff lim sup � � < ∞

𝑛→∞ � 𝑔(𝑛) �
Variants: “Big-Omega”
� � � �
� 𝑓 (𝑛) = Ω 𝑔(𝑛) iff 𝑔(𝑛) = 𝑂 𝑓 (𝑛)
� � � � � �
� 𝑓 (𝑛) = Θ 𝑔(𝑛) iff 𝑓 (𝑛) = 𝑂 𝑔(𝑛) and 𝑓 (𝑛) = Ω 𝑔(𝑛)
� �
� 𝑓 (𝑛) �
“Big-Theta”
� � � �
� “Little-Oh Notation:” 𝑓 (𝑛) = 𝑜 𝑔(𝑛) iff lim � � = 0
𝑛→∞ � 𝑔(𝑛) �
� �
𝑓 (𝑛) = 𝜔 𝑔(𝑛) if lim = ∞

10
Asymptotic tools – Intuition
𝑐 𝑔(𝑛)
� 𝑓 (𝑛) = 𝑂(𝑔(𝑛)): 𝑓 (𝑛) is at most 𝑔(𝑛) 𝑐 𝑔 (𝑛 )
𝑓 (𝑛 ) ≤
up to constant factors and 𝑓 (𝑛)

for sufficiently large 𝑛

𝑛
𝑛0

𝑐 2 𝑔(𝑛)
� 𝑓 (𝑛) = Θ(𝑔(𝑛)): 𝑓 (𝑛) is equal to 𝑔(𝑛) )
𝑓 (𝑛 ) ≤ 𝑐 2 𝑔 (𝑛
up to constant factors and 𝑐 1 𝑔 (𝑛 ) ≤
𝑓 (𝑛)
for sufficiently large 𝑛 𝑐 1 𝑔(𝑛)

𝑛
𝑛0
Plots can be misleading! Example

11
Clicker Question

Assume 𝑓 (𝑛) ∈ 𝑂(𝑔(𝑛)). What can we say about 𝑔(𝑛)?

A 𝑔(𝑛) = 𝑂( 𝑓 (𝑛))

B 𝑔(𝑛) = Ω( 𝑓 (𝑛))

C 𝑔(𝑛) = Θ( 𝑓 (𝑛))

D Nothing (it depends on 𝑓 and 𝑔)

pingo.upb.de/622222
12
Clicker Question

Assume 𝑓 (𝑛) ∈ 𝑂(𝑔(𝑛)). What can we say about 𝑔(𝑛)?

A 𝑔(𝑛) = 𝑂( 𝑓 (𝑛))

B 𝑔(𝑛) = Ω( 𝑓 (𝑛)) �
C 𝑔(𝑛) = Θ( 𝑓 (𝑛))

D Nothing (it depends on 𝑓 and 𝑔)

pingo.upb.de/622222
12
Clicker Question

Assume 𝑓 (𝑛) ∈ 𝑂(𝑔(𝑛)). What can we say about 𝑔(𝑛)?

A 𝑔(𝑛) = 𝑂( 𝑓 (𝑛))

B 𝑔(𝑛) = Ω( 𝑓 (𝑛)) � (if 𝑓 (𝑛) ≠ 0)

C 𝑔(𝑛) = Θ( 𝑓 (𝑛))

D Nothing (it depends on 𝑓 and 𝑔) �

pingo.upb.de/622222
12
Asymptotics – Example 1
Basic examples:
� 20𝑛 3 + 10𝑛 ln(𝑛) + 5 ∼ 20𝑛 3 = Θ(𝑛 3 )

� 3 lg(𝑛 2 ) + lg(lg(𝑛)) = Θ(log 𝑛)

� 10100 = 𝑂(1)

Use wolfram alpha to compute/check limits.

13
Clicker Question

Is (sin(𝑛) + 2)𝑛 2 = Θ(𝑛 2 )?

A Yes B No

pingo.upb.de/622222
14
Clicker Question

Is (sin(𝑛) + 2)𝑛 2 = Θ(𝑛 2 )?

A Yes � B No

pingo.upb.de/622222
14
Asymptotics – Frequently used facts
� Rules:
� 𝑐 · 𝑓 (𝑛) = Θ( 𝑓 (𝑛)) for constant 𝑐 ≠ 0
� Θ( 𝑓 + 𝑔) = Θ(max{ 𝑓 , 𝑔}) largest summand determines order of growth

� Frequently used orders of growth:

� logarithmic Θ(log 𝑛) Note: 𝑎, 𝑏 > 0 constants � Θ(log𝑎 (𝑛)) = Θ(log𝑏 (𝑛))

� linear Θ(𝑛)
� linearithmic Θ(𝑛 log 𝑛)
� quadratic Θ(𝑛 2 )
� polynomial 𝑂(𝑛 𝑐 ) for constant 𝑐
� exponential 𝑂(𝑐 𝑛 ) for constant 𝑐 Note: 𝑎 > 𝑏 > 0 constants � 𝑏 𝑛 = 𝑜(𝑎 𝑛 )

15
Asymptotics – Example 2
Square-and-multiply algorithm
for computing 𝑥 𝑚 with 𝑚 ∈ ℕ 1 double pow(double base, boolean[] exponentBits) {
2 double res = 1;
Inputs:
3 for (boolean bit : exponentBits) {
� 𝑚 as binary number (array of bits) 4 res ∗= res;
� 𝑛 = #bits in 𝑚 5 if (bit) res ∗= base;
6 }
� 𝑥 a floating-point number
7 return res;
8 }

� Cost: 𝐶 = # multiplications
� 𝐶 = 𝑛 (line 4) + #one-bits binary representation of 𝑚 (line 5)
� 𝑛 ≤ 𝐶 ≤ 2𝑛

16
Clicker Question

We showed 𝑛 ≤ 𝐶(𝑛) ≤ 2𝑛; what is the most precise

asymptotic approximation for 𝐶(𝑛) that we can make?
√
Write e. g. O(n^2) for 𝑂(𝑛 2 ) or Theta(sqrt(n)) for Θ( 𝑛).

pingo.upb.de/622222
17
Asymptotics – Example 2 shift-and-multiply-exponentiation

Square-and-multiply algorithm 20

for computing 𝑥 𝑚 with 𝑚 ∈ ℕ 1 double pow(double base, boolean[] exponentBits) {

2 double res = 1;
Inputs: 15
3 for (boolean bit : exponentBits) {
� 𝑚 as binary number (array of bits) 4 res ∗= res;
� 𝑛 = #bits in 𝑚 510 if (bit) res ∗= base;
6 }
� 𝑥 a floating-point number
7 return res;
8 }
5

� Cost: 𝐶 = # multiplications 0
0 200 400 600 800 1000

� 𝐶 = 𝑛 (line 4) + #one-bits binary representation of 𝑚 (line 5)

� 𝑛 ≤ 𝐶 ≤ 2𝑛
� 𝐶 = Θ(𝑛) = Θ(log 𝑚)

Note: Often, you can pretend Θ is “like ∼ with an unknown constant”

but in this case, no such constant exists!

18