Data Structures & Algorithms

Lecture 12 and 13: Algorithms

(Program Efficiency & Asymptotic Complexity)
Dr. Ahsan Saadat
[email protected]

Department Of Computing (DOC),

School of Electrical Engineering & Computer Science (SEECS),
National University of Sciences & Technology (NUST)
 Recap

▪ Basic data structures including

► Arrays & structures

► Linked Lists (single, double & circular)

► Stacks (and its applications)

► Queues

- Array based implementation

- Circular queue

2
 Algorithm

▪ An algorithm takes the input to a problem and

transforms it to the output

► A mapping of input to output

▪ A problem can have many algorithms

▪ How do you know which algorithm is better than the

other for the same problem?

3
 Algorithm Properties

▪ An algorithm possesses the following properties:

► It must be correct

► It must be composed of a series of concrete steps

► There can be no ambiguity as to which step will be

performed next

► It must be composed of a finite number of steps

► It must terminate

4
 Questions to be addressed!

▪ What is a “good” or "efficient" program?

▪ How to measure the efficiency of a program?

▪ How to analyse a simple program?

5
 Efficiency

▪ A solution is said to be efficient if it solves the problem

within its resource constraints

▪ Complexity types
► Space (Space Complexity)

► Time (Time Complexity)

▪ Which measure is more important?

► Answer often depends on the limitations of the
technology available at the time of analysis

6
 Measuring Efficiency

▪ Ways of measuring efficiency:

► Run the program and see how long it takes

► Run the program and see how much memory it

uses

▪ Lots of variables to control:

► What is the input data?

► What is the hardware platform?

► What is the programming language/compiler?

7
 Running time: Empirical Method

▪ Write a program that implements the algorithm

▪ Run the program with inputs of varying size and

composition

▪ Use a timing function to get an accurate measure of

actual running time

▪ Plot the results

8
 Empirical Method: Example

▪ Computer A (state-of-the-art machine)

► Linear search algorithm

▪ Computer B (slower machine)

► Binary search algorithm

Computer A run-time Computer B run-time

n (list size)
(in nanoseconds) (in nanoseconds)
15 7 ns 100,000 ns
65 32 ns 150,000 ns
250 128 ns 200,000 ns
1,000 500 ns 250,000 ns

9
 Empirical Method: Example

Computer A run-time Computer B run-time

n (list size)
(in nanoseconds) (in nanoseconds)
15 7 ns 100,000 ns
65 32 ns 150,000 ns
250 125 ns 200,000 ns
1,000 500 ns 250,000 ns
... ... ...
1,000,000 500,000 ns 500,000 ns
4,000,000 2,000,000 ns 550,000 ns
16,000,000 8,000,000 ns 600,000 ns
... ... ...
31,536 * 1012 ns, 1,375,000 ns,
63,072 * 1012
or 1 year or 1.375 milliseconds

10
 Empirical Method: Example

▪ Computer A
► Linear growth rate
► Program's run-time is directly proportional to its input size

► Doubling the input size doubles the run time,

quadrupling the input size quadruples the run-time …

▪ Computer B
► Logarithmic growth rate
► Doubling the input size only increases the run time by a
constant amount (i.e. 25,000 ns)
► Even though Computer A is a faster machine, Computer B
will inevitably surpass Computer A in run-time because it's
running an algorithm with a much slower growth rate

11
 Limitations of Empirical Method

▪ Necessary to implement and test the algorithm in order

to determine its running time, which may be difficult

▪ Experiments can be done only on a limited set of

inputs, and may not be indicative of the running time
on other inputs not included in the experiment

▪ In order to compare two algorithms, the same

hardware and software environments must be used

12
 Limitations of Empirical Method –
Algorithm Properties

▪ Need to study algorithms so that they are language

and machine independent

▪ I.e., we need techniques that enable us to compare

algorithms without implementing them

► RAM model of computation

► Asymptotic analysis of worst-case complexity

13
 RAM Model

▪ Has one processor

▪ Executes one instruction at a time
▪ Each instruction takes "unit time“
▪ Has fixed-size operands, and
▪ Has fixed size storage (RAM and disk)

14
 A Simple Example
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A

int Sum(int A[], int N){

int s=0;

for (int i=0; i< N; i++)

s = s + A[i];

return s;
}

How should we analyse this?

15
 A Simple Example: Analysis of Sum

▪ Describe the size of the input in terms of one or more

parameters:
► Input to Sum is an array of N ints, so size is N

▪ Then, count how many steps are used for an input of

that size:
► "simple" operation (+, <, =, A[i], -, if, call etc.) takes
exactly 1 step
► Loops and subroutine calls are not simple
operations, instead they are the composition of
many single-steps operations

16
 A Simple Example: Analysis of Sum
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A

int Sum(int A[], int N){

int s=0; 1

for (int i=0; i< N; i++)

2 3 4
s = s + A[i];
5
return s; 6 7
}
8

1,2,8: Once
3,4,5,6,7: Once per each iteration of for loop, N iteration
Total: 5N + 3
The complexity function of the algorithm is : f(N) = 5N +3

17
 How 5N+3 grows?

Estimated running time for different values of N:

N = 10 5(10)+3 = 53 steps
N = 100 5(100)+3 = 503 steps
N = 1,000 5(1000)+3 = 5003 steps
N = 1,000,000 5(1000000)+3 = 5,000,003 steps

As N grows, the number of steps grow in linear proportion

to N for this Sum function

18
 Which term Dominates?

▪ What about the 5 in 5N+3? What about the +3?

► As N gets large, the +3 becomes insignificant
► 5 is inaccurate, as different operations require varying
amounts of time

▪ What is fundamental is that the time is linear w.r.t. N

Asymptotic Complexity:
▪ As N gets large, concentrate on the highest order term:
▪ Drop lower order terms such as +3
▪ Drop the constant coefficient of the highest order term
i.e. 5

19
 Asympototic Complexity

▪ The 5N+3 time bound is said to "grow asymptotically"

like N

▪ This gives us an approximation of the complexity of the

algorithm
(i.e. f(n) = n )

▪ Ignores lots of (machine dependent) details, concentrate

on the bigger picture

20
 Growth Rates

▪ Growth rates of
functions:
▪ Linear ≈ n
▪ Quadratic ≈ n2
▪ Cubic ≈ n3

▪ In a log-log plot, the

slope of the line
corresponds to the
growth rate of the
function

21
 Constant Factors

▪ The growth rate is not

affected by
▪ constant factors or
▪ lower-order terms

▪ Examples
▪ 102n + 105 is a
linear function
▪ 105n2 + 108n is a
quadratic function

22
 Machine Independent Time

▪ Ignore machine-dependent constants

▪ Look at growth of T(n) as n→∞

“Asymptotic Analysis”

23
 Orders of Growth

Slowest growth rate Fastest growth rates

log2n 2n, n!

Algorithms with an exponential rate of growth are practical

for solving only problems of very small size

24
 Summary

▪ When a problem's input size is small, most algorithms

will exhibit similar efficiencies

▪ For large input size, different algorithms will exhibit

significantly different behaviour

▪ Therefore, for large values of n, the function's order of

growth becomes important

25
 Recap

▪ To compare the efficiency of algorithms, a measure of the degree

of difficulty of an algorithm called computational complexity is
used

▪ Computational complexity indicates how much effort is needed to

apply an algorithm or how costly it is

► Two most important criterion to measure cost include: time

and memory which an algorithm takes to execute

► Efficiency considerations usually focus on the amount of time

elapsed when processing data

► However, run time is always system-dependent consequently

requiring same platforms and programming language
26
 Recap
▪ Two ways to perform computational complexity analysis:
► Empirical method

- Write a program that implements the algorithm

- Run the program with inputs of varying size and composition
- Use a timing function to measure of actual running time
- Plot the results
- Limitations:
- Necessary to implement and test the algorithm in order
to determine its running time, which may be difficult
- Experiments can be done only on a limited set of inputs,
and may not be indicative of the running time on other
inputs not included in the experiment
- In order to compare two algorithms, the same hardware
and software environments must be used
► Asymptotic analysis

- Ignore machine-dependent constants

- Look at growth of T(n) as n→∞
27
 Asymptotic Analysis

▪ Asymptotic analysis is a useful tool to help to structure our

thinking toward better algorithm

▪ We shouldn’t ignore asymptotically slower algorithms,

however
► Need to understand relationship between problem size and
running time

▪ For a given input size n we express the time T to run the

algorithm as a function T(n)

▪ Concept of growth rate allows us to compare running time of

two algorithms without writing two programs

28
 Asymptotic complexity

▪ To evaluate an algorithm’s efficiency, real-time units such as

microseconds and nanoseconds should not be used

▪ Rather, logical units that express a relationship between the

size n of a file or an array and the amount of time t required
to process the data should be used

► I.e., if there is a linear relationship between the size n and

time t (or t1 = cn1), then an increase of data by factor of 5
results in the increase of the execution time by same factor

► Or if n2 = 5n1, then t2 = 5t1

► Similarly, if t1 = log2n, then doubling n increases t by only

one unit of time, i.e., if t2 = log2(2n), then t2 = t1 + 1
29
 Asymptotic complexity

▪ Any terms that do not substantially change the function’s

magnitude are eliminated from the function

▪ The resulting function gives only an approximate

measure of efficiency of the original function

▪ However, this approximation is sufficiently close to the

original, especially for a function that processes large
quantities of data

▪ Such a measure of efficiency is called asymptotic

complexity

30
Asymptotic complexity – Growth rate

f (n) = n2 + 100n + log10n + 1,000

Growth rate of all terms of function f(n)

31
 Asymptotic Analysis

▪ The asymptotic analysis of an algorithm determines

the running time in a particular notation (Big-O, Omega
or Theta notation)

▪ To perform the analysis

► Find the number of primitive operations executed
as a function of the input size
► Express this function with some notation

▪ For a given input size n, we express the time T to run

the algorithm as a function T(n)

32
 Big-O notation

▪ Most commonly used notation for specifying asymptotic

complexity—i.e., for estimating the rate of function
growth—is the big-O notation introduced in 1894 by
Paul Bachmann

Definition
▪ Given two positive-valued functions f and g:
f (n) is O(g(n))
if there exist positive numbers c and N such that
f (n) ≤ cg(n) for all n ≥ N
f is big-O of g if there is a positive number c such that f
is not larger than cg for sufficiently large ns; that is, for
all ns larger than some number N
33
 Relationship b/w f and g

▪ The relationship between f and g can be expressed by

stating either that g(n) is an upper bound on the value
of f (n) or that, in the long run, f grows at most as fast
as g

c g(n)

f(n)

n
N

cg(n) is an approximation to f(n), bounding from above

34
 Calculating c and g

▪ Usually infinitely many pairs of cs and Ns that can be

given for the same pair of functions f and g

f (n) = 2n2 + 3n + 1 ≤ cg(n) = cn2 = O(n2)

where g(n) = n2, candidate values for c and N are

Different values of c and N for function f (n) = 2n2 + 3n + 1 = O(n2)

calculated according to the definition of big-O

35
 Calculating c and g

▪ We obtain these values by solving the inequality:

2n2 + 3n + 1 ≤ cn2 for different ns

▪ Because it is one inequality with two unknowns,

different pairs of constants c and N for the same
function g(= n2) can be determined

Different values of c and N for function f (n) = 2n2 + 3n + 1 = O(n2)

calculated according to the definition of big-O

36
 Calculating c and g

▪ To choose the best c and N, it should be determined

for which N, a certain term in f becomes the largest
and stays the largest
▪ In f(n), the only candidates for the largest term are 2n2
and 3n; these terms can be compared using the
inequality 2n2 > 3n that holds for n > 1.5
▪ Thus, the chosen values are N = 2 and c ≥ 3(3/4)

Different values of c and N for function f (n) = 2n2 + 3n + 1 = O(n2)

calculated according to the definition of big-O

37
 Practical Significance

▪ All of them are related to the same function g(n) = n2

and to the same f (n)

▪ The point is that f and g grow at the same rate

▪ The definition states, however, that g is almost always

greater than or equal to f if it is multiplied by a constant
c where “almost always” means for all ns not less than
a constant N

▪ The crux of the matter is that the value of c depends on

which N is chosen, and vice versa

38
 Practical Significance

▪ E.g., if 1 is chosen as the value of N—that is, if g is

multiplied by c so that cg(n) will not be less than f right
away—then c has to be equal to 6 or greater

▪ Or if cg(n) is greater than or equal to f (n) starting from

n = 2, then it is enough that c is equal to 3(3/4) = 3.75

Different values of c and N for function f (n) = 2n2 + 3n + 1 = O(n2)

calculated according to the definition of big-O

39
 g(n) vs c

N is always a
point where the
functions cg(n)
and f intersect
each other

function g is plotted with different coefficients c

40
 Best g(n)
▪ The inherent imprecision of the big-O notation goes even
further, because there can be infinitely many functions g for
a given function f

▪ E.g., f (n) = 2n2 + 3n + 1 is big-O not only of n2, but also of

n3, n4, . . . , nk, . . . for any k ≥ 2

▪ To avoid this, the smallest function g is chosen, i.e., n2 in

this case

▪ Also, the approximation of f can go further

► I.e., f (n) = 2n2 + 3n + 1 can be approximated as f(n) =
2n2 + O(n)
► Or the function n2 + 100n + log10n + 1,000 can be
approximated as n2 + 100n + O(log10n)
41
 Big-O Examples

▪ Prove that running time T(n) = n3 + 20n + 1 is O(n3)

Proof:
▪ By the Big-O definition, T(n) is O(n3)
if T(n) ≤ c·n3 for some n ≥ N
▪ Check the condition: n3 + 20n + 1 ≤ c·n3
or equivalently 1 + 20/n2 + 1/n3 ≤ c
▪ Therefore, the Big-O condition holds for n ≥ N = 1 and
c ≥ 22 (= 1 + 20 + 1)
▪ Larger values of N result in smaller factors c (e.g., for
N = 10, c ≥ 1.201 and so on) but in any case the above
statement is valid

42
 Big-O Examples

▪ Prove that running time T(n) = n3 + 20n + 1 is not O(n2)

Proof:
▪ By the Big-O definition, T(n) is O(n2)
if T(n) ≤ c·n2 for some n ≥ N

▪ Check the condition: n3 + 20n + 1 ≤ c·n2

or equivalently n + 20/n + 1/n2 ≤ c

▪ Therefore, the Big-O condition cannot hold since the left

side of the latter inequality is growing infinitely, i.e., there
is no such constant factor c

43
 Big-O Examples

T(n) examples
▪ 3n + 4 → O(n) Linear
▪ 4n2 + 17n + 5344 → O(n2) Quadratic
▪ 6n3+ 3n2 + 5n + 57 → O(n3) Cubic
▪ 15log2(n) + 37 → O(log(n)) Logarithmic
▪ 15 x 2n + 4n57 +16 → O(2n) Exponential
▪ 37 → O(1) Constant time

44
Typical functions applied in Big-O estimates

45
 Execution times of typical functions

Classes of algorithms and their execution times on a computer

executing 1 million operations per second

46
 Big-O Notation

▪ Important
► Big-O is not a function!

► Never read = as "equals”

▪ Examples
► 5n + 3 = O(n)

(i.e., 5n + 3 grows asymptotically no faster than n)

► 7n2 +2n + 1 = O(n2)

(i.e., 7n2 +2n + 1 grows asymptotically no faster than n2)

▪ Big-O is used as an upper bound estimate so it is ok to

say something like running time is at worse O(n)
▪ It is not ok to say “running time is at least O(n)”
47
 Properties of Big-O Notation

▪ Transitive
If f(n) is O(g(n)) and g(n) is O(h(n)) f(n) is O(h(n))
▪ Additive
if f(n) is O(h(n)) and g(n) is O(h(n)) f(n) + g(n) is O(h(n))
▪ Power rule
ank is O(nk) for c ≥ a
▪ Another power rule
nk is O(nk+j ), j>0

It follows from all these facts that every polynomial is big-O of n

raised to the largest power, or
f (n) = aknk + ak-1nk-1 + · · · + a1n + a0 is O(nk)

48
 Other Notations

▪ Two other notations:

► Big Omega Notation 

► Theta Notation 

49