Lecture 02: Time Complexity CS221: Data Structures & Algo.

Time Complexity

Dr. Zahid Halim

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Algorithm

• Finite sequence of instructions.

• Each instruction having a clear meaning.

• Each instruction requiring finite amount of effort.

• Each instruction requiring finite time to complete.

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Algorithm

• Finite sequence of instructions.

An input should not take the program in an infinite loop

• Each instruction having a clear meaning.

Very subjective. What is clear to me, may not be clear to you.

• Each instruction requiring finite amount of effort.

Very subjective. Finite on a super computer or a P4?

• Each instruction requiring finite time to complete.

Very subjective. 1 min, 1 hr, 1 year or a lifetime?

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Algorithm Definition
A finite set of statements that guarantees an optimal solution in
finite interval of time

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Good Algorithms?
• Run in less time
• Consume less memory

But computational resources (time complexity) is usually more

important

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Measuring Efficiency
• The efficiency of an algorithm is a measure of the amount of resources
consumed in solving a problem of size n.

•
The resource we are most interested in is time
• We can use the same techniques to analyze the consumption of other resources,
such as memory space.

• It would seem that the most obvious way to measure the efficiency of an
algorithm is to run it and measure how much processor time is needed

• Is it correct

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Factors
• Hardware
• Operating System
• Compiler
• Size of input
• Nature of Input
• Algorithm

Which should be improved?

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Running Time of an Algorithm

• Depends upon
• Input Size
• Nature of Input
• Generally time grows with size of input, so running time of an
algorithm is usually measured as function of input size.
• Running time is measured in terms of number of steps/primitive
operations performed
• Independent from machine, OS

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Finding running time of an Algorithm / Analyzing an Algorithm

• Running time is measured by number of steps/primitive

operations performed
• Steps means elementary operation like
– ,+, *,<, =, A[i] etc
• We will measure number of steps taken in term of size of input

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Simple Example (1)

// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A

int Sum(int A[], int N)

{
1. int s=0;
2. for (int i=0; i< N; i++)
3. s = s + A[i];
4. return s;
}
How should we analyse this?

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Simple Example (2)

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int N){

int s=0; 1
for (int i=0; i< N; i++)
2 3 4
s = s + A[i];
5 6 7 1,2,8: Once
return s; 3,4,5,6,7: Once per each iteration
}
8 of for loop, N iteration
Total: 5N + 3
The complexity function of the
algorithm is : f(N) = 5N +3

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.
Simple Example (3)
Growth of 5n+3
Estimated running time for different values of N:

N = 10 => 53 steps
N = 100 => 503 steps
N = 1,000 => 5003 steps
N = 1,000,000 => 5,000,003 steps

As N grows, the number of steps grow in linear proportion to N for this

function “Sum”

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

What Dominates in Previous Example?

What about the +3 and 5 in 5N+3?
– As N gets large, the +3 becomes insignificant
– 5 is inaccurate, as different operations require varying amounts of time and also
does not have any significant importance

What is fundamental is that the time is linear in N.

Asymptotic Complexity: As N gets large, concentrate on the highest order
term:
• Drop lower order terms such as +3
• Drop the constant coefficient of the highest order term i.e. N

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Be careful to differentiate between

Performance: how much time/memory/disk/... is actually used
when a program is run. This depends on the machine, compiler,
etc. as well as the code.

Complexity: how do the resource requirements of a program or

algorithm scale, i.e., what happens as the size of the problem
being solved gets larger.

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Asymptotic Complexity
• The 5N+3 time bound is said to "grow asymptotically" like N
• This gives us an approximation of the complexity of the
algorithm
• Ignores lots of (machine dependent) details, concentrate on
the bigger picture

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Comparing Functions: Asymptotic Notation

• Big Oh Notation: Upper bound

• Omega Notation: Lower bound
• Theta Notation: Tighter bound

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Big Oh Notation
If f(N) and g(N) are two complexity functions, we say

f(N) = O(g(N))

(read "f(N) is order g(N)", or "f(N) is big-O of g(N)")

if there are constants c and N0 such that for N > N0,
f(N) > c * g(N)
for all sufficiently large N.
• O(f(n)) =
{g(n) : there exists positive constants c and n0 such that 0 <= g(n) <= c f(n) }
• O(f(n)) is a set of functions.
• n = O(n2) means that function n belongs to the set of functions O(n2)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Example : Comparing Functions

4000
• Which function is better?
10 n2 Vs n3 3500

3000

2500

10 n^2
2000
n^3

1500

1000

500

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Comparing Functions
• As inputs get larger, any algorithm of a smaller order will be more efficient than an
algorithm of a larger order

0.05 N2 = O(N2)
Time (steps)

3N = O(N)

N = 60 Input (size)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Big-Oh Notation
• Even though it is correct to say “7n - 3 is O(n3)”, a better statement
is “7n - 3 is O(n)”, that is, one should make the approximation as tight as
possible
• Simple Rule:
Drop lower order terms and constant factors
7n-3 is O(n)
8n2log n + 5n2 + n is O(n2log n)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Performance Classification
f(n) Classification

1 Constant: run time is fixed, and does not depend upon n. Most instructions are executed once, or only a few times,
regardless of the amount of information being processed

log n Logarithmic: when n increases, so does run time, but much slower. Common in programs which solve large problems
by transforming them into smaller problems.

n Linear: run time varies directly with n. Typically, a small amount of processing is done on each element.

n log n When n doubles, run time slightly more than doubles. Common in programs which break a problem down into smaller
sub-problems, solves them independently, then combines solutions

n2 Quadratic: when n doubles, runtime increases fourfold. Practical only for small problems; typically the program
processes all pairs of input (e.g. in a double nested loop).

n3 Cubic: when n doubles, runtime increases eightfold

2n Exponential: when n doubles, run time squares. This is often the result of a natural, “brute force” solution.

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Size does matter

What happens if we double the input size N?

N log2N 5N N log2N N2 2N
8 3 40 24 64 256
16 4 80 64 256 65536
32 5 160 160 1024 ~109
64 6 320 384 4096 ~1019
128 7 640 896 16384 ~1038
256 8 1280 2048 65536 ~1076

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Size does matter

• Suppose a program has run time O(n!) and the run time for
n = 10 is 1 second

For n = 12, the run time is 2 minutes

For n = 14, the run time is 6 hours
For n = 16, the run time is 2 months
For n = 18, the run time is 50 years
For n = 20, the run time is 200 centuries

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Standard Analysis Techniques

• Constant time statements
• Analyzing Loops
• Analyzing Nested Loops
• Analyzing Sequence of Statements
• Analyzing Conditional Statements

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Constant time statements

• Simplest case: O(1) time statements
• Assignment statements of simple data types
int x = y;
• Arithmetic operations:
x = 5 * y + 4 - z;
• Array referencing:
A[j] = 5;
• Array assignment:
 j, A[j] = 5;
• Most conditional tests:
if (x < 12) ...

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Analyzing Loops
• Any loop has two parts:
– How many iterations are performed?
– How many steps per iteration?

int sum = 0,j;

for (j=0; j < N; j++)
sum = sum +j;
– Loop executes N times (0..N-1)
– 4 = O(1) steps per iteration
• Total time is N * O(1) = O(N*1) = O(N)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Analyzing Loops

• What about this for loop?

int sum =0, j;
for (j=0; j < 100; j++)
sum = sum +j;
• Loop executes 100 times
• 4 = O(1) steps per iteration
• Total time is 100 * O(1) = O(100 * 1) = O(100) = O(1)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Analyzing Nested Loops

• Treat just like a single loop and evaluate each level of nesting as needed:
int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;
• Start with outer loop:
– How many iterations? N
– How much time per iteration? Need to evaluate inner loop
• Inner loop uses O(N) time
• Total time is N * O(N) = O(N*N) = O(N2)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Analyzing Nested Loops

• What if the number of iterations of one loop depends on the
counter of the other?
int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;
• Analyze inner and outer loop together:
• Number of iterations of the inner loop is:
• 0 + 1 + 2 + ... + (N-1) = O(N2)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Analyzing Sequence of Statements

• For a sequence of statements, compute their complexity
functions individually and add them up
for (j=0; j < N; j++)
for (k =0; k < j; k++)
O(N2)
sum = sum + j*k;
for (l=0; l < N; l++) O(N)
sum = sum -l;
cout<<“Sum=”<<sum; O(1)

Total cost is O(N2) + O(N) +O(1) = O(N2)

SUM RULE
Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi
Lecture 02: Time Complexity CS221: Data Structures & Algo.

Analyzing Conditional Statements

What about conditional statements such as

if (condition)
statement1;
else
statement2;
where statement1 runs in O(N) time and statement2 runs in O(N2) time?

We use "worst case" complexity: among all inputs of size N, that is the maximum
running time?
The analysis for the example above is O(N2)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi

Lecture 02: Time Complexity CS221: Data Structures & Algo.

Time complexity familiar tasks

Task Growth rate

Getting a specific element from a list (array) O(1)
Dividing a list in half, dividing one halve in half, etc O(log2N)
Binary Search O(log2N)
Scanning (brute force search) a list O(N)
Nested for loops (k levels) O(Nk)
MergeSort O(N log2N)
BubbleSort O(N2)
Generate all subsets of a set of data O(2N)
Generate all permutations of a set of data O(N!)

Ghulam Ishaq Khan Institute of Engineering Sciences and Technology, Topi