Analysis of Algorithms | Big-O analysis

In our previous articles on Analysis of Algorithms, we had discussed asymptotic notations, their worst
and best case performance etc. in brief. In this article, we discuss analysis of algorithm using Big – O
asymptotic notation in complete details.
Big-O Analysis of Algorithms
The Big O notation defines an upper bound of an algorithm, it bounds a function only from above. For
example, consider the case of Insertion Sort. It takes linear time in best case and quadratic time in worst
case. We can safely say that the time complexity of Insertion sort is O(n^2). Note that O(n^2) also covers
linear time.

The Big-O Asymptotic Notation gives us the Upper Bound Idea, mathematically described below:
f(n) = O(g(n)) if there exists a positive integer n0 and a positive constant c, such that f(n)≤c.g(n) ∀ n≥n0
The general step wise procedure for Big-O runtime analysis is as follows:
1. Figure out what the input is and what n represents.
2. Express the maximum number of operations, the algorithm performs in terms of n.
3. Eliminate all excluding the highest order terms.
4. Remove all the constant factors.
Some of the useful properties on Big-O notation analysis are as follow:
▪ Constant Multiplication:
If f(n) = c.g(n), then O(f(n)) = O(g(n)) ; where c is a nonzero constant.
▪ Polynomial Function:
If f(n) = a0  + a1.n + a2.n2  + —- + am.nm, then O(f(n)) = O(nm).
▪ Summation Function:
If f(n) = f1(n) + f2(n) + —- + fm(n) and fi(n)≤fi+1(n) ∀ i=1, 2, —-, m,
then O(f(n)) = O(max(f1(n), f2(n), —-, fm(n))).
▪ Logarithmic Function:
If f(n) = logan and g(n)=logbn, then O(f(n))=O(g(n))
; all log functions grow in the same manner in terms of Big-O.
Basically, this asymptotic notation is used to measure and compare the worst-case scenarios of
algorithms theoretically. For any algorithm, the Big-O analysis should be straightforward as long as we
correctly identify the operations that are dependent on n, the input size.
Runtime Analysis of Algorithms
In general cases, we mainly used to measure and compare the worst-case theoretical running time
complexities of algorithms for the performance analysis.
The fastest possible running time for any algorithm is O(1), commonly referred to as Constant Running
Time. In this case, the algorithm always takes the same amount of time to execute, regardless of the input
size. This is the ideal runtime for an algorithm, but it’s rarely achievable.
In actual cases, the performance (Runtime) of an algorithm depends on n, that is the size of the input or
the number of operations is required for each input item.
The algorithms can be classified as follows from the best-to-worst performance (Running Time
Complexity):
▪ A logarithmic algorithm – O(logn)
Runtime grows logarithmically in proportion to n.
▪ A linear algorithm – O(n)
Runtime grows directly in proportion to n.
▪ A superlinear algorithm – O(nlogn)
Runtime grows in proportion to n.
▪ A polynomial algorithm – O(nc)
Runtime grows quicker than previous all based on n.
▪ A exponential algorithm – O(cn)
Runtime grows even faster than polynomial algorithm based on n.
▪ A factorial algorithm – O(n!)
Runtime grows the fastest and becomes quickly unusable for even
small values of n.
Where, n is the input size and c is a positive constant.

Algorithmic Examples of Runtime Analysis:

Some of the examples of all those types of algorithms (in worst-case scenarios) are mentioned below:
▪ Logarithmic algorithm – O(logn) – Binary Search.
▪ Linear algorithm – O(n) – Linear Search.
▪ Superlinear algorithm – O(nlogn) – Heap Sort, Merge Sort.
▪ Polynomial algorithm – O(n^c) – Strassen’s Matrix Multiplication, Bubble Sort, Selection Sort, Insertion
Sort, Bucket Sort.
▪ Exponential algorithm – O(c^n) – Tower of Hanoi.
▪ Factorial algorithm – O(n!) – Determinant Expansion by Minors, Brute force Search algorithm for
Traveling Salesman Problem.
Mathematical Examples of Runtime Analysis:
The performances (Runtimes) of different orders of algorithms separate rapidly as n (the input size) gets
larger. Let’s consider the mathematical example:
If n = 10, If n=20,
log(10) = 1; log(20) = 2.996;
10 = 10; 20 = 20;
10log(10)=10; 20log(20)=59.9;
102=100; 202=400;
210=1024; 220=1048576;
10!=3628800; 20!=2.432902e+18 18;
Memory Footprint Analysis of Algorithms
For performance analysis of an algorithm, runtime measurement is not only relevant metric but also we
need to consider the memory usage amount of the program. This is referred to as the Memory Footprint
of the algorithm, shortly known as Space Complexity.
 Here also, we need to measure and compare the worst case theoretical space complexities of algorithms
for the performance analysis.
It basically depends on two major aspects described below:
 Firstly, the implementation of the program is responsible for memory usage. For example, we can
assume that recursive implementation always reserves more memory than the corresponding
iterative implementation of a particular problem.
 And the other one is n, the input size or the amount of storage required for each item. For
example, a simple algorithm with a high amount of input size can consume more memory than a
complex algorithm with less amount of input size.
Algorithmic Examples of Memory Footprint Analysis: The algorithms with examples are classified from the
best-to-worst performance (Space Complexity) based on the worst-case scenarios are mentioned below:
▪ Ideal algorithm - O(1) - Linear Search, Binary Search,
Bubble Sort, Selection Sort, Insertion Sort, Heap Sort, Shell Sort.
▪ Logarithmic algorithm - O(log n) - Merge Sort.
▪ Linear algorithm - O(n) - Quick Sort.
▪ Sub-linear algorithm - O(n+k) - Radix Sort.
Space-Time Tradeoff and Efficiency
There is usually a trade-off between optimal memory use and runtime performance.
In general for an algorithm, space efficiency and time efficiency reach at two opposite ends and each
point in between them has a certain time and space efficiency. So, the more time efficiency you have, the
less space efficiency you have and vice versa.
For example, Mergesort algorithm is exceedingly fast but requires a lot of space to do the operations. On
the other side, Bubble Sort is exceedingly slow but requires the minimum space.
At the end of this topic, we can conclude that finding an algorithm that works in less running time and also
having less requirement of memory space, can make a huge difference in how well an algorithm
performs.

Recommended Posts:

 Analysis of Algorithms | Set 1 (Asymptotic Analysis)

 Analysis of Algorithms | Set 4 (Analysis of Loops)
 Analysis of Algorithm | Set 5 (Amortized Analysis Introduction)
 Analysis of Algorithms | Set 5 (Practice Problems)
 Analysis of algorithms | little o and little omega notations
 Analysis of Algorithms | Set 3 (Asymptotic Notations)
 Analysis of Algorithms | Set 2 (Worst, Average and Best Cases)
 Asymptotic Analysis and comparison of sorting algorithms
 Algorithms Sample Questions | Set 3 | Time Order Analysis
 Algorithms | Analysis of Algorithms | Question 3
 Algorithms | Analysis of Algorithms | Question 19
 Algorithms | Analysis of Algorithms | Question 8
 Algorithms | Analysis of Algorithms | Question 16
 Algorithms | Analysis of Algorithms | Question 13
 Algorithms | Analysis of Algorithms | Question 17
If you like GeeksforGeeks and would like to contribute, you can also write an article
using contribute.geeksforgeeks.org or mail your article to [email protected]. See your
article appearing on the GeeksforGeeks main page and help other Geeks.
Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.

omplexity of Algorithms
Complexity
 The whole point of the big-O/Ω/Θ stuff was to be able
to say something useful about algorithms.
o So, let's return to some algorithms and see if we
learned anything.

 Consider this simple procedure that sums a list (of

numbers, we assume):
 procedure sum(list)
 total = 0
 for i from 0 to length(list)-1
 total += list[i]
return total

o First: is the algorithm correct? Does it solve the

problem specified?
o Second: is it fast?

 To evaluate the running time of an algorithm, we will

simply ask how many “steps” it takes.
o In this case, we can count the number of times it
runs the += line.
o For a list with \(n\) elements, it takes \(n\) steps.

 Or is counting the += line the right thing to do?

 o When implementing the for loop, each iteration
requires an add (for the loop index) and a
comparison (to check the exit condition). We
should count those.
o Also, the variable initialization and return steps.
o So, \(3n+2\) steps.

 But, not all of those steps are the same.

o How long does an x86 ADD instruction take
compared to a CMP or RET instruction?
o Will the compiler keep both i and total in
registers, or will one/both be in RAM? (A factor of
~10 difference.)
o How do those instructions interact in the
pipeline? Which can be sent through parallel
pipelines in the processor and executed
concurrently?
o The answer to those is simple: I don't know and
you don't either.
o That's part of the reason we're asking about
algorithms, not programs.

 But both \(n\) and \(3n+2\) are perfectly reasonable

proposals for the answer.
o Deciding between them requires more knowledge
about the actual implementation details than we
have.

 Good thing we have the function growth notation.

o Remember: this is easy for \(n=5\) elements. A
good or bad algorithm will both be fast then.
o We want to know how the algorithm behaves for
large \(n\).

 Finally our answer: the sum procedure has running time

\(\Theta(n)\).
 o We'll say that this algorithm has time complexity \
(\Theta(n)\), or “runs in linear time”.
o Both \(n\) and \(3n+2\) are \(\Theta(n)\), and so is
any other “exact” formula we could come up with.
o The easy answer (count the += line) was just as
correct as the very careful one.
o The big-Θ notation hides all of the details we
can't figure out anyway.

 Another example: print out the sum of each two

numbers in a list.
o That is, given the list [1,2,3,4,5], we want to find
1+2, 1+3, 1+4, 1+5, 2+3, 2+4,….
o Pseudocode:
o procedure sum_pairs(list)
o for i from 0 to length(list)-2
o for j from i+1 to length(list)-1
print list[i] + list[j]

o For a list with \(n\) elements, the for j loop

iterates \(n-1\) times when it is called with i==0,
then \(n-2\) times, then \(n-3\) times,…
o So, the total number of times the print step runs
is \[\begin{align*} (n-1)+(n-2)+\cdots+2+1 &=
\sum_{k=1}^{n-1} k\\ &= \frac{n(n-1)}{2}\\ &=
\frac{n^2}{2}-\frac{n}{2}\,. \end{align*}\]
o If we had counted the initialization of
the for loops, counter incrementing, etc, we
might have come up with something more like \
(\frac{3}{2}n^2 + \frac{1}{2}n + 1\).
o Either way, the answer we give is that it takes \
(\Theta(n^2)\) steps.
o Or, the algorithm “has time complexity \
(\Theta(n^2)\)” or “has \(\Theta(n^2)\) running
time” or “has quadratic running time”.
 The lesson: when counting running time, you can be a
bit sloppy.
o We only need to worry about the inner-most
loop(s), not the number of steps in there, or work
in the outer levels.
o … because they are going to disappear anyway as
constant factors and lower-order terms when they
go into a big-O/Ω/Θ anyway.

Average and Worst Case

 Consider a linear search: we want to find an element
in a list and return its (first) position, or -1 if it's not
there.
 procedure linear_search(list, value)
 for i from 0 to length(list)-1
 if list[i] == value
 return i
return -1

o How many steps there?

 The answer is: it depends.

o If the thing we're looking for is in the first
position, it takes \(\Theta(1)\) steps.
o If it's at the end, or not there, it takes \(\Theta(n)\)
steps.

 The easiest thing to count is usually the worst case:

what is the maximum steps required for any input of
size \(n\)?
o The worst case is that we go all the way to the
end of the list, but don't find it and return -1.
o The only line that makes sense to count here is
the if line. It's in the inner-most loop, and is
executed for every iteration.
 o We could also count the number
of comparisons made: the == and the implicit
comparison in the for loop.
o That is either \(n\) or \(2n+1\) steps, so \(\Theta(n)\)
complexity.

 The other useful option is the average case: what is

the average steps required over all inputs of size \
(n\)?
o Much harder to calculate, since you need to
consider every possible input to the algorithm.
o Even if we assume the element is found, the
possible number of comparisons are:

Found in position Comparisons

1 2
2 4
⋮ ⋮
\(n\) \(2n\)

o On average, the number of comparisons is: \

[\frac{2+4+\cdots+2n}{n} = n+1\,.\]
o Again, we have \(\Theta(n)\) complexity.
o … but it's a good thing we checked. Some
algorithms are different.

Good/bad times
 We have said that these running times are important
when it comes to running times of algorithm.

 But we are throwing away a lot of information when

we look only at big-O/Ω/Θ.
o The lower-order terms must mean something.
o The leading constants definitely do.
 Assuming one million operations per second, this is
the approximate running time of an algorithm given
running time, with an input of size \(n\):

\(\log_2 \(n\log_2
\(n\) \(n\) \(n^2\) \(n^{3}\) \(2^n\)
n\) n\)
\(10\) 3.3 μs 10 μs 33 μs 100 μs 1 ms 1 ms
100 \(4\times 10^{16}\)
\(10^2\) 6.6 μs 664 μs 10 ms 1s
μs years
10 1.7
\(10^4\) 13 μs 133 ms 11.6 days \(10^{2997}\) years
ms minutes
11.6 32000 \(10^{300000}\)
\(10^6\) 20 μs 1s 20 s
days years years

 Maybe that gives a little idea why we'll only worry

about complexity
o … at least at first.

 A summary:
o If you can get \(O(\log n)\) life is good: hand it in
and go home.
o \(O(n\log n)\) is pretty good: hard to complain
about it.
o \(O(n^k)\) could be bad, depending on \(k\): you
won't be solving huge problems. These
are polynomial complexity algorithms for \(k\ge
1\).
o \(\Omega(k^n)\) is a disaster: almost as bad as no
algorithm at all if you have double-digit input
sizes. These are exponential
complexity algorithms for \(k\gt 1\).
o See also: Numbers everyone should know

 A problem that has a polynomial-time algorithm is

called tractable.
o No polynomial time algorithm: intractable.
o There is a big category of problems that nobody
has a polynomial-time algorithm for, but also can't
 prove that none exists: the NP-complete
problems.
o Some examples: Boolean satisfiability, travelling
salesman, Hamiltonian path, many scheduling
problems, Sudoku (size \(n\)).

 If you have an algorithm with a higher complexity than

necessary, no amount of clever programming will
make up for it.
o No combination of these will make a \(O(n^2)\)
algorithm faster than an \(O(n\log n)\): faster
language, better optimizer, hand-optimization of
code, faster processor.

 Important point: the complexity notations only say

things about large \(n\).
o If you always have small inputs, you might not
care.
o Algorithms with higher complexity class might be
faster in practice, if you always have small
inputs.
o e.g. Insertion sort has running time \(\Theta(n^2)\)
but is generally faster than \(\Theta(n\log n)\)
sorting algorithms for lists of around 10 or fewer
elements.

 The most important info that the complexity notations

throw away is the leading constant.
o There is a difference between \(n^2\) instructions
and \(100n^2\) instructions to solve a problem.
o Once you have the right big-O, then it's time to
worry about the constants.
o That's what clever programming can do.

 When we're talking about algorithms (and not

programming), the constants don't usually matter
much.
 o It's rare to have an algorithm with a big leading
constant.
o So it's not really possible to decide between the
algorithms.
o Usually it's a choice between \(4n\log n\) or \
(5n\log n\): you probably have to implement,
compile, and profile to decide for sure.

 Example: sorting algorithms. There are several

algorithms to sort a list/array.
o Insertion/Selection/Bubble Sorts: \(\Theta(n^2)\).
o Merge/Heap Sorts: \(\Theta(n\log n)\).
o Quicksort: \(\Theta(n\log n)\) average case but
(very rarely) \(\Theta(n^2)\) worst case.
o But quicksort is usually faster in practice.
o … except when it isn't.
o Several recent languages/libraries have
implemented a heavily-optimized mergesort (e.g.
Python, Perl, Java ≥JDK1.3, Haskell, some STL
implementations) instead of Quicksort (C, Ruby,
some other STL implementations).

Space Complexity
 We have only been talking about running time/speed so
far.
o It also makes good sense to talk about the
complexity of other things.

 Most notably, memory use by an algorithm.

o An algorithm that uses \(\Theta(n^{3})\) space is
bad. Maybe as bad as \(\Theta(n^{3})\) time.
o An algorithm that uses \(O(1)\) extra space (in
addition to space needed to store the input) is
called in-place.
 o e.g. selection sort is in-place, but mergesort (\
(\Theta(n)\) extra space) and Quicksort (\
(\Theta(\log n)\) extra space, average case) aren't.

Return to the course notes front page. Copyright © 2013, Greg Baker.