The objective of this chapter is to explain the importance of the analysis of algorithms, their

notations, relationships and solving as many problems as possible. Let us first focus on
understanding the basic elements of algorithms, the importance of algorithm analysis, and then
slowly move toward the other topics as mentioned above. After completing this chapter, you
should be able to find the complexity of any given algorithm (especially recursive functions).

1.1 Variables

Before going to the definition of variables, let us relate them to old mathematical equations. All of
us have solved many mathematical equations since childhood. As an example, consider the below
equation:
We don’t have to worry about the use of this equation. The important thing that we need to
understand is that the equation has names (x and y), which hold values (data). That means the
names (x and y) are placeholders for representing data. Similarly, in computer science
programming we need something for holding data, and variables is the way to do that.

1.2 Data Types

In the above-mentioned equation, the variables x and y can take any values such as integral
numbers (10, 20), real numbers (0.23, 5.5), or just 0 and 1. To solve the equation, we need to
relate them to the kind of values they can take, and data type is the name used in computer science
programming for this purpose. A data type in a programming language is a set of data with
predefined values. Examples of data types are: integer, floating point, unit number, character,
string, etc.

Computer memory is all filled with zeros and ones. If we have a problem and we want to code it,
it’s very difficult to provide the solution in terms of zeros and ones. To help users, programming
languages and compilers provide us with data types. For example, integer takes 2 bytes (actual
value depends on compiler), float takes 4 bytes, etc. This says that in memory we are combining
2 bytes (16 bits) and calling it an integer. Similarly, combining 4 bytes (32 bits) and calling it a
float. A data type reduces the coding effort. At the top level, there are two types of data types:
• System-defined data types (also called Primitive data types)
• User-defined data types

System-defined data types (Primitive data types)

Data types that are defined by system are called primitive data types. The primitive data types
provided by many programming languages are: int, float, char, double, bool, etc. The number of
bits allocated for each primitive data type depends on the programming languages, the compiler
and the operating system. For the same primitive data type, different languages may use different
sizes. Depending on the size of the data types, the total available values (domain) will also
change.

For example, “int” may take 2 bytes or 4 bytes. If it takes 2 bytes (16 bits), then the total possible
values are minus 32,768 to plus 32,767 (-215 to 215-1). If it takes 4 bytes (32 bits), then the
possible values are between -2,147,483,648 and +2,147,483,647 (-231 to 231-1). The same is the
case with other data types.

User defined data types

If the system-defined data types are not enough, then most programming languages allow the users
to define their own data types, called user – defined data types. Good examples of user defined
data types are: structures in C/C + + and classes in Java. For example, in the snippet below, we
are combining many system-defined data types and calling the user defined data type by the name
“newType”. This gives more flexibility and comfort in dealing with computer memory.

1.3 Data Structures

Based on the discussion above, once we have data in variables, we need some mechanism for
manipulating that data to solve problems. Data structure is a particular way of storing and
organizing data in a computer so that it can be used efficiently. A data structure is a special
format for organizing and storing data. General data structure types include arrays, files, linked
lists, stacks, queues, trees, graphs and so on.

Depending on the organization of the elements, data structures are classified into two types:
1) Linear data structures: Elements are accessed in a sequential order but it is not
compulsory to store all elements sequentially. Examples: Linked Lists, Stacks and
Queues.
2) Non – linear data structures: Elements of this data structure are stored/accessed in a
non-linear order. Examples: Trees and graphs.

1.4 Abstract Data Types (ADTs)

Before defining abstract data types, let us consider the different view of system-defined data
types. We all know that, by default, all primitive data types (int, float, etc.) support basic
operations such as addition and subtraction. The system provides the implementations for the
primitive data types. For user-defined data types we also need to define operations. The
implementation for these operations can be done when we want to actually use them. That means,
in general, user defined data types are defined along with their operations.

To simplify the process of solving problems, we combine the data structures with their operations
and we call this Abstract Data Types (ADTs). An ADT consists of two parts:
1. Declaration of data
 2. Declaration of operations

Commonly used ADTs include: Linked Lists, Stacks, Queues, Priority Queues, Binary Trees,
Dictionaries, Disjoint Sets (Union and Find), Hash Tables, Graphs, and many others. For
example, stack uses LIFO (Last-In-First-Out) mechanism while storing the data in data structures.
The last element inserted into the stack is the first element that gets deleted. Common operations
of it are: creating the stack, pushing an element onto the stack, popping an element from stack,
finding the current top of the stack, finding number of elements in the stack, etc.

While defining the ADTs do not worry about the implementation details. They come into the
picture only when we want to use them. Different kinds of ADTs are suited to different kinds of
applications, and some are highly specialized to specific tasks. By the end of this book, we will
go through many of them and you will be in a position to relate the data structures to the kind of
problems they solve.

1.5 What is an Algorithm?

Let us consider the problem of preparing an omelette. To prepare an omelette, we follow the
steps given below:
1) Get the frying pan.
2) Get the oil.
a. Do we have oil?
i. If yes, put it in the pan.
ii. If no, do we want to buy oil?
1. If yes, then go out and buy.
2. If no, we can terminate.
3) Turn on the stove, etc...

What we are doing is, for a given problem (preparing an omelette), we are providing a step-by-
step procedure for solving it. The formal definition of an algorithm can be stated as:

An algorithm is the step-by-step unambiguous instructions to solve a given problem.

In the traditional study of algorithms, there are two main criteria for judging the merits of
algorithms: correctness (does the algorithm give solution to the problem in a finite number of
steps?) and efficiency (how much resources (in terms of memory and time) does it take to execute
the).

Note: We do not have to prove each step of the algorithm.

1.6 Why the Analysis of Algorithms?

To go from city “A” to city “B”, there can be many ways of accomplishing this: by flight, by bus,
by train and also by bicycle. Depending on the availability and convenience, we choose the one
that suits us. Similarly, in computer science, multiple algorithms are available for solving the
same problem (for example, a sorting problem has many algorithms, like insertion sort, selection
sort, quick sort and many more). Algorithm analysis helps us to determine which algorithm is
most efficient in terms of time and space consumed.

1.7 Goal of the Analysis of Algorithms

The goal of the analysis of algorithms is to compare algorithms (or solutions) mainly in terms of
running time but also in terms of other factors (e.g., memory, developer effort, etc.)

1.8 What is Running Time Analysis?

It is the process of determining how processing time increases as the size of the problem (input
size) increases. Input size is the number of elements in the input, and depending on the problem
type, the input may be of different types. The following are the common types of inputs.
• Size of an array
• Polynomial degree
• Number of elements in a matrix
• Number of bits in the binary representation of the input
• Vertices and edges in a graph.

1.9 How to Compare Algorithms

To compare algorithms, let us define a few objective measures:

Execution times? Not a good measure as execution times are specific to a particular computer.

Number of statements executed? Not a good measure, since the number of statements varies
with the programming language as well as the style of the individual programmer.

Ideal solution? Let us assume that we express the running time of a given algorithm as a function
of the input size n (i.e., f(n)) and compare these different functions corresponding to running
times. This kind of comparison is independent of machine time, programming style, etc.

1.10 What is Rate of Growth?

The rate at which the running time increases as a function of input is called rate of growth. Let us
assume that you go to a shop to buy a car and a bicycle. If your friend sees you there and asks
what you are buying, then in general you say buying a car. This is because the cost of the car is
high compared to the cost of the bicycle (approximating the cost of the bicycle to the cost of the
car).

For the above-mentioned example, we can represent the cost of the car and the cost of the bicycle
in terms of function, and for a given function ignore the low order terms that are relatively
insignificant (for large value of input size, n). As an example, in the case below, n4, 2n2, 100n
and 500 are the individual costs of some function and approximate to n4 since n4 is the highest
rate of growth.

1.11 Commonly Used Rates of Growth

The diagram below shows the relationship between different rates of growth.
Below is the list of growth rates you will come across in the following chapters.
1.12 Types of Analysis

To analyze the given algorithm, we need to know with which inputs the algorithm takes less time
(performing wel1) and with which inputs the algorithm takes a long time. We have already seen
that an algorithm can be represented in the form of an expression. That means we represent the
algorithm with multiple expressions: one for the case where it takes less time and another for the
case where it takes more time.

In general, the first case is called the best case and the second case is called the worst case for
the algorithm. To analyze an algorithm we need some kind of syntax, and that forms the base for
asymptotic analysis/notation. There are three types of analysis:
• Worst case
○ Defines the input for which the algorithm takes a long time (slowest
time to complete).
○ Input is the one for which the algorithm runs the slowest.
• Best case
○ Defines the input for which the algorithm takes the least time (fastest
time to complete).
○ Input is the one for which the algorithm runs the fastest.
• Average case
○ Provides a prediction about the running time of the algorithm.
○ Run the algorithm many times, using many different inputs that come
from some distribution that generates these inputs, compute the total
running time (by adding the individual times), and divide by the
number of trials.
○ Assumes that the input is random.

Lower Bound <= Average Time <= Upper Bound

For a given algorithm, we can represent the best, worst and average cases in the form of
expressions. As an example, let f(n) be the function which represents the given algorithm.

Similarly for the average case. The expression defines the inputs with which the algorithm takes
the average running time (or memory).

1.13 Asymptotic Notation

Having the expressions for the best, average and worst cases, for all three cases we need to
identify the upper and lower bounds. To represent these upper and lower bounds, we need some
kind of syntax, and that is the subject of the following discussion. Let us assume that the given
algorithm is represented in the form of function f(n).

1.14 Big-O Notation [Upper Bounding Function]

This notation gives the tight upper bound of the given function. Generally, it is represented as f(n)
= O(g(n)). That means, at larger values of n, the upper bound of f(n) is g(n). For example, if f(n)
= n4 + 100n2 + 10n + 50 is the given algorithm, then n4 is g(n). That means g(n) gives the
maximum rate of growth for f(n) at larger values of n.
Let us see the O–notation with a little more detail. O–notation defined as O(g(n)) = {f(n): there
exist positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) for all n > n0}. g(n) is an asymptotic
tight upper bound for f(n). Our objective is to give the smallest rate of growth g(n) which is
greater than or equal to the given algorithms’ rate of growth /(n).

Generally we discard lower values of n. That means the rate of growth at lower values of n is not
important. In the figure, n0 is the point from which we need to consider the rate of growth for a
given algorithm. Below n0, the rate of growth could be different. n0 is called threshold for the
given function.

Big-O Visualization

O(g(n)) is the set of functions with smaller or the same order of growth as g(n). For example;
O(n2) includes O(1), O(n), O(nlogn), etc.

Note: Analyze the algorithms at larger values of n only. What this means is, below n0 we do not
care about the rate of growth.

Big-O Examples

Example-1 Find upper bound for f(n) = 3n + 8

Solution: 3n + 8 ≤ 4n, for all n ≥ 8
∴ 3n + 8 = O(n) with c = 4 and n0 = 8
Example-2 Find upper bound for f(n) = n2 + 1
Solution: n2 + 1 ≤ 2n2, for all n ≥ 1
∴ n2 + 1 = O(n2) with c = 2 and n0 = 1

Example-3 Find upper bound for f(n) = n4 + 100n2 + 50

Solution: n4 + 100n2 + 50 ≤ 2n4, for all n ≥ 11
∴ n4 + 100n2 + 50 = O(n4 ) with c = 2 and n0 = 11

Example-4 Find upper bound for f(n) = 2n3 – 2n2

Solution: 2n3 – 2n2 ≤ 2n3, for all n > 1
∴ 2n3 – 2n2 = O(n3 ) with c = 2 and n0 = 1
Example-5 Find upper bound for f(n) = n
Solution: n ≤ n, for all n ≥ 1
∴ n = O(n) with c = 1 and n0 = 1
Example-6 Find upper bound for f(n) = 410
Solution: 410 ≤ 410, for all n > 1
∴ 410 = O(1) with c = 1 and n0 = 1

No Uniqueness?

There is no unique set of values for n0 and c in proving the asymptotic bounds. Let us consider,
100n + 5 = O(n). For this function there are multiple n0 and c values possible.
Solution1: 100n + 5 ≤ 100n + n = 101n ≤ 101n, for all n ≥ 5, n0 = 5 and c = 101 is a solution.
Solution2: 100n + 5 ≤ 100n + 5n = 105n ≤ 105n, for all n > 1, n0 = 1 and c = 105 is also a
solution.

1.15 Omega-Q Notation [Lower Bounding Function]

Similar to the O discussion, this notation gives the tighter lower bound of the given algorithm and
we represent it as f(n) = Ω(g(n)). That means, at larger values of n, the tighter lower bound of
f(n) is g(n). For example, if f(n) = 100n2 + 10n + 50, g(n) is Ω(n2).
The Ω notation can be defined as Ω(g(n)) = {f(n): there exist positive constants c and n0 such that
0 ≤ cg(n) ≤ f(n) for all n ≥ n0}. g(n) is an asymptotic tight lower bound for f(n). Our objective is
to give the largest rate of growth g(n) which is less than or equal to the given algorithm’s rate of
growth f(n).

Ω Examples

Example-1 Find lower bound for f(n) = 5n2.

Solution: ∃ c, n0 Such that: 0 ≤ cn2≤ 5n2 ⇒ cn2 ≤ 5n2 ⇒ c = 5 and n0 = 1
∴ 5n2 = Ω(n2) with c = 5 and n0 = 1

Example-2 Prove f(n) = 100n + 5 ≠ Ω(n2).

Solution: ∃ c, n0 Such that: 0 ≤ cn2 ≤ 100n + 5
100n + 5 ≤ 100n + 5n(∀n ≥ 1) = 105n
cn2 ≤ 105n ⇒ n(cn - 105) ≤ 0
Since n is positive ⇒cn - 105 ≤0 ⇒ n ≤105/c
⇒ Contradiction: n cannot be smaller than a constant

Example-3 2n = Q(n), n3 = Q(n3), = O(logn).

1.16 Theta-Θ Notation [Order Function]

This notation decides whether the upper and lower bounds of a given function (algorithm) are the
same. The average running time of an algorithm is always between the lower bound and the upper
bound. If the upper bound (O) and lower bound (Ω) give the same result, then the Θ notation will
also have the same rate of growth.

As an example, let us assume that f(n) = 10n + n is the expression. Then, its tight upper bound
g(n) is O(n). The rate of growth in the best case is g(n) = O(n).

In this case, the rates of growth in the best case and worst case are the same. As a result, the
average case will also be the same. For a given function (algorithm), if the rates of growth
(bounds) for O and Ω are not the same, then the rate of growth for the Θ case may not be the same.
In this case, we need to consider all possible time complexities and take the average of those (for
example, for a quick sort average case, refer to the Sorting chapter).

Now consider the definition of Θ notation. It is defined as Θ(g(n)) = {f(n): there exist positive
constants c1,c2 and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0}. g(n) is an asymptotic
tight bound for f(n). Θ(g(n)) is the set of functions with the same order of growth as g(n).

Θ Examples
Example 1 Find Θ bound for

Solution: for all, n ≥ 2

∴ with c1 = 1/5,c2 = 1 and n0 = 2

Example 2 Prove n ≠ Θ(n2)

Solution: c1 n2 ≤ n ≤ c2n2 ⇒ only holds for: n ≤ 1/c1
∴ n ≠ Θ(n2)

Example 3 Prove 6n3 ≠ Θ(n2)

Solution: c1 n2≤ 6n3 ≤ c2 n2 ⇒ only holds for: n ≤ c2 /6
∴ 6n3 ≠ Θ(n2)

Example 4 Prove n ≠ Θ(logn)

Solution: c1logn ≤ n ≤ c2logn ⇒ c2 ≥ , ∀ n ≥ n0 – Impossible

1.17 Important Notes

For analysis (best case, worst case and average), we try to give the upper bound (O) and lower
bound (Ω) and average running time (Θ). From the above examples, it should also be clear that,
for a given function (algorithm), getting the upper bound (O) and lower bound (Ω) and average
running time (Θ) may not always be possible. For example, if we are discussing the best case of
an algorithm, we try to give the upper bound (O) and lower bound (Ω) and average running time
(Θ).

In the remaining chapters, we generally focus on the upper bound (O) because knowing the lower
bound (Ω) of an algorithm is of no practical importance, and we use the Θ notation if the upper
bound (O) and lower bound (Ω) are the same.

1.18 Why is it called Asymptotic Analysis?

From the discussion above (for all three notations: worst case, best case, and average case), we
can easily understand that, in every case for a given function f(n) we are trying to find another
function g(n) which approximates f(n) at higher values of n. That means g(n) is also a curve
which approximates f(n) at higher values of n.

In mathematics we call such a curve an asymptotic curve. In other terms, g(n) is the asymptotic
curve for f(n). For this reason, we call algorithm analysis asymptotic analysis.

1.19 Guidelines for Asymptotic Analysis

There are some general rules to help us determine the running time of an algorithm.

1) Loops: The running time of a loop is, at most, the running time of the statements
inside the loop (including tests) multiplied by the number of iterations.

Total time = a constant c × n = c n = O(n).

2) Nested loops: Analyze from the inside out. Total running time is the product of the
sizes of all the loops.

Total time = c × n × n = cn2 = O(n2).

3) Consecutive statements: Add the time complexities of each statement.

Total time = c0 + c1n + c2n2 = O(n2).

4) If-then-else statements: Worst-case running time: the test, plus either the then part
or the else part (whichever is the larger).

Total time = c0 + c1 + (c2 + c3) * n = O(n).

5) Logarithmic complexity: An algorithm is O(logn) if it takes a constant time to cut

the problem size by a fraction (usually by ½). As an example let us consider the
following program:
 If we observe carefully, the value of i is doubling every time. Initially i = 1, in next step i
= 2, and in subsequent steps i = 4,8 and so on. Let us assume that the loop is executing
some k times. At kth step 2k = n, and at (k + 1)th step we come out of the loop. Taking
logarithm on both sides, gives

Total time = O(logn).

Note: Similarly, for the case below, the worst case rate of growth is O(logn). The same
discussion holds good for the decreasing sequence as well.

Another example: binary search (finding a word in a dictionary of n pages)

• Look at the center point in the dictionary
• Is the word towards the left or right of center?
• Repeat the process with the left or right part of the dictionary until the word is found.

1.20 Simplyfying properties of asymptotic notations

• Transitivity: f(n) = Θ(g(n)) and g(n) = Θ(h(n)) ⇒ f(n) = Θ(h(n)). Valid for O and Ω
as well.
• Reflexivity: f(n) = Θ(f(n)). Valid for O and Ω.
• Symmetry: f(n) = Θ(g(n)) if and only if g(n) = Θ(f(n)).
• Transpose symmetry: f(n) = O(g(n)) if and only if g(n) = Ω(f(n)).
• If f(n) is in O(kg(n)) for any constant k > 0, then f(n) is in O(g(n)).
• If f1(n) is in O(g1(n)) and f2(n) is in O(g2(n)), then (f1 + f2)(n) is in O(max(g1(n)),
(g1(n))).
• If f1(n) is in O(g1(n)) and f2(n) is in O(g2(n)) then f1(n) f2(n) is in O(g1(n) g1(n)).

1.21 Commonly used Logarithms and Summations

Logarithms
Arithmetic series

Geometric series

Harmonic series

Other important formulae

1.22 Master Theorem for Divide and Conquer Recurrences

All divide and conquer algorithms (also discussed in detail in the Divide and Conquer chapter)
divide the problem into sub-problems, each of which is part of the original problem, and then
perform some additional work to compute the final answer. As an example, a merge sort
algorithm [for details, refer to Sorting chapter] operates on two sub-problems, each of which is
half the size of the original, and then performs O(n) additional work for merging. This gives the
running time equation:

The following theorem can be used to determine the running time of divide and conquer
algorithms. For a given program (algorithm), first we try to find the recurrence relation for the
problem. If the recurrence is of the below form then we can directly give the answer without fully
solving it. If the recurrence is of the form , where a ≥ 1,b >
1,k ≥ 0 and p is a real number, then:
1) If a > bk , then
2) If a= bk
a. If p > –1, then
b. If p = –1, then
c. If p < –1, then
3) If a < bk
a. If p ≥ 0, then T(n) = Θ(nk logpn)
b. If p < 0, then T(n) = O(nk )

1.23 Divide and Conquer Master Theorem: Problems & Solutions

For each of the following recurrences, give an expression for the runtime T(n) if the recurrence
can be solved with the Master Theorem. Otherwise, indicate that the Master Theorem does not
apply.

Problem-1 T(n) = 3T (n/2) + n2

Solution: T(n) = 3T (n/2) + n2 => T (n) =Θ(n2) (Master Theorem Case 3.a)

Problem-2 T(n) = 4T (n/2) + n2

Solution: T(n) = 4T (n/2) + n2 => T (n) = Θ(n2logn) (Master Theorem Case 2.a)

Problem-3 T(n) = T(n/2) + n2

Solution: T(n) = T(n/2) + n2 => Θ(n2) (Master Theorem Case 3.a)

Problem-4 T(n) = 2nT(n/2) + nn

Solution: T(n) = 2nT(n/2) + nn => Does not apply (a is not constant)
Problem-5 T(n) = 16T(n/4) + n
Solution: T(n) = 16T (n/4) + n => T(n) = Θ(n2) (Master Theorem Case 1)
Problem-6 T(n) = 2T(n/2) + nlogn
Solution: T(n) = 2T(n/2) + nlogn => T(n) = Θ(nlog2n) (Master Theorem Case 2.a)
Problem-7 T(n) = 2T(n/2) + n/logn
Solution: T(n) = 2T(n/2)+ n/logn =>T(n) = Θ(nloglogn) (Master Theorem Case 2. b)

Problem-8 T(n) = 2T (n/4) + n051

Solution: T(n) = 2T(n/4) + n051 => T (n) = Θ(n0.51) (Master Theorem Case 3.b)
Problem-9 T(n) = 0.5T(n/2) + 1/n
Solution: T(n) = 0.5T(n/2) + 1/n => Does not apply (a < 1)

Problem-10 T (n) = 6T(n/3)+ n2 logn

Solution: T(n) = 6T(n/3) + n2logn => T(n) = Θ(n2logn) (Master Theorem Case 3.a)

Problem-11 T(n) = 64T(n/8) – n2logn

Solution: T(n) = 64T(n/8) – n2logn => Does not apply (function is not positive)

Problem-12 T(n) = 7T(n/3) + n2

Solution: T(n) = 7T(n/3) + n2 => T(n) = Θ(n2) (Master Theorem Case 3.as)
Problem-13 T(n) = 4T(n/2) + logn
Solution: T(n) = 4T(n/2) + logn => T(n) = Θ(n2) (Master Theorem Case 1)
Problem-14 T(n) = 16T (n/4) + n!
Solution: T(n) = 16T (n/4) + n! => T(n) = Θ(n!) (Master Theorem Case 3.a)
Problem-15 T(n) = T(n/2) + logn
Solution: T(n) = T(n/2) + logn => T(n) = Θ( ) (Master Theorem Case 1)

Problem-16 T(n) = 3T(n/2) + n

Solution: T(n) = 3T(n/2) + n =>T(n) = Θ(nlog3) (Master Theorem Case 1)
Problem-17 T(n) = 3T(n/3) +
Solution: T(n) = 3T(n/3) + => T(n) = Θ(n) (Master Theorem Case 1)

Problem-18 T(n) = 4T(n/2) + cn

Solution: T(n) = 4T(n/2) + cn => T(n) = Θ(n2) (Master Theorem Case 1)
Problem-19 T(n) = 3T(n/4) + nlogn
Solution: T(n) = 3T(n/4) + nlogn => T(n) = Θ(nlogn) (Master Theorem Case 3.a)
Problem-20 T (n) = 3T(n/3) + n/2
Solution: T(n) = 3T(n/3)+ n/2 => T (n) = Θ(nlogn) (Master Theorem Case 2.a)

1.24 Master Theorem for Subtract and Conquer Recurrences

Let T(n) be a function defined on positive n, and having the property

for some constants c,a > 0,b ≥ 0,k ≥ 0, and function f(n). If f(n) is in O(nk ), then

1.25 Variant of Subtraction and Conquer Master Theorem

The solution to the equation T(n) = T(α n) + T((1 – α)n) + βn, where 0 < α < 1 and β > 0 are
constants, is O(nlogn).

1.26 Method of Guessing and Confirming

Now, let us discuss a method which can be used to solve any recurrence. The basic idea behind
this method is:

guess the answer; and then prove it correct by induction.

In other words, it addresses the question: What if the given recurrence doesn’t seem to match with
any of these (master theorem) methods? If we guess a solution and then try to verify our guess
inductively, usually either the proof will succeed (in which case we are done), or the proof will
fail (in which case the failure will help us refine our guess).

As an example, consider the recurrence . This doesn’t fit into the form
required by the Master Theorems. Carefully observing the recurrence gives us the impression that
it is similar to the divide and conquer method (dividing the problem into subproblems each
with size ). As we can see, the size of the subproblems at the first level of recursion is n. So,
let us guess that T(n) = O(nlogn), and then try to prove that our guess is correct.

Let’s start by trying to prove an upper bound T(n) < cnlogn:

The last inequality assumes only that 1 ≤ c. .logn. This is correct if n is sufficiently large and for
any constant c, no matter how small. From the above proof, we can see that our guess is correct
for the upper bound. Now, let us prove the lower bound for this recurrence.

The last inequality assumes only that 1 ≥ k. .logn. This is incorrect if n is sufficiently large and
for any constant k. From the above proof, we can see that our guess is incorrect for the lower
bound.

From the above discussion, we understood that Θ(nlogn) is too big. How about Θ(n)? The lower
bound is easy to prove directly:

Now, let us prove the upper bound for this Θ(n).

From the above induction, we understood that Θ(n) is too small and Θ(nlogn) is too big. So, we
need something bigger than n and smaller than nlogn. How about ?

Proving the upper bound for :

Proving the lower bound for :

The last step doesn’t work. So, Θ( ) doesn’t work. What else is between n and nlogn?
How about nloglogn? Proving upper bound for nloglogn:

Proving lower bound for nloglogn:

From the above proofs, we can see that T(n) ≤ cnloglogn, if c ≥ 1 and T(n) ≥ knloglogn, if k ≤ 1.
Technically, we’re still missing the base cases in both proofs, but we can be fairly confident at
this point that T(n) = Θ(nloglogn).

1.27 Amortized Analysis

Amortized analysis refers to determining the time-averaged running time for a sequence of
operations. It is different from average case analysis, because amortized analysis does not make
any assumption about the distribution of the data values, whereas average case analysis assumes
the data are not “bad” (e.g., some sorting algorithms do well on average over all input orderings
but very badly on certain input orderings). That is, amortized analysis is a worst-case analysis,
but for a sequence of operations rather than for individual operations.

The motivation for amortized analysis is to better understand the running time of certain
techniques, where standard worst case analysis provides an overly pessimistic bound. Amortized
analysis generally applies to a method that consists of a sequence of operations, where the vast
majority of the operations are cheap, but some of the operations are expensive. If we can show
that the expensive operations are particularly rare we can change them to the cheap operations,
and only bound the cheap operations.

The general approach is to assign an artificial cost to each operation in the sequence, such that the
total of the artificial costs for the sequence of operations bounds the total of the real costs for the
sequence. This artificial cost is called the amortized cost of an operation. To analyze the running
time, the amortized cost thus is a correct way of understanding the overall running time – but note
that particular operations can still take longer so it is not a way of bounding the running time of
any individual operation in the sequence.

When one event in a sequence affects the cost of later events:

• One particular task may be expensive.
• But it may leave data structure in a state that the next few operations become easier.

Example: Let us consider an array of elements from which we want to find the kth smallest
element. We can solve this problem using sorting. After sorting the given array, we just need to
return the kth element from it. The cost of performing the sort (assuming comparison based sorting
algorithm) is O(nlogn). If we perform n such selections then the average cost of each selection is
O(nlogn/n) = O(logn). This clearly indicates that sorting once is reducing the complexity of
subsequent operations.

1.28 Algorithms Analysis: Problems & Solutions

Note: From the following problems, try to understand the cases which have different
complexities (O(n), O(logn), O(loglogn) etc.).
Problem-21 Find the complexity of the below recurrence:
Solution: Let us try solving this function with substitution.
T(n) = 3T(n – 1)

T(n) = 3(3T(n – 2)) = 32T(n – 2)

T(n) = 32(3T(n – 3))

.
.

T(n) = 3nT(n – n) = 3nT(0) = 3n

This clearly shows that the complexity of this function is O(3n).

Note: We can use the Subtraction and Conquer master theorem for this problem.
Problem-22 Find the complexity of the below recurrence:

Solution: Let us try solving this function with substitution.

T(n) = 2T(n – 1) – 1

T(n) = 2(2T(n – 2) – 1) – 1 = 22T(n – 2) – 2 – 1

T(n) = 22(2T(n – 3) – 2 – 1) – 1 = 23T(n – 4) – 22 – 21 – 20

T(n) = 2nT(n – n) – 2n–1 – 2n–2 – 2n–3 .... 22 – 21 – 20

T(n) =2n – 2n–1 – 2n–2 – 2n – 3 .... 22 – 21 – 20

T(n) =2n – (2n – 1) [note: 2n–1 + 2n–2 + ··· + 20 = 2n]

T(n) = 1

∴ Time Complexity is O(1). Note that while the recurrence relation looks exponential, the
solution to the recurrence relation here gives a different result.
Problem-23 What is the running time of the following function?
Solution: Consider the comments in the below function:

We can define the ‘s’ terms according to the relation si = si–1 + i. The value oft’ increases by 1
for each iteration. The value contained in ‘s’ at the ith iteration is the sum of the first ‘(‘positive
integers. If k is the total number of iterations taken by the program, then the while loop terminates
if:

Problem-24 Find the complexity of the function given below.

Solution:

In the above-mentioned function the loop will end, if i2 > n ⇒ T(n) = O( ). This is similar to
Problem-23.
Problem-25 What is the complexity of the program given below:

Solution: Consider the comments in the following function.

The complexity of the above function is O(n2logn).

Problem-26 What is the complexity of the program given below:
Solution: Consider the comments in the following function.

The complexity of the above function is O(nlog2n).

Problem-27 Find the complexity of the program below.

Solution: Consider the comments in the function below.

The complexity of the above function is O(n). Even though the inner loop is bounded by n, due to
the break statement it is executing only once.
Problem-28 Write a recursive function for the running time T(n) of the function given below.
Prove using the iterative method that T(n) = Θ(n3).

Solution: Consider the comments in the function below:

The recurrence for this code is clearly T(n) = T(n – 3) + cn2 for some constant c > 0 since each
call prints out n2 asterisks and calls itself recursively on n – 3. Using the iterative method we get:
T(n) = T(n – 3) + cn2. Using the Subtraction and Conquer master theorem, we get T(n) = Θ(n3).

Problem-29 Determine Θ bounds for the recurrence relation:

Solution: Using Divide and Conquer master theorem, we get O(nlog2n).

Problem-30 Determine Θ bounds for the recurrence:

Solution: Substituting in the recurrence equation, we get:

, where k is a constant. This clearly
says Θ(n).
Problem-31 Determine Θ bounds for the recurrence relation: T(n) = T(⌈n/2⌉) + 7.
Solution: Using Master Theorem we get: Θ(logn).
Problem-32 Prove that the running time of the code below is Ω(logn).

Solution: The while loop will terminate once the value of ‘k’ is greater than or equal to the value
of ‘n’. In each iteration the value of ‘k’ is multiplied by 3. If i is the number of iterations, then ‘k’
has the value of 3i after i iterations. The loop is terminated upon reaching i iterations when 3i ≥ n
↔ i ≥ log3 n, which shows that i = Ω(logn).

Problem-33 Solve the following recurrence.

Solution: By iteration:

Note: We can use the Subtraction and Conquer master theorem for this problem.
Problem-34 Consider the following program:

Solution: The recurrence relation for the running time of this program is: T(n) = T(n – 1) + T(n –
2) + c. Note T(n) has two recurrence calls indicating a binary tree. Each step recursively calls the
program for n reduced by 1 and 2, so the depth of the recurrence tree is O(n). The number of
leaves at depth n is 2n since this is a full binary tree, and each leaf takes at least O(1)
computations for the constant factor. Running time is clearly exponential in n and it is O(2n).
Problem-35 Running time of following program?
Solution: Consider the comments in the function below:

In the above code, inner loop executes n/i times for each value of i. Its running time is
.
Problem-36 What is the complexity of
Solution: Using the logarithmic property, logxy = logx + logy, we can see that this problem is
equivalent to

This shows that the time complexity = O(nlogn).

Problem-37 What is the running time of the following recursive function (specified as a
function of the input value n)? First write the recurrence formula and then find its
complexity.

Solution: Consider the comments in the below function:

We can assume that for asymptotical analysis k = ⌈k⌉ for every integer k ≥ 1. The recurrence for
this code is . Using master theorem, we get T(n) = Θ(n).

Problem-38 What is the running time of the following recursive function (specified as a
function of the input value n)? First write a recurrence formula, and show its solution using
induction.

Solution: Consider the comments in the function below:

The if statement requires constant time [O(1)]. With the for loop, we neglect the loop overhead
and only count three times that the function is called recursively. This implies a time complexity
recurrence:

Using the Subtraction and Conquer master theorem, we get T(n) = Θ(3n).
Problem-39 Write a recursion formula for the running time T(n) of the function whose code
is below.

Solution: Consider the comments in the function below:

The recurrence for this piece of code is T(n) = T(.8n) + O(n) = T(4/5n) + O(n) =4/5 T(n) + O(n).
Applying master theorem, we get T(n) = O(n).
Problem-40 Find the complexity of the recurrence: T(n) = 2T( ) + logn
Solution: The given recurrence is not in the master theorem format. Let us try to convert this to the
master theorem format by assuming n = 2m. Applying the logarithm on both sides gives, logn =
mlogl ⇒ m = logn. Now, the given function becomes:

To make it simple we assume

.

Applying the master theorem format would result in S(m) = O(mlogm).

If we substitute m = logn back, T(n) = S(logn) = O((logn) loglogn).
Problem-41 Find the complexity of the recurrence: T(n) = T( )+1

Solution: Applying the logic of Problem-40 gives . Applying the master

theorem would result in S(m) = O(logm). Substituting m = logn, gives T(n) = S(logn) =
O(loglogn).
Problem-42 Find the complexity of the recurrence: T(n) = 2T( )+1

Solution: Applying the logic of Problem-40 gives: . Using the master

theorem results S(m) = . Substituting m = logn gives T(n) =O(logn).
Problem-43 Find the complexity of the below function.

Solution: Consider the comments in the function below:

For the above code, the recurrence function can be given as: T(n) = T( ) + 1. This is same as
that of Problem-41.
Problem-44 Analyze the running time of the following recursive pseudo-code as a function of
n.

Solution: Consider the comments in below pseudo-code and call running time of function(n) as
T(n).
T(n) can be defined as follows:

Using the master theorem gives: .

Problem-45 Find the complexity of the below pseudocode:

Solution: Consider the comments in the pseudocode below:

The recurrence for this function is T(n) = T(n/2) + n. Using master theorem, we get T(n) = O(n).
Problem-46 Running time of the following program?

Solution: Consider the comments in the below function:

Complexity of above program is: O(nlogn).

Problem-47 Running time of the following program?

Solution: Consider the comments in the below function:

The time complexity of this program is: O(n2).

Problem-48 Find the complexity of the below function:
Solution: Consider the comments in the below function:

The recurrence for this function is: . Using master theorem, we get T(n) =
O(n).
Problem-49 Find the complexity of the below function:
Solution:

Time Complexity: O(logn * logn) = O(log2n).

Problem-50 ∑i≤k≤n O(n), where O(n) stands for order n is:
(A) O(n)
(B) O(n2)
(C) O(n3)
(D) O(3n2)
(E) O(1.5n2)

Solution: (B). ∑i≤k≤n O(n) = O(n) ∑i≤k≤n 1 = O(n2).

Problem-51 Which of the following three claims are correct?

I (n + k) = Θ(nm), where k and m are constants
m

II 2n+1 = O(2n)
III 22n+1 = O(2n)
(A) I and II
(B) I and III
(C) II and III
(D) I, II and III

Solution: (A). (I) (n + k)m =nh + c1*nk–1 + ... km = Θ(nh) and (II) 2n+1 = 2*2n = O(2n)
Problem-52 Consider the following functions:
f(n) = 2n
g(n) = n!
h(n) = nlogn
Which of the following statements about the asymptotic behavior of f(n), g(n), and h(n) is
true?
(A) f(n) = O(g(n)); g(n) = O(h(n))
(B) f(n) = Ω (g(n)); g(n) = O(h(n))
 (C) g(n) = O(f(n)); h(n) = O(f(n))
(D) h(n) = O(f(n)); g(n) = Ω (f(n))

Solution: (D). According to the rate of growth: h(n) < f(n) < g(n) (g(n) is asymptotically greater
than f(n), and f(n) is asymptotically greater than h(n)). We can easily see the above order by
taking logarithms of the given 3 functions: lognlogn < n < log(n!). Note that, log(n!) = O(nlogn).
Problem-53 Consider the following segment of C-code:

The number of comparisons made in the execution of the loop for any n > 0 is:
(A)
(B) n
(C)
(D)

Solution: (a). Let us assume that the loop executes k times. After kth step the value of j is 2k .
Taking logarithms on both sides gives . Since we are doing one more comparison for
exiting from the loop, the answer is .

Problem-54 Consider the following C code segment. Let T(n) denote the number of times the
for loop is executed by the program on input n. Which of the following is true?

(A) T(n) = O( ) and T(n) = Ω( )

(B) T(n) = O( ) and T(n) = Ω(1)
(C) T(n) = O(n) and T(n) = Ω( )
(D) None of the above

Solution: (B). Big O notation describes the tight upper bound and Big Omega notation describes
the tight lower bound for an algorithm. The for loop in the question is run maximum times and
minimum 1 time. Therefore, T(n) = O( ) and T(n) = Ω(1).

Problem-55 In the following C function, let n ≥ m. How many recursive calls are made by
this function?

(A)
(B) Ω(n)
(C)
(D) Θ(n)
Solution: No option is correct. Big O notation describes the tight upper bound and Big Omega
notation describes the tight lower bound for an algorithm. For m = 2 and for all n = 2i, the running
time is O(1) which contradicts every option.
Problem-56 Suppose T(n) = 2T(n/2) + n, T(O)=T(1)=1. Which one of the following is false?
(A) T(n) = O(n2)
(B) T(n) = Θ(nlogn)
(C) T(n) = Q(n2)
(D) T(n) = O(nlogn)

Solution: (C). Big O notation describes the tight upper bound and Big Omega notation describes
the tight lower bound for an algorithm. Based on master theorem, we get T(n) = Θ(nlogn). This
indicates that tight lower bound and tight upper bound are the same. That means, O(nlogn) and
Ω(nlogn) are correct for given recurrence. So option (C) is wrong.
Problem-57 Find the complexity of the below function:
Solution:

Time Complexity: O(n5).

Problem-58 To calculate 9n, give an algorithm and discuss its complexity.

Solution: Start with 1 and multiply by 9 until reaching 9n.

Time Complexity: There are n – 1 multiplications and each takes constant time giving a Θ(n)
algorithm.
Problem-59 For Problem-58, can we improve the time complexity?
Solution: Refer to the Divide and Conquer chapter.
Problem-60 Find the time complexity of recurrence .
Solution: Let us solve this problem by method of guessing. The total size on each level of the
recurrance tree is less than n, so we guess that f(n) = n will dominate. Assume for all i < n that
c1n ≤ T(i) < c2n. Then,
If c1 ≥ 8k and c2 ≤ 8k, then c1n = T(n) = c2n. So, T(n) = Θ(n). In general, if you have multiple
recursive calls, the sum of the arguments to those calls is less than n (in this case ),
and f(n) is reasonably large, a good guess is T(n) = Θ(f(n)).
Problem-61 Solve the following recurrence relation using the recursion tree method:
.

Solution: How much work do we do in each level of the recursion tree?

In level 0, we take n2 time. At level 1, the two subproblems take time:

At level 2 the four subproblems are of size and respectively. These two
subproblems take time:
Similarly the amount of work at level k is at most .

Let , the total runtime is then:

That is, the first level provides a constant fraction of the total runtime.

Problem-62 Rank the following functions by order of growth: (n + 1)!, n!, 4n, n × 3n, 3n + n2
+ 20n, , n2 + 200, 20n + 500, 2lgn, n2/3, 1.

Solution:
Problem-63 Find the complexity of the below function:

Solution: Consider the worst-case.

Time Complexity: O(n2).
Problem-64 Can we say ?

Solution: Yes: because

Problem-65 Can we say 23n = O(2n)?

Solution: No: because 23n = (23)n = 8n not less than 2n.