2.

Declaration of operations

Commonly used ADTs include: Linked Lists, Stacks, Queues, Priority Queues, Binary Trees,
Dictionaries, Disjoint Sets (Union and Find), Hash Tables, Graphs, and many others. For
example, stack uses LIFO (Last-In-First-Out) mechanism while storing the data in data structures.
The last element inserted into the stack is the first element that gets deleted. Common operations
of it are: creating the stack, pushing an element onto the stack, popping an element from stack,
finding the current top of the stack, finding number of elements in the stack, etc.

While defining the ADTs do not worry about the implementation details. They come into the
picture only when we want to use them. Different kinds of ADTs are suited to different kinds of
applications, and some are highly specialized to specific tasks. By the end of this book, we will
go through many of them and you will be in a position to relate the data structures to the kind of
problems they solve.

1.5 What is an Algorithm?

Let us consider the problem of preparing an omelette. To prepare an omelette, we follow the
steps given below:
1) Get the frying pan.
2) Get the oil.
a. Do we have oil?
i. If yes, put it in the pan.
ii. If no, do we want to buy oil?
1. If yes, then go out and buy.
2. If no, we can terminate.
3) Turn on the stove, etc...

What we are doing is, for a given problem (preparing an omelette), we are providing a step-by-
step procedure for solving it. The formal definition of an algorithm can be stated as:

An algorithm is the step-by-step unambiguous instructions to solve a given problem.

In the traditional study of algorithms, there are two main criteria for judging the merits of
algorithms: correctness (does the algorithm give solution to the problem in a finite number of
steps?) and efficiency (how much resources (in terms of memory and time) does it take to execute
the).

Note: We do not have to prove each step of the algorithm.

1.6 Why the Analysis of Algorithms?

To go from city “A” to city “B”, there can be many ways of accomplishing this: by flight, by bus,
by train and also by bicycle. Depending on the availability and convenience, we choose the one
that suits us. Similarly, in computer science, multiple algorithms are available for solving the
same problem (for example, a sorting problem has many algorithms, like insertion sort, selection
sort, quick sort and many more). Algorithm analysis helps us to determine which algorithm is
most efficient in terms of time and space consumed.

1.7 Goal of the Analysis of Algorithms

The goal of the analysis of algorithms is to compare algorithms (or solutions) mainly in terms of
running time but also in terms of other factors (e.g., memory, developer effort, etc.)

1.8 What is Running Time Analysis?

It is the process of determining how processing time increases as the size of the problem (input
size) increases. Input size is the number of elements in the input, and depending on the problem
type, the input may be of different types. The following are the common types of inputs.
• Size of an array
• Polynomial degree
• Number of elements in a matrix
• Number of bits in the binary representation of the input
• Vertices and edges in a graph.

1.9 How to Compare Algorithms

To compare algorithms, let us define a few objective measures:

Execution times? Not a good measure as execution times are specific to a particular computer.

Number of statements executed? Not a good measure, since the number of statements varies
with the programming language as well as the style of the individual programmer.

Ideal solution? Let us assume that we express the running time of a given algorithm as a function
of the input size n (i.e., f(n)) and compare these different functions corresponding to running
times. This kind of comparison is independent of machine time, programming style, etc.

1.10 What is Rate of Growth?

The rate at which the running time increases as a function of input is called rate of growth. Let us
assume that you go to a shop to buy a car and a bicycle. If your friend sees you there and asks
what you are buying, then in general you say buying a car. This is because the cost of the car is
high compared to the cost of the bicycle (approximating the cost of the bicycle to the cost of the
car).

For the above-mentioned example, we can represent the cost of the car and the cost of the bicycle
in terms of function, and for a given function ignore the low order terms that are relatively
insignificant (for large value of input size, n). As an example, in the case below, n4, 2n2, 100n
and 500 are the individual costs of some function and approximate to n4 since n4 is the highest
rate of growth.

1.11 Commonly Used Rates of Growth

The diagram below shows the relationship between different rates of growth.
Below is the list of growth rates you will come across in the following chapters.
1.12 Types of Analysis

To analyze the given algorithm, we need to know with which inputs the algorithm takes less time
(performing wel1) and with which inputs the algorithm takes a long time. We have already seen
that an algorithm can be represented in the form of an expression. That means we represent the
algorithm with multiple expressions: one for the case where it takes less time and another for the
case where it takes more time.

In general, the first case is called the best case and the second case is called the worst case for
the algorithm. To analyze an algorithm we need some kind of syntax, and that forms the base for
asymptotic analysis/notation. There are three types of analysis:
• Worst case
○ Defines the input for which the algorithm takes a long time (slowest
time to complete).
○ Input is the one for which the algorithm runs the slowest.
• Best case
○ Defines the input for which the algorithm takes the least time (fastest
time to complete).
○ Input is the one for which the algorithm runs the fastest.
• Average case
○ Provides a prediction about the running time of the algorithm.
○ Run the algorithm many times, using many different inputs that come
from some distribution that generates these inputs, compute the total
running time (by adding the individual times), and divide by the
number of trials.
○ Assumes that the input is random.

Lower Bound <= Average Time <= Upper Bound

For a given algorithm, we can represent the best, worst and average cases in the form of
expressions. As an example, let f(n) be the function which represents the given algorithm.

Similarly for the average case. The expression defines the inputs with which the algorithm takes
the average running time (or memory).

1.13 Asymptotic Notation

Having the expressions for the best, average and worst cases, for all three cases we need to
identify the upper and lower bounds. To represent these upper and lower bounds, we need some
kind of syntax, and that is the subject of the following discussion. Let us assume that the given
algorithm is represented in the form of function f(n).

1.14 Big-O Notation [Upper Bounding Function]

This notation gives the tight upper bound of the given function. Generally, it is represented as f(n)
= O(g(n)). That means, at larger values of n, the upper bound of f(n) is g(n). For example, if f(n)
= n4 + 100n2 + 10n + 50 is the given algorithm, then n4 is g(n). That means g(n) gives the
maximum rate of growth for f(n) at larger values of n.
Let us see the O–notation with a little more detail. O–notation defined as O(g(n)) = {f(n): there
exist positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) for all n > n0}. g(n) is an asymptotic
tight upper bound for f(n). Our objective is to give the smallest rate of growth g(n) which is
greater than or equal to the given algorithms’ rate of growth /(n).

Generally we discard lower values of n. That means the rate of growth at lower values of n is not
important. In the figure, n0 is the point from which we need to consider the rate of growth for a
given algorithm. Below n0, the rate of growth could be different. n0 is called threshold for the
given function.

Big-O Visualization

O(g(n)) is the set of functions with smaller or the same order of growth as g(n). For example;
O(n2) includes O(1), O(n), O(nlogn), etc.

Note: Analyze the algorithms at larger values of n only. What this means is, below n0 we do not
care about the rate of growth.

Big-O Examples

Example-1 Find upper bound for f(n) = 3n + 8

Solution: 3n + 8 ≤ 4n, for all n ≥ 8
∴ 3n + 8 = O(n) with c = 4 and n0 = 8
Example-2 Find upper bound for f(n) = n2 + 1
Solution: n2 + 1 ≤ 2n2, for all n ≥ 1
∴ n2 + 1 = O(n2) with c = 2 and n0 = 1

Example-3 Find upper bound for f(n) = n4 + 100n2 + 50

Solution: n4 + 100n2 + 50 ≤ 2n4, for all n ≥ 11
∴ n4 + 100n2 + 50 = O(n4 ) with c = 2 and n0 = 11

Example-4 Find upper bound for f(n) = 2n3 – 2n2

Solution: 2n3 – 2n2 ≤ 2n3, for all n > 1
∴ 2n3 – 2n2 = O(n3 ) with c = 2 and n0 = 1
Example-5 Find upper bound for f(n) = n
Solution: n ≤ n, for all n ≥ 1
∴ n = O(n) with c = 1 and n0 = 1
Example-6 Find upper bound for f(n) = 410
Solution: 410 ≤ 410, for all n > 1
∴ 410 = O(1) with c = 1 and n0 = 1

No Uniqueness?

There is no unique set of values for n0 and c in proving the asymptotic bounds. Let us consider,
100n + 5 = O(n). For this function there are multiple n0 and c values possible.
Solution1: 100n + 5 ≤ 100n + n = 101n ≤ 101n, for all n ≥ 5, n0 = 5 and c = 101 is a solution.
Solution2: 100n + 5 ≤ 100n + 5n = 105n ≤ 105n, for all n > 1, n0 = 1 and c = 105 is also a
solution.

1.15 Omega-Q Notation [Lower Bounding Function]

Similar to the O discussion, this notation gives the tighter lower bound of the given algorithm and
we represent it as f(n) = Ω(g(n)). That means, at larger values of n, the tighter lower bound of
f(n) is g(n). For example, if f(n) = 100n2 + 10n + 50, g(n) is Ω(n2).
The Ω notation can be defined as Ω(g(n)) = {f(n): there exist positive constants c and n0 such that
0 ≤ cg(n) ≤ f(n) for all n ≥ n0}. g(n) is an asymptotic tight lower bound for f(n). Our objective is
to give the largest rate of growth g(n) which is less than or equal to the given algorithm’s rate of
growth f(n).

Ω Examples

Example-1 Find lower bound for f(n) = 5n2.

Solution: ∃ c, n0 Such that: 0 ≤ cn2≤ 5n2 ⇒ cn2 ≤ 5n2 ⇒ c = 5 and n0 = 1
∴ 5n2 = Ω(n2) with c = 5 and n0 = 1

Example-2 Prove f(n) = 100n + 5 ≠ Ω(n2).

Solution: ∃ c, n0 Such that: 0 ≤ cn2 ≤ 100n + 5
100n + 5 ≤ 100n + 5n(∀n ≥ 1) = 105n
cn2 ≤ 105n ⇒ n(cn - 105) ≤ 0
Since n is positive ⇒cn - 105 ≤0 ⇒ n ≤105/c
⇒ Contradiction: n cannot be smaller than a constant

Example-3 2n = Q(n), n3 = Q(n3), = O(logn).

1.16 Theta-Θ Notation [Order Function]

This notation decides whether the upper and lower bounds of a given function (algorithm) are the
same. The average running time of an algorithm is always between the lower bound and the upper
bound. If the upper bound (O) and lower bound (Ω) give the same result, then the Θ notation will
also have the same rate of growth.

As an example, let us assume that f(n) = 10n + n is the expression. Then, its tight upper bound
g(n) is O(n). The rate of growth in the best case is g(n) = O(n).

In this case, the rates of growth in the best case and worst case are the same. As a result, the
average case will also be the same. For a given function (algorithm), if the rates of growth
(bounds) for O and Ω are not the same, then the rate of growth for the Θ case may not be the same.
In this case, we need to consider all possible time complexities and take the average of those (for
example, for a quick sort average case, refer to the Sorting chapter).

Now consider the definition of Θ notation. It is defined as Θ(g(n)) = {f(n): there exist positive
constants c1,c2 and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0}. g(n) is an asymptotic
tight bound for f(n). Θ(g(n)) is the set of functions with the same order of growth as g(n).

Θ Examples
Example 1 Find Θ bound for

Solution: for all, n ≥ 2

∴ with c1 = 1/5,c2 = 1 and n0 = 2

Example 2 Prove n ≠ Θ(n2)

Solution: c1 n2 ≤ n ≤ c2n2 ⇒ only holds for: n ≤ 1/c1
∴ n ≠ Θ(n2)

Example 3 Prove 6n3 ≠ Θ(n2)

Solution: c1 n2≤ 6n3 ≤ c2 n2 ⇒ only holds for: n ≤ c2 /6
∴ 6n3 ≠ Θ(n2)

Example 4 Prove n ≠ Θ(logn)

Solution: c1logn ≤ n ≤ c2logn ⇒ c2 ≥ , ∀ n ≥ n0 – Impossible

1.17 Important Notes

For analysis (best case, worst case and average), we try to give the upper bound (O) and lower
bound (Ω) and average running time (Θ). From the above examples, it should also be clear that,
for a given function (algorithm), getting the upper bound (O) and lower bound (Ω) and average
running time (Θ) may not always be possible. For example, if we are discussing the best case of
an algorithm, we try to give the upper bound (O) and lower bound (Ω) and average running time
(Θ).

In the remaining chapters, we generally focus on the upper bound (O) because knowing the lower
bound (Ω) of an algorithm is of no practical importance, and we use the Θ notation if the upper
bound (O) and lower bound (Ω) are the same.

1.18 Why is it called Asymptotic Analysis?

From the discussion above (for all three notations: worst case, best case, and average case), we
can easily understand that, in every case for a given function f(n) we are trying to find another
function g(n) which approximates f(n) at higher values of n. That means g(n) is also a curve
which approximates f(n) at higher values of n.

In mathematics we call such a curve an asymptotic curve. In other terms, g(n) is the asymptotic
curve for f(n). For this reason, we call algorithm analysis asymptotic analysis.

1.19 Guidelines for Asymptotic Analysis

There are some general rules to help us determine the running time of an algorithm.

1) Loops: The running time of a loop is, at most, the running time of the statements
inside the loop (including tests) multiplied by the number of iterations.

Total time = a constant c × n = c n = O(n).

2) Nested loops: Analyze from the inside out. Total running time is the product of the
sizes of all the loops.

Total time = c × n × n = cn2 = O(n2).

3) Consecutive statements: Add the time complexities of each statement.

Total time = c0 + c1n + c2n2 = O(n2).

4) If-then-else statements: Worst-case running time: the test, plus either the then part
or the else part (whichever is the larger).

Total time = c0 + c1 + (c2 + c3) * n = O(n).

5) Logarithmic complexity: An algorithm is O(logn) if it takes a constant time to cut

the problem size by a fraction (usually by ½). As an example let us consider the
following program:
 If we observe carefully, the value of i is doubling every time. Initially i = 1, in next step i
= 2, and in subsequent steps i = 4,8 and so on. Let us assume that the loop is executing
some k times. At kth step 2k = n, and at (k + 1)th step we come out of the loop. Taking
logarithm on both sides, gives

Total time = O(logn).

Note: Similarly, for the case below, the worst case rate of growth is O(logn). The same
discussion holds good for the decreasing sequence as well.

Another example: binary search (finding a word in a dictionary of n pages)

• Look at the center point in the dictionary
• Is the word towards the left or right of center?
• Repeat the process with the left or right part of the dictionary until the word is found.

1.20 Simplyfying properties of asymptotic notations

• Transitivity: f(n) = Θ(g(n)) and g(n) = Θ(h(n)) ⇒ f(n) = Θ(h(n)). Valid for O and Ω
as well.
• Reflexivity: f(n) = Θ(f(n)). Valid for O and Ω.
• Symmetry: f(n) = Θ(g(n)) if and only if g(n) = Θ(f(n)).
• Transpose symmetry: f(n) = O(g(n)) if and only if g(n) = Ω(f(n)).
• If f(n) is in O(kg(n)) for any constant k > 0, then f(n) is in O(g(n)).
• If f1(n) is in O(g1(n)) and f2(n) is in O(g2(n)), then (f1 + f2)(n) is in O(max(g1(n)),
(g1(n))).
• If f1(n) is in O(g1(n)) and f2(n) is in O(g2(n)) then f1(n) f2(n) is in O(g1(n) g1(n)).

1.21 Commonly used Logarithms and Summations

Logarithms