Data Structures and Algorithmic Thinking with Python Introduction

Chapter

Introduction 1
The objective of this chapter is to explain the importance of the analysis of algorithms, their notations, relationships and solving as many
problems as possible. Let us first focus on understanding the basic elements of algorithms, the importance of algorithm analysis, and then
slowly move toward the other topics as mentioned above. After completing this chapter, you should be able to find the complexity of any
given algorithm (especially recursive functions).

1.1 Variables
Before going to the definition of variables, let us relate them to old mathematical equations. All of us have solved many mathematical equations
since childhood. As an example, consider the below equation:
+2 −2 =1
We don’t have to worry about the use of this equation. The important thing that we need to understand is that the equation has names ( and
), which hold values (data). That means the ( and ) are placeholders for representing data. Similarly, in computer science
programming we need something for holding data, and is the way to do that.

1.2 Data Types

In the above-mentioned equation, the variables and can take any values such as integral numbers (10, 20), real numbers (0.23, 5.5), or
just 0 and 1. To solve the equation, we need to relate them to the kind of values they can take, and is the name used in computer
science programming for this purpose. A in a programming language is a set of data with predefined values. Examples of data
types are: integer, floating point, unit number, character, string, etc.
Computer memory is all filled with zeros and ones. If we have a problem and we want to code it, it’s very difficult to provide the solution in
terms of zeros and ones. To help users, programming languages and compilers provide us with data types. For example, takes 2
bytes (actual value depends on compiler), takes 4 bytes, etc. This says that in memory we are combining 2 bytes (16 bits) and calling it
an . Similarly, combining 4 bytes (32 bits) and calling it a . A data type reduces the coding effort. At the top level, there are two
types of data types:
· System-defined data types (also called data types)
· User-defined data types

System-defined data types (Primitive data types)

Data types that are defined by system are called data types. The primitive data types provided by many programming languages
are: int, float, char, double, bool, etc. The number of bits allocated for each primitive data type depends on the programming languages, the
compiler and the operating system. For the same primitive data type, different languages may use different sizes. Depending on the size of the
data types, the total available values (domain) will also change.
For example, “ ” may take 2 bytes or 4 bytes. If it takes 2 bytes (16 bits), then the total possible values are minus 32,768 to plus 32,767 (-
2 2 -1). If it takes 4 bytes (32 bits), then the possible values are between −2,147,483,648 and +2,147,483,647 (-2 2 -1). The
same is the case with other data types.

User defined data types

If the system-defined data types are not enough, then most programming languages allow the users to define their own data types, called
− . Good examples of user defined data types are: classes in . For example, in the snippet below, we are
combining many system-defined data types and calling the user defined data type by the name “ ”. This gives more flexibility and
comfort in dealing with computer memory.
class NewType(object):
def __init__(self, datainput1, datainput2, datainput3):
self.data1 = datainput1
self.data2 = datainput2
self.data3 = datainput3

1.1 Variables 19
 Data Structures and Algorithmic Thinking with Python Introduction

1.3 Data Structures

Based on the discussion above, once we have data in variables, we need some mechanism for manipulating that data to solve problems.
is a particular way of storing and organizing data in a computer so that it can be used efficiently. A is a
special format for organizing and storing data. General data structure types include arrays, files, linked lists, stacks, queues, trees, graphs and
so on. Depending on the organization of the elements, data structures are classified into two types:
1) : Elements are accessed in a sequential order but it is not compulsory to store all elements sequentially.
: Linked Lists, Stacks and Queues.
2) − : Elements of this data structure are stored/accessed in a non-linear order. : Trees and
graphs.

1.4 Abstract Data Types (ADTs)

Before defining abstract data types, let us consider the different view of system-defined data types. We all know that, by default, all primitive
data types (int, float, etc.) support basic operations such as addition and subtraction. The system provides the implementations for the primitive
data types. For user-defined data types we also need to define operations. The implementation for these operations can be done when we
want to actually use them. That means, in general, user defined data types are defined along with their operations.
To simplify the process of solving problems, we combine the data structures with their operations and we call this
(ADTs). An ADT consists of parts:
1. Declaration of data
2. Declaration of operations
Commonly used ADTs : Linked Lists, Stacks, Queues, Priority Queues, Binary Trees, Dictionaries, Disjoint Sets (Union and Find),
Hash Tables, Graphs, and many others. For example, stack uses LIFO (Last-In-First-Out) mechanism while storing the data in data structures.
The last element inserted into the stack is the first element that gets deleted. Common operations of it are: creating the stack, pushing an
element onto the stack, popping an element from stack, finding the current top of the stack, finding number of elements in the stack, etc.
While defining the ADTs do not worry about the implementation details. They come into the picture only when we want to use them.
Different kinds of ADTs are suited to different kinds of applications, and some are highly specialized to specific tasks. By the end of this
book, we will go through many of them and you will be in a position to relate the data structures to the kind of problems they solve.

1.5 What is an Algorithm?

Let us consider the problem of preparing an . To prepare an omelet, we follow the steps given below:
1) Get the frying pan.
2) Get the oil.
a. Do we have oil?
i. If yes, put it in the pan.
ii. If no, do we want to buy oil?
1. If yes, then go out and buy.
2. If no, we can terminate.
3) Turn on the stove, etc...
What we are doing is, for a given problem (preparing an omelet), we are providing a step-by-step procedure for solving it. The formal definition
of an algorithm can be stated as:
An algorithm is the step-by-step unambiguous instructions to solve a given problem.
In the traditional study of algorithms, there are two main criteria for judging the merits of algorithms: correctness (does the algorithm give
solution to the problem in a finite number of steps?) and efficiency (how much resources (in terms of memory and time) does it take to
execute the).
: We do not have to prove each step of the algorithm.

1.6 Why the Analysis of Algorithms?

To go from city “ ” to city “ ”, there can be many ways of accomplishing this: by flight, by bus, by train and also by bicycle. Depending on
the availability and convenience, we choose the one that suits us. Similarly, in computer science, multiple algorithms are available for solving
the same problem (for example, a sorting problem has many algorithms, like insertion sort, selection sort, quick sort and many more).
Algorithm analysis helps us to determine which algorithm is most efficient in terms of time and space consumed.

1.7 Goal of the Analysis of Algorithms

The goal of the ℎ is to compare algorithms (or solutions) mainly in terms of running time but also in terms of other
factors (e.g., memory, developer effort, etc.)

1.8 What is Running Time Analysis?

It is the process of determining how processing time increases as the size of the problem (input size) increases. Input size is the number of
elements in the input, and depending on the problem type, the input may be of different types. The following are the common types of inputs.

1.3 Data Structures 20

 Data Structures and Algorithmic Thinking with Python Introduction

· Size of an array
· Polynomial degree
· Number of elements in a matrix
· Number of bits in the binary representation of the input
· Vertices and edges in a graph.

1.9 How to Compare Algorithms

To compare algorithms, let us define a few :
as execution times are specific to a particular computer.
, since the number of statements varies with the programming language as well as the
style of the individual programmer.
? Let us assume that we express the running time of a given algorithm as a function of the input size (i.e., ( )) and compare
these different functions corresponding to running times. This kind of comparison is independent of machine time, programming style, etc.

1.10 What is Rate of Growth?

The rate at which the running time increases as a function of input is called ℎ. Let us assume that you go to a shop to buy a
car and a bicycle. If your friend sees you there and asks what you are buying, then in general you say . This is because the cost
of the car is high compared to the cost of the bicycle (approximating the cost of the bicycle to the cost of the car).
= _ _ + _ _
≈ _ _ ( )
For the above-mentioned example, we can represent the cost of the car and the cost of the bicycle in terms of function, and for a given function
ignore the low order terms that are relatively insignificant (for large value of input size, ). As an example, in the case below, , 2 , 100
and 500 are the individual costs of some function and approximate to since is the highest rate of growth.
+ 2 + 100 + 500 ≈

1.11 Commonly Used Rates of Growth

The diagram below shows the relationship between different rates of growth.
2

4 D
e
c
2
r
e
a
s
i
log n
log ( !) g

R
a
t
e
2 s

O
f

G
r
o
w
log log t
h

1.9 How to Compare Algorithms 21

 Data Structures and Algorithmic Thinking with Python Introduction

Below is the list of growth rates you will come across in the following chapters.
Time Complexity Name Example
1 Constant Adding an element to the front of a linked list
Logarithmic Finding an element in a sorted array
Linear Finding an element in an unsorted array
Linear Logarithmic Sorting n items by ‘divide-and-conquer’ - Mergesort
Quadratic Shortest path between two nodes in a graph
Cubic Matrix Multiplication
2 Exponential The Towers of Hanoi problem

1.12 Types of Analysis

To analyze the given algorithm, we need to know with which inputs the algorithm takes less time (performing well) and with which inputs the
algorithm takes a long time. We have already seen that an algorithm can be represented in the form of an expression. That means we represent
the algorithm with multiple expressions: one for the case where it takes less time and another for the case where it takes more time.
In general, the first case is called the and the second case is called the for the algorithm. To analyze an algorithm, we
need some kind of syntax, and that forms the base for asymptotic analysis/notation. There are three types of analysis:
·
o Defines the input for which the algorithm takes a long time (slowest time to complete).
o Input is the one for which the algorithm runs the slowest.
·
o Defines the input for which the algorithm takes the least time (fastest time to complete).
o Input is the one for which the algorithm runs the fastest.
·
o Provides a prediction about the running time of the algorithm.
o Run the algorithm many times, using many different inputs that come from some distribution that generates these
inputs, compute the total running time (by adding the individual times), and divide by the number of trials.
o Assumes that the input is random.
<= <=
For a given algorithm, we can represent the best, worst and average cases in the form of expressions. As an example, let ( ) be the function
which represents the given algorithm.
( )= + 500, for worst case
( )= + 100 + 500, for best case
Similarly, for the average case. The expression defines the inputs with which the algorithm takes the average running time (or memory).

1.13 Asymptotic Notation

Having the expressions for the best, average and worst cases, for all three cases we need to identify the upper and lower bounds. To represent
these upper and lower bounds, we need some kind of syntax, and that is the subject of the following discussion. Let us assume that the given
algorithm is represented in the form of function ( ).

1.14 Big-O Notation

This notation gives the ℎ upper bound of the given function. Generally, it is represented as ( ) = O( ( )). That means, at larger
values of , the upper bound of ( ) is ( ). For example, if ( ) = + 100 + 10 + 50 is the given algorithm, then is ( ).
That means ( ) gives the maximum rate of growth for ( ) at larger values of .

( ) ( )

Rate of growth

Input size,

1.12 Types of Analysis 22

 Data Structures and Algorithmic Thinking with Python Introduction

Let us see the O−notation with a little more detail. O−notation defined as O( ( )) = { ( ): there exist positive constants and such
that 0 ≤ ( ) ≤ ( ) for all ≥ }. ( ) is an asymptotic tight upper bound for ( ). Our objective is to give the smallest rate of
growth ( ) which is greater than or equal to the given algorithms’ rate of growth ( ).
Generally, we discard lower values of . That means the rate of growth at lower values of is not important. In the figure, is the point
from which we need to consider the rate of growth for a given algorithm. Below , the rate of growth could be different. is called threshold
for the given function.

Big-O Visualization
O( ( )) is the set of functions with smaller or the same order of growth as ( ). For example; O( ) includes O(1), O( ), O( ), etc.

O(1): 100,1000, 200,1,20, . O( ):3 + 100, 100 , 2 − 1, 3, .

O( ): 5 , 3 − 100, 2 − O( ) , 5 − 10, 100, − 2 + 1,

1, 100, 100 , . 5, .

Note: Analyze the algorithms at larger values of only. What this means is, below we do not care about the rate of growth.

Big-O Examples
Find upper bound for ( ) = 3 + 8
3 + 8 ≤ 4 , for all ≥ 8
∴ 3 + 8 = O( ) with c = 4 and =8
Find upper bound for ( ) = + 1
+ 1 ≤ 2 , for all ≥ 1
∴ + 1 = O( ) with = 2 and =1
Find upper bound for ( )= + 100 + 50
+ 100 + 50 ≤ 2 , for all ≥ 11
∴ + 100 + 50 = O( ) with = 2 and = 11
Find upper bound for ( )=2 − 2
2 − 2 ≤ 2 , for all ≥ 1
∴ 2 − 2 = O( ) with = 2 and =1
Find upper bound for ( ) =
≤ , for all ≥ 1
∴ = O( ) with = 1 and =1
Find upper bound for ( ) = 410
410 ≤ 410, for all ≥ 1
∴ 410 = O(1 ) with = 1 and =1

No Uniqueness?
There is no unique set of values for and in proving the asymptotic bounds. Let us consider, 100 + 5 = O( ). For this function there
are multiple and values possible.
100 + 5 ≤ 100 + = 101 ≤ 101 , for all ≥ 5, = 5 and = 101 is a solution.
100 + 5 ≤ 100 + 5 = 105 ≤ 105 , for all ≥ 1, = 1 and = 105 is also a solution.

1.15 Omega-Ω Notation

Similar to the O discussion, this notation gives the tighter lower bound of the given algorithm and we represent it as ( ) = W( ( )). That
means, at larger values of , the tighter lower bound of ( ) is ( ). The Ω notation can be defined as Ω( ( )) = { ( ): there exist positive
constants c and such that 0 ≤ ( ) ≤ ( ) for all n ≥ }. ( ) is an asymptotic tight lower bound for ( ). Our objective is to
give the largest rate of growth ( ) which is less than or equal to the given algorithm’s rate of growth ( ).
For example, if ( ) = 100 + 10 + 50, ( ) is W( ).

1.15 Omega-Ω Notation 23

 Data Structures and Algorithmic Thinking with Python Introduction

( )
( ))

Rate of growth

Input size,
Ω Examples
Find lower bound for ( )=5 .

, Such that: 0 £ £5 Þ £5 Þ = 5 and =1

∴5 = W( ) with = 5 and =1
Prove ( ) = 100 + 5 ≠ W( ).
c, Such that: 0 £ £ 100 + 5
100 + 5 £ 100 + 5 ( ³ 1) = 105
£ 105 Þ ( – 105) £ 0
Since is positive Þ – 105 £ 0 Þ £ 105/
Þ Contradiction: cannot be smaller than a constant
2 = W( ), = W( ), = W( ).

1.16 Theta-Q Notation

c ( )

( )

Rate of growth c ( )

Input size,

This notation decides whether the upper and lower bounds of a given function (algorithm) are the same. The average running time of an
algorithm is always between the lower bound and the upper bound. If the upper bound (O) and lower bound (W) give the same result, then
the Q notation will also have the same rate of growth. As an example, let us assume that ( ) = 10 + is the expression. Then, its tight
upper bound ( ) is O( ). The rate of growth in the best case is ( ) = O( ).
In this case, the rates of growth in the best case and worst case are the same. As a result, the average case will also be the same. For a given
function (algorithm), if the rates of growth (bounds) for O and W are not the same, then the rate of growth for the Q case may not be the same.
In this case, we need to consider all possible time complexities and take the average of those (for example, for a quick sort average case, refer
to the chapter).
Now consider the definition of Q notation. It is defined as Q( ( )) = { ( ): there exist positive constants , and such that 0 ≤
( ) ≤ ( ) ≤ ( ) for all ≥ }. ( ) is an asymptotic tight bound for ( ). Q( ( )) is the set of functions with the same
order of growth as ( ).

Q Examples
Find Q bound for ( ) = −
≤ − ≤ , for all, ≥ 2
∴ − = Q( ) with = 1/5, = 1 and =2
Prove ≠ Q( )

1.16 Theta- Notation 24

 Data Structures and Algorithmic Thinking with Python Introduction

c 2 ≤ ≤ c 2Þ only holds for: ≤ 1/c1

∴ ≠ Q( )
Prove 6 ≠ Q( )
≤6 ≤c Þ only holds for: ≤ c2 /6
∴6 ≠ Q( )
Prove ≠ Q( )
Solution c ≤ ≤ c Þc ≥ log
, ≥ 0 – Impossible

Important Notes
For analysis (best case, worst case and average), we try to give the upper bound (O) and lower bound (W) and average running time (Q). From
the above examples, it should also be clear that, for a given function (algorithm), getting the upper bound (O) and lower bound (W) and
average running time (Q) may not always be possible. For example, if we are discussing the best case of an algorithm, we try to give the upper
bound (O) and lower bound (W) and average running time (Q).
In the remaining chapters, we generally focus on the upper bound (O) because knowing the lower bound (W) of an algorithm is of no practical
importance, and we use the Q notation if the upper bound (O) and lower bound (W) are the same.

1.17 Why is it called Asymptotic Analysis?

From the discussion above (for all three notations: worst case, best case, and average case), we can easily understand that, in every case for a
given function ( ) we are trying to find another function ( ) which approximates ( ) at higher values of . That means ( ) is also a
curve which approximates ( ) at higher values of .
In mathematics we call such a curve an . In other terms, ( ) is the asymptotic curve for ( ). For this reason, we call
algorithm analysis .

1.18 Guidelines for Asymptotic Analysis

There are some general rules to help us determine the running time of an algorithm.
: The running time of a loop is, at most, the running time of the statements inside the loop (including tests) multiplied by the
number of iterations.
# executes times
for i in range(0,n):
print ('Current Number :', i, sep=" ") #constant time
Total time = a constant × = = O( ).
Analyze from the inside out. Total running time is the product of the sizes of all the loops.
# outer loop executed times
for i in range(0,n):
# inner loop executes n times
for j in range(0,n):
print ('i value %d and j value %d' % (i,j)) #constant time
Total time = × × = = O( ).
Add the time complexities of each statement.
n = 100
# executes times
for i in range(0,n):
print ('Current Number :', i, sep=" ") #constant time
# outer loop executed n times
for i in range(0,n):
# inner loop executes n times
for j in range(0,n):
print ('i value %d and j value %d' % (i,j)) #constant time
Total time = + + = O( ).
Worst-case running time: the test, plus ℎ the ℎ part or the part (whichever is the larger).
if n == 1: #constant time
print ("Wrong Value")
print (n)
else:
for i in range(0,n): #n times
print ('Current Number :', i, sep=" ") #constant time

1.17 Why is it called Asymptotic Analysis? 25

 Data Structures and Algorithmic Thinking with Python Introduction

Total time = + ∗ = O( ).
5) An algorithm is O( ) if it takes a constant time to cut the problem size by a fraction (usually by ½).
As an example, let us consider the following program:
def logarithms(n):
i=1
while i <= n:
i= i * 2
print (i)
logarithms(100)
If we observe carefully, the value of is doubling every time. Initially = 1, in next step = 2, and in subsequent steps = 4, 8 and
so on. Let us assume that the loop is executing some times. At step 2 = , and at ( + 1) step we come out of the .
Taking logarithm on both sides, gives
2 =
2=
= //if we assume base-2
Total time = O( ).
Similarly, for the case below, the worst-case rate of growth is O( ). The same discussion holds good for the decreasing sequence as
well.
def logarithms(n):
i=n
while i >= 1:
i= i // 2
print (i)
logarithms(100)
Another example: binary search (finding a word in a dictionary of pages)
· Look at the center point in the dictionary
· Is the word towards the left or right of center?
· Repeat the process with the left or right part of the dictionary until the word is found.

1.19 Simplifying properties of asymptotic notations

· Transitivity: ( ) = Q( ( )) and ( ) = Q(ℎ( )) Þ ( ) = Q(ℎ( )). Valid for O and W as well.
· Reflexivity: ( ) = Q( ( )). Valid for O and W.
· Symmetry: ( ) = Q( ( )) if and only if ( ) = Q( ( )).
· Transpose symmetry: ( ) = O( ( )) if and only if ( ) = W( ( )).
· If ( ) is in O( ( )) for any constant > 0, then ( ) is in O( ( )).
· If ( ) is in O( ( )) and ( ) is in O( ( )), then ( + )( ) is in O(max( ( ), ( ))).
· If ( ) is in O( ( )) and ( ) is in O( ( )) then ( ) ( ) is in O( ( ) ( )).

1.21 Commonly used Logarithms and Summations

Logarithms
= =
= + = ( )
= ( ) = –
= =
Arithmetic series
( + 1)
= 1 + 2 + ⋯+ =
2
Geometric series
−1
=1+ + …+ = ( ≠ 1)
−1
Harmonic series
1 1 1
= 1 + + …+ ≈
2

1.19 Simplifying properties of asymptotic notations 26

 Data Structures and Algorithmic Thinking with Python Introduction

Other important formulae

1
= 1 + 2 + ⋯+ ≈
+1

1.21 Master Theorem for Divide and Conquer Recurrences

All divide and conquer algorithms (Also discussed in detail in the chapter) divide the problem into sub-problems, each
of which is part of the original problem, and then perform some additional work to compute the final answer. As an example, a merge sort
algorithm [for details, refer to chapter] operates on two sub-problems, each of which is half the size of the original, and then performs
O( ) additional work for merging. This gives the running time equation:
T( ) = 2 + O( )
The following theorem can be used to determine the running time of divide and conquer algorithms. For a given program (algorithm), first
we try to find the recurrence relation for the problem. If the recurrence is of the below form then we can directly give the answer without fully
solving it. If the recurrence is of the form T( ) = ( ) + Q( ), where ≥ 1, > 1, ≥ 0 and is a real number, then:
1) If > , then ( ) = Θ
2) If =
a. If > −1, then ( ) = Θ
b. If = −1, then ( ) = Θ
c. If < −1, then ( ) = Θ
3) If <
a. If ≥ 0, then ( ) = Θ( )
b. If < 0, then ( ) = O( )

1.22 Divide and Conquer Master Theorem: Problems & Solutions

For each of the following recurrences, give an expression for the runtime ( ) if the recurrence can be solved with the Master Theorem.
Otherwise, indicate that the Master Theorem does not apply.
( ) = 3 ( /2) +
( ) = 3 ( /2) + => ( ) =Θ( ) (Master Theorem Case 3.a)
( ) = 4 ( /2) +
( ) = 4 ( /2) + => ( ) = Θ( ) (Master Theorem Case 2.a)
( ) = ( /2) +
( ) = ( /2) + => Θ( ) (Master Theorem Case 3.a)
( ) = 2 ( /2) +
( ) = 2 ( /2) + => Does not apply ( is not constant)
( ) = 16 ( /4) +
( ) = 16 ( /4) + => ( ) = Θ( ) (Master Theorem Case 1)
( ) = 2 ( /2) +
( ) = 2 ( /2) + => ( ) = Θ( ) (Master Theorem Case 2.a)
( ) = 2 ( /2) + /
( ) = 2 ( /2) + / => ( ) = Θ( ) (Master Theorem Case 2.b)
.
( ) = 2 ( /4) +
( ) = 2 ( /4) + . => ( ) = Θ( .
) (Master Theorem Case 3.b)
( ) = 0.5 ( /2) + 1/
( ) = 0.5 ( /2) + 1/ => Does not apply ( < 1)
( ) = 6 ( /3) +
( ) = 6 ( /3) + => ( ) = Θ( ) (Master Theorem Case 3.a)
( ) = 64 ( /8) −
( ) = 64 ( /8) − => Does not apply (function is not positive)

1.21 Master Theorem for Divide and Conquer Recurrences 27

 Data Structures and Algorithmic Thinking with Python Introduction

( ) = 7 ( /3) +
( ) = 7 ( /3) + => ( ) = Θ( ) (Master Theorem Case 3.as)
( ) = 4 ( /2) +
( ) = 4 ( /2) + => ( ) = Θ( ) (Master Theorem Case 1)
( ) = 16 ( /4) + !
( ) = 16 ( /4) + ! => ( ) = Θ( !) (Master Theorem Case 3.a)
( ) = √2 ( /2) +
( ) = √2 ( /2) + => ( ) = Θ(√ ) (Master Theorem Case 1)
( ) = 3 ( /2) +
( ) = 3 ( /2) + => ( ) = Q( ) (Master Theorem Case 1)
( ) = 3 ( /3) + √
( ) = 3 ( /3) + √ => ( ) = Θ( ) (Master Theorem Case 1)
( ) = 4 ( /2) +
( ) = 4 ( /2) + => ( ) = Q( ) (Master Theorem Case 1)
( ) = 3 ( /4) +
( ) = 3 ( /4) + => ( ) = Θ( ) (Master Theorem Case 3.a)
( ) = 3 ( /3) + /2
( ) = 3 ( /3) + /2 => ( ) = Θ( ) (Master Theorem Case 2.a)

1.23 Master Theorem for Subtract and Conquer Recurrences

Let ( ) be a function defined on positive , and having the property
, if ≤ 1
( )=
( − ) + ( ), if > 1
for some constants , > 0, > 0, ≥ 0, and function ( ). If ( ) is in O( ), then
O( ), if a < 1
( )= O( ), if a = 1
O , if a > 1

1.24 Variant of Subtraction and Conquer Master Theorem

The solution to the equation ( ) = ( ) + ((1 − ) ) + , where 0 < < 1 and > 0 are constants, is O( ).

1.25 Method of Guessing and Confirming

Now, let us discuss a method which can be used to solve any recurrence. The basic idea behind this method is:
the answer; and then it correct by induction.
In other words, it addresses the question: What if the given recurrence doesn’t seem to match with any of these (master theorem) methods?
If we guess a solution and then try to verify our guess inductively, usually either the proof will succeed (in which case we are done), or the
proof will fail (in which case the failure will help us refine our guess).
As an example, consider the recurrence T( ) = √ T(√ ) + . This doesn’t fit into the form required by the Master Theorems. Carefully
observing the recurrence gives us the impression that it is similar to the divide and conquer method (dividing the problem into √ subproblems
each with size √ ). As we can see, the size of the subproblems at the first level of recursion is . So, let us guess that T( ) = O( ), and
then try to prove that our guess is correct.
Let’s start by trying to prove an bound T( ) ≤ :
T( ) = √ T(√ ) +
≤ √ . √ √ +
= . √ +
= .c. . +
≤
The last inequality assumes only that 1 ≤ c. . . This is correct if is sufficiently large and for any constant , no matter how small. From
the above proof, we can see that our guess is correct for the upper bound. Now, let us prove the bound for this recurrence.

1.23 Master Theorem for Subtract and Conquer Recurrences 28

 Data Structures and Algorithmic Thinking with Python Introduction

T( ) = √ T(√ ) +
≥ √ . √ √ +
= . √ +
= . . . +
≥
The last inequality assumes only that 1 ≥ . . . This is incorrect if is sufficiently large and for any constant . From the above proof,
we can see that our guess is incorrect for the lower bound.
From the above discussion, we understood that Θ( ) is too big. How about Θ( )? The lower bound is easy to prove directly:
T( ) = √ T(√ ) + ≥
Now, let us prove the upper bound for this Θ( ).
T( )= √ T(√ ) +
≤ √ . .√ +
= . +
= ( + 1)
≰
From the above induction, we understood that Θ( ) is too small and Θ( ) is too big. So, we need something bigger than and smaller
than . How about ?
Proving the upper bound for :
T( ) = √ T(√ ) +
≤ √ . .√ √ +
= . . √ +
√
≤ √
Proving the lower bound for :
T( ) = √ T(√ ) +
≥ √ . .√ √ +
= . . √ +
√
≱ √
The last step doesn’t work. So, Θ( ) doesn’t work. What else is between and ? How about ?
Proving upper bound for :
T( ) = √ T(√ ) +
≤ √ . .√ √ +
= . . - . +
≤ , if ≥ 1
Proving lower bound for :
T( ) = √ T(√ ) +
≥ √ . .√ √ +
= . . - . +
≥ , if ≤ 1
From the above proofs, we can see that T( ) ≤ , if ≥ 1 and T( ) ≥ , if ≤ 1. Technically, we’re still missing the
base cases in both proofs, but we can be fairly confident at this point that T( ) = Θ( ).

1.26 Amortized Analysis

Amortized analysis refers to determining the time-averaged running time for a sequence of operations. It is different from average case
analysis, because amortized analysis does not make any assumption about the distribution of the data values, whereas average case analysis
assumes the data are not "bad" (e.g., some sorting algorithms do well on over all input orderings but very badly on certain input
orderings). That is, amortized analysis is a worst-case analysis, but for a sequence of operations rather than for individual operations.
The motivation for amortized analysis is to better understand the running time of certain techniques, where standard worst-case analysis
provides an overly pessimistic bound. Amortized analysis generally applies to a method that consists of a sequence of operations, where the
vast majority of the operations are cheap, but some of the operations are expensive. If we can show that the expensive operations are
particularly rare, we can ℎ ℎ to the cheap operations, and only bound the cheap operations.

1.26 Amortized Analysis 29

 Data Structures and Algorithmic Thinking with Python Introduction

The general approach is to assign an artificial cost to each operation in the sequence, such that the total of the artificial costs for the sequence
of operations bounds the total of the real costs for the sequence. This artificial cost is called the amortized cost of an operation. To analyze
the running time, the amortized cost thus is a correct way of understanding the overall running time — but note that particular operations can
still take longer so it is not a way of bounding the running time of any individual operation in the sequence.
When one event in a sequence affects the cost of later events:
· One particular task may be expensive.
· But it may leave data structure in a state that the next few operations become easier.
: Let us consider an array of elements from which we want to find the smallest element. We can solve this problem using sorting.
After sorting the given array, we just need to return the element from it. The cost of performing the sort (assuming comparison-based
sorting algorithm) is O( ). If we perform such selections then the average cost of each selection is O( / ) = O( ). This
clearly indicates that sorting once is reducing the complexity of subsequent operations.

1.27 Algorithms Analysis: Problems & Solutions

From the following problems, try to understand the cases which have different complexities (O( ), O( ), O( ) etc.).
Find the complexity of the below recurrence:
3 ( − 1), > 0,
( )=
1, ℎ
Let us try solving this function with substitution.
( ) = 3 ( − 1)
( ) = 3 3 ( − 2) = 3 ( − 2)
( ) = 3 (3 ( − 3))
.
.

( )=3 ( − )=3 (0) = 3

This clearly shows that the complexity of this function is O(3 ).
We can use the master theorem for this problem.
Find the complexity of the below recurrence:
2 ( − 1) − 1, > 0,
( )=
1, ℎ
Let us try solving this function with substitution.
( ) = 2 ( − 1) − 1
( ) = 2(2 ( − 2) − 1) − 1 = 2 ( − 2) − 2 − 1
( ) = 2 (2 ( − 3) − 2 − 1) − 1 = 2 ( − 4) − 2 − 2 − 2
( )=2 ( − )−2 −2 −2 ….2 − 2 −2
( ) = 2 −2 −2 −2 ….2 −2 −2
( ) = 2 − (2 − 1) [ :2 +2 +⋯+2 = 2 ]
( )=1
∴ Time Complexity is O(1). Note that while the recurrence relation looks exponential, the solution to the recurrence relation here gives a
different result.
What is the running time of the following function?
def function(n):
i=s=1
while s < n:
i = i+1
s = s+i
print("*")
function(20)
Consider the comments in the below function:
def function(n):
i=s=1
while s < n: # s is increasing not at rate 1 but i
i = i+1
s = s+i
1.27 Algorithms Analysis: Problems & Solutions 30
 Data Structures and Algorithmic Thinking with Python Introduction

print("*")
function(20)
We can define the ‘ ’ terms according to the relation = + . The value of ‘ ’ increases by 1 for each iteration. The value contained in ‘ ’
at the iteration is the sum of the first ‘ ’ positive integers. If is the total number of iterations taken by the program, then the ℎ loop
terminates if:
( )
1 + 2+ ...+ = > ⟹ = O(√ ).
Find the complexity of the function given below.
def function(n):
i=1
count = 0
while i*i <n:
count = count +1
i=i+1
print(count)
function(20)
In the above-mentioned function the loop will end, if > ⟹ ( ) = O(√ ). This is similar to Problem-23.
What is the complexity of the program given below?
def function(n):
count = 0
for i in range(n/2, n):
j=1
while j + n/2 <= n:
k=1
while k <= n:
count = count + 1
k=k*2
j=j+1
print (count)
function(20)
Observe the comments in the following function.
def function(n):
count = 0
for i in range(n/2, n): #Outer loop execute n/2 times
j=1
while j + n/2 <= n: #Middle loop executes n/2 times
k=1
while k <= n: #Inner loop executes times
count = count + 1
k=k*2
j=j+1
print (count)
function(20)
The complexity of the above function is O( ).
What is the complexity of the program given below?
def function(n):
count = 0
for i in range(n/2, n):
j=1
while j + n/2 <= n:
k=1
while k <= n:
count = count + 1
k=k*2
j=j*2
print (count)
function(20)
Consider the comments in the following function.
def function(n):
count = 0
for i in range(n/2, n): #Outer loop execute n/2 times
j=1
while j + n/2 <= n: #Middle loop executes logn times

1.27 Algorithms Analysis: Problems & Solutions 31

 Data Structures and Algorithmic Thinking with Python Introduction

k=1
while k <= n: #Inner loop executes logn times
count = count + 1
k=k*2
j=j*2
print (count)
function(20)
The complexity of the above function is O( ).
Find the complexity of the program below.
def function(n):
count = 0
for i in range(n/2, n):
j=1
while j + n/2 <= n:
break
j=j*2
print (count)
function(20)
Consider the comments in the function below.
def function(n):
count = 0
for i in range(n/2, n): #Outer loop execute n/2 times
j=1
while j + n/2 <= n: #Middle loop has break statement
break
j=j*2
print (count)
function(20)
The complexity of the above function is O( ). Even though the inner loop is bounded by , but due to the break statement it is executing
only once.
Write a recursive function for the running time ( ) of the function given below. Prove using the iterative method that
( ) = Θ( ).
def function(n):
count = 0
if n <= 0:
return
for i in range(0, n):
for j in range(0, n):
count = count + 1
function(n-3)
print (count)
function(20)
Consider the comments in the function below:
def function(n):
count = 0
if n <= 0:
return
for i in range(0, n): #Outer loop executes n times
for j in range(0, n): #Outer loop executes n times
count = count + 1
function(n-3) #Recursive call
print (count)
function(20)
The recurrence for this code is clearly T( ) = ( − 3) + for some constant > 0 since each call prints out asterisks and calls
itself recursively on - 3. Using the iterative method, we get: ( ) = ( − 3) + . Using the master
theorem, we get ( ) = Θ( ).
Determine Θ bounds for the recurrence relation: ( ) = 2 + .
Using Divide and Conquer master theorem, we get: O( ).
Determine Θ bounds for the recurrence: ( ) = + + + .

1.27 Algorithms Analysis: Problems & Solutions 32

 Data Structures and Algorithmic Thinking with Python Introduction

Substituting in the recurrence equation, we get: ( ) ≤ 1 ∗ + 2 ∗ + 3 ∗ + ≤ ∗ , where is a constant. This

clearly says Θ( ).
Determine Θ bounds for the recurrence relation: ( ) = é /2ù + 7.
Using Master Theorem we get: Θ( ).
Prove that the running time of the code below is Ω( ).
def Read(n):
k=1
while k < n:
k = 3*k
The ℎ loop will terminate once the value of ‘ ’ is greater than or equal to the value of ‘ ’. In each iteration the value of ‘ ’ is
multiplied by 3. If is the number of iterations, then ‘ ’ has the value of 3 after iterations. The loop is terminated upon reaching iterations
when 3 ≥ ↔ ≥ log , which shows that = Ω ( ).
Solve the following recurrence.
1, =1
( ) =
( − 1) + ( − 1), ≥2
By iteration:
( ) = ( − 2) + ( − 1)( − 2) + ( − 1)
…

( ) = (1) + ( − 1)

( ) = (1) + −

(( + 1)(2 + 1) ( + 1)
( )=1+ −
6 2
( ) =Q( )
We can use the master theorem for this problem.
Consider the following program:
def Fib(n):
if n == 0: return 0
elif n == 1: return 1
else: return Fib(n-1)+ Fib(n-2)
print(Fib(3))
The recurrence relation for the running time of this program is: ( ) = ( − 1) + ( − 2) + . Note T( ) has two recurrence
calls indicating a binary tree. Each step recursively calls the program for reduced by 1 and 2, so the depth of the recurrence tree is O( ).
The number of leaves at depth is 2 since this is a full binary tree, and each leaf takes at least O(1) computations for the constant factor.
Running time is clearly exponential in and it is O(2 ).
Running time of following program?
def function(n):
count = 0
if n <= 0:
return
for i in range(0, n):
j=1
while j <n:
j=j+i
count = count + 1
print (count)
function(20)
Consider the comments in the function below:
def function(n):
count = 0
if n <= 0:
return
for i in range(0, n): #Outer loop executes n times
j=1 #Inner loop executes j increase by the rate of i
while j <n:

1.27 Algorithms Analysis: Problems & Solutions 33

 Data Structures and Algorithmic Thinking with Python Introduction

j=j+i
count = count + 1
print (count)
function(20)
In the above code, inner loop executes / times for each value of . Its running time is × (∑ni=1 n/i) = O( ).
What is the complexity of ∑ ?
Using the logarithmic property, = + , we can see that this problem is equivalent to

= 1+ 2 + ⋯+ = (1 × 2 × … × ) = ( !) ≤ ( )≤

This shows that the time complexity = O( ).

What is the running time of the following recursive function (specified as a function of the input value )? First write the
recurrence formula and then find its complexity.
def function(n):
if n <= 0:
return
for i in range(0, 3):
function(n/3)
function(20)
Consider the comments in the below function:
def function(n):
if n <= 0:
return
for i in range(0, 3): #This loop executes 3 times with recursive value of value
function(n/3)
function(20)
We can assume that for asymptotical analysis = é ù for every integer ≥ 1. The recurrence for this code is ( ) = 3 ( ) + Θ(1).
Using master theorem, we get ( ) = Θ( ).
What is the running time of the following recursive function (specified as a function of the input value )? First write a
recurrence formula, and show its solution using induction.
def function(n):
if n <= 0:
return
for i in range(0, 3): #This loop executes 3 times with recursive value of value
function(n-1)
function(20)
Consider the comments in the function below:
def function(n):
if n <= 0:
return
for i in range(0, 3): #This loop executes 3 times with recursive value of − 1 value
function(n-1)
function(20)
The statement requires constant time [O(1)]. With the loop, we neglect the loop overhead and only count three times that the function
is called recursively. This implies a time complexity recurrence:
( ) = , ≤ 1;
= + 3 ( − 1), > 1.
Using the master theorem, we get ( ) = Θ(3 ).
Write a recursion formula for the running time ( ) of the function whose code is below.
def function3(n):
if n <= 0:
return
for i in range(0, 3): #This loop executes 3 times with recursive value of n/3 value
function3(0.8 * n)
function3(20)
Consider the comments in the function below:
def function3(n):
1.27 Algorithms Analysis: Problems & Solutions 34
 Data Structures and Algorithmic Thinking with Python Introduction

if n <= 0:
return
for i in range(0, 3): #This loop executes 3 times with recursive value of 0.8n value
function3(0.8 * n)
function3(20)
The recurrence for this piece of code is ( ) = (. 8 ) + O( ) = T(4/5 ) + O( ) = 4/5 T( ) + O( ). Applying master theorem, we get
T( ) = O( ).
Find the complexity of the recurrence: ( ) = 2 (√ ) +
The given recurrence is not in the master theorem format. Let us try to convert this to the master theorem format by assuming =
2 . Applying the logarithm on both sides gives, = 2⟹ = . Now, the given function becomes:
( ) = (2 ) = 2 √2 + =2 2 + .
To make it simple we assume ( ) = (2 ) ⟹ ( ) = (2 ) ⟹ ( ) = 2 + .
Applying the master theorem format would result in ( ) = O( ).
If we substitute = back, ( ) = ( ) = O(( ) ).
Find the complexity of the recurrence: ( ) = (√ ) + 1
Applying the logic of Problem-40 gives ( ) = + 1. Applying the master theorem would result in ( ) = O( ).
Substituting = , gives ( ) = ( ) = O( ).
Find the complexity of the recurrence: ( ) = 2 (√ ) + 1
Applying the logic of Problem-40 gives: ( ) = 2 + 1. Using the master theorem results ( ) = O = O( ).
Substituting = gives ( ) = O( ).
Find the complexity of the below function.
import math
count = 0
def function(n):
global count
if n <= 2:
return 1
else:
function(round(math.sqrt(n)))
count = count + 1
return count
print(function(200))
Consider the comments in the function below:
import math
count = 0
def function(n):
global count
if n <= 2:
return 1
else:
function(round(math.sqrt(n))) #Recursive call with √ value
count = count + 1
return count
print(function(200))
For the above code, the recurrence function can be given as: ( ) = (√ ) + 1. This is same as that of Problem-41.
Analyze the running time of the following recursive pseudo-code as a function of .
def function(n):
if (n < 2):
return
else:
counter = 0
for i in range(0,8):
function (n/2)
for i in range(0,n**3):
counter = counter + 1
Consider the comments in below pseudo-code and call running time of function(n) as ( ).
def function(n):
if (n < 2): # Constant time

1.27 Algorithms Analysis: Problems & Solutions 35

 Data Structures and Algorithmic Thinking with Python Introduction

return
else:
counter = 0 # Constant time
for i in range(0,8): # This loop executes 8 times with n value half in every call
function (n/2)
for i in range(0,n**3): # This loop executes times with constant time loop
counter = counter + 1
( ) can be defined as follows:
( ) = 1 < 2,
=8 ( ) + + 1 ℎ .
2
Using the master theorem gives: ( ) = Θ( ) = Θ( ).
Find the complexity of the below pseudocode.
count = 0
def function(n):
global count
count = 1
if n <= 0:
return
for i in range(0, n):
count = count + 1
n = n//2
function(n)
print (count)
function(200)
Consider the comments in the pseudocode below:
count = 0
def function(n):
global count
count = 1
if n <= 0:
return
for i in range(1, n): # This loops executes n times
count = count + 1
n = n//2 #Integer Division
function(n) #Recursive call with value
print (count)
function(200)
The recurrence for this function is ( ) = ( /2) + . Using master theorem, we get ( ) = O( ).
Running time of the following program?
def function(n):
for i in range(1, n):
j=1
while j <= n:
j=j*2
print("*")
function(20)
Consider the comments in the below function:
def function(n):
for i in range(1, n): # This loops executes n times
j=1
while j <= n: # This loops executes times from our logarithm’s guideline
j=j*2
print("*")
function(20)
Complexity of above program is: O( ).
Running time of the following program?
def function(n):
for i in range(0, n/3):
j=1
while j <= n:
j=j+4

1.27 Algorithms Analysis: Problems & Solutions 36

 Data Structures and Algorithmic Thinking with Python Introduction

print("*")
function(20)
Consider the comments in the below function:
def function(n):
for i in range(0, n/3): #This loops executes n/3 times
j=1
while j <= n: #This loops executes n/4 times
j=j+4
print("*")
function(20)
The time complexity of this program is: O( ).
Find the complexity of the below function:
def function(n):
if n <= 0:
return
print ("*")
function(n/2)
function(n/2)
print ("*")
function(20)
Consider the comments in the below function:
def function(n):
if n <= 0: #Constant time
return
print ("*") #Constant time
function(n/2) #Recursion with n/2 value
function(n/2) #Recursion with n/2 value
print ("*")
function(20)
The recurrence for this function is: ( ) = 2 + 1. Using master theorem, we get ( ) = O( ).
Find the complexity of the below function:
count = 0
def logarithms(n):
i=1
global count
while i <= n:
j=n
while j > 0:
j = j//2
count = count + 1
i= i * 2
return count
print(logarithms(10))

count = 0
def logarithms(n):
i=1
global count
while i <= n:
j=n
while j > 0:
j = j//2 # This loops executes times from our logarithm’s guideline
count = count + 1
i= i * 2 # This loops executes times from our logarithm’s guideline
return count
print(logarithms(10))
Time Complexity: O( ∗ ) = O( ).
∑ ( ), where O( ) stands for order is:
(a) O( ) (b) O( ) (c) O( ) (d) O(3 ) (e) O(1.5 )
( ). ∑ ( ) = O( ) ∑ 1 = O( ).
1.27 Algorithms Analysis: Problems & Solutions 37
 Data Structures and Algorithmic Thinking with Python Introduction

Which of the following three claims are correct?

I ( + ) = Q( ), where and are constants II 2 = O(2 ) III 2 = O(2 )
(a) I and II (b) I and III (c) II and III (d) I, II and III
( ). (I) ( + ) = + c1* + ... = Q( ) and (II) 2 = 2*2 = O(2 )
Consider the following functions:
f( ) = 2 g( ) = ! h( ) =
Which of the following statements about the asymptotic behavior of f( ), g( ), and h( ) is true?
(A) f( ) = O(g( )); g( ) = O(h( )) (B) f( ) = W (g( )); g( ) = O(h( ))
(C) g( ) = O(f( )); h( ) = O(f( )) (D) h( ) = O(f( )); g( ) = W (f( ))
: ( ). According to the rate of growth: h( ) < f( ) < g( ) (g( ) is asymptotically greater than f( ), and f( ) is asymptotically greater than
h( )). We can easily see the above order by taking logarithms of the given 3 functions: < < ( !). Note that, ( !) =
O( ).
Consider the following segment of C-code:
j=1
while j <=n:
j = j*2
The number of comparisons made in the execution of the loop for any > 0 is:
(A) ceil( )+ 1 (B) (C) ceil( ) (D) floor( )+1
( ). Let us assume that the loop executes times. After step the value of is 2 . Taking logarithms on both sides gives = .
Since we are doing one more comparison for exiting from the loop, the answer is ceil( )+ 1.
Consider the following C code segment. Let T( ) denote the number of times the for loop is executed by the program
on input . Which of the following is true?
import math
def IsPrime(n):
for i in range(2, math.sqrt(n)):
if n%i == 0:
print(“Not Prime”)
return 0
return 1
(A) T( ) = O(√ ) and T( ) = W(√ ) (B) T( ) = O(√ ) and T( ) = W(1)
(C) T( ) = O( ) and T( ) = W(√ ) (D) None of the above
( ). Big O notation describes the tight upper bound and Big Omega notation describes the tight lower bound for an algorithm. The
loop in the question is run maximum √ times and minimum 1 time. Therefore, T( ) = O(√ ) and T( ) = W(1).
In the following C function, let ≥ . How many recursive calls are made by this function?
def gcd(n,m):
if n%m ==0:
return m
n = n%m
return gcd(m,n)
(A) Q( ) (B) W( ) (C) Q( ) (D) Q( )
No option is correct. Big O notation describes the tight upper bound and Big Omega notation describes the tight lower bound for
an algorithm. For = 2 and for all = 2 , the running time is O(1) which contradicts every option.
Suppose ( ) = 2 ( /2) + , T(0)=T(1)=1. Which one of the following is false?
(A) ( ) = O( ) (B) ( ) = Q( ) (C) ( ) = W( ) (D) ( ) = O( )
( ). Big O notation describes the tight upper bound and Big Omega notation describes the tight lower bound for an algorithm.
Based on master theorem, we get ( ) = Q( ). This indicates that tight lower bound and tight upper bound are the same. That means,
O( ) and W( ) are correct for given recurrence. So, option (C) is wrong.
Find the complexity of the below function:
def function(n):
for i in range(1, n):
j=i
while j <i*i:
j=j+1
if j %i == 0:
for k in range(0, j):
print(" * ")
function(10)

1.27 Algorithms Analysis: Problems & Solutions 38

 Data Structures and Algorithmic Thinking with Python Introduction

def function(n):
for i in range(1, n): # Executes n times
j=i
while j <i*i: # Executes n*n times
j=j+1
if j %i == 0:
for k in range(0, j): #Executes j times = (n*n) times
print(" * ")
function(10)
Time Complexity: O( ).
To calculate 9 , give an algorithm and discuss its complexity.
Start with 1 and multiply by 9 until reaching 9 .
Time Complexity: There are − 1 multiplications and each takes constant time giving a Q( ) algorithm.
For Problem-58, can we improve the time complexity?
Refer to the chapter.
Find the complexity of the below function:
def function(n):
sum = 0
for i in range(0, n-1):
if i > j:
sum = sum + 1
else:
for k in range(0, j):
sum = sum - 1
print (sum)
function(10)
Consider the − and we can ignore the value of j.
def function(n):
sum = 0
for i in range(0, n-1): # Executes times
if i > j:
sum = sum + 1 # Executes times
else:
for k in range(0, j): # Executes times
sum = sum - 1
print (sum)
function(10)
Time Complexity: O( ).
Solve the following recurrence relation using the recursion tree method: T( )=T( ) +T( )+ .
How much work do we do in each level of the recursion tree?
T( )

T( ) T( )

2
T( ) T( ) T( ) T( )
2 3

T( ) T( ) T( ) T( ) 2 T( ) T( ) T( ) T( ) 2
2 3 2 3
In level 0, we take time. At level 1, the two subproblems take time:

1.27 Algorithms Analysis: Problems & Solutions 39

 Data Structures and Algorithmic Thinking with Python Introduction

1 2 1 4 25
+ = + =
2 3 4 9 36
At level 2 the four subproblems are of size , , , and respectively. These two subproblems take time:

1 1 1 4 625 25
+ + + = =
4 3 3 9 1296 36

Similarly, the amount of work at level is at most .

Let = , the total runtime is then:

T( ) ≤

1
=
1−∝
1
= 25
1−
36
1
= 11
36
36
=
11
= O( )
That is, the first level provides a constant fraction of the total runtime.
Find the time complexity of recurrence T(n) = T( ) + T( ) + T( ) + .
: Let us solve this problem by method of guessing. The total size on each level of the recurrence tree is less than , so we guess that
( ) = will dominate. Assume for all < that ≤ T( ) ≤ . Then,
+ + + ≤ T( ) ≤ + + +
( + + + ) ≤ T( ) ≤ ( + + + )
( + ) ≤ T( ) ≤ ( + )

If ≥ 8k and ≤ 8k, then = T( ) = . So, T( ) = Θ( ). In general, if you have multiple recursive calls, the sum of the arguments to
those calls is less than n (in this case + + < ), and ( ) is reasonably large, a good guess is T( ) = Θ(f( )).

Rank the following functions by order of growth: ( + 1)!, n!, 4 , ×3 ,3 + + 20 , ( ) , 4 ,4 , + 200,

20 + 500, 2 , / , 1.
:
Function Rate of Growth
( + 1)! O( !)
! O( !)
4 O(4 )
×3 O( 3 )
3 + + 20 O(3 )
3 O(( ) )
( ) Decreasing rate of growths
2
4 O( )
4 O( )
+ 200 O( )
20 + 500 O( )
2 O( )
/
O( / )
1 O(1)
.
Can we say 3 = O(3 )?
.
: Yes: because 3 < 3 .
Can we say 2 = O(2 )?
: No: because 2 = (2 ) = 8 not less than 2 .

1.27 Algorithms Analysis: Problems & Solutions 40

 Data Structures and Algorithmic Thinking with Python Recursion and Backtracking

Chapter

Recursion and
Backtracking 2
2.1 Introduction
In this chapter, we will look at one of the important topics, “ ”, which will be used in almost every chapter, and also its relative
“ ”.

2.2 What is Recursion?

Any function which calls itself is called . A recursive method solves a problem by calling a copy of itself to work on a smaller
problem. This is called the recursion step. The recursion step can result in many more such recursive calls.
It is important to ensure that the recursion terminates. Each time the function calls itself with a slightly simpler version of the original problem.
The sequence of smaller problems must eventually converge on the base case.

2.3 Why Recursion?

Recursion is a useful technique borrowed from mathematics. Recursive code is generally shorter and easier to write than iterative code.
Generally, loops are turned into recursive functions when they are compiled or interpreted.
Recursion is most useful for tasks that can be defined in terms of similar subtasks. For example, sort, search, and traversal problems often
have simple recursive solutions.

2.4 Format of a Recursive Function

A recursive function performs a task in part by calling itself to perform the subtasks. At some point, the function encounters a subtask that it
can perform without calling itself. This case, where the function does not recur, is called the . The former, where the function calls
itself to perform a subtask, is referred to as the . We can write all recursive functions using the format:
if(test for the base case):
return some base case value
elif(test for another base case):
return some other base case value
# the recursive case
else:
return (some work and then a recursive call)
As an example, consider the factorial function: ! is the product of all integers between and 1. The definition of recursive factorial looks
like:
! = 1, if = 0
! = ∗ ( − 1)! if > 0

This definition can easily be converted to recursive implementation. Here the problem is determining the value of !, and the subproblem is
determining the value of ( − )!. In the recursive case, when is greater than 1, the function calls itself to determine the value of ( − )! and
multiplies that with .
In the base case, when is 0 or 1, the function simply returns 1. This looks like the following:
// calculates factorial of a positive integer
def factorial(n):
if n == 0: return 1
return n*factorial(n-1)
print(factorial(6))

2.1 Introduction 41
 Data Structures and Algorithmic Thinking with Python Recursion and Backtracking

2.5 Recursion and Memory (Visualization)

Each recursive call makes a new copy of that method (actually only the variables) in memory. Once a method ends (that is, returns some
data), the copy of that returning method is removed from memory. The recursive solutions look simple but visualization and tracing takes
time. For better understanding, let us consider the following example.
def printFunc(n):
if n == 0: # this is the terminating base case
return 0
else:
print (n)
return printFunc(n-1) # recursive call to itself again
print(printFunc(4))
For this example, if we call the print function with n=4, visually our memory assignments may look like:
printFunc(4)

printFunc(3)

Returns 0 printFunc(2)

printFunc(1)
Returns 0

Returns 0 to main function printFunc(0)

Returns 0

Returns 0

Now, let us consider our factorial function. The visualization of factorial function with = 4 will look like:

4!

4* 3!

4*6=24 is returned 3*2!

3*2=6 is returned 2*1!

Returns 24 to main function 2*1=2 is returned 1

Returns 1
2.6 Recursion versus Iteration
While discussing recursion, the basic question that comes to mind is: which way is better? – iteration or recursion? The answer to this question
depends on what we are trying to do. A recursive approach mirrors the problem that we are trying to solve. A recursive approach makes it
simpler to solve a problem that may not have the most obvious of answers. But recursion adds overhead for each recursive call (needs space
on the stack frame).

Recursion
· Terminates when a base case is reached.
· Each recursive call requires extra space on the stack frame (memory).
· If we get infinite recursion, the program may run out of memory and result in stack overflow.
· Solutions to some problems are easier to formulate recursively.

Iteration
· Terminates when a condition is proven to be false.
· Each iteration does not require extra space.
· An infinite loop could loop forever since there is no extra memory being created.
· Iterative solutions to a problem may not always be as obvious as a recursive solution.

2.7 Notes on Recursion

· Recursive algorithms have two types of cases, recursive cases and base cases.
· Every recursive function case must terminate at a base case.
· Generally, iterative solutions are more efficient than recursive solutions [due to the overhead of function calls].
· A recursive algorithm can be implemented without recursive function calls using a stack, but it’s usually more trouble than its worth.
That means any problem that can be solved recursively can also be solved iteratively.

2.5 Recursion and Memory (Visualization) 42

 Data Structures and Algorithmic Thinking with Python Recursion and Backtracking

· For some problems, there are no obvious iterative algorithms.

· Some problems are best suited for recursive solutions while others are not.

2.8 Example Algorithms of Recursion

· Fibonacci series, factorial finding
· Merge sort, quick sort
· Binary search
· Tree traversals and many tree problems: InOrder, PreOrder PostOrder
· Graph traversals: DFS [Depth First Search] and BFS [Breadth First Search]
· Dynamic programming examples
· Divide and conquer algorithms
· Towers of Hanoi
· Backtracking algorithms [we will discuss in next section]

2.9 Recursion: Problems & Solutions

In this chapter we cover a few problems with recursion and we will discuss the rest in other chapters. By the time you complete reading the
entire book, you will encounter many recursion problems.
Discuss Towers of Hanoi puzzle.
The Towers of Hanoi is a mathematical puzzle. It consists of three rods (or pegs or towers) and a number of disks of different sizes
which can slide onto any rod. The puzzle starts with the disks on one rod in ascending order of size, the smallest at the top, thus making a
conical shape. The objective of the puzzle is to move the entire stack to another rod, satisfying the following rules:
· Only one disk may be moved at a time.
· Each move consists of taking the upper disk from one of the rods and sliding it onto another rod, on top of the other disks that may
already be present on that rod.
· No disk may be placed on top of a smaller disk.

· Move the top − 1 disks from to tower,

· Move the disk from to tower,
· Move the − 1disks from tower to tower.
· Transferring the top − 1 disks from to tower can again be thought of as a fresh problem and can be solved
in the same manner. Once we solve with three disks, we can solve it with any number of disks with the above
algorithm.
def towersOfHanoi(numberOfDisks, startPeg=1, endPeg=3):
if numberOfDisks:
towersOfHanoi (numberOfDisks-1, startPeg, 6-startPeg-endPeg)
print ("Move disk %d from peg %d to peg %d" % (numberOfDisks, startPeg, endPeg))
towersOfHanoi (numberOfDisks-1, 6-startPeg-endPeg, endPeg)
towersOfHanoi (numberOfDisks=4)
Given an array, check whether the array is in sorted order with recursion.
:
def isArrayInSortedOrder(A):
# Base case
if len(A) == 1:
return True
return A[0] <= A[1] and isArrayInSortedOrder(A[1:])
A = [127, 220, 246, 277, 321, 454, 534, 565, 933]
print(isArrayInSortedOrder(A))
Time Complexity: O( ). Space Complexity: O( ) for recursive stack space.

2.10 What is Backtracking?

Backtracking is an improvement of the brute force approach. It systematically searches for a solution to a problem among all available options.
In backtracking, we start with one possible option out of many available options and try to solve the problem if we are able to solve the
problem with the selected move then we will print the solution else we will backtrack and select some other option and try to solve it. If none
if the options work out, we will claim that there is no solution for the problem.
Backtracking is a form of recursion. The usual scenario is that you are faced with a number of options, and you must choose one of these.
After you make your choice you will get a new set of options; just what set of options you get depends on what choice you made. This
procedure is repeated over and over until you reach a final state. If you made a good sequence of choices, your final state is a goal state; if you
2.8 Example Algorithms of Recursion 43
 Data Structures and Algorithmic Thinking with Python Recursion and Backtracking

didn't, it isn't. Backtracking can be thought of as a selective tree/graph traversal method. The tree is a way of representing some initial starting
position (the root node) and a final goal state (one of the leaves). Backtracking allows us to deal with situations in which a raw brute-force
approach would explode into an impossible number of options to consider. Backtracking is a sort of refined brute force. At each node, we
eliminate choices that are obviously not possible and proceed to recursively check only those that have potential.
What’s interesting about backtracking is that we back up only as far as needed to reach a previous decision point with an as-yet-unexplored
alternative. In general, that will be at the most recent decision point. Eventually, more and more of these decision points will have been fully
explored, and we will have to backtrack further and further. If we backtrack all the way to our initial state and have explored all alternatives
from there, we can conclude the particular problem is unsolvable. In such a case, we will have done all the work of the exhaustive recursion
and known that there is no viable solution possible.
· Sometimes the best algorithm for a problem is to try all possibilities.
· This is always slow, but there are standard tools that can be used to help.
· Tools: algorithms for generating basic objects, such as binary strings [2 possibilities for -bit string], permutations
[ !], combinations [ !/ ! ( − )!], general strings [ −ary strings of length has possibilities], etc...
· Backtracking speeds the exhaustive search by pruning.

2.11 Example Algorithms of Backtracking

· Binary Strings: generating all binary strings
· Generating −ary Strings
· N-Queens Problem
· The Knapsack Problem
· Generalized Strings
· Hamiltonian Cycles [refer ℎ chapter]
· Graph Coloring Problem

2.12 Backtracking: Problems & Solutions

Generate all the binary strings with bits. Assume [0. . − 1] is an array of size .

def appendAtFront(x, L):

return [x + element for element in L]
def bitStrings(n):
if n == 0: return []
if n == 1: return ["0", "1"]
else:
return (appendAtFront("0", bitStrings(n-1)) + appendAtFront("1", bitStrings(n-1)))
print (bitStrings(4))

def bitStrings(n):
if n == 0: return []
if n == 1: return ["0", "1"]
return [ digit+bitstring for digit in bitStrings(1)
for bitstring in bitStrings(n-1)]
print (bitStrings(4))
Let ( ) be the running time of ( ). Assume function takes time O(1).
, if < 0
( )=
2 ( − 1) + , otherwise
Using Subtraction and Conquer Master theorem we get: ( ) = O(2 ). This means the algorithm for generating bit-strings is optimal.
Generate all the strings of length drawn from 0. . . − 1.
Let us assume we keep current k-ary string in an array [0. . − 1]. Call function - (n, k):
def rangeToList(k):
result = []
for i in range(0,k):
result.append(str(i))
return result
def baseKStrings (n,k):

2.11 Example Algorithms of Backtracking 44

 Data Structures and Algorithmic Thinking with Python Recursion and Backtracking

if n == 0: return []
if n == 1: return rangeToList(k)
return [ digit+bitstring for digit in baseKStrings (1,k)
for bitstring in baseKStrings (n-1,k)]
print (baseKStrings (4,3))
Let ( ) be the running time of − ( ). Then,
, <0
( )=
( − 1) + , ℎ
Using Subtraction and Conquer Master theorem we get: ( ) = O( ).
For more problems, refer to ℎ chapter.
Solve the recurrence T( ) = 2T( − 1) + 2 .
: At each level of the recurrence tree, the number of problems is double from the previous level, while the amount of work being
done in each problem is half from the previous level. Formally, the level has 2 problems, each requiring 2 work. Thus the level
requires exactly 2 work. The depth of this tree is , because at the level, the originating call will be T( − ). Thus, the total complexity
for T( ) is T( 2 ).
: Given a matrix, each of which may be 1 or
0. The filled cells that are connected form a region. Two cells are said to be connected if they are adjacent to each other horizontally,
vertically or diagonally. There may be several regions in the matrix. How do you find the largest region (in terms of number of cells) in the
matrix?
Sample Input 11000 Sample Output: 5
01100
00101
10001
01011
The simplest idea is: for each location traverse in all 8 directions and in each of those directions keep track of maximum region
found.
def getval(A, i, j, L, H):
if (i< 0 or i >= L or j< 0 or j >= H):
return 0
else:
return A[i][j]
def findMaxBlock(A, r, c, L, H, size):
global maxsize
global cntarr
if ( r >= L or c >= H):
return
cntarr[r][c]=1
size += 1
if (size > maxsize):
maxsize = size
#search in eight directions
direction=[[-1,0],[-1,-1],[0,-1],[1,-1],[1,0],[1,1],[0,1],[-1,1]]
for i in range(0,7):
newi =r+direction[i][0]
newj=c+direction[i][1]
val=getval (A, newi, newj, L, H)
if (val>0 and (cntarr[newi][newj]==0)):
findMaxBlock(A, newi, newj, L, H, size)
cntarr[r][c]=0
def getMaxOnes(A, rmax, colmax):
global maxsize
global size
global cntarr
for i in range(0,rmax):
for j in range(0,colmax):
if (A[i][j] == 1):
findMaxBlock (A, i, j, rmax, colmax, 0)
return maxsize
zarr=[[1,1,0,0,0],[0,1,1,0,1],[0,0,0,1,1],[1,0,0,1,1],[0,1,0,1,1]]
rmax = 5

2.12 Backtracking: Problems & Solutions 45