Design and Analysis of Algorithms

Unit-I: Introduction

Dr. Dwiti Krishna Bebarta

Text Books:
1.Ellis Horowitz, SartajSahni, Sanguthevar Rajasekaran, “Fundamentals of
Computer Algorithms”, 2nd Edition, Universities Press.
2.Introduction to Algorithms Thomas H. Cormen, PHI Learning
3.Harsh Bhasin, “Algorithms Design & Analysis”, Oxford University Press.
 UNIT I: Introduction to
Problem Solving Concepts

1. Problem definition
2. Algorithm definition
3. Algorithm specification
4. Performance analysis of an algorithm
5. Performance Measurement
6. Asymptotic notation
7. Randomized Algorithms
 Problem Definition

What is the task to be accomplished?

Calculate the average of the grades for a given student
Understand the talks given out by politicians and
translate them in Chinese
What are the time / space / speed / performance
requirements ?
Algorithm Definition
Algorithm: Finite set of instructions that, if followed,
accomplishes a particular task.
Describe: in natural language / pseudo-code / diagrams /
etc.
Criteria to follow:
Input: Zero or more quantities (externally produced)
Output: One or more quantities
Definiteness: Clarity, precision of each instruction
Finiteness: The algorithm has to stop after a finite (may
be very large) number of steps
Effectiveness: Each instruction has to be basic enough
and feasible

• Understand speech
• Translate to Chinese
Algorithm Specification

Algorithm can be specified in three ways

Natural Language like English
Graphics representations like Flowcharts
Pseudo-code Method
Pseudo-code Method
We present most of our algorithms using a pseudo-code that
resembles C and Pascal.
Comments begin with // and continue until the end of line.
Blocks are indicated with matching braces: { and }
A compound statement(i.e., a collection of simple
statements) can be represented as a block.
An identifier begins with a letter. The data types of variables
are not explicitly declared. The types will be clear from the
context. Whether a variable is global or local to a procedure
will also be evident from the context. We assume simple
data types such as integer, float, char, boolean, and so on.
Compound data types can be formed with records. Here is
an example:
node= record { datatype_1 data_1; datatype_n data_n; node
*link; }
Pseudo-code Method Cont.
Assignment of values to variables is done using the
assignment statement (variable):=(expression);
There are two boolean values true and false. In order to
produce these values, the logical operators and, or, and not
and the relational operators <, >, etc. are provided.
Elements of multidimensional arrays for example, if A is a two
dimensional array, the (i,j)th element of the array is denoted
as-A[i,j]. Array indices start at zero.
The following looping statements are employed: for, while,
and repeat- until. The while loop takes the following form:
while (condition)
do { (statement 1) ---- (statement n) }
Pseudo-code Method Cont.
for variable:=value-l to value-2 increment/decrement
do { (statement-1) ----- (statement-n) }
Here value-l,value-2,and increment/decrement are arithmetic
expression

A repeat-until statement is constructed as follows: repeat

{statement-1) ----- {statement-n)
until{condition)
Pseudo-code Method Cont.
A conditional statement has the following forms:
if {condition) then {statement)
if {condition) then {statement-1) else (statement-2)
We also employ the following case statement:
case { :{condition-1) : (statement-1)
-------------
:(condition-n) : (statement-n)
:else: (statement n + 1)
}
Pseudo-code Method Cont.
As an example, the following algorithm finds and returns the
maximum of n given numbers:
Algorithm Max (A, n)
{
// A is an array of size n.
Result:=A[l];
for i :=2 to n do
if A[i] >Result then
Result:=A[I];
return Result;
}
In this algorithm (named Max), A and n are procedure
parameters. Result and i are local variables.
Performance Analysis
There are many criteria upon which we can judge an algorithm.
For instance:
Does it do what we want it to do?
Does it work correctly according to the original specifications of
the task?
Is there documentation that describes how to use it and how it
works?
Are procedures created in such a way that they perform logical
sub-functions
Is the code readable?
Performance analysis can be loosely divided into two major phases:
(1) a priori estimates and
(2) a posteriori testing.
We refer to these as performance analysis and performance
measurement respectively
 3. Algorithm Analysis

Space complexity
How much space is required
Time complexity
How much time does it take to run the algorithm
Often, we deal with estimates!
 Space Complexity

Space complexity = The amount of memory required

by an algorithm to run to completion
[Core dumps = the most often encountered cause is
“memory leaks” – the amount of memory required larger
than the memory available on a given system]
Some algorithms may be more efficient if data
completely loaded into memory
Need to look also at system limitations
E.g. Classify 2GB of text in various categories [politics,
tourism, sport, natural disasters, etc.] – can I afford to load
the entire collection?
 Space Complexity (cont’d)
1. Fixed part: The size required to store certain
data/variables, that is independent of the size of the
problem:

2. Variable part: Space needed by variables, whose size is

dependent on the size of the problem:
- e.g. actual text
- load 2GB of text VS. load 1MB of text
Space Complexity (cont’d)
S(P) = c + S(instance characteristics)
c = constant
Example:
int square(int a) { return a*a; }
In the above piece of code, it requires 2 bytes of memory to
store variable 'a' and another 2 bytes of memory is used
for return value.
That means, totally it requires 4 bytes of memory to
complete its execution. And this 4 bytes of memory is fixed
for any input value of 'a'. This space complexity is said to
be Constant Space Complexity.
E.g.: 1
Algorithm abc(a, b, c)
return a + b+b*c+(a+b-c) / (a+b)+40

The problem instance is characterized by the specific values

of a, b, and c.
Hence the space needed for a, b, c is independent of the
values of a, b, c.
Therefore, S(instance Characteristics) = 0 and c=3
S(p)=c + S(instance characteristics)=3+0=3
Space Complexity (cont’d)
S(P) = c + S(instance characteristics)
c = constant
Example:
int sum(int A[ ], int n)
{ int sum = 0, i;
for(i = 0; i < n; i++)
sum = sum + A[i];
return sum;
}
In the above piece of code it requires
'n*2' bytes of memory to store array variable 'a[ ]'
2 bytes of memory for integer parameter 'n'
4 bytes of memory for local integer variables 'sum' and 'i' (2 bytes each)
2 bytes of memory for return value.
That means, totally it requires '2n+8' bytes of memory to complete its
execution. Here, the total amount of memory required depends on the
value of 'n'. As 'n' value increases the space required also increases
proportionately. This type of space complexity is said to be Linear Space
Complexity.
 Example: 3
Algorithm R-sum (A, n)
if (n≤0) then return 0
else return R-sum(A, n-1)+A[n]

The problem instances are characterized by n. The recursion

stack space includes space for the for the formal parameters, the
local variables, and the return address.
Each call to R-sum requires at least three words including space
for the values of n, the return address, and a pointer to A[ ] ).
Since the depth of recursion is n+1, the recursion stack space
needed is ≥ 3 (n+1)
S (R-sum ) = 3 (n+1)
Time Complexity
Most algorithms transform input objects into output
objects.
The running time of an algorithm typically grows with the
input size.
Average case time is often difficult to determine.
We focus on the worst case running time.
best case
average case
worst case
120

100
Running Time

80

60

40

20

0
1000 2000 3000 4000
Input Size
Use a Theoretical Approach
Based on high-level description of the algorithms, rather
than language dependent implementations
Makes possible an evaluation of the algorithms that is
independent of the hardware and software environments
Pseudo-Code = a description of an algorithm that is
more structured than usual prose but
less formal than a programming language
Example: find the maximum element of an array.
Algorithm arrayMax(A, n):
Input: An array A storing n integers.
Output: The maximum element in A.
currentMax  A[0]
for i 1 to n -1 do
if currentMax < A[i] then currentMax  A[i]
return currentMax
Counting Primitive Operations
Example: find the maximum element of an array.
Algorithm arrayMax(A, n) No. of Operations
//Input: An array A storing n integers. 00
//Output: The maximum element in A. 00
currentMax  A[0] 01 time
for i 1 to n -1 do n times
if currentMax < A[i] then (n-1) times
currentMax  A[i] (n-1) times
return currentMax 01 time
Total: 3n times
Estimating Running Time

Algorithm arrayMax executes 3n primitive operations in

the worst case. Define:
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive operation
Let T(n) be worst-case time of arrayMax.
a (8n  2)  T(n)  b(8n  2)
Hence, the running time T(n) is bounded by two linear
functions
Growth Rate of Running Time: The linear growth rate of
the running time T(n) is an intrinsic property of algorithm
arrayMax
Seven functions that often appear in algorithm analysis:
Constant  1
Logarithmic  log n
Linear  n
N-Log-N  n log n
Quadratic  n2
Cubic  n3
Exponential  2n
Asymptotic Complexity
Running time of an algorithm as a function of input size n
for large n.
Expressed using only the highest-order term in the
expression for the exact running time.
Written using Asymptotic Notation Q, O, W, o, w
The notations describe different rate-of-growth relations
between the defining function and the defined set of
functions.
 Theta (Q) -notation
For function g(n), we define Q(g(n)),
big-Theta of n, as the set:

Q(g(n)) =
f(n) :  positive constants c1, c2, and
n0, such that n  n0,
we have 0  c1g(n)  f(n)  c2g(n)

f(n) and g(n) are nonnegative, for large n.

 Q(g(n)) = {f(n) :  positive constants c1, c2, and n0, such
that n  n0, 0  c1g(n)  f(n)  c2g(n)}

Example:
10n2 - 3n = Θ(n2)
What constants for n0, c1, and c2 will work?
Make c1 a little smaller than the leading coefficient, and c2 a
little bigger.
To compare orders of growth, look at the leading term.
Solution: c1n2 ≤ 10n2 −3n ≤ c2n2
c1 ≤ 10−3/n ≤ c2
For n0=1, c1 ≤ 7 and c2 ≥ 10
c1n2 ≤ n2/2−3n ≤ c2n2
Exercise: Prove that n2/2-3n= Θ(n2) c1 ≤ 1/2−3/n ≤ c2
For n0=7,
c1 ≤ 1/14 and c2 ≥ 1/2
 Big-Oh (O) - notation
For a function having only
asymptotic upper bound, Big Oh
„O‟ notation is used. Let a given
function g(n), O(g(n))) is the set
of functions f(n) defined as
O(g(n))={f(n): if there exist
positive constant c and n0 such
that 0≤f(n) ≤cg(n) for all n, n n0}

g(n) is an asymptotic upper bound for f(n).

Show that: 10n2 + 4n + 2 = O(n2 )
Show that: 6*2n + n2 = O(2n )

10n2 + 4n + 2 ≤ C*n2
10n2 + 4n + 2 ≤ 10n2 + n2 ≤ 11n2
For n=1, 16 ≤11 not true
For n=5, 272 ≤ 275 true
So c = 11 ∀n ≥ 5, O(n2 )
6*2n + n2 ≤ 6*2n + 2n
6*2n + n2 ≤ 7*2n
6*2n + n2 ≤ C*2n
For n=1, 13 ≤ 14 not true
For n=2, 28 ≤ 28 true
So C=7 and ∀n ≥ 2, O(2n)
 Big Omega (Ω) - Notation
For a function having only
asymptotic lower bound, Ω
notation is used. Let a given
function g(n), Ω(g(n))) is the set
of functions f(n) defined as
Ω(g(n)) ={f(n): if there exist
positive constant c and n0 such
that 0≤ cg(n) ≤f(n) for all n, n n0}
g(n) is an asymptotic lower bound for f(n).
f(n)=3n2+n
f(n)>=c*g(n) => 3n2+n>=3n2= Ω(n2)

Where n<=n0 and n=1

Definition (Little–o, o()): Let f(n) and g(n) be functions that map
positive integers to positive real numbers. We say that f(n) is o(g(n))
(or f(n) ∈ o(g(n))) if for any real constant c > 0, there exists an integer
constant n0 ≥ 1 such that f(n) < c ∗ g(n) for every integer n ≥ n0.

Note: Little o notation is used to describe an upper bound that cannot

be tight. In other words, loose upper bound of f(n).

f(n) = o(g(n))
If f(n) = n2 and g(n) = n3 then
check whether f(n) = o(g(n))
or not.
Definition (Little–Omega, ω()): Let f(n) and g(n) be functions that
map positive integers to positive real numbers. We say that f(n) is
ω(g(n)) (or f(n) ∈ ω(g(n))) if for any real constant c > 0, there exists an
integer constant n0 ≥ 1 such that f(n)>c*g(n) for every integer n ≥ n0.

Examples
•2≠ω(1)
•4x+2≠ω(x)
•4x+2=ω(1)
•3x2+4x+2≠ω(x2)
•3x2+4x+2=ω(x)
What is Randomized Algorithm?
We call an algorithm randomized if its behaviour is
determined not only by its input but also by values
produced by a random-number generator
An algorithm that uses random numbers to decide
what to do next anywhere in its logic is called
Randomized Algorithm.
For example, in Randomized Quick Sort, we use a
random number to pick the next pivot (or we
randomly shuffle the array). Typically, this
randomness is used to reduce time complexity or
space complexity in other standard algorithms.
the worst case running time is O(n2).
Linear Search Algorithm

Linear-Search (A[ ],x, n)

i=1
while (i <=n) do
if A[i] = x then
return i
i=i+1
if (i>n) then
Print element not found
else
Print element x found at index i
Exit