UNIT I
UNIT I
Input: there are zero or more quantities, which are externally supplied;
Output: at least one quantity is produced
Definiteness: each instruction must be clear and unambiguous;
Finiteness: if we trace out the instructions of an algorithm, then for all cases the
algorithm will terminate after a finite number of steps;
Effectiveness: every instruction must be sufficiently basic that it can in principle be
carried out by a person using only pencil and paper. It is not enough that each
operation be definite, but it must also be feasible.
2. Graphic representation called flowchart: This method will work well when the algorithm
is small& simple.
Pseudo-Code Conventions:
3. An identifier begins with a letter. The data types of variables are not explicitly declared.
1
4. Compound data types can be formed with records. Here is an example,
Node. Record
{
data type – 1 data-1;
.
.
.
data type – n data – n;
node * link;
}
Here link is a pointer to the record type node. Individual data items of a record can
be accessed with and period.
<statement-n>
}
For Loop:
For variable: = value-1 to value-2 step step do
{
<statement-1>
.
.
.
<statement-n>
}
repeat-until:
repeat
<statement-1>
.
.
.
2
<statement-n>
until<condition>
Case statement:
Case
{
: <condition-1> : <statement-1>
.
.
.
: <condition-n> : <statement-n>
: else : <statement-n+1>
}
9. Input and output are done using the instructions read & write.
As an example, the following algorithm fields & returns the maximum of ‘n’ given
numbers:
1. Algorithm Max(A,n)
2. // A is an array of size n
3. {
4. Result := A[1];
5. for I:= 2 to n do
6. if A[I] > Result then
7. Result :=A[I];
8. return Result;
9. }
In this algorithm (named Max), A & n are procedure parameters. Result & I are Local
variables.
Algorithm:
Performance Analysis:
The performance of a program is the amount of computer memory and time needed to
run a program. We use two approaches to determine the performance of a program. One
is analytical, and the other experimental. In performance analysis we use analytical
methods, while in performance measurement we conduct experiments.
Time Complexity:
The time needed by an algorithm expressed as a function of the size of a problem is
called the time complexity of the algorithm. The time complexity of a program is the
amount of computer time it needs to run to completion.
The limiting behavior of the complexity as size increases is called the asymptotic time
complexity. It is the asymptotic complexity of an algorithm, which ultimately determines
the size of problems that can be solved by the algorithm.
1. Algorithm Sum(a,n) 0 - 0
2.{ 0 - 0
3. S=0.0; 1 1 1
4. for I=1 to n do 1 n+1 n+1
5. s=s+a[I]; 1 n n
6. return s; 1 1 1
7. } 0 - 0
4
Space Complexity:
The space complexity of a program is the amount of memory it needs to run to
completion. The space need by a program has the following components:
Instruction space: Instruction space is the space needed to store the compiled
version of the program instructions.
Data space: Data space is the space needed to store all constant and variable
values. Data space has two components:
Space needed by constants and simple variables inprogram.
Space needed by dynamically allocated objects such as arrays and class
instances.
Environment stack space: The environment stack is used to save information
needed to resume execution of partially completed functions.
Instruction Space: The amount of instructions space that is needed depends on
factors such as:
The compiler used to complete the program into machinecode.
The compiler options in effect at the time ofcompilation
The target computer.
The space requirement s(p) of any algorithm p may therefore be written as,
S(P) = c+ Sp(Instance characteristics)
Where ‘c’ is a constant.
Example 2:
Algorithm sum(a,n)
{
s=0.0;
for I=1 to n do
s= s+a[I];
return s;
}
The problem instances for this algorithm are characterized by n,the number of
elements to be summed. The space needed d by ‘n’ is one word, since it is of type
integer.
The space needed by ‘a’a is the space needed by variables of tyepe array of floating
point numbers.
This is atleast ‘n’ words, since ‘a’ must be large enough to hold the ‘n’ elements to be
summed.
So,we obtain Ssum(n)>=(n+s)
[ n for a[],one each for n,I a& s]
Complexity of Algorithms
The complexity of an algorithm M is the function f(n) which gives the running time
and/or storage space requirement of the algorithm in terms of the size ‘n’ of the input
data. Mostly, the storage space required by an algorithm is simply a multiple of the data
size ‘n’. Complexity shall refer to the running time of thealgorithm.
The function f(n), gives the running time of an algorithm, depends not only on the
size ‘n’ of the input data but also on the particular data. The complexity function f(n) for
certain cases are:
1. Best Case : The minimum possible value of f(n) is called the best case.
1. Big–OH (O)
2. Big–OMEGA (Ω),
3. Big–THETA (Θ) and
4. Little–OH (o)
f(n) = O(g(n)), (pronounced order of or big oh), says that the growth rate of f(n) is less
than or equal (<) that of g(n).
f(n) = Ω (g(n)) (pronounced omega), says that the growth rate of f(n) is greater than or
equal to (>) that of g(n).
6
Big–THETA Θ (Same order)
f(n) = Θ (g(n)) (pronounced theta), says that the growth rate of f(n) equals (=) the
growth rate of g(n) [if f(n) = O(g(n)) and T(n) = Θ (g(n)].
little-o notation
Definition: A theoretical measure of the execution of an algorithm, usually the time or memory needed,
given the problem size n, which is usually the number of items. Informally, saying some equation f(n) =
o(g(n)) means f(n) becomes insignificant relative to g(n) as n approaches infinity. The notation is read, "f
of n is little oh of g of n".
Formal Definition: f(n) = o(g(n)) means for all c > 0 there exists some k > 0 such that 0 ≤ f(n) < cg(n) for
all n ≥ k. The value of k must not depend on n, but may depend on c.
O(1), O(log2 n), O(n), O(n. log2 n), O(n2), O(n3), O(2n), n! and nn
Classification of Algorithms
If ‘n’ is the number of data items to be processed or degree of polynomial or the size of
the file to be sorted or searched or the number of nodes in a graph etc.
1 Next instructions of most programs are executed once or at most only a few
times. If all the instructions of a program have this property, we say that its
running time is a constant.
Log n When the running time of a program is logarithmic, the program gets
slightly slower as n grows. This running time commonly occurs in
programs that solve a big problem by transforming it into a smaller
problem, cutting the size by some constant fraction., When n is a million,
log n is a doubled. Whenever n doubles, log n increases by a constant, but
log n does not double until n increases to n2.
n When the running time of a program is linear, it is generally the case that a
small amount of processing is done on each input element. This is the
optimal situation for an algorithm that must process n inputs.
7
o. log n This running time arises for algorithms that solve a problem by
breaking it up into smaller sub-problems, solving then independently, and
then combining the solutions. When n doubles, the running time more
than doubles.
The execution time for six of the typical functions is given below:
n log2 n n*log2n n2 n3 2n
1 0 0 1 1 2
2 1 2 4 8 4
4 2 8 16 64 16
8 3 24 64 512 256
16 4 64 256 4096 65,536
32 5 160 1024 32,768 4,294,967,296
64 6 384 4096 2,62,144 Note 1
128 7 896 16,384 2,097,152 Note 2
256 8 2048 65,536 1,677,216 ????????
General Method:
Divide and conquer is a design strategy which is well known to breaking down
efficiency barriers. When the method applies, it often leads to a large improvement in
time complexity. For example, from O (n 2) to O (n log n) to sort theelements.
Divide and conquer strategy is as follows: divide the problem instance into two or
more smaller instances of the same problem, solve the smaller instances recursively,
and assemble the solutions to form a solution of the original instance. The recursion
stops when an instance is reached which is too small to divide. When dividing the
instance, one can either use whatever division comes most easily to hand or invest
21
time in making the division carefully so that the assembly is simplified.
Divide : Divide the problem into a number of sub problems. The sub problems
are solved recursively.
Conquer : The solution to the original problem is then formed from the solutions
to the sub problems (patching together the answers).
Traditionally, routines in which the text contains at least two recursive calls are called
divide and conquer algorithms, while routines whose text contains only one recursive
call are not. Divide–and–conquer is a very powerful use of recursion.
DANDC (P)
{
if SMALL (P) then return S (p);
else
{
divide p into smaller instances p1, p2, …. Pk, k 1;
apply DANDC to each of these sub problems;
return (COMBINE (DANDC (p1) , DANDC (p2),…., DANDC (pk));
}
}
SMALL (P) is a Boolean valued function which determines whether the input size is
small enough so that the answer can be computed without splitting. If this is so
function ‘S’ is invoked otherwise, the problem ‘p’ into smaller sub problems. These
sub problems p1, p2, . . . , pk are solved by recursive application of DANDC.
22
If the sizes of the two sub problems are approximately equal then the computing
time of DANDC is:
g (n)
n small
T (n) = 2 T(n/2) f (n)
otherwise
Binary Search:
If we have ‘n’ records which have been ordered by keys so that x 1 < x2 < … < xn .
When we are given a element ‘x’, binary search is used to find the corresponding
element from the list. In case ‘x’ is present, we have to determine a value ‘j’ such
that a[j] = x (successful search). If ‘x’ is not in the list then j is to set to zero (un
successful search).
In Binary search we jump into the middle of the file, where we find key a[mid], and
compare ‘x’ with a[mid]. If x = a[mid] then the desired record has been found.
If x < a[mid] then ‘x’ must be in that portion of the file that precedes a[mid], if there
at all. Similarly, if a[mid] > x, then further search is only necessary in that past of
the file which follows a[mid]. If we use recursive procedure of finding the middle key
a[mid] of the un-searched portion of a file, then every un-successful comparison of
‘x’ with a[mid] will eliminate roughly half the un-searched portion from consideration.
Since the array size is roughly halved often each comparison between ‘x’ and
a[mid], and since an array of length ‘n’ can be halved only about log 2n times before
reaching a trivial length, the worst case complexity of Binary search is about log2n
low and high are integer variables such that each time through the loop either ‘x’
is found or low is increased by at least one or high is decreased by at least one. Thus
we have two sequences of integers approaching each other and eventually low will
become greater than high causing termination in a finite number of steps if ‘x’ is not
present.
23
24
Example for Binary Search
Index 1 2 3 4 5 6 7 8 9
Elements -15 -6 0 7 9 23 54 82 101
Number of comparisons = 4
Continuing in this manner the number of element comparisons needed to find each of
nine elements is:
Index 1 2 3 4 5 6 7 8 9
Elements -15 -6 0 7 9 23 54 82 101
Comparisons 3 2 3 4 1 3 2 3 4
There are ten possible ways that an un-successful search may terminate depending
upon the value of x.
25
If x < a[1], a[1] < x < a[2], a[2] < x < a[3], a[5] < x < a[6], a[6] < x < a[7] or
a[7] < x < a[8] the algorithm requires 3 element comparisons to determine that ‘x’
is not present. For all of the remaining possibilities BINSRCH requires 4 element
comparisons. Thus the average number of element comparisons for an unsuccessful
search is:
(3 + 3 + 3 + 4 + 4 + 3 + 3 + 3 + 4 + 4) / 10 = 34/10 = 3.4
The time complexity for a successful search is O(log n) and for an unsuccessful
search is Θ(log n).
Therefore,
T(0) = 0
T(n) = 1 if x = a [mid]
= 1 + T([(n + 1) / 2] – 1) if x < a [mid]
= 1 + T(n – [(n + 1)/2]) if x > a [mid]
2K – 1 - 1 2K – 1 - 1
2K 1
n 1
Algebraically this is 2K 1 1 = 2K – 1
for K > 1
2 2
Giving,
T(0) = 0
T(2k – 1) = 1 if x = a [mid]
K - 1
= 1 + T(2 – 1) if x < a [mid]
= 1 + T(2 k - 1
– 1) if x > a [mid]
In the worst case the test x = a[mid] always fails, so
w(0) = 0
w(2k – 1) = 1 + w(2k - 1
– 1)
This is now solved by repeated substitution:
w(2k – 1) = 1 + w(2k - 1
– 1)
26
= 1 + [1 + w(2k -2
–1)]
= 1 + [1 + [1 + w(2k - 3 –1)]]
= ........
= ........
= i + w(2k - i
– 1)
Although it might seem that the restriction of values of ‘n’ of the form 2 K–1 weakens
the result. In practice this does not matter very much, w(n) is a monotonic
increasing function of ‘n’, and hence the formula given is a good approximation even
when ‘n’ is not of the form 2K–1.
Merge Sort:
Merge sort algorithm is a classic example of divide and conquer. To sort an array,
recursively, sort its left and right halves separately and then merge them. The time
complexity of merge mort in the best case, worst case and average case is O(n log n)
and the number of comparisons used is nearly optimal.
This strategy is so simple, and so efficient but the problem here is that there seems
to be no easy way to merge two adjacent sorted arrays together in place (The result
must be build up in a separate array).
The fundamental operation in this algorithm is merging two sorted lists. Because the
lists are sorted, this can be done in one pass through the input, if the output is put in
a third list.
Algorithm
27
Algorithm MERGE (low, mid, high)
// a (low : high) is a global array containing two sorted subsets
// in a (low : mid) and in a (mid + 1 : high).
// The objective is to merge these sorted sets into single sorted
// set residing in a (low : high). An auxiliary array B is used.
{
h :=low; i := low; j:= mid + 1;
while ((h < mid) and (J < high)) do
{
if (a[h] < a[j]) then
{
b[i] := a[h]; h := h + 1;
}
else
{
b[i] :=a[j]; j := j + 1;
}
i := i + 1;
}
if (h > mid) then
for k := j to high do
{
b[i] := a[k]; i := i + 1;
}
else
for k := h to mid do
{
b[i] := a[K]; i := i + l;
}
for k := low to high do
a[k] := b[k];
}
Example
28
Tree Calls of MERGESORT(1, 8)
The following figure represents the sequence of recursive calls that are produced by
MERGESORT when it is applied to 8 elements. The values in each node are the values
of the parameters low and high.
1, 8
1, 4 5, 8
1, 2 3, 4 5, 6 7, 8
1, 1 2, 2 3, 3 4, 4 5, 5 6, 6 7, 7 8, 8
1, 1, 2 3, 3, 4 5, 5, 6 7, 7, 8
1, 2, 4 5, 6, 8
1, 4, 8
We will assume that ‘n’ is a power of 2, so that we always split into even halves, so
we solve for the case n = 2k.
T(1) = 1
T(n) = 2 T(n/2) + n
This is a standard recurrence relation, which can be solved several ways. We will
solve by substituting recurrence relation continually on the right–hand side.
29
Since we can substitute n/2 into this main equation
T(n/4) = 2 T(n/8) + n
T(n) = 8 T(n/8) + 3n
T(n) = 2k T(n/2k) + K. n
2
= n T(1) + n log n
= n log n + n
We have assumed that n = 2k. The analysis can be refined to handle cases when ‘n’
is not a power of 2. The answer turns out to be almost identical.
Although merge sort’s running time is O(n log n), it is hardly ever used for main
memory sorts. The main problem is that merging two sorted lists requires linear
extra memory and the additional work spent copying to the temporary array and
back, throughout the algorithm, has the effect of slowing down the sort considerably.
The Best and worst case time complexity of Merge sort is O(n log n).
The matrix multiplication of algorithm due to Strassens is the most dramatic example
of divide and conquer technique (1969).
The usual way to multiply two n x n matrices A and B, yielding result matrix ‘C’ as
follows :
for i := 1 to n do
for j :=1 to n do
c[i, j] := 0;
for K: = 1 to n do
c[i, j] := c[i, j] + a[i, k] * b[k, j];
30
This algorithm requires n3 scalar multiplication’s (i.e. multiplication ofsingle
numbers) and n3 scalar additions. So we naturally cannot improve upon.
We apply divide and conquer to this problem. For example let us considers three
multiplication like this:
A 11 A 12 B 11 B 12 C 12
C 11
A
A B B C C
21 22 21 22 21 22
Strassens insight was to find an alternative method for calculating the C ij, requiring
seven (n/2) x (n/2) matrix multiplications and eighteen (n/2) x (n/2) matrix
additions and subtractions:
C11 = P + S – T + V
C12 = R + T
C21 = Q + S
C22 = P + R - Q + U.
This method is used recursively to perform the seven (n/2) x (n/2) matrix
multiplications, then the recurrence equation for the number of scalar multiplications
performed is:
31
T(1) = 1
T(n) = 7 T(n/2)
T(2k) = 7 T(2k–1)
= 72 T(2k-2)
= ------
= ------
= 7i T(2k–i)
Put i = k
= 7k T(1)
= 7k
= n log 7
2
log 7
= O(n 2 ) = O(2n.81)
So, concluding that Strassen’s algorithm is asymptotically more efficient than the
standard algorithm. In practice, the overhead of managing the many small matrices
does not pay off until ‘n’ revolves the hundreds.
Quick Sort
The main reason for the slowness of Algorithms like SIS is that all comparisons and
exchanges between keys in a sequence w1, w2, . . . . , wn take place between
adjacent pairs. In this way it takes a relatively long time for a key that is badly out of
place to work its way into its proper position in the sorted sequence.
Hoare his devised a very efficient way of implementing this idea in the early 1960’s
that improves the O(n2) behavior of SIS algorithm with an expected performance that
is O(n log n).
In essence, the quick sort algorithm partitions the original array by rearranging it
into two groups. The first group contains those elements less than some arbitrary
chosen value taken from the set, and the second group contains those elements
greater than or equal to the chosen value.
The chosen value is known as the pivot element. Once the array has been rearranged
in this way with respect to the pivot, the very same partitioning is recursively applied
to each of the two subsets. When all the subsets have been partitioned and
rearranged, the original array is sorted.
The function partition() makes use of two pointers ‘i’ and ‘j’ which are moved toward
each other in the following fashion:
32
If j > i, interchange a[j] with a[i]
Repeat the steps 1, 2 and 3 till the ‘i’ pointer crosses the ‘j’ pointer. If ‘i’
pointer crosses ‘j’ pointer, the position for pivot is found and place pivot
element in ‘j’ pointer position.
The program uses a recursive function quicksort(). The algorithm of quick sort
function sorts all elements in an array ‘a’ between positions ‘low’ and ‘high’.
It terminates when the condition low >= high is satisfied. This condition
will be satisfied only when the array is completely sorted.
Here we choose the first element as the ‘pivot’. So, pivot = x[low]. Now it
calls the partition function to find the proper position j of the element
x[low] i.e. pivot. Then we will have two sub-arrays x[low], x[low+1], . . . .
. . . x[j-1] and x[j+1], x[j+2], x[high].
33
Example
Select first element as the pivot element. Move ‘i’ pointer from left to right in search
of an element larger than pivot. Move the ‘j’ pointer from right to left in search of an
element smaller than pivot. If such elements are found, the elements are swapped.
This process continues till the ‘i’ pointer crosses the ‘j’ pointer. If ‘i’ pointer crosses ‘j’
pointer, the position for pivot is found and interchange pivot and element at ‘j’
position.
Let us consider the following example with 13 elements to analyze quick sort:
1 2 3 4 5 6 7 8 9 10 11 12 13 Remarks
38 08 16 06 79 57 24 56 02 58 04 70 45
pivot i j swap i & j
04 79
i j swap i & j
34
02 57
j i
swap pivot
(24 08 16 06 04 02) 38 (56 57 58 79 70 45)
&j
swap pivot
pivot j, i
&j
(02 08 16 06 04) 24
pivot, swap pivot
i
j &j
02 (08 16 06 04)
pivot i j swap i & j
04 16
j i
swap pivot
(06 04) 08 (16)
&j
pivot,
i
j
swap pivot
(04) 06
&j
04
pivot,
j, i
16
pivot,
j, i
(02 04 06 08 16 24) 38
(56 57 58 79 70 45)
pivot i j swap i & j
45 57
j i
swap pivot
(45) 56 (58 79 70 57)
&j
45
pivot, swap pivot
j, i &j
(58 79 57)
pivot i 70 j swap i & j
57 79
j i
swap pivot
(57) 58 (70 79)
&j
57
pivot,
j, i
(70 79)
pivot, swap pivot
i
j &j
70
79
pivot,
j, i
(45 56 57 58 70 79)
02 04 06 08 16 24 38 45 56 57 58 70 79
35
Analysis of Quick Sort:
Like merge sort, quick sort is recursive, and hence its analysis requires solving a
recurrence formula. We will do the analysis for a quick sort, assuming a random pivot
(and no cut off for small files).
The running time of quick sort is equal to the running time of the two recursive calls
plus the linear time spent in the partition (The pivot selection takes only constant
time). This gives the basic quick sort relation:
The pivot is the smallest element, all the time. Then i=0 and if we ignore T(0)=1,
which is insignificant, the recurrence is:
T (n – 2) = T (n – 3) + C (n – 2)
------- -
n
T (n) T (1)
i2
i
= O (n2) - (3)
36
Best and Average Case Analysis
T(n) = (n+1)2[ 2 k n 1
1/ k
=2(n+1) [ ]
=2(n+1)[log (n+1) – log 2]
=2n log (n+1) + log (n+1)-2n log 2 –log 2
T(n)= O(n log n)