Algorithm Analysis and Design Lecture Notes

----------------------------------------------------------------------------------------------------------------------------------------------------------
PART ONE
BASICS OF ALGORITHMS ANALYSIS
1. INTRODUCTION TO ALGORITHM ANALYSIS
Introduction
An algorithm is a set of steps of operations to solve a problem performing calculation, data processing, and
automated reasoning tasks. An algorithm is an efficient method that can be expressed within finite amount of
time and space.
An algorithm is the best way to represent the solution of a particular problem in a very simple and efficient
way. If we have an algorithm for a specific problem, then we can implement it in any programming language,
meaning that the algorithm is independent from any programming languages.

Algorithm Design
The important aspects of algorithm design include creating an efficient algorithm to solve a problem in an
efficient way using minimum time and space.
To solve a problem, different approaches can be followed. Some of them can be efficient with respect to time
consumption, whereas other approaches may be memory efficient.
However, one has to keep in mind that both time consumption and memory usage cannot be optimized
simultaneously. If we require an algorithm to run in lesser time, we have to invest in more memory and if we
require an algorithm to run with lesser memory, we need to have more time.

Problem Development Steps

The following steps are involved in solving computational problems.
Problem definition
Development of a model
Specification of an Algorithm
Designing an Algorithm
Checking the correctness of an Algorithm
Analysis of an Algorithm
Implementation of an Algorithm
Program testing
Documentation

Characteristics of Algorithms
The main characteristics of algorithms are as follows:
 Algorithms must have a unique name
 Algorithms should have explicitly defined set of inputs and outputs
 Algorithms are well-ordered with unambiguous operations
 Algorithms halt in a finite amount of time. Algorithms should not run for infinity,
i.e., an algorithm must end at some point

An algorithm is the best way to represent the solution of a particular problem in a very simple and efficient
way. If we have an algorithm for a specific problem, then we can implement it in any programming language,
meaning that the algorithm is independent from any programming languages.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 1 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
The study of algorithms includes many important and active areas of research. There are perhaps five distinct
areas of study one can identify:
(i} How to devise algorithms-The act of creating an algorithm is an art which may never be fully automated.
A major goal of this is to study various design techniques which have proven to be useful in that they have
often yielded good algorithms. By mastering these design strategies, it will become easier for you to devise
new and useful algorithms.
(ii) How to express algorithms-The structured programming "movement" has as its central concern the clear
and concise expression of algorithms in a programming language.
(iii) How to validate algorithms-Once an algorithm is devised it is necessary to show that it computes the
correct answer for all possible legal inputs. We refer to this process as algorithm validation. The algorithm
need not as yet be expressed as a program. It is sufficient to state it in any precise way. The purpose of the
validation is to assure us that this algorithm will work correctly independent of the issues concerning the
programming language it will eventually be written in.
(iv) How to analyze algorithms-This field of study is called analysis of algorithms. As an algorithm is
executed, it makes use of the computer's central processing unit ( cpu) to perform operations and it uses the
memory (both immediate and auxiliary) to hold the program and its data. Analysis of algorithms refers to the
process of determining how much computing time and storage an algorithm will require.
(v) How to test a program-Testing a program really consists of two phases: debugging and profiling.
Debugging is the process of executing programs on sample data sets to determine if faulty results occur and, if
so, to correct them. However, as E. Dijkstra has pointed out, "debugging can only point to the presence of
errors, but not to their absence.

What kinds of problems are solved by algorithms?

Sorting is by no means the only computational problem for which algorithms have been developed.
Practical applications of algorithms are ubiquitous and include the following examples:
 The Human Genome Project has the goals of identifying all the 100,000 genes in human DNA,
determining the sequences of the 3 billion chemical base pairs that make up human DNA, storing this
information in databases, and developing tools for data analysis.
 The Internet enables people all around the world to quickly access and retrieve large amounts of
information. In order to do so, clever algorithms are employed to manage and manipulate this large
volume of data. Examples of problems which must be solved include finding good routes on which the
data will travel
 Electronic commerce enables goods and services to be negotiated and exchanged electronically. The
ability to keep information such as credit card numbers, passwords, and bank statements private is
essential if electronic commerce is to be used widely. Public-key cryptography and digital signatures •
 In manufacturing and other commercial settings, it is often important to allocate scarce resources in the
most beneficial way. An oil company may wish to know where to place its wells in order to maximize
its expected profit. A candidate for the presidency of the United States may want to determine where to
spend money buying campaign advertising in order to maximize the chances of winning an election.

Complexity Analysis
In order to solve a problem there are many possible algorithms that differ in efficiency. One has to be able to
choose the best algorithm for the problem at hand using some scientific method. In order to say an algorithm is
good, we must analyze its resource requirements such as running time, memory usage, and communication
band width. Algorithm analysis refers to the process of determining the amount of computing time
andstorage space required by different algorithms. It is a study of computer program performance and
resource usage.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 2 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
There are two approaches to measure the efficiency of algorithms (run time of an algorithm)
1. Empirical: - Programmers computing algorithms and trying them on different instances. Use the method
like clock() or system.currentTime.millis() to get an accurate measure of the running time. However, it is
difficult to use actual clock- time (execution time) as a consistent measure of algorithm efficiency, because
clock- time can vary based on many things such as:

 Based on system used

 Processor speed
 Current CPU load
 Input size
 Operating Environment
 Language in which a given algorithm is written
Therefore real time units should not be used to evaluate algorithm efficiency. Rather logical units that express
a relationship between the size n of a file and amount of time required to process the data should be used (i.e.
the number of operation required). This show how an algorithm‟s efficiency changes according to the size of
the input.

2. Theoretical: Determine the quantity of resources required mathematically ( execution time, memory
needed) by each algorithm, independent from hardware and soft ware environment It uses a high level
description of the algorithm instead of testing one of its implementation. Takes an algorithm and produce, a
function T(n) which depends on the number of operations, which are expressed in time units. That is running
time is expressed as T(n) for some function T on input size n.

Complexity analysis analyzes the cost of solving interesting problems by measuring amount of resources
needed such as execution time and space required to store data. The complexity of an algorithm is a function
describing the efficiency of an algorithm in terms of the amount of data the algorithm must process. There are
two main complexity measure of the efficiency of an algorithm.
a) Time complexity is a function describing the amount of time an algorithm takes in-terms of the amount
of input to the algorithm. Time can mean the number of memory accesses performed, the number of
comparisons between integers, the number of times some inner loop is executed, or some other natural
unit related to the amount of real time the algorithm will take. It is most important one it does not use
real time.
b) Space complexity is a function describing the amount of memory (space) an algorithm takes in termsof
the amount of input to the algorithm.
The only factor that affects measure of complexity is the input size of the algorithm. To avoid this, measure
complexity of an algorithm for arbitrary number n as n tends to infinity ( n→ ∞ ) that means for both
complexity of an algorithm we are interested in the asymptotic complexity of an algorithm, when n( the
number of items or input) goes to infinity, what happens to the performance of the algorithm?
For large enough inputs, the multiplicative constants and lower order terms of an exact running time are
dominated by the effects of the input size itself. When quantifying the performance of the algorithm in terms of
n T(n), we are mainly interested in how the time and space requirements change as n grows large. When we try
to quantify the performance, there are three kinds of analysis of program efficiency

The Need for Analysis

In this chapter, we will discuss the need for analysis of algorithms and how to choose a better algorithm for a
particular problem as one computational problem can be solved by different algorithms.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 3 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
By considering an algorithm for a specific problem, we can begin to develop pattern recognition so that similar
types of problems can be solved by the help of this algorithm.
Algorithms are often quite different from one another, though the objective of these algorithms is the same. For
example, we know that a set of numbers can be sorted using different algorithms. Number of comparisons
performed by one algorithm may vary with others for the same input. Hence, time complexity of those
algorithms may differ. At the same time, we need to calculate the memory space required by each algorithm.
Analysis of algorithm is the process of analyzing the problem-solving capability of the algorithm in terms of
the time and size required (the size of memory for storage while implementation). However, the main concern
of analysis of algorithms is the required time or performance. Generally, we perform the following types of
analysis:

 Worst-case: The maximum number of steps taken on any instance of size a.

 Best-case: The minimum number of steps taken on any instance of size a.
 Average case: An average number of steps taken on any instance of size a.
 Amortized: A sequence of operations applied to the input of size a averaged over time.

Methodology of Analysis
To measure resource consumption of an algorithm, different strategies
A) Asymptotic Analysis
The asymptotic behavior of a function (𝒏) refers to the growth of (𝒏) as n gets large.
We typically ignore small values of n, since we are usually interested in estimating how slow the program will
be on large inputs.
A good rule of thumb is that the slower the asymptotic growth rate, the better the algorithm. Though it‟s not
always true.
For example, a linear algorithm (𝒏) = 𝒅 ∗ 𝒏 + 𝒌 is always asymptotically better than a quadratic one, (𝒏) = 𝒄.
𝒏𝟐 + 𝒒.
B) Solving Recurrence Equations
A recurrence is an equation or inequality that describes a function in terms of its value on smaller inputs.
Recurrences are generally used in divide-and-conquer paradigm.
Let us consider (𝒏) to be the running time on a problem of size n.
If the problem size is small enough, say 𝒏 < 𝒄 where c is a constant, the straightforward solution takes constant
time, which is written as Ɵ(𝟏). If the division of the problem yields a number of sub-problems with size 𝒏/b
To solve the problem, the required time is 𝒂. 𝑻(𝒏/𝒃). If we consider the time required for division is (𝒏) and
the time required for combining the results of sub-problems is (𝒏), the recurrence relation can be represented
as:

A recurrence relation can be solved using the following methods:

 Substitution Method ─ In this method, we guess a bound and using mathematical induction we prove
that our assumption was correct.
 Recursion Tree Method ─ In this method, a recurrence tree is formed where each node represents the
cost.
 Master’s Theorem ─ This is another important technique to find the complexity of a recurrence
relation.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 4 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

Order of Magnitude Analysis:

This phase analyze the function T(n) to determine the general complexity category to which it belongs. Refers
to the rate at which the storage or time grows as a function of problem size. At this phase we take timing
function T(n) as input and determine order of T(n) written as O( T(n)) as out put. The out put helps to
determine the category to which these complexities are known through research and experiment.

Functions whose complexity is known through research and experiment are listed as follow:

Constant function –T(n) εO(1) :Great. This means your algorithm takes only constant time.
 loglogn–T(n) εO(loglogn):super fast! For all intents this is as fast as a constant time
 Logarithmic time –T(n) εO(logn):Very good.
 Poly logarithmictime–T(n) εO((logn)k):(where k is a constant ). Not bad, when simple logarithmic
is not achievable
 Linear time: –T(n) εO(n) : It is about the best that one can hope for if your algorithm has to look at all
the data. –In data structure the game is usually to avoid this through
 nlogn–T(n) εO(nlogn) : This one is famous, because this is the time needed to sort a list of numbers.
 Quadratic time: –T(n) εO(n2) : is not okay if n increasing
 Polynomial time: –T(n) εO(nk) : (where k is a constant). Practical if k is not too large
 Exponential time: –T(n) εO(2n) , T(n) εO(nn) , T(n) εO(n!) : algorithms taking this much time are
only practical for smallest values of n.
Those function ordered according to their order of growth T(n) has higher order of growth than g(n) which
means that for any positive constant c T(n)>c g(n) for large n. Two functions that differ only by a constant
factor have the same order of growth.

Rules for computing complexity

To analyze an algorithm written in pseudo-code in order to determine its complexity, determine the number of
operations on the underlying model of computation. To obtain an estimate of this numbers, we count primitive
operations, chosen to reflect the complexity of the total cost of all operations. Primitive operation refers to
basic operations performed by an algorithm identifiable in pseudo code. Examples of basic operations
o an assignment
o calling a method
o Returning from a method
o Evaluating an expression- An arithmetic operation between two variable
o Comparison between two variables
o Indexing into an array
We measure execution (running) time in terms of any of the above operation.

We assume an arbitrary time unit

1. Execution of one of the following operations takes time 1 (counts as 0(1))1
o Assignment statement
o Single read/write statement (I/0)
o Single Boolean expression Evaluation
o Single arithmetic operation
o Function return
2. Running time of a selection statement (if, switch) is the time for the condition evaluation plus the
maximum of the running times for the individual clauses in the selection.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 5 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

Running time of if- then- else statement is, worst case running time, the test, plus either the then part or the else
part (whichever is the larger).

3. Loops: Running time for a loop is equal to the running time for the statements inside the loop multiplied by
number of iteration. The total running time of a statement inside a group of nested loops is the running time of
the statement multiplied by the product of the sizes of all the loops. For nested loops, analyze inside to out.
Always assume that the loop executes their maximum number of iterations possible
4. Running time of a function call is 1 for setup plus the time for any parameter calculations plus the time
required for the execution of the function body.

5. Sequences of statements

Use order arithmetic addition rule( O( f(n)+g(n)) = max ( f(n), g(n)) ) and add the time complexities of each
statement

Limitation theoretical analysis: - it assume all operations take exactly the same time unit but not in real
world.

Example 1
I=1
P=1
while (I < N) do
{
P=P*I
I=I+1
}
T(n) = 2 + (N - I) * (2 + 2)+ N
= 2 + (N - 1) * (4) + N
= O(2 + (5N - 4))
= O(N)

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 6 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
Hint
2 for assignment
N testing condition in the while loop
(N-I) assignment & multiplication in the loops
(N-I) assignment & multiplication in loop

Example 2
int count ( )
{
int k=0 - 1 assignment
cout<< “ Enter an integer”; - 1 out put
Cin>>n; - 1 for input
For (i=0;i<n; i++)In the for loop;
K++; - 1 assignment
Return 0; - n increments
}
- n + 1 test
- n increment inside the loop
- n for return statement
T(n)= 1+1+1+(1+n+1+n)+n+n= 4n+5=O(n)

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 7 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
Example 4

Additional Examples

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 8 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 9 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

Asymptotic Analysis
Asymptotic analysis is concerned with how the running time of an algorithm increases with the size of the
input in the limit, as the size of input increases without bound, (estimating the rate of function growth). There
are five notations used for specifying asymptotic complexity (estimating a rate of running time
functiongrowth)
o Big –oh notation ( O)
o Big – Omega Notation (𝛀)
o Theta Notation (Ɵ)
o Little 0 Notation (o)
o ω − Little omega
O: Asymptotic Upper Bound

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 10 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
„O‟ (Big Oh) is the most commonly used notation. A function (𝐧) can be represented is the order of (𝒏) that is
𝑶(𝒈(𝒏)), if there exists a value of positive integer n as n0 and a positive constant c such that:
𝒇(𝒏) ≤ 𝒄. (𝒏) for 𝒏 > 𝒏𝟎 in all case.
Hence, function (𝒏) is an upper bound for function (𝒏), as 𝒈(𝒏) grows faster than 𝒇(𝒏).

Example
Let us consider a given function, (𝒏) = 𝟒. 𝒏𝟑 + 𝟏𝟎. 𝒏𝟐 + 𝟓. 𝒏 + 𝟏.
Considering (𝒏) = 𝒏𝟑,
𝒇(𝒏) ≤ 𝟓. (𝒏) for all the values of 𝒏 > 𝟐.
Hence, the complexity of (𝒏) can be represented as (𝒈(𝒏)), i.e. 𝑶(𝒏𝟑).

Ω: Asymptotic Lower Bound

We say that (𝒏) = (𝐠(𝒏)) when there exists constant c that 𝒇(𝒏) ≥ 𝒄. (𝒏) for all sufficiently large value of n.
Here n is a positive integer. It means function g is a lower
bound for function f; after a certain value of n, f will never go below g.
Example
Let us consider a given function, (𝒏) = 𝟒. 𝒏𝟑 + 𝟏𝟎. 𝒏𝟐 + 𝟓. 𝒏 + 𝟏.
Considering (𝒏) = 𝒏𝟑, (𝒏) ≥ 𝟒. (𝒏) for all the values of 𝒏 > 𝟎.
Hence, the complexity of (𝒏) can be represented as (𝒈(𝒏)), i.e. 𝛀(𝒏𝟑)
Ɵ: Asymptotic Tight Bound
We say that (𝑛) = Ɵ(g(𝑛)) when there exist constants c1 and c2 that 𝑐1. (𝑛) ≤ (𝑛) ≤ 𝑐2. (𝑛) for all sufficiently
large value of n. Here n is a positive integer.
This means function g is a tight bound for function f.
Example
Let us consider a given function, (𝒏) = 𝟒. 𝒏𝟑 + 𝟏𝟎. 𝒏𝟐 + 𝟓. 𝒏 + 𝟏.
Considering (𝒏) = 𝒏3, 𝟒. 𝒈(𝒏) ≤ 𝒇(𝒏) ≤ 𝟓. (𝒏) for all the large values of n.
Hence, the complexity of (𝒏) can be represented as Ɵ(𝐠(𝒏)), i.e. Ɵ(𝒏3).
O-Notation
The asymptotic upper bound provided by O-notation may or may not be asymptotically tight. The bound 𝟐.
𝒏𝟐 = (𝒏2) is asymptotically tight, but the bound 𝟐. 𝒏 = (𝒏2) is not.
We use o-notation to denote an upper bound that is not asymptotically tight.
We formally define ((𝒏)) (little-oh of g of n) as the set 𝒇(𝒏) = 𝒐(𝒈(𝒏)) for any positive constant 𝒄 > 𝟎 and there
exists a value 𝒏𝟎 > 𝟎, such that 𝟎 ≤ 𝒇(𝒏) ≤ 𝒄. 𝒈(𝒏).
Intuitively, in the o-notation, the function (𝒏) becomes insignificant relative to (𝒏) as n approaches infinity.

ω-Notation
We use ω-notation to denote a lower bound that is not asymptotically tight. Formally, however, we define
⍵(𝒈(𝒏)) (little-omega of g of n) as the set 𝒇(𝒏) = ⍵(𝒈(𝒏)) for any positive constant 𝒄 > 𝟎 and there exists a
value 𝒏𝟎 > 𝟎, such that 𝟎 ≤ 𝒄. 𝒈(𝒏) < 𝒇(𝒏).

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 11 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

Asymptotic notation

Comparison of functions
Many of the relational properties of real numbers apply to asymptotic comparisons as well. For the following,
assume that f (n) and g(n) are asymptotically positive.

Because these properties hold for asymptotic notations, one can draw an analogy between the asymptotic
comparison of two functions f and g and the comparison of two real numbers a and b:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 12 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

We say that f (n) is asymptotically smaller than g(n) if f (n) = o(g(n)), and f (n) is asymptotically larger than
g(n) if f (n) = ω(g(n)).

2. ELEMENTARY DATA STRUCTURES AND ALGORITHMS

In this section, we examine the representation of dynamic sets by simple data structures that use pointers.
Although many complex data structures can be fashioned using pointers, we present only the redimentary ones:
stacks, queues, linked lists, and rooted trees. We also discuss a method by which objects and pointers can be
synthesized from arrays.

Now that we have presented the fundamental methods we need to express and analyze algorithms you might
feel all set to begin. But alas we need to make one last diversion to which we devote this chapter, and that is a
discussion of data structures. One of the basic techniques for improving algorithms is to structure the data in
such a way that the resulting operations can be efficiently carried out. Though we can't possibly survey here all
of the techniques that are known, in this chapter we have selected several which we feel occur most frequently.
Maybe you have already seen these techniques in a course on data structures (hopefully having used
Fundamentals of data structures). If so, you may either skip this chapter or scan it briefly. If you haven't been
exposed to the ideas of stack, queues, sets, trees, graphs, heaps, or hashing then lets begin our study of
algorithms right now with some interesting problems from the field of data structures.

One of the most common forms of data organization in computer programs is the ordered or linear list, which
is often written as A = (a i. a 2, ••• an). The a;s are referred to as atoms and they are chosen from some set. The
null or empty list has n = 0 elements. A stack is an ordered list in which all insertions and deletions are made at
one end, called the top. A queue is an ordered list in which all insertions take place at one end, the rear, while
all deletions take place at the other end, the.front.

Stacks and queues

Stacks and queues are dynamic sets in which the element removed from the set by the
DELETE operation is prespecified. In a stack, the element deleted from the set is the one most recently
inserted: the stack implements a last-in, first-out, or LIFO, policy. Similarly, in a queue, the element deleted
is always the one that has been in the set for the longest time: the queue implements a first-in, first out, or
FIFO, policy. There are several efficient ways to implement stacks and queues on a computer. In this section
we show how to use a simple array to implement each.

Stacks:
The INSERT operation on a stack is often called PUSH, and the DELETE operation, which does not take an
element argument, is often called POP. These names are allusions to physical stacks, such as the spring-loaded
stacks of plates used in cafeterias. The order in which plates are popped from the stack is the reverse of the
order in which they were pushed onto the stack, since only the top plate is accessible.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 13 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
The stack operations can each be implemented with a few lines of code.
STACK-EMPTY(S)
1 if top[S] = 0
2 then return TRUE
3 else return FALSE
PUSH(S, x)
1 top[S] ← top[S] + 1
2 S[top[S]] ← x
POP(S)
1 if STACK-EMPTY(S)
2 then error "underflow"
3 else top[S] ← top[S] - 1
4 return S[top[S] + 1]

The code shows the effects of the modifying operations PUSH and POP. Each of the three stack operations
takes O(1) time.

Application of Stack
• Tracking function calls
• Dealing with undo/redo operations
• Reverse-Polish calculators
• Assembly language

Queues:
We call the INSERT operation on a queue ENQUEUE, and we call the DELETE operation
DEQUEUE; like the stack operation POP, DEQUEUE takes no element argument. The FIFO property of a
queue causes it to operate like a line of people in the registrar's office. The queue has a head and a tail. When
an element is enqueued, it takes its place at the tail of the queue, just as a newly arriving student takes a place
at the end of the line. The element dequeued is always the one at the head of the queue, like the student at the
head of the line who has waited the longest. (Fortunately, we don't have to worry about computational
elements cutting into line.)

ENQUEUE(Q, x)
1 Q[tail[Q]] ← x
2 if tail[Q] = length[Q]
3 then tail[Q] ← 1
4 else tail[Q] ← tail[Q] + 1
DEQUEUE(Q)
1 x ← Q[head[Q]]
2 if head[Q] = length[Q]
3 then head[Q] ← 1
4 else head[Q] ← head[Q] + 1
5 return x
The code shows the effects of the ENQUEUE and DEQUEUE operations. Each operation takes O(1) time.

Applications Queue
The most common application is in client-server models
• Multiple clients may be requesting services from one or more servers
• Some clients may have to wait while the servers are busy
• Those clients are placed in a queue and serviced in the order of arrival
Grocery stores, banks, and airport security use queues
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 14 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
The SSH Secure Shell and SFTP are clients
Most shared computer services are servers:
• Web, file, ftp, database, mail, printers, WOW, etc.

Linked lists:
A linked list is a data structure in which the objects are arranged in a linear order. Unlike an array, though, in
which the linear order is determined by the array indices, the order in a linked list is determined by a pointer in
each object. Linked lists provide a simple, flexible representation for dynamic sets, supporting (though not
necessarily efficiently).

Searching a linked list

The procedure LIST-SEARCH(L, k) finds the first element with key k in list L by a simple linear search,
returning a pointer to this element. If no object with key k appears in the list, then NIL is returned. For the
linked list the call LIST-SEARCH(L, 4) returns a pointer to the third element, and the call LIST-SEARCH(L,
7) returns NIL.
LIST-SEARCH(L, k)
1 x ← head[L]
2 while x ≠ NIL and key[x] ≠ k
3 do x ← next[x]
4 return x
To search a list of n objects, the LIST-SEARCH procedure takes Θ(n) time in the worst case, since it may have
to search the entire list.

Inserting into a linked list

Given an element x whose key field has already been set, the LIST-INSERT procedure "splices" x onto the
front of the linked list, as shown in code.
LIST-INSERT(L, x)
1 next[x] ← head[L]
2 if head[L] ≠ NIL
3 then prev[head[L]] ← x
4 head[L] ← x
5 prev[x] ← NIL
The running time for LIST-INSERT on a list of n elements is O(1).

Deleting from a linked list

The procedure LIST-DELETE removes an element x from a linked list L. It must be given a pointer to x, and it
then "splices" x out of the list by updating pointers. If we wish to delete an element with a given key, we must
first call LIST-SEARCH to retrieve a pointer to the element.
LIST-DELETE(L, x)
1 if prev[x] ≠ NIL
2 then next[prev[x]] ← next[x]
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 15 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
3 else head[L] ← next[x]
4 if next[x] ≠ NIL
5 then prev[next[x]] ← prev[x]

Code above shows how an element is deleted from a linked list. LIST-DELETE runs in O(1) time, but if we
wish to delete an element with a given key, Θ(n) time is required in the worst case because we must first call
LIST-SEARCH.

BASIC SORTING ALGORITHMS

Why sorting?
Many computer scientists consider sorting to be the most fundamental problem in the study of algorithms.
There are several reasons:
o Sometimes the need to sort information is inherent in an application. For example, in order to prepare
customer statements, banks need to sort checks by check number.
o Algorithms often use sorting as a key subroutine. For example, a program that renders graphical
objects that are layered on top of each other might have to sort the objects according to an "above"
relation so that it can draw these objects from bottom to top.
o There is a wide variety of sorting algorithms, and they use a rich set of techniques. In fact, many
important techniques used throughout algorithm design are represented in the body of sorting
algorithms that have been developed over the years.
a)Bubble Sort
Bubble Sort is an elementary sorting algorithm, which works by repeatedly exchanging adjacent elements, if
necessary. When no exchanges are required, the file is sorted. This is the simplest technique among all sorting
algorithms.
Algorithm: Sequential-Bubble-Sort (A)
fori← 1 to length [A] do
for j ← length [A] down-to i +1 do
if A[A] < A[j-1] then
Exchange A[j] ↔ A[j-1]
Implementation
Void bubbleSort(int numbers[], intarray_size)
{
inti, j, temp;
for (i = (array_size - 1); i>= 0; i--)
for (j = 1; j <= i; j++)
if (numbers[j-1] > numbers[j])
{
temp = numbers[j-1];
numbers[j-1] = numbers[j];
numbers[j] = temp;}}

Analysis
Here, the number of comparisons is

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 16 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
 + +  + . . . + ( − ) = (− )/ = ()
Clearly, the graph shows the n2 nature of the bubble sort.
In this algorithm, the number of comparison is irrespective of the data set, i.e. whether the provided input
elements are in sorted order or in reverse order or at random.

There is no change in 3rd, 4th, 5th and 6th iteration.

b) Insertion sort

Insertion sort is a very simple method to sort numbers in an ascending or descending order. This method
follows the incremental method. It can be compared with the technique how cards are sorted at the time of
playing a game.
The numbers, which are needed to be sorted, are known as keys. Here is the algorithm of the insertion sort
method.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 17 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
Algorithm: Insertion-Sort(A)
for j = 2 to A.length
key = A[j]
i=j–1
while i > 0 and A[i] > key
A[i + 1] = A[i]
i = i -1
A[i + 1] = key

Analysis
Run time of this algorithm is very much dependent on the given input.
If the given numbers are sorted, this algorithm runs in () time. If the given numbers are in reverse order, the
algorithm runs in () time.
Example

c) Selection Sort
This type of sorting is called Selection Sort as it works by repeatedly sorting elements.
It works as follows: first find the smallest in the array and exchange it with the element in the first position,
then find the second smallest element and exchange it with the element in the second position, and continue in
this way until the entire array is sorted.

Algorithm: Selection-Sort (A)

fori← 1 to n-1 do
min j ←i;
min x ← A[i]
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 18 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
for j ←i + 1 to n do
if A[j] < min x then
min j ← j
min x ← A[j]
A[min j] ← A [i]
A[i] ← min x

Selection sort is among the simplest of sorting techniques and it works very well for small files. It has a quite
important application as each item is actually moved at the most once. Section sort is a method of choice for
sorting files with very large objects (records) and small keys. The worst case occurs if the array is already
sorted in a descending order and we want to sort them in an ascending order.
Nonetheless, the time required by selection sort algorithm is not very sensitive to the original order of the array
to be sorted: the test if [] <  is executed exactly the same number of times in every case.
Selection sort spends most of its time trying to find the minimum element in the unsorted part of the array. It
clearly shows the similarity between Selection sort and Bubble sort.

 Bubble sort selects the maximum remaining elements at each stage, but wastes some effort imparting
some order to an unsorted part of the array.
 Selection sort is quadratic in both the worst and the average case, and requires no extra memory.

For each i from 1 to n - 1, there is one exchange and n - i comparisons, so there is a total of n - 1 exchanges
and ( − ) + (− ) + . . . +  +  = (− )/ comparisons.
These observations hold, no matter what the input data is.
In the worst case, this could be quadratic, but in the average case, this quantity is O(n log n). It implies that the
running time of Selection sort is quite insensitive to the input.

Implementation
Void Selection-Sort(int numbers[], int array_size)
{
int i, j;
int min, temp;
for (i = 0; I < array_size-1; i++)
{
min = i;
for (j = i+1; j < array_size; j++)
if (numbers[j] < numbers[min])
min = j;
temp = numbers[i];
numbers[i] = numbers[min];
numbers[min] = temp;
}
}

Example

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 19 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

Quick Sort
It is used on the principle of divide-and-conquer. Quick sort is an algorithm of choice in many situations as it is
not difficult to implement. It is a good general purpose sort and it consumes relatively fewer resources during
execution.
Advantages
 It is in-place since it uses only a small auxiliary stack.
 It requires only  () time to sort n items.
 It has an extremely short inner loop.
 This algorithm has been subjected to a thorough mathematical analysis, a very precise statement can be
made about performance issues.

Disadvantages
 It is recursive. Especially, if recursion is not available, the implementation is extremely complicated.
 It requires quadratic (i.e., n2) time in the worst-case.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 20 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
 It is fragile, i.e. a simple mistake in the implementation can go unnoticed and cause it to perform badly.
Quick sort works by partitioning a given array A[p ... r] into two non-empty sub array A[p ... q] and A[q+1 ...
r] such that every key in A[p ... q] is less than or equal to every key in A[q+1 ... r].
Then, the two sub-arrays are sorted by recursive calls to Quick sort. The exact position of the partition depends
on the given array and index q is computed as a part of the partitioning procedure.

Algorithm: Quick-Sort (A, p, r)

if p < r then
q Partition (A, p, r)
Quick-Sort (A, p, q)
Quick-Sort (A, q + r, r)

Note that to sort the entire array, the initial call should be Quick-Sort (A, 1, length[A]) As a first step, Quick
Sort chooses one of the items in the array to be sorted as pivot.
Then, the array is partitioned on either side of the pivot. Elements that are less than or equal to pivot will move
towards the left, while the elements that are greater than or equal to pivot will move towards the right.
Analysis
The worst case complexity of Quick-Sort algorithm is O(n2). However using this technique, in average cases
generally we get the output in O(n log n) time.

RECURRENCE RELATIONS
Recurrence Relation for a sequence of numbers S is a formula that relates all but a finite number of terms of S
to previous terms of the sequence, namely, {a0, a1, a2, . . . . . , an-1}, for all integers n with n ≥ n0, where n0
is a nonnegative integer. Recurrence relations are also called as difference equations.
Sequences are often most easily defined with a recurrence relation; however the calculation of terms by
directly applying a recurrence relation can be time consuming. The process of determining a closed form
expression for the terms of a sequence from its recurrence relation is called solving the relation. Some guess
and check with respect to solving recurrence relation are as follows:
 Make simplifying assumptions about inputs
 Tabulate the first few values of the recurrence
 Look for patterns, guess a solution
 Generalize the result to remove the assumptions Examples: Factorial, Fibonnaci, Quick sort, Binary
search etc.

Recurrence relation is an equation, which is defined in terms of itself. There is no single technique or algorithm
that can be used to solve all recurrence relations. In fact, some recurrence relations cannot be solved. Most of
the recurrence relations that we encounter are linear recurrence relations with constant coefficients.
Several techniques like substitution, induction, characteristic roots and generating function are available to
solve recurrence relations.

The Iterative Substitution Method:

One way to solve a divide-and-conquer recurrence equation is to use the iterative substitution method. This is a
“plug-and-chug” method. In using this method, we assume that the problem size n is fairly large and we than
substitute the general form of the recurrence for each occurrence of the function T on the right-hand side. For
example, performing such a substitution with the merge sort recurrence equation yields the equation.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 21 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

The hope in applying the iterative substitution method is that, at some point, we will see a pattern that can be converted into a general
closed-form equation (with T only appearing on the left-hand side). In the case of merge-sort recurrence equation, the general form
is:

The Recursion Tree:

Another way of characterizing recurrence equations is to use the recursion tree method. Like the iterative
substitution method, this technique uses repeated substitution to solve a recurrence equation, but it differs from
the iterative substitution method in that, rather than being an algebraic approach, it is a visual approach.
In using the recursion tree method, we draw a tree R where each node represents a different substitution of the
recurrence equation. Thus, each node in R has a value of the argument n of the function T (n) associated with
it. In addition, we associate an overhead with each node v in R, defined as the value of the non-recursive part
of the recurrence equation for v.
For divide-and-conquer recurrences, the overhead corresponds to the running time needed to merge the
subproblem solutions coming from the children of v. The recurrence equation is then solved by summing the
overheads associated with all the nodes of R. This is commonly done by first summing values across the levels
of R and then summing up these partial sums for all the levels of R.

For example, consider the following recurrence equation:

This is the recurrence equation that we get, for example, by modifying the merge sort algorithm so that we divide an unsorted
sequence into three equal – sized sequences, recursively sort each one, and then do a three-way merge of three sorted sequences to
produce a sorted version of the original sequence. In the recursion tree R for this recurrence, each internal node v has three children
and has a size and an overhead associated with it, which corresponds to the time needed to merge the sub- problem solutions
produced by v‟s children. We illustrate the tree R as follows:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 22 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

Constructing a recursion tree for the recurrence T(n)=3T(n/4)+cn 2

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 23 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 24 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

The Master Theorem Method:

Each of the methods described above for solving recurrence equations is ad hoc and requires mathematical
sophistication in order to be used effectively. There is, nevertheless, one method for solving divide-and-
conquer recurrence equations that is quite general and does not require explicit use of induction to apply
correctly. It is the master method. The master method is a “cook-book” method for determining the asymptotic
characterization of a wide variety of recurrence equations. It is used for recurrence equations of the form:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 25 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 26 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 27 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

SOLVING RECURRENCE RELATIONS

Example 2.13.1. Solve the following recurrence relation:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 28 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 29 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

This implies that T (n) is O (n).

Example 2.13.3. Write a recurrence relation and solve the recurrence relation for the following fragment of
program:
Write T(n) in terms of T for fractions of n, for example, T(n/k) or T(n - k).
int findmax (int a[ ], int start, int end)
{
int mid, max1, max2;
if (start == end) // easy case (2 steps) return (a [start]);
mid = (start + end) / 2; // pre-processing (3)
max1 = findmax (a, start, mid); // recursive call: T(n/2) + 1 max2 = findmax (a, mid+1, end); // recursive
call: T(n/2) + 1 if (max1 >= max2) //post-processing (2)
return (max1);
else
}
return (max2);
Solution:
The Recurrence Relation is:

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 30 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 31 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

PART TWO

ALGORITHM DESIGN STRATEGIES

There are many ways to design algorithms.There are many ways to design algorithms. Insertion sort uses an
incremental approach:having sorted the subarray A[1 _ j - 1], we insert the single element A[j] into its proper
place,yielding the sorted subarray A[1 _ j].

In this section, we examine an alternative design approach, known as "divide-and-conquer." We shall use
divide-and-conquer to design a sorting algorithm whose worst-case running time is much less than that of
insertion sort. One advantage of divide-and-conquer algorithms is that their running times are often easily
determined using techniques.

1. Divide & Conquer

Many useful algorithms are recursive in structure: to solve a given problem, they call themselves recursively
one or more times to deal with closely related subproblems. These algorithms typically follow a divide-and-
conquer approach: they break the problem into several subproblems that are similar to the original problem but
smaller in size, solve the subproblems recursively, and then combine these solutions to create a solution to the
original problem.

Given a function to compute on n inputs the divide-and-conquer strategy suggests splitting the inputs into k
distinct subsets, 1 < k ~ n yielding k subproblems. These subproblems must be solved and then a method must
be found to combine subsolutions into a solution of the whole. If the subproblems are still relatively large, then
the divide-and-conquer strategy may possibly be reapplied. Often the subproblems resulting from a divideand-
conquer design are of the same type as the original problem. For those cases the reapplication of the divide-
and-conquer principle is naturally expressed by a recursive procedure. Now smaller and smaller subproblems
of the same kind are generated, eventually producing subproblems that are small enough to be solved without
splitting.

To be more precise suppose we consider the divide-and-conquer strategy when it splits the input into two
subproblems of the same kind as the original problem. This splitting is typical of many of the problems we will
see here. We can write a control abstraction which mirrors the way an actual program based upon divide-and-
conquer will look. By a control abstraction we informally mean a procedure whose flow of control is clear, but
whose primary operations are specified by other procedures whose precise meaning is left undefined.

Many algorithms are recursive in nature to solve a given problem recursively dealing with sub-problems.In
divide and conquer approach, a problem is divided into smaller problems, then the smaller problems are
solved independently, and finally the solutions of smaller problems are combined into a solution for the large
problem.

Generally, divide-and-conquer algorithms have three parts −

 Divide the problem into a number of sub-problems that are smaller instances of the same problem.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 32 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
 Conquer the sub-problems by solving them recursively. If they are small enough, solve the sub-
problems as base cases.
 Combine the solutions to the sub-problems into the solution for the original problem.

The merge sort algorithm closely follows the divide-and-conquer paradigm. Intuitively, it operates as follows.
o Divide: Divide the n-element sequence to be sorted into two subsequences of n/2 elements each.
o Conquer: Sort the two subsequences recursively using merge sort.
o Combine: Merge the two sorted subsequences to produce the sorted answer.

Pros and cons of Divide and Conquer Approach

Divide and conquer approach supports parallelism as sub-problems are independent. Hence, an algorithm,
which is designed using this technique, can run on the multiprocessor system or in different machines
simultaneously. In this approach, most of the algorithms are designed using recursion, hence memory
management is very high. For recursive function stack is used, where function state needs to be stored.

Application of Divide and Conquer Approach

Following are some problems, which are solved using divide and conquer approach.

 Finding the maximum and minimum of a sequence of numbers

 Strassen‟s matrix multiplication
 Merge sort
 Binary search

Max-Min Problem:
Let us consider a simple problem that can be solved by divide and conquer technique.

Problem Statement
The Max-Min Problem in algorithm analysis is finding the maximum and minimum value in an array.

Solution
To find the maximum and minimum numbers in a given array of size n, the following algorithm can be used.
First we are representing the naive method and then we will present divide and conquer approach.

Naïve Method
Naïve method is a basic method to solve any problem. In this method, the maximum and minimum number can
be found separately. To find the maximum and minimum numbers,

the following straightforward algorithm can be used.

Algorithm: Max-Min-Element (numbers[])

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 33 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
max := numbers[1]
min := numbers[1]
for i = 2 to n do
if numbers[i] > max then
max := numbers[i]
if numbers[i] < min then
min := numbers[i]
return (max, min)

Analysis
The number of comparison in Naive method is  − .
The number of comparisons can be reduced using the divide and conquer approach.

Following is the technique.

Divide and Conquer Approach

In this approach, the array is divided into two halves. Then using recursive approach maximum and minimum
numbers in each halves are found. Later, return the maximum of two maxima of each half and the minimum of
two minima of each half.

Algorithm: Max-Min(x, y)
if x –y ≤ 1 then
return (max(numbers[x], numbers[y]), min((numbers[x], numbers[y]))
else
(max1, min1):= maxmin(x, ⌊ ((x+y)/2)⌋ )
(max2, min2):= maxmin(⌊ ((x+y)/2) + 1)⌋ ,y)

return (max(max1, max2), min(min1, min2))

Analysis

Merge Sort:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 34 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
Problem Statement

The problem of sorting a list of numbers lends itself immediately to a divide-and-conquer strategy: split the list
into two halves, recursively sort each half, and then merge the two sorted sub-lists.

Solution

In this algorithm, the numbers are stored in an array numbers[]. Here, p and q represents the start and end
index of a sub-array.

Algorithm: Merge-Sort (numbers[], p, r)

if p < r then
q = ⌊ (p + q) / 2⌋
Merge-Sort (numbers[], p, q)
Merge-Sort (numbers[], q + 1, r)
Merge (numbers[], p, q, r)
Function: Merge (numbers[], p, q, r)
n1 = q – p + 1
n2 = r – q
declare leftnums[1…n1 + 1] and rightnums[1…n2 + 1] temporary arrays
for i = 1 to n1
leftnums[i] = numbers[p + i - 1]
for j = 1 to n2
leftnums[j] = numbers[q+ j]
leftnums[n1 + 1] = ∞
rightnums[n2 + 1] = ∞
i=1
j=1
for k = p to r
if leftnums[i] ≤ rightnums[j]
numbers[k] = leftnums[i]
i=i+1
else
numbers[k] = rightnums[j]
j=j+1

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 35 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------

Analysis

Let us consider, the running time of Merge-Sort as (). Hence,

Example2: Merge-Sort

In the following example, we have shown Merge-Sort algorithm step by step. First, every iteration array is
divided into two sub-arrays, until the sub-array contains only one element. When these sub-arrays cannot be
divided further, then merge operations are performed.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 36 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
The following figure gives a high-level view of the algorithm. The “divide” phase is shown on the left. It
works top-down splitting up the list into smaller sublists. The “conquer and combine” phases are shown on the
right. They work bottom-up, merging sorted lists together into larger sorted lists.

Problem Statement: Binary-Search algorithm based on divide and conquers method

Binary search can be performed on a sorted array. In this approach, the index of an element x is determined if
the element belongs to the list of elements. If the array is unsorted, linear search is used to determine the
position.

Solution

In this algorithm, we want to find whether element x belongs to a set of numbers stored in an array numbers[].
Where l and r represent the left and right index of a sub-array in which searching operation should be
performed.

Algorithm: Binary-Search(numbers[], x, l, r)
if l = r then
return l
else
m := ⌊ (l + r) / 2⌋
if x ≤ numbers[m] then
return Binary-Search(numbers[], x, l, m)
else
return Binary-Search(numbers[], x, m+1, r)

Analysis

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 37 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
Linear search runs in O(n) time. Whereas binary search produces the result in O(log n) time

Let T(n) be the number of comparisons in worst-case in an array of n elements.

Hence,

T(n)={0T(n2)+1ifn=1 otherwise

Using this recurrence relation T(n)=logn

Therefore, binary search uses O(logn) time.

Example

In this example, we are going to search element 63.

2. THE GREEDY METHOD

Among all the algorithmic approaches, the simplest and straightforward approach is the Greedy method. In this
approach, the decision is taken on the basis of current available information without worrying about the effect
of the current decision in future.
Greedy algorithms build a solution part by part, choosing the next part in such a way, that it gives an
immediate benefit. This approach never reconsiders the choices taken previously. This approach is mainly used
to solve optimization problems. Greedy method
is easy to implement and quite efficient in most of the cases. Hence, we can say that
Greedy algorithm is an algorithmic paradigm based on heuristic that follows local optimal choice at each step
with the hope of finding global optimal solution.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 38 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
In many problems, it does not produce an optimal solution though it gives an approximate (near optimal)
solution in a reasonable time.

Components of Greedy Algorithm

Greedy algorithms have the following five components:
 A candidate set: A solution is created from this set.
 A selection function: Used to choose the best candidate to be added to the solution.
 A feasibility function: Used to determine whether a candidate can be used to contribute to the solution.
 An objective function: Used to assign a value to a solution or a partial solution.
 A solution function: Used to indicate whether a complete solution has been reached.

Areas of Application
Greedy approach is used to solve many problems, such as
 Finding the shortest path between two vertices using Dijkstra‟s algorithm.
 Finding the minimal spanning tree in a graph using Prim‟s /Kruskal‟s algorithm, etc.

Where Greedy Approach Fails

In many problems, Greedy algorithm fails to find an optimal solution, moreover it may produce a worst
solution. Problems like Travelling Salesman and Knapsack cannot be solved using this approach.

Example: Spanning Tree

A spanning tree is a subset of an undirected Graph that has all the vertices connected by minimum number of
edges. If all the vertices are connected in a graph, then there exists at least one spanning tree.
In a graph, there may exist more than one spanning tree.

Properties
 A spanning tree does not have any cycle.
 Any vertex can be reached from any other vertex.

Tree
A Minimum Spanning Tree (MST) is a subset of edges of a connected weighted undirected graph that
connects all the vertices together with the minimum possible total edge weight. To derive an MST, Prim‟s
algorithm or Kruskal‟s algorithm can be used.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 39 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
Example

In the following graph, the highlighted edges form a spanning tree.

Minimum Spanning Tree

A Minimum Spanning Tree (MST) is a subset of edges of a connected weighted undirected graph that
connects all the vertices together with the minimum possible total edge weight. To derive an MST, Prim‟s
algorithm or Kruskal‟s algorithm can be used. Hence, we will discuss Prim‟s algorithm in this chapter.

As we have discussed, one graph may have more than one spanning tree. If there are n number of vertices, the
spanning tree should have n - 1 number of edges. In this context, if each edge of the graph is associated with a
weight and there exists more than one spanning tree, we need to find the minimum spanning tree of the graph.

Moreover, if there exist any duplicate weighted edges, the graph may have multiple minimum spanning tree.

In the above graph, we have shown a spanning tree though it‟s not the minimum spanning tree. The cost of this spanning
tree is (5 + 7 + 3 + 3 + 5 + 8 + 3 + 4) = 38.

We will use Prim‟s algorithm to find the minimum spanning tree.

Prim’s Algorithm

Prim‟s algorithm is a greedy approach to find the minimum spanning tree. In this algorithm, to form a MST we
can start from an arbitrary vertex.

Algorithm: MST-Prim’s (G, w, r)

for each u є G.V
u.key = ∞
u.∏ = NIL
r.key = 0
Q = G.V
while Q ≠ Ф
u = Extract-Min (Q)
for each v є G.adj[u]
if each v є Q and w(u, v) < v.key
v.∏ = u
v.key = w(u, v)

The function Extract-Min returns the vertex with minimum edge cost. This function works on min-heap.

Example

Using Prim‟s algorithm, we can start from any vertex, let us start from vertex 1.

Vertex 3 is connected to vertex 1 with minimum edge cost, hence edge (1, 2) is added to the spanning tree.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 40 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
Next, edge (2, 3) is considered as this is the minimum among edges {(1, 2), (2, 3), (3, 4), (3, 7)}.

In the next step, we get edge (3, 4) and (2, 4) with minimum cost. Edge (3, 4) is selected at random.

In a similar way, edges (4, 5), (5, 7), (7, 8), (6, 8) and (6, 9) are selected. As all the vertices are visited, now the
algorithm stops.

The cost of the spanning tree is (2 + 2 + 3 + 2 + 5 + 2 + 3 + 4) = 23. There is no more spanning tree in this
graph with cost less than 23.

Here are some examples:

To explain further upon the Minimum Spanning Tree, and what it applies to, let's consider a couple of real-
world examples:
1. One practical application of a MST would be in the design of a network. For instance, a group of
individuals, who are separated by varying distances, wish to be connected together in a telephone network.
Although MST cannot do anything about the distance from one connection to another, it can be used to
determine the least cost paths with no cycles in this network, thereby connecting everyone at a minimum cost.

2. Another useful application of MST would be finding airline routes. The vertices of the graph would
represent cities, and the edges would represent routes between the cities. Obviously, the further one has to
travel, the more it will cost, so MST can be applied to optimize airline routes by finding the least costly paths
with no cycles.

3. Dynamic Programming
Dynamic programming, like the divide-and-conquer method, solves problems by combining the solutions to
subproblems. ("Programming" in this context refers to a tabular method, not to writing computer code.)
Dynamic Programming is also used in optimization problems. Like divide-and-conquer method, Dynamic
Programming solves problems by combining the solutions of sub problems. Moreover, Dynamic Programming
algorithm solves each sub-problem just once and then saves its answer in a table, thereby avoiding the work of
re-computing the answer every time. Two main properties of a problem suggest that the given problem can be
solved using Dynamic Programming. These properties are overlapping sub-problems and optimal
substructure.

Overlapping Sub-Problems: similar to Divide-and-Conquer approach, Dynamic Programming also combines

solutions to sub-problems. It is mainly used where the solution of one sub-problem is needed repeatedly. The
computed solutions are stored in a table, so that these don‟t have to be re-computed. Hence, this technique is
needed where overlapping sub-problem exists.

For example, Binary Search does not have overlapping sub-problem. Whereas recursive program of Fibonacci
numbers have many overlapping sub-problems.

Optimal Sub-Structure: A given problem has Optimal Substructure Property, if the optimal solution of the
given problem can be obtained using optimal solutions of its sub-problems.

For example, the Shortest Path problem has the following optimal substructure property −

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 41 of 42
 Algorithm Analysis and Design Lecture Notes
----------------------------------------------------------------------------------------------------------------------------------------------------------
If a node x lies in the shortest path from a source node u to destination node v, then the shortest path from u to
v is the combination of the shortest path from u to x, and the shortest path from x to v.

The standard All Pair Shortest Path algorithms like Floyd-Warshall and Bellman-Ford are typical examples of
Dynamic Programming.

Steps of Dynamic Programming Approach

Dynamic Programming algorithm is designed using the following four steps −

 Characterize the structure of an optimal solution.

 Recursively define the value of an optimal solution.
 Compute the value of an optimal solution, typically in a bottom-up fashion.
 Construct an optimal solution from the computed information.

Applications of Dynamic Programming Approach

 Matrix Chain Multiplication

 Longest Common Subsequence
 Travelling Salesman Problem

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Prepared by Gezae G. Page 42 of 42