ES106: Programming for Engineers

Lecture 9, 10, 11: Basic Algorithm

Basic Searching Algorithm
How do you find someone’s phone number in the phone book? How do you find your keys when you’ve
misplaced them? If a deck of cards has less than 52 cards, how do you determine which card is missing?

Searching for items and sorting through items are tasks that we do every day. Searching and
sorting are also common tasks in computer programs. We search for all occurrences of a word in a file in
order to replace it with another word. We sort the items on a list into alphabetical or numerical order.
Because searching and sorting are common computer tasks, we have well-known algorithms, or recipes,
for doing searching and sorting.

Searching is important in many areas of computer science. Many large computer systems include
databases that must be searched when an inquiry is made. In many examples and in many other situations,
such as looking up account balances, finding plane reservations, retrieving on-line orders, checking license
plates, and many, many others, the first task is to find an entry in a database that is indexed by a target
number, and it is always accomplished by a search algorithm.

There are two types of search algorithms: algorithms that don’t make any assumptions about the
order of the list, and algorithms that assume the list is already in order.

Linear search:
The simplest search algorithm is sequential search, also known as linear search. In this method, the target
number that is being sought is compared with the first member of the database, then with the second,
etc., until either the number is found, or the end of the database is reached, whichever comes first.

In linear search, we look at each item in the list in turn, quitting once we find an item that matches the
search term or once we’ve reached the end of the list. Our “return value” is the index at which the search
term was found, or some indicator that the search term was not found in the list.

Let’s work on the simplest version of the search problem. We are given a vector of numbers and a target
value. Our task is to search for that target value in the vector. If we find it, then the answer we give is the
index of the element that we found. If we do not find it, we return an impossible index, say, −1, as a flag
that indicates that the search failed. That flag is crucial to the search algorithm because there must be
some means of informing the searcher that the target is not there.

Faster searching methods are available, but only if the list has been sorted before the search is undertaken.
For an unsorted list, the sequential search is the only possible choice. Here is the pseudocode for Linear
Search followed by flow chart shown in Figure 1.

Pseudocode for Linear Search

For all items in the list
Compare the item with the desired item
If the item is found
Return the index value of the current item

ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 1 of 16
 Endif
EndFor
Return -1 if the item is not found
Flow chart for Linear search

Figure 1 Flow chart for Linear Search

Example of Linear search

The function written in MATLAB, sequential_search, below carries out the sequential search:
function index = sequential_search ...
(vector,target,first,last)
%SEQUENTIAL_SEARCH
% SEQUENTIAL_SEARCH(VECTOR,TARGET,FIRST,LAST) returns
% smallest index for which TARGET == VECTOR(index) or
% -1, if TARGET not found within VECTOR(FIRST:LAST).
found = false; % Assume the target is not in vector
for n = first:last
if target == vector(n)

ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 2 of 16
 found = true; % We found it...
break; % so we quit looking for it!
end
end
index = n;
if ~found
index = -1;
end
We note that that there are four arguments. Argument 1 is the vector to be searched; Argument 2 is the
target that we are searching for; Arguments 3 and 4 give the beginning and the end, respectively, of the
range of indices over which we wish to search. Most of the time, when we call the function, we will set
first to 1 and last to the full length of vector, but sometimes we may wish to search only a limited range.
Arguments 3 and 4 are there to provide this option.

So, for example, if we wish to search all of a vector, named accounts, for the number stored in smith, we
would give the call,
>> sequential_search(accounts, smith, 1, length(accounts))

but, if we wish to search only within, say, accounts (450:5600), we would give the call,
>> sequential_search(accounts, smith, 450, 5600)

If the target is on the list, then sequential search returns the first position at which it finds the target. So,
for example, suppose the function is applied to the list [45 23 17 17 –2 100 34]. In that case, the number
17 would be found at index 3 after the target had been compared with three numbers: 45, 23, and 17 (the
first 17, that is). When searching for a target that is not on the list, 82, for example, the function will have
to compare the target with every number on the list, requiring 7 comparisons. These few comparisons
will not take long, but if there are a million numbers on the list, time can become a significant factor.

Performance of linear search:

When comparing search algorithms, we only look at the number of comparisons, since we don’t swap any
values while searching. Often, when comparing performance, we look at three cases:

• Best case: What is the fewest number of comparisons necessary to find an item?
• Worst case: What is the greatest number of comparisons necessary to find an item?
• Average case: On average, how many comparisons does it take to find an item in the list?

For linear search, our cases look like this:

• Best case: The best case occurs when the search term is in the first slot in the array. The
number of comparisons in this case is 1.
• Worst case: The worst case occurs when the search term is in the last slot in the array, or is
not in the array. The number of comparisons in this case is equal to the size of the array. If
our array has N items, then it takes N comparisons in the worst case.

ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 3 of 16
 • Average case: On average, the search term will be somewhere in the middle of the array. The
number of comparisons in this case is approximately N/2.

In both the worst case and the average case, the number of comparisons is proportional to the
number of items in the array, N. Thus, we say in these two cases that the number of comparisons is order
N, or O(N) for short. For the best case, we say the number of comparisons is order 1, or O(1).

Linear search works well in many cases, particularly if we don’t know if our list is in order. It’s one
drawback is that it can be slow. If N, the number of items in our list, is 1,000,000, then it can take a long
time on average to find the search term in the list (on average, it will take 500,000 comparisons).

Binary search:
If a list has been sorted before the search is undertaken, searches can be carried out much more quickly.
The search method that should be used on a sorted list is the binary search. It is more complicated than
the sequential search method, but the complication is well worth it: A binary search requires no more
than 40 comparisons to find a number in a list of one million numbers!

If our list is already in order, for example, think about looking up a name in the phone book. It
makes more sense to exploit the ordering of the names, start our search somewhere near the starting
alphabet, and refine the search from there rather than by starting at the beginning and looking at each
name in turn.

Binary search exploits the ordering of a list. The idea behind binary search is that each time we
make a comparison, we eliminate half of the list, until we either find the search term or determine that
the term is not in the list. We do this by looking at the middle item in the list, and determining if our search
term is higher or lower than the middle item. If it’s lower, we eliminate the upper half of the list and
repeat our search starting at the point halfway between the first item and the middle item. If it’s in higher,
we eliminate the lower half of the list and repeat our search starting at the point halfway between the
middle item and the last item.

Here's the pseudocode for binary search followed by flow chart shown in , for searching in an
array. The inputs are the array, which we call array; the number n of elements in array; and target,
the number being searched for. The output is the index in array of target:

Pseudocode for Binary Search:

1. Let min = 0 and max = n-1.

2. Compute guess as the average of max and min, rounded down (so that it is an integer).
3. If array[guess] equals target, then stop. You found it! Return guess.
4. If the guess was too low, that is, array[guess] < target, then set min =
guess + 1.
5. Otherwise, the guess was too high. Set max = guess - 1.
6. Go back to step 2.

Flow chart for Binary search

ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 4 of 16
 Figure 2 Flow chart for Binary Search

Example of Binary search:

Recursive implementation of the binary search: When you use this approach, you are carrying out your
own recursive approximation of the binary search, and that is the version of binary search that we will
consider first. Here is a recursive function that implements the binary search algorithm, providing the
same interface (arguments) as that used by sequential_search:

1. function index = binary_search_recursive ...

2. (vector,target,first,last)
3. % BINARY_SEARCH_RECURSIVE
4. % BINARY_SEARCH_RECURSIVE(VECTOR,TARGET,FIRST,LAST)
ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 5 of 16
5. % returns an index for which TARGET == VECTOR(index)
6. % or -1, if TARGET not found in VECTOR(FIRST:LAST).
7.
8. mid = fix( (first + last)/2 );
9. if ~(first <= last) % If first and last out of order..
10. index = -1; % ..then target not on the list!
11. elseif target == vector(mid)
12. index = mid; % found it!
13. elseif target < vector(mid)
14. index = binary_search_recursive ...
15. (vector,target,first, mid-1);
16. else
17. index = binary_search_recursive ...
18. (vector,target,mid+1, last);
19. end
The function works as follows:

Line 8: The middle of the range is determined. If first is odd and last is even, or vice versa, then that
average will not be an integer. Since mid must be an integer in order to be an index, the fix() function
is used to discard the fractional part (which is always 0.5), if there is one.

Lines 9-10: If first and last are out of order, then the range is empty, so target cannot possibly be
in the range. Therefore, failure is flagged by returning −1.

Lines 11-12: The value of target is compared to the element at the middle of the range. If the target
and the element are equal, then the search is successful, and the middle index is returned.

Line 13: If target is less than that middle number, then the middle element and all the elements that
come after the middle element can be eliminated. A recursive call is made to search only the range of
elements that come before the middle. Whatever index is received from this recursive call is returned.

Lines 14-15: Else the target must be greater than the middle element. That means that the middle
element and all the elements that come before the middle element are eliminated. A recursive call is made
to search only the range of elements that come after the middle. Whatever index is received from this
recursive call is returned.

Line 16: This search algorithm is in fact named “binary” search because the list is repeatedly divided into
two parts. Subsequent searching in one of those subranges is accomplished by specifying the desired
subrange via the third and fourth arguments and applying the binary search to it.

Lines 17-18: It is informative to follow the algorithm as it works its way through a few specific searches.
Suppose we are searching for targets in a sorted vector named A that contains 11 elements:

ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 6 of 16
Example: Find target number 88 in

In our first search let’s look for the target number 88:
A = [2 17 17 18 24 43 74 77 80 88 97] .

Answer:
>> binary_search_recursive(A,88,1,length(A))
ans =
10

The following is a summary of the search progression showing the search ranges for each successive
function call along with the element A(mid) in the middle of the range that was compared to the target:
Range= A(1)= 2 to A(11)= 97, mid element A(6)= 43
Range= A(7)= 74 to A(11)= 97, mid element A(9)= 80
Range= A(10)= 88 to A(11)= 97, mid element A(10)= 88

Thus, the three numbers, 43, 80, and 88, were compared with the target.

Iterative implementation of the binary search

function index = binary_search_iterative ...
(vector,target,first,last)
%BINARY_SEARCH_ITERATIVE
% BINARY_SEARCH_ITERATIVE(VECTOR,TARGET,FIRST,LAST)
% smallest index for which TARGET == VECTOR(index) or
% returns an index for which TARGET == VECTOR(index)
% or -1, if TARGET not found in VECTOR(FIRST:LAST).
found = false;
while first <= last && ~found
mid = fix( (first + last) /2 );
if target < vector(mid)
last = mid - 1;
elseif target > vector(mid)
first = mid + 1 ;
else
found = true;
end
end
if found
ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 7 of 16
 index = mid;
else
index = -1;
end
As in the iterative version, the list is repeatedly divided into smaller and smaller parts, but instead of
executing a recursive call after each division, another iteration of the while-loop is executed.
Performance of Binary search:

The best case for binary search still occurs when we find the search term on the first try. In this
case, the search term would be in the middle of the list.
The worst case for binary search occurs when the search term is not in the list, or when the search
term is one item away from the middle of the list, or when the search term is the first or last item in the
list.
The average case occurs when the search term is anywhere else in the list. The number of
comparisons is roughly the same as for the worst case, so it also is O(log N). In general, anytime an
algorithm involves dividing a list in half, the number of operations is O(log N).

Binary search certainly seems faster and more efficient than linear search. But binary search has
an unfair advantage: it assumes the list is already in sorted order.

Basic Sorting Algorithm

Sorting refers to arranging data in a particular format. Sorting algorithm specifies the way to arrange data
in a particular order. The importance of sorting lies in the fact that data searching can be optimized to a
very high level, if data is stored in a sorted manner. Sorting is also used to represent data in more readable
formats. In this section, we’ll look at such algorithms in detail, and determine the performance of each
algorithm in terms of the number of data comparisons and the number of times we swap list items.

Selection sort:
The idea behind selection sort is that we put a list in order by placing each item in turn. In other words,
we put the smallest item at the start of the list, then the next smallest item at the second position in the
list, and so on until the list is in order.

Pseudocode for Selection sort

Pseudocode for selection sort is written here which is followed by the flow chart for the same (Refer Figure
3).

For i = 1 to n-1

min = i

For j = i+1 to n

If A[j] < A[min] then

ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 8 of 16
 min=j

End for

If min ≠ i then

Interchange A[i] and A[min]

End for

Flow chart for Selection sort

Figure 3 Flow chart for Selection Sort

A sorting algorithm that is not very efficient, but is simple enough to be readily understood is the selection
sort. Here is a function written in MATLAB that implements the Selection Sort algorithm:
function v = selection_sort(v)
%SELECTION_SORT sort in ascending order
% V = SELECTION_SORT(V) sorts vector V into
% ascending order. The method used is
% selection sort.
ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 9 of 16
for m = 1:length(v)-1
m_min = m;
for n = m+1:length(v)
if v(n) < v(m_min)
m_min = n;
end
end
if m_min ~= m
temp = v(m);
v(m) = v(m_min);
v(m_min) = temp;
end
end
It is used as follows,
>> vs = selection_sort(vu);

where vu is the vector to be sorted and vs is a vector containing the same elements as vs but arranged
in ascending order.

Consider the following numbers are stored in an array:

Original Array: 77,33,44,11,88,22,66,55

Pass 1 :11,33,44,77,88,22,66,55
Pass 2 :11,22,44,77,88,33,66,55
Pass 3 :11,22,33,77,88,44,66,55
Pass 4 :11,22,33,44,88,77,66,55
Pass 5 :11,22,33,44,55,77,66,88
Pass 6 :11,22,33,44,55,66,77,88

Performance of Selection sort:

The best case for selection sort occurs when the list is already sorted. In this case, the number of swaps is
zero. We still have to compare each item in the list to each other item in the list on each pass through the
algorithm. The first time through, we compare the first item in the list to all other items in the list, so the
number of comparisons is (N-1). The second time through, we compare the second item in the list to the
remaining items in the list, so the number of comparisons is (N-2). It turns out that the total number of
comparisons for selection sort in the best case is (N − 1) + (N − 2) + ... + 2 + 1. This equation simplifies to
(N (N + 1)/2) − 1, which is approximately N2. Thus, even in the best case, selection sort requires O(N2)
comparisons.

The worst case for selection sort occurs when the first item in the list is the largest, and the rest of the list
is in order. In this case, we perform one swap on each pass through the algorithm, so the number of swaps
is N. The number of comparisons is the same as in the best case, O(N2).

The average case requires the same number of comparisons, O(N2), and roughly N/2 swaps. Thus, the
number of swaps in the average case is O(N).
ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 10 of 16
In summary, we say that selection sort is O(N) in swaps and O(N2) in comparisons.

Bubble sort:
Bubble sort is a simple sorting algorithm. This sorting algorithm is comparison-based algorithm in which
each pair of adjacent elements is compared and the elements are swapped if they are not in order. The
idea behind bubble sort is similar to the idea behind selection sort: on each pass through the algorithm,
we place at least one item in its proper location. The differences between bubble sort and selection sort
lie in how many times data is swapped and when the algorithm terminates. Bubble sort performs more
swaps in each pass, in the hopes that it will finish sorting the list sooner than selection sort will.

Like selection sort, bubble sort works by comparing two items in the list at a time. Unlike selection sort,
bubble sort will always compare two consecutive items in the list, and swap them if they are out of order.
If we assume that we start at the beginning of the list, this means that at each pass through the algorithm,
the largest remaining item in the list will be placed at its proper location in the list.

Pseudocode for Bubble sort

Pseudocode for Bubble sort is written here which is followed by the flow chart for the same (Refer Figure
4).

Procedure bubblesort (A : List of sortable items)

N = length(A)

Repeat

newn = 0

for i = 1 to n-1 inclusive do

if A[i-1] > A[i] then

swap ( A[i-1], A[i] )

newn = i

end if

end for

n = newn

until n = 0

end Procedure

Here is a function written in MATLAB that implements the Bubble Sort algorithm:
% This program shows you how to put a vector of numbers in
% an ascending order using the bubble sort method
clc
clear all
disp(‘This program shows the bubble sort method’)
ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 11 of 16
disp(‘to put a vector of numbers in an ‘)
disp(‘ascending order’)
disp(‘Matlab 2007a’)
disp(‘Authors : Autar Kaw’)
disp(‘Last Revised : November 8, 2009’)
disp(‘http://numericalmethods.eng.usf.edu&#8217;)
disp(‘ ‘)
%% INPUTS
% The vector of numbers
disp (‘INPUTS’)
disp(‘Input the vector of numbers’)
A=[18 7 6 15 4 13];
disp(A)
%% SOLUTION
% Number of entries, n
n=length(A);
% making (n-1) passes
for j=1:1:n-1
% comparing each number with the next and swapping
for i=1:1:n-1
if A(i)>A(i+1);
% temp is a variable where the numbers are kept
% temporarily for the switch
temp=A(i);
A(i)=A(i+1);
A(i+1)=temp;
end
end
end

Example of Bubble sort

Consider the following numbers are stored in an array:

Original Array: 32,51,27,85,66,23,13,57

Pass 1: 32,27,51,66,23,13,57,85
Pass 2: 27,33,51,23,13,57,66,85
Pass 3: 27,33,23,13,51,57,66,85
Pass 4: 27,23,13,33,51,57,66,85
Pass 5: 23,13,27,33,51,57,66,85
Pass 6: 13,23,27,33,51,57,66,85

Flow chart for Bubble sort

ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 12 of 16
 Figure 4 Flow chart for Bubble Sort

Performance of Bubble sort:

The best case for bubble sort occurs when the list is already sorted. In this case, bubble sort makes one
pass through the list, performing no swaps and N-1 comparisons.

The worst case for bubble sort occurs when the list is in reverse order. In this case, every item will have
to be swapped with every other item on every pass through the algorithm. As in selection sort, there will
be O(N2) comparisons. The number of swaps in the worst case is greater than in selection sort: each
comparison results in a swap, so there are O(N2) swaps in bubble sort!

The average case looks like the worst case: O(N2) comparisons and O(N2) swaps. The trade-off is that we
may be able to do half as many iterations, on average, as we do in selection sort. From a mathematical
standpoint, however, bubble sort performs worse than selection sort in terms of the number of swaps,
and the same as selection sort in terms of the number of comparisons.

Insertion sort:
This is an in-place comparison-based sorting algorithm. Here, a sub-list is maintained which is always
sorted. For example, the lower part of an array is maintained to be sorted. An element which is to be
ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 13 of 16
'inserted’ in this sorted sub-list, has to find its appropriate place and then it has to be inserted there.
Hence the name, insertion sort.

The array is searched sequentially and unsorted items are moved and inserted into the sorted sub-list (in
the same array). This algorithm is not suitable for large data sets as its average and worst-case complexity
are of Ο(n2), where n is the number of items.

Pseudocode for Insertion sort

Pseudocode for Insertion sort is written here which is followed by the flow chart for the same (Refer Figure
5).

Insertion_sort[A]

for (j = 2, j ≤ length(A), j = j+1)

key = A[j]

i = j-1

while ( i > 0 and A[i] > key )

A[i+1] = A[i]

i=i–1

end while

A[i+1] = key

end for

Flow chart for Insertion sort

ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 14 of 16
 Figure 5 Flow chart for Insertion Sort

Here is a function written in MATLAB that implements the Insertion Sort algorithm:
function sorted = insertion_sort(array)
% Insertion sort in ascending order
% @input - Array to be sorted
% @output - Sorted Array
% Usage:
% input_array = [ 9 8 7 6 5 4 3 2 1 0];
% output_array = insertion_sort(input_array);
if(size(array,1)>1)
error('Input must be a 1xN vector');
end
if(isempty(array))
ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 15 of 16
 error('Input should not be empty');
end
disp(['Array to be sorted: ' num2str(array)]);
n = length(array);
for i = 2:n
d = i;
while((d > 1) && (array(d) < array(d-1)))
temp = array(d);
array(d) = array(d-1);
array(d-1) = temp;
d = d-1;
end
end
sorted = array;
disp(['Sorted Array: ' num2str(array)]);
end

Example of Insertion sort

Consider the following numbers are stored in an array:

Original Array: 77,33,44,11,88,22,66,55

Pass 1: 33,77,44,11,88,22,66,55
Pass 2: 33,44,77,11,88,22,66,55
Pass 3: 11,33,44,77,88,22,66,55
Pass 4: 11,22,33,44,77,88,66,55
Pass 5: 11,22,33,44,55,77,88,66
Pass 6: 11,22,33,44,55,66,77,88

Performance of Insertion sort:

The best case input is an array that is already sorted. In this case insertion sort has a linear running time
(i.e., O(n)). During each iteration, the first remaining element of the input is only compared with the right-
most element of the sorted subsection of the array.

The simplest worst case input is an array sorted in reverse order. The set of all worst case inputs consists
of all arrays where each element is the smallest or second-smallest of the elements before it. In these
cases every iteration of the inner loop will scan and shift the entire sorted subsection of the array before
inserting the next element. This gives insertion sort a quadratic running time (i.e., O(n2)).

The average case is also quadratic, which makes insertion sort impractical for sorting large arrays.
However, insertion sort is one of the fastest algorithms for sorting very small arrays.

It is efficient for smaller data sets, but very inefficient for larger lists. Insertion Sort is adaptive, that means
it reduces its total number of steps if a partially sorted array is provided as input, making it efficient. It is
better than Selection Sort and Bubble Sort algorithms. Its space complexity is less. It is a stable sorting
technique, as it does not change the relative order of elements which are equal.
ES106: Programming for Engineers, Lecture Note, Prof. (Dr.) Vinay J. Patel 16 of 16