Searching algorithms

Searching is also a common and well-studied task. This task can be described formally as follows:

Given a list of values, a function that compares two values and a desired value, find the position of
the desired value in the list.

We will look at two algorithms that perform this task:

• linear search, which simply checks the values in sequence until the desired value is found
• binary search, which requires a sorted input list, and checks for the value in the middle of
the list, repeatedly discarding the half of the list which contains values which are definitely
either all larger or all smaller than the desired value

There are numerous other searching techniques. Often they rely on the construction of more complex
data structures to facilitate repeated searching. Examples of such structures are hash tables (such as
Python’s dictionaries) and prefix trees. Inexact searches that find elements similar to the one being
searched for are also an important topic.

Linear search

Linear search is the most basic kind of search method. It involves checking each element of the list in
turn, until the desired element is found.

For example, suppose that we want to find the number 3.8 in the following list:

We start with the first element, and perform a comparison to see if its value is the value that we want.
In this case, 1.5 is not equal to 3.8, so we move onto the next element:
We perform another comparison, and see that 2.7 is also not equal to 3.8, so we move onto the next
element:

We perform another comparison and determine that we have found the correct element. Now we can
end the search and return the position of the element (index 2).

We had to use a total of 3 comparisons when searching through this list of 4 elements. How many
comparisons we need to perform depends on the total length of the list, but also whether the element
we are looking for is near the beginning or near the end of the list. In the worst-case scenario, if our
element is the last element of the list, we will have to search through the entire list to find it.

If we search the same list many times, assuming that all elements are equally likely to be searched
for, we will on average have to search through half of the list each time. The cost (in comparisons) of
performing linear search thus scales linearly with the length of the list.

Exercise (Linear Search)

1. Write a function which implements linear search. It should take a list and an element as a
parameter, and return the position of the element in the list. If the element is not in the list,
the function should raise an exception. If the element is in the list multiple times, the
function should return the first position.

Binary search

Binary search is a more efficient search algorithm which relies on the elements in the list being
sorted. We apply the same search process to progressively smaller sub-lists of the original list,
starting with the whole list and approximately halving the search area every time.

We first check the middle element in the list.

• If it is the value we want, we can stop.

• If it is higher than the value we want, we repeat the search process with the portion of the
list before the middle element.
 • If it is lower than the value we want, we repeat the search process with the portion of the
list after the middle element.

For example, suppose that we want to find the value 3.8 in the following list of 7 elements:

First we compare the element in the middle of the list to our value. 7.2 is bigger than 3.8, so we need
to check the first half of the list next.

Now the first half of the list is our new list to search. We compare the element in the middle of this list
to our value. 2.7 is smaller than 3.8, so we need to search the second half of this sublist next.

The second half of the last sub-list is just a single element, which is also the middle element. We
compare this element to our value, and it is the element that we want.

We have performed 3 comparisons in total when searching this list of 7 items. The number of
comparisons we need to perform scales with the size of the list, but much more slowly than for linear
search – if we are searching a list of length N, the maximum number of comparisons that we will have
to perform is log2N.
Exercise (Binary Search)

1. Write a function which implements binary search. You may assume that the input list will be
sorted. Hint: this function is often written recursively.

Algorithm complexity and Big O notation

We commonly express the cost of an algorithm as a function of the number N of elements that the
algorithm acts on. The function gives us an estimate of the number of operations we have to perform
in order to use the algorithm on N elements – it thus allows us to predict how the number of required
operations will increase as N increases. We use a function which is an approximation of the exact
function – we simplify it as much as possible, so that only the most important information is
preserved.

For example, we know that when we use linear search on a list of N elements, on average we will
have to search through half of the list before we find our item – so the number of operations we will
have to perform is N/2. However, the most important thing is that the algorithm scales linearly – as N
increases, the cost of the algorithm increases in proportion to N, not N2 or N3. The constant factor of
1/2 is insignificant compared to the very large differences in cost between – for example – N and N2,
so we leave it out when we describe the cost of the algorithm.

We thus write the cost of the linear search algorithm as O(N) – we say that the cost is on the order of
N, or just order N. We call this notation big O notation, because it uses the capital O symbol
(for order).

We have dropped the constant factor 1/2. We would also drop any lower-order terms from an
expression with multiple terms – for example, O(N3 + N2) would be simplified to O(N3).

In the example above we calculated the average cost of the algorithm, which is also known as
the expected cost, but it can also be useful to calculate the best case and worst case costs. Here are
the best case, expected and worst case costs for the sorting and searching algorithms we have
discussed so far:

Algorithm Best case Expected Worst case

Selection sort O(N2) O(N2) O(N2)

Merge sort O(N log N) O(N log N) O(N log N)

Linear search O(1) O(N) O(N)

Binary search O(1) O(log N) O(log N)

 What does O(1) mean? It means that the cost of an algorithm is constant, no matter what the value of
N is. For both these search algorithms, the best case scenario happens when the first element to be
tested is the correct element – then we only have to perform a single operation to find it.

In the previous table, big O notation has been used to describe the time complexity of algorithms. It
can also be used to describe their space complexity – in which case the cost function represents the
number of units of space required for storage rather than the required number of operations. Here are
the space complexities of the algorithms above (for the worst case, and excluding the space required
to store the input):

Algorithm Space complexity

Selection sort O(1)

Merge sort O(N)

Linear search O(1)

Binary search O(1)

None of these algorithms require a significant amount of storage space in addition to that used by the
input list, except for the merge sort – which, as we saw in a previous section, requires temporary
storage which is the same size as the input (and thus scales linearly with the input size).

Note

The Python wiki has a summary of the time complexities of common operations on collections. You
may also wish to investigate the collections module, which provides additional collection classes
which are optimised for particular tasks.
Note

Computational complexity theory studies the inherent complexity of tasks themselves. Sometimes it is
possible to prove that any algorithm that can perform a given task will require some minimum number
of steps or amount of extra storage. For example, it can be shown that, given a list of arbitrary objects
and only a comparison function with which to compare them, no sorting algorithm can use fewer than
O(N log N) comparisons.
Exercise (Algorithm complexity and Big O notation)
1. We can see from the comparison tables above that binary search is more efficient than
linear search. Why would we ever use linear search? Hint: what property must a list have
for us to be able to use a binary search on it?
2. Suppose that each of the following functions shows the average number of operations
required to perform some algorithm on a list of length N. Give the big O notation for the
time complexity of each algorithm:
1. 4N2 + 2N + 2
2. N + log N
3. N log N
4. 3