ESO211 (Data Structures and Algorithms) Lectures 1 to 3

Shashank K Mehta

Random Access Machine

The computers of today can be simplied into the following model. These have a control unit, a memory where locatons have address 0 to N, and a data-path where data manipulation is performed. Initially the program (a sequence of instructions) is placed in the memory, relevent data is placed in the memory (it is ensured that two remain disjoint), and the control unit is set with the memory address of the rst instruction (by setting the address in the PC register.) The control unit reads the address of the next instruction in PC, brings it from the memory, reads it and brings all the data referred in this instruction, sets up the datapath to perform the necessary manipulations and nally it restores the results back into the memory in the location mentioned by the instruction. In this cycle the PC is also suitably updated. Computer repeats this innitely. It is important to remember that accessing (reading/writing) any data item takes a xed amount of time. Similarly each data manipulation (arithmatic, logic operation) has a xed time requirement.

The Concept of Problem Size

A problem for which a program is written is a family of problem instances. For example a program that computes the sum of two numbers, say a + b, has one instance for each pair of values of a and b. Now consider two instances: 1 + 1 and 1000000000000000+100000000000000000. Do you think both instances will take the same amount of time? In some way the latter instance appears to be larger. So we need to dene a notion of the size of the instances. We might dene the number of digits in the larger of the two numbers as the size of the instance. In general the size represents the data size which is crucial in deciding the time it takes perform the computation. Note: In our discussion size will always be an integer.

3
3.1

Complexity
Time Complexity

We can determine the time it takes to execute a program in a given computer. But there are variety of computers and each type of machine has dierent capability. The execution time on each machine is likely to be dierent. Thus how to dene the time of a program or algorithm independent of an underlying machine? Each machine has a datapath which performs basic manipulation of the data. The datapath has some things called functional units which perform arithmatic and logic computations. It is possible that some machines may have complex functional units and some may only do simple functions. To overcome this problem arising out of diverse machines we use the following criterion to measure the time to execute a program/algorithm. We assume that each basic arithmatic or logic operation on a data items of one word size takes an unknown but a xed amount of time. Suppose the algorithm performs 3 additions and 2 multiplications then we say that it takes 3a + 2m time. Let c be the maximum of a and m, then we say that 5c is the upperbound for the time taken by this algorithm. We denote it by writing that the time complexity of this algorithm is O(5c). We futher simplify it and only write it as O(5). It simply means that the time it takes is at most 5c for some positive constant c. If n is a size parameter for the problem then the time complexity will be states as O(f (n)) which means that the program takes at most c.f (n) time for some constant c. Essentailly f () measures the number of operations. If we want to state a lowerbound for the time a program will take, then it is denoted by (g(n)) which means that for some constant positive c the program will take at least c.g(n) time. If we know that there exists positive constants c1 and c2 and a function f () such that the time is always going to be between c1.f (n) and c2.f (n), then we state it by notation (f (n)).

3.2

Space Complexity

Once again the diversity in the word sizes of machines makes it dicult to give a machine independent measure of the memory it takes to execute a program. In this case also we use the same approach as in the time complexity.

Time/Space for Instances of the Same Size

In general there can be many instances of a problem of the same size. Also the time and space requirements for these instances is usually not same. Consider the required time (space) as a function of two parameters: size and the instances of the same size. The surface dened by this function is normally not smooth in the problems of discrete nature. Often it is hard to come up with a mathematical expression for such surfaces. For example consider usual long-hand algorithm for GCD(n, m) and Squarer oot(n). Compare the times of 2

GCD(4, 16) and GCD(4, 15). Also compare the times of Squarer oot(15) with that of Squarer oot(16). Generally it may not be possible to determine this function in its full glory nor it is our objective. Our goal is to nd how does this function behave asyptotically.

4.1

Worst Case And Average Case

We want to getrid of the second parameter of the function. Teher are two ways to do it: worst case analysis and the average case analysis. Here we take average of the time/space of each instance usually with equal probability distribution for each instance. Example: Search a number N in an array of n number. Observe that it may take any thing between 1 comparison to n comparisons, depending on the position of the target number.

Asymptotic Complexity

After removing the second parameter, we have only size as the parameter. We intend to bound this curve by a smooth and simple function. Usually we like to bound by a polynomial or exponential function. If these functions do not bound tightly, then we try to nd a suitable function which should be a tight t asymptotically, even if it does not give a good bound for the smaller sizes. Hence the time complexity T (n) is O(f (n)) means that there exists a constant c and an integer n0 such that T (n) <= c.f (n) for all n >= n0 . Similarly for and T heta.

5.1

Alternative Interpretation of O, ,

It is possible that the function T (n) is extremely bad behaves, having large and frequent spikes. In this case we want a function f (n) such that T (n) <= c.f (n) innitely often. Similarly for Omega and Theta. Note: We will not adopt this denition since such T (n) are rare and we will not come accross any such case.

Where Time Complexity is Not a Deciding Factor

3. Too complex algorithms may not be desirable as one needs to maintain. 4. In numerical algorithms accuracy and stability are very important.

Analysis of Some Simple Algorithms and Introduction to Pseudo Language

(a) Bubble Sort: Left as an exercise. (b) Factorial: recursive and non-recursive: Left as an exercise. (c) GCD(a0 .a1 ): GCD(a0 , a1 ) /* where a0 a1 */ x := a0 ; y := a1 ; while (y > 0) do temp := y; y := remainder(xdivy); x := temp; return x; The algorithm computes as follows, where qi denote the respective quotients. a2 = a0 q1 a1 a3 = a1 q2 a2 ... an = an2 qn2 an1 an+1 = an1 qn1 an = 0 This reduction takes n steps to compute the gcd = an . So an1 = qn1 an an2 = qn2 an1 + an = qn1 qn2 an an3 = (qn1 qn2 + 1)an an4 = (qn1 qn2 qn3 + qn3 + qn1 qn2 )an ... a0 = .... For each value of an and qi s gives an a0 which requires n reduction steps. We can get the minimum value of a0 if all qi and an are taken to be 1. Let the values computed under these assumptions be denoted by bj . Then bn1 = 1 = f0 bn2 = 1 = f1 bn3 = f0 + f1 = 2 = f2 bn4 = f1 + f2 = f3 ... b0 = fn1 Since we know that a0 b0 = fn1 , we have the following result. Lemma 1: If GCD(a0 , a1 ) requires n steps, then a0 fn1 . Following corollary is an equivalent statement. 4

Corollary 1: If a0 < fn1 , then GCD(a0 , a1 ) requires less than n steps, where a1 is any integer less than or equal to a0 . Let x be any integer and x denote the nearest Fibonacci number to x which is greater than or equal to x. Let F denote the Fibonacci function, i.e., F (n) = fn .SoF 1 (fn ) = n. We restate the above result as follows. Corollary 2: GCD(x, y) requires at most F 1 (x ) + 1 steps, where x y. Theorem 1A: The GCD(x, y) algorithm given above has time complexity O(F 1 (x ) + 1). As mentioned in the class, we prefer complexity in terms of polynomials, exponential etc functions. Next we will nd the bound in one such function. To nd a tight bound for the function F 1 , revisit the Fibonacci series. f0 = 1, f1 = 1, f2 = 2, f3 = 3, f4 = 5, f5 = 8, f6 = 13, . . . we can see that for all i > 0, fi+1 /fi is bounded between 1.5and 2 in the rst few steps. We will now show that 1.5 fi+1 /fi 2 for all i > 0. To prove by induction we observe that it holds for i=1 so base case is taken care of. Now for the induction case assume that it holds for 1 i k 1. So fk+1 /fk = fk1 /fk + 1. Since 1/2 fk1 /fk 2/3, 1/2 + 1 fk1 /fk + 1 2/3 + 1. Hence 1.5 fk+1 /fk 1.6666 < 2. Thus the claim is proven. We have fn = f1 (f2 /f1 ) ... (fn /fn1 ). So using the bounds we get (1.5)n1 fn 2n1 . So log2 (fn ) n 1 log1.5 (fn ). Hence log2 (fn ) + 1 F 1 (fn ) <= log1.5 (fn ) + 1. Thus we have the nal result. Here we use the fact that since every successive Fibonacci number is less than twice the previous Fibonacci number, x < 2x. Theorem 1B: The GCD(x, y) algorithm given above has time complexity O(log1.5 (2x)). We prefer to use log base 2 for the reason described below. Since log1.5 (z) = log2 (z)log1.5 (2) and since log1.5 (2) is just a constant, we have the third version of the theorem. Theorem 1C: The GCD(x, y) algorithm given above has time complexity O(log2 (2x)) or O(log2 (x) + 1) or O(log2 (x)). Finally a remark about this version of the theorm. Recall that we wish to capture time complexity in terms of the size of the problem. So what is the size of the problem? Since the input is a pair of integers, the size may be taken to be the number of bits in the larger of the two numbers. And that is 1 + log2 (x). So theorem 1C states that the time complexity of the algorithm is linear in the size of the input (number of bits).