Data Structure- 301

Data Structures and Programming

Lecture 1
Steven S. Skiena

Why Data Structures?

In my opinion, there are only three important ideas which must be mastered to

write interesting programs.

 Iteration - Do, While, Repeat, If

 Data Representation - variables and pointers
 Subprograms and Recursion - modular design and abstraction

At this point, I expect that you have mastered about 1.5 of these 3.

It is the purpose of Computer Science II to finish the job.

Data types vs. Data Structures

A data type is a well-defined collection of data with a well-defined set of operations

on it.

A data structure is an actual implementation of a particular abstract data type.

Example: The abstract data type Set has the operations EmptySet(S), Insert(x,S),
Delete(x,S), Intersection(S1,S2), Union(S1,S2), MemberQ(x,S), EqualQ(S1,S2),
SubsetQ(S1,S2).

This semester, we will learn to implement such abstract data types by building data
structures from arrays, linked lists, etc.

Modula-3 Programming

Control Structures: IF-THEN-ELSE, CASE-OF

Iteration Constructs: REPEAT-UNTIL (at least once), WHILE-DO (at least 0), FOR-
DO (exactly n times).

Elementary Data Types: INTEGER, REAL, BOOLEAN, CHAR

Enumerated Types: COINSIDE = {HEADS, TAIL, SIDE}

Operations: +, -, <, >, #

Elementary Data Structures

ArraysThese let you access lots of data fast. (good)

You can have arrays of any other data type. (good)

However, you cannot make arrays bigger if your program decides it needs more
space. (bad)

RecordsThese let you organize non-homogeneous data into logical packages to keep
everything together. (good)

These packages do not include operations, just data fields (bad, which is why we need
objects)

Records do not help you process distinct items in loops (bad, which is why arrays of
records are used)

SetsThese let you represent subsets of a set with such operations as intersection,
union, and equivalence. (good)

Built-in sets are limited to a certain small size. (bad, but we can build our own set
data type out of arrays to solve this problem if necessary)

Subroutines

Subprograms allow us to break programs into units of reasonable size and complexity,
allowing us to organize and manage even very long programs.

This semester, you will first encounter programs big enough that modularization will
be necessary for survival.

Functions are subroutines which return values, instead of communicating by

parameters.
Abstract data types have each operation defined by a subroutine.

Subroutines which call themselves are recursive. Recursion provides a very powerful

way to solve problems which takes some getting used to.

Such standard data structures as linked lists and trees are inherently recursive data
structures.

Parameter Passing

There are two mechanisms for passing data to a subprogram, depending upon whether
the subprogram has the power to alter the data it is given.

In pass by value, a copy of the data is passed to the subroutine, so that no matter what
happens to the copy the original is unaffected.

In pass by reference, the variable argument is renamed, not copied. Thus any changes
within the subroutine effect the original data. These are the VAR parameters.

Example: suppose the subroutine is declared Push(VAR s:stack, e:integer) and called

with Push(t,x). Any changes with Push to e have no effect on x, but changes
to s effect t.

Generic Modula-3 Program I

MODULE Prim EXPORTS Main;
(* Prime number testing with Repeat *)

IMPORT SIO;

VAR candidate, i: INTEGER;

BEGIN
SIO.PutText("Prime number test\n");
REPEAT
SIO.PutText("Please enter a positive number; enter 0 to quit. ");
candidate:= SIO.GetInt();
IF candidate > 2 THEN
i:= 1;
REPEAT
i:= i + 1
UNTIL ((candidate MOD i) = 0) OR (i * i > candidate);
IF (candidate MOD i) = 0 THEN
SIO.PutText("Not a prime number\n")
ELSE
SIO.PutText("Prime number\n")
END; (*IF (candidate MOD i) = 0 ...*)
ELSIF candidate > 0 THEN
 SIO.PutText("Prime number\n") (*1 and 2 are prime*)
END; (*IF candidate > 2*)
UNTIL candidate <= 0;
END Prim.

Generic Modula-3 Program II

MODULE Euclid2 EXPORTS Main; (*17.05.94. LB*)
(* The Euclidean algorithm (with controlled input):
Compute the greatest common divisor (GCD) *)

IMPORT SIO;

VAR
a, b: INTEGER; (* input values *)
x, y: CARDINAL; (* working variables *)

<*FATAL SIO.Error*>

BEGIN (*statement part*)

SIO.PutText("Euclidean algorithm\nEnter 2 positive numbers: ");

a:= SIO.GetInt();
WHILE a <= 0 DO
SIO.PutText("Please enter a positive number: ");
a:= SIO.GetInt();
END; (*WHILE a < 0*)

b:= SIO.GetInt();
WHILE b <= 0 DO
SIO.PutText("Please enter a positive number: ");
b:= SIO.GetInt();
END; (*WHILE b < 0*)

x:= a; y:= b; (*x and y can be changed by the algorithm*)

WHILE x # y DO
IF x > y THEN x:= x - y ELSE y:= y - x END;
END; (*WHILE x # y*)

SIO.PutText("Greatest common divisor = ");

SIO.PutInt(x);
SIO.Nl();
END Euclid2.

Programming Proverbs

KISS - ``Keep it simple, stupid.'' - Don't use fancy features when simple ones suffice.

RTFM - ``Read the fascinating manual.'' - Most complaints from the compiler can be
solved by reading the book. Logical errors are something else.

Make your documentation short but sweet. - Always document your variable
declarations, and tell what each subprogram does.
Every subprogram should do something and hide something - If you cannot concisely
explain what your subprogram does, it shouldn't exist. This is why I write the header
comments before I write the subroutine.

Program defensively - Add the debugging statements and routines at the beging,
because you know you are going to need them later.

A good program is a pretty program. - Remember that you will spend more time
reading your programs than we will.

Perfect Shuffles
1 1 1 1 1 1 1 1 1
2 27 14 33 17 9 5 3 2
3 2 27 14 33 17 9 5 3
4 28 40 46 49 25 13 7 4
5 3 2 27 14 33 17 9 5
6 29 15 8 30 41 21 11 6
7 4 28 40 46 49 25 13 7
8 30 41 21 11 6 29 15 8
9 5 3 2 27 14 33 17 9
10 31 16 34 43 22 37 19 10
11 6 29 15 8 30 41 21 11
12 32 42 47 24 38 45 23 12
13 7 4 28 40 46 49 25 13
14 33 17 9 5 3 2 27 14
15 8 30 41 21 11 6 29 15
16 34 43 22 37 19 10 31 16
17 9 5 3 2 27 14 33 17
18 35 18 35 18 35 18 35 18
19 10 31 16 34 43 22 37 19
20 36 44 48 50 51 26 39 20
21 11 6 29 15 8 30 41 21
22 37 19 10 31 16 34 43 22
23 12 32 42 47 24 38 45 23
24 38 45 23 12 32 42 47 24
25 13 7 4 28 40 46 49 25
26 39 20 36 44 48 50 51 26
27 14 33 17 9 5 3 2 27
28 40 46 49 25 13 7 4 28
29 15 8 30 41 21 11 6 29
30 41 21 11 6 29 15 8 30
31 16 34 43 22 37 19 10 31
32 42 47 24 38 45 23 12 32
33 17 9 5 3 2 27 14 33
34 43 22 37 19 10 31 16 34
35 18 35 18 35 18 35 18 35
36 44 48 50 51 26 39 20 36
37 19 10 31 16 34 43 22 37
38 45 23 12 32 42 47 24 38
39 20 36 44 48 50 51 26 39
40 46 49 25 13 7 4 28 40
41 21 11 6 29 15 8 30 41
 42 47 24 38 45 23 12 32 42
43 22 37 19 10 31 16 34 43
44 48 50 51 26 39 20 36 44
45 23 12 32 42 47 24 38 45
46 49 25 13 7 4 28 40 46
47 24 38 45 23 12 32 42 47
48 50 51 26 39 20 36 44 48
49 25 13 7 4 28 40 46 49
50 51 26 39 20 36 44 48 50
51 26 39 20 36 44 48 50 51
52 52 52 52 52 52 52 52 52

Software Engineering and Top-Down Design

Lecture 2
Steven S. Skiena

Software Engineering and Saddam Hussain

Think about the Patriot missiles which tried to shoot down SCUD missiles in the
Persian Gulf war and think about how difficult it is to produce working software!

1. How do you test a missile defense system?

2. How do you satisfy such tight constraints as program speed, computer
size/weight, flexibility to recognize different types of missiles?
3. How do you get hundreds of people to work on the same program without
getting chaos?

Even today, there is great controversy about how well the missiles actually did in the
war.

Testing and Verification

How do you know that your program works? Not by testing it!

``Testing reveals the presence, but not the absence of bugs.'' - Dijkstra

Still, it is important to design test cases which exercise the boundary conditions of the
program.

Example: Linked list insertion. The boundary cases include:

 insertion before the first element.

  insertion after the last element.
 insertion into the empty list.
 insertion between element (the general case).

Test Case Generation

In the Microsoft Excel group, there is one tester for each programmer! Types of test
cases include:

Boundary cases - Make sure that each line of code and branch of IF is executed at
least once.

Random data - Automatically generated test data can be useful to test user patterns
you might otherwise not consider, but you must verify that the results are correct!

Other users - People who were not involved in writing the program will have vastly
different ideas of how to use it.

Adversaries - People who want to attack your program and have access to the source
for often find bugs by reading it.

Verification

But how can we know that our program works? The ideal way is to mathematically
prove it.

For each subprogram, there is a precise set of preconditions which we assume is

satisfied by the input parameters, and a precise set of post-conditions which are
satisfied by the output parameters.

If we can show that any input satisfying the preconditions is always transformed to
output satisfying the post conditions, we have proven the subprogram correct.

Top-Down Refinement

To correctly build a complicated system requires first setting broad goals and refining
them over time. Advantages include:

A hierarchy hides information - This permits you to focus attention on only a

manageable amount of detail.
With the interfaces defined by the hierarchy, changes can be made without effecting
the rest of the structure - Thus systems can be maintained without being ground to a
halt.

Progress can be made in parallel by having different people work on different

subsections - Thus you can organize to build large systems.

Stepwise Refinement in Programming

The best way to build complicated programs is to construct the hierarchy one level at
a time, finally writing the actual functions when the task is small enough to be easily
done.

 Build a prototype to throw away, because you will, anyway.

 Anything difficult put off for last. If necessary, decompose it into another level
of detail.

Most of software engineering is just common sense, but it is very easy to ignore
common sense.

Building a Military Threat

Module Build-Military: The first decision is now to organize it, not what type of tank
to buy.

Several different organizations are possible, and in planning we should investigate

each one:

 Offense, Defense
 Army, Navy, Air Force, Marines, Coast Guard ...

Procedure Army: Tanks, Troops, Guns ...

Procedure Troops: Training, Recruitment, Supplies

Top-Down Design Example

``Teaching Software Engineering is like telling children to brush their teeth.'' -

anonymous professor.

To make this more concrete, lets outline how a non-trivial program should be
structured.
Suppose that you wanted to write a program to enable a person to play the game
Battleship against a computer.

Tell me what to do!

What is Battleship?

Each side places 5 ships on a   grid, and then takes turns guessing grid points
until one side has covered all the ships:

For each query, the answer ``hit'', ``miss'', or ``you sunk my battleship'' must be given.

There are two distinct views of the world, one reflecting the truth about the board, the
other reflecting what your opponent knows.

Program: Battleship

Interesting subproblems are: display board, generate query, respond to query, generate
initial configuration, move-loop (main routine).

What data structure should we use? Two-dimensional arrays.

How do I enforce separation between my view and your view?

Data Structures and Programming

Lecture 3
Steven S. Skiena

Stacks and Queues

The first data structures we will study this semester will be lists which have the
property that the order in which the items are used is determined by the order they
arrive.

 Stacks are data structures which maintain the order of last-in, first-out

 Queues are data structures which maintain the order of first-in, first-out
Queues might seem fairer, which is why lines at stores are organized as queues
instead of stacks, but both have important applications in programs as a data structure.

Operations on Stacks

The terminology associated with stacks comes from the spring loaded plate containers
common in dining halls.

When a new plate is washed it is pushed on the stack.

When someone is hungry, a clean plate is popped off the stack.

A stack is an appropriate data structure for this task since the plates don't care about
when they are used!

Maintaining Procedure Calls

Stacks are used to maintain the return points when Modula-3 procedures call other
procedures which call other procedures ...

Jacob and Esau

In the biblical story, Jacob and Esau were twin brothers where Esau was born first and
thus inherited Issac's birthright. However, Jacob got Esau to give it away for a bowl of
soup, and so Jacob went to become a patriarch of Israel.

But why was Jacob justified in so tricking his brother???

Rashi, a famous 11th century Jewish commentator, explained the problem by saying
Jacob was conceived first, then Esau second, and Jacob could not get around the
narrow tube to assume his rightful place first in line!

 Therefore Rebecca was modeled by a stack.

 ``Push'' Issac, Push ``Jacob'', Push ``Esau'', Pop ``Esau'', Pop ``Jacob''

Abstract Operations on a Stack

 Push(x,s) and Pop(x,s) - Stack s, item x. Note that there is no search operation.
 Initialize(s), Full(s), Empty(s), - The latter two are Boolean queries.

Defining these abstract operations lets us build a stack module to use and reuse
without knowing the details of the implementation.
The easiest implementation uses an array with an index variable to represent the top of
the stack.

An alternative implementation, using linked lists is sometimes better, for it can't ever
overflow. Note that we can change the implementations without the rest of the
program knowing!

Declarations for a stack

INTERFACE Stack; (*14.07.94 RM, LB*)
(* Stack of integer elements *)

TYPE ET = INTEGER; (*element type*)

PROCEDURE Push(elem : ET); (*adds element to top of stack*)

PROCEDURE Pop(): ET; (*removes and returns top element*)
PROCEDURE Empty(): BOOLEAN; (*returns true if stack is empty*)
PROCEDURE Full(): BOOLEAN; (*returns true if stack is full*)

END Stack.

Stack Implementation
MODULE Stack; (*14.07.94 RM, LB*)
(* Implementation of an integer stack *)

CONST
Max = 8; (*maximum number of elements on stack*)

TYPE
S = RECORD
info: ARRAY [1 .. Max] OF ET;
top: CARDINAL := 0; (*initialize stack to empty*)
END; (*S*)

VAR stack: S; (*instance of stack*)

PROCEDURE Push(elem:ET) =
(*adds element to top of stack*)
BEGIN
INC(stack.top); stack.info[stack.top]:= elem
END Push;

PROCEDURE Pop(): ET =
(*removes and returns top element*)
BEGIN
DEC(stack.top); RETURN stack.info[stack.top + 1]
END Pop;

PROCEDURE Empty(): BOOLEAN =

(*returns true if stack is empty*)
BEGIN
RETURN stack.top = 0
 END Empty;

PROCEDURE Full(): BOOLEAN = (*returns true if stack is full*)

BEGIN
RETURN stack.top = Max
END Full;

BEGIN
END Stack.

Using the Stack Type

MODULE StackUser EXPORTS Main; (*14.02.95. LB*)
(* Example client of the integer stack *)

FROM Stack IMPORT Push, Pop, Empty, Full;

FROM SIO IMPORT Error, GetInt, PutInt, PutText, Nl;
<* FATAL Error *> (*suppress warning*)

BEGIN
PutText("Stack User. Please enter numbers:\n");
WHILE NOT Full() DO
Push(GetInt()) (*add entered number to stack*)
END;
WHILE NOT Empty() DO
PutInt(Pop()) (*remove number from stack and return it*)
END;
Nl();
END StackUser.

FIFO Queues

Queues are more difficult to implement than stacks, because action happens at both
ends.

The easiest implementation uses an array, adds elements at one end, and moves all

elements when something is taken off the queue.

It is very wasteful moving all the elements on each DEQUEUE. Can we do better?

More Efficient Queues

Suppose that we maintaining pointers to the first (head) and last (tail) elements in the
array/queue?

Note that there is no reason to explicitly clear previously unused cells.

Now both ENQUEUE and DEQUEUE are fast, but they are wasteful of space. We
need a array bigger than the total number of ENQUEUEs, instead of the maximum
number of items stored at a particular time.

Circular Queues

Circular queues let us reuse empty space!

Note that the pointer to the front of the list is now behind the back pointer!

When the queue is full, the two pointers point to neighboring elements.

There are lots of possible ways to adjust the pointers for circular queues. All are
tricky!

How do you distinguish full from empty queues, since their pointer positions might be
identical? The easiest way to distinguish full from empty is with a counter of how
many elements are in the queue.

FIFO Queue Interface

INTERFACE Fifo; (*14.07.94 RM, LB*)
(* A queue of text elements *)

TYPE ET = TEXT; (*element type*)

PROCEDURE Enqueue(elem:ET); (*adds element to end*)

PROCEDURE Dequeue(): ET; (*removes and returns first element*)
PROCEDURE Empty(): BOOLEAN; (*returns true if queue is empty*)
PROCEDURE Full(): BOOLEAN; (*returns true if queue is full*)

END Fifo.

Priority Queue Implementation

MODULE Fifo; (*14.07.94 RM, LB*)
(* Implementation of a fifo queue of text elements *)

CONST
Max = 8; (*Maximum number of elements in FIFO queue*)

TYPE
Fifo = RECORD
info: ARRAY [0 .. Max - 1] OF ET;
in, out, n: CARDINAL := 0;
END; (*Fifo*)

VAR w: Fifo; (*contains a FIFO queue*)

 PROCEDURE Enqueue(elem:ET) =
(*adds element to end*)
BEGIN
w.info[w.in]:= elem; (*stores new element*)
w.in:= (w.in + 1) MOD Max; (*increments in-pointer in ring*)
INC(w.n); (*increments number of stored elements*)
END Enqueue;

PROCEDURE Dequeue(): ET =
(*removes and returns first element*)
VAR e: ET;
BEGIN
e:= w.info[w.out]; (*removes oldest element*)
w.out:= (w.out + 1) MOD Max; (*increments out-pointer in ring*)
DEC(w.n); (*decrements number of stored elements*)
RETURN e; (*returns the read element*)
END Dequeue;

Utility Routines

PROCEDURE Empty(): BOOLEAN =

(*returns true if queue is empty*)
BEGIN
RETURN w.n = 0;
END Empty;

PROCEDURE Full(): BOOLEAN =

(*returns true if queue is full*)
BEGIN
RETURN w.n = Max
END Full;

BEGIN
END Fifo.

User Module
MODULE FifoUser EXPORTS Main; (*14.07.94. LB*)
(* Example client of the text queue. *)

FROM Fifo IMPORT Enqueue, Dequeue, Empty, Full; (* operations of the queue
*)
FROM SIO IMPORT Error, GetText, PutText, Nl;
<* FATAL Error *> (*supress warning*)

BEGIN
PutText("FIFO User. Please enter texts:\n");
WHILE NOT Full() DO
Enqueue(GetText())
END;
WHILE NOT Empty() DO
PutText(Dequeue() & " ")
 END;
Nl();
END FifoUser.

Other Queues

Double-ended queues - These are data structures which support both push and pop
and enqueue/dequeue operations.

Priority Queues(heaps) - Supports insertions and ``remove minimum'' operations

which useful in simulations to maintain a queue of time events.

We will discuss simulations in a future class.

Pointers and Dynamic Memory Allocation

Lecture 4
Steven S. Skiena

Pointers and Dynamic Memory Allocation

Although arrays are good things, we cannot adjust the size of them in the middle of
the program.

If our array is too small - our program will fail for large data.

If our array is too big - we waste a lot of space, again restricting what we can do.

The right solution is to build the data structure from small pieces, and add a new piece
whenever we need to make it larger.

Pointers are the connections which hold these pieces together!

Pointers in Real Life

In many ways, telephone numbers serve as pointers in today's society.

 To contact someone, you do not have to carry them with you at all times. All
you need is their number.
  Many different people can all have your number simultaneously. All you need
do is copy the pointer.
 More complicated structures can be built by combining pointers. For example,
phone trees or directory information.

Addresses are a more physically correct analogy for pointers, since they really are
memory addresses.

Linked Data Structures

All the dynamic data structures we will build have certain shared properties.

 We need a pointer to the entire object so we can find it. Note that this is a
pointer, not a cell.
 Each cell contains one or more data fields, which is what we want to store.
 Each cell contains a pointer field to at least one ``next'' cell. Thus much of the
space used in linked data structures is not data!
 We must be able to detect the end of the data structure. This is why we need the
NIL pointer.

Pointers in Modula-3

A node in a linked list can be declared:

type
pointer = REF node;
node = record
info : item;
next : pointer;
end;

var
p,q,r : pointer; (* pointers *)
x,y,z : node; (* records *)

Note circular definition. Modula-3 lets you get away with this because it is a reference
type. Pointers are the same size regardless of what they point to!

We want dynamic data structures, where we make nodes as we need them. Thus

declaring nodes as variables are not the way to go!

Dynamic Allocation

To get dynamic allocation, use new:

 p := New(ptype);

New(ptype) allocates enough space to store exactly one object of the type ptype.
Further, it returns a pointer to this empty cell.

Before a new or otherwise explicit initialization, a pointer variable has an arbitrary

value which points to trouble!

Warning - initialize all pointers before use. Since you cannot initialize them to explicit
constants, your only choices are

 NIL - meaning explicitly nothing.

 New(ptype) - a fresh chunk of memory.
 assignment to some previously initialized pointer of the same type.

Pointer Examples

Example: p := new(node); q := new(node);

p.x grants access to the field x of the record pointed to by p.

p^.info := "music";
q^.next := nil;

The pointer value itself may be copied, which does not change any of the other fields.

Note this difference between assigning pointers and what they point to.
p := q;

We get a real mess. We have completely lost access to music and can't get it back!
Pointers are unidirectional.

Alternatively, we could copy the object being pointed to instead of the pointer itself.
p^ := q^;

What happens in each case if we now did:

p^.info := "data structures";

Where Does the Space Come From?

Can we really get as much memory as we want without limit just by using New?
No, because there are the physical limits imposed by the size of the memory of the
computer we are using. Usually Modula-3 systems let the dynamic memory come
from the ``other side'' of the ``activation record stack'' used to maintain procedure
calls:

Just as the stack reuses memory when a procedure exits, dynamic storage must be
recycled when we don't need it anymore.

Garbage Collection

The Modula-3 system is constantly keeping watch on the dynamic memory which it
has allocated, making sure that something is still pointing to it. If not, there is no way
for you to get access to it, so the space might as well be recycled.

The garbage collector automatically frees up the memory which has nothing pointing

to it.

It frees you from having to worry about explicitly freeing memory, at the cost of
leaving certain structures which it can't figure out are really garbage, such as a circular
list.

Explicit Deallocation

Although certain languages like Modula-3 and Java support garbage collection, others
like C++ require you to explicitly deallocate memory when you don't need it.

Dispose(p) is the opposite of New - it takes the object which is pointed to by p and
makes it available for reuse.

Note that each dispose takes care of only one cell in a list. To dispose of an entire
linked structure we must do it one cell as a time.

Note we can get into trouble with dispose:

Of course, it is too late to dispose of music, so it will endure forever without garbage
collection.

Suppose we dispose(p), and later allocation more dynamic memory with new. The cell
we disposed of might be reused. Now what does q point to?
Answer - the same location, but it means something else! So called dangling
references are a horrible error, and are the main reason why Modula-3 supports
garbage collection.

A dangling reference is like a friend left with your old phone number after you move.
Reach out and touch someone - eliminate dangling references!

Security in Java

It is possible to explicitly dispose of memory in Modula-3 when it is really necessary,

but it is strongly discouraged.

Java does not allow one to do such operations on pointers at all. The reason
is security.

Pointers allow you access to raw memory locations. In the hands of skilled but evil
people, unchecked access to pointers permits you to modify the operating system's or
other people's memory contents.

Java is a language whose programs are supposed to be transferred across the Internet
to run on your computer. Would you allow a stranger's program to run on your
machine if they could ruin your files?

Linked Stacks and Queues

Lecture 5
Steven S. Skiena

Pointers about Pointers

var p, q : ^node;

p = new(node) creates a new node and sets p to point to it.

p   describes the node which is pointed to by p.

p   .item describes the item field of the node pointed to by p.

dispose(p) returns to the system the memory used by the node pointed to by p. This is
not used because of Modula-3 garbage collection.

NIL is the only value a pointer can have which is not an address.

Linked Stacks

The problem with array-based stacks are that the size must be determined at compile
time. Instead, let's use a linked list, with the stack pointer pointing to the top element.

To push a new element on the stack, we must do:

p^.next = top;
top = p;

Note this works even for the first push if top is initialized to NIL!

Popping from a Linked Stack

To pop an item from a linked stack, we just have to reverse the operation.
p = top;
top = top^.next;
p^.next = NIL; (*avoid dangling reference*)

Note again that this works in the boundary case of one item on the stack.

Note that to check we don't pop from an empty stack, we must test
whether top = NIL before using top as a pointer. Otherwise things crash or
segmentation fault.

Linked Stack in Modula-3

MODULE Stacks; (*14.07.94 RM, LB*)
(* Implementation of the abstract, generic stack. *)
REVEAL
T = BRANDED REF RECORD
info: ET; next: T;
END; (*T*)

PROCEDURE Create(): T = (*creates and intializes a new stack*)

BEGIN
RETURN NIL; (* a new, empty stack is simply NIL *)
END Create;

PROCEDURE Push(VAR stack: T; elem:ET) =

(*adds element to stack*)
VAR new: T := NEW(T, info:= elem, next:= stack); (*create element*)
 BEGIN
stack:= new (*add element at top*)
END Push;

PROCEDURE Pop(VAR stack: T): ET =

(*removes and returns top element, or NIL for empty stack*)
VAR first: ET := NIL; (* Pop returns NIL for empty stack*)
BEGIN
IF stack # NIL THEN
first:= stack.info; (*copy info from first element*)
stack:= stack.next; (*remove first element*)
END; (*IF stack # NIL*)
RETURN first;
END Pop;

PROCEDURE Empty(stack: T): BOOLEAN =

(*returns TRUE for empty stack*)
BEGIN
RETURN stack = NIL
END Empty;

BEGIN
END Stacks.

Generic Stack Interface

INTERFACE Stacks; (*14.07.94 RM, LB*)
(* Abstract generic stack. *)

TYPE
T <: REFANY; (*type of stack*)
ET = REFANY; (*type of elements*)

PROCEDURE Create(): T; (*creates and intializes a new

stack*)

PROCEDURE Push(VAR stack: T; elem: ET); (*adds element to stack*)

PROCEDURE Pop(VAR stack: T): ET; (*removes and returns top element,
or
NIL for empty stack*)
PROCEDURE Empty(stack: T): BOOLEAN; (*returns TRUE for empty stack*)

END Stacks.

Generic Stacks Client

MODULE StacksClient EXPORTS Main; (* LB *)
(* Example client of both the generic stack and the type FractionType.
This program builds up a stack of fraction numbers as well as of
complex numbers.
*)

IMPORT Stacks;
IMPORT FractionType;
FROM Stacks IMPORT Push, Pop, Empty;
 FROM SIO IMPORT PutInt, PutText, Nl, PutReal, PutChar;

TYPE
Complex = REF RECORD r, i: REAL END;

VAR
stackFraction: Stacks.T:= Stacks.Create();
stackComplex : Stacks.T:= Stacks.Create();

c: Complex;
f: FractionType.T;

BEGIN (*StacksClient*)
PutText("Stacks Client\n");

FOR i:= 1 TO 4 DO
Push(stackFraction, FractionType.Create(1, i)); (*stores numbers 1/i*)
END;

FOR i:= 1 TO 4 DO
Push(stackComplex, NEW(Complex, r:= FLOAT(i), i:= 1.5 * FLOAT(i)));
END;

WHILE NOT Empty(stackFraction) DO

f:= Pop(stackFraction);
PutInt(FractionType.Numerator(f));
PutText("/");
PutInt(FractionType.Denominator(f), 1);
END;
Nl();

WHILE NOT Empty(stackComplex) DO

c:= Pop(stackComplex);
PutReal(c.r);
PutChar(':');
PutReal(c.i);
PutText(" ");
END;
Nl();

END StacksClient.

Linked Queues

Queues in arrays were ugly because we need wrap around for circular queues. Linked
lists make it easier.

We need two pointers to represent our queue - one to the rear for enqueue operations,

and one to the front for dequeue operations.

Note that because both operations move forward through the list, no back pointers are
necessary!
Enqueue and Dequeue

To enqueue an item   :
p^.next := NIL;
if (back = NIL) then begin (* empty queue *)
front := p; back := p;
end else begin (* non-empty queue *)
back^.next := p;
back := p;
end;

To dequeue an item:
p := front;
front := front^.next;
p^.next := NIL;
if (front = NIL) then back := NIL; (* now-empty queue *)

Building the Calculator

Lecture 6
Steven S. Skiena

Reverse Polish Notation

HP Calculators use reverse Polish notation or postfix notation. Instead of the

conventional a + b, we write A B +.

Our calculator will do the same. Why? Because it is the easiest notation to implement!

The rule for conversion is to read the expression from left to right. When we see a
number, push it on the operation stack. When we see an operation, pop the last two
numbers on stack, do the operation, and push the result on the stack.

Look Ma, no parentheses!

Algorithms for the calculator

To implement addition, we add digits from right to left, with the carry one place if the
sum is greater than 10.

Note that the last carry can go beyond one or both numbers, so you must handle this
special case.

To implement subtraction, we work on digits from right to left, and borrow 10 if

necessary from the digit to the left.

A borrow from the leftmost digit is complicated, since that gives a negative number.

This is why I suggest completing addition first before worrying about subtraction.

I recommend to test which number has a larger absolute value, subtract from that, and
then adjust the sign accordingly.

Parsing the Input

There are several possible ways to handle the problem of reading in the input line
and parsing it, i.e. breaking it into its elementary components of numbers and
operators.

The way that seems best to me is to read the entire line as one character string in a
variable of type TEXT.

As detailed in your book, you can use the function Text.Length(S) to get the length of
this string, and the function Text.GetChar(S,i) to retreive any given character.

Useful functions on characters include the function ORD(c), which returns the integer
character code of c. Thus ORD(c) - ORD('0') returns the numerical value of a digit
character.

You can test characters for equality to identify special symbols.

Standard I/O

The easiest way to read and write from the files is to use I/O redirection from UNIX.

Suppose calc is your binary program, and it expects input from the keyboard and
output to the screen. By running calc < filein at the command prompt, it will take its
input from the file filein instead of the keyboard.
Thus by writing your program to read from regular I/O, you can debug it interactively
and also run my test files.

Programming Hints

1. Write the comments first, for your sake.

2. Make sure your main routine is abstract enough that you can easily see what the
program does.
3. Isolate the details of your data structures to a few abstract operations.
4. Build good debug print routines first.

List Insertion and Deletion

Lecture 7
Steven S. Skiena

Search, Insert, Delete

There are three fundamental operations we need for any database:

 Insert: add a new record at a given point

 Delete: remove an old record
 Search: find a record with a given key

We will see a wide variety of different implementation of these operations over the
course of the semester.

How would you implement these using an array?

With linked lists, we can creating arbitrarily large structures, and never have to move
any items.

Most of these operations should be pretty simple now that you understand pointers!

Searching in a Linked List

Procedure Search(head:pointer, key:item):pointer;
Var
p:pointer;
found:boolean;
Begin
found:=false;
p:=head;
 While (p # NIL) AND (not found) Do
Begin
If (p^.info = key) then
found = true;
Else
p = p^.next;
End;
return p;
END;

Search performs better when the item is near the front of the list than the back.

What happens when the item isn't found?

Insertion into a Linked List

The easiest way to insert a new node p into a linked list is to insert it at the front of the
list:
p^.next = front;
front = p;

To maintain lists in sorted order, however, we will want to insert a node between the
two appropriate nodes. This means that as we traverse the list we must keep pointers
to both the current node and the previous node.
MODULE Intlist; (*16.07.94. RM, LB*)
(* Implementation of sorted integer lists. *)

REVEAL (*reveal inner structure of T*)

T = BRANDED REF RECORD
key: INTEGER; (*key value*)
next: T := NIL; (*pointer to next element*)
END; (*T*)

PROCEDURE Create(): T =
(* returns a new, empty list *)
BEGIN
RETURN NIL; (*creation is trivial; empty list is NIL*)
END Create;

PROCEDURE Insert(VAR list: T; value:INTEGER) =

(* inserts new element in list and maintains order *)
VAR
current, previous: T;
new: T := NEW(T, key:= value); (*create new element*)
BEGIN
IF list = NIL THEN list:= new (*first element*)
ELSIF value < list.key THEN (*insert at beginning*)
new.next:= list; list:= new;
ELSE (*find position for insertion*)
 current:= list;
previous:= current;
WHILE (current # NIL) AND (current.key <= value) DO
previous:= current; (*previous hobbles after*)
current:= current.next;
END;
(*after the loop previous points to the insertion point*)
new.next:= current;
(*current = NIL if insertion point is the end*)
previous.next:= new; (*insert new element*)
END; (*IF list = NIL*)
END Insert;

Make sure you understand where these cases come from and can verify why all of
them work correct.

Deletion of a Node

To delete a node from a singly linked-list, we must have pointers to both the node-to-
die and the previous node, so we can reconnect the rest of the list.
PROCEDURE Remove(VAR list: T; value:INTEGER; VAR found: BOOLEAN) =
(* deletes (first) element with value from sorted list,
or returns false in found if the element was not found *)
VAR
current, previous: T;
BEGIN
IF list = NIL THEN found:= FALSE
ELSE (*start search*)
current:= list; previous:= current;
WHILE (current # NIL) AND (current.key # value) DO
previous:= current; (*previous hobbles after*)
current:= current.next;
END;
(*holds: current = NIL or current.key = value, but not both*)
IF current = NIL THEN
found:= FALSE (*value not found*)
ELSE
found:= TRUE; (*value found*)
IF current = list THEN
list:= current.next (*element found at beginning*)
ELSE
previous.next:= current.next
END;
END; (*IF current = NIL*)
END; (*IF list = NIL*)
END Remove;

Passing Procedures as Arguments

Note the passing of a procedure as a parameter - it is legal, and useful to make more
general functions, for example a sort routine for both increasing and decreasing order,
or any order.

PROCEDURE Iterate(list: T; action: Action) =

(* applies action to all elements (with key value as parameter) *)
BEGIN
WHILE list # NIL DO
action(list.key);
list:= list.next;
END;
END Iterate;

BEGIN (* Intlist *)
END Intlist.

Pointers and Parameter Passing

Pointers provide, for better or (usually) worse, and alternate way to modify
parameters. Let us look at two different ways to swap the ``values'' of two pointers.
Procedure Swap1(var p,q:pointer);
Var r:pointer;
begin
r:=q;
q:=p;
p:=r;
end;

This is perhaps the simplest and best way - we just exchange the values of the
pointers...

Alternatively, we could swap the values of what is pointed to, and leave the pointers
unchanged.
Procedure Swap2(p,q : pointer);
var tmp : node;
begin
tmp := q^; (*1*)
q^ := p^; (*2*)
p^ := tmp; (*3*)
end;

After step (*1*):

After step (*2*):

After step (*3*):

Side Effects of Pointers

If swap2, since we do not change the values of p and q, they do not need to be var
parameters!

However, copying the values did not do the same thing as copying the pointers,
because in the first case the physical location of the data changed, while in the second
the data stayed put.

If data which is pointed to moves, the value of what is pointed to can change!

Moral: you must be careful about the side effects of pointer operations!!!

C language does not have var parameters. All side effects are done by passing
pointers. Additional pointer operations in C language help make this practical.

Doing the Shuffle

Lecture 8
Steven S. Skiena

Programming Style

Although programming style (like writing style) is a somewhat subjective thing, there
is a big difference between good and bad.

The good programmer doesn't just strive for something that works, but something that
works elegantly and efficiently; something that can be maintained and understood by
others.

Just like a good writer rereads and rewrites their prose, a good programmer rewrites
their program to make it cleaner and better.

To get a better sense of programming style, let's critique some representative solutions
to the card-shuffling assignment to see what's good and what can be done better.

Ugly looking main

MODULE card EXPORTS Main;
IMPORT SIO;
TYPE
index=[1..200];
Start=ARRAY index OF INTEGER;
Left=ARRAY index OF INTEGER;
Right=ARRAY index OF INTEGER;
Final=ARRAY index OF INTEGER;
VAR
i,j,times,mid,k,x: INTEGER;
start: Start;
left: Left;
right: Right;
final: Final;
BEGIN
SIO.PutText("deck size shuffles\n");
SIO.PutText("--------- --------\n");
SIO.PutText(" 200 ");
SIO.PutInt( times );

REPEAT (*Repeat the following until perfect shuffle*)

i:=1; (*original deck*)

WHILE i<=200 DO
start[i]:=i;
i:=i+1;
END;

j:=1; (*splits into two decks*)

mid:=100;
WHILE (j<=100) DO
left[j]:=start[j];
right[j]:=start[j+mid];
j:=j+1;
END;
x:=1;
k:=1; (*shuffle them into one deck*)
WHILE k<=200 DO
final[k]:=left[x];
final[k+1]:=right[x];
k:=k+2;
END;

UNTIL start[2]=final[2]; (*check if complete shuffle*)

times:=times+1;
END card.

There are no variable or block comments. This program would be hard to understand.

This is an ugly looking program - the structure of the program is not reflected by the
white space.

Indentation and blank lines appear to be added randomly.

There are no subroutines used, so everything is one big mess.

See how the dependence on the number of cards is used several times within the body
of the program, instead of just in one CONST.

What Does shufs Do?

PROCEDURE shufs( nn : INTEGER )= (* shuffling procedure *)

VAR
i : INTEGER; (* index variable *)
count : INTEGER; (* COUNT variable *)

BEGIN

FOR i := 1 TO 200 DO (* reset this array *)

shuffled[i] := i;
END;

count := 0; (* start counter from 0 *)

REPEAT
count := count + 1;
FOR i := 1 TO 200 DO (* copy shuffled -> tempshuf *)
tempshuf[i] := shuffled[i];
END;

FOR i := 1 TO nn DO (* shuffle 1st half *)

shuffled[2*i-1] := tempshuf[i];
END;

FOR i := nn+1 TO 2*nn DO (* shuffle 2nd half *)

shuffled[2*(i-nn)] := tempshuf[i];
END;

UNTIL shuffled = unshuffled ; (* did it return to original? *)

(* print out the data *)

Wr.PutText(Stdio.stdout , "2*n= " & Fmt.Int(2*nn) & " \t" );
Wr.PutText(Stdio.stdout , Fmt.Int(count) & "\n" );

END shufs;

Every subroutine should ``do'' something that is easily described. What does shufs do?

The solution to such problems is to write the block comments for the subroutine
does before writing the subroutine.

If you can't easily explain what it does, you don't understand it.

How many comments are enough?

MODULE Shuffles EXPORTS Main;
IMPORT SIO;
TYPE
Array= ARRAY [1..200] OF INTEGER; (*Create an integer array from *)
(*1 to 200 and called Array *)
VAR
original, temp1, temp2: Array; (*Declare original,temp1 and *)
(*temp2 to be Array *)
counter: INTEGER; (*Declare counter to be integer*)

(********************************************************************)
(* This is a procedure called shuffle used to return a number of *)
(* perfect shuffle. It input a number from the main and run the *)
(* program with it and then return the final number of perfect shuffle *)
(********************************************************************)

PROCEDURE shuffle(total: INTEGER) :INTEGER =

VAR
half, j, p: INTEGER; (*Declare half, j, p to be integer *)
BEGIN
FOR j:= 1 TO total DO
original[j] := j;
temp1[j] := j;
END; (*for*)
half := total DIV 2;
REPEAT
j := 0;
p := 1;
REPEAT
j := j + 1;
temp2[p] := temp1[j]; (* Save the number from the first half *)
(* of the original array into temp2 *)
p := p + 1;
temp2[p] := temp1[half+j]; (* Save the number from the last half*)
(* of the original array into temp2 *)
p := p + 1;
UNTIL p = total + 1; (*REPEAT_UNTIL used to make a new array of temp1*)
INC (counter); (* increament counter when they shuffle once *)
FOR i := 1 TO total DO
temp1[i] := temp2[i];
END; (* FOR loop used to save all the elements from temp2 to temp1 *)
UNTIL temp1 = original; (* REPEAT_UNTIL, when two array match exactly *)
(* same then quick *)
RETURN counter; (* return the counter *)
END shuffle; (* end procedure shuffle *)

(********************************************************************)
(* This is the Main for shuffle program that prints out the numbers *)
(* of perfect shuffles necessary for a deck of 2n cards *)
(********************************************************************)

BEGIN
...
END Shuffles. (* end the main program called Shuffles *)

This program has many comments which should be obvious to anyone who can read
Modula-3.
More useful would be enhanced block comments telling you what the program is done
and how it works.

The ``is it completely reshuffled yet?'' test is done cleanly, although all of the 200
cards are tested regardless of deck size.

The shuffle algorithm is too complicated. Algorithms must be pretty, too

MODULE prj1 EXPORTS Main;
IMPORT SIO;
CONST
n : INTEGER = 100; (*size of split deck*)

TYPE
nArray = ARRAY[1..n] OF INTEGER; (*n sized deck type*)
twonArray = ARRAY[1..2*n] OF INTEGER; (*2n sized deck type*)

VAR
merged : twonArray; (*merged deck*)
count : INTEGER;

PROCEDURE shuffle(size:INTEGER; VAR merged:twonArray)=

VAR
topdeck, botdeck : nArray; (*arrayed split decks*)
BEGIN
FOR i := 1 TO size DO
topdeck[i] := merged[i]; (*split entire deck*)
botdeck[i] := merged[i+size]; (*into top, bottom decks*)
END;
FOR j := 1 TO size DO
merged[2*j-1] := topdeck[j]; (*If odd then 2*n-1 position.*)
merged[2*j] := botdeck[j]; (*If even then 2*n position*)
END;
END shuffle;

PROCEDURE printout(count:INTEGER; size:INTEGER)=

BEGIN
SIO.PutInt(size);
SIO.PutText(" ");
SIO.PutInt(count);
SIO.PutText(" \n");
END printout;

PROCEDURE checkperfect(merged:twonArray; i:INTEGER) : BOOLEAN=

VAR
size : INTEGER;
check : BOOLEAN;
BEGIN
check := FALSE;
size := 0;
REPEAT
 INC(size, 1); (*check to see if*)
IF merged[size+1] - merged[size] = 1 THEN (*deck is perfectly*)
check := TRUE; (*shuffled, if so *)
END; (*card progresses by 1*)
UNTIL (check = FALSE OR size - 1 = i);
RETURN check;
END checkperfect;

Checkperfect is much more complicated than it need be; just check

whether merged[i] = i. You can return without the BOOLEAN variable.

A good thing is that the deck size is all a function of a CONST.

The shuffle is slightly wasteful of space - two extra full arrays instead of two extra
half arrays.

Why does this work correctly?

BEGIN
SIO.PutLine("Welcome to Paul's card shuffling program!");
SIO.PutLine(" DECK SIZE NUMBER OF SHUFFLES ");
SIO.PutLine(" _________________________________ ");
num_cards := 2;
REPEAT
counter := 0;
FOR i := 1 TO (num_cards) DO
deck[i] :=i;
END; (*initializes deck*)
REPEAT
deck := Shuffle(deck,num_cards);
INC(counter);
UNTIL deck[2] = 2;
SIO.PutInt(num_cards,16); SIO.PutInt(counter,19);
SIO.PutText("\n");
INC(num_cards,2);
(*increments the number of cards in deck by 2.*)
UNTIL ( num_cards = ((2*n)+2));
END ShuffleCards.

Why we know that this stopping condition suffices to get us all the cards in the right
position. This should be proven prior to use.

Why use a Repeat loop when For will do?

Program Defensively

I am starting to see the wreckage of several programs because students are not
building their programs to be debugged.
  Add useful debug print statements! Have your program describe what it is
doing!
 Document what you think your program does! Otherwise, how do you know
whine it is doing it!
 Build your program in stages! Thus you localize your bugs, and make sure you
understand simple things before going on to complicated things.
 Use spacing to show the structure of your program. A good program is a pretty
program!

Recursive and Doubly Linked Lists

Lecture 9
Steven S. Skiena

Recursive List Implementation

The basic insertion and deletion routines for linked lists are more elegantly written
using recursion.
PROCEDURE Insert(VAR list: T; value:INTEGER) =
(* inserts new element in list and maintains order *)
VAR new: T; (*new node*)
BEGIN
IF list = NIL THEN
list:= NEW(T, key := value) (*list is empty*)
ELSIF value < list.key THEN (*proper place found: insert*)
new := NEW(T, key := value);
new.next := list;
list := new;
ELSE (*seek position for insertion*)
Insert(list.next, value);
END; (*IF list = NIL*)
END Insert;

PROCEDURE Remove(VAR list:T; value:INTEGER; VAR found:BOOLEAN) =

(* deletes (first) element with value from sorted list,
or returns false in found if the element was not found *)
BEGIN
IF list = NIL THEN (*empty list*)
found := FALSE
ELSIF value = list.key THEN (*elemnt found*)
found := TRUE;
list := list.next
ELSE (*seek for the element to delete*)
Remove(list.next, value, found);
END;
END Remove;
Doubly Linked Lists

Often it is necessary to move both forward and backwards along a linked list. Thus we
need another pointer from each node, to make it doubly linked.

List types are analogous to dance structures:

 Conga line - singly linked list.

 Chorus line - doubly linked list.
 Hora circle - double linked circular list.

Extra pointers allow the flexibility to have both forward and backwards linked lists:
type
pointer = REF node;
node = record
info : item;
front : pointer;
back : pointer;
end;

Insertion

How do we insert p between nodes q and r in a doubly linked list?

p^.front = r;
p^.back = q;
r^.back = p;
q^.front = p;

It is not absolutely necessary to have pointer r, since r = q .front, but it makes it

cleaner.

The boundary conditions are inserting before the first and after the last element.

How do we insert before the first element in a doubly linked list (head)?
p^.back = NIL;
p^.front = head;
head^.back = p;
head = p; (* must point to entire structure *)

Inserting at the end is similar, except head doesn't change, and a back pointer is set to
NIL.

Linked Lists: Pro or Con?

The advantages of linked lists include:

 Overflow can never occur unless the memory is actually full.

 Insertions and deletions are easier than for contiguous (array) lists.
 With large records, moving pointers is easier and faster than moving the items
themselves.

The disadvantages of linked lists include:

 The pointers require extra space.

 Linked lists do not allow random access.
 Time must be spent traversing and changing the pointers.
 Programming is typically trickier with pointers.

Recursion and Backtracking

Lecture 10
Steven S. Skiena

Recursion

Recursion is a wonderful, powerful way to solve problems.

Elegant recursive procedures seem to work by magic, but the magic is same
reason mathematical induction works!

Example: Prove   .

For n=1,   , so its true. Assume it is true up to n-1.

Example: All horses are the same color! (be careful of your basis cases!)

The Tower of Hanoi

MODULE Hanoi EXPORTS Main; (*18.07.94*)
(* Implementation of the game Towers of Hanoi. *)
 PROCEDURE Transfer(from, to: Post) =
(*moves a disk from post "from" to post "to"*)
BEGIN
WITH f = posts[from], t = posts[to] DO
INC(t.top);
t.disks[t.top]:= f.disks[f.top];
f.disks[f.top]:= 0;
DEC(f.top);
END; (*WITH f, t*)
END Transfer;

PROCEDURE Tower(height:[0..Height] ; from, to, between: Post) =

(*Does the job through recursive calls on itself*)
BEGIN
IF height > 0 THEN
Tower(height - 1, from, between, to);
Transfer(from, to);
Display();
Tower(height - 1, between, to, from);
END;
END Tower;

BEGIN (*main program Hanoi*)

posts[Post.Start].top:= Height;
FOR h:= 1 TO Height DO
posts[Post.Start].disks[h]:= Height - (h - 1)
END;
Tower(Height, Post.Start, Post.Finish, Post.Temp);
END Hanoi.

To count the number of moves made,

Recursion not only made a complicated problem understandable, it made it easy to

understand.

Combinatorial Objects

Many mathematical objects have simple recursive definitions which can be exploited
algorithmically.

Example: How can we build all subsets of n items? Build all subsets of n-1 items,
copy the subsets, and add item n to each of the subsets in one copy but not the other.

Once you start thinking recursively, many things have simpler formulations, such as
traversing a linked list or binary search.

Gray codes
We saw how to generate subsets recursively. Now let us generate them in an
interesting order.

All subsets of   can be represented as binary strings of length n, where

bit i tells whether i is in the subset or not.

Obviously, all subsets must differ in at least one element, or else they would be
identical. An order where they differ by exactly one from each other is called a Gray
code.

For n=1, {},{1}.

For n=2, {},{1},{1,2},{2}.

For n=3, {},{1},{1,2},{2},{2,3},{1,2,3},{1,3},{3}

Recursive construction algorithm: Build a Gray Code of   , make a

reverse copy of it, append n to each subset in the reverse copy, and stick the two
together!

Formulating Recursive Programs

Think about the base cases, the small cases where the problem is simple enough to
solve.

Think about the general case, which you can solve if you can solve the smaller cases.

Unfortunately, many of the simple examples of recursion are equally well done by
iteration, making students suspicious.

Further, many of these classic problems have hidden costs which make
recursion seem expensive, but don't be fooled!

Factorials
PROCEDURE Factorial (n: CARDINAL): CARDINAL =
BEGIN
IF n = 0 THEN
RETURN 1 (* trivial case *)
ELSE
RETURN n * Factorial(n-1) (* recursive branch *)
END (* IF*)
END Factorial;
Be sure you understand how the parameter passing mechanism works.

Would this program work if n was a VAR parameter?

Fibonacci Numbers

The Fibonacci numbers are given by the recurrence relation   .

PROCEDURE Fibonacci(n : CARDINAL) : CARDINAL =
BEGIN (* Fibonacci *)
IF n <= 1 THEN
RETURN 1
ELSE
RETURN Fibonacci(n-1) + Fibonacci(n-2) (*n > 1*)
END (* IF *)
END Fibonacci;

How much time does this elementary Fibonacci function take?

Implementing Recursion

Part of the mystery of recursion is the question of how the machine keeps everything
straight.

How come local variables don't get trashed?

The answer is that whenever a procedure or function is called, the local variables
are pushed on a stack, so the new recursive call is free to use them.

When a procedure ends, the variables are popped off the stack to restore them to
where they were before the call.

Thus the space used is equal to the depth of the recursion, since stack space is reused.

Tail Recursion

Tail recursion costs space, but not time. It can be removed mechanically and is by
some compilers.

Moral: Do not be afraid to use recursion if the algorithm is efficient.

The overhead of recursion vs. maintaining your own stack is too small to worry about.
By being clever, you can sometimes save stack space. Consider the following
variation of Quicksort:

If (p-1 < h-p) then

Qsort(1,p)

Qsort(p,h)

else

Qsort(p,h)

Qsort(1,p)

By doing the smaller half first, the maximum stack depth is   in the worst
case.

Applications of Recursion

You may say, ``I just want to get a job and make lots of money. What can recursion
do for me?

We will look at three applications

 Backtracking
 Game Tree Search
 Recursion Descent Compilation

The N-Queens Problem

Backtracking is a way to solve hard search problems.

For example, how can we put n queens on an   board so that no two queens
attack each other?

Tree Pruning

Backtracking really pays off when we can prove a node early in the search tree.

Thus we need never look at its children, or grandchildren, or great....

We apply backtracking to big problems, so the more clever we are, the more time we
save.

There are   total sets of eight squares but no two queens can be in the same row.
There are   ways to place eight queens in different rows. However, since no two
queens can be in the same column, there are only 8! permutations of columns, or only
40,320 possibilities.

We must also be clever to test as quickly as possible the new queen does not violate a
diagonal constraint

Applications of Recursion
Lecture 11
Steven S. Skiena

Game Trees

Chess playing programs work by constructing a tree of all possible moves from a
given position, so as to select the best possible path.

The player alternates at each level of the tree, but at each node the player whose move
it is picks the path that is best for them.

A player has a forced loss if lead down a path where the other guy wins if they play
correctly.
This is a recursive problem since we can always maximize, by just changing
perspective.

In a game like chess, we will never reach the bottom of the tree, so we must stop at a
particular depth.

Alpha-beta Pruning

Sometimes we don't have to look at the entire game tree to get the right answer:

No matter what the red score is, it cannot help max and thus need not be looked at.

An advanced strategy called alpha-beta running reduces search accordingly.

Recursive Descent Compilation

Compilers do two useful things

 They identify whether a program is legal in the language.

 They translate it into assembly language.

To do either, we need a precise description of the language, a BNF grammar which

gives the syntax. A grammar for Modula-3 is given throughout your text.

The language definition can be recursive!!

Our compiler will follow the grammar to break the program into smaller and smaller
pieces.

When the pieces get small enough, we can spit out the appropriate chunk of assembly
code.

To avoid getting into infinite loops, we place our trust in the fellow who wrote the
grammar. Proper design can ensure that there are no such troubles.

Abstraction and Modules

Lecture 12
Steven S. Skiena
Abstract Data Types

It is important to structure programs according to abstract data types: collections of

data with well-defined operations on it

Example: Stack or Queue. Data: A sequence of items Operations: Initialize, Empty?,

Full?, Push, Pop, Enqueue, Dequeue

Example: Infinite Precision Integers. Data: Linked list of digits with sign

bit. Operations: Print number, Read Number, Add, Subtract, Multiply, Divide,
Exponent, Module, Compare.

Abstract data types add clarity by separating the definitions from the implementations.

What Do We Want From Modules?

Separate Compilation - We should be able to break the program into smaller files.
Further, we shouldn't need the source for each Module to link it together, just the
compiled object code files.

Communicate Desired Information Between Modules - We should be able to define a

type or procedure in one module and use it in another.

Information Hiding - We should be able to define a type or procedure in one module

and forbid using it in another! Thus we can clearly separate the definition of an
abstract data type from its implementation!

Modula-3 supports all of these goals by separating interfaces (.i3 files) from
implementations (.m3 files).

Example: The Piggy Bank

Below is an interface file to:

INTERFACE PiggyBank; (*RM*)
(* Interface to a piggy bank:

You can insert money with "Deposit". The only other permissible
operation is smashing the piggy bank to get the ``money back''
The procedure "Smash" returns the sum of all deposited amounts
and makes the piggy bank unusable.
*)

PROCEDURE Deposit(cash: CARDINAL);

PROCEDURE Smash(): CARDINAL;
END PiggyBank.

Note that this interface does not reveal where or how the total value is stored, nor how
to initialize it.

These are issues to be dealt with within the implementation of the module.

Piggy Bank Implementation

MODULE PiggyBank; (*RM/CW*)
(* Implementation of the PiggyBank interface *)

VAR contents: INTEGER; (* state of the piggy bank *)

PROCEDURE Deposit(cash: CARDINAL) =

(* changes the state of the piggy bank *)
BEGIN
<*ASSERT contents >= 0*> (* piggy bank still okay? *)
contents := contents + cash
END Deposit;

PROCEDURE Smash(): CARDINAL =

VAR oldContents: CARDINAL := contents; (* contents before smashing *)
BEGIN
contents := -1; (* smash piggy bank *)
RETURN oldContents
END Smash;

BEGIN
contents := 0 (* initialization of state variables in body *)
END PiggyBank.

A Client Program for the Bank

MODULE Saving EXPORTS Main; (*RM*)
(* Client of the piggy bank:

In a loop the user is prompted for the amount of deposit.

Entering a negative amount smashes the piggy bank.
*)

FROM PiggyBank IMPORT Deposit, Smash;

FROM SIO IMPORT GetInt, PutInt, PutText, Nl, Error;

<*FATAL Error*>

VAR cash: INTEGER;

BEGIN (* Saving *)
PutText("Amount of deposit (negative smashes the piggy bank): \n");
REPEAT
cash := GetInt();
IF cash >= 0 THEN
 Deposit(cash)
ELSE
PutText("The smashed piggy bank contained $");
PutInt(Smash());
Nl()
END;
UNTIL cash < 0
END Saving.

Interface File Conventions

Imports describe what procedures a given module makes available.

Exports describes what we are willing to make public, ultimately including the

``MAIN'' program.

By naming files with the same .m3 and .i3 names, the ``ezbuild'' make command can
start from the file with the main program, and final all other relevant files.

Ideally, the interface file should hide as much detail about the internal implementation
of a module from its users as possible. This is not easy without sophisticated language
features.

Hiding the Details

INTERFACE Fraction; (*RM*)
(* defines the data type for rational numbers *)

TYPE T = RECORD
num : INTEGER;
den : INTEGER;
END;

PROCEDURE Init (VAR fraction: T; num: INTEGER; den: INTEGER := 1);

(* Initialize "fraction" to be "num/den" *)

PROCEDURE Plus (x, y : T) : T; (* x + y *)

PROCEDURE Minus (x, y : T) : T; (* x - y *)
PROCEDURE Times (x, y : T) : T; (* x * y *)
PROCEDURE Divide (x, y : T) : T; (* x / y *)

PROCEDURE Numerator (x : T): INTEGER; (* returns the numerator of x *)

PROCEDURE Denominator (x : T): INTEGER; (* returns the denominator of x *)

END Fraction.

Note that there is a dilemma here. We must make type T public so these procedures
can use it, but would like to prevent users from accessing (or even knowing about) the
fields num and dem directly.
Subtypes and REFANY

Modula-3 permits one to declare subtypes of types, A <: B, which means that anything

of type A is of type B, but everything of type type B is not necessarily of type A.

This proves important in implementing advanced object-oriented features like

inheritance.

REFANY is a pointer type which is a supertype of any other pointer. Thus a variable
of type REFANY can store a copy of any other pointer.

This enables us to define public interface files without actually revealing the guts of
the fraction type implementation.

Fraction type with REFANY

INTERFACE FractionType; (*19.12.94. RM, LB*)
(* defines the data type of rational numbers, compare with Example 10.10! *)

TYPE T <: REFANY; (*T is a subtype of Refany; its structure is hidden*)

PROCEDURE Create (numerator: INTEGER; denominator: INTEGER := 1): T;

PROCEDURE Plus (x, y : T) : T; (* x + y *)
PROCEDURE Minus (x, y : T) : T; (* x - y *)
PROCEDURE Mult (x, y : T) : T; (* x * y *)
PROCEDURE Divide (x, y : T) : T; (* x : y *)
PROCEDURE Numerator (x : T): INTEGER;
PROCEDURE Denominator (x : T): INTEGER;

END FractionType.

Somewhere within a module we must reveal the implementation of type T. This is

done with a REVEAL statement:
MODULE FractionType; (*19.12.94. RM, LB*)
(* Implementation of the type FractionType. Compare with Example
10.12. In this version the structure of elements of the type is
hidded in the interface. The structure is revealed here.
*)

REVEAL T = BRANDED REF RECORD (*opaque structure of T*)

num, den: INTEGER
END; (*T*)

...

The Key Idea about REFANY

With generic pointers, it becomes necessary for type checking to be done a run-time,
instead of at compile-time as done to date.

This gives more flexibility, but much more room for you to hang yourself. For
example:
TYPE
Student = REF RECORD lastname,firstname:TEXT END;
Address = REF RECORD street:TEXT; number:CARDINAL END;

VAR
r1 : Student;
r2 := NEW(Student, firstname:="Julie", lastname:="Tall");
adr := NEW(Address, street:="Washington", number:="21");
any := REFANY;

BEGIN
any := r2; (* always a safe assignment *)
r1 := any; (* legal because any is of type student *)
adr := any; (* produces a run-time error, not compile-time *)

You should worry about the ideas behind generic implementations (why does
Modula-3 do it this way?) more than the syntactic details (how does Modula-3 let you
do this?). It is very easy to get overwhelmed by the detail.

Generic Types

When we think about the abstract data type ``Stack'' or ``Queue'', the implementation
of th the data structure is pretty much the same whether we have a stack
of integers or reals.

Without generic types, we are forced to declare the type of everything at compile
time. Thus we need two distinct sets of functions, like PushInteger and PushReal for
each operation, which is waste.

Object-Oriented programming languages provide features which enable us to create

abstract data types which are more truly generic, making it cleaner and easier to reuse
code.

Object-Oriented Programming
Lecture 13
 Steven S. Skiena

Why Objects are Good Things

Modules provide a logical grouping of procedures on a related topic.

Objects provide a logical grouping of data and associated operations.

The emphasis of modules is on procedures; the emphasis of objects is on data.

Modules are verbs followed by nouns: Push(S,x), while objects are nouns followed by
verbs: S.Push(x).

This provides only an alternate notation for dealing with things, but different

notations can sometimes make it easier to understand things - the history of Calculus
is an example.

Objects do a great job of encapsulating the data items within, because the only access
to them is through the methods, or associated procedures.

Stack Object
MODULE StackObj EXPORTS Main; (*24.01.95. LB*)
(* Stack implemented as object type. *)

IMPORT SIO;

TYPE
ET = INTEGER; (*Type of elements*)
Stack = OBJECT
top: Node := NIL; (*points to stack*)
METHODS
push(elem:ET):= Push; (*Push implements push*)
pop() :ET:= Pop; (*Pop implements pop*)
empty(): BOOLEAN:= Empty; (*Empty implements empty*)
END; (*Stack*)
Node = REF RECORD
info: ET; (*Stands for any information*)
next: Node (*Points to the next node in the
stack
*)
END; (*Node*)

PROCEDURE Push(stack: Stack; elem:ET) =

(*stack: receiver object (self)*)
VAR
new: Node := NEW(Node, info:= elem); (*Element instantiate*)
BEGIN
new.next:= stack.top;
stack.top:= new; (*new element added to top*)
END Push;
 PROCEDURE Pop(stack: Stack): ET =
(*stack: receiver object (self)*)
VAR first: ET;
BEGIN
first:= stack.top.info; (*Info copied from first element*)
stack.top:= stack.top.next; (*first element removed*)
RETURN first
END Pop;

PROCEDURE Empty(stack: Stack): BOOLEAN =

(*stack: receiver object (self)*)
BEGIN
RETURN stack.top = NIL
END Empty;

VAR
stack1, stack2: Stack := NEW(Stack); (*2 stack objects created*)
i1, i2: INTEGER;
BEGIN
stack1.push(2); (*2 pushed onto stack1*)
stack2.push(6); (*6 pushed onto stack2*)
i1:= stack1.pop(); (*pop element from stack1*)
i2:= stack2.pop(); (*pop element from stack2*)
SIO.PutInt(i1);
SIO.PutInt(i2);
SIO.Nl();
END StackObj.

Object-Oriented Programming

Object-oriented programming is a popular, recent way of thinking about program

organization.

OOP is typically characterized by three major ideas:

 Encapsulation - objects incorporate both data and procedures.

 Inheritance - classes (object types) are arranged in a hierarchy, and each class
inherits but specializes methods and data from its ancestors.
 Polymorphism - a particular object can take on different types at different
times. We saw this with REFANY variables whose types depend upon what is
assigned it it (dynamic binding).

Inheritance

When we define an object type (class), we can specify that it be derived from (subtype
to) another class. For example, we can specialize the Stack object into a GarbageCan:
TYPE
GarbageCan = Stack OBJECT
 OVERRIDES
pop():= Yech; (* Remove something from can?? *)
dump():= RemoveAll; (* Discard everything from can *)
END; (*GarbageCan*)

The GarbageCan type is a form of stack (you can still push in it the same way), but we
have modified the pop and dump methods.

This subtype-supertype relation defines a hierarchy (rooted tree) of classes. The

appropriate method for a given object is determined at run time (dynamic binding)
according to the first class at or above the current class to define the method.

OOP and the Calculator Program

How might object-oriented programming ideas have helped in writing the calculator
program?

Many of you noticed that the linked stack type was similar to the long integer type,
and wanted to reuse the code from one in another.

The following type hierarchy shows one way we could have exploited this, by
creating special stack methods push and pop, and overwriting
the add and subtract methods for general long-integers.

Philosophical issue: should Long-Integer be a subtype of Positive-Long-Integer or

visa versa?

Why didn't I urge you to do it this way? In my opinion, the complexity of mastering
and using the OOP features of Modula-3 would very much overwhelm the code
savings from such a small program. Object-oriented features differ significantly from
language to language, but the basic principles outlined here are fairly common.

However, you should see why inheritance can be a big win in organizing larger
programs.

Simulations
Lecture 14
Steven S. Skiena
Simulations

Often, a system we are interested in may be too complicated to readily understand,

and too expensive or big to experiment with.

 What direction will an oil spill move in the Persian Gulf, given certain weather
conditions?
 How much will increases in the price of oil change the American
unemployment rate?
 Now much traffic can an airport accept before long delays become common?

We can often get good insights into hard problems by performing mathematical
simulations.

Scoring in Jai-alai

Jai-alai is a Basque variation of handball, which is important because you can bet on it
in Connecticut. What is the best way to bet?

The scoring system in use in Connecticut is very interesting. Eight players or teams
appear in each match, numbered 1 to 8. The players are arranged in a queue, and the
top two players in the queue play each other. The winner gets a point and keeps
playing, the loser goes to the end of the queue. Winner is the first one to get to 7
points.

This scoring obviously favors the low numbered players. For fairness, after the first
trip through the queue, each point counts two.

But now is this scoring system fair?

Simulating Jai-Alai
1 PLAYS 2
1 WINS THE POINT, GIVING HIM 1

1 PLAYS 3
3 WINS THE POINT, GIVING HIM 1

4 PLAYS 3
3 WINS THE POINT, GIVING HIM 2

5 PLAYS 3
3 WINS THE POINT, GIVING HIM 3

6 PLAYS 3
3 WINS THE POINT, GIVING HIM 4

7 PLAYS 3
3 WINS THE POINT, GIVING HIM 5

8 PLAYS 3
8 WINS THE POINT, GIVING HIM 1

8 PLAYS 2
2 WINS THE POINT, GIVING HIM 2

1 PLAYS 2
2 WINS THE POINT, GIVING HIM 4

4 PLAYS 2
2 WINS THE POINT, GIVING HIM 6

5 PLAYS 2
5 WINS THE POINT, GIVING HIM 2

5 PLAYS 6
5 WINS THE POINT, GIVING HIM 4

5 PLAYS 7
7 WINS THE POINT, GIVING HIM 2

3 PLAYS 7
7 WINS THE POINT, GIVING HIM 4

8 PLAYS 7
7 WINS THE POINT, GIVING HIM 6

1 PLAYS 7
7 WINS THE POINT, GIVING HIM 8

WIN-PLACE-SHOW IS 7 2 3

BETTER THAN AVERAGE TRIFECTAS: 1 TRIALS

WIN PLACE SHOW OCCURRENCES

7 2 3

Is the Scoring Fair?

How can we test if the scoring system is fair?

We can simulate a lot of games and see how often each player wins the game!

But when player A plays a point against player B, how do we decide who wins? If the
players are all equally matched, we can flip a coin to decide. We can use a random
number generator to flip the coin for us!
What data structures do we need?

 A queue to maintain the order of who is next to play.

 An array to keep track of each player's score during the game.
 A array to keep track of how often a player has won so far.

Simulation Results
Jai-alai Simulation Results

Pos win %wins place %places show %shows

1 16549 16.55 17989 17.99 15123 15.12
2 16207 16.21 17804 17.80 15002 15.00
3 13584 13.58 16735 16.73 14551 14.55
4 12349 12.35 13314 13.31 13786 13.79
5 10103 10.10 10997 11.00 13059 13.06
6 10352 10.35 7755 7.75 11286 11.29
7 9027 9.03 8143 8.14 9007 9.01
8 11829 11.83 7263 7.26 8186 8.19

total games = 100000

Compare these to the actual win results from Berenson's Jai-alai 1983-1986:

1 14.1%, 2 14.6%, 3 12.8%, 4 11.5%, 5 12.0%, 6 12.4%, 7 11.1%, 8 11.3%

Were these results good?

Yes, but not good enough to bet with! The matchmakers but the best players in the
middle, so as to even the results. A more complicated model will be necessary for
better results.

Limitations of Simulations

Although simulations are good things, there are several reasons to be skeptical of any
results we get.

Is the underlying model for the simulation accurate?

Are the implicit assumptions reasonable, or are there biases?

How do we know the program is an accurate implementation of the given model?

After all, we wrote the simulation because we do not know the answers! How do you
debug a simulation of two galaxies colliding or the effect of oil price increases on the
economy?
So much rides on the accuracy of simulations it is critical to build in self-verification
tests, and prove the correctness of implementation.

Random Number Generator

We have shown that random numbers are useful for simulations, but how do we get
them?

First we must realize that there is a philosophical problem with

generating random numbers on a deterministic machine.

``Anyone who considers arithmetical methods of producing random digits is , of

course, in a state of sin.'' - John Von Neumann

What we really want is a good way to generate pseudo-random numbers, a sequence

which has the same properties as a truly random source.

This is quite difficult - people are lousy at picking random numbers. Note that the
following sequence produces 0's + 1's with equal frequency but does not look like a
fair coin:

Even recognizing random sequences is hard. Are the digits of   pseudo-random?

Should all those palindromes (535, 979, 46264, 383) be there?

The Middle Square Method

Von Neumann suggested generating random numbers by taking a big integer,

squaring it, and using the middle digits as the seed/random number.

It looks random to me... But what happens when the middle digits just happen to be
0000000000? From then on, all digits will be zeros!

Linear Congruential Generators

The most popular random number generators, because of simplicity, quality, and
small state requirements are linear congruential generators.

If   is the last random number we generated, then

The quality of the numbers generated depends upon careful selection of the seed   
and the constants a, c, and m.

Why does it work? Clearly, the numbers are between 0 and m-1. Taking the remainder
mod m is like seeing where a roulette ball drops in a wheel with m slots.

SUNY at Stony BrookMidterm 1

CSE 214 - Data Structures October 10, 1997

Midterm Exam

Name: Signature:
ID #: Section #:

INSTRUCTIONS:

 You may use either pen or pencil.

 Check to see that you have 4 exam pages plus this cover (5 total).
 Look over all problems before starting work.
 Your signature above signs the CSE 214 Honor Pledge: ``On my honor as a
student I have neither given nor received aid on this exam.''
 Think before you write.
 Good luck!!

1) (25 points) Assume that you have the linked structure on the left, where each node
contains a .next field consisting of a pointer, and the pointer p points to the structure
as shown. Describe the sequence of Modula-3 pointer manipulations necessary to
convert it to the linked structure on the right. You may not change any of
the .info fields, but you may use temporary pointers tmp1, tmp2, and tmp3 if you wish.

Many different solutions were possible, including:

tmp1 := p;
p := p.next;
p^.next^.next := tmp1;

2) (30 points) Write a procedure which ``compresses'' a linked list by deleting

consecutive copies of the same character from it. For example, the
list (A,B,B,C,A,A,A,C,A) should be compressed to (A,B,C,A,C,A). Thus the same
character can appear more than once in the compressed list, only not successively.
Your procedure must have one argument as defined below, a VAR
parameter head pointing to the front of the linked list. Each node in the list
has .info and .next fields.
PROCEDURE compress(VAR head : pointer);

Many different solutions are possible, but recursive solutions are particularly clean
and elegant.
PROCEDURE compress(VAR head : pointer);
VAR
second : pointer; (* pointer to next element *)

BEGIN
IF (head # NIL) THEN
second := head^.next;
IF (second # NIL)
IF (head^.info = second^.info) THEN
head^.next = second^.next;
compress(head);
ELSE
compress(head^.next);
END;
END;
END;
END;

3) (20 points) Provide the output of the following program:

MODULE strange; EXPORTS main;

IMPORT SIO;

TYPE
ptr_to_integer = REF INTEGER;
VAR
a, b : ptr_to_integer;
 PROCEDURE modify(x : ptr_to_integer; VAR y : ptr_to_integer);
begin
x^ := 3;
SIO.PutInt(a^);
SIO.PutInt(x^); SIO.Nl();
y^ := 4;
SIO.PutInt(b^);
SIO.PutInt(y^); SIO.Nl();
end;

begin
a := NEW(INTEGER); b := NEW(INTEGER);
a^ := 1;
b^ := 2;
SIO.PutInt(a^);
SIO.PutInt(b^); SIO.Nl();
modify(a,b);
SIO.PutInt(a^);
SIO.PutInt(b^); SIO.Nl();

end.

Answers:
1 2
3 3
4 4
3 4

4) (25 points)

Write brief essays answering the following questions. Your answer must fit
completely in the space allowed

(a) Explain the difference between objects and modules? ANSWER: Several answers
possible, but the basic differences are (1) the notation to use them, and (2) that objects
encapsulate both procedures and data where modules are procedure oriented. (b) What
is garbage collection? ANSWER: The automatic reuse of dynamic memory which,
because of pointer dereferencing, is no longer accessible. (c) What might be an
advantage of a doubly-linked list over a singly-linked list for certain applications?
ANSWER: Additional flexibility in moving both forward and in reverse on a linked
list. Specific advantages include being able to delete a node from a list given just a
pointer to the node, and efficiently implementing double-ended queues (supporing
push, pop, enqueue, and dequeue).
 Asymptotics
Lecture 15
Steven S. Skiena

Analyzing Algorithms

There are often several different algorithms which correctly solve the same problem.
How can we choose among them? There can be several different criteria:

 Ease of implementation
 Ease of understanding
 Efficiency in time and space

The first two are somewhat subjective. However, efficiency is something

we can study with mathematical analysis, and gain insight as to which is the fastest
algorithm for a given problem.

Time Complexity of Programs

What would we like as the result of the analysis of an algorithm? We might hope for a
formula describing exactly how long a program implementing it will run.

Example: Binary search will take   milliseconds on an array

of n elements.

This would be great, for we could predict exactly how long our program will take. But
it is not realistic for several reasons:

1. Dependence on machine type - Obviously, binary search will run faster on a

CRAY than a PC. Maybe binary search will now take   ms?
2. Dependence on language/compiler - Should our time analysis change when
someone uses an optimizing compiler?
3. Dependence of the programmer - Two different people implementing the same
algorithm will result in two different programs, each taking slightly differed
amounts of time.
4. Should your time analysis be average or worst case? - Many algorithms return
answers faster in some cases than others. How did you factor this in? Exactly
what do you mean by average case?
 5. How big is your problem? - Sometimes small cases must be treated different
from big cases, so the same formula won't work.

Time Complexity of Algorithms

For all of these reasons, we cannot hope to analyze the performance

of programs precisely. We can analyze the underlying algorithm, but at a less precise
level.

Example: Binary search will use about   iterations, where each iteration takes
time independent of n, to search an array of n elements in the worst case.

Note that this description is true for all binary search programs regardless of language,
machine, and programmer.

By describing the worst case instead of the average case, we saved ourselves some

nasty analysis. What is the average case?

Algorithms for Multiplications

Everyone knows two different algorithms for multiplication: repeated addition and
digit-by-digit multiplication.

Which is better? Let's analyze the complexity of multiplying an n-digit number by

an m-digit number, where   .

In repeated addition, we explicity use that   . Thus adding

an n-digit + m-digit number,   requires ``about'' n+m steps, one for each digit.

How many additions can we do in the worst case? The biggest n-digit number is all
nines, and   .

The total time complexity is the cost per addition times the number of additions, so
the total complexity   .

Digit-by-Digit Multiplication

Since multiplying one digit by one other digit can be done by looking up in a
multiplication table (2D array), each step requires a constant amount of work.
Thus to multiply an n-digit number by one digit requires ``about'' n steps.
With m ``extra'' zeros (in the worst case), ``about'' n + m steps certainly suffice.

We must do m such multiplications and add them up - each add costs as much as the
multiplication.

The total complexity is the cost-per-multiplication * number-of-multiplications +

cost-per-addition * number-of- multiplication   .

Which is faster?

Clearly the repeated addition method is much slower by our analysis, and the
difference is going to increase rapidly with n...

Further, it explains the decline and fall of Roman empire - you cannot do digit-by-
digit multiplication with Roman numbers! 

Growth Rates of Functions

To compare the efficiency of algorithms then, we need a notation to

classify numerical functions according to their approximate rate of growth.

We need a way of exactly comparing approximately defined functions. This is the big

Oh Notation:

If f(n) and g(n) are functions defined for positive integers, then f(n)= O(g(n)) means
that there exists a constant c such that   for all sufficiently
large positive integers.

The idea is that if f(n)=O(g(n)), then f(n) grows no faster (and possibly slower)
than g(n).

Note this definition says nothing about algorithms - it is just a way to compare

numerical functions!
Examples

Example:   is   . Why? For all n > 100,

clearly   , so it satisfies the definition for c=100.

Example:   is not   . Why? No matter what value of c you

pick,   is not true for n>c!

In the big Oh Notation, multiplicative constants and lower order terms are
unimportant. Exponents are important.

Ranking functions by the Big Oh

The following functions are different according to the big Oh notation, and are ranked
in increasing order:

O(1) Constant growth

 Logarithmic growth (note:independent of base!)

 Polynomial growth: ordered by exponent

O(n) Linear Growth

 Quadratic growth

 Exponential growth

Why is the big Oh a Big Deal?

Suppose I find two algorithms, one of which does twice as many operations in solving
the same problem. I could get the same job done as fast with the slower algorithm if I
buy a machine which is twice as fast.

But if my algorithm is faster by a big Oh factor - No matter how much faster you
make the machine running the slow algorithm the fast-algorithm, slow
machine combination will eventually beat the slow algorithm, fast
machine combination.

I can search faster than a supercomputer for a large enough dictionary, If I use binary
search and it uses sequential search!

An Application: The Complexity of Songs

Suppose we want to sing a song which lasts for n units of time. Since n can be large,
we want to memorize songs which require only a small amount of brain space, i.e.
memory.    

Let S(n) be the space complexity of a song which lasts for n units of time.

The amount of space we need to store a song can be measured in either the words or
characters needed to memorize it. Note that the number of characters is   
since every word in a song is at most 34 letters long -
Supercalifragilisticexpialidocious!

What bounds can we establish on S(n)? S(n) = O(n), since in the worst case we must
explicitly memorize every word we sing - ``The Star-Spangled Banner''

The Refrain

Most popular songs have a refrain, which is a block of text which gets repeated after
each stanza in the song:  

Bye, bye Miss American pie

Drove my chevy to the levy but the levy was dry
Them good old boys were drinking whiskey and rye
Singing this will be the day that I die.

Refrains made a song easier to remember, since you memorize it once yet sing it O(n)
times. But do they reduce the space complexity?

Not according to the big oh. If

Then the space complexity is still O(n) since it is only halved (if the verse-size =
refrain-size):

The k Days of Christmas

To reduce S(n), we must structure the song differently.

Consider ``The k Days of Christmas''. All one must memorize is:

On the kth Day of Christmas, my true love gave to me, 

On the First Day of Christmas, my true love gave to me, a partridge in a pear tree

But the time it takes to sing it is

If   , then   , so   . 100 Bottles of Beer

What do kids sing on really long car trips?

n bottles of beer on the wall,

n bottles of beer.
You take one down and pass it around
n-1 bottles of beer on the ball.

All you must remember in this song is this template of size   , and the current
value of n. The storage size for n depends on its value, but   bits suffice.

This for this song,   .

Uh-huh, uh-huh

Is there a song which eliminates even the need to count?

That's the way, uh-huh, uh-huh

I like it, uh-huh, huh

Reference: D. Knuth, `The Complexity of Songs', Comm. ACM, April 1984, pp.18-24

Introduction to Sorting
Lecture 16
 Steven S. Skiena

Sorting

Sorting is, without doubt, the most fundamental algorithmic problem

1. Supposedly, 25% of all CPU cycles are spent sorting

2. Sorting is fundamental to most other algorithmic problems, for example binary
search.
3. Many different approaches lead to useful sorting algorithms, and these ideas
can be used to solve many other problems.

What is sorting? It is the problem of taking an arbitrary permutation of n items

and rearranging them into the total order,

Knuth, Volume 3 of ``The Art of Computer Programming is the definitive reference

of sorting.

Issues in Sorting

Increasing or Decreasing Order? - The same algorithm can be used by both all we
need do is change   to   in the comparison function as we desire.

What about equal keys? - Does the order matter or not? Maybe we need to sort on
secondary keys, or leave in the same order as the original permutations.

What about non-numerical data? - Alphabetizing is sorting text strings, and libraries
have very complicated rules concerning punctuation, etc. Is Brown-Williams before or
after Brown America before or after Brown, John?

We can ignore all three of these issues by assuming a comparison function which

depends on the application. Compare (a,b) should return ``<'', ``>'', or ''=''.

Applications of Sorting

One reason why sorting is so important is that once a set of items is sorted, many
other problems become easy.  
SearchingBinary search lets you test whether an item is in a dictionary in   
time.  

Speeding up searching is perhaps the most important application of sorting.

Closest pairGiven n numbers, find the pair which are closest to each other.  

Once the numbers are sorted, the closest pair will be next to each other in sorted order,
so an O(n) linear scan completes the job.

Element uniquenessGiven a set of n items, are they all unique or are there any
duplicates?    

Sort them and do a linear scan to check all adjacent pairs.

This is a special case of closest pair above.

Frequency distribution - ModeGiven a set of n items, which element occurs the largest
number of times?   

Sort them and do a linear scan to measure the length of all adjacent runs.

Median and SelectionWhat is the kth largest item in the set?   

Once the keys are placed in sorted order in an array, the kth largest can be found in
constant time by simply looking in the kth position of the array.

How do you sort?

There are several different ideas which lead to sorting algorithms:

 Insertion - putting an element in the appropriate place in a sorted list yields a

larger sorted list.
 Exchange - rearrange pairs of elements which are out of order, until no such
pairs remain.
 Selection - extract the largest element form the list, remove it, and repeat.
 Distribution - separate into piles based on the first letter, then sort each pile.
 Merging - Two sorted lists can be easily combined to form a sorted list.

Selection Sort
In my opinion, the most natural and easiest sorting algorithm is selection sort, where
we repeatedly find the smallest element, move it to the front, then repeat...
* 5 7 3 2 8
2 * 7 3 5 8
2 3 * 7 5 8
2 3 5 * 7 8
2 3 5 7 * 8

If elements are in an array, swap the first with the smallest element- thus only one
array is necessary.

If elements are in a linked list, we must keep two lists, one sorted and one unsorted,
and always add the new element to the back of the sorted list.

Selection Sort Implementation

MODULE SimpleSort EXPORTS Main; (*1.12.94. LB*)
(* Sorting and text-array by selecting the smallest element *)

TYPE
Array = ARRAY [1..N] OF TEXT;
VAR
a: Array; (*the array in which to search*)
x: TEXT; (*auxiliary variable*)
last, (*last valid index *)
min: INTEGER; (* current minimum*)

BEGIN

...

FOR i:= FIRST(a) TO last - 1 DO

min:= i; (*index of smallest element*)
FOR j:= i + 1 TO last DO
IF Text.Compare(a[j], a[min]) = -1 THEN (*IF a[i] < a[min]*)
min:= j
END;
END; (*FOR j*)
x:= a[min]; (* swap a[i] and a[min] *)
a[min]:= a[i];
a[i]:= x;
END; (*FOR i*)

...

END SimpleSort.

The Complexity of Selection Sort

One interesting observation is that selection sort always takes the same time no matter
what the data we give it is! Thus the best case, worst case, and average cases are all
the same!

Intuitively, we make n iterations, each of which ``on average'' compares n/2, so we

should make about   comparisons to sort n items.

To do this more precisely, we can count the number of comparisons we make.

To find the largest takes (n-1) steps, to find the second largest takes (n-2) steps, to find
the third largest takes (n-3) steps, ... to find the last largest takes 0 steps.

An advantage of the big Oh notation is that fact that the worst case   time is
obvious - we have n loops of at most n steps each.

If instead of time we count the number of data movements, there are n-1, since there is
exactly one swap per iteration.

Insertion Sort

In insertion sort, we repeatedly add elements to a sorted subset of our data, inserting
the next element in order:

* 5 7 3 2 8
5 * 7 3 2 8
3 5 * 7 2 8
2 3 5 * 7 8
2 3 5 7 * 8

InsertionSort(A)
 for i = 1 to n-1 do

j=i

while (A[j] > A[j-1]) do swap(A[j],A[j-1])

In inserting the element in the sorted section, we might have to move many elements
to make room for it.

If the elements are in an array, we scan from bottom to top until we find the j such
that   , then move from j+1 to the end down one to make
room.

If the elements are in a linked list, we do the sequential search until we find where the
element goes, then insert the element there. No other elements need move!

Complexity of Insertion Sort

Since we do not necessarily have to scan the entire sorted section of the array, the
best, worst, and average cases for insertion sort all differ!

Best case: the element always gets inserted at the end, so we don't have to move
anything, and only compare against the last sorted element. We have (n-1) insertions,
each with exactly one comparison and no data moves per insertion!

What is this best case permutation? It is when the array or list is already sorted! Thus
insertion sort is a great algorithm when the data has previously been ordered, but
slightly messed up.

Worst Case Complexity

Worst case: the element always gets inserted at the front, so all the sorted elements
must be moved at each insertion. The ith insertion requires (i-1) comparisons and
moves so:
What is the worst case permutation? When the array is sorted in reverse order.

This is the same number of comparisons as with selection sort, but uses more
movements. The number of movements might get important if we were
sorting large records.

Average Case Complexity

Average Case: If we were given a random permutation, the chances of the ith
insertion requiring   comparisons are equal, and hence 1/i.

The expected number of comparisons is for the ith insertion is:

Summing up over all n keys,

So we do half as many comparisons/moves on average!

Can we use binary search to help us get below   time?

Mergesort and Quicksort

Lecture 17
Steven S. Skiena

Faster than O(   ) Sorting?

Can we find a sorting algorithm which does significantly better than comparing each
pair of elements? If not, we are doomed to quadratic time complexity....

Since sorting large numbers of items is such an important problem - an   

algorithm is the way to go!

Logarithms

It is important to understand deep in your bones what logarithms are and where they
come from.   

A logarithm is simply an inverse exponential function. Saying   is equivalent

to saying that   .

Exponential functions, like the amount owed on a n year mortgage at an interest rate
of   per year, are functions which grow distressingly fast. Thus inverse exponential
functions, ie. logarithms, grow refreshingly slowly.

Binary search is an example of an   algorithm. After each comparison, we can

throw away half the possible number of keys. Thus twenty comparisons suffice to find
any name in the million-name Manhattan phone book!

If you have an algorithm which runs in   time, take it, because this is
blindingly fast even on very large instances.

Properties of Logarithms

Recall the definition,   .

Asymptotically, the base of the log does not matter: 

Thus,   , and note
that   is just a constant.

Asymptotically, any polynomial function of n does not matter:Note that

since   , and   .

Any exponential dominates every polynomial. This is why we will seek to avoid

exponential time algorithms.

Federal Sentencing Guidelines

2F1.1. Fraud and Deceit; Forgery; Offenses Involving Altered or Counterfeit

Instruments other than Counterfeit Bearer Obligations of the United States.  

(a) Base offense Level: 6

(b) Specific offense Characteristics

(1) If the loss exceeded $2,000, increase the offense level as follows:
The federal sentencing guidelines are designed to help judges be consistent in
assigning punishment. The time-to-serve is a roughly linear function of the total level.

However, notice that the increase in level as a function of the amount of money you
steal grows logarithmically in the amount of money stolen.  

This very slow growth means it pays to commit one crime stealing a lot of money,
rather than many small crimes adding up to the same amount of money, because the
time to serve if you get caught is much less.

The Moral: ``if you are gonna do the crime, make it worth the time!''

Mergesort

Given two sorted lists with a total of n elements, at most n-1 comparisons are required
to merge them into one sorted list. Repeatedly compare the top elements on each list.
Example:   and   .

No more comparisons are needed once the list is empty.

Fine, but how do we get the smaller sorted lists to start with? We do merges of even
smaller lists!

Working backwards, we eventually get to lists of one element, which are by definition

sorted!

Mergesort Example

Note that on each iteration, the size of the sorted lists doubles, form 1 to 2 to 4 to 8 to
16 ...to n.

How many doublings (or iterations) does it take before the entire array of size n is
sorted? Answer:   .

How much work do we do per iteration?

In merging the lists of 1 element, we have   merges, each requiring 1

comparison, for a total of   comparisons.
In merging the lists of 2 elements, we have   merges, each requiring at most 3
comparisons, for a total of   comparisons.

...

In merging the lists of 2 elements, we have   merges, each requiring at

most   comparisons, for a total of   .

This is always less than n per stage!!! If we make at most n comparisons in each

of   stages, we make at most   comparisons in total!

Make sure you understand why mergesort is   - it is the conceptually

simplest   algorithm we will see.

Space Requirements for Mergesort

How much extra space (over the space used to represent the input elements) do we
need to do mergesort?

It is easy to merge two sorted linked lists without using any extra space.

However, to merge two sorted arrays (or portions of an array), we must use a third
array to store the result of the merge. This avoids steping on elements we have not
needed yet:

Example: Merge ((4,5,6), (1,2,3)).

QuickSort

Although Mergesort is   , it is somewhat inconvienient to implementate

using arrays, since we need space to merge.

In practice, the fastest sorting algorithm is Quicksort, which uses partitioning as its
main idea.

Example: Pivot about 10.

17 12 6 19 23 8 5 10 - before

6 8 5 10 23 19 12 17 - after

Partitioning places all the elements less than the pivot in the left part of the array, and
all elements greater than the pivot in the right part of the array. The pivot fits in the
slot between them.

Note that the pivot element ends up in the correct place in the total order!

Partitioning the elements

Once we have selected a pivot element, we can partition the array in one linear scan,
by maintaining three sections of the array: < pivot, > pivot, and unexplored.

Example: pivot about 10

| 17 12 6 19 23 8 5 | 10
| 5 12 6 19 23 8 | 17
5 | 12 6 19 23 8 | 17
5 | 8 6 19 23 | 12 17
5 8 | 6 19 23 | 12 17
5 8 6 | 19 23 | 12 17
5 8 6 | 23 | 19 12 17
5 8 6 ||23 19 12 17
5 8 6 10 19 12 17 23

As we scan from left to right, we move the left bound to the right when the element is
less than the pivot, otherwise we swap it with the rightmost unexplored element and
move the right bound one step closer to the left.

Since the partitioning step consists of at most n swaps, takes time linear in the number
of keys. But what does it buy us?

1. The pivot element ends up in the position it retains in the final sorted order.
2. After a partitioning, no element flops to the other side of the pivot in the final
sorted order.

Thus we can sort the elements to the left of the pivot and the right of the pivot
independently!

This gives us a recursive sorting algorithm, since we can use the partitioning approach
to sort each subproblem.

Quicksort Implementation
MODULE Quicksort EXPORTS Main; (*18.07.94. LB*)
(* Read in an array of integers, sort it using the Quicksort algorithm,
and output the array.

See Chapter 14 for the explanation of the file handling and Chapter
15 for exception handling, which is used in this example.
*)

IMPORT SIO, SF;

VAR
out: SIO.Writer;

TYPE
ElemType = INTEGER;
VAR
array: ARRAY [1 .. 10] OF ElemType;

PROCEDURE InArray(VAR a: ARRAY OF ElemType) RAISES {SIO.Error} =

(*Reads a sequence of numbers. Passes SIO.Error for bad file format.*)
VAR
in:= SF.OpenRead("vector"); (*open input file*)
BEGIN
FOR i:= FIRST(a) TO LAST(a) DO a[i]:= SIO.GetInt(in) END;
END InArray;

PROCEDURE OutArray(READONLY a: ARRAY OF ElemType) =

(*Outputs an array of numbers*)
BEGIN
FOR i:= FIRST(a) TO LAST(a) DO SIO.PutInt(a[i], 4, out) END;
SIO.Nl(out);
END OutArray;

PROCEDURE Quicksort(VAR a: ARRAY OF ElemType; left, right: CARDINAL) =

VAR
i, j: INTEGER;
x, w: ElemType;
BEGIN

(*Partitioning:*)
i:= left; (*i iterates upwards from left*)
j:= right; (*j iterates down from right*)
x:= a[(left + right) DIV 2]; (*x is the middle element*)
REPEAT
WHILE a[i] < x DO INC(i) END; (*skip elements < x in left part*)
WHILE a[j] > x DO DEC(j) END; (*skip elements > x in right part*)
IF i <= j THEN
w:= a[i]; a[i]:= a[j]; a[j]:= w; (*swap a[i] and a[j]*)
INC(i);
DEC(j);
END; (*IF i <= j*)
UNTIL i > j;

(*recursive application of partitioning to subarrays:*)

IF left < j THEN Quicksort(a, left, j) END;

 IF i < right THEN Quicksort(a, i, right) END;

END Quicksort;

BEGIN
TRY (*grasps bad file format*)
InArray(array); (*read an array in*)
out:= SF.OpenWrite(); (*create output file*)
OutArray(array); (*output the array*)
Quicksort(array, 0, NUMBER(array) - 1); (*sort the array*)
OutArray(array); (*display the array*)
SF.CloseWrite(out); (*close output file to make it
permanent*)
EXCEPT
SIO.Error => SIO.PutLine("bad file format");
END; (*TRY*)
END Quicksort.

Best Case for Quicksort

Since each element ultimately ends up in the correct position, the algorithm correctly
sorts. But how long does it take?

The best case for divide-and-conquer algorithms comes when we split the input as

evenly as possible. Thus in the best case, each subproblem is of size n/2.

The partition step on each subproblem is linear in its size. Thus the total effort in

partitioning the   problems of size   is O(n).

The recursion tree for the best case looks like this:

The total partitioning on each level is O(n), and it take   levels of perfect partitions
to get to single element subproblems. When we are down to single elements, the
problems are sorted. Thus the total time in the best case is   .

Worst Case for Quicksort

Suppose instead our pivot element splits the array as unequally as possible. Thus
instead of n/2 elements in the smaller half, we get zero, meaning that the pivot
element is the biggest or smallest element in the array.

Now we have n-1 levels, instead of   , for a worst case time of   , since the
first n/2 levels each have   elements to partition.
Thus the worst case time for Quicksort is worse than Heapsort or Mergesort.

To justify its name, Quicksort had better be good in the average case. Showing this
requires some fairly intricate analysis.

The divide and conquer principle applies to real life. If you will break a job into
pieces, it is best to make the pieces of equal size!

Intuition: The Average Case for Quicksort

The book contains a rigorous proof that quicksort is   in the average case.
I will instead give an intuitive, less formal explanation of why this is so.

Suppose we pick the pivot element at random in an array of n keys.

Half the time, the pivot element will be from the center half of the sorted array.

Whenever the pivot element is from positions n/4 to 3n/4, the larger remaining
subarray contains at most 3n/4 elements.

If we assume that the pivot element is always in this range, what is the maximum
number of partitions we need to get from n elements down to 1 element?

good partitions suffice.

At most   levels of decent partitions suffices to sort an array of n elements.  

But how often when we pick an arbitrary element as pivot will it generate a decent
partition?

Since any number ranked between n/4 and 3n/4 would make a decent pivot, we get
one half the time on average.
If we need   levels of decent partitions to finish the job, and half of random
partitions are decent, then on average the recursion tree to quicksort the array
has   levels.

Since O(n) work is done partitioning on each level, the average time is   .

More careful analysis shows that the expected number of comparisons

is   .

What is the Worst Case?

The worst case for Quicksort depends upon how we select our partition or pivot
element. If we always select either the first or last element of the subarray, the worst-
case occurs when the input is already sorted!
A B D F H J K
B D F H J K
D F H J K
F H J K
H J K
J K
K

Having the worst case occur when they are sorted or almost sorted is very bad, since
that is likely to be the case in certain applications.

To eliminate this problem, pick a better pivot:

1. Use the middle element of the subarray as pivot.

2. Use a random element of the array as the pivot.
3. Perhaps best of all, take the median of three elements (first, last, middle) as the
pivot. Why should we use median instead of the mean?

Whichever of these three rules we use, the worst case remains   . However,
because the worst case is no longer a natural order it is much more difficult to occur.

Is Quicksort really faster than Mergesort?

Since Mergesort is   and selection sort is   , there is no debate about
which will be better for decent-sized files.
But how can we compare two   algorithms to see which is faster? Using the
RAM model and the big Oh notation, we can't!

When Quicksort is implemented well, it is typically 2-3 times faster than mergesort or
heapsort. The primary reason is that the operations in the innermost loop are simpler.
The best way to see this is to implement both and experiment with different inputs.

Since the difference between the two programs will be limited to a multiplicative
constant factor, the details of how you program each algorithm will make a big
difference.

If you don't want to believe me when I say Quicksort is faster, I won't argue with you.
It is a question whose solution lies outside the tools we are using. The best way to tell
is to implement them and experiment.

Combining Quicksort and Insertion Sort

When we compare the expected number of comparisons for Quicksort + Insertion

sort, a funny thing happens for small n:

Why not take advantage of this, and switch over to insertion sort when the size of the
subarray falls below a certain threshhold?

Why not indeed? But how do we find the right switch point to optimize performance?
Experiments are more useful than analysis here.

Randomization

Suppose you are writing a sorting program, to run on data given to you by your worst
enemy. Quicksort is good on average, but bad on certain worst-case instances.  

If you used Quicksort, what kind of data would your enemy give you to run it on?
Exactly the worst-case instance, to make you look bad.
But instead of picking the median of three or the first element as pivot, suppose you
picked the pivot element at random.

Now your enemy cannot design a worst-case instance to give to you, because no
matter which data they give you, you would have the same probability of picking a
good pivot!

Randomization is a very important and useful idea. By either picking a random pivot
or scrambling the permutation before sorting it, we can say:

``With high probability, randomized quicksort runs in   time.''

Where before, all we could say is:

``If you give me random input data, quicksort runs in expected   time.''

Since the time bound how does not depend upon your input distribution, this means
that unless we are extremely unlucky (as opposed to ill prepared or unpopular) we will
certainly get good performance.

Randomization is a general tool to improve algorithms with bad worst-case but good
average-case complexity.

The worst-case is still there, but we almost certainly won't see it.

Priority Queues and Heapsort

Lecture 18
Steven S. Skiena

Who's Number 2?

In most sports playoffs, a single elimination tournament is used to decide the

championship.

The Marlins were clearly the best team in the 1997 World Series, since they were the
only one without a loss. But who is number 2? The Giants, Braves, and Indians all
have equal claims, since only the champion beat them!
Each game can be thought of as a comparison. Given n keys, we would like to
determine the k largest values. Can we do better than just sorting all of them?

In the tournament example, each team represents an leaf of the tree and each game is

an internal node of the tree. Thus there are n-1 games/comparisons for n teams/leaves.

Note that the champion is identified even though no team plays more than   
games!

Lewis Carroll, author of ``Alice in Wonderland'', studied this problem in the 19th
century in order to design better tennis tournaments!

We will seek a data structure which will enable us to repeatedly identify

the largest key, and then delete it to retrieve the largest remaining key.

This data structure is called a heap, as in ``top of the heap''.

Binary Heaps

A binary heap is defined to be a binary tree with a key in each node such that:

1. All leaves are on, at most, two adjacent levels.

2. All leaves on the lowest level occur to the left, and all levels except the lowest
are completely filled.
3. The key in the root is   all its children, and the left and right subtrees are again
binary heaps. (This is a recursive definition)

Conditions 1 and 2 specify the shape of the tree, while condition 3 describes the
labeling of the nodes tree.

Unlike the tournament example, each label only appears on one node.

Note that heaps are not binary search trees, but they are binary trees.

Heap Test

Where is the largest element in a heap?

Answer - the root.

Where is the second largest element?

Answer - as the root's left or right child.

Where is the smallest element?

Answer - it is one of the leaves.

Can we do a binary search to find a particular key in a heap?

Answer - No! A heap is not a binary search tree, and cannot be effectively used for
searching.

Why Do Heaps Lean Left?

As a consequence of the structural definition of a heap, each of the n items can be

assigned a number from 1 to n with the property that the left child of node
number k has a number 2k and the right child number 2k+1.

Thus we can store the heap in an n element array without pointers!

If we did not enforce the left constraint, we might have holes, and need room for   
elements to store n things.

This implicit representation of trees saves memory but is less flexible than using

pointers. For this reason, we will not be able to use them when we discuss binary
search trees.

Constructing Heaps

Heaps can be constructed incrementally, by inserting new elements into the left-most
open spot in the array.

If the new element is greater than its parent, swap their positions and recur.

Since at each step, we replace the root of a subtree by a larger one, we preserve the
heap order.

Since all but the last level is always filled, the height h of an n element heap is
bounded because:
so   .

Doing n such insertions takes   , since each insertion takes at

most   time.

Deleting the Root

The smallest (or largest) element in the heap sits at the root.

Deleting the root can be done by replacing the root by the nth key (which must be a
leaf) and letting it percolate down to its proper position!

The smallest element of (1) the root, (2) its left child, and (3) its right child is moved
to the root. This leaves at most one of the two subtrees which is not in heap order, so
we continue one level down.

After   steps of O(1) time each, we reach a leaf, so the deletion is completed
in   time.

This percolate-down operation is called often Heapify, for it merges two heaps with a
new root.

Heapsort

An initial heap can be constructed out on n elements by incremental insertion

in   time:

Build-heap(A)

for i = 2 to n do
 HeapInsert(A[i], A)

Exchanging the maximum element with the last element and calling heapify
repeatedly gives an   sorting algorithm, named Heapsort.

Heapsort(A)

Build-heap(A)

for i = n to 1 do

swap(A[1],A[i])

n = n - 1

Heapify(A,1)

Advantages of heapsort include:

 No extra space (Quicksort needs a stack)

 No worst case trouble.
 Simpler to get fast and correct than Quicksort.

The Lesson of Heapsort

Always ask yourself, ``Can we use a different data structure?''

Selection sort scans throught the entire array, repeatedly finding the smallest
remaining element.

For i = 1 to n

A: Find the smallest of the first n-i+1 items.

B: Pull it out of the array and put it first.

Using arrays or unsorted linked lists as the data structure, operation A takes O(n) time
and operation B takes O(1).

Using heaps, both of these operations can be done within   time, balancing
the work and achieving a better tradeoff.

Priority Queues

A priority queue is a data structure on sets of keys supporting the operations: Insert(S,

x) - insert x into set S, Maximum(S) - return the largest key in S, and ExtractMax(S) -
return and remove the largest key in S

These operations can be easily supported using a heap.

 Insert - use the trickle up insertion in   .

 Maximum - read the first element in the array in O(1).
 Extract-Max - delete first element, replace it with the last, decrement the
element counter, then heapify in   .

Application: Heaps as stacks or queues

 In a stack, push inserts a new item and pop removes the most recently pushed

item.
  In a queue, enqueue inserts a new item and dequeue removes the least recently
enqueued item.

Both stacks and queues can be simulated by using a heap, when we add a
new time field to each item and order the heap according it this time field.

 To simulate the stack, increment the time with each insertion and put the
maximum on top of the heap.
 To simulate the queue, decrement the time with each insertion and put the
maximum on top of the heap (or increment times and keep the minimum on
top)

This simulation is not as efficient as a normal stack/queue implementation, but it is a

cute demonstration of the flexibility of a priority queue.

Discrete Event Simulations

In simulations of airports, parking lots, and jai-alai - priority queues can be used to
maintain who goes next.

In a simulation, we often need to schedule events according to a clock. When

someone is born, we may then immediately decide when they will die, and we will
have to be reminded when to bury them!

The stack and queue orders are just special cases of orderings. In real life, certain
people cut in line.

Sweepline Algorithms in Computational Geometry

In the priority queue, we will store the points we have not yet encountered, ordered
by x coordinate. and push the line forward one stop at a time.

Greedy Algorithms

In greedy algorithms, we always pick the next thing which locally maximizes our
score. By placing all the things in a priority queue and pulling them off in order, we
can improve performance over linear search or sorting, particularly if the weights
change.  

Example: Sequential strips in triangulations.

 Sequential and Binary Search
Lecture 19
Steven S. Skiena

Sequential Search

The simplest algorithm to search a dictionary for a given key is to test successively
against each element.

This works correctly regardless of the order of the elements in the list. However, in
the worst case all elements might have to be tested.
Procedure Search(head:pointer, key:item):pointer;
Var
p:pointer;
found:boolean;
Begin
found:=false;
p:=head;
While (p # NIL) AND (not found) Do
Begin
If (p^.info = key) then
found = true;
Else
p = p^.next;
End;
return p;
END;

With and Without Sentinels

A sentinel is a value placed at the end of an array to insure that the normal case of
searching returns something even if the item is not found. It is a way to simplify
coding by eliminating the special case.
MODULE LinearSearch EXPORTS Main; (*1.12.94. LB*)
(* Linear search without a sentinel *)

...

i:= FIRST(a);
WHILE (i <= last) AND NOT Text.Equal(a[i], x) DO INC(i) END;

IF i > last THEN

SIO.PutText("NOT found");
ELSE
SIO.PutText("Found at position: ");
 SIO.PutInt(i)
END; (*IF i > last*)
SIO.Nl();
END LinearSearch.

The sentinel insures that the search will eventually succeed:

MODULE SentinelSearch EXPORTS Main; (*27.10.93. LB*)

(* Linear search with sentinel. *)

...

(* Do search *)
a[LAST(a)]:= x; (*sentinel at position N+1*)
i:= FIRST(a);
WHILE x # a[i] DO INC(i) END;

(* Output result *)
IF i = LAST(a) THEN
SIO.PutText("NOT found");
ELSE
SIO.PutText("Found at position: "); SIO.PutInt(i)
END;
SIO.Nl();
END SentinelSearch.

Weighted Sequential Search

Sometimes sequential search is not a bad algorithm, especially when the list isn't long.
After all, sequential search is easier to implement than binary search, and does not
require the list to be sorted.

However, if we are going to do a sequential search, what order do we want the

elements? Sorted order in a linked list doesn't really help, except maybe to help us
stop early if the item isn't in the list.

Suppose you were organizing your personal phone book for sequential search. You
would want your most frequently called friends to be at the front: In sequential
search, you want the keys ordered by frequency of use!

Why? If   is the probability of searching for the ith key, which is a distance   from
the front, the expected search time is
which is minimized by placing the list in decreasing probability of access order.

For the list (Cheryl,0.4), (Lisa,0.25), (Lori,0.2), (Lauren,0.15), the expected search

time is:

If access probability had been uniform, the expected search time would have been

So I win using this order, and win even more if the access probabilities are furthered
skewed.

But how do I find the probabilities?

Self-Organizing Lists

Since it is often impractical to compute usage frequencies, and because usage

frequencies often change in the middle of a program (locality), we would like our data
structure to automatically adjust to the distribution.

Such data structures are called self-organizing.

The idea is to use a heuristic to move an element forward in the list whenever it is
accessed. There are two possibilities:

 Move forward one is the ``conservative'' approach. (1,2,3,4,5) becomes

(1,2,4,3,5) after a Find(4).
 Move to front is the ``liberal'' approach. (1,2,3,4,5) becomes (4,1,2,3,5) after
a Find(4).

Which Heuristic is Better?

Move-forward-one can get caught in traps which won't fool move-to-front:

For list (1,2,3,4,5,6,7,8), the queries Find(8), Find(7), Find(8), Find(7), ... will search
the entire list every time. With move-to-front, it averages only two comparisons per
query!

In fact, it can be shown that the total search time with move-to-front is never more
than twice the time if you knew the actual probabilities in advance!!
We will see self-organization again later in the semester when we talk about splay
trees.

Let's Play 20 Questions!

1. 11.
2. 12.
3. 13.
4. 14.
5. 15.
6. 16.
7. 17.
8. 18.
9. 19.
10. 20.

Binary Search

Binary Search is an incredibly powerful technique for searching an ordered list. It is

familiar to everyone who uses a telephone book!

The basic algorithm is to find the middle element of the list, compare it against the
key, decide which half of the list must contain the key, and repeat with that half.

Two requirements to support binary search:

 Random access of the list elements, so we need arrays instead of linked lists.
 The array must contain elements in sorted order by the search key.

Why Do Twenty Questions Suffice?

With one question, I can distinguish between two words: A and B; ``Is the key   ?''

With two questions, I can distinguish between four words: A,B,C,D; ``Is the   ?''

Each question I ask em doubles the number of words I can search in my dictionary.

 , which is much larger than any portable dictionary!

Thus I could waste my first two questions

because   .
Exponents and Logs

Recall the definitions of exponent and logarithm from high school:

Thus exponentiation and logarithms are inverse functions, since   .

Note that the logarithm of a big number is a much smaller number.

Thus the number of questions we must ask is the base two logarithm of the size of the
dictionary.

Implementing Binary Search

Although the algorithm is simple to describe informally, it is tricky to produce a

working binary search function. The first published binary search algorithm appeared
in 1946, but the first correct published program appeared in 1962!

The difficulty is maintaining the following two invariants with each iteration:

 The key must always remain between the low and high indices.
 The low or high indice must advance each iteration.

The boundary cases are very tricky: zero elements left, one elements left, two
elements left, and an even or odd number of elements!

Versions of Binary Search

There are at least two different versions of binary search, depending upon whether we
want to test for equality at each query or only at the end.

For the later, suppose we want to search for ``k'':

iteration bottom top mid
---------------------------------------
1 2 14 (1+14)/2=7
2 1 7 (1+7)/2=4
3 5 7 (5+7)/2=6
4 6 7 (7+7)/2=7
Since   , 7 is the right spot. However, we must now test if entry[7]='k'.
If not, the item isn't in the array.

Alternately, we can test for equality at each comparison. Suppose we search for ``c'':
iteration bottom top mid
------------------------------------
1 1 14 (1+14)/2 = 7
2 1 6 (1+6)/2 = 3
3 1 2 (1+2)/2 = 1
4 2 2 (2+2)/2 = 2

Now it will be found!

Recursive Binary Search Implementation

PROCEDURE Search( READONLY array: ARRAY [0 .. MaxInd - 1] OF INTEGER;
left, right: [0 .. MaxInd - 1];
argument: INTEGER): [0..MaxInd] =
(*Implements binary search in an array*)
VAR
middle := left + (right - left) DIV 2;

BEGIN (* binary search *)

IF argument = array[middle] THEN (*found*)
RETURN middle
ELSIF argument < array[middle] THEN (*search in left half*)
IF left < middle THEN
RETURN Search(array, left, middle - 1, argument)
ELSE (*left boundary reaches middle: not
found*)
RETURN MaxInd
END (*IF left < middle*)
ELSE (*search in right half*)
IF middle < right THEN
RETURN Search(array, middle + 1, right, argument)
ELSE (*middle reaches right boundary: not
found*)
RETURN MaxInd
END (*IF middle < right*)
END (*IF argument = array[middle]*)
END Search;

Arrays and Access Formulas

Lecture 20
Steven S. Skiena
One-dimensional Arrays

The easiest way to view a one - dimensional array is as a contiguous block of memory
locations of length (# of array elements)   (size of each element)

Because the size (in bytes) of each element is the same, the compiler can
translated A[500] into the address of the record

If A points to the first location of   of k-byte records, then

This is the access formula for a one-dimensional array.

Two-Dimensional Arrays

How does the compiler know where to store element A[i,j] of a two-dimensional

array? By chopping the matrix into rows, it can be stored like a one- dimensional
array:

If A points to the first location of A[l1..h1,l2..h2] of k-byte records, then:

Is this access formula for row-major or column-major order, assuming the first index
gives the row?

For three dimensions, cut the matrix into two dimensional slabs, and use the previous
formula. For k-dimensional arrays, we can find a similar formula by induction.

Thus we can access any element in a k-dimensional array in O(k) time, which is
constant for any reasonably dimension.

Fortran stores its arrays in column-major order, while most other languages use row-
major order. But why might we really need to know what is going on under the hood?

In C language, pointers are usually used to cruise through arrays. Cruising through a
2D array meaningfully requires knowing the order of the elements.

Also, in a computer with virtual memory or a cache, it is often faster to access

elements if they are close to the last one we have read. Knowing the access function
lets us choose the right way to order nested loops.
(*row-major*)
(*column-major*)

Do i=1 to n
Do j=1 to n

Do j=1 to n
Do i=1 to n

A[i,j] = 0
A[i,j] = 0

Triangular Tables

By playing with our own access functions we can build efficient arrays of whatever
shape we want, including triangular and banded arrays.

Triangular tables prove useful for representing any symmetric function, such as the
distance from A to B, D[a,b] = D[b,a]. Thus we can save almost half the memory of a
rectangular array by storing it as a triangle

The access formula is:

since the identity   can be proven by induction.

Faster than Binary Search?

Binary search takes   time to find a particular key in a sorted array. It can be
shown that, in the worst case, no faster algorithm exists. So how might we do faster?
This is not a contradiction. Suppose we wanted to search on a field containing an ID
number between 1 and the number of records. Rather than doing a binary search on
this field, why not use it as an index in an array!

Accessing such an array element is O(1) instead of   !

Interpolation Search

Binary search is only optimal when you know nothing about your data except that it is
sorted!

When you look up AAA in the telephone book, you don't start in the middle. We use
our understanding of how things are named in the real world to choose where to prove
next. Such an algorithm is called an interpolation search, since we are
interpolating(guessing) where the key should be.

Interpolation search is only as good as our guesses. If we do not understand the data
as well as you think, interpolation search can be very slow - recall the Shifflett's of
Charlottesville!

With interpolation search, the cost of making a good guess might overwhelm the
reduction in the number of guesses, so watch out!

The Key Ideas on Access Formulas

A pointer tells us exactly where in memory an item is.

An array reference A[i] lets us quickly calculate exactly where the ith element of A is
in memory, knowing only i, the starting location of A, and the size of each array item.

Any time we can compute the exact position for an item in memory by a simple
access formula, we can find it as quickly as we can compute the formula!

Must Array Indices be Integers?

We have seen that binary search is slower than table lookup. Why can't the entire
world be one big array?

One reason is that many of the fields we wish to search on are not integers, for
example, names in a telephone book. What address in the machine is defined by
``Skiena''?
To compute the appropriate address we need a function to map arbitrary keys to
addresses. Such hash functions form the basis of an important search
technique, hashing!

Hashing
Lecture 21
Steven S. Skiena

Hashing

One way to convert form names to integers is to use the letters to form a base
``alphabet-size'' number system:

To convert ``STEVE'' to a number, observe that e is the 5th letter of the alphabet, s is
the 19th letter, t is the 20th letter, and v is the 22nd letter.

Thus

``Steve'' 

Thus one way we could represent a table of names would be to set aside an array big
enough to contain one element for each possible string of letters, then store data in the
elements corresponding to real people. By computing this function, it tells us where
the person's phone number is immediately!!

What's the Problem?

Because we must leave room for every possible string, this method will use an
incredible amount of memory. We need a data structure to represent a sparse table,
one where almost all entries will be empty.

We can reduce the number of boxes we need if we are willing to put more than one
thing in the same box!

Example: suppose we use the base alphabet number system, then take the
remainder 
Now the table is much smaller, but we need a way to deal with the fact that more than
one, (but hopefully every few) keys can get mapped to the same array element.

The Basics of Hashing

The basics of hashing is to apply a function to the search key so we can

determine where the item is without looking at the other items. To make the table of
reasonable size, we must allow for collisions, two distinct keys mapped to the same
location.

 We a special hash function to map keys (hopefully uniformly) to integers in a

certain range.
 We set up an array as big as this range, and use the valve of the function as the
index to store the appropriate key. Special care must be taken to handle
collisions when they occur.

There are several clever techniques we will see to develop good hash functions and
deal with the problems of duplicates.

Hash Functions

The verb ``hash'' means ``to mix up'', and so we seek a function to mix up keys as well
as possible.

The best possible hash function would hash m keys into n ``buckets'' with no more
than   keys per bucket. Such a function is called a perfect hash function

How can we build a hash function?

Let us consider hashing character strings to integers. The ORD function returns the
character code associated with a given character. By using the ``base character size''
number system, we can map each string to an integer.

The First Three SSN digits Hash

The first three digits of the Social Security Number  

The last three digits of the Social Security Number

What is the big picture?

 1. A hash function which maps an arbitrary key to an integer turns searching into
array access, hence O(1).
2. To use a finite sized array means two different keys will be mapped to the same
place. Thus we must have some way to handle collisions.
3. A good hash function must spread the keys uniformly, or else we have a linear
search.

Ideas for Hash Functions

 Truncation - When grades are posted, the last four digits of your SSN are used,
because they distribute students more uniformly than the first four digits.
 Folding - We should get a better spread by factoring in the entire key. Maybe
subtract the last four digits from the first five digits of the SSN, and take the
absolute value?
 Modular Arithmetic - When constructing pseudorandom numbers, a good trick
for uniform distribution was to take a big number mod the size of our range.
Because of our roulette wheel analogy, the numbers tend to get spread well if
the tablesize is selected carefully.

Prime Numbers are Good Things

Suppose we wanted to hash check totals by the dollar value in pennies mod 1000.
What happens?

 ,   , and 

Prices tend to be clumped by similar last digits, so we get clustering.

If we instead use a prime numbered Modulus like 1007, these clusters will get
broken:   ,   , and   .

In general, it is a good idea to use prime modulus for hash table size, since it is less
likely the data will be multiples of large primes as opposed to small primes - all
multiples of 4 get mapped to even numbers in an even sized hash table!

The Birthday Paradox

No matter how good our hash function is, we had better be prepared for collisions,
because of the birthday paradox.
Assuming 365 days a year, what is the probability that exactly two people share a
birthday? Once the first person has fixed their birthday, the second person has 365
possible days to be born to avoid a collision, or a 365/365 chance.

With three people, the probability that no two share is   . In

general, the probability of there being no collisions after n insertions into an m-
element table is

When m = 366, this probability sinks below 1/2 when N = 23 and to almost 0
when   .

The moral is that collisions are common, even with good hash functions.

What about Collisions?

No matter how good our hash functions are, we must deal with collisions. What do we
do when the spot in the table we need is occupied?

 Put it somewhere else! - In open addressing, we have a rule to decide where to

put it if the space is already occupied.
 Keep a list at each bin! - At each spot in the hash table, keep a linked list of
keys sharing this hash value, and do a sequential search to find the one we
need. This method is called chaining.

Collision Resolution by Chaining

The easiest approach is to let each element in the hash table be a pointer to a list of
keys.  

Insertion, deletion, and query reduce to the problem in linked lists. If the n keys are
distributed uniformly in a table of size m/n, each operation takes O(m/n) time.

Chaining is easy, but devotes a considerable amount of memory to pointers, which

could be used to make the table larger. Still, it is my preferred method.

Open Addressing
We can dispense with all these pointers by using an implicit reference derived from a
simple function:

If the space we want to use is filled, we can examine the remaining locations:

1. Sequentially 

2. Quadratically 
3. Linearly 

The reason for using a more complicated scheme is to avoid long runs from similarly
hashed keys.

Deletion in an open addressing scheme is ugly, since removing one element can break
a chain of insertions, making some elements inaccessible.

Performance on Set Operations

With either chaining or open addressing:

 Search - O(1) expected, O(n) worst case.

 Insert - O(1) expected, O(n) worst case.
 Delete - O(1) expected, O(n) worst case.

Pragmatically, a hash table is often the best data structure to maintain a dictionary.
However, the worst-case running time is unpredictable.

The best worst-case bounds on a dictionary come from balanced binary trees, such as
red-black trees.

Tree Structures
Lecture 21
Steven S. Skiena

Trees
``I think that I shall never see a poem as lovely as a tree.
Poems are wrote by fools like me, but only G-d can make a tree.''
- Joyce Kilmer

We have seen many data structures which allow fast search, but not fast, flexible
update.

Sorted Tables -   search, O(n) insertion, O(n) deletion.

Hash Tables - The number of insertions are essentially bounded by the table size,
which must be specified in advance. Worst case O(n) search.

Binary trees will enable us to search, insert, and delete fast, without predefining the
size of our data structure!

How can we get this flexibility?

The only data structure we have seen which allows fast insertion/ deletion is
the linked list, with updates in O(1) time but search in O(n) time.

To get   search time, we used binary search, meaning we always had a

choice of two next elements to look at.

To combine these ideas, we want a ``linked list'' with two pointers per node! This is
the basic idea behind search trees!

Rooted Trees

We can use a recursive definition to specify what we mean by a ``rooted tree''.

A rooted tree is either (1) empty, or (2) consists of a node called the root, together
with two rooted trees called the left subtree and right subtree of the root.

A binary tree is a rooted tree where each node has at most two descendants, the left
child and the right child.

A binary tree can be implemented where each node has left and right pointer fields, an

(optional) parent pointer, and a data field.

Rooted trees in Real Life

Rooted trees can be used to model corporate heirarchies and family trees.

Note the inherently recursive structure of rooted trees. Deleting the root gives rise to a
certain number of smaller subtrees.

In a rooted tree, the order among ``brother'' nodes matters. Thus left is different from
right. The five distinct binary trees with five nodes:

Binary Search Trees

A binary search tree is a binary tree where each node contains a key such that:

 All keys in the left subtree precede the key in the root.
 All keys in the right subtree succeed the key in the root.
 The left and right subtrees of the root are again binary search trees.

Left: A binary search tree. Right: A heap but not a binary search tree.

For any binary tree on n nodes, and any set of n keys, there is exactly one labeling to
make it a binary search tree!!

Binary Tree Search

Searching a binary tree is almost like binary search! The difference is that instead of
searching an array and defining the middle element ourselves, we just follow the
appropriate pointer!

The type declaration is simply a linked list node with another pointer. Left and right
pointers are identical types.
TYPE
T = BRANDED REF RECORD
key: ElemT;
left, right: T := NIL;
END; (*T*)

Dictionary search operations are easy in binary trees. The algorithm works because
both the left and right subtrees of a binary search tree are binary search trees -
recursive structure, recursive algorithm.

Search Implementation
PROCEDURE Search(tree: T; e: ElemT): BOOLEAN =
(*Searches for an element e in tree.
Returns TRUE if present, else FALSE*)
BEGIN
IF tree = NIL THEN
RETURN FALSE (*not found*)
ELSIF tree.key = e THEN
RETURN TRUE (*found*)
ELSIF e < tree.key THEN
RETURN Search(tree.left, e) (*search in left tree*)
ELSE
RETURN Search(tree.right, e) (*search in right tree*)
END; (*IF tree...*)
END Search;

This takes time proportional to the height of the tree, O(h). Good, balanced trees have
height   , while bad, unbalanced trees have height O(n).

Building Binary Trees

To insert a new node into an existing tree, we search for where it should be, then
replace that NIL pointer with a pointer to the new node.

Each NIL pointer defines a gap in the space of keys!

The pointer in the parent node must be modified to remember where we put the new
node.

Insertion Routine
PROCEDURE Insert(VAR tree: T; e: ElemT) =
BEGIN
IF tree = NIL THEN
tree:= NEW(T, key:= e); (*insert at proper place*)
ELSIF e < tree.key THEN
Insert(tree.left, e) (*search place in left tree*)
ELSE
Insert(tree.right, e) (*search place in right tree*)
END; (*IF tree...*)
END Insert;

Tree Shapes and Sizes

Suppose we have a binary tree with n nodes.

How many levels can it have? At least   and at most n.

How many pointers are in the tree? There are n nodes in tree, each of which has 2
pointers, for a total of 2n pointers regardless of shape.
How many pointers are NIL, i.e ``wasted''? Except for the root, each node in the tree
is pointed to by one tree pointer Thus the number of NILs
is   , for   .

Traversal of Binary Trees

How can we print out all the names in a family tree?

An essential component of many algorithms is to completely traverse a tree data

structure. The key is to make sure we visit each node exactly once.

The order in which we explore each node and its children matters for many
applications.

There are six permutations of {left, right, node} which define traversals. The most
interesting traversals are inorder {left, node, right}, preorder {node, left,
right}, postorder {left, right, node},

Why do we care about different traversals? Depending on what the tree represents,
different traversals have different interpretations.

An in-order traversals of a binary serach tree sorts the keys!

Inorder traversal: 748251396, Preorder traversal: 124785369, Postorder traversal:

784529631

Reverse Polish notation is simply a post order traversal of an expression tree, like the
one below for expression 2+3*4+(3*4)/5.
PROCEDURE Traverse(tree: T; action: Action;
order := Order.In; direction := Direction.Right) =

PROCEDURE PreL(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
action(x.key, depth);
PreL(x.left, depth + 1);
PreL(x.right, depth + 1);
END; (*IF x # NIL*)
END PreL;

PROCEDURE PreR(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
action(x.key, depth);
PreR(x.right, depth + 1);
 PreR(x.left, depth + 1);
END; (*IF x # NIL*)
END PreR;

PROCEDURE InL(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
InL(x.left, depth + 1);
action(x.key, depth);
InL(x.right, depth + 1);
END; (*IF x # NIL*)
END InL;

PROCEDURE InR(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
InR(x.right, depth + 1);
action(x.key, depth);
InR(x.left, depth + 1);
END; (*IF x # NIL*)
END InR;

PROCEDURE PostL(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
PostL(x.left, depth + 1);
PostL(x.right, depth + 1);
action(x.key, depth);
END; (*IF x # NIL*)
END PostL;

PROCEDURE PostR(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
PostR(x.right, depth + 1);
PostR(x.left, depth + 1);
action(x.key, depth);
END; (*IF x # NIL*)
END PostR;

BEGIN (*Traverse*)
IF direction = Direction.Left THEN
CASE order OF
| Order.Pre => PreL(tree, 0);
| Order.In => InL(tree, 0);
| Order.Post => PostL(tree, 0);
END (*CASE order*)
ELSE (* direction = Direction.Right*)
CASE order OF
| Order.Pre => PreR(tree, 0);
| Order.In => InR(tree, 0);
| Order.Post => PostR(tree, 0);
END (*CASE order*)
END (*IF direction*)
END Traverse;
Deletion from Binary Search Trees

Insertion was easy because the new node goes in as a leaf and only its parent is
affected.

Deletion of a leaf is just as easy - set the parent pointer to NIL. But what if the node to
be deleted is an interior node? We have two pointers to connect to only one parent!!

Deletion is somewhat more tricky than insertion, because the node to die may not be a
leaf, and thus effect other nodes.  

Case (a), where the node is a leaf, is simple - just NIL out the parents child pointer.

Case (b), where a node has one chld, the doomed node can just be cut out.

Case (c), relabel the node as its predecessor (which has at most one child when z has
two children!) and delete the predecessor!
PROCEDURE Delete(VAR tree: T; e: ElemT): BOOLEAN =
(*Deletes an element e in tree.
Returns TRUE if present, else FALSE*)

PROCEDURE LeftLargest(VAR x: T) =
VAR y: T;
BEGIN
IF x.right = NIL THEN (*x points to largest element left*)
y:= tree; (*y now points to target node*)
tree:= x; (*tree assumes the largest node to the
left*)
x:= x.left; (*Largest node left replaced by its left
subtree*)
tree.left:= y.left; (*tree assumes subtrees ...*)
tree.right:= y.right; (*... of deleted node*)
ELSE (*Largest element left not found*)
LeftLargest(x.right) (*Continue search to the right*)
END;
END LeftLargest;

BEGIN
IF tree = NIL THEN RETURN FALSE
ELSIF e < tree.key THEN RETURN Delete(tree.left, e)
ELSIF e > tree.key THEN RETURN Delete(tree.right, e)
ELSE (*found*)
IF tree.left = NIL THEN
tree:= tree.right;
ELSIF tree.right = NIL THEN
tree:= tree.left;
ELSE (*Target node has two nonempty subtrees*)
LeftLargest(tree.left) (*Search in left subtree*)
END; (*IF tree.left...*)
RETURN TRUE
 END; (*IF tree...*)
END Delete;

SUNY at Stony BrookMidterm 2

CSE 214 - Data Structures November 21, 1997

Midterm Exam

Name: Signature:
ID #: Section #:

INSTRUCTIONS:

 You may use either pen or pencil.

 Check to see that you have 5 exam pages plus this cover (6 total).
 Look over all problems before starting work.
 Your signature above signs the CSE 214 Honor Pledge: ``On my honor as a
student I have neither given nor received aid on this exam.''
 Think before you write.
 Good luck!!

1) (20 points) Show the state of the array after each pass by the following sorting
routines. You do not have to show the array after every move or comparison, but only
after each execution of the main sorting loop or recursive call. Sort in increasing
order.
-------------------------------------------------------------
| 34 | 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |
-------------------------------------------------------------

(a) Insertion Sort

10 points
-------------------------------------------------------------
| 34 | 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |
| 34 \ 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |
 | 34 | 125 \ 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |
| 5 | 34 | 125 \ 19 | 87 | 243 | 19 | -3 | 117 | 36 |
| 5 | 19 | 34 | 125 \ 87 | 243 | 19 | -3 | 117 | 36 |
| 5 | 19 | 34 | 87 | 125\ 243 | 19 | -3 | 117 | 36 |
| 5 | 19 | 34 | 87 | 125| 243 \ 19 | -3 | 117 | 36 |
| 5 | 19 | 19 | 34 | 87 | 125| 243 \ -3 | 117 | 36 |
| -3 | 5 | 19 | 19 | 34 | 87 | 125| 243 \ 117 | 36 |
| -3 | 5 | 19 | 19 | 34 | 87 | 117| 125| 243 \ 36 |
| -3 | 5 | 19 | 19 | 34 | 36 | 87 | 117| 125| 243 |
-------------------------------------------------------------

(b) Quicksort (pivot on rightmost element)

10 points - there are many variants

------------------------------------------------------------
| 34 | 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |
| 34 | 5 | 19 | 19 | -3 | 36 | 125 | 87 | 243 | 117 |
| -3 | 34 | 5 | 19 | 19 | 36 | 87 | 117 | 125 | 243 |
| -3 | 5 | 19 | 19 | 34 | 36 | 87 | 117 | 125 | 243 |
| -3 | 5 | 19 | 19 | 34 | 36 | 87 | 117| 125| 243 |
-------------------------------------------------------------

2) (25 points) In class, we discussed two different heuristics for self-organizing

sequential search. With the move-to-front heuristic, on a query we perform a
sequential search for the appropriate element, which when found is moved from its
current position to the front of the list. With the move-forward-one heuristic, on a
query we perform a sequential search for the appropriate element, which when found
is moved one element closer to the front of the list. For both algorithms, if the search
key is already at the front of the list, no change occurs.

(a) Write a function to implement sequential search in a linked list, with the move-to-
front heuristic. You may assume that there are at least two elements in the list and that
the item is always found.
PROCEDURE ListSearch (VAR p : pointer) : pointer =

var q, head:pointer;

head := p;
q := p;

if (q^.info = key) then

return(q);
else
p := p^.next;
while (p^.info # key) do
p := p^.next;
q := q^.next;
end
q^.next := p^.next;
 p^.next := head;
head := p;
return (p);
end
points for searching, 10 points for move to front.

(b) Which of these two heuristics is better suited for implementation with arrays?
Why?

5 points

move-forward-one is better for arrays since it can be done via one swap.

3) (15 points) Assume you have an array with 11 elements that is to be used to store
data as an hash table. The hash function computes the number mod 11. Given the
following list of insertions to the table:
2 4 13 18 22 31 33 34 42 43 49
Show the resulting table after the insertions for each of the following hashing collision
handling methods.

a) Show the resulting table after the insertions for chaining. (array of linked lists)

10 points

0 - 22, 33 1 - 34 2 - 2, 13 3 - 4 - 4 5 - 49 6 - 7 - 18 8 - 9 - 31, 42 10 - 43

b) List an advantage and a disadvantage of chaining compared to open addressing

5 points.

Advantages - deletion is easier and hash table cannot be filled.

Disadvantages - the links use up memory which can go to a bigger hash table.

4) (20 points) Write brief essays answering the following questions. Your answer
must fit completely in the space allowed

(a) Is f(n) = O(g(n)) if   and   ? Show why or why not. points

No! There is no constant such that   .

(b) Consider the following variant of insertion sort. Instead of using sequential search
to find the position of the next element we insert into the sorted array, we use a binary
search. We then move the appropriate elements over to create room for the new
insertion. What is the worst case number of element comparisons performed using
this version of insertion sort on n items (big Oh)? points

(c) What is the worst case number of element movements performed using the above
version of insertion sort on n items (big Oh)?

6 points

I took off more points for inconsistancies between the answers...

5) (20 points) The integer square root of integer n (SQRT(n)) is the largest

integer x such that   . For example, SQRT(8) = 2, while SQRT(9) = 3.

Write a Modula-3 function to compute SQRT(n). For full credit, your algorithm

should run in   time. Partial credit will be given for an   algorithm.

(Hint: think about the ideas behind binary search)

PROCEDURE sqrt(n : INTEGER):INTEGER =

var
low, high, mid : integer;

low := 1;
high := n;

while (high - low) > 1 do

mid := (high+low) div 2;
if (mid * mid) > n then low := mid+1;
else high := mid;
end;

return (mid);
end;

Random Search Trees

Lecture 23
Steven S. Skiena
How good are Random Trees?

Who have seen that binary trees can have heights ranging from   to n. How tall
are they on average?

By using an intuitive argument, like I did with quicksort. I will convince you a
random tree is usually quite close to balanced. The text contains a more rigorous
proof, which you should look at.

Consider the first insertion into an empty tree. This node becomes the root and never
changes. Since in a binary search tree all keys less than the root go in the left subtree,
the root acts as a partition or pivot element!

Let's say a key is a 'good' pivot element if it is in the center half of the sorted space of
keys. Half of the time, our root will be a 'good' pivot element.

The next insertion will form the root of a subtree, and will be drawn at random from
the items either > root or < root. Again, half the time each insertion will be a 'good'
partition of the appropriate subset of keys.

The bigger half of a good partition contains at most 3n/4 items. Thus the maximum
depth of good splits k is:

so   .

Doubling the depth to account for bad splits still makes in   on average!

On average, random search trees are very good - more careful analysis shows the
average height after n insertions is   . Since   , this is
only 39% more than a perfectly balanced tree.

Of course, if we get unlucky and insert keys in sorted order, we are doomed to the
worst case performance.

insert(a)
insert(b)

insert(c)

insert(d)

What we want is an insertion/deletion procedure which adjusts the tree a little after

each insertion, keeping it close enough to balanced so the maximum height
is logarithmic, but flexible enough so we can still update fast!

Perfectly Balanced Trees

Perfectly balanced trees require a lot of work to maintain:

If we insert the key 1, we must move every single node in the tree to rebalance it,
taking   time.

Therefore, when we talk about "balanced" trees, we mean trees whose height
is   , so all dictionary operations (insert, delete, search, min/max,
successor/predecessor) take   time.

Red-Black trees are binary search trees where each node is assigned a color, where the
coloring scheme helps us maintain the height as   .

AVL Trees
Lecture 24
Steven S. Skiena

AVL Trees
An AVL tree is a binary search tree in which the heights of the left and right subtrees
of the root differ by at most 1, and the left and right subtrees are again AVL trees.

Therefore, we can label each node of an AVL tree with a balance factor as well as a
key:

 ``='' - both subtrees of the node are of equal height

 ``/'' - the left subtree is one taller than the right subtree
 ``
'' - the right subtree is one taller than the left subtree.

AVL trees are named after their inventors, the Russians G.M. Adel'son-Velshi, and
E.M. Laudis in 1962.

These are the most unbalanced possible AVL trees with a skew always to the right.

By maintaining the balance of each node (i.e. the subtree below it) when we insert a
new node, we can easily see whether or not to take action!

The balance is more useful than maintaining the height of each node because it is
a relative, not absolute measure. Thus we can move subtrees around without affecting
their balance, even if they end up at different heights.

How good are AVL trees?

To find out how bad they can be, we want to find what the minimum number of modes
a tree of height h can have. If   is a minimum node AVL tree, its left and right
subtrees must themselves be minimum node AVL trees of smaller size. Further, they
should differ in height by 1 to take advantage of AVL freedom.

Counting the root node,

Such trees are called Fibonacci trees and   .

Thus the worse case AVL tree is almost as good as a random tree - on average it is
very close to an optional tree.

Why are Fibonacci trees of logarithmic height?

Recall that the Fibonacci numbers are defined   ,   
,   .

Since we are adding the last two numbers together, we are more than doubling the
next-to-last and somewhat less that doubling the last number.

In fact,   , so a tree with   nodes has height

AVL Trees Interface

INTERFACE AVLTree; (*08.07.94. CW, LB*)

(* Balanced binary search tree, subtype of "BinaryTree.T" *)

IMPORT BinaryTree;
TYPE T <: BinaryTree.T; (*T is a subtype of BinaryTree.T *)

END AVLTree.

AVL Trees Implementation

MODULE AVLTree EXPORTS AVLTree, AVLTreeRep; (*08.07.94. CW*)
(* Implementation of the balanced binary search tree as subtype of
"BinaryTree.T". The methods "insert" and "delete" are overwritten
to keep the tree balanced when elements are inserted or
deleted. The other methods are inhereted from the supertype.
*)

IMPORT BinaryTree, BinTreeRep;

REVEAL
T = BinaryTree.T BRANDED OBJECT
OVERRIDES
delete:= Delete;
insert:= Insert;
END;

PROCEDURE Insert(tree: T; e: REFANY) =

PROCEDURE RR (VAR root: BinTreeRep.NodeT) =

(*simple rotation right*)
VAR left:= root.left;
BEGIN
root.left:= left.right;
 left.right:= root;
NARROW(root, NodeT).balance:= 0;
root:= left;
END RR;

PROCEDURE RL (VAR root: BinTreeRep.NodeT) =

(*simple rotation left*)
VAR right:= root.right;
BEGIN
root.right:= right.left;
right.left:= root;
NARROW(root, NodeT).balance:= 0;
root:= right;
END RL;

PROCEDURE RrR (VAR root: BinTreeRep.NodeT) =

(*double rotation right*)
VAR right:= root.left.right;
BEGIN
root.left.right:= right.left;
right.left:= root.left;
IF NARROW(right, NodeT).balance = -1
THEN NARROW(root, NodeT).balance:= +1
ELSE NARROW(root, NodeT).balance:= 0
END;
IF NARROW(right, NodeT).balance = +1
THEN NARROW(root.left, NodeT).balance:= -1
ELSE NARROW(root.left, NodeT).balance:= 0
END;
root.left:= right.right;
right.right:= root;
root:= right;
END RrR;

PROCEDURE RrL (VAR root: BinTreeRep.NodeT) =

(*double rotation left*)
VAR left:= root.right.left;
BEGIN
root.right.left:= left.right;
left.right:= root.right;
IF NARROW(left, NodeT).balance = +1
THEN NARROW(root, NodeT).balance:= -1
ELSE NARROW(root, NodeT).balance:= 0
END;
IF NARROW(left, NodeT).balance = -1
THEN NARROW(root.right, NodeT).balance:= +1
ELSE NARROW(root.right, NodeT).balance:= 0
END;
root.right:= left.left;
left.left:= root;
root:= left;
END RrL;

PROCEDURE InsertBal(VAR root: BinTreeRep.NodeT; new: REFANY;

VAR bal: BOOLEAN) =
BEGIN
IF root = NIL
 THEN
root:= NEW(NodeT, info:= new, balance:= 0);

ELSIF tree.compare(new, root.info)<0 THEN

InsertBal(root.left, new, bal);
IF NOT bal THEN (* bal stops recursion*)
WITH done=NARROW(root, NodeT).balance DO
CASE done OF
|+1=> done:= 0; bal:= TRUE; (*insertion ok*)
| 0=> done:= -1; (*still balanced, but continue*)
|-1=>
IF NARROW(root.left, NodeT).balance = -1
THEN RR(root)
ELSE RrR(root)
END;
NARROW(root, NodeT).balance:= 0;
bal:= TRUE; (*after rotation tree ok*)
END; (*CASE*)
END (*WITH*)
END (*IF*)

ELSE
InsertBal(root.right, new, bal);
IF NOT bal THEN (* bal is set to stop the recurs. adjustm. of
balance *)
WITH done=NARROW(root, NodeT).balance DO
CASE done OF
|-1=> done:= 0; bal:= TRUE; (*insertion ok *)
| 0=> done:= +1; (*still balanced, but continue*)
|+1=>
IF NARROW(root.right, NodeT).balance = +1
THEN RL(root)
ELSE RrL(root)
END;
NARROW(root, NodeT).balance:= 0;
bal:= TRUE; (*after rotation tree ok*)
END; (*CASE*)
END (*WITH*)
END (*IF*)
END;
END InsertBal;

VAR balanced:= FALSE;

BEGIN (*Insert*)
InsertBal(tree.root, e, balanced)
END Insert;

PROCEDURE Delete(tree: T; e: REFANY): REFANY =

PROCEDURE RR (VAR root: BinTreeRep.NodeT;

VAR bal: BOOLEAN) =
(*simple rotation right*)
VAR left:= root.left;
BEGIN
root.left:= left.right;
left.right:= root;
IF NARROW(left, NodeT).balance = 0
 THEN
NARROW(root, NodeT).balance:= -1;
NARROW(left, NodeT).balance:= +1;
bal:= TRUE;
ELSE
NARROW(root, NodeT).balance:= 0;
NARROW(left, NodeT).balance:= 0; (*depth changed: continue*)
END;
root:= left;
END RR;

PROCEDURE RL (VAR root: BinTreeRep.NodeT;

VAR bal: BOOLEAN) =
(*simple rotation left*)
VAR right:= root.right;
BEGIN
root.right:= right.left;
right.left:= root;
IF NARROW(right, NodeT).balance = 0
THEN
NARROW(root, NodeT).balance:= +1;
NARROW(right, NodeT).balance:= -1;
bal:= TRUE;
ELSE
NARROW(root, NodeT).balance:= 0;
NARROW(right, NodeT).balance:= 0; (*depth changed: continue*)
END;
root:= right;
END RL;

PROCEDURE RrR (VAR root: BinTreeRep.NodeT) =

(*double rotation right*)
VAR right:= root.left.right;
BEGIN
root.left.right:= right.left;
right.left:= root.left;
IF NARROW(right, NodeT).balance = -1
THEN NARROW(root, NodeT).balance:= +1
ELSE NARROW(root, NodeT).balance:= 0
END;
IF NARROW(right, NodeT).balance = +1
THEN NARROW(root.left, NodeT).balance:= -1
ELSE NARROW(root.left, NodeT).balance:= 0
END;
root.left:= right.right;
right.right:= root;
root:= right;
NARROW(right, NodeT).balance:= 0;
END RrR;

PROCEDURE RrL (VAR root: BinTreeRep.NodeT) =

(*double rotation left*)
VAR left:= root.right.left;
BEGIN
root.right.left:= left.right;
left.right:= root.right;
IF NARROW(left, NodeT).balance = +1
 THEN NARROW(root, NodeT).balance:= -1
ELSE NARROW(root, NodeT).balance:= 0
END;
IF NARROW(left, NodeT).balance = -1
THEN NARROW(root.right, NodeT).balance:= +1
ELSE NARROW(root.right, NodeT).balance:= 0
END;
root.right:= left.left;
left.left:= root;
root:= left;
NARROW(left, NodeT).balance:= 0;
END RrL;

PROCEDURE BalanceLeft(VAR root: BinTreeRep.NodeT;

VAR bal: BOOLEAN) =
BEGIN
WITH done = NARROW(root, NodeT).balance DO
CASE done OF
|-1=> done:= 0; (*new depth: continue*)
| 0=> done:= 1; bal:= TRUE; (*balanced ->ok*)
|+1=> (*balancing needed*)
IF NARROW(root.right, NodeT).balance >= 0
THEN RL(root, bal)
ELSE RrL(root)
END
END (*CASE*)
END (*WITH*)
END BalanceLeft;

PROCEDURE BalanceRight(VAR root: BinTreeRep.NodeT;

VAR bal: BOOLEAN) =
BEGIN
WITH done = NARROW(root, NodeT).balance DO
CASE done OF
|+1=> done:= 0; (*new depth: continue*)
| 0=> done:= -1; bal:= TRUE; (*balanced ->ok*)
|-1=> (*balancing needed*)
IF NARROW(root.left, NodeT).balance <= 0
THEN RR(root, bal)
ELSE RrR(root)
END
END (*CASE*)
END (*WITH*)
END BalanceRight;

PROCEDURE DeleteSmallest(VAR root: BinTreeRep.NodeT;

VAR bal: BOOLEAN): REFANY =
VAR deleted: REFANY;
BEGIN
IF root.left = NIL
THEN
deleted:= root.info;
root:= root.right;
RETURN deleted;
ELSE
deleted:= DeleteSmallest(root.left, bal);
IF NOT bal THEN BalanceLeft(root, bal) END;
 RETURN deleted;
END;
END DeleteSmallest;

PROCEDURE Delete(VAR root: BinTreeRep.NodeT; elm: REFANY;

VAR bal: BOOLEAN): REFANY =
VAR deleted: REFANY;
BEGIN
IF root = NIL
THEN RETURN NIL

ELSIF tree.compare(root.info, elm)>0 THEN

deleted:= Delete(root.left, elm, bal);
IF deleted # NIL
THEN
IF NOT bal THEN BalanceLeft(root, bal) END;
RETURN deleted;
ELSE RETURN NIL;
END

ELSIF tree.compare(root.info, elm)<0 THEN

deleted:= Delete(root.right, elm, bal);
IF deleted # NIL
THEN
IF NOT bal THEN BalanceRight(root, bal) END;
RETURN deleted;
ELSE RETURN NIL;
END

ELSE
deleted:= root.info;
IF root.left = NIL
THEN
root:= root.right;
ELSIF root.right = NIL THEN
root:= root.left;
ELSE
root.info:= DeleteSmallest(root.right, bal);
IF NOT bal THEN BalanceRight(root, bal) END;
END;
RETURN deleted;
END;
END Delete;

VAR balanced:= FALSE;

BEGIN (*Delete*)
RETURN Delete(tree.root, e, balanced)
END Delete;

BEGIN
END AVLTree.

Deletion from AVL Trees

We have seen that AVL trees are   for insertion and query.
But what about deletion?

Don't ask! Actually, you can rebalance an AVL tree in   but it is more
complicated than insertion.

We will later study B-trees, where deletion is simpler, so don't worry about the details
of deletions form AVL trees.

Red-Black Trees
Lecture 25
Steven S. Skiena

Red-Black Tree Definition

Red-black trees have the following properties:

1. Every node is colored either red or black.

2. Every leaf (NIL pointer) is black.
3. If a node is red then both its children are black.
4. Every single path from a node to a decendant leaf contains the same number of
black nodes.

What does this mean?

If the root of a red-black tree is black can we just color it red?

No! For one of its children might be red.

If an arbitrary node is red can we color it black?

No! Because now all nodes may not have the same black height.

What tree maximizes the number of nodes in a tree of black height h?

What does a red-black tree with two real nodes look like?

Not (1) - consecutive reds Not (2), (4) - Non-Uniform black height
Red-Black Tree Height

Lemma: A red-black tree with n internal nodes has height at most   .

Proof: Our strategy; first we bound the number of nodes in any subtree, then we
bound the height of any subtree.

We claim that any subtree rooted at x has at least   - 1 internal nodes,

where bh(x) is the black height of node x.

Proof, by induction:

Now assume it is true for all tree with black height < bh(x).

If x is black, both subtrees have black height bh(x)-1. If x is red, the subtrees have
black height bh(x).

Therefore, the number of internal nodes in any subtree is

Now, let h be the height of our red-black tree. At least half the nodes on any single
path from root to leaf must be black if we ignore the root.

Thus   and   , so   .

This implies that   ,so   . height6pt width4pt

Therefore red-black trees have height at most twice optimal. We have a balanced
search tree if we can maintain the red-black tree structure under insertion and deletion.

Rotations

The basic restructuring step for binary search trees are left and right rotation:

1. Rotation is a local operation changing O(1) pointers.

2. An in-order search tree before a rotation stays an in-order search tree.
 3. In a rotation, one subtree gets one level closer to the root and one subtree one
level further from the root.

Rotation Implementation
PROCEDURE RR (VAR root: BinTreeRep.NodeT) =
(*simple rotation right*)
VAR left:= root.left;
BEGIN
root.left:= left.right;
left.right:= root;
root:= left;
END RR;

PROCEDURE RL (VAR root: BinTreeRep.NodeT) =

(*simple rotation left*)
VAR right:= root.right;
BEGIN
root.right:= right.left;
right.left:= root;
root:= right;
END RL;

Note the in-order property is preserved.

Red-Black Insertion

Since red-black trees have   height, if we can preserve all properties of such
trees under insertion/deletion, we have a balanced tree!

Suppose we just did a regular insertion. Under what conditions does it stay a red-black
tree?

Since every insertion take places at a leaf, we will change a black NIL pointer to a
node with two black NIL pointers.

To preserve the black height of the tree, the new node must be red. If its new parent is
black, we can stop, otherwise we must restructure! How can we fix two reds in a row?

It depends upon our uncle's color:

If our uncle is red, reversing our relatives' color either solves the problem or pushes it
higher!

Note that after the recoloring:

 1. The black height is unchanged.
2. The shape of the tree is unchanged.
3. We are done if our great-grandparent is black.

If we get all the way to the root, recall we can always color a red-black tree's root
black. We always will, so initially it was black, and so this process terminates.

The Case of the Black Uncle

If our uncle was black, observe that all the nodes around us have to be black:

Solution - rotate right about B:

Since the root of the subtree is now black with the same black-height as before, we
have restored the colors and can stop!

Double Rotations

A double rotation can be required to set things up depending upon the left-right turn
sequence, but the principle is the same.

Deletion from Red-Black Trees

Recall the three cases for deletion from a binary tree:

Case (a) The node to be deleted was a leaf;

Case (b) The node to be deleted had one child;

Case (c) relabel to node as its successor and delete the successor.

Deletion Color Cases

Suppose the node we remove was red, do we still have a red-black tree?

Yes! No two reds will be together, and the black height for each leaf stays the same.

However, if the dead node y was black, we must give each of its decendants another
black ancestor. If an appropriate node is red, we can simply color it black otherwise
we must restructure.

Case (a) black NIL becomes ``double black'';

Case (b) red   becomes black and black   becomes ``double black'';

Case (c) red   becomes black and black   becomes ``double black''.

Our goal will be to recolor and restructure the tree so as to get rid of the ``double
black'' node.

In setting up any case analysis, we must be sure that:

1. All possible cases are covered.

2. No case is covered twice.

In the case analysis for red-black trees, the breakdown is:

Case 1: The double black node x has a red brother.

Case 2: x has a black brother and two black nephews.

Case 3: x has a black brother, and its left nephew is red and its right nephew is black.

Case 4: x has a black brother, and its right nephew is red (left nephew can be any
color).

Conclusion

Red-Black trees let us implement all dictionary operations in   . Further, in

no case are more than 3 rotations done to rebalance. Certain very advanced data
structures have data stored at nodes which requires a lot of work to adjust after a
rotation -- red-black trees ensure it won't happen often.

Example: Each node represents the endpoint of a line, and is augmented with a list of
segments in its subtree which it intersects.

We will not study such complicated structures, however.

Splay Trees
Lecture 26
Steven S. Skiena
What about non-uniform access?

AVL/red-black trees give us worst case   query and update operations, by

keeping a balanced search tree. But when I access with non-uniform probability, a
skewed tree might be better:

 I call Eve with probability .75

 I call Lisa with probability .05
 I call Wendy with probability .20

Expected cost of left tree: 

Expected cost of right tree: 

In real life, it is difficult to obtain the actual probabilities, and they keep changing.
What can we do?

Self-organizing Search Trees

We can apply our self-organizing heuristics to search trees, as we did with linked lists.
Whenever we access a node, we can either:

 Move-forward-one (conservative heuristic)

 Move-to-front (liberal heuristic)

Once again, move-to-front proves better at adjusting to changing distributions.

Moving a made to the front of a search tree means making it the root!

To get a particular node to the root we can do a sequence of rotations!

Splay trees use the move-to-front heuristic on each search / query.

Splay Trees

To search or insert into a splay tree, we first perform the operation as if it was a
random tree. After it is found or inserted, perform a splay operation to move the given
key to the root.
A splay operation consists of a sequence of double rotations until the node is within
one level of the root, where at most one single rotation suffices to finish the job.

The choice of which double rotation to do depends upon our relationship to

our grandparent - a single rotation is performed only when we have no grandparent!

The cases:   and   .

Splay Tree Example

Example: Splay(a)

At the conclusion, a is the root and the tree is more balanced.

Note that the tree would not have become more balanced had we just used single
rotations to promote a to the root, instead of double rotations.

How good are Splay Trees?

Sleator and Tarjan showed that if the keys are accessed with a uniform distribution,
the cost for any sequence of n splay operations is   , so the amortized
cost is   per operation!

This is better than expected   since there is no probability involved! If we

get an expensive splay step (i.e. moving up an non-balanced tree) it meant we did
enough cheap operations before this that we can pay for the differences out of our
savings!

Further, if the distribution is non-uniform, we get amortized costs within a constant

factor of the best possible tree!

All of this is done without keeping any balance or color information - amazing!

Graphs
Lecture 27
Steven S. Skiena
Graphs

A graph G consists of a set of vertices V together with a set E of vertex pairs or edges.

Graphs are important because any binary relation is a graph, so graphs can be used to
represent essentially any relationship.

Example: A network of roads, with cities as vertices and roads between cities as
edges.

Example: An electronic circuit, with junctions as vertices as components as edges.

To understand many problems, we must think of them in terms of graphs!

The Friendship Graph

Consider a graph where the vertices are people, and there is an edge between two
people if and only if they are friends.  

This graph is well-defined on any set of people: SUNY SB, New York, or the world.

What questions might we ask about the friendship graph?

 If I am your friend, does that mean you are my friend?

A graph is undirected if (x,y) implies (y,x). Otherwise the graph is directed. The
``heard-of'' graph is directed since countless famous people have never heard of
me! The ``had-sex-with'' graph is presumably undirected, since it requires a
partner.

 Am I my own friend?

An edge of the form (x,x) is said to be a loop. If x is y's friend several times
over, that could be modeled using multiedges, multiple edges between the same
pair of vertices. A graph is said to be simple if it contains no loops and multiple
edges.

 Am I linked by some chain of friends to the President?

A path is a sequence of edges connecting two vertices. Since Mel Brooks is my

father's-sister's-husband's cousin, there is a path between me and him!

 How close is my link to the President?

 If I were trying to impress you with how tight I am with Mel Brooks, I would
be much better off saying that Uncle Lenny knows him than to go into the
details of how connected I am to Uncle Lenny. Thus we are often interested in
the shortest path between two nodes.

 Is there a path of friends between any two people?

A graph is connected if there is a path between any two vertices. A directed

graph is strongly connected if there is a directed path between any two vertices.

 Who has the most friends?

The degree of a vertex is the number of edges adjacent to it.

 What is the largest clique?

A social clique is a group of mutual friends who all hang around together. A
graph theoretic clique is a complete subgraph, where each vertex pair has an
edge between them. Cliques are the densest possible subgraphs. Within the
friendship graph, we would expect that large cliques correspond to workplaces,
neighborhoods, religious organizations, schools, and the like.

 How long will it take for my gossip to get back to me?

A cycle is a path where the last vertex is adjacent to the first. A cycle in which
no vertex repeats (such as 1-2-3-1 verus 1-2-3-2-1) is said to be simple. The
shortest cycle in the graph defines its girth, while a simple cycle which passes
through each vertex is said to be a Hamiltonian cycle.

Data Structures for Graphs

There are two main data structures used to represent graphs.

Adjacency MatricesAn adjacency matrix is an   matrix, where M[i,j] = 0 iff

there is no edge from vertex i to vertex j  

It takes   time to test if (i,j) is in a graph represented by an adjacency matrix.

Can we save space if (1) the graph is undirected? (2) if the graph is sparse?
Adjacency ListsAn adjacency list consists of a   array of pointers, where the ith
element points to a linked list of the edges incident on vertex i.

To test if edge (i,j) is in the graph, we search the ith list for j, which takes   ,
where   is the degree of the ith vertex.

Note that   can be much less than n when the graph is sparse. If necessary, the
two copies of each edge can be linked by a pointer to facilitate deletions.

Tradeoffs Between Adjacency Lists and Adjacency Matrices

Both representations are very useful and have different properties, although adjacency
lists are probably better for most problems.

Traversing a Graph

One of the most fundamental graph problems is to traverse every edge and vertex in a
graph. Applications include:

 Printing out the contents of each edge and vertex.

 Counting the number of edges.
 Identifying connected components of a graph.

For efficiency, we must make sure we visit each edge at most twice.

For correctness, we must do the traversal in a systematic way so that we don't miss

anything.
Since a maze is just a graph, such an algorithm must be powerful enough to enable us
to get out of an arbitrary maze.

Marking Vertices

The idea in graph traversal is that we must mark each vertex when we first visit it,
and keep track of what have not yet completely explored.

For each vertex, we can maintain two flags:

 discovered - have we ever encountered this vertex before?

 completely-explored - have we finished exploring this vertex yet?

We must also maintain a structure containing all the vertices we have discovered but
not yet completely explored.

Initially, only a single start vertex is considered to be discovered.

To completely explore a vertex, we look at each edge going out of it. For each edge
which goes to an undiscovered vertex, we mark it discovered and add it to the list of
work to do.

Note that regardless of what order we fetch the next vertex to explore, each edge is
considered exactly twice, when each of its endpoints are explored.

Correctness of Graph Traversal

Every edge and vertex in the connected component is eventually visited.

Suppose not, ie. there exists a vertex which was unvisited whose neighbor was visited.
This neighbor will eventually be explored so we would visit it:

Traversal Orders

The order we explore the vertices depends upon what kind of data structure is used:

 Queue - by storing the vertices in a first-in, first out (FIFO) queue, we explore
the oldest unexplored vertices first. Thus our explorations radiate out slowly
from the starting vertex, defining a so-called breadth-first search.
 Stack - by storing the vertices in a last-in, first-out (LIFO) stack, we explore the
vertices by lurching along a path, constantly visiting a new neighbor if one is
available, and backing up only if we are surrounded by previously discovered
 vertices. Thus our explorations quickly wander away from our starting point,
defining a so-called depth-first search.

The three possible colors of each node reflect if it is unvisited (white), visited but
unexplored (grey) or completely explored (black).

Breadth-First Search

BFS(G,s)

for each vertex do

color[u] = white

, ie. the distance from s

p[u] = NIL, ie. the parent in the BFS tree

color[u] = grey

d[s] = 0

p[s] = NIL
while do

u = head[Q]

for each do

if color[v] = white then

color[v] = gray

d[v] = d[u] + 1

p[v] = u

enqueue[Q,v]

dequeue[Q]

color[u] = black

Depth-First Search

DFS has a neat recursive implementation which eliminates the need to explicitly use a
stack.

Discovery and final times are sometimes a convenience to maintain.

DFS(G)

for each vertex do

color[u] = white

parent[u] = nil

time = 0

for each vertex do

if color[u] = white then DFS-VISIT[u]

Initialize each vertex in the main routine, then do a search from each connected
component. BFS must also start from a vertex in each component to completely visit
the graph.

DFS-VISIT[u]

color[u] = grey (*u had been white/undiscovered*)

discover[u] = time

time = time+1

for each do

if color[v] = white then

parent[v] = u

DFS-VISIT(v)

color[u] = black (*now finished with u*)

finish[u] = time

time = time+1