1.

Chapter 1 Analysis Basics

a. Introduction  there are many algorithms that can solve a given problem; to determine if an algorithm is good or
bad, we need to analyze it; a good algorithm is a fast one; analyzing an algorithm determines the amount of time
that algorithm takes to execute; time refers to the # of operations that an algorithm performs; time refers to
computational complexity; we do not care about the exact # of seconds because we are concerned with the relative
efficiency of an algorithm
i. Size of input data is used as a variable, and we need to determine an equation that relates the # of operations
that a particular algorithm does to the size of the input; n refers to the # of elements
b. What is analysis? For each algorithm, we need an estimate of how long it will take to solve a problem has a set of N
input values
c. Do we need to count the exact # of operations as a function of N? No, not useful
d. Space complexity  if an algorithm needs extra space to do their work, space complexity is high; the ideal case
algorithm should work in place, meaning there is no need to include a temporary variable
e. Significant operations
i. Comparison (==, !=, <, >, <=, >=)
ii. Arithmetic (additive [+, -, ++, --] multiplicative [*, /, %])
f. Cases to consider  we will not consider just one input set when we analyze an algorithm; best case (the input that
requires the algorithm to take the shortest time); worst case (most time an algorithm will ever take; identify the
input values that cause an algorithm to do the most work); average case (determining the # of different groups into
which all possible input sets can be divided; determine the probability that the input will come from each of these
groups; determine how long the algorithm will run for each of these groups)
g. Mathematical background  floor of a # is the largest integer that is less than or equal to the #; ceiling of a # is the
smallest integer that is greater than or equal to the #; factorial of a # is the product of all of the #s between 1 and the
#; logarithms base y of a # x is the power of y that will produce the # x; log is base 10 and lg is base 2; logarithms
are a strictly increasing function; binary tree is a structure in which each node has at most 2 nodes as its children
and each node has exactly 1 parent node; top node is the root of the tree; probability is the likelihood of a certain set
of input; 0 will never occur and 1 always occur; summations are the sum of the values from 1 through N
h. Rates of growth  our concern is the rate of increase in operations for an algorithm to solve a problem as the size
of the problem increases; we will use the equation of N to represent the # of operations for that algorithm; the rate
of growth is dominated by the largest term in an equation; discard the terms that grow slowly (order of the
function); Big Omega represents the class of functions that grow at least as fast as the function f; all functions in
this class will grow as fast as f or faster; Big O represents the class of functions that grow no faster than f; the class
O(f) has f as an upper bound, so none of the functions in this class grow faster than f; Big Theta represents the class
of functions that grow at the same rate as the function f; all of the functions in this class have values that are about
the same as f
i. Finding Big O, we can find if a function is in O(f), if the limit of g(n)/f(n) is some real # less than infinity, g is in
O(f)
j. Tournament Method can be used to solve problems where information from a 1st pass through the data can help to
make later passes more efficient; steps: #1) Build a binary tree with all of the elements in the leaves #2) At each
level, 2 elements are paired and the larger of the 2 gets copied into the parent node #3) Process continues until the
root node is reached; building the tournament tree will take N-1 comparisons. Why? Every comparison produces 1
winner and 1 loser; the losers are eliminated and only the winners move up in the tree; each element, except for the
largest must lose one comparison
k. Lower bounds  an algorithm is optimal when there is no algorithm that will work more quickly; how do we know
when we have found an algorithm that is optimal? We need to know the absolute smallest # of operations needed to
solve a particular problem; this must be determined by looking at the problem itself and not any particular
algorithm to solve it; we can use a binary tree to help us analyze the process of a sorting a list; labeling each
internal node with the 2 elements of the list that would be compared; ordering of the elements that would be
necessary to move from the root to that leaf would be in the leaf of the tree
l. Sorting algorithm analysis  each sort algorithm produces a different decision tree based on the elements that it
compares; worst case is the longest path from root to leaf; best case is the shortest path; average case is the total #
of edges in the decision tree divided by the # of leaves in the tree; how can decision trees be used to give us an idea
of the bounds on an algorithm? At least 1 leaf for every possible permutation of input values, which means that
there must be at least N! leaves in the decision tree; find the shortest tree as that many leaves; the minimum depth
of the decision tree for sorting problems, is of order O(NlgN); any sort that is of order O(NlgN) is the best we will
be able to do and it can be considered optimal; we also know that any sort that runs faster than O(NlgN) must not
work
2. Chapter 2 Recursive Algorithms
a. Apply themselves to a part of the problem and then using the solution to solve the larger problem
b. 2 types  reduce and conquer algorithm: part of the problem is 1 less element than the full problem; divide and
conquer algorithm: part of the problem is half the elements in the full problem
c. Analyze  how did we count the comparisons in loops? How many comparisons inside the loop; how many times
the loop is executed; the algorithm itself does not indicate how deeply the recursion will go
d. Efficiency  map the steps of the algorithm into the 4 steps shown in the generic algorithm (direct solution,
division of the input, # of recursive calls, combination of the solution)
e. Recurrence relations does not allow us to quickly determine how efficient the algorithm is; we prefer a form that is
dependent only on N and not other function calls; we need to convert the set of recursive equations into closed form
by removing the recursive nature
f. Applications
i. Closest pair problem; brute force way is calculating the distance from every point to every other point and
finding the smallest distance; brute force solution is O(N^2); we can calculate the smallest distance in
O(NlogN) time using divide and conquer strategy; Step 1: Create 2 lists of the N points (1st will be sorted by
increasing values of the x coordinate; 2nd will be sorted by increasing values of the y coordinate; the base
case for the recursive algorithm will be when there are 3 or fewer points; more than 3 points, divide the set
of points along a vertical line, so that half points in the left part and half points are in the right part); Step 2
(recursively call the algorithm for the left and right parts until each part has no more than 3 points; let d =
min(dL, dR); it is possible that the distance between a point in the left part and a point in the right part is
shorter than d; create a new set of points that have x coordinates in the middle range; find if there is a pair
that has a shorter distance than d; if there is a pair of points closer than d in any section, those points must be
within 8 positions of each other in the middle strip
ii. Convex region is if a straight line connecting any 2 points in the region stays within the region; none of the
interior angles between 2 adjacent sides is greater than 180 degrees
iii. Convex hull for a set of points is the smallest convex region that contains all of the points; a straight line
through 2 points in a pane can be defined by the equation: ax + by – c = 0; a line divides the plane into 2
halves; the brute force method is computationally complex; in trying to find the edges of the convex hull,
we consider every pair of points as forming a potential edge; there are O(N^2) pairs of points from a set of
N points; all the other points are compared against the line formed by each of the pairs; algorithm is O(N^3)
1. Divide and conquer solution  sorting the points in order based on x coordinate; first and last points
(p1, pN) must be extreme points; use the 2 points to form a line and divide the other points into 2
sets (pL, pR) based on whether they are on the right or the left side; the convex hull of pl/pR will be
the line from p1 to pN along with the upper/lower hull of the entire set of points; upper and lower
hulls can be found by a recursive process (input is the line dividing the larger set of points and the
subset of points on one side of the line; find the point that is farthest from the line; connect farthest
point with the 2 end points of the line; check other points whether they are all inside the triangle; if
not, algorithm is called recursively with the one or two sets of points that are outside of the triangle);
using master method, the complexity on average is O(NlgN) the worst case is O(N^2)
3. Chapter 3 Searching and Selection Algorithms
a. Searching  if a list is unsorted, search sequentially—look through the list for the item we want (not very
efficient); if a list of elements is sorted, more options—binary search
b. Sequential search  assume the list is unsorted and the key values are unique; the task for a search algorithm is to
identify the location of the target so it returns the index in the range 1 to N of where the record is located; return 0 if
the target is not in the list; further down the list a particular key value is, the longer it will take to find that key value
i. Worst case  case 1 (target is last element in the list) case 2 (target is not in the list); N comparisons will be
taken; N comparisons is the upper bound for any searching algorithms for a list of N elements
ii. Average-case  case 1 (target is in the list; if the target is index 1, 1 comparison; if target is at index N, N
comparison; average # of comparisons A(N) = (N+1)/2) case 2 ( target may not be in the list; N+1
possibilities; # of comparisons = N; increased by ½ when we consider this amount relative to the size of the
list, ½ is not significant)
c. Binary search  elements of the list is sorted; compare the target with the element in the middle; the target
matches; target is less than or greater than the element; if no match, half of the list can be eliminated
i. Worst-case  power of 2 is decreased by 1 each pass of the loop; last pass of the loop occurs when the list
has a size of 1, which occurs when j is 1 (2^1 – 1); so there are at most k passes when N = 2^k – 1; worst
case is k = lg(N+1)
 ii. Decision tree  nodes of the decision tree would have the element that is checked at each pass; those
elements that would be checked if the target is less than the current element would go into the left subtree;
those would be checked when the target is greater would go into the right subtree
iii. Average-case  case 1 (target always in list; N possible locations for target; one comparison is done to find
the element that is in the root of the tree on level 1; 2 comparisons are done to find the elements that are in
the nodes on level 2; i comparisons are done to find the elements that are in the nodes on level i; average #
of comparisons: k-1 lg(N+1)-1) case 2 (target may not be in list; there are N+1 of these possibilities; target
can be smaller than the element in location 1, larger than the element in location 1 but smaller than the one
in location 2; larger than the element in location 2 but smaller than the one in location 3 and so on the target
is larger than the element in location it takes k comparisons to learn that the target is not in the list; # of
comparisons: lg(N+1) – ½ )
d. Selection is finding an element in a list that has a particular property; smallest, largest, median value; kth largest
value in the list
i. Finding kth largest value  solution 1 (sort the list in decreasing order; kth largest value will be in position
K) solution 2 (find the largest value and then move it to the last location in the list; we again look for the
largest value in the list, ignoring the value we already found, we get the 2nd largest value, which can be
moved to the 2nd last location in the list; continue process until finding kth largest value on kth pass);
complexity (O(K*N)) solution 3 (partition; choose an element of the list and partition list into 2 parts—
those values that are greater than the one chosen and those that are less than it; rearrange the list so that all
of the larger values are before the chosen value and all of the smaller ones are after it, the chosen value will
wind up in some position P in our list meaning it is the Pth largest value; to do this portioning, we will need
to compare this value with all of the others, doing N-1 comparisons; if we are lucky and P = K, we are done;
if K is less than P, we want a larger value and will repeat on 1st partition; if K is greater than P, we want a
smaller value and will use the 2nd partition, but we need to reduce K by the # of values we have eliminated
in the larger partition)
ii. Complexity  linear and independent of K
4. Chapter 4 Sorting algorithms
a. Insertion sort  you have a list that is sorted and you add a new element into the list; the list is one element larger,
next insert another element into the larger group; continue until all elements are in the group; the 1 st element of any
list is always a sorted list of size 1; insert the 2nd element into it; insert the 3rd element into the sorted list containing
the 1st two elements; repeat until all of the elements have been put into the expanding sorted potion of the list
i. Worst-case  for one pass, the new element to be added is smaller than all of the elements already in the
sorted part of the list; the most work the entire algorithm will do is in the case where every new element is
added to the front of the list; the list must be in decreasing order when we start; 1st element is compared to
one other element at most; 2nd element is compared to the 2 previous elements; this process is repeated for
N-1 elements; O(N^2)
ii. Average-case  adding the ith element to the sorted part of the list does at most i comparisons; O(N^2)
b. Bubble sort  start each of the passes at the beginning of the list and compare the elements in location 1 and 2,
then the elements in locations 2 and 3, then 3 and 4, and so one swapping those that are out of order; once the
algorithm reaches the largest element, it will be swapped with all of the remaining elements, moving it to the end of
the list after the 1st pass; 2nd pass will move the 2nd largest element down until it is in the second to last location
i. Best-case  on the first pass, the for loop must fully execute so this algorithm does at least N-1
comparisons; if there are no swaps, swappedElements will still be false and the algorithm will end; N-1
comparisons; this happens when the data values are already in order
ii. Worst-case  when data is in reverse order, on the 1st pass the largest value is first it will be swapped with
every other element down the list; on the 2nd pass the second largest element is now in the first position, and
will be swapped with every other element in the list until it is in the second to last position; O(N^2)
iii. Average-case  assume that it is equally likely that on any of these passes there will be no swaps done; if
we stop after one pass, we have done N-1 comparisons; if we stop after 2 passes, we have done N-1+N-2
comparisons; let C(i) be how many comparisons are done on the first i passes; O(N^2)
c. Shellsort  begins by considering the full list of values as a set of interleaved sublists; on the 1st pass, it may deal
with sublists that are just pairs of elements; on the 2nd pass, it could deal with groups of four elements each; process
repeats, increasing the # of elements per sublist and decreasing the # of sublists; starts with an increment that is 1
less than the largest power of 2 that is smaller than the size of the list; if the list has 1000 elements, our 1 st
increment will be 511; the increment also indicates the # of sublists; 1st sublist has the elements in locations 1 and
1+increment, the last sublist has to start in location increment
i. Complexity  very complex; worst case is O(N^3/2)
 ii. Choice of increment sequence can have a major effect on the order of shellsort; attempts at finding an
optimal increment sequence have not been successful
d. Radix sort  does not actually comparing key values to each other; create a set of buckets and distribute the entries
into the buckets based on their key values; after collecting the values and repeating this process for successive parts
of the key and create a sorted list; each key is looked at once for each digit of the longest key; so if the longest key
has M digits and there are N keys, radix sort has order O(M*N); M is relatively small; this algorithm is of linear
complexity O(N); if we use arrays for the buckets, we will need 10N additional space if the keys are numeric; why
is N the size of each array? We can’t assume that the keys will be uniformly distributed among the buckets
e. Heapsort  based on a special type of binary tree called a heap; heap: for every subtree the value at the root is
larger than all the values in the two children; store a heap as a list (for the node at location I, we will store its two
children at locations 2*i and 2*i+1); fix a heap (when we take the largest element out of the root and move it to the
list, this leaves the root vacant); remove the largest element (element to be removed is in the root; removing it
leaves the root empty; how do we maintain a nearly complete tree? The only node we can delete from the tree, and
still have a complete tree is the last node); insert a new element (there is only one place where we can insert a new
node and still have a nearly complete binary tree; the only place the heap property can possibly fail is at the new
node; we compare the element with the parent node and find that the heap property does not fail at the node)
f. Mergesort  merging 2 sorted arrays; how do you sort each half? Recursion; you divide the half into 2 quarters,
sort each of the quarters, and merge them to make a sorted half; divide the array again and again until you reach a
subarray with only one element; this is the base case; it’s assumed an array with one element is already sorted
i. Analysis  breaking the list into 2 sublists, and each one is half the size of the original (N/2); the combine
step will take N/2 comparisons in the best case and N/2+N/2-1 comparisons in the worst case; recurrence
relations for worst and best:

Both worst case complexity and best-case complexity are O(NlgN)

g. Quick sort  in the majority of situations, it’s the fastest operating in O(NlogN) time; operates by portioning an
array into 2 subarrays and then calling itself recursively to quicksort each of these subarrays; 3 basic steps: 1.
Partition the array or subarray into left smaller keys and right larger keys groups; 2. Call ourselves to sort the left
group; 3. Call ourselves again to sort the right group; after a partition, all the items in the left subarray are smaller
than those on the right; if we then sort the left subarray and sort the right subarray, the entire array will be sorted;
how do we sort these subarrays? By calling recursively
i. Pivot value should be the key value of an actual data item; can pick a data item to be the pivot at random;
after the partition, if the pivot is inserted at the boundary between the left and right subarrays, it will be in its
final sorted position
ii. Splitting the list  pick the 1st element of the list as its pivot element and will set the pivot point as the 1 st
location of the list; moves through the list comparing this pivot element to the rest of the elements;
whenever it finds an element that is smaller than the pivot element, it will increment the pivot point location
and then swap this element into the new pivot point location
5. Chapter 5 Numeric Algorithms
a. Matrix multiplication; matrix operations are done frequently so faster matrix multiplication results in faster
programs
b. Calculating polynomials  assume that the coefficient values of a`n through a`0 are all known , constant and will
be stored in an array; our evaluation of a polynomial has only the value of x as its input and will return the resulting
polynomial value as its output; standard evaluation algorithm (the for loop has 2 multiplications and is done N-1
times; there is one multiplication done before the loop, giving a total of 2N-1 multiplications; there is one addition
done inside the loop and one done before it, giving N additions); Horner’s method (based on recognizing that the
polynomial equation can be factored into a certain form; the loop is done N times and that there is one addition and
one multiplication done in the loop; this means that there are N multiplications and N additions done by Horner’s
method)
c. Preprocessed coefficients  it is possible to express a polynomial as a factorization into 2 polynomials of lesser
degree; requirements (polynomial is monic; its largest degree equal to 1 less than a power of 2)
6. Chapter 6
a. Linear equations
i. a set of N equations with N unknown quantities; target is to find the value of x1 through xn
 ii. Solution  1.) Do substitutions of the equations (take the 2nd equation and rewrite it; then substitute this
into the other 3 equations in place of x1; now we have 3 equations with 3 unknowns), 2.) Do the same for
x2, 3.) Do the same for x3, 4.) Get x4, 5.) Substituting x4 to get x3, 6.) Substituting x3 and x4 and get x2,
7.) Using these 3 values in one of our original equations would give us the value of x1
iii. Gauss-Jordan method  consider the system of linear equations as a matrix with N rows and N+1 columns;
when the first n rows and columns represent the identity matrix, the final column will have the x values that
we want; Step 1 (eliminate the first value for rows 2 and 4; divide the 1st row by the value in the first column
and then subtract multiples o this new first row from each of the other rows); Step 2 (eliminate the second
value for rows 1, 3, 4; divide the 2nd row by the value in the second column and then subtract multiples of
this new first row from each of the other rows); Step 3 (eliminate the third value for rows 1, 2, 4; divide the
3rd row by the value in the third column and then subtract multiples of this new first row from each of the
other rows); Step 4 (eliminate the fourth value for rows 1, 2, 3; divide the fourth row by the value in the
second column and then subtract multiples of this new first row from each of the other rows)
b. Formal language basics
i. Regular languages are the simplest class of formal languages; used in the programming language compiler
to separate the program into a series of tokens that can be processed later; regular languages can be specified
by regular expressions
ii. A symbol is the simplest element; ex: a, b, c,…
iii. Alphabet  is a finite set of symbols; ex:  = {a, b, c}
iv. A word/string is a sequence of 0+ symbols from an alphabet; ex: babb, abab, bbbaaa
v. Empty word  is a word that has no symbols
vi. A letter can refer to a word; ex: w = bacb
vii. To represent the # of symbols of a word w   w = 4
viii. Word concatenation  w = bacb, x = aab, wx = bacbaab
ix. Reverse operator  x = aab, x r=baa
x. Concatenate a word itself 0+ times  x = aab, x 2=aabaab x 0=¿
xi. Star closure (Kleene star) w* represents the set containing 0+ concatenations of the word w; w = ab; w* =
{, ab, abab, ababab, abababab, …}
¿
xii. Star closure can be applied to alphabets;  = {a, b}, ❑ ={, a , b , aa , bb , ab , ba , aaa , bbb , aab … }
xiii. A language over an alphabet  is a subset of ❑¿; ex: L1 is a language over the alphabet  = {a,b}, then L1 
❑ , L1 = {a, ab, abb, bab, bba, abbb, babb, …}, it contains all words with one a, L1 = { x : ¿a ( x ) =1}
¿

xiv. Other examples: L2  ❑¿, L2 = {a n b n : n0 } this includes and contains all words with n number of letters;
L3  ❑¿, L3 = { x : ¿a ( x ) =2¿ b ( x ) } this includes and will include 2x more a’s for each b
xv. Language are sets; we can perform any of the normal set operations on languages; union, intersection,
complement; we can concatenate 2 languages; ex: L1 L2 = { x 1 x 2 : x 1 L1 , x2 L2 } we take any element from
L1 and concatenate it with the element of L2 and combine together
xvi. Lexical ordering  the words are grouped in order of increasing length, and within each group the words
are alphabetical; the lexical order of ❑¿ is , a, b, aa, ab, ba, bb, aaa, aab, aba, abb, baa, bab, bba, bbb, …
xvii. Language classifications  regular languages; context-free languages; a language that is regular is also
context-free but there are context-free languages that are not regular
xviii. Grammar  a grammar is a set of rules; a rule has 1+ symbols on the left side and 0+ on the right side;
grammar is used to derive words of a language; repeatedly replace symbols based on the production rules;
the language is the set of all the words that can be derived from the rules of the grammar; ex: {S aSb, S 
} generates and the same set of a’s and b’s {, ab , aabb ,aaabbb , …}, {SaSb, SbSa, SSS,S}
generates and same # of a’s and b’s are added
c. Finite automata  determines the words that are in a language; works using internal states/finite, an input tape, and
transition function; the automaton examines the current state and the next input symbol, transition function is used
to determine the new state; will output a simple accept or reject to indicate whether the input word is in the
language that the finite automaton is designed for; some of the states are identified as accepting state; if a finite
automaton stops in an accepting state after reading all of the input symbols, the word is accepted and is in the
language
i. Graph representation of finite automata  draw a transition graph; draw a circle for each internal state;
draw an arrow for each transition from the old state to the new state, place the input symbol above the
arrow; double circle indicates the accepting state(s)
ii. Finite automaton: configuration  a pair of elements (x, y); the 1st element represent the current state; the
2nd element represents the part of the input word yet to be examined; (current state, current string)
 iii. Regular languages  finite automata can be written to accept a variety of languages; but not all languages
in ❑¿; languages that we can write finite automata for are called regular languages; regular languages
include the set of reserved words for programming languages and the set of possible variable names,
operators
iv. Regular expressions  a regular language can be described by a regular expression; rules for creating
regular expressions: 1.) , , and all symbols  are primitive regular expressions, 2.) If r, r1, r2 are regular
expressions, then so are r1, +r2, r1*r2 and (r), 3.) A string represents a regular expression if and only if
symbols of step 1 and repeated application of step 2 can create it;  represents the empty language that has
no words,  represents an empty element in a language; regular expressions are associative operations (+
operator: union, . operator: concatenation, * operator: star closure); ex: (aaa)* includes  and any # of aaa’s,
a*b* includes  and any # of a’s and b’s, (a+b)* bbb(a+b)* includes any string made up of a’s and b’s but
must include bbb in the middle
v. Deterministic finite automata (DFA) means every state has exactly one edge leaving it for each of the
symbols of the alphabet; an input word will cause the exact same behavior every time a deterministic finite
automaton is run on that word
vi. Nondeterministic finite automata (NFA)  a state can have no edges for a symbol or can have multiple
edges for a symbol; it may have edges labeled , such edge can be taken without consuming any of the input
symbols; NFA will accept a word if there is at least one path through the automaton that consumes all of the
symbols of the input and winds up in an accepting state; for a word to be rejected, all possible paths through
the automaton must reject the word; any language that can be decided by a DFA can also be decided by a
NFA and vice versa
vii. Transition function  for DFA ((s1, a) = s2; each old state and input symbol combination leads to exactly
one new state) for NFA ((s1, a) = S; S is the set of states that can result from an old state of s1 and an input
symbol of a; Set S can have 0+ elements corresponding to the # of edges leaving state s1 that are labeled
with an a symbol)
d. Design finite automata  figure what words are in and not in the language and what is the pattern of the language
words; create the states and transitions we need to accept these words; finite automata restrictions (limits to the type
of languages they can accept; good at finding patterns and can count but up to known #s; they do not have any
memory and cannot compare one part of a word with another)
e. Design regular grammar  think of each of the nonterminal symbols as having a meaning or purpose; A  aaB, A
has the purpose of adding two a symbols to the word;
f. Finite automata equivalence and limitations DFA and NFA are equivalent; we can prove this by showing that we
can convert any NFA to DFA; subsets of states will be used as labels for the states of a new and equivalent DFA;
starting state of new DFA is labeled with the set containing the NFA initial state and all of the states that can be
reached from it with just  edges, then from a state in the new automata, which is a set of old states and an input
symbol, the destination new state is the union of all the places we can get from old states in the set using the
symbols, continue to examine the reachable destinations for every symbol and each new set we produce until we
stop producing new sets