Classification of problem & problem solving

strategies

classification of time complexities (linear, logarithmic

etc)

Problem subdivision –Divide and Conquer strategy.

Asymptotic notations, lower bound and upper bound:

Best case, worst case, average case analysis,

amortized analysis.

Performance analysis of basic programming

constructs.
 Problem solving is the application of
ideas, skills, or factual information to
achieve the solution to a problem or to
reach a desired outcome. Let's talk
about different types of problems and
different types of solutions.
 A well-defined problem is one that has
a clear goal or solution, and problem
solving strategies are easily developed.
In contrast,
 poorly-defined problem is the
opposite. It's one that is unclear,
abstract, or confusing, and that does not
have a clear problem solving strategy.
 routine problem is one that is typical and
has a simple solution. In contrast,
 non-routine problem is more abstract or
subjective and requires a strategy to solve.
 The first strategy you might try when solving
a routine problem is called an algorithm.
Algorithms are step-by-step strategies or
processes for how to solve a problem or
achieve a goal.
 Another solution that many people use to solve
problems is called heuristics. Heuristics are
general strategies used to make quick, short-
cut solutions to problems that sometimes lead
to solutions but sometimes lead to errors.
Heuristics are based on past experiences.
 Identify a problem
 Understand the problem

 Identify alternative ways to solve a

problem
 Select beat way to solve a problem from
the
list of alternative solutions
 Evaluate the solution
What is an
algorithm?
An algorithm is a finite set of precise
instructions for performing a computation or
for solving a problem.

This is a rather vague definition. You will get

to know a more precise and mathematically
useful definition when you attend CS420.

But this one is good enough for now…

CMSC 203 - Discrete Fall 2002 8

Structures
Properties of
algorithms:
 Input from a specified set,
 Output from a specified set (solution),
 Definiteness of every step in the
computation,
 Correctness of output for every possible
input,
 Finiteness of the number of calculation
steps,
 Effectiveness of each calculation step and

CMSC 203 - Discrete Fall 2002 9

Structures
 Why we should analyze
algorithms?
◦ Predict the resources that the algorithm
requires
 Computational time (CPU consumption)
 Memory space (RAM consumption)
 Communication bandwidth consumption
◦ The running time of an algorithm is:
 The total number of operations executed
 Also known as algorithm complexity

10
Interna Externa
l l
Factors Factors
Space Processo
Complexit r
y Quality

CPU
Time Speed
Complexit
y 11
 Time complexity of an algorithm signifies the
total time required by the program to run to
 completion.
Time complexity of an algorithm is measured by
its
rate of growth relative to the standard function.
 Cases of time complexity are:
 Worst-case
◦ An upper bound on the running time for any input
of given size
 Average-case
◦ Assume all inputs of a given size are equally likely
 Best-case
◦ The lower bound on the running time

12
 Sequential search in a list of size
n ◦ Worst-case:
 n comparisons
◦ Best-case:
 1 comparison
◦ Average-case:
 n/2 comparisons
 The algorithm runs in linear
time
◦ Linear number of operations

13
Notation Complexit Description Example
y
O(1) Constant Simple statement Addition

O(log(n)) Logarithmic Divide in half Binary search

O(n) Linear loop Linear search
O(n*log(n) Linearithmic Divide & Conquer Merge sort
)
O(n2) Quadratic Double loop Check all pairs
O(n3) Cubic Triple loop Check all triples
O(2n) Exponential Exhaustive search Check all
subsets
O(n!) Factorial Recursive function Factorial

14
 Statement
S
1tatement 2
.
.
.
Statement k

 Total time=time(statement 1)+time(statement 2)

….
+time(statement k)
 Each statement is simple so takes only
o(1) ie constant time
 For (i=1;i<n;i+
+)
{
Printf
(i)
}

Runs in O(n) where n is the size of the

array
 Int i=1
While(i<10
)
{
printf(i)
;
i=i*2;
}

Runs in logarithmic time O(log

n)
 algo Sum
{
for (int x=0; x<n; x+
+) for (int y=0;
y<n; y++)
sum += x*y;

Runs in quadratic time

O(n2)
 algo Sum
{
int sum = 0;
for (int a=0; a<n; a++)
for (int b=0; b<n;
b++)
for (int c=0; c<n;
c++) sum +=
a*b*c;
}

 Runs in cubic time

O(n3)
 decimal Fibonacci(int
{
n)
if (n == 0)
return 1;
else if (n ==
1)
return
1; else
retu
rn
Fibonacc
 Runs in exponential time O(2n)
i(n-1) +
 The number of elementary steps
Fibonacc ~
i(n-2);
isFib(n+1)
}
where Fib(k) is the k-th
Fibonacci's
number
 Running time of an algorithm as a function
of input size n.
 Expressed using only the highest-order
term in the expression for the exact
running time.
 Describes behavior of function in the
limit.
 Written using Asymptotic Notation.

Comp 122
 Big Oh
 Big thetaONotation : 
Notation:
 Big Omega Notation 
 Small Oh Notation : o

 Small Omega Notation :

 Defined for functions over the
natural numbers.
 Define a set of functions; in practice used
to compare two function sizes.The
notations describe different rate-of-
growth.
Comp 122
Let f(n) and g(n) be
two
functions then we can
say that f(n) = O(g(n))
if and only if there
exists positive
constants c and n0,
such that
f(n) ≤ cg(n) for all n≥n0
Ans: In the given
Problem
Ans: In the given Problem
Let f(n) and g(n) be
two
functions then we can
say that f(n) = (g(n))
if and only if there
exists positive
constants c and n0,
such that
f(n) ≥ cg(n) for all
n≥n0
Ans:
Ans:
Let f(n) and g(n) be
two functions then we
can
say that f(n) = (g(n))
if and only if there
exists positive
constants c1,c2 and n0,
such that
0≤ c1g(n) ≤ f(n) ≤
c2g(n) for all n≥n0
Ans:
Key point: The time required to perform
a sequence of data structure
operations is averaged over all
operations performed
 Amortized
◦ The average analysis can
cost of an be used
operation to
is small
show If that
one averages over a sequence of operations
even though a single operation might be
expensive
The most common three
◦ The aggregate method
techniques
◦ The accounting method
◦ The potential method
If there are several types of operations in
a sequence
 The aggregate method assigns
◦ The same amortized cost to each operation
 The accounting method and the
potential method may assign
◦ Different amortized costs to different types
of
operations
 Show that sequence of n operations
takes
◦ Worst case time T(n) in total for all n
 The amortized cost (average cost in the
worst case) per operation is therefore
T(n)n
 This amortized cost applies to each
operation
◦ Even when there are several types of operations
in
the sequence
PUSH(S, x): pushed object x onto stack
POP(S): pops the top of the stack S and
returns the
popped object
MULTIPOP(S, k): removes the k top objects of
the
stack S or pops the entire stack if |S |  k
 PUSH and POP runs in (1) time
◦ The total cost of a sequence of n PUSH and
POP operations is therefore (n)
 The running time of MULTIPOP(S, k) is
◦ (min(s, k)) where s  | S |
 Let us analyze a sequence of n POP, PUSH,
MULTIPOP operations on an initially empty stack
and
 The worst case of a MULTIPOP operation in
the sequence is O(n)
 Hence, a sequence of n operations costs
O(n2)
◦ we may have n MULTIPOP operations each costing
O(n)
 The analysis is correct, however,
◦ Considering worst-case cost of each operation, it is
not tight
 We can obtain a better bound by using
aggregate method of amortized analysis
Aggregate method considers the
entire
◦ Although aof
sequence n operations
single MULTIPOP can be expensive
◦ Any sequence of n POP, PUSH, and MULTIPOP
operations on an initially empty sequence can cost
at most O(n)

Proof: each object can be popped once for each time it

is pushed. Hence the number of times that POP can be
called on a nonempty stack including the calls within
MULTIPOP is at most the number of PUSH operations,
which is at most n
The amortized cost of an operation is the average
O(n)n
 O(1)
 The Accounting
Method:
 We assign different charges to different operations
with some operations charged more or less than they
actually cost
 The amount we charge an operation is called
its amortized cost
 When the amortized cost of an operation exceeds its
actual cost the difference is assigned to specific
objects in the data structure as credit
 Credit can be used later to help pay for operations
whose amortized cost is less than their actual cost
 That is, amortized cost of an operation can be
considered as being split between its actual cost
and credit (either deposited or used)
 The Accounting
Method:
Stack Operations
Assign the following amortized costs:

Push: 2 Pop: 0 Multipop: 0 We

start with an empty stack of plates

When we push a plate on the stack
• we use $1 to pay the actual cost of the push
operation
• we put a credit of $1 on top of the pushed plate
At any time point, every plate on the stack has a $1 of credit
on it. The $1 stored on the plate is a prepayment for the cost
of popping it.
In order to pop a plate from the stack
• we take $1 of credit off the plate
• and use it to pay the actual cost of the pop operation
 The Accounting Method:
Stack Operations
Thus by charging the push operation a little bit
more we don’t need to charge anything from
the pop & multipop operations
We have ensured that the amount of credits is
always nonnegative
Thus, for any sequence of n push, pop,
multipop operations the total amortized cost
is an upper bound on the total actual cost
 The Potential
Method:
 Potential method represents the prepaid work
as potential energy that can be released to
pay for the future operations
 The potential is associated with the data
structure as a whole rather than with specific
objects within the data structure
 The Potential Method
Initial Datastructure on which we perform n
D0
operations
:C : the actual cost of the i-th operation
i

Di: data structure that results after applying i-th operation to

Di1
 : potential function that maps each data structure D to a
real i
number  (D )
i
(Dˆi)::the
amortized cost
potential of the i-th
associated with data structure Di
C operation
i
 The Potential
Method
Ciˆ  Ci   (Di )   (Di1 )
actual
increase in potential
cost
due to the operation

The total amortized cost of n

operations
n is:n
 Cˆ 
i i1
C i


 (Dn )
  (D0
 Define  (S)| S |, the number of objects
inthe stack
 For the initial empty stack, we have  (D0)

0
 Since |S|  0, stack Di that results after i
th operation has nonnegative potential
 total amortized cost is an upper bound on
total
actual cost
 Let us compute the amortized costs of stack
operations where i th operation is
performed on a stack with s objects
PUSH  
Cˆi  Ci  (Di)  (Di1)  1  i -(i-1) 
(S): 2
MULTIPOP(S, k): suppose k’ elements are popped

Cˆi  Ci   (D )  (D
i i1 )  k' +(i-k' )-j 0

POP (S): Cˆi  Ci   ( D )  ( D

i i1 )  1  (i -1)-i  0

 The amortized cost of each operation is O(1),

and thus the total amortized cost of a
sequence of n operations is O(n)
 7 29 4  2 4 7 9

72  2 7 94  4 9

77 22 99 44

Divide-and conquer is a general algorithm
design
paradigm:
◦ Divide: divide the input data S in two or more
disjoint subsets S1, S2, …
◦ Conquer : solve the subproblems recursively
◦ Combine: combine the solutions for S1, S2, …, into
a solution for S
Analysis can be done using recurrence
equations
 Merge-sort on an input Algorithm mergeSort(S, C)
sequence S with n Input sequence S with n
elements
elements consists of
Output sequence S sorted
three steps: if S.size() > 1
◦ Divide: partition S into (S1, S2)  partition(S, n/2)
two sequences S1 and S2
mergeSort(S1, C)
of about n2 elements
mergeSort(S2, C)
each
◦ Recur: recursively sort S1 S  merge(S1, S2)
and S2
◦ Conquer: merge S1 and
S2 into a unique sorted
sequence
 The conquer step of merge-sort consists of merging two
sorted sequences, each with n2 elements and
implemented by means of a doubly linked list, takes at
most bn steps, for some constant b.
 Likewise, the basis case (n < 2) will take at b most steps.
 Therefore, if we let T(n) denote the running time of
merge- sort:

 b if
T (n) 
n
 2T (n / 2) 
 We can therefore analyze the running  2
time if
of merge-sort
by finding a closedbn form solution to the above
equation.
n
◦ That is, a solution that has T(n) only on the left-hand side.
2
 Draw the recursion tree for the recurrence relation and
look for a pattern:
 b if n  2
T (n) 
if n 
 2T (n / 2) 
bn 2 time
depth T’s size
0 1 n bn

1 2 n2 bn

2 4 n4 bn

i n2i …
2i

Total time = bn + bn log n

Divide-and-Conquer 53
Comp 122, Spring 2004 19 June 2015
 Given: a divide and conquer
algorithm
◦ An algorithm that divides the problem of size n
into
a subproblems, each of size n/b
◦ Let the cost of each stage (i.e., the work to
divide the problem + combine solved
subproblems) be described by the function f(n)

 Then, the Master Theorem gives us a

cookbook for the algorithm’s running
time:
 Many divide-and-conquer recurrence
equations have the form:
 c if n 
T (n) 
ifd n 
 aT (n / b) f (n)
d
 The Master Theorem:
1. if f (n) is O(n lo g b a 
), then T (n) is (n lo g a ) b

2. if f (n) is (n lo g a log k n), then T (n) is (n lo g a log k 1

b b

n)
3. if f (n) is (n lo g b a 
), then T (n) is ( f (n)),
provided af (n / b)  f (n) for some   1. for all
n≥d Divide-and-Conquer 56
 if T(n) = aT(n/b) + f(n)
then
 

  
logb a
f (n)  O n logb a 

n 
    0

T (n)   n lo g a log
b
f (n)   n 
lo g a
b

n c

 f 
f (n)   n lo g a 
b 1


(n)  (n / b)  cf (n) for large n
af AND
 T(n) = 9T(n/3) +
n
◦ a=9, b=3, f(n) = n

◦nlog b a = nlog3 9 = n2

◦ Since f(n) = O(nlog 3 9- ), where =1, case 1

applies:


T (n )   n lo g b a
w hen 
f (n )  O n lo g b

a 

  The Master Theorem:
log a 
1. if f (n) is O(n b
), then T (n) is (n lo g a ) b

2. if f (n) is (n lo gb a log k n), then T (n) is (n lo g a log k 1 n)

b

3. if f (n) is (n l o g b a   ), then T (n) is ( f (n)),

provided af (n / b)  f (n) for for all n≥d

some   1.
 Example: T (n)  4T (n / 2)
Solution
n
:so case 1 says T(n) is
n logb a = nlog2 4 = n2
n )
2

Divide-and-Conquer 59
  The Master Theorem:
1. if f (n ) is O (n lo g b a 
), t hen T (n ) is  (n lo g a )
b

2. if f (n ) is  (n lo g b a
log k n ), t hen T (n ) is  (n lo g b a
log k 1
n)
3. if f (n ) is  (n lo g b a 
), t hen T (n ) is  ( f (n )),
provided af (n / b)   f (n) for some   1. for all n≥d

Example: T (n)  2T (n / 2)  n
log n
Solution: nlogb a = nlog2 2 = n,
so case 2 says: T(n) = (n
log2 n).
60
 The Master Theorem:
a 
1. if f (n) is O (n lo g b
), then T (n) is  (n lo g a )
b

2. if f (n) is  (n lo g b a
log k n), then T (n) is  (n lo g b a
log k 1
n)
a 
3. if f (n) is  (n lo g b
), then T (n) is  ( f
(n)), provided af (n / b)   f (n) for some
   1.
Example: T (n)  T (n / 3)  n
log n
Solution: nlogb a = nlog3 1 = n0,
so case 3 says T(n) is (n log
n).
Divide-and-Conquer 61

Example:
T (n)  8T (n / 2) 
n2
Solution: logba=3, so case 1 says T(n) is  (n3).

T (n)  9T (n / 3) 
n3
Solution: logba=2, so case 3 says T(n) is  (n3).

T (n)  T (n / 2) 1
Solution: logba=0, so case 2 says T(n) is  (log