Revisited:

How To Multiply Two Numbers

2X2=5
 Time complexity of
addition
***********
+ *********** T(n) = The amount of
time addition uses to
***********
add two n-bit
numbers
************

We saw that T(n) was linear.

T(n) = Θ(n)
 Time complexity of
school multiplication
X ********
********
T(n) = The amount of
********
********
******** time grade school
********
n2 ********
********
multiplication uses to
********
******** add two n-bit
****************
numbers

We saw that T(n) was quadratic.

T(n) = Θ(n2)
 addition

Addition is
linear time.

Is there a sub-linear time

method for addition?
Any addition algorithm takes Ω(n) time

Claim: Any algorithm for addition must read

all of the input bits

Proof: Suppose there is a mystery algorithm

A that does not examine each bit

Give A a pair of numbers. There must be

some unexamined bit position i in one of
the numbers
Any addition algorithm takes Ω(n) time

********* A did not

********* read this bit
at position i
**********
• If A is not correct on the inputs, we found a bug

• If A is correct, flip the bit at position i and give A

the new pair of numbers. A gives the same answer
as before, which is now wrong.
Addition can’t be
improved upon by
more than a
constant factor.
 Multiplication

Multiplication: Θ(n2) time

Is there a clever algorithm to

multiply two numbers in linear
time?
Repetitive addition

int mult( int n, int m)

{
if( m==0 ) return 0;
return mult(n, m-1) + n;
}

What is the complexity of

this algorithm?
Peasant multiplication

x1011
1001
------
1011
0000
0000
1011
-----------
1100011
Peasant multiplication

Why is it called the “Russian”

peasant’s algorithm?
Peasant multiplication

It’s also called the Egyptian

multiplication.
But if the number does not
fit the CPU?

Can we do something very

different than grade school
multiplication?
 Divide And Conquer

An approach to faster algorithms:

1. DIVIDE a problem into smaller

subproblems

2. CONQUER them recursively

3. GLUE the answers together so as to obtain

the answer to the larger problem
 Multiplication of 2 n-bit numbers

n bits

X= a X b
Y= c Y d
n/2 bits n/2 bits

X = a 2n/2 + b Y = c 2n/2 + d

X × Y = ac 2n + (ad + bc) 2n/2 + bd

 Same thing for decimals

n digits

X= a b
Y= c d
n/2 digits n/2 digits

X = a 10n/2 + b Y = c 10n/2 + d

X × Y = ac 10n + (ad + bc) 10n/2 + bd

Multiplying (Divide & Conquer style)

12345678 * 21394276

*4276 5678*2139 5678*4276

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

12*21 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

12*21 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

12*21 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

12*21 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

12*21 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

12*21 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

1*2 1*1 2*2 2*1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

12*21 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

2 1*1
1*2 1 2*2
4 2
2*1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Hence: 12*21 = 2*102 + (1 + 4)101 + 2 = 252

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

252 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

2 1*1
1*2 1 2*2
4 2
2*1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Hence: 12*21 = 2*102 + (1 + 4)101 + 2 = 252

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

252 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

1*2 1*1 2*2 2*1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Hence: 12*21 = 2*102 + (1 + 4)101 + 2 = 252

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

252 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

1*3 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1*9 2*3 2*9

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

242 12*39 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

3 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
9 6 18
*102 + *101 + *101 + *1

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

242 468 34*21 34*39 xxxxxxxxxxxxxxxxxxxxxxxxx

3 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
9 6 18
*102 + *101 + *101 + *1

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

252 12*39
12*21 468 714 34*39
34*21 1326 xxxxxxxxxxxxxxxxxxxxxxxxx

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

252 12*39
12*21 468 714 34*39
34*21 1326 xxxxxxxxxxxxxxxxxxxxxxxxx
*10 + *10 + *10 + *1
4 2 2

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Multiplying (Divide & Conquer style)

12345678 * 21394276

1234*2139 1234*4276 5678*2139 5678*4276

252 12*39
12*21 468 714 34*39
34*21 1326 xxxxxxxxxxxxxxxxxxxxxxxxx
*10 + *10 + *10 + *1
4 2 2

= 2639526

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
Multiplying (Divide & Conquer style)

12345678 * 21394276

2639526 1234*4276 5678*2139 5678*4276

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
Multiplying (Divide & Conquer style)

12345678 * 21394276

2639526 5276584 12145242 24279128

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
Multiplying (Divide & Conquer style)

12345678 * 21394276

2639526 5276584 12145242 24279128

*108 + *104 + *104 + *1

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
Multiplying (Divide & Conquer style)

12345678 * 21394276

2639526 5276584 12145242 24279128

*108 + *104 + *104 + *1

= 264126842539128

X= a b
Y= c d
X × Y = ac 10n + (ad + bc) 10n/2 + bd
 Divide, Conquer, and Glue

MULT(X,Y):
X=a;b Y=c;d
 Divide, Conquer, and Glue

MULT(X,Y):
X=a;b Y=c;d

Mult(a,c) Mult(a,d) Mult(b,c) Mult(b,d)

 Divide, Conquer, and Glue

MULT(X,Y):
X=a;b Y=c;d

Mult(a,d) Mult(b,c) Mult(b,d)

Mult(a,c)
 Divide, Conquer, and Glue

MULT(X,Y):
X=a;b Y=c;d
ac

ac
Mult(a,d) Mult(b,c) Mult(b,d)
 Divide, Conquer, and Glue

MULT(X,Y):
X=a;b Y=c;d
ac,

ac
Mult(b,c) Mult(b,d)

Mult(a,d)
 Divide, Conquer, and Glue

MULT(X,Y):
X=a;b Y=c;d
ac, ad,

ac
ad Mult(b,d)
Mult(b,c)
 Divide, Conquer, and Glue

MULT(X,Y):
X=a;b Y=c;d
ac, ad,

ac
ad Mult(b,d)

Mult(b,c)
 Divide, Conquer, and Glue

MULT(X,Y):
X=a;b Y=c;d
ac, ad, bc,

ac bc
ad Mult(b,d)
 Divide, Conquer, and Glue

MULT(X,Y):
X=a;b Y=c;d
ac, ad, bc,

ac bc
ad

Mult(b,d)
 Divide, Conquer, and Glue
XY =
ac2n
+(ad+bc)
MULT(X,Y): 2n/2
X=a;b Y=c;d + bd
ac, ad, bc, bd

ac bd
ad bc
 Time required by MULT

T(n) = time taken by MULT on two n-bit

numbers

What is T(n)? What is its growth rate?

Big Question: Is it Θ(n2)?
 Time required by MULT

X=a;b Y=c;d

XY = ac2n + (ad + bc)2n/2 + bd

ac bd
ad bc

T(n/2) T(n/2) T(n/2) T(n/2)

 T(n) = 4 T(n/2) + (c1 n + c2)

Conquering divide and

time glue
X=a; b Y=c; d

XY = ac2n + (ad + bc)2n/2 + bd

ac bd
ad bc

T(n/2) T(n/2) T(n/2) T(n/2)

 Recurrence Relation

T(1) = c3

T(n) = 4 T(n/2) + c1 n + c2

What is the growth rate of T(n)?

 Let’s keep it simple

T(1) = 1

T(n) = 4 T(n/2) + n

What is the growth rate of T(n)?

 Technique:
Guess and Verify
Guess: T(n) = 2n2 – n

Verify: T(1) = 1 and T(n) = 4 T(n/2) + n

2n2 – n = 4 [2(n/2)2 – n/2] + n

= 2n2 – 2n +n
= 2n2 – n
 Recursion Tree Representation

We visualize the
recursion as a tree,
where each node
represents a
recursive call. The
root is the initial call.
Leaves correspond to
the exit condition.
 T(n) = 4 T(n/2) + n

T(n)

T(n/2) T(n/2) T(n/2) T(n/2)

T(n/4) T(n/4) T(n/4) T(n/4)

 Recursion Tree
each node
reflects the
n combining
step

n/2 n/2 n/2 n/2

n/4 n/4 n/4 n/4

 Recursion Tree
write down
the work at n
each level n

n/2 n/2 n/2 n/2 2n

n/4 n/4 n/4 n/4 4n

 0 n = 1*n

1 n/2 + n/2 + n/2 + n/2 = 2*n

2 n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 = 4*n

i Level i is the sum of 4i copies of n/2i = 2*in

. . . . . . . . . . . . . . . . . . . . . . . . . .
log2n
1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+ 1+1+1+1+1+1+1+1+1+1+1+1
= n*n

T(n)=n (1+2+4+8+ . . . +2h), where h = log2n

T(n) = n(2n-1) = Θ(n2)

 Xn+1 - 1
1 + X1 + X2 + X3 + … + Xn-1 + Xn =
X- 1

The Geometric Series

T(n)=n (1+2+4+8+ . . . +2h), where h = log2n

T(n) = n(2n-1) = Θ(n2)

Divide and Conquer MULT: Θ(n2) time
Multiplication: Θ(n2) time

All that work

for nothing!
Karatsuba, Anatolii Alexeevich (1937-)

In 1962 Karatsuba
had formulated the
first algorithm to
break n2 barrier!
 Multiplication of 2 n-bit numbers
X = a 2n/2 + b Y = c 2n/2 + d
X × Y = ac 2n + (ad + bc) 2n/2 + bd

z1 = (a + b) (c + d) = ac + ad + bc + bd
z2 = ac
z3 = bd
z4 = z2 – z3 = ac - bd
z5 = z1 – z2 – z3 = ad + bc
X × Y = z2 2n + z5 2n/2 + z3
 Karatsuba (1962)

T(1) = 1
T(n) = 3 T(n/2) + n

What is the growth rate of T(n)?

T(n) = n

T(n/2) T(n/2) T(n/2)

T(n) = n

n/2

T(n/2) T(n/2)
T(n/4)T(n/4)T(n/4)
T(n) = n

n/2 n/2 n/2

T(n/4)T(n/4)T(n/4) T(n/4)T(n/4)T(n/4) T(n/4)T(n/4)T(n/4)

 0 n

1 n/2 + n/2 + n/2

2 n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4

i Level i is the sum of 3i copies of n/2i

. . . . . . . . . . . . . . . . . . . . . . . . . .
log2n 1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1
 0 n = 1n
1 n/2 + n/2 + n/2 = 3/2n
2 n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 + n/4 = 9/4n

i Level i is the sum of 3i copies of n/2i = (3/2)i n

. . . . . . . . . . . . . . . . . . . . . . . . . .
log2n
1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+1+ 1+1+1+1+1+1+1+1+1+1+1+1
= (3/2)log n n

T(n) = n (1+3/2+(3/2)2+ . . . + (3/2)log2 n) = ??

 The Geometric Series
Xk+1 - 1
1 + X1 + X2 + X3 + … + Xk-1 + Xk =
X- 1

We have: X = 3/2 k = log2 n

(3/2) × (3/2)log2 n - 1
= 3 × (3log2 n/2log2 n) - 2
½
= 3 × (3log2 n/n) - 2
 Dramatic improvement for large n

T(n) = 3 nlog2 3 – 2n = Θ(n log2 3) = Θ(n1.58…)

A huge savings over Θ(n2) when n gets large.

 Recursion Tree Representation

T(n) = a T(n/b) + f(n)

f(n)

a nodes
f(n/b) f(n/b) f(n/b) f(n/b)

f(n/b2) f(n/b2) f(n/b2) f(n/b2)

 Exercise

Solve the following

recurrence

T(n) = p T(n/3) + n

where p>0.
 T(n) = p T(n/3) + n

Let p>3 be an arbitrary parameter.

Using the tree method, we obtain

T(n) = n(1 + p/3 + p2/9 + …+ (p/3)h)
where the tree height is log3n.
It follows
T(n) = Θ(nlog3 p)
Recurrence

T(n) = p T(n/3) + n

has the following

asymptotic solution

T(n) = Θ (n log3 p)

when p>3.
 T(n) = p T(n/3) + n

Cases p =1, 2 and 3 should

be considered separately

If p =3, then
T(n) = Θ (n log n)
3-Way Multiplication

X = a1 2n/4 + a2 2n/2 + a3

Y = b1 2n/4 + b2 2n/2 + b3

T(1) = 1
T(n) = 9 T(n/3) + n
 3-Way Multiplication

T(n) = p T(n/3) + n

T(n) = Θ(nlog3p)

To get an improvement over

Karatsuba’s, we have to decrease
the number of multiplications to
at least 5.
Toom and Cook (1963, 1966)

T(1) = 1
T(n) = 5 T(n/3) + n

T(n) = Θ(n log3 5) = Θ(n1.46…)

 3-Way Multiplication

X = a1 2n/4 + a2 2n/2 + a3

Y = b1 2n/4 + b2 2n/2 + b3

Is it possible to reduce the number of

multiplications to 5?
 Multiplication Algorithms

Grade School n2

Karatsuba n1.58…

3-way n1.46…

FFT n log2n

Schönhage and Strassen n logn loglogn