CS 473: Algorithms

Ruta Mehta
University of Illinois, Urbana-Champaign

Spring 2021

Ruta (UIUC) CS473 1 Spring 2021 1 / 55

CS 473: Algorithms, Spring 2021

Polynomials, Convolutions
and FFT
Lecture 2
Jan 26/28, 2021

Most slides are courtesy Prof. Chekuri

Ruta (UIUC) CS473 2 Spring 2021 2 / 55
Outline
Discrete Fourier Transfor (DFT) and Fast Fourier Transform (FFT)
have many applications and are connected to important mathematics.

“One of top 10 Algorithms of 20th Century” according to IEEE.

Gilbert Strang: “The most important numerical algorithm of our
lifetime”.

Our goal:
Multiplication of two degree n polynomials in O(n log n) time.
Surprising and non-obvious.
Algorithmic ideas
change in representation
mathematical properties of polynomials
divide and conquer

Ruta (UIUC) CS473 3 Spring 2021 3 / 55

 Part I

Polynomials, Convolutions and FFT

Ruta (UIUC) CS473 4 Spring 2021 4 / 55

Polynomials
Definition
A polynomial is a function of one variable built from additions,
subtractions and multiplications (but no divisions).
n−1
X
p(x) = aj x j
j =0

The numbers a0 , a1 , . . . , an are the coefficients of the polynomial.

The degree is the highest power of x with a non-zero coefficient.

Example
p(x) = 3 − 4x + 5x 3
a0 = 3, a1 = −4, a2 = 0, a3 = 5 and deg (p) = 3
Ruta (UIUC) CS473 5 Spring 2021 5 / 55
Polynomials

Definition
A polynomial is a function of one variable built from additions,
subtractions and multiplications (but no divisions).
n−1
X
p(x) = aj x j
j =0

The numbers a0 , a1 , . . . , an are the coefficients of the polynomial.

The degree is the highest power of x with a non-zero coefficient.

Coefficient Representation
Polynomials represented by vector a = (a0 , a1 , . . . an−1 ) of
coefficients.

Ruta (UIUC) CS473 5 Spring 2021 5 / 55

Operations on Polynomials

Evaluate Given a polynomial p and a value α, compute p(α)

Add Given (representations of) polynomials p, q, compute
(reprsentation of) polynomial p + q
Multiply Given (representation of) polynomials p, q, compute
(representation of) polynomial p · q.
Roots Given p find all roots of p.

Ruta (UIUC) CS473 6 Spring 2021 6 / 55

Evaluation
Compute value of polynomial a = (a0 , a1 , . . . an−1 ) at α
power = 1
value = 0
for j = 0 to n − 1
// invariant: power = αj
value = value + aj · power
power = power · α
end for
return value
How many additions?

Ruta (UIUC) CS473 7 Spring 2021 7 / 55

Evaluation
Compute value of polynomial a = (a0 , a1 , . . . an−1 ) at α
power = 1
value = 0
for j = 0 to n − 1
// invariant: power = αj
value = value + aj · power
power = power · α
end for
return value
How many additions? n

Ruta (UIUC) CS473 7 Spring 2021 7 / 55

Evaluation
Compute value of polynomial a = (a0 , a1 , . . . an−1 ) at α
power = 1
value = 0
for j = 0 to n − 1
// invariant: power = αj
value = value + aj · power
power = power · α
end for
return value
How many additions? n
How many multiplications?

Ruta (UIUC) CS473 7 Spring 2021 7 / 55

Evaluation
Compute value of polynomial a = (a0 , a1 , . . . an−1 ) at α
power = 1
value = 0
for j = 0 to n − 1
// invariant: power = αj
value = value + aj · power
power = power · α
end for
return value
How many additions? n
How many multiplications? 2n

Ruta (UIUC) CS473 7 Spring 2021 7 / 55

Evaluation
Compute value of polynomial a = (a0 , a1 , . . . an−1 ) at α
power = 1
value = 0
for j = 0 to n − 1
// invariant: power = αj
value = value + aj · power
power = power · α
end for
return value
How many additions? n
How many multiplications? 2n
Horner’s rule can be used to cut the multiplications in half

a(x) = a0 + x(a1 + x(a2 + x(· · · + xan−1 ) · · · ))

Ruta (UIUC) CS473 7 Spring 2021 7 / 55

Evaluation: Numerical Issues

Question
How long does evaluation really take? O(n) time?

Bits to represent αn is n log α while bits to represent α is only

log α. Thus, need to pay attention to size of numbers and
multiplication complexity.

Ignore this issue for now. Can get around it for applications of
interest where one typically wants to compute p(α) mod m for
some number m.

Ruta (UIUC) CS473 8 Spring 2021 8 / 55

Addition

Compute the sum of polynomials

a = (a0 , a1 , . . . an−1 ) and b = (b0 , b1 , . . . bn−1 )

Ruta (UIUC) CS473 9 Spring 2021 9 / 55

Addition

Compute the sum of polynomials

a = (a0 , a1 , . . . an−1 ) and b = (b0 , b1 , . . . bn−1 )
a + b = (a0 + b0 , a1 + b1 , . . . an−1 + bn−1 ). Takes O(n) time.

Ruta (UIUC) CS473 9 Spring 2021 9 / 55

Multiplication

Compute the product of polynomials

a = (a0 , a1 , . . . an ) and b = (b0 , b1 , . . . bm )
Recall a · b = (c0 , c1 , . . . cn+m ) where
X
ck = ai · bj
i ,j : i +j =k

Takes Θ(nm) time; Θ(n2 ) when n = m.

Ruta (UIUC) CS473 10 Spring 2021 10 / 55

Multiplication

Compute the product of polynomials

a = (a0 , a1 , . . . an ) and b = (b0 , b1 , . . . bm )
Recall a · b = (c0 , c1 , . . . cn+m ) where
X
ck = ai · bj
i ,j : i +j =k

Takes Θ(nm) time; Θ(n2 ) when n = m.

We will obtain a better algorithm!

Ruta (UIUC) CS473 10 Spring 2021 10 / 55

Multiplication

Compute the product of polynomials

a = (a0 , a1 , . . . an ) and b = (b0 , b1 , . . . bm )
Recall a · b = (c0 , c1 , . . . cn+m ) where
X
ck = ai · bj
i ,j : i +j =k

Takes Θ(nm) time; Θ(n2 ) when n = m.

We will obtain a better algorithm!
Better/Efficient/Easy (today’s lecture): preferably O(n + m), but
O(n log n) is also okay.

Ruta (UIUC) CS473 10 Spring 2021 10 / 55

Convolutions

Definition
The convolution of vectors a = (a0 , a1 , . . . an ) and
b = (b0 , b1 , . . . bm ) is the vector c = (c0 , c1 , . . . cn+m ) where
X
ck = ai · bj
i ,j : i +j =k

Convolution of vectors a and b is denoted by a ∗ b. In other words,

the convolution is the coefficients of the product of the two
polynomials.

Ruta (UIUC) CS473 11 Spring 2021 11 / 55

Revisiting Polynomial Representations

Representation
Polynomials represented by vector a = (a0 , a1 , . . . an−1 ) of
coefficients.

Ruta (UIUC) CS473 12 Spring 2021 12 / 55

Revisiting Polynomial Representations

Representation
Polynomials represented by vector a = (a0 , a1 , . . . an−1 ) of
coefficients.

Question
Are there other useful ways to represent polynomials?

Ruta (UIUC) CS473 12 Spring 2021 12 / 55

Representing Polynomials by Roots
Root of a polynomial p(x): r such that p(r ) = 0. If
r1 , r2 , . . . , rn−1 are roots then
p(x) = an−1 (x − r1 )(x − r2 ) . . . (x − rn−1 ).

Valid representation because of:

Theorem (Fundamental Theorem of Algebra)

Every polynomial p(x) of degree d has exactly d roots r1 , r2 , . . . , rd
where the roots can be complex numbers and can be repeated.

Ruta (UIUC) CS473 13 Spring 2021 13 / 55

Representing Polynomials by Roots

Representation
Polynomials represented by vector scale factor an−1 and roots
r1 , r2 , . . . , rn−1 .

Ruta (UIUC) CS473 14 Spring 2021 14 / 55

Representing Polynomials by Roots

Representation
Polynomials represented by vector scale factor an−1 and roots
r1 , r2 , . . . , rn−1 .

Evaluating p at a given x is easy. Why?

Ruta (UIUC) CS473 14 Spring 2021 14 / 55

Representing Polynomials by Roots

Representation
Polynomials represented by vector scale factor an−1 and roots
r1 , r2 , . . . , rn−1 .

Evaluating p at a given x is easy. Why?

Multiplication: given p, q with roots r1 , . . . , rn−1 and
s1 , . . . , sm−1 the product p · q has roots
r1 , . . . , rn−1 , s1 , . . . , sm−1 . Easy! O(n + m) time.

Ruta (UIUC) CS473 14 Spring 2021 14 / 55

Representing Polynomials by Roots

Representation
Polynomials represented by vector scale factor an−1 and roots
r1 , r2 , . . . , rn−1 .

Evaluating p at a given x is easy. Why?

Multiplication: given p, q with roots r1 , . . . , rn−1 and
s1 , . . . , sm−1 the product p · q has roots
r1 , . . . , rn−1 , s1 , . . . , sm−1 . Easy! O(n + m) time.
Addition: requires Ω(nm) time?

Ruta (UIUC) CS473 14 Spring 2021 14 / 55

Representing Polynomials by Roots

Representation
Polynomials represented by vector scale factor an−1 and roots
r1 , r2 , . . . , rn−1 .

Evaluating p at a given x is easy. Why?

Multiplication: given p, q with roots r1 , . . . , rn−1 and
s1 , . . . , sm−1 the product p · q has roots
r1 , . . . , rn−1 , s1 , . . . , sm−1 . Easy! O(n + m) time.
Addition: requires Ω(nm) time?
Given coefficient representation, how do we go to root
representation? No finite algorithm because of potential for
irrational roots.

Ruta (UIUC) CS473 14 Spring 2021 14 / 55

Representing Polynomials by Samples

Let p be a polynomial of degree n − 1.

Pick n distinct samples x0 , x1 , x2 , . . . , xn−1
Let y0 = p(x0 ), y1 = p(x1 ), . . . , yn−1 = p(xn−1 ).

Representation
Polynomials represented by (x0 , y0 ), (x1 , y1 ), . . . , (xn−1 , yn−1 ).

Ruta (UIUC) CS473 15 Spring 2021 15 / 55

Representing Polynomials by Samples

Let p be a polynomial of degree n − 1.

Pick n distinct samples x0 , x1 , x2 , . . . , xn−1
Let y0 = p(x0 ), y1 = p(x1 ), . . . , yn−1 = p(xn−1 ).

Representation
Polynomials represented by (x0 , y0 ), (x1 , y1 ), . . . , (xn−1 , yn−1 ).

Is the above a valid representation?

Ruta (UIUC) CS473 15 Spring 2021 15 / 55

Representing Polynomials by Samples

Let p be a polynomial of degree n − 1.

Pick n distinct samples x0 , x1 , x2 , . . . , xn−1
Let y0 = p(x0 ), y1 = p(x1 ), . . . , yn−1 = p(xn−1 ).

Representation
Polynomials represented by (x0 , y0 ), (x1 , y1 ), . . . , (xn−1 , yn−1 ).

Is the above a valid representation? Why do we use 2n numbers

instead of n numbers for coefficient and root representation?

Ruta (UIUC) CS473 15 Spring 2021 15 / 55

Sample Representation

Theorem
Given a list {(x0 , y0 ), (x1 , y1 ), . . . (xn−1 , yn−1 )} there is exactly
one polynomial p of degree n − 1 such that p(xj ) = yj for
j = 0, 1, . . . , n − 1.

Ruta (UIUC) CS473 16 Spring 2021 16 / 55

Sample Representation

Theorem
Given a list {(x0 , y0 ), (x1 , y1 ), . . . (xn−1 , yn−1 )} there is exactly
one polynomial p of degree n − 1 such that p(xj ) = yj for
j = 0, 1, . . . , n − 1.

So representation is valid.

Ruta (UIUC) CS473 16 Spring 2021 16 / 55

Sample Representation

Theorem
Given a list {(x0 , y0 ), (x1 , y1 ), . . . (xn−1 , yn−1 )} there is exactly
one polynomial p of degree n − 1 such that p(xj ) = yj for
j = 0, 1, . . . , n − 1.

So representation is valid.
Can use same x0 , x1 , . . . , xn−1 for all polynomials of degree n − 1.
No need to store them explicitly and hence need only n numbers
y0 , y1 , . . . , yn−1 .
(

Ruta (UIUC) CS473 16 Spring 2021 16 / 55

Lagrange Interpolation
Given (x0 , y0 ), . . . , (xn−1 , yn−1 ) the following polynomial p satisfies
the property that p(xj ) = yj for j = 0, 1, 2, . . . , n − 1.
 
n−1
X yj Y
p(x) = Q (x − xk )
j =0 k6=j (xj − xk ) k6=j

Ruta (UIUC) CS473 17 Spring 2021 17 / 55

Lagrange Interpolation
Given (x0 , y0 ), . . . , (xn−1 , yn−1 ) the following polynomial p satisfies
the property that p(xj ) = yj for j = 0, 1, 2, . . . , n − 1.
 
n−1
X yj Y
p(x) = Q (x − xk )
j =0 k6=j (xj − xk ) k6=j

For n = 3, p(x) =

(x − x1 )(x − x2 ) (x − x0 )(x − x2 ) (x − x0 )(x − x1 )

y0 +y1 +y2
(x0 − x1 )(x0 − x2 ) (x1 − x0 )(x1 − x2 ) (x2 − x0 )(x2 − x1 )

Ruta (UIUC) CS473 17 Spring 2021 17 / 55

Lagrange Interpolation
Given (x0 , y0 ), . . . , (xn−1 , yn−1 ) the following polynomial p satisfies
the property that p(xj ) = yj for j = 0, 1, 2, . . . , n − 1.
 
n−1
X yj Y
p(x) = Q (x − xk )
j =0 k6=j (xj − xk ) k6=j

For n = 3, p(x) =

(x − x1 )(x − x2 ) (x − x0 )(x − x2 ) (x − x0 )(x − x1 )

y0 +y1 +y2
(x0 − x1 )(x0 − x2 ) (x1 − x0 )(x1 − x2 ) (x2 − x0 )(x2 − x1 )

Easy to verify that p(xj ) = yj ! Thus there exists one polynomial of

degree n − 1 that interpolates the values (x0 , y0 ), . . . , (xn−1 , yn−1 ).

Ruta (UIUC) CS473 17 Spring 2021 17 / 55

Lagrange Interpolation

Given (x0 , y0 ), . . . , (xn−1 , yn−1 ) there is a polynomial p(x) such

that p(xi ) = yi for 0 ≤ i < n. Can there be two distinct
polynomials?

Ruta (UIUC) CS473 18 Spring 2021 18 / 55

Lagrange Interpolation

Given (x0 , y0 ), . . . , (xn−1 , yn−1 ) there is a polynomial p(x) such

that p(xi ) = yi for 0 ≤ i < n. Can there be two distinct
polynomials?

No! Use Fundamental Theorem of Algebra to prove it — exercise.

Ruta (UIUC) CS473 18 Spring 2021 18 / 55

Addition and Multiplication with Sample
Representation

Let a = {(x0 , y0 ), (x1 , y1 ), . . . (xn−1 , yn−1 )} and

b = {(x0 , y00 ), (x1 , y10 ), . . . (xn−1 , yn−1
0
)} be two polynomials
of degree n − 1 in sample representation.

Ruta (UIUC) CS473 19 Spring 2021 19 / 55

Addition and Multiplication with Sample
Representation

Let a = {(x0 , y0 ), (x1 , y1 ), . . . (xn−1 , yn−1 )} and

b = {(x0 , y00 ), (x1 , y10 ), . . . (xn−1 , yn−1
0
)} be two polynomials
of degree n − 1 in sample representation.
a + b can be represented by

Ruta (UIUC) CS473 19 Spring 2021 19 / 55

Addition and Multiplication with Sample
Representation

Let a = {(x0 , y0 ), (x1 , y1 ), . . . (xn−1 , yn−1 )} and

b = {(x0 , y00 ), (x1 , y10 ), . . . (xn−1 , yn−1
0
)} be two polynomials
of degree n − 1 in sample representation.
a + b can be represented by
{(x0 , (y0 + y00 )), (x1 , (y1 + y10 )), . . . (xn−1 , (yn−1 + yn−1
0
))}
Thus, can be computed in O(n) time

Ruta (UIUC) CS473 19 Spring 2021 19 / 55

Addition and Multiplication with Sample
Representation

Let a = {(x0 , y0 ), (x1 , y1 ), . . . (xn−1 , yn−1 )} and

b = {(x0 , y00 ), (x1 , y10 ), . . . (xn−1 , yn−1
0
)} be two polynomials
of degree n − 1 in sample representation.
a + b can be represented by
{(x0 , (y0 + y00 )), (x1 , (y1 + y10 )), . . . (xn−1 , (yn−1 + yn−1
0
))}
Thus, can be computed in O(n) time
a · b can be evaluated at n samples

Ruta (UIUC) CS473 19 Spring 2021 19 / 55

Addition and Multiplication with Sample
Representation

Let a = {(x0 , y0 ), (x1 , y1 ), . . . (xn−1 , yn−1 )} and

b = {(x0 , y00 ), (x1 , y10 ), . . . (xn−1 , yn−1
0
)} be two polynomials
of degree n − 1 in sample representation.
a + b can be represented by
{(x0 , (y0 + y00 )), (x1 , (y1 + y10 )), . . . (xn−1 , (yn−1 + yn−1
0
))}
Thus, can be computed in O(n) time
a · b can be evaluated at n samples
{(x0 , (y0 · y00 )), (x1 , (y1 · y10 )), . . . (xn−1 , (yn−1 · yn−1
0
))}
Can be computed in O(n) time.

Ruta (UIUC) CS473 19 Spring 2021 19 / 55

Addition and Multiplication with Sample
Representation

Let a = {(x0 , y0 ), (x1 , y1 ), . . . (xn−1 , yn−1 )} and

b = {(x0 , y00 ), (x1 , y10 ), . . . (xn−1 , yn−1
0
)} be two polynomials
of degree n − 1 in sample representation.
a + b can be represented by
{(x0 , (y0 + y00 )), (x1 , (y1 + y10 )), . . . (xn−1 , (yn−1 + yn−1
0
))}
Thus, can be computed in O(n) time
a · b can be evaluated at n samples
{(x0 , (y0 · y00 )), (x1 , (y1 · y10 )), . . . (xn−1 , (yn−1 · yn−1
0
))}
Can be computed in O(n) time.
But what if p, q are given in coefficient form? Convolution requires
p, q to be in coefficient form.

Ruta (UIUC) CS473 19 Spring 2021 19 / 55

Recall

Goal: given polynomials a = (a0 , . . . , an−1 ) and

b = (b0 , . . . , bn−1 ) in coefficient representation, compute a · b in
coefficient form (convolution).

Ruta (UIUC) CS473 20 Spring 2021 20 / 55

Recall

Goal: given polynomials a = (a0 , . . . , an−1 ) and

b = (b0 , . . . , bn−1 ) in coefficient representation, compute a · b in
coefficient form (convolution).
Sample representation:
Fix x0 , . . . , xn−1 .
a0 = (x0 , a(x0 )), . . . , (xn−1 , a(xn−1 )), similarly b 0 from b.
Theorem. Unique degree (n − 1) polynomial corresponding to
any given n samples.

Ruta (UIUC) CS473 20 Spring 2021 20 / 55

Recall

Goal: given polynomials a = (a0 , . . . , an−1 ) and

b = (b0 , . . . , bn−1 ) in coefficient representation, compute a · b in
coefficient form (convolution).
Sample representation:
Fix x0 , . . . , xn−1 .
a0 = (x0 , a(x0 )), . . . , (xn−1 , a(xn−1 )), similarly b 0 from b.
Theorem. Unique degree (n − 1) polynomial corresponding to
any given n samples. a 0 is a valid representation of a.
a0 · b 0 requires O(n) multiplications.

Ruta (UIUC) CS473 20 Spring 2021 20 / 55

Recall

Goal: given polynomials a = (a0 , . . . , an−1 ) and

b = (b0 , . . . , bn−1 ) in coefficient representation, compute a · b in
coefficient form (convolution).
Sample representation:
Fix x0 , . . . , xn−1 .
a0 = (x0 , a(x0 )), . . . , (xn−1 , a(xn−1 )), similarly b 0 from b.
Theorem. Unique degree (n − 1) polynomial corresponding to
any given n samples. a 0 is a valid representation of a.
a0 · b 0 requires O(n) multiplications.
Plan. Convert to sample representation. Multiply. Convert back to
coefficient representation.

Ruta (UIUC) CS473 20 Spring 2021 20 / 55

Coefficient representation to Sample representation

Given a polynomial a as (a0 , a1 , . . . , an−1 ) can we obtain a sample

representation (x0 , y0 ), . . . , (xn−1 , yn−1 ) quickly? Also can we
invert the representation quickly?

Ruta (UIUC) CS473 21 Spring 2021 21 / 55

Coefficient representation to Sample representation

Given a polynomial a as (a0 , a1 , . . . , an−1 ) can we obtain a sample

representation (x0 , y0 ), . . . , (xn−1 , yn−1 ) quickly? Also can we
invert the representation quickly?

Suppose we choose x0 , x1 , . . . , xn−1 arbitrarily.

Take O(n) time to evaluate yj = a(xj ) given (a0 , . . . , an−1 ).
Total time is Ω(n2 )
Inversion via Lagrange interpolation also Ω(n2 )

Ruta (UIUC) CS473 21 Spring 2021 21 / 55

Key Idea

Can choose x0 , x1 , . . . , xn−1 carefully!

Total time to evaluate a(x0 ), a(x1 ), . . . , a(xn−1 ) should be better

than evaluating each separately.

Ruta (UIUC) CS473 22 Spring 2021 22 / 55

Key Idea

Can choose x0 , x1 , . . . , xn−1 carefully!

Total time to evaluate a(x0 ), a(x1 ), . . . , a(xn−1 ) should be better

than evaluating each separately.

How do we choose x0 , x1 , . . . , xn−1 to save work?

Ruta (UIUC) CS473 22 Spring 2021 22 / 55

A Simple Start

a(x) = a0 + a1 x + a2 x 2 + a3 x 3 + . . . + an−1 x n−1

Assume n is a power of 2 for rest of the discussion.

Observation: (−x)2j = x 2j . Can we exploit this?

Ruta (UIUC) CS473 23 Spring 2021 23 / 55

A Simple Start

a(x) = a0 + a1 x + a2 x 2 + a3 x 3 + . . . + an−1 x n−1

Assume n is a power of 2 for rest of the discussion.

Observation: (−x)2j = x 2j . Can we exploit this?

Example
3+4x +6x 2 +2x 3 +x 4 +10x 5 = (3+6x 2 +x 4 )+x(4+2x 2 +10x 4 )

Ruta (UIUC) CS473 23 Spring 2021 23 / 55

A Simple Start

a(x) = a0 + a1 x + a2 x 2 + a3 x 3 + . . . + an−1 x n−1

Assume n is a power of 2 for rest of the discussion.

Observation: (−x)2j = x 2j . Can we exploit this?

Example
3+4x +6x 2 +2x 3 +x 4 +10x 5 = (3+6x 2 +x 4 )+x(4+2x 2 +10x 4 )

a(c) = (3 + 6c 2 + c 4 ) + c(4 + 2c 2 + 10c 4 )

Ruta (UIUC) CS473 23 Spring 2021 23 / 55

A Simple Start

a(x) = a0 + a1 x + a2 x 2 + a3 x 3 + . . . + an−1 x n−1

Assume n is a power of 2 for rest of the discussion.

Observation: (−x)2j = x 2j . Can we exploit this?

Example
3+4x +6x 2 +2x 3 +x 4 +10x 5 = (3+6x 2 +x 4 )+x(4+2x 2 +10x 4 )

a(c) = (3 + 6c 2 + c 4 ) + c(4 + 2c 2 + 10c 4 )

a(−c) =

Ruta (UIUC) CS473 23 Spring 2021 23 / 55

A Simple Start

a(x) = a0 + a1 x + a2 x 2 + a3 x 3 + . . . + an−1 x n−1

Assume n is a power of 2 for rest of the discussion.

Observation: (−x)2j = x 2j . Can we exploit this?

Example
3+4x +6x 2 +2x 3 +x 4 +10x 5 = (3+6x 2 +x 4 )+x(4+2x 2 +10x 4 )

a(c) = (3 + 6c 2 + c 4 ) + c(4 + 2c 2 + 10c 4 )

a(−c) = (3 + 6c 2 + c 4 ) − c(4 + 2c 2 + 10c 4 )

Ruta (UIUC) CS473 23 Spring 2021 23 / 55

Odd and Even Decomposition

Let a = (a0 , a1 , . . . an−1 ) be a polynomial.

Let aodd = (a1 , a3 , a5 , . . .) be the (n/2 − 1) degree
polynomial defined by the odd coefficients; so

aodd (x) = a1 + a3 x + a5 x 2 + · · ·

Ruta (UIUC) CS473 24 Spring 2021 24 / 55

Odd and Even Decomposition

Let a = (a0 , a1 , . . . an−1 ) be a polynomial.

Let aodd = (a1 , a3 , a5 , . . .) be the (n/2 − 1) degree
polynomial defined by the odd coefficients; so

aodd (x) = a1 + a3 x + a5 x 2 + · · ·

Similarly, let aeven (x) = a0 + a2 x + . . . be the (n/2 − 1)

degree polynomial defined by the even coefficients.

Ruta (UIUC) CS473 24 Spring 2021 24 / 55

Odd and Even Decomposition

Let a = (a0 , a1 , . . . an−1 ) be a polynomial.

Let aodd = (a1 , a3 , a5 , . . .) be the (n/2 − 1) degree
polynomial defined by the odd coefficients; so

aodd (x) = a1 + a3 x + a5 x 2 + · · ·

Similarly, let aeven (x) = a0 + a2 x + . . . be the (n/2 − 1)

degree polynomial defined by the even coefficients.
Observe
a(x) = aeven (x 2 ) + xaodd (x 2 )
Thus, evaluating a at x can be reduced to evaluating lower
degree polynomials plus constantly many arithmetic operations.

Ruta (UIUC) CS473 24 Spring 2021 24 / 55

Exploiting Odd-Even Decomposition

a(x) = aeven (x 2 ) + xaodd (x 2 )

Choose n samples
x0 , x1 , x2 , . . . , xn/2−1 , −x0 , −x1 , . . . , −xn/2−1
Evaluate aeven and aodd at x02 , x12 , x22 , . . . , xn/2−1
2
.

Ruta (UIUC) CS473 25 Spring 2021 25 / 55

Exploiting Odd-Even Decomposition

a(x) = aeven (x 2 ) + xaodd (x 2 )

Choose n samples
x0 , x1 , x2 , . . . , xn/2−1 , −x0 , −x1 , . . . , −xn/2−1
Evaluate aeven and aodd at x02 , x12 , x22 , . . . , xn/2−1
2
.
For each i = 0 to (n/2 − 1), evaluate
a(xi ) = aeven (xi2 ) + xi aodd (xi2 )
a(−xi ) = aeven (xi2 ) − xi aodd (xi2 )

Ruta (UIUC) CS473 25 Spring 2021 25 / 55

Exploiting Odd-Even Decomposition

a(x) = aeven (x 2 ) + xaodd (x 2 )

Choose n samples
x0 , x1 , x2 , . . . , xn/2−1 , −x0 , −x1 , . . . , −xn/2−1
Evaluate aeven and aodd at x02 , x12 , x22 , . . . , xn/2−1
2
.
For each i = 0 to (n/2 − 1), evaluate
a(xi ) = aeven (xi2 ) + xi aodd (xi2 )
a(−xi ) = aeven (xi2 ) − xi aodd (xi2 )
Total of O(n) work!

Ruta (UIUC) CS473 25 Spring 2021 25 / 55

Exploiting Odd-Even Decomposition

a(x) = aeven (x 2 ) + xaodd (x 2 )

Choose n samples
x0 , x1 , x2 , . . . , xn/2−1 , −x0 , −x1 , . . . , −xn/2−1
Evaluate aeven and aodd at x02 , x12 , x22 , . . . , xn/2−1
2
.
For each i = 0 to (n/2 − 1), evaluate
a(xi ) = aeven (xi2 ) + xi aodd (xi2 )
a(−xi ) = aeven (xi2 ) − xi aodd (xi2 )
Total of O(n) work!
Suppose we can make this work recursively. Then
T (n) = 2T (n/2) + O(n) which implies T (n) = O(n log n)

Ruta (UIUC) CS473 25 Spring 2021 25 / 55

Collapsible sets

Definition
Given a set X of numbers square(X ) (for square of X ) is the set
{x 2 | x ∈ X }.

Ruta (UIUC) CS473 26 Spring 2021 26 / 55

Collapsible sets

Definition
Given a set X of numbers square(X ) (for square of X ) is the set
{x 2 | x ∈ X }.

Definition
A set X of n numbers is collapsible if square(X ) ⊂ X and
|square(X )| = n/2.

Ruta (UIUC) CS473 26 Spring 2021 26 / 55

Collapsible sets

Definition
Given a set X of numbers square(X ) (for square of X ) is the set
{x 2 | x ∈ X }.

Definition
A set X of n numbers is collapsible if square(X ) ⊂ X and
|square(X )| = n/2.

Definition
A set X of n numbers (for n a power of 2) is recursively collapsible if
n = 1 or if X is collapsible and square(X ) is recursively collapsible.

Ruta (UIUC) CS473 26 Spring 2021 26 / 55

Divide and Conquer assuming collapsible set
Given a recursively collapsible set X of size n, compute sample
representation of polynomial a of degree (n − 1) as follows:
SampleRepresentation(a, X , n)
If n = 1 return a(x0 ) where X = {x0 }
Compute square(X ) in O(n) time %note:|square(X )| = n/2

Ruta (UIUC) CS473 27 Spring 2021 27 / 55

Divide and Conquer assuming collapsible set
Given a recursively collapsible set X of size n, compute sample
representation of polynomial a of degree (n − 1) as follows:
SampleRepresentation(a, X , n)
If n = 1 return a(x0 ) where X = {x0 }
Compute square(X ) in O(n) time %note:|square(X )| = n/2
{y0 , y1 , . . . , yn/2−1 } =SampleRepresentation(aodd , square(X ), n/2)
{y00 , y10 , . . . , yn/2−1
0
} =SampleRepresentation(aeven , square(X ), n/2)

Ruta (UIUC) CS473 27 Spring 2021 27 / 55

Divide and Conquer assuming collapsible set
Given a recursively collapsible set X of size n, compute sample
representation of polynomial a of degree (n − 1) as follows:
SampleRepresentation(a, X , n)
If n = 1 return a(x0 ) where X = {x0 }
Compute square(X ) in O(n) time %note:|square(X )| = n/2
{y0 , y1 , . . . , yn/2−1 } =SampleRepresentation(aodd , square(X ), n/2)
{y00 , y10 , . . . , yn/2−1
0
} =SampleRepresentation(aeven , square(X ), n/2)

For each i from 0 to (n − 1) compute

zi = aeven (xi2 ) + xi aodd (xi2 )

Return {z0 , z1 , . . . , zn−1 }

Ruta (UIUC) CS473 27 Spring 2021 27 / 55

Divide and Conquer assuming collapsible set
Given a recursively collapsible set X of size n, compute sample
representation of polynomial a of degree (n − 1) as follows:
SampleRepresentation(a, X , n)
If n = 1 return a(x0 ) where X = {x0 }
Compute square(X ) in O(n) time %note:|square(X )| = n/2
{y0 , y1 , . . . , yn/2−1 } =SampleRepresentation(aodd , square(X ), n/2)
{y00 , y10 , . . . , yn/2−1
0
} =SampleRepresentation(aeven , square(X ), n/2)

For each i from 0 to (n − 1) compute

zi = aeven (xi2 ) + xi aodd (xi2 )

Return {z0 , z1 , . . . , zn−1 }

Exercise: show that algorithm runs in O(n log n) time

Ruta (UIUC) CS473 27 Spring 2021 27 / 55

Are there collapsible sets?

n samples x0 , x1 , x2 , . . . , xn/2−1 , −x0 , −x1 , . . . , −xn/2−1

Next step in recursion x02 , x12 , . . . , xn/2−1
2

Ruta (UIUC) CS473 28 Spring 2021 28 / 55

Are there collapsible sets?

n samples x0 , x1 , x2 , . . . , xn/2−1 , −x0 , −x1 , . . . , −xn/2−1

Next step in recursion x02 , x12 , . . . , xn/2−1
2

To continue recursion, we need

{x02 , x12 , . . . , x 2n −1 } = {z0 , z1 , . . . , z n4 −1 , −z0 , −z1 , . . . , −z n4 −1 }

2

Ruta (UIUC) CS473 28 Spring 2021 28 / 55

Are there collapsible sets?

n samples x0 , x1 , x2 , . . . , xn/2−1 , −x0 , −x1 , . . . , −xn/2−1

Next step in recursion x02 , x12 , . . . , xn/2−1
2

To continue recursion, we need

{x02 , x12 , . . . , x 2n −1 } = {z0 , z1 , . . . , z n4 −1 , −z0 , −z1 , . . . , −z n4 −1 }

2

√
If z0 = x02 and −z0 = xn/4
2
then x0 = −1xn/4 That is
x0 = ixn/4 where i is the imaginary number.

Ruta (UIUC) CS473 28 Spring 2021 28 / 55

Are there collapsible sets?

n samples x0 , x1 , x2 , . . . , xn/2−1 , −x0 , −x1 , . . . , −xn/2−1

Next step in recursion x02 , x12 , . . . , xn/2−1
2

To continue recursion, we need

{x02 , x12 , . . . , x 2n −1 } = {z0 , z1 , . . . , z n4 −1 , −z0 , −z1 , . . . , −z n4 −1 }

2

√
If z0 = x02 and −z0 = xn/4
2
then x0 = −1xn/4 That is
x0 = ixn/4 where i is the imaginary number.
Can continue recursion but need to go to complex numbers.

Ruta (UIUC) CS473 28 Spring 2021 28 / 55

Complex Numbers

Notation
√
For the rest of lecture, i stands for −1

Definition
Complex numbers are points lying in the complex plane represented as
√
Cartesian a + ib = a2 + b 2 e (arctan(b/a))i
Polar re θi = r (cos θ + i sin θ)
Thus, e πi = −1 and e 2πi = 1.

Ruta (UIUC) CS473 29 Spring 2021 29 / 55

Power Series for Functions (Recall)

What is e z when z is a real number? When z is a complex number?

e z = 1 + z/1! + z 2 /2! + . . . + z j /j ! + . . .

Therefore

e i θ = 1 + i θ/1! + (i θ)2 /2! + (i θ)3 /3! + . . .

= (1 − θ 2 /2! + θ 4 /4! − . . . +) + i (θ − θ 3 /3! + . . . +)
= cos θ + i sin θ

Ruta (UIUC) CS473 30 Spring 2021 30 / 55

Complex Roots of Unity
What are the roots of the polynomial x k − 1? (e 2πi = 1)
Clearly 1 is a root.

Ruta (UIUC) CS473 31 Spring 2021 31 / 55

Complex Roots of Unity
What are the roots of the polynomial x k − 1? (e 2πi = 1)
Clearly 1 is a root.
Suppose re θi is a root then r k e kθi = 1 which implies that
r = 1 and kθ = 2π ⇒ θ = 2π/k

Ruta (UIUC) CS473 31 Spring 2021 31 / 55

Complex Roots of Unity
What are the roots of the polynomial x k − 1? (e 2πi = 1)
Clearly 1 is a root.
Suppose re θi is a root then r k e kθi = 1 which implies that
r = 1 and kθ = 2π ⇒ θ = 2π/k
Let ωk = e 2πi /k . The roots are 1 = ωk0 , ωk2 , . . . , ωkk−1 where
ωkj = e 2πji /k .

Ruta (UIUC) CS473 31 Spring 2021 31 / 55

Complex Roots of Unity
What are the roots of the polynomial x k − 1? (e 2πi = 1)
Clearly 1 is a root.
Suppose re θi is a root then r k e kθi = 1 which implies that
r = 1 and kθ = 2π ⇒ θ = 2π/k
Let ωk = e 2πi /k . The roots are 1 = ωk0 , ωk2 , . . . , ωkk−1 where
ωkj = e 2πji /k .

Proposition
Let ωk be e 2πi /k . The equation x k = 1 has k distinct complex roots
given by ωkj = e (2πj )i /k for j = 0, 1, . . . k − 1

Proof.
(ωkj )k = (e 2πji /k )k = e 2πji = (e 2πi )j = (1)j = 1
Ruta (UIUC) CS473 31 Spring 2021 31 / 55
Roots of unity form a collapsible set
Observation 1: ωkj = ωkj mod k

Ruta (UIUC) CS473 32 Spring 2021 32 / 55

Roots of unity form a collapsible set
Observation 1: ωkj = ωkj mod k

Lemma
Assume n is a power of 2. The n’th roots of unity are a recursively
collapsible set.

Proof.
Let Xn = {1, ωn , ωn2 , . . . , ωnn−1 }.

Ruta (UIUC) CS473 32 Spring 2021 32 / 55

Roots of unity form a collapsible set
Observation 1: ωkj = ωkj mod k

Lemma
Assume n is a power of 2. The n’th roots of unity are a recursively
collapsible set.

Proof.
Let Xn = {1, ωn , ωn2 , . . . , ωnn−1 }. (ωnn/2+j )2 = ωnn+2j = ωn2j ,

Ruta (UIUC) CS473 32 Spring 2021 32 / 55

Roots of unity form a collapsible set
Observation 1: ωkj = ωkj mod k

Lemma
Assume n is a power of 2. The n’th roots of unity are a recursively
collapsible set.

Proof.
Let Xn = {1, ωn , ωn2 , . . . , ωnn−1 }. (ωnn/2+j )2 = ωnn+2j = ωn2j , for
each j < n/2.

Ruta (UIUC) CS473 32 Spring 2021 32 / 55

Roots of unity form a collapsible set
Observation 1: ωkj = ωkj mod k

Lemma
Assume n is a power of 2. The n’th roots of unity are a recursively
collapsible set.

Proof.
Let Xn = {1, ωn , ωn2 , . . . , ωnn−1 }. (ωnn/2+j )2 = ωnn+2j = ωn2j , for
each j < n/2.

X1 = {1}, X2 = {1, −1}

X4 = {1, −1, i , −i }
X8 = {1, −1, i , −i , √12 (±1 ± i )}

Ruta (UIUC) CS473 32 Spring 2021 32 / 55

Discrete Fourier Transform

Definition
Given vector a = (a0 , a1 , . . . , an−1 ) the Discrete Fourier Transform
(DFT) of a is the vector a0 = (a00 , a10 , . . . , an−1
0
) where aj0 = a(ωnj )
for 0 ≤ j < n.

a0 is a sample representation of polynomial with coefficient

reprentation a at n’th roots of unity.

Ruta (UIUC) CS473 33 Spring 2021 33 / 55

Discrete Fourier Transform

Definition
Given vector a = (a0 , a1 , . . . , an−1 ) the Discrete Fourier Transform
(DFT) of a is the vector a0 = (a00 , a10 , . . . , an−1
0
) where aj0 = a(ωnj )
for 0 ≤ j < n.

a0 is a sample representation of polynomial with coefficient

reprentation a at n’th roots of unity.

We have shown that a0 can be computed from a in O(n log n) time.

This divide and conquer algorithm is called the Fast Fourier
Transform (FFT).

Ruta (UIUC) CS473 33 Spring 2021 33 / 55

Back to Convolutions and Polynomial
Multiplication

Convolutions (products)
Compute convolution c = (c0 , c1 , . . . , c2n−2 ) of
a = (a0 , a1 , . . . an−1 ) and b = (b0 , b1 , . . . bn−1 )
1 Evaluate a and b at some n sample points.
2 Compute sample representation of product. That is
c 0 = (a00 b00 , a10 b10 , . . . , an−1
0 0
bn−1 ).
3 Compute coefficients of unique polynomial associated with
sample representation of product. That is compute c from c 0 .

Ruta (UIUC) CS473 34 Spring 2021 34 / 55

Back to Convolutions and Polynomial
Multiplication

Convolutions (products)
Compute convolution c = (c0 , c1 , . . . , c2n−2 ) of
a = (a0 , a1 , . . . an−1 ) and b = (b0 , b1 , . . . bn−1 )
1 Evaluate a and b at the nth roots of unity.
2 Compute sample representation of product. That is
c 0 = (a00 b00 , a10 b10 , . . . , an−1
0 0
bn−1 ).
3 Compute coefficients of unique polynomial associated with
sample representation of product. That is compute c from c 0 .

Can we really compute c from c 0 ?

Ruta (UIUC) CS473 34 Spring 2021 34 / 55

Back to Convolutions and Polynomial
Multiplication

Convolutions (products)
Compute convolution c = (c0 , c1 , . . . , c2n−2 ) of
a = (a0 , a1 , . . . an−1 ) and b = (b0 , b1 , . . . bn−1 )
1 Evaluate a and b at the nth roots of unity.
2 Compute sample representation of product. That is
c 0 = (a00 b00 , a10 b10 , . . . , an−1
0 0
bn−1 ).
3 Compute coefficients of unique polynomial associated with
sample representation of product. That is compute c from c 0 .

Can we really compute c from c 0 ? We only have n sample points

and c 0 has 2n − 1 coefficients!

Ruta (UIUC) CS473 34 Spring 2021 34 / 55

Convolutions and Polynomial Multiplication
Convolutions
Compute convolution c = (c0 , c1 , . . . , c2n−2 ) of
a = (a0 , a1 , . . . an−1 ) and b = (b0 , b1 , . . . bn−1 )
1 Pad a with n zeroes to make it a (2n − 1) degree polynomial
a = (a0 , a1 , . . . , an−1 , an , an+1 , . . . , a2n−1 ). Similarly for b.

Ruta (UIUC) CS473 35 Spring 2021 35 / 55

Convolutions and Polynomial Multiplication
Convolutions
Compute convolution c = (c0 , c1 , . . . , c2n−2 ) of
a = (a0 , a1 , . . . an−1 ) and b = (b0 , b1 , . . . bn−1 )
1 Pad a with n zeroes to make it a (2n − 1) degree polynomial
a = (a0 , a1 , . . . , an−1 , an , an+1 , . . . , a2n−1 ). Similarly for b.
2 Compute values of a and b at the 2nth roots of unity.
3 Compute sample representation of product. That is
c 0 = (a00 b00 , a10 b10 , . . . , an−1
0 0
bn−1 0
, . . . , a2n−1 0
b2n−1 ).
4 Compute coefficients of unique polynomial associated with
sample representation of product. That is compute c from c 0 .

Ruta (UIUC) CS473 35 Spring 2021 35 / 55

Convolutions and Polynomial Multiplication
Convolutions
Compute convolution c = (c0 , c1 , . . . , c2n−2 ) of
a = (a0 , a1 , . . . an−1 ) and b = (b0 , b1 , . . . bn−1 )
1 Pad a with n zeroes to make it a (2n − 1) degree polynomial
a = (a0 , a1 , . . . , an−1 , an , an+1 , . . . , a2n−1 ). Similarly for b.
2 Compute values of a and b at the 2nth roots of unity.
3 Compute sample representation of product. That is
c 0 = (a00 b00 , a10 b10 , . . . , an−1
0 0
bn−1 0
, . . . , a2n−1 0
b2n−1 ).
4 Compute coefficients of unique polynomial associated with
sample representation of product. That is compute c from c 0 .

Step 2 takes O(n log n) using divide and conquer algorithm

Step 3 takes O(n) time
Step 4?
Ruta (UIUC) CS473 35 Spring 2021 35 / 55
 Part II

Inverse Fourier Transform

Ruta (UIUC) CS473 36 Spring 2021 36 / 55

Inverse Fourier Transform

Input Given the evaluation of a n − 1-degree polynomial a on

the nth roots of unity specified by vector a0
Goal Compute the coefficients of a
We saw that a0 can be computed from a in O(n log n) time. Can
we compute a from a0 in O(n log n) time?

Ruta (UIUC) CS473 37 Spring 2021 37 / 55

A Matrix Point of View
a(x) = a0 + a1 x + · · · + an−1 x n−1

a00 = a(x0 ), a10 = 0

a(x1 ), . . . , an−1 = a(xn−1 ).

x02 . . . x0n−1
 
1 x0 a0 a00
  
n−1 
 1 x
 1
2
x1 . . . x1  a1   a10 
 .. .
. .. .. ..   ..   ...
   
 . . . . .  .

 =  a0
   
n−1  
 2

  aj 
 1 xj xj . . . xj

  .j
 
.

 .. .. .. ... ..  
.   ..   ..

 . . . 
n−1 0
1 xn−1 2
xn−1 . . . xn−1 an−1 an−1

Ruta (UIUC) CS473 38 Spring 2021 38 / 55

A Matrix Point of View
a(x) = a0 + a1 x + · · · + an−1 x n−1

Denote ω = ωn1 = e 2π/n . Let xj = ω j

a00 = a(1), a10 = a(ω), . . . , an−1
0
= a(ω n−1 ).

a0 a00
    
1 1 1 ... 1
 1 ω ω2 ... ω n−1  a1   a10 
 .. .. .. .. ..  ..   .. 

. . . . .

.
  . 
 =  a0
    
1 ωj ω 2j ... ω j (n−1)   aj
  
  .j
   
 .. .. .. .. ..  .
  ..   ..

 . . . . . 
0
1 ω n−1 ω 2(n−1) ... ω (n−1)(n−1)
an−1 an−1

Ruta (UIUC) CS473 39 Spring 2021 39 / 55

Inverting the Matrix

−1 
a0 a00
   
1 1 1 ... 1
 a1   1 ω ω2 ω n−1
...   a10 
 ..   .. .. .. ..
..
  .. 

.
 
. . . ..   . 
=
     
j 2j j (n−1)  a0
 aj   1 ω ω ... ω
  
 .j
  
 .   . .. .. .. ..
 ..   ..  ..
 
. . . .  
n−1 2(n−1) (n−1)(n−1) 0
an−1 1 ω ω ... ω an−1

Ruta (UIUC) CS473 40 Spring 2021 40 / 55

Inverting the Matrix

 1 1 1 ... 1 −1  1 1 1 ... 1 

 1 ω ω2 ... ω n−1   1 ω −1 ω −2 ... ω −(n−1) 
   
 . . . . .   . . . . . 
 . . . . .   . . . . . 
 . . . . .  1 
 . . . . . 
=
  
 1 ωj ω 2j ... ω j (n−1) −j −2j −j (n−1)
1 ω ω ... ω
   


 n 



 . . . . .   . . . . . 
 . . . . .   . . . . . 
 . . . . .   . . . . . 
−(n−1) −2(n−1)
1 ω n−1
ω 2(n−1)
... ω (n−1)(n−1)
1 ω ω ... ω −(n−1)(n−1)

Replace ω by ω −1 which is also a root of unity!

Since ω j = ω j mod n , we get ω −j = e −j 2π/n = ω (n−j )2π/n .

Ruta (UIUC) CS473 41 Spring 2021 41 / 55

Inverting the Matrix

 1 1 1 ... 1 −1  1 1 1 ... 1 

 1 ω ω2 ... ω n−1   1 ω −1 ω −2 ... ω −(n−1) 
   
 . . . . .   . . . . . 
 . . . . .   . . . . . 
 . . . . .  1 
 . . . . . 
=
  
 1 ωj ω 2j ... ω j (n−1) −j −2j −j (n−1)
1 ω ω ... ω
   


 n 



 . . . . .   . . . . . 
 . . . . .   . . . . . 
 . . . . .   . . . . . 
−(n−1) −2(n−1)
1 ω n−1
ω 2(n−1)
... ω (n−1)(n−1)
1 ω ω ... ω −(n−1)(n−1)

Replace ω by ω −1 which is also a root of unity!

Since ω j = ω j mod n , we get ω −j = e −j 2π/n = ω (n−j )2π/n .
Inverse matrix is simply a permutation of the original matrix modulo
scale factor 1/n.

Ruta (UIUC) CS473 41 Spring 2021 41 / 55

Why does it work?
Check VV −1 = I where I is the n × n identity matrix.

Ruta (UIUC) CS473 42 Spring 2021 42 / 55

Why does it work?
Check VV −1 = I where I is the n × n identity matrix.
Pn−1 j )s
Observation: s=0 (ω = (1 + ω j + ω 2j + . . . + ω (n−1)j ) = 0, j 6= 0

Ruta (UIUC) CS473 42 Spring 2021 42 / 55

Why does it work?
Check VV −1 = I where I is the n × n identity matrix.
n−1
(ω j )s = (1 + ω j + ω 2j + . . . + ω (n−1)j ) = 0, j 6= 0
P
Observation: s=0
ω j is root of x n − 1 = (x − 1)(x n−1 + x n−2 + . . . + 1)
Thus, ω j is root of (x n−1 + x n−2 + . . . + 1)

Ruta (UIUC) CS473 42 Spring 2021 42 / 55

 Why does it work?
Check VV −1 = I where I is the n × n identity matrix.
n−1
(ω j )s = (1 + ω j + ω 2j + . . . + ω (n−1)j ) = 0, j 6= 0
P
Observation: s=0
ω j is root of x n − 1 = (x − 1)(x n−1 + x n−2 + . . . + 1)
Thus, ω j is root of (x n−1 + x n−2 + . . . + 1)
n−1
X
−k −2k −k(n−1)
j 2j
(1, ω , ω , . . . , ω j (n−1)
)·(1, ω ,ω ,...,ω )= ω s(j −k)
s=0

Note that ω j −k is a n’th root of unity. If j = k then sum is n,

otherwise by previous observation sum is 0.

Ruta (UIUC) CS473 42 Spring 2021 42 / 55

 Why does it work?
Check VV −1 = I where I is the n × n identity matrix.
n−1
(ω j )s = (1 + ω j + ω 2j + . . . + ω (n−1)j ) = 0, j 6= 0
P
Observation: s=0
ω j is root of x n − 1 = (x − 1)(x n−1 + x n−2 + . . . + 1)
Thus, ω j is root of (x n−1 + x n−2 + . . . + 1)
n−1
X
−k −2k −k(n−1)
j 2j
(1, ω , ω , . . . , ω j (n−1)
)·(1, ω ,ω ,...,ω )= ω s(j −k)
s=0

Note that ω j −k is a n’th root of unity. If j = k then sum is n,

otherwise by previous observation sum is 0.

Rows of matrix V (and hence also those of V −1 ) are orthogonal.

Thus a0 = Va can be thought of transforming the vector a into a
new Fourier basis with basis vectors corresponding to rows of V .
Ruta (UIUC) CS473 42 Spring 2021 42 / 55
Inverse Fourier Transform

Input Given the evaluation of a n − 1-degree polynomial a on

the nth roots of unity specified by vector a0
Goal Compute the coefficients of a
We saw that a0 can be computed from a in O(n log n) time. Can
we compute a from a0 in O(n log n) time?

Yes! a = V −1 a0 which is simply a permuted and scaled version of

DFT. Hence can be computed in O(n log n) time.

Ruta (UIUC) CS473 43 Spring 2021 43 / 55

Convolutions Once More

Convolutions
Compute convolution of a = (a0 , a1 , . . . an−1 ) and
b = (b0 , b1 , . . . bn−1 )
1 Compute values of a and b at the 2nth roots of unity
2 Compute sample representation c 0 of product c = a · b
3 Compute c from c 0 using inverse Fourier transform.

Step 1 takes O(n log n) using two FFTs

Step 2 takes O(n) time
Step 3 takes O(n log n) using one FFT

Ruta (UIUC) CS473 44 Spring 2021 44 / 55

 FFT Circuit

ithms Lecture 3: Fast Fourier Transforms

000 000 000

001 010 100

U FFT(n/2) U* 010 100 010

011 110 110

P P* 100 001 001

101 011 101

110 101 011

V FFT(n/2) V*
111 111 111

bit reversal permutation butterfly network

The recursive structure of the FFT algorithm.

we expand this recursive structure completely, we see that the circuit splits naturally into
arts. The left half computes the bit-reversal permutation of the input. To find the position of
n this permutation, write k in binary, and then read the bits backward. For example, in an
ment bit-reversal permutation, P [3] = P [0112 ] ends up in position 6 = 1102 . The right half of
butterfly
T circuit is aRuta (UIUC) network. Butterfly CS473
networks are45often used to route between processors
Spring 2021 45 / 55
Numerical Issues

As noted earlier evaluating a polynomial p at a point x makes

numbers big
Are we cheating when we say O(n log n) algorithm for
convolution?
Can get around numerical issues — work in finite fields and
avoid numbers growing too big.
Outside the scope of lecture
We will assume for reductions that convolution can be done in
O(n log n) time.

Ruta (UIUC) CS473 46 Spring 2021 46 / 55

Numerical Issues: Puzzle

Ruta (UIUC) CS473 47 Spring 2021 47 / 55

 Part III

Application to String Matching

Ruta (UIUC) CS473 48 Spring 2021 48 / 55

Basic string matching problem:
Input Given a pattern string P on length m and a text string
T of length n over a fixed alphabet Σ
Goal Does P occur as a substring of T ? Find all “matches”
of P in T .

Ruta (UIUC) CS473 49 Spring 2021 49 / 55

Basic string matching problem:
Input Given a pattern string P on length m and a text string
T of length n over a fixed alphabet Σ
Goal Does P occur as a substring of T ? Find all “matches”
of P in T .
Several generalizations. Matching with don’t cares.
Input Given a pattern string P on length m over Σ ∪ {∗} (∗
is a don’t care) and a text string T of length n over Σ
Goal Find all “matches” of P in T . ∗ matches with any
character of Σ

Ruta (UIUC) CS473 49 Spring 2021 49 / 55

Basic string matching problem:
Input Given a pattern string P on length m and a text string
T of length n over a fixed alphabet Σ
Goal Does P occur as a substring of T ? Find all “matches”
of P in T .
Several generalizations. Matching with don’t cares.
Input Given a pattern string P on length m over Σ ∪ {∗} (∗
is a don’t care) and a text string T of length n over Σ
Goal Find all “matches” of P in T . ∗ matches with any
character of Σ
Example: P = a ∗ ∗, T = aardvark
Matches?

Ruta (UIUC) CS473 49 Spring 2021 49 / 55

Shifted products via Convolution
Given two arrays A and B with say with A[0..m − 1] and
B[0..n − 1] with m ≤ n
Input Two arrays: A[0..(m − 1)] and B[0..(n − 1)].
Goal Compute all shifted products in array
C [0..(n − m − 1)] where C [i ] = m−1
P
j =0 A[j ]B[i + j ].

Ruta (UIUC) CS473 50 Spring 2021 50 / 55

Shifted products via Convolution
Given two arrays A and B with say with A[0..m − 1] and
B[0..n − 1] with m ≤ n
Input Two arrays: A[0..(m − 1)] and B[0..(n − 1)].
Goal Compute all shifted products in array
C [0..(n − m − 1)] where C [i ] = m−1
P
j =0 A[j ]B[i + j ].

Example: A = [0, 1, 1, 0], B = [0, 0, 1, 1, 1, 0, 1]

C =

Ruta (UIUC) CS473 50 Spring 2021 50 / 55

Shifted products via Convolution
Given two arrays A and B with say with A[0..m − 1] and
B[0..n − 1] with m ≤ n
Input Two arrays: A[0..(m − 1)] and B[0..(n − 1)].
Goal Compute all shifted products in array
C [0..(n − m − 1)] where C [i ] = m−1
P
j =0 A[j ]B[i + j ].

Example: A = [0, 1, 1, 0], B = [0, 0, 1, 1, 1, 0, 1]

C =
Lemma
Reverse of C is the convolution of the vectors A and reverse of B.

Proof.
Exercise.

Ruta (UIUC) CS473 50 Spring 2021 50 / 55

Reduction of pattern matching to shifted products
Assume first that Σ = {0, 1}
Goal:
Convert P = a0 a1 . . . am−1 to binary array A of size m.
Convert T = b0 b1 . . . bn−1 to binary array B of size n.
So that we can use shifted product C of A and B to count
“mismatches”.

Ruta (UIUC) CS473 51 Spring 2021 51 / 55

Reduction of pattern matching to shifted products
Assume first that Σ = {0, 1}
Goal:
Convert P = a0 a1 . . . am−1 to binary array A of size m.
Convert T = b0 b1 . . . bn−1 to binary array B of size n.
So that we can use shifted product C of A and B to count
“mismatches”.
Type 1 mismatches: C [i ] counts # j ’s where P[j ] = 0 and
T [i + j ] = 1, when P is aligned with T at T [i ].

Ruta (UIUC) CS473 51 Spring 2021 51 / 55

Reduction of pattern matching to shifted products
Assume first that Σ = {0, 1}
Goal:
Convert P = a0 a1 . . . am−1 to binary array A of size m.
Convert T = b0 b1 . . . bn−1 to binary array B of size n.
So that we can use shifted product C of A and B to count
“mismatches”.
Type 1 mismatches: C [i ] counts # j ’s where P[j ] = 0 and
T [i + j ] = 1, when P is aligned with T at T [i ].
T = 10110010 . . .
Example:
P = 010

Ruta (UIUC) CS473 51 Spring 2021 51 / 55

Reduction of pattern matching to shifted products
Assume first that Σ = {0, 1}
Goal:
Convert P = a0 a1 . . . am−1 to binary array A of size m.
Convert T = b0 b1 . . . bn−1 to binary array B of size n.
So that we can use shifted product C of A and B to count
“mismatches”.
Type 1 mismatches: C [i ] counts # j ’s where P[j ] = 0 and
T [i + j ] = 1, when P is aligned with T at T [i ].
T = 10110010 . . .
Example:
P = 010
Finding Type 1 mismatches:
B[j ] = T [j ]
If P[j ] = 0 set A[j ] = 1, if P[j ] = 1 or ∗ set A[j ] = 0.
Ruta (UIUC) CS473 51 Spring 2021 51 / 55
Reduction of pattern matching to shifted products

Type 2 mismatches: C [i ] counts # j ’s where P[j ] = 1 and

T [i + j ] = 0, when P is aligned with T at T [i ].

Ruta (UIUC) CS473 52 Spring 2021 52 / 55

Reduction of pattern matching to shifted products

Type 2 mismatches: C [i ] counts # j ’s where P[j ] = 1 and

T [i + j ] = 0, when P is aligned with T at T [i ].

T = 10110010 . . .
Example:
P = 010

Ruta (UIUC) CS473 52 Spring 2021 52 / 55

Reduction of pattern matching to shifted products

Type 2 mismatches: C [i ] counts # j ’s where P[j ] = 1 and

T [i + j ] = 0, when P is aligned with T at T [i ].

T = 10110010 . . .
Example:
P = 010

Finding Type 2 mismatches:

B[j ] = (1 − T [j ]) (flip the bits)
If P[j ] = 0 or ∗ set A[j ] = 0, if P[j ] = 1 set A[j ] = 1.

Ruta (UIUC) CS473 52 Spring 2021 52 / 55

Reduction of pattern matching to shifted products

Type 2 mismatches: C [i ] counts # j ’s where P[j ] = 1 and

T [i + j ] = 0, when P is aligned with T at T [i ].

T = 10110010 . . .
Example:
P = 010

Finding Type 2 mismatches:

B[j ] = (1 − T [j ]) (flip the bits)
If P[j ] = 0 or ∗ set A[j ] = 0, if P[j ] = 1 set A[j ] = 1.

There is a match at position i of T iff both types of mismatches are

0.

Ruta (UIUC) CS473 52 Spring 2021 52 / 55

Running time analysis

Reducing to shift product is O(n).

Need to compute two convolutions with polynomials of size n
and m. Total run time is O(n log n) (here we assume m ≤ n).

Ruta (UIUC) CS473 53 Spring 2021 53 / 55

Running time analysis

Reducing to shift product is O(n).

Need to compute two convolutions with polynomials of size n
and m. Total run time is O(n log n) (here we assume m ≤ n).
Can reduce to O(n log m) as follows. Break text T into
O(n/m) overlapping substrings of length 2m each and compute
matches of P with these substrings. Total time is O(n log m).

Exercise: work out the details of this improvement.

Ruta (UIUC) CS473 53 Spring 2021 53 / 55

General Alphabet
If Σ is not binary replace each character α ∈ Σ by its binary
representation. Need s = dlog |Σ|e bits. Running time increases to
O(n log m log s).

Can remove dependence on s and obtain O(n log m) time where

m = |P| using more advanced ideas and/or randomization.

Ruta (UIUC) CS473 54 Spring 2021 54 / 55

Trivia
FFT algorithm is used billions of times everyday: image/sound
processing – jpeg, mp3, MRI scans, etc.

Even your brain is running FFT!

A fun video on FFT applications:

https://www.youtube.com/watch?v=aqa6vyGSdos

Ruta (UIUC) CS473 55 Spring 2021 55 / 55