Other Optimization Techniques

Conjugate Gradient
Similar to steepest descent, but slightly different way of choosing
direction of next step:




x r +1 = x r +  r sr



s0 =
g0




sr +1 =
gr +1 +  r +1sr

{
new term




 r is chosen to minimize h( x r +1 ). This yields grt +1gr = 0


Here we allow a further step in the sr direction. One choice
(Fletcher - Reeves) for
r +1 is
gr2+1

r +1 = 2
gr
Winter Semester 2006/7 Computational Physics I Lecture 12 1
 Newton-Raphson

Assume the function that we want to minimize is twice

differentiable. Then, a Taylor expansion gives



t
1
t

h( x +
)  a + b
+
C

2
where






2
h 
a = h( x ), b = h( x ) = g( x ), C =   =H

x i
x j 






Now h( x +
)  b + C
Because C is symmetric (check)

1


For an extremum, we have b + C = 0  =
C b

or x r +1 = x r
H( x r )
1 g( x r )

Winter Semester 2006/7 Computational Physics I Lecture 12 2

 Newton-Raphson

x r +1 = x r
H( x r )
1 g( x r )



1


i.e., the search direction is s = H( x r ) g( x r ) and  = 1
This converges quickly (if you start with a good guess), but the
penalty is that the Hessian needs to be calculated (usually
numerically)


Again, convergence is when s is sufficiently small

How would we calculate the Hessian numerically ? Use Lagrange

polynomial in several dimensions and work it out

Winter Semester 2006/7 Computational Physics I Lecture 12 3

 Bounded Regions

The standard tool for minimization in particle physics is the

MINUIT program (CERN library). It has also made its way well
outside the particle physics community.

Author: Fred James

Here is how MINUIT handles bounded search regions - it

transforms the parameter to be optimized as follows:
 
a  b
a
  = arcsin 2
1
 b
a 
 = a+ (sin   + 1)
2
 is the exernal (user) parameter
  is the internal parameter
MINUIT is available within PAW, ROOT …

Winter Semester 2006/7 Computational Physics I Lecture 12 4

 MINUIT

MINUIT uses a (variable metric) conjugate gradient search

algorithm (along with others). Basic idea:

• assume that the function to minimize can be approximated by a

quadratic form near the minimum
• build up iteratively an approximation for the inverse of the
Hessian matrix. Recall






1
t

h( x +
)  h( x ) + h( x ) 
+
H

2
the approximation for the Hessian is updated as follows:





( x i+1
x i )  ( x i+1
x i ) [ H i  (hi+1
hi )]  [ H i  (hi+1
hi )]





H i+1 = H i +









( x i+1
x i )  (hi+1
hi ) (hi+1
hi )  H i  (hi+1
hi )
where the  symbol represents an outer product of two vectors (a matrix)



(a  b )ij = aib j
Winter Semester 2006/7 Computational Physics I Lecture 12 5
 Fourier Transforms

Fourier transforms are very important

• as a way of summarizing the data with a few parameters
• because the transform of the data is itself very interesting (e.g.,
power spectrum, momentumcoordinate space
representation,…)

H( f ) =  h(t) e 2ift dt H( f ) frequency domain representation



h(t) =  H( f ) e
2ift dt h(t) time domain representation



Warning: there is no unanimity on 2
factors in front of the

integral. Often the angular frequency is used  = 2
f
 1 
it
it
H( ) =  h(t) e dt h(t) =  H( ) e d

 2

Winter Semester 2006/7 Computational Physics I Lecture 12 6
 Fourier Transform

Fourier Transform is a linear operation:

• transform of the sum of two functions is the sum of the
transforms
• the transform of a constant times a function is constant times the
transform
H(
f ) = [ H( f )]
*
h(t) real
H(
f ) =
[ H( f )]
*
h(t) imaginary
h(t) even H(
f ) = H( f )
h(t) odd H(
f ) =
H( f )
h(t) real,even H( f ) real, even
h(t) real,odd H( f ) imaginary, odd
h(t) imaginary,even H( f ) imaginary, even
h(t) imaginary,odd H( f ) real, odd
Winter Semester 2006/7 Computational Physics I Lecture 12 7
 Fourier Transform

Further properties: 1  f
h(at)  H
a  a
1  t
h  H (bf )
b  b
h(t
t 0 )  H ( f )e 2ift 0

h(t)e
2if t  H( f
f 0 )
0

We are typically interested in the Fourier analysis of a discretely

sampled data set. Define the time step (taken to be constant
here) as
. The sampling rate (frequency) is 1/
. Define the
samples as
hn = h(n) n =
,
3,
2,
1,0,1,2,3,

Winter Semester 2006/7 Computational Physics I Lecture 12 8

 Nyquist frequency

1
fc  Nyquist frequency
2

This is the highest frequency which can be resolved with a

sampling frequency f=1/
. If a continuous function h(t) is limited
in frequency components to frequencies less than fc, then h(t) is
completely determined by its samples hn. It can then be written
as follows:
 sin[2f c (t
n)]
h(t) =   hn
n =
  (t
n)

However, if there are frequency components which are higher

than fc, then they will be spuriously moved in the range f<fc
(aliasing).
Winter Semester 2006/7 Computational Physics I Lecture 12 9
 Example
Data Fit Aliasing

10 Hz sampling 5 Hz sampling
Conditions are:
Sine wave with f=4 Hz, phase offset
=0.1 i.e.,
h(t) = sin(
+ 2ft) = sin(0.1 + 8t)

Winter Semester 2006/7 Computational Physics I Lecture 12 10

 Example

H( f ) = 
 sin(0.1 + 8t)e 2ift dt
 
= 
 sin(0.1)cos(8t)e 2ift dt + 
 cos(0.1)sin(8t)e 2ift dt
8it
8it 8it
8it
 e + e  e
e
= sin(0.1) 
 e 2ift dt + cos(0.1) 
 e 2ift dt
2 2
Recall the relation:
 2ifx

 e df =  (x) where  (x) is the Dirac Delta function

so we have
8it
 e + e
8it 2ift  e
8it

e
8it 2ift
H( f ) = sin(0.1) 
 e dt + cos(0.1) 
 e dt
2 2
sin(0.1)  2it( f + 4 ) 2it( f
4 ) cos(0.1)  2it( f + 4 ) 2it( f
4 )
= 
 e +e dt + 
 e
e dt
2 2
sin(0.1) cos(0.1)
= [ ( f + 4) +  ( f
4)] + [ ( f + 4)
 ( f
4)]
2 2
Winter Semester 2006/7 Computational Physics I Lecture 12 11
 Discrete Fourier Transform

Suppose we have N consecutive sampled points

hk  h(t k ), t k  k, k = 0,1,2,
,N
1
We can extract the amplitude for N frequency components since
we have N data points. Define the frequency components as
n  1 N N
fn  n =
,..., (take N even)
N   2 2
Note: there are N+1 frequencies, but we will find that the two at
the ends are equal, so only N independent. Negative frequencies
allows us to include sine and cosine terms. So
N
1 N
1 N
1
 2ifn t 2ifn tk 2ifn k
H( f n ) = 
 h(t)e dt   hk e  =   hk e =   hk e 2ikn / N
k=0 k=0 k=0
N
1
H n   hk e 2ikn / N Discrete Fourier Transform
k=0

Winter Semester 2006/7 Computational Physics I Lecture 12 12

 Discrete Fourier Transform
The discrete fourier transform does not depend on any
dimensional parameters.

Note H
n = HN
n [
e 2ik( N
n ) / N = e 2ik e
2ikn / N = e
2ikn / N ]
In particular H
N / 2 = HN / 2

We can therefore rewrite the sum as follows

N
1
H n   hk e 2ikn / N n = 0,
,N
1
k=0

Discrete inverse Fourier transform

1 N
1
hk =  H n e
2ikn / N k = 0,
,N
1
N n=0
Winter Semester 2006/7 Computational Physics I Lecture 12 13
 Discrete Fourier Transform

*
* Get the discrete Fourier components
*
Do n=0,63
Hn(n,1)=0.D0
Hn(n,2)=0.D0
Do k=0,63
Hn(n,1)=Hn(n,1)
& +amplitude(k,1)*dcos(twopi*k*n/64.)
& -amplitude(k,2)*dsin(twopi*k*n/64.) amplitude(k,1) real components
Hn(n,2)=Hn(n,2) amplitude(k,2) imaginary components
& +amplitude(k,1)*dsin(twopi*k*n/64.)
& +amplitude(k,2)*dcos(twopi*k*n/64.)
Enddo
Write (11,*) N,Hn(N,1),Hn(N,2)
Enddo

Winter Semester 2006/7 Computational Physics I Lecture 12 14

 Discrete Fourier Transform

Let’s try it out on our sine wave data:

Recall, signal f=4 Hz
Here’s the result (64 points):

Large components are:

25 26
f 25 = = 3.9, f 26 = = 4.06
64  0.1 64  0.1
f 38 = f 64
26 = f
26 f 39 = f 64
25 = f
25

cos and sin needed because of

phase offset.

Winter Semester 2006/7 Computational Physics I Lecture 12 15

 Discrete Fourier Transform

Here’s the inverse transform

*
* Now we try the inverse transform
*
Do n=0,63
amplitude(n,1)=0.D0
amplitude(n,2)=0.D0
Do k=0,63
amplitude(n,1)=amplitude(n,1)
& +Hn(k,1)*dcos(twopi*k*n/64.)
& +Hn(k,2)*dsin(twopi*k*n/64.)
amplitude(n,2)=amplitude(n,2)
& -Hn(k,1)*dsin(twopi*k*n/64.)
& +Hn(k,2)*dcos(twopi*k*n/64.)
Enddo
Write (12,*) N,amplitude(n,1)/64.,amplitude(n,2)/64.
Enddo
*

Winter Semester 2006/7 Computational Physics I Lecture 12 16

 Fast Fourier Transform
The discrete Fourier transforms as we described it requires a sum
over N terms for each of the N components. I.e., the number of
operations scales as N2. A large part of the success of Fourier
transforms for analysis of electronic signals, optical images, x-ray
tomography,…, results from the fact that a numerical algorithm
was found which requires of order Nlog2N operations - the so-
called Fast Fourier Transform (FFT). Here is how it works:
N
1 N / 2
1 N / 2
1
2ijk / N 2ik(2 j ) / N
Fk =  e fj =  e f 2 j +  e 2ik(2 j +1) / N f 2 j +1
j=0 j=0 j=0
N / 2
1 N / 2
1
2ikj /(N / 2) 2ikj /(N / 2)
=  e f2 j + W k
 e f 2 j +1
j=0 j=0

= Fke + W k Fko where W  e 2i / N

What is won ? The sums in the individual terms in the last line
only have 1/2 as many terms, and the same factor appears
Winter Semester 2006/7 Computational Physics I Lecture 12 17
 Fast Fourier Transform
To see how this works in detail, we take an explicit example of
having 8 data points (taking a power of 2 is important ! If you don’t
have enough data, pad with zeroes).
Fk = Fke + W k Fko
3 3
2
i / N
where W  e , Fke  W 2kj
f 2 j , Fko   W 2kj f 2 j +1
j=0 j=0

Now use a binary representation for the index k=4k2+2k1+k0

where the ki’s are 0,1. Then,

W 2kj
= e ( )
2
i / 8 2(4 k2 + 2k1 + k0 ) j
= e 2
i(k 2
+ k1 / 2+ k0 / 4 )
= e 2
i(k / 2+ k
1 0
/4)
= W 2 j(2k + k
1 0
)

i.e., the k2 bit is irrelevant. So,

Fk = F(ke ,k ) + W k F(ok ,k )
1 0 1 0

3 3
2 j(2k1 + k0 )
F(ke 1 ,k0 )
W f2 j F(ko1 ,k0 )
 W 2 j( 2k + k ) f 2 j +1
1 0

j=0 j=0
Winter Semester 2006/7 Computational Physics I Lecture 12 18
 Fast Fourier Transform

Let’s try again:

3 1 1
2 j(2k1 + k0 ) 2(2 j )(2k1 + k0 )
Fke =
W f2 j =
W f 2(2 j ) +
W 2( 2 j +1)(2k + k ) f 2(2 j +1)
1 0

j=0 j=0 j=0

1 1
2(2 j )(2k1 + k0 ) 2( 2k1 + k0 ) 2(2 j )(2k + k )
=
W f 2( 2 j ) +W
W 1
f 2(2 j +1)
0

j=0 j=0

Fke = Fkee + W 2(2k + k )Fkeo 1 0

W 4 j(2k1 + k0 )
= e ( )
2
i / 8 (8k1 + 4 k0 ) j
= W 2 jk 0

and Fko = Fkoe + W 2(2k + k )Fkoo 1 0

Can perform one more step:

Fkee = F eee + W 4 k F eeo 0
F eee = f 0 F eeo = f 4
The sums have disappeared !
Winter Semester 2006/7 Computational Physics I Lecture 12 19
 Fast Fourier Transform

The final pieces are:

Fkee = F eee + W 4 k F eeo = f 0 + W 4 k f 4
0 0 Note k0=0,1
So, need 8 multiplications
Fkeo = F eoe + W 4 k F eoo = f 2 + W 4 k f 6
0 0

and 8 additions for this

Fkoe = F oee + W 4 k F oeo = f 1 + W 4 k f 5
0 0 step

Fkoo = F ooe + W 4 k F ooo = f 3 + W 4 k f 7

0 0

Then, Fke = F ee + W 2(2k + k )F eo

1 0 Here k0=0,1 k1=0,1
So, again 8 multiplications
Fko = F oe + W 2(2k + k )F oo
1 0

and 8 additions for this

step

Finally Fk = F e + W k F o again 8 multiplications and

8 additions for this step

Winter Semester 2006/7 Computational Physics I Lecture 12 20

 Fast Fourier Transform

So we need 2N operations per level, and there are log2N levels.

The scaling of the computational time is therefore Nlog2N rather
than N2.

E.g., N=1000 Nlog2N
1000*10=104 N2=106

How to implement in practice. Note that the trick is to find out

which value of n corresponds to which pattern of e,o in

F eoeooeoe... = f n
Answer: reverse pattern of e,o. Assign e=0, o=1, and the binary
value gives n.
eee
eee
000
0
examples
oeo
eoe
010
2
Winter Semester 2006/7 Computational Physics I Lecture 12 21
 Some examples

. Agric. Food Chem., 52 (20), 6055 -6060, 2004. 10.1021/jf049240e S0021-8561(04)09240-4

Web Release Date: September 9, 2004
Copyright © 2004 American Chemical Society

Discrimination of Olives According to Fruit Quality Using Fourier Transform Raman Spectroscopy and Pattern
Recognition Techniques

Barbara Muik, Bernhard Lendl, Antonio Molina-Díaz, Domingo Ortega-Calderón,# and María José Ayora-Cañada*

Department of Physical and Analytical Chemistry, University of Jaén, Paraje las Lagunillas s/n, E-23071 Jaén,
Spain; Institute of Chemical Technologies and Analytics, Vienna University of Technology, Getreidemarkt 9/164,
A-1060 Wien, Austria; and CIFA Venta del Llano, IFAPA, Ctra. Bailén-Motril km 18.5, E-23620 Mengíbar, Jaén,
Spain

Multiplication of large integers

The fastest known algorithms for the multiplication of large integers or polynomials are based on the discrete
Fourier transform: the sequences of digits or coefficients are interpreted as vectors whose convolution needs to
be computed; in order to do this, they are first Fourier-transformed, then multiplied component-wise, then
transformed back.

Winter Semester 2006/7 Computational Physics I Lecture 12 22

 Power Spectrum

The autocorrelation of a function is


Corr[y]( ) = 
 y(t)* y(t +  ) dt
and the power spectrum is defined as the Fourier transform of
the autocorrelation

PS[y]( f ) = 
 Corr[y]( )e 2if d

For a periodic function, the correlation is often defined as the

expectation value. There is no convention on the normalization,
so be careful about the values. Best to see which frequencies
dominate a given spectrum. Here is a practical approach for a
discretely sampled function:
N
1
Ck =  c j e 2ijk / N k = 0,1,…,N
1
j=0

Winter Semester 2006/7 Computational Physics I Lecture 12 23

 Power Spectrum
1 2 where only positive
P(0) = P( f 0 ) = 2 C0
N frequencies are
1
[ 2
P( f k ) = 2 Ck + CN
k
N
]2
k = 1,2,…,
N 


1
2 
considered:
k k N
1
P( f c ) = P( f N / 2 ) = 2 CN / 2
2 fk  = 2 fc k = 0,1,…,
N N
N 2

Let’s try it out on our example:

The 4Hz frequency is picked out.

Winter Semester 2006/7 Computational Physics I Lecture 12 24

 Noise
We now add some noise to our spectrum (Gaussian smearing
with
=0.5) and see what happens:

Winter Semester 2006/7 Computational Physics I Lecture 12 25

 Exercizes

1. Solve the
2 minimization problem from last lecture with

MINUIT.

2. Generate 64 data points using

f (t) = cos( /4 + 2f1t) + cos(2f 2 t) f1 = 0.5, f 2 = 1
= 0.2

and fit with a discrete Fourier transform. Extract the power

spectrum.

Winter Semester 2006/7 Computational Physics I Lecture 12 26