Simulations of 3-D Electromagnetic

Fields in Accelerator Structures

國家同步輻射研究中心
周炳榮 Ping J. Chou
[email protected]

PHYS553000— 05b

Last update: 2019-11-15

1
 十年樹木，百年樹人
Important Notes to Students: The sole purpose of this lecture notes is meant for classroom
use only. Some photographs and graphic illustrations are adapted from various reference
literatures, which are NOT to be distributed beyond the classroom use.

[References]
1. S. Ramo, J.R. Whinnery, T. Van Duzer, Fields and Waves in Communication Electronics, 3rd ed.,
(Wiley, 1994).
2. A. Taflove & S.C. Hagness, Computational Electrodynamics: the finite-difference time-domain
method, 3rd ed. (Artech House, 2005)
3. A. Peterson et al., Computational Methods for Electromagnetics (IEEE, 1998).
4. M.N.O. Sadiku, Numerical Techniques in Electromagnetics, 2nd ed., (CRC, 2000)
5. D. Eadline, High Performance Computing for Dummies (Sun & AMD).

2
Computer lab for PHYS553000-05a
 Homework 1) compute the cutoff frequencies of the four lowest propagating modes for a
rectangular waveguide. Compare the results for analytical formulas vs. numerical simulations.
Find out the dependence of computational accuracy vs. mesh size. Compare the simulated field
patterns with textbook results.
 Homework 2) compute the cutoff frequencies of the four lowest propagating modes for a
cylindrical waveguide. Compare the results for analytical formulas vs. numerical simulations.
Find out the dependence of computational accuracy vs. mesh size. Compare the simulated field
patterns with textbook results.
 Homework 3) compute the resonant frequencies of the two lowest TM- and TE-modes for a
rectangular cavity. Compare the results for analytical formulas vs. numerical simulations. Find
out the dependence of computational accuracy vs. mesh size. Compare the simulated field
patterns with textbook results.
 Homework 4) compute the resonant frequencies of the two lowest TM- and TE-modes for a
pillbox cavity. Compare the results for analytical formulas vs. numerical simulations. Find out the
dependence of computational accuracy vs. mesh size. Compare the simulated field patterns with
textbook results.
 Homework 5) using the computer simulations to perform a bead-pull measurement for the
fundament accelerating mode of a pillbox cavity. Compare the results of analytical formula vs.
numerical simulations. Find out the amplitude profile of fundamental accelerating mode from
both analytical formulas and numerical simulations.

3
 Computer Exercise:
 Homework 1) Please calculate the cutoff frequencies for TE10, TE01, and TM11 modes for a
rectangular waveguide by using the GdfidL code. Then, you compare the results of simulation
with these calculated from analytical formulas of waveguide. What are the field patterns of these
waveguide modes?

 Homework 2) Please calculate the cutoff frequencies for TE11 and TM01 modes for a circular
waveguide by using the GdfidL code. Then, you compare the results of simulation with these
calculated from analytical formulas of waveguide. What are the field patterns of these waveguide
modes?

 Homework 3) Please calculate the frequency of fundamental mode for a rectangular cavity by
using the GdfidL code. Then, you compare the results of simulation with these calculated from
analytical formulas. What are the field patterns of the fundamental mode?

 Homework 4) Continue with the previous homework, let’s introduce a small perturbation to
the metallic boundary of the rectangular cavity. Let’s place a small cube on the end plane of the
cavity. Please calculate the frequency of fundamental mode for a rectangular cavity by using the
GdfidL code. Then, you compare the results of simulation with these calculated from the
analytical formula given by the Slater theorem . What are the field patterns of the fundamental
mode with this small cube?

4
What to do in this computer lab:
1. Log in the NSRRC Wi-Fi “NSRRC-staff ”
2. Divide students into 2 persons/team
3. Assign guest account to ‘yum01’ or ‘yum02’ for each team
4. Log in the guest account
5. Type command $ source .bashrc
6. Each team create its own directory under /home/yum01 or /home/yum02
e.g. $ mkdir team1
7. Copy examples from /home/yum01/RFlab to your own working directory
8. Edit the example input files:
-general
outfile= /home/yum01/team1/output/team1-result
scratch= /dev/shm/scratch-team1
9. Edit the dimensions of waveguides or cavities as you wish
10. Run the example input file after proper revision,
$ gd1< filename.gdf | tee filename.log
11. Compare the simulated results with your analytical calculation

5
Basic physics related to wake fields and impedances:
1. T. Weiland, Part. Accelerator, vol.15 (1984) 245-292, "On the numerical solution of Maxwell's
equations and applications in the field of accelerator physics"
2. T. Weiland, R. Wanzenberg, “wake fields and impedances”, in M. Dienes et al.(eds.), Frontier
of Particle Beams: Intensity Limitations (Springer-Verlag, 1992)
3. B.W. Zotter and S.A. Kheifets, Impedances and Wakes in High-Energy Particle Accelerators
(World Scientific, 1998).

Learning about Unix/Linux:

1. 鳥哥的私房菜 http://linux.vbird.org
2. A Practical Guide to Linux, (Addison, 1997), Mark G. Sobell
3. A Practical Guide to the UNIX System, 3rd ed., (Addison, 1994), Mark G. Sobell

6
 Basic principles related to numerical simulation codes employing the
finite difference method :
1. T. Weiland, "On the numerical solution of Maxwell's equations and applications in the field
of accelerator physics" , Part. Accelerator, vol.15(1984) 245-292
2. W. Bruns, “GdfidL: a finite difference program with reduced memoery and CPU usage”, Proc.
Part. Accel. Conf. 1997 (Vancouver), p.2651.
3. Syntax and examples for the simulation code ‘GdfidL’ can be found at GdfidL homepage:
http://www.gdfidl.de
4. Numerical Techniques in Electromagnetics, 2nd ed. (CRC,2000), M.N.O. Sadiku
5. Computational Electrodynamics: the finite-difference time-domain method, 3rd ed., (Artech
House, 2005)A. Taflove and S.C. Hagness

Introductory readings for computer clusters:

• Scientific American (Aug. 2001), p.64. “Next-Generation Supercomputers”
• High Performance Linux Clusters, (O’Reilly, 2005), Joseph D. Sloan

We use the 3D electromagnetic simulation code GdfidL(parallel version) at NSRRC. GdfidL is developed
by Dr. Warner Bruns (Technical University in Berlin, Germany). It is developed for Unix/Linux operating
systems  No Microsoft Windows 

7
Basic Concepts of 3-D Computational
Electrodynamics

8
 Solving Maxwell’s Equations with Numerical Methods

  D
 Maxwell’s equations can be approximately solved by
 D  discretizing the curl operators
J   H
t

B 
   E  x   i  A triple numerical index is assigned to each field
t    
  y    j  component in the computational volume
0  B  x   k  accordingly.
   

 Finite Difference Method:

For example a simple difference equation can be obtained for the time derivative of B-field,

Bz E y Ex
  
t x y
E y E yi , j 1,k  E yi , j ,k Ex Exi 1, j ,k  Exi , j ,k
 , 
x x y y
i 1, j , k
d i , j ,k  x ( Ex  Exi , j ,k )   y ( E yi , j 1,k  E yi , j ,k )
Bz 
dt  x y 9
•The advantage of a simple finite difference method: it’s easy to understand and implemented in a
computer program by declaring a triple indexed discrete variables, i.e. (I, j, k).
•The disadvantage of a simple finite difference method: for most geometries the grid cells are also
assigned to regions where there are no electromagnetic fields (ideal conducting material). This will
increase the consumption of computer memory and decrease the computational speed.
•W. Bruns’ code GdfidL uses the linked list method instead of the simple finite- difference scheme as
described in the previous example. This results in a significant reduction of the number of grid cells
which is required to model the geometry and speed up the computation as well.

Finite-difference time-domain method (Kane Yee, 1966)

+ Finite Integration Theory (T. Weiland, 1977)

(i+1,j+1,k+1)
Maxwell’s equations (integral form) 
Hz Maxwell’s Grid Equations (Matrix representation)

(i,j,k+1) grid-based numerical modeling method,
Maxwell's equations are modified to central-
Hx difference equations, discretized, and implemented
Ez Hy (i+1,j+1,k)
in software.
Ey
(i,j,k) (i+1,j,k)
Ex

10
 Comparison between various simulation codes for electromagnetic fields

mesh physics engineering

method Ease/quality Freq./time/PIC S-parameter
MAFIA v3.2 FD Good/good Good/good/yes Fair
CST Particle Studio FD? Good/no No knowledge No knowledge
knowledge
GdfidL (parallel) FD Good/good Good/good/yes? Good

HFSS FE good/excellent yes/good/N.A. excellent

ACE3P (parallel) FE Good/Excellent Excellent No knowledge

ABCI/ FD 2D(axis- N.A./yes/N.A. N.A.

ABCI_MP (parallel) symmetrical)

•HFSS and MAFIA v4.0 are both GUI programs, less control for users. GdfidL is text-driven but
more control for users, the input scripts can be reused later on.
•ACE3P is implemented on NERSC’s IBM SP2 supercomputer (ranked No. 5 in the June 2003 TOP
500 list), one of DOE’s grand challenge projects for computational accelerator physics, a large
collaboration between major US national labs and universities. Major development is led by
SLAC  We have no access to this wonderful tool 

11
 GdfidL uses the generalized diagonal fillings 

GdfidL uses the linked lists to store EM-fields of grid cells, less memory is required. It supports
inhomogenous meshes for better modeling of sharp edge or intersecting planes.

•An example of good agreements between rf measurements and GdfidL can be found at
http://link.aps.org/abstract/PRSTAB/v5/e033501

12
 You should think of computer simulations as numerical experiments.
Do NOT blindly believe in anything you get from the computer simulations !!

Numerical experiments Traditional experiments

(Computers) (Lab apparatus: microscope, oscilloscope, …etc.)

Both methods have NOISE !

Dipole excitation in
 Example of numerical noise (adapted from presentation pillbox cavity
materials of Advanced Computations Dept., SLAC)

Noise arising from the late time instabilities for

unstructured grids

Signal response
Signal sepctrum Noise

Mode frequency

t
f
[Ref.] V. Srinivas, M. Wolf, et al., Tau3P, SciDAC TSTT-SLAC meeting (1 Oct., 2003)
http://www-group.slac.stanford.edu/acd/Presentations.html 13
  Another Approach of Discretization: Finite Element Method
•The development of finite element method can be traced back
to Alexander Hrennikoff (1941) and Richard Courant (1942).
Courant’s approach divides a domain into finite triangular
subregions. The development of finite element method began
in 1950s driven by engineering related problems. Since then it
has been generalized into a branch of applied mathematics for
numerical modeling of various physical systems.
•Finite element method can provide a better and more
accurate mesh for curved geometries than the finite difference
method.
An example of discrete mesh
(triangular mesh)
[Ref.] http://en.wikipedia.org/wiki/Finite_element

Richard Courant Ernest Courant

(father) (son)
mathematician accelerator physicist

14
[Ref.] http://mathdl.maa.org/mathDL/1//convergence/
[Ref.] http://www.pas.rochester.edu/~bodek/dept/courant.gif
Basic Concepts of Parallel Computing on a
PC Cluster

15
High-performance Computing on Linux Clusters

The basic structure of von Neumann computer:

CPU
The overall speed of a computer depends on
both the speed of CPU and the speed of
communication between memory and CPU.

communication channel (bus)

Memory bandwidth is the rate at which bits
Instructions and data are are transferred from memory over the bus.
memory stored in the memory The increase of memory bandwidth is slower
than the increase of CPU speed.

The speed of data communication can become the

CPU bottleneck of overall system performance.

cache memory
faster but more expensive •modern configuration to improve
bus the overall performance

slower memory 16
cheaper [Ref.] J.D. Sloan, High Performance Linux Clusters, (O’Reilly, 2005)
The architect of multiprocessor machine (UMA)

CPU CPU CPU

cache memory cache memory cache memory

bus

memory

This is the generic structure for the uniform memory access (UMA) machines. There is a common
shared memory. The UMA machines also called symmetric multiprocessors (SMP) machines.

•A cluster is a multicomputer configuration, a group of computers that work together.

•Basic elements of a cluster: collection of individual computers, a network to connect them, and software
that makes these computers to share work among themselves through the network.

 What is the difference between the clusters and the grid computing?
Clusters: computers are on the same subnetwork (LAN)
Grid computing: computers distribute across a WAN or internet (on a larger scale)
LAN: local area network (the same subnetwork)
17
 Amdahl’s Law
The speed of parallel computation will not
always scale up as the number of
processors increase. The serial portion of
a program will be the limiting factor of
speedup one can gain by using multiple

Speed up
computers Amdahl’s Law (Gene Amdahl
in 1960s)

Network speed and data I/O play

important roles in limiting the speedup
one can get from a cluster of computers. Number of processors
[Ref.] Wikipedia contributor, “Amdahl’s Law”, Wikipedia,
http://en.wikipedia.org/wiki/Amdahl%27s_law

Node (slave)
NFS
Node (slave)
User Head (master)

asymmetric,
I/O server Node (slave)
expanded
External network/ Internet
Node (slave) 18
 Basic configuration of the Linux cluster ‘ wilson’ at NSRRC
master node
External network
(local area
network, internet)

using real IP address using virtual IP address, message

(172.18.120.4) passing protocols (PVM, MPI,
OpenMP…)

Private network network switch

using virtual IP address to communicate with each other

19
slave node slave node slave node slave node slave node slave node slave node
Network configuration

There are two network interfaces in a computer cluster: external(public) interface and
internal(private) interface. There are reserved address spaces that we can use to configure the
private interface, a.k.a. the virtual IP.

Table. A List of private IP address spaces

Address spaces
10.0.0.0 to 10.255.255.255
172.16.0.0 to 172.31.255.255
192.168.0.0 to 192.168.255.255

The usual IP configuration constraints also apply to our cluster configuration. For example, we should
not assign the broadcast address to a computer. In this case, we want to avoid 192.168.1.0 and
possibly 192.168.1.255.

20
Linux cluster in T192 computer lab (wilson: 172.18.120.4)

RF group

http://bd08.nsrrc.org.tw/cluster.html

Linux cluster supports the computing needs of staff in Division-A (master node: 12 CPUs, 80
GB; 25 computing nodes: total 240 CPUs, 768 GB; one dedicated for Matlab)

Software installed: parallel Elegant, parallel GdfidL, parallel TRACY, Mathematica, Matlab,
CERN ROOT
21
How to Run GdfidL Jobs on a Single Node

22
How to Use GdfidL at NSRRC
•Environment (Linux or Unix)
•Basic structure of input file
•Post processing
•Parallel computing
•My own learning experience (a suggestion only)

User manual and examples for GdfidL can be found from

http://www.gdfidl.de

Environment:
Specify your environment settings for using GdfidL in your own login shell, e.g. “.cshrc” in C shell
or “.bashrc” in Bourne Again Shell

23
If your default login shell is Bash shell instead, you need to add the following lines to your “.bashrc” file.

###
### set up the GdfidL environment, from Warner Bruns
###
export GDFIDL_VERSION=180819
export GDFIDL_HOME=/usr/local/gd1
export GDFIDL_LICENSE=$GDFIDL_HOME/gdX-license
export PATH=$GDFIDL_HOME/Scripts/:$PATH

## Open MPI version

export PATH=/usr/local/openmpi/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/openmpi/lib/:$LD_LIBRARY_PATH

###
### end of GdfidL setup
###

## set up PBS/Torque
export PATH="${PATH}:/usr/local/bin:/usr/local/sbin"

24
Basic structure of input file:
•Define variables and parameters
•Define the location of output files and temporary scratch
•Define boundary conditions
•Define the geometry of objects to be studied
•Display: define plotting options
•Solver: define actions/commands to be executed (doit)
Frequency-domain
Time-domain
•End of input script use ‘gd1’ or ‘single.gd1’

Post processing:
•Define the location of your output files
•Define the physics/rf engineering parameters of your interest
•Define the options for data display
•End of post processing script use ‘gd1.pp’

Tutorial materials can be found in my account on the NSRRC cluster ‘virus’:

/home/pjchou/gd1/Tutorial or, /usr/local/gd1/Tutorial-SRRC
◆It is recommend that you go through the tutorial training in /usr/local/gd1/Tutorial-SRRC

25
 Parallel version Serial version

Set up environment for Prepare an input file for the

Open MPI version solver module of GdfidL (gd1)

Check the status of available

computing nodes first! Solve the electromagnetic field
http://bd08.nsrrc.org.tw/cluster.html, of your 3-D structure with ‘gd1’
http://172.18.120.4/ganglia/

Prepare an input file for Prepare an input file for the postprocessor
the solver module ‘gd1’, module of GdfidL (gd1.pp)
then submit the parallel
job with a batch system
from the master node using
Display the electromagnetic fields or deduce
the example script
physical parameters with the postprocessor
‘OpenMPI-Job.pbs’
module ‘gd1.pp’

You need to prepare two separate input files for GdfidL. One is used by the solver module gd1, and
another one is used by the postprocessor module gd1.pp .
26
BD/LSD @ NSRRC
http://bd08.nsrrc.org.tw/cluster.html

27
wilson-ganglia
http://172.18.120.4/ganglia/

28
How to run GdfidL in the serial mode on a single node
First , log in any slave node to edit the input file with the following commands. Using the slave
node ‘node23’ as an example.
pjchou@wilson$ ssh –X node23
pjchou@node23$ cd YourWorkingDirectory
pjchou@node23$ gedit square01.gdf &
Bla bla bla….
pjchou@node23$ gd1 < square01.gdf | tee square01.log
or,
pjchou@node23$ single.gd1 < square01.gdf | tee square01.log

Example: /home/pjchou/gd1/Tutorial/square01.gdf
## File= square01.gdf
## inputfile for a square prism cavity(Eigenmode calculation).
## BoxWidth
## 10:40, 6-17-2002 y
## GapLength
#
# Some helpful symbols:
# x
z
define(EL, 1) define(MAG, 2)
define(INF, 1000)
define(STPSZE, 5e-3)
29
…continued to the next page
###
### We enter the section "-general"
### Here we define the name of the database where the
### results of the computation shall be written to.
### (outfile= )
### We also define what names shall be used for scratchfiles.
### (scratchbase= )
###
-general
outfile= /home/pjchou/output/results
scratchbase= /dev/shm/scratch-xxx
text()= STPSZE: mesh size This will affect the speed of
text()= DT: dimensional deviation parallel computation!
text()= BoxWidth: size of box cavity

###
### We define the borders of the computational volume,
### we define the default mesh-spacing,
### and we define the conditions at the borders:
###

-mesh
spacing= STPSZE
pxlow= eval(-1*(BoxWidth/2+1e-2) )
pylow= eval(-1*(GapLength/2+1e-2) )
pzlow= eval(-1*(BoxWidth/2+1e-2) )
pxhigh= eval(BoxWidth/2+1e-2 )
pyhigh= eval(GapLength/2+1e-2 )
pzhigh= eval(BoxWidth/2+1e-2 )
30
…continued to the next page
#
# We define symbols that will be used to describe our cavity:
# The names of the symbols can be up to 32 characters long,
# The values of the symbols can be up to 128 characters long.
#
define(DT , 3e-6 ) #dimensional deviation due to temperature
define(GapLength , eval(DT+30.0e-2) ) #length in y-axis
define(BoxWidth , eval(DT+42.43e-2) ) #length in x and z-axis

#
# The conditions to use at the borders of the computational volume:
#
cxlow= electric, cxhigh= electric
cylow= electric, cyhigh= electric
czlow= electric, czhigh= electric

######
######
#
# we fill the universe with metal
#

-brick
material= EL
xlow= -INF, xhigh= INF
ylow= -INF, yhigh= INF
zlow= -INF, zhigh= INF
doit
31
…continued to the next page
#
# we carve out the square box cavity
#
-brick
material= 0
xlow= -BoxWidth/2, xhigh= BoxWidth/2
ylow= -GapLength/2, yhigh= GapLength/2
zlow= -BoxWidth/2, zhigh= BoxWidth/2
doit

-mesh
#
# enforce 2 meshplanes: at both ends of cavity side opening
#
xfixed(2, eval(-(BoxWidth/2) ), eval(BoxWidth/2) )
yfixed(2, eval(-(GapLength/2)), eval(GapLength/2) )
zfixed(2, eval(-(BoxWidth/2) ), eval(BoxWidth/2) )

-volumeplot
showpartition= yes
plotopts= -geometry 600x480
doit
The dimension of matrix to be solved
-eigenvalues
How to give a good guess for estimated frequency?
solutions= 30
estimation= 1.3e9 # the estimated highest frequency
passes= 3
doit

32
…the end
How to run parallel GdfidL on a single node without the batch job system

※Prepare the script for using Open MPI-1.8.8

1. Logging in a slave node to edit the script, e.g. node23.
pjchou@wilson$ ssh –X node23
2. Edit the script file for running GdfidL-ompi version, e.g. ‘ompiJob.x’ as shown below:

#!/bin/bash

PBS_JOBID=PJChou

rm -f $PBS_JOBID-uniq

for N in 23
do
echo node$N >> $PBS_JOBID-uniq
## echo node$N.cluster >> $PBS_JOBID-uniq
done

# How many Nodes?

NHOSTS=`wc -l < $PBS_JOBID-uniq`

# Start the MPI-Job:

BINARY=$GDFIDL_HOME/Linux-x86_64/ompi188-gd1-$GDFIDL_VERSION

…continued to the next page 33

 /opt/openmpi-1.8.8/bin/mpirun \
-np $NHOSTS \
-npernode 1 \
-hostfile $PBS_JOBID-uniq \
--show-progress \
--verbose \
$BINARY \
< ./doris04_qsub.gdf | tee doris04_qsub.log

rm -f $PBS_JOBID-uniq

echo finished.
# exit

###------------------------------------------------------------------------
### P.J. Chou, 17:53, 2018-8-30
…the end of the script file ‘ompiJob.x’

After the script is completed, change its permission to an executable file. Run the following
commands:
pjchou@node23$ chmod u+x ompiJob.x
pjchou@node23$ ./ompiJob.x

34
3. Another example of time-domain simulation using the input file ‘hkicker-v3.3.gdf’. You need to
modify the name of input file in the script file ‘ompiJob.x’ first. You can run the job either from
the master node or a slave node with the following command:
pjchou@node23$ ./ompiJob.x
or, pjchou@wilson$ ./ompiJob.x

#!/bin/bash
Bla bla bla…
…
Bla bla bla…
/opt/openmpi-1.8.8/bin/mpirun \
-np $NHOSTS \
-npernode 1 \
-hostfile $PBS_JOBID-uniq \
--show-progress \
--verbose \
$BINARY \
< ./hkicker-v3.3.gdf | tee hkicker-v3.3.log

rm -f $PBS_JOBID-uniq

echo finished.
# exit

###------------------------------------------------------------------------
### P.J. Chou, 17:53, 2018-8-30

…the end of the script file ‘ompiJob.x’ 35

How to run parallel GdfidL on a single node (batch job mode with Torque)

※Prepare the script and submit a batch job from the master node (required)
1. Edit the script ‘OpenMPI-Job.pbs’ in your working directory on the mater node as shown
below:
#!/bin/bash
### --------------------------------------------------------------------------
### This PBS script file is provided by Warner Bruns in Nov. 2015 .
### 14:58 Sept. 4, 2018
### --------------------------------------------------------------------------
### Usage: qsub -X ./OpenMPI-Job.pbs

## --- Queue name --- How to get X-display?

#PBS -q batch-two
## --- Use one node to run the batch job --- pjchou@wilson$ ssh –X node23
#PBS -l nodes=node23:ppn=1 pjchou@node23$ echo $DISPLAY
### --- X11 DISPLAY setting --- localhost:11.0
export DISPLAY=localhost:11.0 pjchou@node23$ exit
pjchou@wilson$
# Set the Time which this Job will not exceed:
#PBS -l walltime=24:00:00

# Set the Name of this Job:

#PBS -N PJ_gd1

…continued to the next page

36
# Mail Alert at Begin, End and Abort of Execution:
#PBS -m bea

# Send Mail to this Address: Change This! <---

#PBS -M [email protected]

################
# Start Job from the Directory it was submitted.
cd $PBS_O_WORKDIR

echo PBS_JOBID is $PBS_JOBID

echo PBS_WORKDIR is $PBS_WORKDIR
echo PBS_NODEFILE is $PBS_NODEFILE
# echo its Content is
# cat $PBS_NODEFILE

# Create a File which contains the Nodes without Duplicates.

sort < $PBS_NODEFILE | uniq > $PBS_JOBID-uniq

# How many total Cores, how many Nodes?

NSLOTS=`wc -l < $PBS_NODEFILE`
NHOSTS=`wc -l < $PBS_JOBID-uniq`
echo NSLOTS is $NSLOTS
echo NHOSTS is $NHOSTS

# Compute NrOfThreads per Node:

NTHREADS=`echo eval\( $NSLOTS / $NHOSTS \) | m4`
echo NTHREADS is $NTHREADS
…continued to the next page 37
 # Start the MPI-Job:
export GDFIDL_HOME=/usr/local/gd1/
export GDFIDL_VERSION=180819
# export BINARY=$GDFIDL_HOME/Linux-x86_64/single.ompi188-gd1-
$GDFIDL_VERSION
export BINARY=$GDFIDL_HOME/Linux-x86_64/ompi188-gd1-$GDFIDL_VERSION
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/openmpi-1.8.8/lib/"

/opt/openmpi-1.8.8/bin/mpirun \
-np $NHOSTS -hostfile $PBS_JOBID-uniq \
-npernode 1 \
$BINARY \
-DNROFTHREADS=$NTHREADS \
Single precision version, if
-DOUTFILE=$PBS_O_WORKDIR/Result \
you want to speed up the -DSCRATCH=/dev/shm/$PBS_JOBID \
time-domain calculation. -DLSSUBCWD=$PBS_O_WORKDIR/ \
< ./doris04_qsub.gdf | tee doris04_qsub.log

rm -f $PBS_JOBID-uniq

echo finished.
exit

…the end of the script file ‘OpenMPI-Job.pbs’

38
3. After you edit the script ‘OpenMPI-Job.pbs’, submit the batch job from the master node with
the following commands:
pjchou@wilson$ qsub –X OpenMPI-Job.pbs
396.wilson.cluster

PBS Job ID

You can view the job status by the following command,

pjchou@wilson$ qstat -ans
The X-window displays of GdfidL plots will popped up during the calculation. When these X-
window plots disappear, it means the simulation is finished. You can view the results and any
error messages from these two files ‘OpenMPI-job.o396’ and ‘OpenMPI-Job.e396’, respectively.

For the time-domain simulation, you edit the name of input file in the script ‘OpenMPI-Job.pbs’,
then submit the batch job on the master node. An example input file ‘hkicker-v3.3.gdf’ is used
for demonstration.

39
16 cores were used during
the calculation

40
How to Run GdfidL Jobs on Multi Nodes

41
How to run parallel GdfidL on multi nodes (batch job mode with Torque)

※Prepare the script and submit a batch job from the master node (required)
1. Edit the script ‘OpenMPI-Job.pbs’ in your working directory on the mater node. We use
the example input file ‘doris04_qsub_openmpi.gdf’. Besides the name of input file that you
need to change, the major change you need to do in the script file is shown below:
pjchou@wilson$ gedit OpenMPI-Job.pbs

#!/bin/bash
### --------------------------------------------------------------------------
### This PBS script file is provided by Warner Bruns in Nov. 2015 .
### 14:58 Sept. 4, 2018
### --------------------------------------------------------------------------
### Usage: qsub -X ./OpenMPI-Job.pbs

## --- Queue name ---

#PBS -q batch-two
## --- Choose which nodes to be used for compuation: Change this! <---
#PBS -l nodes=node23:ppn=1+node24:ppn=1 major change you need to do
## --- set up X11 DISPLAY for one node: Change this! <---
export DISPLAY=localhost:11.0

## --- Set the Time which this Job will not exceed ---
#PBS -l walltime=24:00:00

…continued to the next page

42
Bla bla bla…
Bla bla bla…

# Start the MPI-Job:

export GDFIDL_HOME=/usr/local/gd1/
export GDFIDL_VERSION=180819
# export BINARY=$GDFIDL_HOME/Linux-x86_64/single.ompi188-gd1-
$GDFIDL_VERSION
export BINARY=$GDFIDL_HOME/Linux-x86_64/ompi188-gd1-$GDFIDL_VERSION
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/openmpi-1.8.8/lib/"

/opt/openmpi-1.8.8/bin/mpirun \
-np $NHOSTS -hostfile $PBS_JOBID-uniq \
-npernode 1 \
$BINARY \
-DNROFTHREADS=$NTHREADS \
-DOUTFILE=$PBS_O_WORKDIR/Result \
-DSCRATCH=/dev/shm/$PBS_JOBID \
-DLSSUBCWD=$PBS_O_WORKDIR/ \
< ./doris04_qsub_ompi.gdf | tee doris04_qsub_ompi.log

rm -f $PBS_JOBID-uniq

echo finished.
exit

…the end of the script file ‘OpenMPI-Job.pbs’

43
All the processors on each node are used for this batch job as shown in the screen shot below

Batch job using the input file ‘doris04_qsub_openmpi.gdf’

This example is not large enough. The calculation is slower when using two nodes! 44
The X-window displays of GdfidL plots will popped up during the calculation. When the X-window displays
disappear, it means the calculation is finished. You can view the results and any error messages from these
two files ‘OpenMPI-job.o***’ and ‘OpenMPI-Job.e***’, respectively.

Batch job using the input file ‘hkicker-v3.3.gdf’

45
How to Be a Smart User of GdfidL

46
How to model your structure with finer meshes and yet less computational
time in a smart way? Tricks of building your model

Experts tend to use inhomogeneous meshes to describe a 3-D structure with sharp
edges or curved features. On the contrary, a novice tends to use homogeneous
meshes to describe the structure. To achieve a better accuracy without big increase of
computation time, you should use inhomogeneous meshes as much as possible. You
need to have a good understanding about the physics of the problem that you are
going to simulate. Simply abuse the computing power of Linux cluster without much
thinking is a brute force approach that is stupid!

A test case I did before can serve as an example of using inhomogeneous meshes to
improve the accuracy and keep computation time as reasonable as possible.

Please refer to /home/pjchou/gd1/Tutorial/rfwindow2.gdf and ./sparm.gdf

The detail of the test run is shown in the following pages.

47
Example: Assembly of SRF Module S0 – RF Window

1. PEP-II type pre-stressed “traveling type” RF

window (without matching posts);
2. Manufacture will be done in 2009.
3. Technical transfer of PEP-II RF cavity from SLAC has
been received. Consultants from retired SLAC
experts would be arranged.

This slide is provided from Chaoen Wang (RF group).

We want to double check simulation results with GdfidL.

Ghost mode at 518 MHz

Pick fc= 500 MHz for cross check 48
Results:
The thickness of ceramic disk= 0.717”, freq_center= 500 MHz
mesh size= 1 mm, uniform mesh in x, y, z direction
total grid points~ 45e6
I didn't try to optimize the mesh for faster calculation.
No symmetry plane was used to speed up the calculation either.
I did test runs on the following two cases:
1) using 4 compute nodes , but CPU and RAM were shared with other users;
computation time~ 2 days and 8 hours
2) using one compute node only; uniform mesh= 1mm, computation time~ 2 days 6 hours
Model of ceramic window:

49
 S11 parameter as computed by GdfidL (t = 0.717”):

RF Ceramic Window, t= 0.717"

▲You can get the calculated results and
0 perform data reduction off-line with
other tools. The results calculated by
-5
gd1.pp are stored in ./scratchdir
-10
S11 in [dB]

-15

-20

-25

-30
300 350 400 450 500 550 600 650
frequency [MHz]

RF Ceramic Window, t= 0.717"

1.0
0.9
0.8
0.7
0.6
S11

0.5
0.4
0.3
0.2
0.1
0.0
300 350 400 450 500 550 600 650
frequency [MHz]
50
Inhomogeneous mesh, zmesh= 1 mm, x/ymesh= 1.25 mm, computation time= 12 hrs.

51
 rf ceramic window, t= 0.717" inhomogeneous mesh

-5
S11 in [dB]

-10

-15

-20

-25
300 350 400 450 500 550 600 650
frequency [MHz]

rf ceramic window, t= 0.717" inhomogeneous mesh

0.9

0.8

0.7

0.6

0.5
S11

0.4

0.3

0.2

0.1

0.0
300 350 400 450 500 550 600 650
frequency [MHz] 52
Example: /home/pjchou/gd1/Tutorial/square01.gdf
The output message displayed on the terminal during the
computation,
System is Linux-x86_64
Version is 2010-02-08

### defining: "@pi", value: "3.14159265358979323846264338327950"

### defining: "@clight", value: "299792458"

### defining: "INF", value: "1e30"

### defining: "BIG", value: "1e30"

##################
# start date : 27/04/2010
# start time : 18:27:46
##################
I was compiled on Mon Feb 8 13:12:25 GMT 2010 on host wb024
This host is: virus
--
-- Your license is going to expire in 613 days.
--
#######################################################################
bla…
bla…
…continued to the next page
53
 i freq(i) acc(i) cont(i)
1 499.58014409e+6 0.0000088784 0.0000221937 # "grep" for me
2 611.87074499e+6 0.0000038001 0.0000101327 # "grep" for me
3 611.87101338e+6 0.0000092365 0.0000307894 # "grep" for me
4 706.52397906e+6 0.0000004300 0.0000015768 # "grep" for me
5 706.52397906e+6 0.0000037623 0.0000163264 # "grep" for me
6 789.79735384e+6 0.0000007873 0.0000036788 # "grep" for me
7 789.80047196e+6 0.0000010278 0.0000054797 # "grep" for me
8 865.20826876e+6 0.0000000658 0.0000003726 # "grep" for me
9 865.21130491e+6 0.0000000230 0.0000001458 # "grep" for me
10 934.54570393e+6 0.0000000432 0.0000002877 # "grep" for me
11 934.54604689e+6 0.0012514099 1.0000000000 # "grep" for me
12 934.54833910e+6 0.0000000135 0.0000000992 # "grep" for me
13 998.99165425e+6 0.0000000092 0.0000000733 # "grep" for me
14 1.05949636e+9 0.0000071575 0.0000621237 # "grep" for me
15 1.05949652e+9 0.0000030006 0.0000280829 # "grep" for me
16 1.11664859e+9 0.0004014902 0.0038918268 # "grep" for me
17 1.11683609e+9 0.0001871118 0.5572846366 # "grep" for me
18 1.11695127e+9 0.0005257445 0.0054598696 # "grep" for me
19 1.17119086e+9 0.0009741136 0.0107482412 # "grep" for me
20 1.22311515e+9 0.1732660204 1.0000000000 # "grep" for me
################################
fmax # outfile : /tmp/pjchou/dummy/square-results
# CPU-seconds for eigenvalues : 742
# Start date : 27/04/2010
# End date : 27/04/2010
# Start time : 18:27:46
# End time : 18:41:13
# The computation of the eigenvalues has finished normally.
# Start the postprocessor to look at the results. 54
stop This is the normal end. …the end of output message
How to calculate eigenmodes with good accuracy in a smart way?

• How accurate the calculated eigenfrequencies should be?

i) For 2-D problems, the accuracy should be ~10-5 or better
ii) For 3-D problems, the accuracy should be ~10-3 or better

• Recipe for improving the accuracy of computed eigenmodes

i) Find the fmax from the output
ii) Set [estimation]= 1.1× fmax
iii) Improve the mesh of your model. No brute force, only smart meshing helps
iv) Increase number of modes by 50% 1.5× [solutions]
v) Increase number of iterations use larger value for [passes], passes= 3 is
large enough

….. In THAT order!

Important note: To compute the eigenmodes, ‘single.gd1’ results in a worse accuracy than ‘gd1’
for the same input file. It is advised to use the double-precision version ‘gd1’.

55
Example : Calculate the cutoff frequency of a circular waveguide
Please refer to /home/pjchou/gd1/Calibration/circular-wvguide.gdf and plotwvguide.gdf

Refer to Table.7 in “phys55300-5a”, for TE11

mode, u’11= 1.8412

R= 0.1 m 
cumn
fc   878.500 MHz
2R

How to do it?

~/gd1/Calibration$ gd1 –host=node23 < circular-wvguide.gdf | tee circular-wvguide.log

56
a) Mesh size= 0.01 m @ central region

Electric field of TE11 mode

STPSZE= 0.01 m

zig-zag due to meshing

mesh= 0.01 m

mesh= 0.00318 m

57
The simulated cutoff frequency for TE11 mode is 875.311 MHz (STPSZE= 0.01 m), the
analytical formula gives 878.500 MHz

It took 6 seconds.
58
b) Mesh size= 0.005 m @ central region

Electric field of TE11 mode

STPSZE= 0.005 m

Improving the mesh within

this region

59
The simulated cutoff frequency for TE11 mode is 874.448 MHz (STPSZE= 0.005 m), the
analytical formula gives 878.500 MHz

It took 9 seconds.
60
c) Mesh size= 0.0025 m @ central region

Electric field of TE11 mode

STPSZE= 0.0025 m

Improving the mesh within

this region

61
The simulated cutoff frequency for TE11 mode is 875.940 MHz (STPSZE= 0.0025 m), the
analytical formula gives 878.500 MHz

It took 26 seconds.

62
 d) Mesh size= 0.005 m @ central region

mesh= 0.00318 m
mesh= 0.005 m
Electric field of TE11 mode

Improving the mesh within

this region

63
The simulated cutoff frequency for TE11 mode is 878.614 MHz (STPSZE= 0.005 m), the
analytical formula gives 878.500 MHz
A proper use of inhomogenous mesh can greatly improve the accuracy and save the
computational time!

It took 15 seconds.
64
 Example : Calculate the loss factor of a step collimator with circular symmetry
Please refer to /home/pjchou/gd1/Calibration/step-collimator.gdf and wake.gdf

~/gd1/Calibration/step-collimator.gdf

Job PID

If you want to kill a parallel GdfidL job, find out the job PID and use the following command, 65
$ kill -9 job_PID
66
 -fdtd
-lcharge
charge= 1e-12
sigma= BunchLength
xposition= BeamX, yposition= BeamY
shigh= RecordLength
napoly= yes
return

-ports
name= beamlow , plane= zlow, doit
name= beamhigh, plane= zhigh, doit
return

doit

62 minutes

67
To view the computed wakepotential and impedance, run the postprocessor gd1.pp
$ gd1.pp < wake.gdf

232.956 1015
 0.233 [V / pC ]
11012
Loss factor

68
69
70
Example : How to model four-button beam position monitor
/home/pjchou/gd1/TPS/BPM/bpm.tps01.gdf

71
Example : A flange gap in standard beam pipe
..\..\..\..\PJ.Chou_Research\CollectiveEffects\ImpedanceBudget\TLS\FlangeJoint

You can only use inhomogeneous mesh

in the beam direction for the frequency-
domain computation.

For the time-domain computation, the

mesh must be uniform in the beam
direction.

72
Refer to /home/pjchou/gd1/3D/flange.gdf and plotwake.gdf
Example: benchmark GdfidL ‘–type=impedance’
Loss factor for a cylindrical resistive pipe: length= 0.1 m, radius= 15 mm, rms bunch length= 3 mm,
made of aluminum (resistivity= 2.62e-8 ohm-m) kloss= 0.84e-3 V/pC (analytical result)

File= /home/pjchou/gd1/TPS/Bellows/cylinder.gdf || ( ) (3 / 4)c Z0


L 4 2b z3/2 2 c
"Handbook of accelerator physics &
engineering, 2nd ed", p.265.

Kloss= 1.37e-3 V/pC

L= 0.1 m, STEPSZE= 1 mm,

xfixed( 31, PipeRadius, 0.6*PipeRadius ) # 0.2 mm

xfixed( 31, -PipeRadius, -0.6*PipeRadius ) # 0.2 mm
yfixed( 31, PipeRadius, 0.6*PipeRadius ) # 0.2 mm
yfixed( 31, -PipeRadius, -0.6*PipeRadius ) # 0.2 mm
zfixed( 201, ZMIN, ZMAX ) # 0.5 mm

Relative error=|GdfidL-analytical|/analytical= 0.631 73

 Kloss= 0.751e-3 V/pC
L= 0.1 m, STEPSZE= 0.5 mm,

xfixed( 31, PipeRadius, 0.6*PipeRadius ) # 0.2 mm

xfixed( 31, -PipeRadius, -0.6*PipeRadius ) # 0.2 mm
yfixed( 31, PipeRadius, 0.6*PipeRadius ) # 0.2 mm
yfixed( 31, -PipeRadius, -0.6*PipeRadius ) # 0.2 mm
zfixed( 201, ZMIN, ZMAX ) # 0.5 mm

Relative error=|GdfidL-analytical|/analytical= 0.106 74

Refine the mesh of transverse footprint greatly improve the accuracy

Kloss= 0.852e-3 V/pC

L= 0.1 m, STEPSZE= 0.3 mm,

xfixed( 61, PipeRadius, 0.6*PipeRadius ) # 0.1 mm

xfixed( 61, -PipeRadius, -0.6*PipeRadius ) # 0.1 mm
yfixed( 61, PipeRadius, 0.6*PipeRadius ) # 0.1 mm
yfixed( 61, -PipeRadius, -0.6*PipeRadius ) # 0.1 mm
zfixed( 201, ZMIN, ZMAX ) # 0.5 mm

Relative error=|GdfidL-analytical|/analytical= 0.014

75
 Numerical instability in time-domain simulations

shigh= 200 m, beam sigma= 4.5 mm, (xb, yb)= (0, 1e-3 m)
W||(s) Re(Z||)

File= hkicker_v2d3.gdf
76
Wx(s) Wy(s)

The numerical buildup is observed! Calculated by

VERSION=180128g

77
When we zoom in the noise buildup, the oscillating frequency is around 1.65 GHz. It seems to
correspond to the dominant resonant peak (1.697 GHz) in Re(Zy). Since the amplitude of this resonant
mode is growing, the spectral magnitude of this mode is not reliable.

78
* Debug the Re(Zy) and the noise buildup in Wy
In the inputfile for gd1.pp, -wakes, shigh= 120, VERSION=180128g

File= hkicker_v2d3.gdf

79
In the inputfile for gd1.pp, -wakes, shigh= 140

80
In the inputfile for gd1.pp, -wakes, shigh= 160

81
In the inputfile for gd1.pp, -wakes, shigh= 180

82
In the inputfile for gd1.pp, -wakes, shigh= 200

As we increase the ‘shigh’ value, the amplitude of dominant mode in Re(Zy) keeps increasing. On the
other hand, the amplitude of dominant mode in Re(Z) reaches a steady value. The noise buildup in Wy
is affecting the accuracy of Re(Zy). Therefore, we can’t determine the transverse resonant modes by
time domain solver based on VERSION=180128g.

* End of debug 83
* Using ‘single.gd1’ and ‘–fdtd, hfdtd= yes’, computation time= 49 h 14 m, VERSION=180510

df= 2.99e6 Hz
Remedy for the numerical instability use the higher-order FDTD
84
-wakes, wantdf= 0.5e6 (shigh= 100 m)

85
* Using ‘single.gd1’ and 2*beampipe, computation time= 12 h 50 m, VERSION=180128g

-wakes, wantdf= 0.5e6

Files= Wz_200m.txt, WY_200m.txt, ReZ_200m.txt, ReZY_200m.txt

Remedy for the numerical instability use a much longer beam pipe
86
 Benchmark GdfidL FDTD using a coaxial line

1) Calculation of the characteristic line impedance (this feature only works in single-node runs)

138
Z0  log( D d )
d r
D 60
 ln( D d )
r

For er= 1, D= 23 mm, d= 10 mm: Z0= 49.97 V/A (formula), Z0=

49.8524 V/A (gdfidl w/ step size= 0.2 mm & inhomogeneous
x/ymesh) 0.24% discrepancy.
File= ~/gd1/Calibration/test_coaxial.gdf
2) Calculation of S-parameter & time-domain signal

File= ~/gd1/Calibration/test_coaxial-2.gdf
Using node22(8 threads): 28 min.

output

120 mm 0.4 ns

input
• Short pulse excitation

Excitation signal (input signal@port_Input) Reflected signal (output signal@port_Input)

2*0.4 ns

File= ~/gd1/Calibration/plot-sparm.gdf
transmitted signal (output signal@port_Output)

delayed by 0.4 ns w.r.t. excitation pulse

S11< 3e-5 S21~ 0.9999
<gd1 inputfile>
-fdtd
-pexcitation
port= Input
mode= 1
amplitude= 1
if(PULSE) then
frequency= FREQEX
bandwidth= 0.7*FREQEX
else
frequency= 0
risetime= 30*STPSZE/3e8
endif

<gd1.pp inputfile>
-sparameter
ports= all
modes= 1
timedata= yes
freqdata= yes # ‘no’ for TDR, excitation_freq= 0
• Step-pulse excitation

Excitation signal (input signal@port_Input) Reflected signal (output signal@port_Input)

File= ~/gd1/Calibration/test_coaxial-2.gdf
Using node22(24 threads): 10 min.
transmitted signal (output signal@port_Output)
How to estimate the required memory (size) for your simulation job?

No. of gridcells= A
No. of gridcells_field= B (i.e. gridcells in a dielectric where fields are nonzero)

For the single-precision version of GdfidL(*):

Total memory required (bytes)= 4×A+64×B
For the double-precision version of GdfidL:
total memory required (bytes)= 4×A+128×B

For a quick estimate of required memory, let’s assume the fill ratio A/B is roughly 0.5. Then, we
have the following relations:

• Single-precision GdfidL≈ 36×A bytes

• Double-precision GdfidL≈ 68×A bytes

For rectangular structures, the fill ratio will be closer to one.

 Using more compute nodes does NOT necessarily speed up the computation!

(*) W. Bruns, private communication (8 Sept., 2003)

95
 My own learning experience (a suggestion only):
{
1. Starting with the tutorial files that I prepared, understand square0*.gdf and pillbox.gdf
to familiarize yourself with the basic syntax of GdfidL.
2. Then, you start with the tutorial provided by W. Bruns (he did a wonderful job!), step-
by-step practice these files ‘doris**.gdf ’ and understand why the inputfiles were
written the way it is.
3. Those files of GdfidL tutorial are located in ‘/usr/local/gd1/tutorial-SRRC’.
4. User manual can be found in ‘/usr/local/gd1’ and GdfidL’s home page.
5. Using the rigorous results for the pillbox cavity in J.D. Jackson’s textbook Classical
Electrodynamics, constructing a model with GdfidL and comparing both results 
confidence builder/verification/validation
}

You are ready to fly !

*Validation of GdifdL: “Benchmark Results of RF Simulation Code GdfidL”, P.J. Chou,

SRRC/OP/99-05 (1999), unpublished. 96
Appendix-1:
Remote computing and local display: X Window system

X window system was created in 1984 at MIT by researchers

working on distributed computing project. X window was
specifically designed to be used over networked environment
rather than on an integral or attached display device. X allows a
user to run a program on a remote system and send the results
to a local display where the end user is located. X uses a client-
server model: an X server communicates with various client
programs. Server is a program controlling the video output, local
X display provides display services to programs. Any remote
program uses this service is a client.

Example:
virus> setenv DISPLAY 140.110.205.206:0.0

[Ref.] "X Window System." Wikipedia, The Free

Encyclopedia. 19 Feb 2009, IP address of your local computer/terminal (your
http://en.wikipedia.org/wiki/X_window physical location)

97
Appendix-2:
Install MobaXterm on your MS Windows (X-server on you PC)

Step.1 Download the program from http://mobaxterm.mobatek.net/

1) After installation, launch the program

2) Press this button

98
Step.2 open another x-terminal by typing the command ‘xterm –sb &’

99
Step.3 Log in the Linux cluster by using the command ‘ssh –X -l username 172.18.120.4’

100
For detailed formulation of FDTD, see
Computational Electrodynamics, A. Taflove (NSRRC, QC760/T34)
Computational Methods for Electromagnetics, A. Peterson (NSRRC, QC760/P48)
Numerical Techniques in Electromagnetics, 2nd ed., M.N.O. Sadiku (CRC,2000)
101