PETSc Tutorial

Numerical Software Libraries for

the Scalable Solution of PDEs
Satish Balay, Kris Buschelman, Bill Gropp,
Dinesh Kaushik, Matt Knepley,
Lois Curfman McInnes, Barry Smith, Hong Zhang
Mathematics and Computer Science Division
Argonne National Laboratory
http://www.mcs.anl.gov/petsc

Intended for use with version 2.1.3 of PETSc

Tutorial Objectives
Introduce the Portable, Extensible Toolkit for
Scientific Computation (PETSc)
Demonstrate how to write a complete parallel
implicit PDE solver using PETSc
Introduce PETSc interfaces to other software
packages
Explain how to learn more about PETSc

The Role of PETSc

Developing parallel, non-trivial PDE solvers that
deliver high performance is still difficult and
requires months (or even years) of concentrated
effort.
PETSc is a toolkit that can ease these difficulties
and reduce the development time, but it is not a
black-box PDE solver nor a silver bullet.

What is PETSc?

A freely available and supported research code

Available via http://www.mcs.anl.gov/petsc

Free for everyone, including industrial users
Hyperlinked documentation and manual pages for all routines
Many tutorial-style examples
Support via email: [email protected]
Usable from Fortran 77/90, C, and C++

Portable to any parallel system supporting MPI, including

Tightly coupled systems
Cray T3E, SGI Origin, IBM SP, HP 9000, Sun Enterprise

Loosely coupled systems, e.g., networks of workstations

Compaq, HP, IBM, SGI, Sun
PCs running Linux or Windows

PETSc history
Begun in September 1991
Now: over 8,500 downloads since 1995 (versions 2.0 and 2.1)

PETSc funding and support

Department of Energy: MICS Program, DOE2000, SciDAC
National Science Foundation, Multidisciplinary Challenge Program, CISE

The PETSc Team

Satish
Balay

Matt
Knepley

Kris
Buschelman

Lois
Curfman
McInnes

Bill
Gropp

Barry
Smith

Dinesh
Kaushik

Hong
Zhang

PETSc Concepts
How to specify the mathematics of the problem
Data objects
vectors, matrices

How to solve the problem

Solvers
linear, nonlinear, and time stepping (ODE) solvers

Parallel computing complications

Parallel data layout
structured and unstructured meshes

Tutorial Topics (Long Course)

Getting started

Solvers (cont.)

Data objects

Data layout and ghost

values

motivating examples
programming paradigm
vectors (e.g., field
variables)
matrices (e.g., sparse
Jacobians)

Viewers

object information
visualization

Solvers

Linear

Profiling and
performance tuning

nonlinear
timestepping (and ODEs)
structured and
unstructured mesh
problems

Putting it all together

a complete example

Debugging and error

handling
New features
Using PETSc with other
software packages

Tutorial Topics:
Using PETSc with Other
Packages
Linear solvers

AMG http://www.mgnet.org/mgnet-codes-gmd.html
BlockSolve95 http://www.mcs.anl.gov/BlockSolve95
ILUTP http://www.cs.umn.edu/~saad/
LUSOL http://www.sbsi-sol-optimize.com
SPAI http://www.sam.math.ethz.ch/~grote/spai
SuperLU http://www.nersc.gov/~xiaoye/SuperLU

Optimization software
TAO http://www.mcs.anl.gov/tao
Veltisto http://www.cs.nyu.edu/~biros/veltisto

Mesh and discretization tools

Overture http://www.llnl.gov/CASC/Overture
SAMRAI http://www.llnl.gov/CASC/SAMRAI
SUMAA3d http://www.mcs.anl.gov/sumaa3d

ODE solvers
PVODE

http://www.llnl.gov/CASC/PVODE

Others
Matlab http://www.mathworks.com
ParMETIS http://www.cs.umn.edu/~karypis/
metis/parmetis

10

CFD on an Unstructured Mesh

3D incompressible Euler
Tetrahedral grid
Up to 11 million unknowns
Based on a legacy NASA
code, FUN3d, developed
by W. K. Anderson
Fully implicit steady-state
Primary PETSc tools:
nonlinear solvers (SNES)
and vector scatters
(VecScatter)

Results courtesy of Dinesh Kaushik

and David Keyes, Old Dominion
Univ., partially funded by NSF and
ASCI level 2 grant

11

Fixed-size Parallel Scaling Results

(GFlop/s)
Dimension=11,047,096

12

Fixed-size Parallel Scaling Results

(Time in seconds)

13

Inside the Parallel Scaling Results on ASCI

Red
ONERA M6 wing test case, tetrahedral grid of 2.8 million vertices (about 11 million
unknowns)
on up to 3072 ASCI Red nodes (each with dual Pentium Pro 333 MHz processors)

14

Multiphase Flow

Oil reservoir simulation: fully implicit, time-dependent

First fully implicit, parallel compositional simulator
3D EOS model (8 DoF per cell)
Structured Cartesian mesh
Over 4 million cell blocks, 32 million DoF
Primary PETSc tools: linear solvers (SLES)
restarted GMRES with Block Jacobi preconditioning
Point-block ILU(0) on each process

Over 10.6 gigaflops sustained performance on 128 nodes of an

IBM SP. 90+ percent parallel efficiency
Results courtesy of collaborators Peng Wang and Jason Abate, Univ. of Texas at
Austin, partially funded by DOE ER FE/MICS

15

PC and SP Comparison
179,000 unknowns (22,375 cell blocks)
1200

Solution time

1000
800
600

PC
SP

400
200
0

16

Number Processors

PC: Fast ethernet (100 Megabits/second) network of 300 Mhz Pentium

PCs with 66 Mhz bus
SP: 128 node IBM SP with 160 MHz Power2super processors and 2
memory cards

16

Speedup

Speedup Comparison
18
16
14
12
10
8
6
4
2
0

PC
SP
ideal

Number Processors

17

Structures Simulations
ALE3D (LLNL structures code) test problems
Simulation with over 16 million degrees of freedom
Run on NERSC 512 processor T3E and LLNL ASCI Blue
Pacific
Primary PETSc tools: multigrid linear solvers (SLES)

Results courtesy of Mark Adams (Univ. of California, Berkeley)

18

ALE3D Test Problem Performance

60
50
40
Time 30
20
10
0

100

200

300

400

Processors

NERSC Cray T3E Scaled Performance

15,000 DoF per processor

500

19

Tutorial Approach
From the perspective of an application programmer:
Beginner

Advanced

basic functionality, intended

for use by most programmers

user-defined customization of
algorithms and data structures

beginner
beginner

advanced
advanced

Intermediate
selecting options, performance
evaluation and tuning
2
intermediate
intermediate

Developer
advanced customizations,
intended primarily for use by
library developers
4
developer
developer

20

Incremental Application
Improvement
Beginner
Get the application up and walking

Intermediate
Experiment with options
Determine opportunities for improvement

Advanced
Extend algorithms and/or data structures as needed

Developer
Consider interface and efficiency issues for integration and
interoperability of multiple toolkits

Full tutorials available at

http://www.mcs.anl.gov/petsc/docs/tutorials

21

Structure of PETSc
PETSc PDE Application Codes
ODE Integrators

Visualization

Nonlinear Solvers,
Interface
Unconstrained Minimization
Linear Solvers
Preconditioners + Krylov Methods
Object-Oriented
Grid
Matrices, Vectors, Indices
Management
Profiling Interface
Computation and Communication Kernels
MPI, MPI-IO, BLAS, LAPACK

22

PETSc Numerical Components

Nonlinear Solvers
Newton-based Methods
Line Search Trust Region

Time Steppers

Other

Backward Pseudo Time

Euler
Stepping

Euler

Other

Krylov Subspace Methods

GMRES

Additive
Schwartz

CG

Block
Jacobi

Compressed
Sparse Row
(AIJ)

CGS

Bi-CG-STAB

TFQMR

Preconditioners
Jacobi

Blocked Compressed
Sparse Row
(BAIJ)

ILU

ICC

Other

LU
(Sequential only)

Others

Matrix-free

Other

Matrices
Block
Diagonal
(BDIAG)

Distributed Arrays
Vectors

Richardson Chebychev

Dense

Index Sets
Indices

Block Indices

Stride

Other

23

What is not in PETSc?

Discretizations
Unstructured mesh generation and refinement tools
Load balancing tools
Sophisticated visualization capabilities

But PETSc does interface to external software that

provides some of this functionality.

24

Flow of Control for PDE Solution

Main Routine

Timestepping Solvers (TS)

Nonlinear Solvers (SNES)
Linear Solvers (SLES)
PC
Application
Initialization

PETSc

KSP
Function
Evaluation
User code

Jacobian
Evaluation
PETSc code

PostProcessing

25

Flow of Control for PDE Solution

Main Routine
Overture

SAMRAI
Timestepping Solvers (TS)
Nonlinear Solvers (SNES)

Linear Solvers (SLES)

PC
Application
Initialization
User code

SPAI

PETSc

KSP
ILUDTP
PETSc code

Function
Evaluation
Other Tools

Jacobian
Evaluation

PostProcessing

PVODE

26

Levels of Abstraction
in Mathematical Software
Application-specific interface

Programmer manipulates objects associated with the

application

High-level mathematics interface

Programmer manipulates mathematical objects, such as

PDEs and boundary conditions

PETSc
emphasis

Algorithmic and discrete mathematics

interface

Programmer manipulates mathematical objects (sparse

matrices, nonlinear equations), algorithmic objects (solvers)
and discrete geometry (meshes)

Low-level computational kernels

e.g., BLAS-type operations

27

Solver Definitions: For Our

Purposes
Explicit: Field variables are updated using
neighbor information (no global linear or
nonlinear solves)
Semi-implicit: Some subsets of variables (e.g.,
pressure) are updated with global solves
Implicit: Most or all variables are updated in a
single global linear or nonlinear solve

28

Focus On Implicit Methods

Explicit and semi-explicit are easier cases
No direct PETSc support for
ADI-type schemes
spectral methods
particle-type methods

29

Numerical Methods Paradigm

Encapsulate the latest numerical algorithms in a
consistent, application-friendly manner
Use mathematical and algorithmic objects, not
low-level programming language objects
Application code focuses on mathematics of the
global problem, not parallel programming details

30

PETSc Programming Aids

Correctness Debugging
Automatic generation of tracebacks
Detecting memory corruption and leaks
Optional user-defined error handlers

Performance Debugging
Integrated profiling using -log_summary
Profiling by stages of an application
User-defined events

31

The PETSc Programming Model

Goals
Portable, runs everywhere
Performance
Scalable parallelism

Approach
Distributed memory, shared-nothing
Requires only a compiler (single node or processor)
Access to data on remote machines through MPI

Can still exploit compiler discovered parallelism on each node

(e.g., SMP)
Hide within parallel objects the details of the communication
User orchestrates communication at a higher abstract level than
message passing

32

Collectivity
MPI communicators (MPI_Comm) specify collectivity
(processes involved in a computation)
All PETSc routines for creating solver and data objects are
collective with respect to a communicator, e.g.,
VecCreate(MPI_Comm comm, int m, int M, Vec *x)
Use PETSC_COMM_WORLD for all processes (like
MPI_COMM_WORLD, but allows the same code to work when
PETSc is started with a smaller set of processes)

Some operations are collective, while others are not, e.g.,

collective: VecNorm()
not collective: VecGetLocalSize()

If a sequence of collective routines is used, they must be called

in the same order by each process.

33

Hello World
#include petsc.h
int main( int argc, char *argv[] )
{
PetscInitialize(&argc,&argv,PETSC_NULL,PETSC_NULL);
PetscPrintf(PETSC_COMM_WORLD,Hello World\n);
PetscFinalize();
return 0;
}

34

Hello World (Fortran)

program main
integer ierr, rank
#include "include/finclude/petsc.h"
call PetscInitialize( PETSC_NULL_CHARACTER, ierr )
call MPI_Comm_rank( PETSC_COMM_WORLD, rank, ierr )
if (rank .eq. 0) then
print *, Hello World
endif
call PetscFinalize(ierr)
end

35

Fancier Hello World

#include petsc.h
int main( int argc, char *argv[] )
{
int rank;
PetscInitialize(&argc,&argv,PETSC_NULL,PETSC_NULL);
MPI_Comm_rank(PETSC_COMM_WORLD,&rank );
PetscSynchronizedPrintf(PETSC_COMM_WORLD,
Hello World from %d\n,rank);
PetscSynchronizedFlush(PETSC_COMM_WORLD);
PetscFinalize();
return 0;
}

36

Data Objects
Vectors (Vec)
focus: field data arising in nonlinear PDEs

Matrices (Mat)
focus: linear operators arising in nonlinear PDEs (i.e., Jacobians)
beginner
beginner

Object creation

beginner
beginner

Object assembly

intermediate
intermediate

Setting options

intermediate
intermediate

Viewing

advanced
advanced

User-defined customizations
tutorial
tutorialoutline:
outline:
data
dataobjects
objects

37

Vectors
What are PETSc vectors?
Fundamental objects for storing field solutions, right-hand
sides, etc.
Each process locally owns a subvector of contiguously
numbered global indices

proc 0
proc 1

Create vectors via

VecCreate(MPI_Comm,Vec *)

proc 2

MPI_Comm - processes that share the vector

VecSetSizes( Vec, int, int )

number of elements local to this process
or total number of elements

VecSetType(Vec,VecType)
Where VecType is

proc 3
proc 4

VEC_SEQ, VEC_MPI, or VEC_SHARED

VecSetFromOptions(Vec) lets you set the type at

runtime

beginner
beginner

data
dataobjects:
objects:
vectors
vectors

38

Creating a vector
Use PETSc to get value
from command line

Vec x;
int n;

PetscInitialize(&argc,&argv,(char*)0,help);
PetscOptionsGetInt(PETSC_NULL,"-n",&n,PETSC_NULL);

VecCreate(PETSC_COMM_WORLD,&x);
VecSetSizes(x,PETSC_DECIDE,n);
VecSetType(x,VEC_MPI);
VecSetFromOptions(x);

Global size
PETSc determines
local size

39

How Can We Use a PETSc Vector

In order to support the distributed memory shared
nothing model, as well as single processors and shared
memory systems, a PETSc vector is a handle to the real
vector
Allows the vector to be distributed across many processes
To access the elements of the vector, we cannot simply do
for (i=0; i<n; i++) v[i] = i;
We do not want to require that the programmer work only with the
local part of the vector; we want to permit operations, such as
setting an element of a vector, to be performed by any process.

Recall how data is stored in the distributed memory

programming model

40

Sample Parallel System

Architecture
Systems have an increasingly deep memory hierarchy (1, 2,
3, and more levels of cache)
Time to reference main memory 100s of cycles
Access to shared data requires synchronization
Better to ensure data is local and unshared when possible

SMP

CPUs

CPUs

Cache

Cache

Main Memory

Main Memory
Interconnect

...

42

Distributed Memory Model

Process 0

Address
Space

Process 1
A(10)

A(10)
Different Variables!
Integer A(10)

print *, A

Integer A(10)
do i=1,10
A(i) = i
enddo
...

This A is completely different from this one

43

Vector Assembly
A three step process
Each process tells PETSc what values to set or add to a vector component.
Once all values provided,
Begin communication between processes to ensure that values end up where
needed
(allow other operations, such as some computation, to proceed)
Complete the communication

VecSetValues(Vec,)

number of entries to insert/add

indices of entries
values to add
mode: [INSERT_VALUES,ADD_VALUES]

VecAssemblyBegin(Vec)
VecAssemblyEnd(Vec)
beginner
beginner

data
dataobjects:
objects:
vectors
vectors

44

Parallel Matrix and Vector

Assembly
Processes may generate any entries in vectors and
matrices
Entries need not be generated on the process on
which they ultimately will be stored
PETSc automatically moves data during the
assembly process if necessary

beginner
beginner

data
dataobjects:
objects:
vectors
vectorsand
and
matrices
matrices

45

One Way to Set the Elements of

A Vector
VecGetSize(x,&N); /* Global size */
MPI_Comm_rank( PETSC_COMM_WORLD, &rank
);
Vector index
if (rank == 0) {
for (i=0; i<N; i++)
VecSetValues
(x,1,&i,&i,INSERT_VALUES);Vector value
}
/* These two routines ensure that the data is
distributed to the other processes */
VecAssemblyBegin(x);
VecAssemblyEnd(x);

46

A Parallel Way to Set the

Elements of A Distributed Vector
VecGetOwnershipRange(x,&low,&high);
for (i=low; i<high; i++)
VecSetValues(x,1,&i,&i,INSERT_VALUES);
/* These two routines must be called (in case
some other process contributed a value owned
by another process) */
VecAssemblyBegin(x);
VecAssemblyEnd(x);

47

Selected Vector Operations

Function Name

Operation

VecAXPY(Scalar *a, Vec x, Vec y)

VecAYPX(Scalar *a, Vec x, Vec y)
VecWAXPY(Scalar *a, Vec x, Vec y, Vec w)
VecScale(Scalar *a, Vec x)
VecCopy(Vec x, Vec y)
VecPointwiseMult(Vec x, Vec y, Vec w)
VecMax(Vec x, int *idx, double *r)
VecShift(Scalar *s, Vec x)
VecAbs(Vec x)
VecNorm(Vec x, NormType type , double *r)

y = y + a*x
y = x + a*y
w = a*x + y
x = a*x
y=x
w_i = x_i *y_i
r = max x_i
x_i = s+x_i
x_i = |x_i |
r = ||x||

beginner
beginner

data
dataobjects:
objects:
vectors
vectors

48

Simple Example Programs

Location: petsc/src/sys/examples/tutorials/
E

ex2.c

- synchronized printing

Location: petsc/src/vec/examples/tutorials/
E

1
beginner
beginner

ex1.c, ex1f.F, ex1f90.F

ex3.c, ex3f.F
layout

1
- basic vector routines
- parallel vector

And many more examples ...

E - on-line exercise

data
dataobjects:
objects:
vectors
vectors

49

A Complete PETSc Program

#include petscvec.h
int main(int argc,char **argv)
{
Vec x;
int n = 20,ierr;
PetscTruth flg;
PetscScalar one = 1.0, dot;

PetscInitialize(&argc,&argv,0,0);
PetscOptionsGetInt(PETSC_NULL,"-n",&n,PETSC_NULL);
VecCreate(PETSC_COMM_WORLD,&x);
VecSetSizes(x,PETSC_DECIDE,n);
VecSetFromOptions(x);
VecSet(&one,x);
VecDot(x,x,&dot);
PetscPrintf(PETSC_COMM_WORLD,"Vector length %dn",(int)dot);
VecDestroy(x);
PetscFinalize();
return 0;

50

Working With Local Vectors

It is sometimes more efficient to directly access the storage for
the local part of a PETSc Vec.
E.g., for finite difference computations involving elements of the
vector

PETSc allows you to access the local storage with

VecGetArray( Vec, double *[] );

You must return the array to PETSc when you are done with it
VecRestoreArray( Vec, double *[] )

This allows PETSc to handle and data structure conversions

For most common uses, these routines are inexpensive and do not
involve a copy of the vector.

51

Example of VecGetArray
Vec
vec;
Double *avec;

VecCreate(PETSC_COMM_SELF,&vec);
VecSetSizes(vec,PETSC_DECIDE,n);
VecSetFromOptions(vec);
VecGetArray(vec,&avec);
/* compute with avec directly, e.g., */
PetscPrintf(PETSC_COMM_WORLD,
First element of local array of vec in each process is %f\n,
avec[0] );
VecRestoreArray(vec,&avec);

52

Matrices
What are PETSc matrices?
Fundamental objects for storing linear operators (e.g., Jacobians)

Create matrices via

MatCreate(,Mat *)
MPI_Comm - processes that share the matrix
number of local/global rows and columns

MatSetType(Mat,MatType)
where MatType is one of
default sparse AIJ: MPIAIJ, SEQAIJ
block sparse AIJ (for multi-component PDEs): MPIAIJ, SEQAIJ
symmetric block sparse AIJ: MPISBAIJ, SAEQSBAIJ
block diagonal: MPIBDIAG, SEQBDIAG
dense: MPIDENSE, SEQDENSE
matrix-free
etc.
MatSetFromOptions(Mat) lets you set the MatType at runtime.
beginner
beginner

data
dataobjects:
objects:
matrices
matrices

53

Matrices and Polymorphism

Single user interface, e.g.,
Matrix assembly
MatSetValues()

Matrix-vector multiplication
MatMult()

Matrix viewing
MatView()

Multiple underlying implementations

AIJ, block AIJ, symmetric block AIJ, block diagonal, dense, matrixfree, etc.

A matrix is defined by its properties, the operations that you

can perform with it.
Not by its data structures
beginner
beginner

data
dataobjects:
objects:
matrices
matrices

54

Matrix Assembly
Same form as for PETSc Vectors:
MatSetValues(Mat,)

number of rows to insert/add

indices of rows and columns
number of columns to insert/add
values to add
mode: [INSERT_VALUES,ADD_VALUES]

MatAssemblyBegin(Mat)
MatAssemblyEnd(Mat)
beginner
beginner

data
dataobjects:
objects:
matrices
matrices

55

Matrix Assembly Example

simple 3-point stencil for 1D discretization
Mat
A;
int
column[3], i;
double value[3];

MatCreate(PETSC_COMM_WORLD,
PETSC_DECIDE,PETSC_DECIDE,n,n,&A);

Choose the global

Size of the matrix

MatSetFromOptions(A);
/* mesh interior */
Let PETSc decide
value[0] = -1.0; value[1] = 2.0; value[2] = -1.0;
to allocate matrix
if (rank == 0) { /* Only one process creates matrix */
for (i=1; i<n-2; i++) {
across processes
column[0] = i-1; column[1] = i; column[2] = i+1;
MatSetValues(A,1,&i,3,column,value,INSERT_VALUES);
}
}
/* also must set boundary points (code for global row 0 and n-1 omitted) */
MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);

beginner
beginner

how

data
dataobjects:
objects:
matrices
matrices

56

Parallel Matrix Distribution

Each process locally owns a submatrix of contiguously
numbered global rows.
proc 0
proc 1
proc 2
proc 3
proc 4

} proc 3: locally owned rows

MatGetOwnershipRange(Mat A, int *rstart, int *rend)

rstart: first locally owned row of global matrix
rend -1: last locally owned row of global matrix
beginner
beginner

data
dataobjects:
objects:
matrices
matrices

57

Matrix Assembly Example With

Parallel Assembly
simple 3-point stencil for 1D discretization
Mat
A;
int
column[3], i, start, end,istart,iend;
double value[3];

MatCreate(PETSC_COMM_WORLD,
PETSC_DECIDE,PETSC_DECIDE,n,n,&A);

Choose the global

Size of the matrix

MatSetFromOptions(A);
MatGetOwnershipRange(A,&start,&end);
Let PETSc decide how
/* mesh interior */
to allocate matrix
istart = start; if (start == 0) istart = 1;
iend = end; if (iend == n-1) iend = n-2;
across processes
value[0] = -1.0; value[1] = 2.0; value[2] = -1.0;
for (i=istart; i<iend; i++) {
column[0] = i-1; column[1] = i; column[2] = i+1;
MatSetValues(A,1,&i,3,column,value,INSERT_VALUES);
}
/* also must set boundary points (code for global row 0 and n-1 omitted) */
MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);
data
MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);
dataobjects:
objects:

beginner
beginner

matrices
matrices

58

Why Are PETSc Matrices The

Way They Are?
No one data structure is appropriate for all problems
Blocked and diagonal formats provide significant performance benefits
PETSc provides a large selection of formats and makes it (relatively) easy
to extend PETSc by adding new data structures

Matrix assembly is difficult enough without being forced to worry

about data partitioning
PETSc provides parallel assembly routines
Achieving high performance still requires making most operations local to
a process but programs can be incrementally developed.

Matrix decomposition by consecutive rows across processes is simple

and makes it easier to work with other codes.
For applications with other ordering needs, PETSc provides Application
Orderings (AO), described later.

59

Blocking: Performance Benefits

More issues discussed in full tutorials available via PETSc web site.
100

3D compressible
Euler code
Block size 5
IBM Power2

80
MFlop
/sec

60
40
20
0

Basic

Matrix-vector products
Triangular solves
beginner
beginner

data
dataobjects:
objects:
matrices
matrices

60

Viewers
beginner
beginner

beginner
beginner
intermediate
intermediate

Printing information about solver and

data objects
Visualization of field and matrix data
Binary output of vector and matrix data

tutorial
tutorialoutline:
outline:
viewers
viewers

61

Viewer Concepts
Information about PETSc objects
runtime choices for solvers, nonzero info for matrices, etc.

Data for later use in restarts or external tools

vector fields, matrix contents
various formats (ASCII, binary)

Visualization
simple x-window graphics
vector fields
matrix sparsity structure

beginner
beginner

viewers
viewers

62

Viewing Vector Fields

VecView(Vec x,PetscViewer v); Solution components,

Default viewers
ASCII (sequential):

using runtime option

-snes_vecmonitor

PETSC_VIEWER_STDOUT_SELF

ASCII (parallel):

PETSC_VIEWER_STDOUT_WORLD

X-windows:

PETSC_VIEWER_DRAW_WORLD

Default ASCII formats

PETSC_VIEWER_ASCII_DEFAULT
PETSC_VIEWER_ASCII_MATLAB
PETSC_VIEWER_ASCII_COMMON
PETSC_VIEWER_ASCII_INFO
etc.

velocity: u

velocity: v

vorticity: temperature: T
beginner
beginner

viewers
viewers

63

Viewing Matrix Data

MatView(Mat A, PetscViewer
v);
Runtime options available after
matrix assembly
-mat_view_info
info about matrix assembly

-mat_view_draw
sparsity structure

-mat_view
data in ASCII

etc.
beginner
beginner

viewers
viewers

64

More Viewer Info

Viewer creation
ViewerASCIIOpen()
ViewerDrawOpen()
ViewerSocketOpen()

Binary I/O of vectors and matrices

output: ViewerBinaryOpen()
input: VecLoad(), MatLoad()

intermediate
intermediate

viewers
viewers

65

Running PETSc Programs

The easiest approach is to use the PETSc Makefile provided with the examples

Make sure that you have your PETSc environment setup correctly:

setenv PETSC_DIR <directory with PETSc installed>

setenv PETSC_ARCH <name of PETSc architecture>

Copy an example file and the makefile from the directory to your local
directory:

cp ${PETSC_DIR}/src/vec/examples/tutorials/ex1.c .
cp ${PETSC_DIR}/src/vec/examples/tutorials/makefile .

Make and run the program:

Ensures that all of the correct libraries are used and that the correct compilation
options are used.

make BOPT=g ex1

mpirun np 2 ex1

Edit the example to explore PETSc

66

End of Day 1

67

Solvers: Usage Concepts

Solver Classes
Linear (SLES)
Nonlinear (SNES)
Timestepping (TS)

important
importantconcepts
concepts

Usage Concepts

Context variables
Solver options
Callback routines
Customization

tutorial
tutorialoutline:
outline:
solvers
solvers

68

Linear PDE Solution

Main Routine

PETSc

Linear Solvers (SLES)

Solve
Ax = b

Application
Initialization

PC

Evaluation of A and b

User code
beginner
beginner

KSP

PostProcessing

PETSc code
solvers:
solvers:
linear
linear

69

Linear Solvers
Goal: Support the solution of linear systems,
Ax=b,
particularly for sparse, parallel problems arising
within PDE-based models
User provides:
Code to evaluate A, b

beginner
beginner

solvers:
solvers:
linear
linear

70

Sample Linear Application:

Exterior Helmholtz Problem
Solution Components

2u k 2u 0
lim r

1/ 2

beginner
beginner

Real

iku 0

Collaborators: H. M. Atassi, D. E. Keyes,

L. C. McInnes, R. Susan-Resiga

Imaginary

solvers:
solvers:
linear
linear

71

Helmholtz: The Linear System

Logically regular grid, parallelized with DAs

Finite element discretization (bilinear quads)
Nonreflecting exterior BC (via DtN map)
Matrix sparsity structure (option: -mat_view_draw)

Natural ordering
beginner
beginner

Close-up

Nested dissection ordering

solvers:
solvers:
linear
linear

72

Linear Solvers (SLES)

SLES: Scalable Linear Equations Solvers
beginner
beginner

Application code interface

beginner
beginner

Choosing the solver

beginner
beginner

Setting algorithmic options

beginner
beginner

Viewing the solver

intermediate
intermediate

Determining and monitoring convergence

intermediate
intermediate

Providing a different preconditioner matrix

advanced
advanced

Matrix-free solvers

advanced
advanced

User-defined customizations

tutorial
tutorialoutline:
outline:
solvers:
solvers:
linear
linear

73

Objects In PETSc
How should a matrix be described in a program?
Old way:
Dense matrix:
double precision A(10,10)
Sparse matrix:
integer ia(11), ja(max_nz)
double precision a(max_nx)

New way:
Mat M

Hides the choice of data structure

Of course, the library still needs to represent the matrix with some choice of data
structure, but this is an implementation detail

Benefit
Programs become independent of particular choices of data structure, making it
easier to modify and adapt

74

Operations In PETSc
How should operations like solve linear system be described in a
program?
Old way:
mpiaijgmres( ia, ja, a, comm, x, b, nlocal, nglobal, ndir, orthmethod, convtol,
&its )

New way:
SLESSolve( sles, b, x, &its )

Hides the choice of algorithm

Algorithms are to operations as data structures are to objects

Benefit
Programs become independent of particular choices of algorithm, making
it easier to explore algorithmic choices and to adapt to new methods.

In PETSc, operations have their own handle, called a context

variable

75

Context Variables
Are the key to solver organization
Contain the complete state of an algorithm,
including
parameters (e.g., convergence tolerance)
functions that run the algorithm (e.g.,
convergence monitoring routine)
information about the current state (e.g., iteration
number)
beginner
beginner

solvers:
solvers:
linear
linear

76

Creating the SLES Context

C/C++ version
ierr = SLESCreate(PETSC_COMM_WORLD,&sles);

Fortran version
call SLESCreate(PETSC_COMM_WORLD,sles,ierr)

Provides an identical user interface for all linear

solvers
uniprocess and parallel
real and complex numbers

beginner
beginner

solvers:
solvers:
linear
linear

77

SLES Structure
Each SLES object actually contains two other
objects:
KSP Krylov Space Method
The iterative method
The context contains information on method parameters (e.g.,
GMRES search directions), work spaces, etc

PC Preconditioners
Knows how to apply a preconditioner
The context contains information on the preconditioner, such
as what routine to call to apply it

78

Linear Solvers in PETSc 2.0

Krylov Methods (KSP)

Conjugate Gradient
GMRES
CG-Squared
Bi-CG-stab
Transpose-free QMR
etc.

beginner
beginner

Preconditioners (PC)
Block Jacobi
Overlapping Additive
Schwarz
ICC, ILU via BlockSolve95
ILU(k), LU (direct solve,
sequential only)
Arbitrary matrix
etc.
solvers:
solvers:
linear
linear

79

Basic Linear Solver Code (C/C++)

SLES sles;
Mat
A;
Vec
x, b;
int
n, its;

/* linear solver context */

/* matrix */
/* solution, RHS vectors */
/* problem dimension, number of iterations */

MatCreate(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE ,n,n,&A);
MatSetFromOptions(A);
/* (code to assemble matrix not shown) */
Indicate whether the preconditioner
VecCreate(PETSC_COMM_WORLD,&x);
has the same nonzero pattern as the
VecSetSizes(x,PETSC_DECIDE, n);
matrix each time a system is
VecSetFromOptions(x);
solved. This default works with all
VecDuplicate(x,&b);
preconditioners. Other values
/* (code to assemble RHS vector not shown)*/
(e.g.,
SAME_NONZERO_PATTERN)
can be used for particular
SLESCreate(PETSC_COMM_WORLD,&sles);
SLESSetOperators(sles,A,A,DIFFERENT_NONZERO_PATTERN); preconditioners. Ignored when
solving only one system
SLESSetFromOptions(sles);

SLESSolve(sles,b,x,&its);
SLESDestroy(sles);
beginner
beginner

solvers:
solvers:
linear
linear

80

Basic Linear Solver Code (Fortran)

SLES
Mat
Vec
integer
call
call
call
call
call
call

sles
A
x, b
n, its, ierr

MatCreate( PETSC_COMM_WORLD,PETSC_DECIDE,n,n,A,ierr )
MatSetFromOptions( A, ierr )
VecCreate( PETSC_COMM_WORLD,x,ierr )
VecSetSizes( x, PETSC_DECIDE, n, ierr )
VecSetFromOptions( x, ierr )
VecDuplicate( x,b,ierr )

then assemble matrix and right-hand-side vector

call
call
call
call
call

beginner
beginner

SLESCreate(PETSC_COMM_WORLD,sles,ierr)
SLESSetOperators(sles,A,A,DIFFERENT_NONZERO_PATTERN,ierr)
SLESSetFromOptions(sles,ierr)
SLESSolve(sles,b,x,its,ierr)
solvers:
SLESDestroy(sles,ierr)
solvers:
linear
linear

81

Customization Options
Command Line Interface
Applies same rule to all queries via a database
Enables the user to have complete control at runtime,
with no extra coding

Procedural Interface
Provides a great deal of control on a usage-by-usage
basis inside a single code
Gives full flexibility inside an application

beginner
beginner

solvers:
solvers:
linear
linear

82

Setting Solver Options at Runtime

-ksp_type [cg,gmres,bcgs,tfqmr,]
-pc_type [lu,ilu,jacobi,sor,asm,]

-ksp_max_it <max_iters>
-ksp_gmres_restart <restart>
-pc_asm_overlap <overlap>
-pc_asm_type
[basic,restrict,interpolate,none]
etc ...
1

beginner
beginner intermediate
intermediate

solvers:
solvers:
linear
linear

83

Linear Solvers: Monitoring

Convergence
-ksp_monitor
-ksp_xmonitor

- Prints preconditioned residual norm

-ksp_truemonitor
-ksp_xtruemonitor

- Prints true residual norm || b-Ax ||

- Plots preconditioned residual norm

- Plots true residual norm || b-Ax ||

User-defined monitors, using callbacks

beginner
beginner intermediate
intermediate advanced
advanced

solvers:
solvers:
linear
linear

84

Setting Solver Options within Code

SLESGetKSP(SLES sles,KSP *ksp)
KSPSetType(KSP ksp,KSPType type)
KSPSetTolerances(KSP ksp,PetscReal rtol,
PetscReal atol,PetscReal dtol, int maxits)
etc....

SLESGetPC(SLES sles,PC *pc)

PCSetType(PC pc,PCType)
PCASMSetOverlap(PC pc,int overlap)
etc....
beginner
beginner

solvers:
solvers:
linear
linear

85

Recursion: Specifying Solvers for

Schwarz Preconditioner Blocks
Specify SLES solvers and options with -sub prefix, e.g.,
Full or incomplete factorization
-sub_pc_type lu
-sub_pc_type ilu -sub_pc_ilu_levels <levels>

Can also use inner Krylov iterations, e.g.,

-sub_ksp_type gmres -sub_ksp_rtol <rtol>
-sub_ksp_max_it <maxit>

beginner
beginner

solvers:
solvers:linear:
linear:
preconditioners
preconditioners

86

Helmholtz: Scalability
128x512 grid, wave number = 13, IBM SP
GMRES(30)/Restricted Additive Schwarz
1 block per proc, 1-cell overlap, ILU(1) subdomain solver
Procs
1
2
4
8
16
32
beginner
beginner

Iterations
221
222
224
228
229
230

Time (Sec)
163.01
81.06
37.36
19.49
10.85
6.37

Speedup
-

2.0
4.4
8.4
15.0
25.6
solvers:
solvers:
linear
linear

87

SLES: Review of Basic Usage

SLESCreate( )
- Create SLES context
SLESSetOperators( ) - Set linear operators
SLESSetFromOptions( ) - Set runtime solver options
for [SLES, KSP,PC]
SLESSolve( )
- Run linear solver
SLESView( )
- View solver options
actually used at runtime
(alternative: -sles_view)
SLESDestroy( )
- Destroy solver

beginner
beginner

solvers:
solvers:
linear
linear

88

SLES: Review of Selected Preconditioner

Options
Functionality

Procedural Interface

Runtime Option

Set preconditioner type PCSetType( )

-pc_type [lu,ilu,jacobi,
sor,asm,]

Set level of fill for ILU

Set SOR iterations
Set SOR parameter
Set additive Schwarz
variant
Set subdomain solver
options

PCILUSetLevels( )
PCSORSetIterations( )
PCSORSetOmega( )
PCASMSetType( )

-pc_ilu_levels <levels>
2
-pc_sor_its <its>
-pc_sor_omega <omega>
-pc_asm_type [basic,
restrict,interpolate,none]

PCGetSubSLES( )

-sub_pc_type <pctype>
-sub_ksp_type
<ksptype>
-sub_ksp_rtol <rtol>

beginner
beginner intermediate
intermediate

And many more options...

solvers:
solvers:linear:
linear:
preconditioners
preconditioners

89

SLES: Review of Selected Krylov Method

Options
Functionality

Procedural Interface

Runtime Option

Set Krylov method

KSPSetType( )

-ksp_type [cg,gmres,bcgs,
tfqmr,cgs,]

Set monitoring
routine

KSPSetMonitor( )

-ksp_monitor, ksp_xmonitor,
-ksp_truemonitor, ksp_xtruemonitor

KSPSetTolerances( )
-ksp_rtol <rt> -ksp_atol <at>
Set convergence
-ksp_max_its <its>
tolerances
KSPGMRESSetRestart( ) -ksp_gmres_restart <restart>
Set GMRES restart
parameter
-ksp_unmodifiedgramschmidt
Set orthogonalization KSPGMRESSet
Orthogonalization( )
-ksp_irorthog
routine for GMRES

beginner
beginner intermediate
intermediate

And many more options...

solvers:
solvers:linear:
linear:
Krylov
methods
Krylov methods

90

Why Polymorphism?
Programs become independent of the choice of
algorithm
Consider the question:
What is the best combination of iterative method and
preconditioner for my problem?

How can you answer this experimentally?

Old way:
Edit code. Make. Run. Edit code. Make. Run. Debug. Edit.

New way:

91

SLES: Runtime Script Example

intermediate
intermediate

solvers:
solvers:
linear
linear

92

Viewing SLES Runtime Options

intermediate
intermediate

solvers:
solvers:
linear
linear

93

Providing Different Matrices to

Define Linear System and
Preconditioner
Solve Ax=b
-1
-1
-1
Precondition via: M L A M R (M R x) = M L b

Krylov method: Use A for matrix-vector products

Build preconditioner using either
A - matrix that defines linear system
or P - a different matrix (cheaper to assemble)

SLESSetOperators(SLES sles,

advanced
advanced

Mat A,
Mat P,
MatStructure flag)

solvers:
solvers:
linear
linear

94

Matrix-Free Solvers
Use shell matrix data structure
MatCreateShell(, Mat *mfctx)

Define operations for use by Krylov methods

MatShellSetOperation(Mat mfctx,
MatOperation MATOP_MULT,
(void *) int (UserMult)(Mat,Vec,Vec))

Names of matrix operations defined in

petsc/include/mat.h
Some defaults provided for nonlinear solver usage
advanced
advanced

solvers:
solvers:
linear
linear

95

User-defined Customizations
Restricting the available solvers
Customize PCRegisterAll( ), KSPRegisterAll( )

Adding user-defined preconditioners via

PCShell preconditioner type

Adding preconditioner and Krylov methods in library style

Method registration via PCRegister( ), KSPRegister( )

Heavily commented example implementations

Jacobi preconditioner: petsc/src/sles/pc/impls/jacobi.c
Conjugate gradient: petsc/src/sles/ksp/impls/cg/cg.c
3

advanced
advanced developer
developer

solvers:
solvers:
linear
linear

96

SLES: Example Programs

Location: petsc/src/sles/examples/tutorials/
ex1.c, ex1f.F - basic uniprocess codes
E
ex23.c
- basic parallel code
ex11.c
- using complex numbers

ex4.c

ex9.c
E systems
ex22.c
ex15.c
1

- using different linear system and

preconditioner matrices
- repeatedly solving different linear
- 3D Laplacian using multigrid

- setting a user-defined preconditioner

And many more examples ...
3

beginner
beginner intermediate
intermediate advanced
advanced

E - on-line exercise

solvers:
solvers:
linear
linear

97

Now What?
If you have a running program, are you done?
Yes, if your program answers your question.
No, if you now need to run much larger problems or many more
problems.

How can you tune an application for performance?

PETSc approach:
Debug the application by focusing on the mathematically operations.
The parallel matrix assembly operations are an example.
Once the code works,
Understand the performance
Identify performance problems
Use special PETSc routines and other techniques to optimize only the
code that is underperforming

98

Profiling and Performance Tuning

Profiling:
beginner
beginner
intermediate
intermediate
intermediate
intermediate

Integrated profiling using

-log_summary
User-defined events
Profiling by stages of an application

Performance Tuning:
intermediate
intermediate

Matrix optimizations

intermediate
intermediate

Application optimizations

advanced
advanced

Algorithmic tuning

tutorial
tutorialoutline:
outline:
profiling
profilingand
and
performance
performancetuning
tuning

99

Profiling
Integrated monitoring of

time
floating-point performance
memory usage
communication

All PETSc events are logged if PETSc was compiled with

-DPETSC_LOG (default); can also profile application code
segments
Print summary data with option: -log_summary
Print redundant information from PETSc routines: -log_info
Print the trace of the functions called: -log_trace
beginner
beginner

profiling
profilingand
and
performance
performancetuning
tuning

100

User-defined Events
int USER_EVENT;
int user_event_flops
PetscLogEventRegister(&USER_EVENT,User
event name, eventColor);
PetscLogEventBegin(USER_EVENT,0,0,0,0);
[ code to monitor]
PetscLogFlops(user_evnet_flops);
PetscLogEvent End(USER_EVENT,0,0,0,0);

intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

101

Nonlinear Solvers (SNES)

SNES: Scalable Nonlinear Equations Solvers
beginner
beginner

Application code interface

beginner
beginner

Choosing the solver

beginner
beginner

Setting algorithmic options

beginner
beginner

Viewing the solver

intermediate
intermediate

Determining and monitoring convergence

advanced
advanced

Matrix-free solvers

advanced
advanced

User-defined customizations
tutorial
tutorialoutline:
outline:
solvers:
solvers:
nonlinear
nonlinear

102

Nonlinear PDE Solution

Main Routine

Nonlinear Solvers (SNES)

Solve
F(u) = 0

Linear Solvers (SLES)

PC
Application
Initialization

PETSc

KSP
Function
Evaluation
User code

beginner
beginner

Jacobian
Evaluation
PETSc code

PostProcessing
solvers:
solvers:
nonlinear
nonlinear

103

Nonlinear Solvers
Goal: For problems arising from PDEs,
support the general solution of F(u) = 0
User provides:
Code to evaluate F(u)
Code to evaluate Jacobian of F(u) (optional)
or use sparse finite difference approximation
or use automatic differentiation
AD support via collaboration with P. Hovland and B. Norris
Coming in next PETSc release via automated interface to
ADIFOR and ADIC (see http://www.mcs.anl.gov/autodiff)

beginner
beginner

solvers:
solvers:
nonlinear
nonlinear

104

Nonlinear Solvers (SNES)

Newton-based methods, including

Line search strategies

Trust region approaches
Pseudo-transient continuation
Matrix-free variants

User can customize all phases of the solution process

beginner
beginner

solvers:
solvers:
nonlinear
nonlinear

105

Sample Nonlinear Application:

Driven Cavity Problem
Velocity-vorticity
formulation
Flow driven by lid and/or
bouyancy
Logically regular grid,
parallelized with DAs
Finite difference
discretization
source code:
petsc/src/snes/examples/tutorials/ex19.c

beginner
beginner

Solution Components

velocity: u

vorticity:

velocity: v

temperature: T

Application code author: D. E. Keyes

solvers:
solvers:
nonlinear
nonlinear

106

Basic Nonlinear Solver Code (C/C++)

snes;
J;
x, F;
n, its;
cationCtx usercontext;

/* nonlinear solver context */

/* Jacobian matrix */
/* solution, residual vectors */
/* problem dimension, number of iterations */
/* user-defined application context */

eate(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE ,n,n,&J);
etFromOptions(J);
eate(PETSC_COMM_WORLD,&x);
tSizes(x,PETSC_DECIDE,n);
tFromOptions(x);
uplicate(x,&F);

Create(PETSC_COMM_WORLD,&snes);
SetFunction(snes,F,EvaluateFunction,usercontext);
SetJacobian(snes,J,J,EvaluateJacobian,usercontext);
SetFromOptions(snes);
Solve(snes,x,&its);
Destroy(snes);
beginner
beginner

solvers:
solvers:
nonlinear
nonlinear

107

Basic Nonlinear Solver Code (Fortran)

SNES snes
Mat
J
Vec
x, F
int
n, its
...

call MatCreate(PETSC_COMM_WORLD,PETSC_DECIDE,
PETSC_DECIDE,n,n,J,ierr)
call MatSetFromOptions( J, ierr )
call VecCreate(PETSC_COMM_WORLD,x,ierr)
call VecSetSizes(x,PETSC_DECIDE,n,ierr)
call VecSetFromOptions(x,ierr)
call VecDuplicate(x,F,ierr)
call
call
call
call
call
call
beginner
beginner

SNESCreate(PETSC_COMM_WORLD, snes,ierr)
SNESSetFunction(snes,F,EvaluateFunction,PETSC_NULL,ierr)
SNESSetJacobian(snes,J,J,EvaluateJacobian,PETSC_NULL,ierr)
SNESSetFromOptions(snes,ierr)
SNESSolve(snes,x,its,ierr)
SNESDestroy(snes,ierr)

solvers:
solvers:
nonlinear
nonlinear

108

Solvers Based on Callbacks

User provides routines to perform actions that the library
requires. For example,
SNESSetFunction(SNES,...)

important
importantconcept
concept
uservector - vector to store function values
userfunction - name of the users function
usercontext - pointer to private data for the users function

Now, whenever the library needs to evaluate the users

nonlinear function, the solver may call the application code
directly with its own local state.
usercontext: serves as an application context object.
Data are handled through such opaque objects; the library
never sees irrelevant application data.
beginner
beginner

solvers:
solvers:
nonlinear
nonlinear

109

Sample Application Context:

Driven Cavity Problem
typedef struct {
/* - - - - - - - - - - - - - - - - basic application data - - - - - - - - - - - - - - - - */
double
lid_velocity, prandtl, grashof;
/* problem
parameters */
int
mx, my;
/* discretization parameters
*/
int
mc;
/* number of DoF
per node */
int
draw_contours;
/* flag - drawing contours */
/* - - - - - - - - - - - - - - - - - - - - parallel data - - - - - - - - - - - - - - - - - - - - */
MPI_Comm comm;
/* communicator */
DA
da;
/* distributed array
*/
Vec
localF, localX;
/* local ghosted vectors */
} AppCtx;
solvers:
solvers:
nonlinear
beginner
nonlinear
beginner

110

Sample Function Evaluation Code:

Driven Cavity Problem
UserComputeFunction(SNES snes, Vec X, Vec F, void *ptr)
{
AppCtx *user = (AppCtx *) ptr;
/* user-defined application context */
int
istart, iend, jstart, jend;
/* local starting and ending grid
points */
Scalar
*f;
/* local vector data */
.
/* (Code to communicate nonlocal ghost point data not shown) */
VecGetArray( F, &f );
/* (Code to compute local function components; insert into f[ ] shown on
next slide) */
VecRestoreArray( F, &f );
.
return 0;
}

beginner
beginner

solvers:
solvers:
nonlinear
nonlinear

111

Sample Local Computational

Loops:
Driven Cavity Problem
#define U(i)
4*(i)
The PDEs 4 components
#define V(i)
4*(i)+1
(U,V,Omega,Temp) are interleaved in the
#define Omega(i) 4*(i)+2
unknown vector.
#define Temp(i)
4*(i)+3
#define statements provide easy access to
.
each component.
for ( j = jstart; j<jend; j++ ) {
row = (j - gys) * gxm + istart - gxs - 1;
for ( i = istart; i<iend; i++ ) {
row++; u = x[U(row)];
uxx = (two * u - x[ U (row-1)] - x [U (row+1) ] ) * hydhx;
uyy = (two * u - x[ U (row-gxm)] - x [ U (row+gxm) ] ) * hxdhy;
f [U(row)] = uxx + uyy
p5 * (x [ (Omega (row+gxm)] - x [Omega (row-gxm)] ) *
hx;
}}
.
beginner
beginner

solvers:
solvers:
nonlinear
nonlinear

112

Finite Difference Jacobian

Computations
Compute and explicitly store Jacobian via 1st-order FD
Dense: -snes_fd, SNESDefaultComputeJacobian()
Sparse via colorings: MatFDColoringCreate(),
SNESDefaultComputeJacobianColor()

Matrix-free Newton-Krylov via 1st-order FD, no

preconditioning unless specifically set by user
-snes_mf

Matrix-free Newton-Krylov via 1st-order FD, user-defined

preconditioning matrix
-snes_mf_operator
beginner
beginner

solvers:
solvers:
nonlinear
nonlinear

113

Uniform access to all linear and

nonlinear solvers
-ksp_type [cg,gmres,bcgs,tfqmr,]
-pc_type [lu,ilu,jacobi,sor,asm,]
-snes_type [ls,tr,]

-snes_line_search <line search method>

-sles_ls <parameters>
2
-snes_convergence <tolerance>
etc...
2

beginner
beginner intermediate
intermediate

solvers:
solvers:
nonlinear
nonlinear

114

Customization via Callbacks:

Setting a user-defined line search routine
SNESSetLineSearch(SNES snes,int(*ls)(),void
*lsctx)

where available line search routines ls( ) include:

2
SNESCubicLineSearch( ) - cubic line search
SNESQuadraticLineSearch( ) - quadratic line search
SNESNoLineSearch( ) - full Newton step
SNESNoLineSearchNoNorms( ) - full Newton step but calculates
no norms (faster in parallel, useful when using a fixed number of
Newton iterations instead of usual convergence testing)

YourOwnFavoriteLineSearchRoutine( )
2

intermediate
intermediate advanced
advanced

solvers:
solvers:
nonlinear
nonlinear

115

SNES: Review of Basic Usage

SNESCreate( ) - Create SNES context
SNESSetFunction( )
- Set function eval.
routine
SNESSetJacobian( )
- Set Jacobian eval.
routine
SNESSetFromOptions( )
- Set runtime solver options
for [SNES,SLES, KSP,PC]
SNESSolve( )
- Run nonlinear solver
SNESView( )
- View solver options
actually
used at runtime
(alternative: -snes_view)
SNESDestroy( )
- Destroy solver
beginner
beginner

solvers:
solvers:
nonlinear
nonlinear

116

SNES: Review of Selected Options

Functionality

Procedural Interface

Runtime Option

Set nonlinear solver

Set monitoring
routine

SNESSetType( )
SNESSetMonitor( )

-snes_type [ls,tr,umls,umtr,]
-snes_monitor
1
snes_xmonitor,

Set convergence
tolerances
Set line search routine
View solver options
Set linear solver options

SNESSetTolerances( )

-snes_rtol <rt> -snes_atol <at>

-snes _max_its <its>
-snes_eq_ls [cubic,quadratic,]
-snes_view
-ksp_type <ksptype>
2
-ksp_rtol <krt>
-pc_type <pctype>

beginner
beginner intermediate
intermediate

SNESSetLineSearch( )
SNESView( )
SNESGetSLES( )
SLESGetKSP( )
SLESGetPC( )

And many more options...

solvers:
solvers:
nonlinear
nonlinear

117

SNES: Example Programs

Location: petsc/src/snes/examples/tutorials/
ex1.c, ex1f.F
ex2.c
E

ex5.c, ex5f.F, ex5f90.F

per node)

1
- basic uniprocess codes
- uniprocess nonlinear PDE
(1 DoF per node)
- parallel nonlinear PDE (1 DoF

ex18.c

ex19.c
with

- parallel radiative transport problem with

2
multigrid
- parallel driven cavity problem
multigrid

And many more examples ...

1

beginner
beginner intermediate
intermediate

E - on-line exercise

solvers:
solvers:
nonlinear
nonlinear

118

Timestepping Solvers (TS)

(and ODE Integrators)
beginner
beginner

Application code interface

beginner
beginner

Choosing the solver

beginner
beginner

Setting algorithmic options

beginner
beginner

Viewing the solver

intermediate
intermediate
advanced
advanced

Determining and monitoring convergence

User-defined customizations
tutorial
tutorialoutline:
outline:
solvers:
solvers:
timestepping
timestepping

119

Time-Dependent PDE Solution

Main Routine

Timestepping Solvers (TS)

Nonlinear Solvers (SNES)

PETSc

Linear Solvers (SLES)

PC
Application
Initialization

Solve
U t = F(U,Ux,Uxx)

KSP
Function
Evaluation
User code

beginner
beginner

Jacobian
Evaluation
PETSc code

PostProcessing
solvers:
solvers:
timestepping
timestepping

120

Timestepping Solvers
Goal: Support the (real and pseudo) time
evolution of PDE systems
Ut = F(U,Ux,Uxx,t)
User provides:
Code to evaluate F(U,Ux,Uxx,t)
Code to evaluate Jacobian of F(U,Ux,Uxx,t)
or use sparse finite difference approximation
or use automatic differentiation (coming soon!)

beginner
beginner

solvers:
solvers:
timestepping
timestepping

Sample Timestepping Application:

Burgers Equation

Ut= U Ux + Uxx
U(0,x) = sin(2x)
U(t,0) = U(t,1)

beginner
beginner

solvers:
solvers:
timestepping
timestepping

122

Actual Local Function Code

Ut = F(t,U) = Ui (Ui+1 - U i-1)/(2h) +

(Ui+1 - 2Ui + U i-1)/(h*h)

Do 10, i=1,localsize
F(i) = (.5/h)*u(i)*(u(i+1)-u(i-1)) +
(e/(h*h))*(u(i+1) - 2.0*u(i) +
u(i-1))
10 continue
beginner
beginner

solvers:
solvers:
timestepping
timestepping

123

Timestepping Solvers

Euler
Backward Euler
Pseudo-transient continuation
Interface to PVODE, a sophisticated parallel ODE
solver package by Hindmarsh et al. of LLNL
Adams
BDF

beginner
beginner

solvers:
solvers:
timestepping
timestepping

124

Timestepping Solvers
Allow full access to all of the PETSc
nonlinear solvers
linear solvers
distributed arrays, matrix assembly tools, etc.

User can customize all phases of the solution

process

beginner
beginner

solvers:
solvers:
timestepping
timestepping

125

TS: Review of Basic Usage

TSCreate( )
TSSetRHSFunction( )
TSSetRHSJacobian( )
TSSetFromOptions( )

TSSolve( )
TSView( )
TSDestroy( )

beginner
beginner

- Create TS context
- Set function eval. routine
- Set Jacobian eval. routine
- Set runtime solver options
for [TS,SNES,SLES,KSP,PC]
- Run timestepping solver
- View solver options
actually used at runtime
(alternative: -ts_view)
- Destroy solver
solvers:
solvers:
timestepping
timestepping

126

TS: Review of Selected Options

Functionality

Procedural
Interface

Set timestepping solver TSSetType( )

TSSetMonitor()
Set monitoring
routine
Set timestep duration

TSSetDuration ( )

TSView( )
View solver options
Set timestepping solver TSGetSNES( )
SNESGetSLES( )
options
SLESGetKSP( )
SLESGetPC( )

beginner
beginner intermediate
intermediate

Runtime Option
-ts_ type [euler,beuler,pseudo,]
1
-ts_monitor
-ts_xmonitor,
-ts_max_steps <maxsteps>
-ts_max_time <maxtime>
-ts_view
-snes_monitor -snes_rtol <rt>
-ksp_type <ksptype>
2
-ksp_rtol <rt>
-pc_type <pctype>

And many more options...

solvers:
solvers:
timestepping
timestepping

127

TS: Example Programs

Location: petsc/src/ts/examples/tutorials/
ex1.c, ex1f.F
E

ex2.c, ex2f.F

ex3.c
ex4.c

- basic uniprocess codes (time1

dependent nonlinear PDE)
- basic parallel codes (time-dependent
nonlinear PDE)
- uniprocess heat equation
- parallel heat equation

And many more examples ...

1

beginner
beginner intermediate
intermediate

E - on-line exercise

solvers:
solvers:
timestepping
timestepping

128

Mesh Definitions:

For Our Purposes

Structured: Determine neighbor relationships

purely from logical I, J, K coordinates
Semi-Structured: In well-defined regions,
determine neighbor relationships purely from
logical I, J, K coordinates
Unstructured: Do not explicitly use logical I, J,
K coordinates

beginner
beginner

data
datalayout
layout

129

Structured Meshes

PETSc support provided via DA objects

beginner
beginner

data
datalayout
layout

130

Unstructured Meshes
One is always free to
manage the mesh data as
if unstructured
PETSc does not currently
have high-level tools for
managing such meshes
PETSc can manage
unstructured meshes
though lower-level
VecScatter utilities
beginner
beginner

data
datalayout
layout

131

Semi-Structured Meshes

No explicit PETSc support

OVERTURE-PETSc for composite meshes
SAMRAI-PETSc for AMR

beginner
beginner

data
datalayout
layout

132

Parallel Data Layout and Ghost

Values: Usage Concepts
Managing field data layout and required ghost values is the key
to high performance of most PDE-based parallel programs.

Mesh Types
Structured
DA objects

Unstructured
VecScatter objects

important
importantconcepts
concepts

Usage Concepts

Geometric data
Data structure creation
Ghost point updates
Local numerical computation

tutorial
tutorialoutline:
outline:
data
layout
data layout

133

Ghost Values
Local node

Ghost
node

Ghost values: To evaluate a local function f(x) , each process

requires its local portion of the vector x as well as its ghost values
or bordering portions of x that are owned by neighboring processes.
beginner
beginner

data
datalayout
layout

Communication and Physical134

Discretization
Communication
Geometric
Data

Data Structure Ghost Point

Creation
Data Structures

stencil
[implicit]

DACreate( )

DA
AO

Ghost Point
Updates

DAGlobalToLocal( )

structured meshes
elements
edges
vertices

VecScatter
VecScatterCreate( ) AO

Loops over
I,J,K
indices

VecScatter( )

unstructured meshes

Local
Numerical
Computation

Loops over
entities

beginner
beginner intermediate
intermediate

data
datalayout
layout

135

DA: Parallel Data Layout and

Ghost Values for Structured
Meshes
beginner
beginner

Local and global indices

beginner
beginner

Local and global vectors

beginner
beginner

DA creation

intermediate
intermediate

Ghost point updates

intermediate
intermediate

Viewing
tutorial
tutorialoutline:
outline:
data
layout:
data layout:
distributed
distributedarrays
arrays

136

Communication and Physical

Discretization:
Structured Meshes
Communication
Geometric
Data
stencil
[implicit]

Data Structure Ghost Point

Creation
Data Structures
DACreate( )

DA
AO

Ghost Point
Updates

DAGlobalToLocal( )

structured meshes

beginner
beginner

Local
Numerical
Computation
Loops over
I,J,K
indices

data
datalayout:
layout:
distributed
distributedarrays
arrays

137

Global and Local

Representations
Local node
Ghost
node

9
0

Global: each process stores a unique

local set of vertices (and each vertex
is owned by exactly one process)
beginner
beginner

Local: each process stores a unique

local set of vertices as well as ghost
nodes from neighboring processes
data
datalayout:
layout:
distributed
distributedarrays
arrays

138

Global and Local Representations (cont.)

Proc 1

9 10 11
6 7 8

Global Representation:

Proc 0

3 4 5
0 1 2

0 1 2 34 5

Proc 1

Proc 0

6 7 8
3 4 5
0 1 2
6 7 8
3 4 5
0 1 2

Proc 0

Proc 1

Local Representations:
Proc 1

3 4 5 6 7 8 9 1 11
0
0 1 2 3 4 5 6 7 8

0 1 23 4 5 6 7 8
0 1 2 3 4

beginner
beginner

6 7 8 9 10 11

Proc 0

5 6 7 8
data
datalayout:
layout:
distributed
distributedarrays
arrays

139

Logically Regular Meshes

DA - Distributed Array: object containing information about vector
layout across the processes and communication of ghost values
Form a DA
DACreate1d(.,DA *)
DACreate2d(.,DA *)
DACreate3d(.,DA *)

Create the corresponding PETSc vectors

DACreateGlobalVector( DA, Vec *) or
DACreateLocalVector( DA, Vec *)

Update ghostpoints (scatter global vector into local parts, including

ghost points)
DAGlobalToLocalBegin(DA, )
DAGlobalToLocalEnd(DA,)
beginner
beginner

data
datalayout:
layout:
distributed
distributedarrays
arrays

140

Distributed Arrays
Data layout and ghost values

Proc 10

Proc 0

Proc 1

Box-type
stencil
beginner
beginner

Proc 10

Proc 0

Proc 1

Star-type
stencil
data
datalayout:
layout:
distributed
distributedarrays
arrays

141

Vectors and DAs

The DA object contains information about the data
layout and ghost values, but not the actual field data,
which is contained in PETSc vectors
Global vector: parallel
each process stores a unique local portion
DACreateGlobalVector(DA da,Vec *gvec);

Local work vector: sequential

each process stores its local portion plus ghost values
DACreateLocalVector(DA da,Vec *lvec);
uses natural local numbering of indices (0,1,nlocal-1)
beginner
beginner

data
datalayout:
layout:
distributed
distributedarrays
arrays

142

DACreate1d(,*DA)
DACreate1d(MPI_Comm
comm,DAPeriodicType wrap,int M,int dof,int
s,int *lc,DA *inra)
MPI_Comm processes containing array
DA_[NONPERIODIC,XPERIODIC]

beginner
beginner

number of grid points in x-direction

degrees of freedom per node
stencil width
Number of nodes for each cell (use PETSC_NULL
for the default)
data
datalayout:
layout:
distributed
distributedarrays
arrays

143

DACreate2d(,*DA)
DACreate2d(MPI_Comm comm,DAPeriodicType
wrap,DAStencilType stencil_type, int M,int N,int
m,int n,int dof,int s,int *lx,int *ly,DA *inra)
DA_[NON,X,Y,XY]PERIODIC
DA_STENCIL_[STAR,BOX]

number of grid points in x- and y-directions

processes in x- and y-directions
degrees of freedom per node
stencil width
Number of nodes for each cell (use PETSC_NULL for the default) as
tensor-product

And similarly for DACreate3d()

beginner
beginner

data
datalayout:
layout:
distributed
distributedarrays
arrays

144

Updating the Local Representation

Two-step process that enables overlapping
computation and communication
DAGlobalToLocalBegin(DA, Vec global_vec,
insert,
Vec local_vec )
global_vec provides data
Insert is either INSERT_VALUES or ADD_VALUES
and specifies how to update values in the local
vector, local_vec (a pre-existing local vector)

DAGlobalToLocal End(DA,)
Takes same arguments
beginner
beginner

data
datalayout:
layout:
distributed
distributedarrays
arrays

145

Ghost Point Scatters:

Burgers Equation Example
call
DAGlobalToLocalBegin(da,u_global,INSERT_VALUES,ulocal,ierr)
call
DAGlobalToLocalEnd(da,u_global,INSERT_VALUES,ulocal,ierr)
call VecGetArray( ulocal, uv, ui, ierr )
#define u(i) uv(ui+i)
C Do local computations (here u and f are local vectors)
do 10, i=1,localsize
f(i) = (.5/h)*u(i)*(u(i+1)-u(i-1)) +
(e/(h*h))*(u(i+1) - 2.0*u(i) + u(i-1))
10 continue
data
datalayout:
layout:
call VecRestoreArray( ulocal, uv, ui, ierr )
beginner
distributed
beginner
distributedarrays
arrays
call DALocalToGlobal(da,f,INSERT_VALUES,f_global,ierr)

146

Global Numbering used by DAs

Proc 2

Proc 3

Proc 2

Proc 3

26 27 28
21 22 23
16 17 18

29 30
24 25
19 20

22 23 24
19 20 21
16 17 18

29 30
27 28
25 26

11 12 13
6 7 8
1 2 3

14 15
9 10
4 5

Proc 0

Proc1

Natural numbering, corresponding

to the entire problem domain
intermediate
intermediate

7
4

8
5

9
6

14 15
12 13

10 11

Proc 0

Proc1

PETSc numbering used by DAs

data
datalayout:
layout:
distributed
distributedarrays
arrays

147

Mapping Between Global

Numberings
Natural global numbering
convenient for visualization of global problem,
specification of certain boundary conditions, etc.

Can convert between various global numbering

schemes using AO (Application Orderings)
DAGetAO(DA da, AO *ao);
AO usage explained in next section

Some utilities (e.g., VecView()) automatically

handle this mapping for global vectors attained from
DAs
intermediate
intermediate

data
datalayout:
layout:
distributed
distributedarrays
arrays

148

Distributed Array Example

src/snes/examples/tutorial/ex5.c, ex5f.F
Uses functions that help construct vectors and matrices
naturally associated a DA
DAGetMatrix
DASetLocalFunction
DASetLocalJacobian

149

End of Day 2

150

Unstructured Meshes
Setting up communication patterns is much more
complicated than the structured case due to
mesh dependence
discretization dependence

cell-centered
vertex-centered
cell and vertex centered (e.g., staggered grids)
mixed triangles and quadrilaterals

Can use VecScatter

Introduction in this tutorial
See additional tutorial material available via PETSc web site
beginner
beginner

data
datalayout:
layout:
vector
vectorscatters
scatters

Communication and Physical151

Discretization
Communication
Geometric
Data

Data Structure Ghost Point

Creation
Data Structures

stencil
[implicit]

DACreate( )

DA
AO

Ghost Point
Updates

DAGlobalToLocal( )

structured mesh
elements
edges
vertices

VecScatter
VecScatterCreate( ) AO

Loops over
I,J,K
indices

VecScatter( )

unstructured mesh

Local
Numerical
Computation

Loops over
entities

beginner
beginner intermediate
intermediate

data
datalayout
layout

152

Unstructured Mesh Concepts

AO: Application Orderings
map between various global numbering schemes

IS: Index Sets

indicate collections of nodes, cells, etc.

VecScatter:
update ghost points using vector scatters/gathers

ISLocalToGlobalMapping
map from a local (calling process) numbering of indices
to a global (across-processes) numbering
intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

153

Setting Up the Communication

Pattern
(the steps in creating VecScatter)

Renumber objects so that contiguous entries are

adjacent
Use AO to map from application to PETSc ordering

Determine needed neighbor values

Generate local numbering
Generate local and global vectors
Create communication object (VecScatter)

intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

154

Cell-based Finite Volume

Application Numbering
Proc 0

Proc1

11

10

6
9

7
8

global indices defined by application

intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

155

PETSc Parallel Numbering

Proc 0

Proc 1

10

11

global indices numbered contiguously on each process

intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

156

Remapping Global Numbers:

An Example
Process 0
nlocal: 6
app_numbers: {1,2,0,5,11,10}
PETSc
petsc_numbers: {0,1,2,3,4,5}numbers

Process 1

application
numbers

n_local: 6
app_numbers: {3,4,6,7,9,8}
petsc_numbers: {6,7,8,9,10,11}
intermediate
intermediate

Proc 0
0

Proc 1

10

11

11 10

3
6
9

4
7
8

data
datalayout:
layout:
vector
vectorscatters
scatters

157

Remapping Numbers (1)

Generate a parallel object (AO) to use in mapping
between numbering schemes
AOCreateBasic( MPI_Comm comm,

intermediate
intermediate

int nlocal,
int app_numbers[ ],
int petsc_numbers[ ],
AO *ao );

data
datalayout:
layout:
vector
vectorscatters
scatters

158

Remapping Numbers (2)

AOApplicationToPetsc(AO *ao,
int number_to_convert,
int indices[ ]);

For example, if indices[ ] contains the cell

neighbor lists in an application numbering, then
apply AO to convert to new numbering

intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

159

Neighbors
using global PETSc numbers (ordering
of the global PETSc vector)
Process 0
Process1

10

10

11

intermediate
intermediate

ghost cells

data
datalayout:
layout:
vector
vectorscatters
scatters

160

Local Numbering
Proc 0

Proc1

At end of local vector

intermediate
intermediate

ghost cells

data
datalayout:
layout:
vector
vectorscatters
scatters

161

Summary of the Different

Orderings
Application Ordering

PETSc Ordering

13

14

15

16

15

16

10

11

12

13

14

11

12

10

Local Ordering on Process

Local Ordering on Process

12

12

11

11

10

10

Local points
at the end

162

Global and Local

Representations
Global representation
parallel vector with no ghost locations
suitable for use by PETSc parallel solvers (SLES,
SNES, and TS)

Local representation
sequential vectors with room for ghost points
used to evaluate functions, Jacobians, etc.

intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

163

Global and Local

Representations
Global representation:
0 1 2 3 4 5

6 7 8 9 10 11

Proc 0

Proc1

Local Representations:
0 1 2 3 4 5 6 8 10
0 1 2 3 4 5 6 7 8
Proc1

intermediate
intermediate

Proc 0

6 7 8 9 10 11 1 3 5
0 1 2 3 4 5 6 7 8
data
datalayout:
layout:
vector
vectorscatters
scatters

164

Creating Vectors
Sequential
VecCreateSeq(PETSC_COMM_SELF, 9, Vec
*lvec);

Parallel
VecCreateMPI(PETSC_COMM_WORLD,
6,PETSC_DETERMINE,Vec
*gvec)

intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

165

Local and Global Indices

Process 0
int indices[] = {0,1,2,3,4,5,6,8,10};
ISCreateGeneral(PETSC_COMM_WORLD, ISCreateGeneral
() - Specify global
9, indices, IS *isg);
numbers
ISCreateStride(PETSC_COMM_SELF, 9,0,1,IS*
isl); of locally
owned cells,
including ghost cells

Process 1

ISCreateStride
int indices = {6,7,8,9,10,11,1,3,5};
() - Specify local
ISCreateGeneral(PETSC_COMM_WORLD,
numbers of
9, indices, IS *isg);
locally owned
ISCreateStride(PETSC_COMM_SELF, 9,0,1,IS*
isl);including
cells,
ghost cells

intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

166

Creating Communication Objects

VecScatterCreate(Vec gvec,

IS gis,
Vec lvec,
IS lis
VecScatter gtol);

Determines all required messages for mapping

data from a global vector to local (ghosted)
vectors
intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

167

Performing a Global-to-Local Scatter

Two-step process that enables overlapping
computation and communication
VecScatterBegin(VecScatter gtol,

Vec gvec,
Vec lvec,
INSERT_VALUES
SCATTER_FORWARD);

VecScatterEnd(...);
intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

168

Performing a Local-to-Global Scatter

VecScatterBegin(VecScatter gtol,

Vec lvec,
Vec gvec,
ADD_VALUES,
SCATTER_REVERSE);

VecScatterEnd(...);

intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

169

Setting Values in Global Vectors

and Matrices using a Local
Numbering
Create mapping
ISLocalToGlobalMappingCreateIS(IS gis,
ISLocalToGlobalMapping *lgmap);

Set mapping
VecSetLocalToGlobalMapping(Vec gvec,
ISLocalToGlobalMapping lgmap);
MatSetLocalToGlobalMapping(Mat gmat,
ISLocalToGlobalMapping lgmap);

Set values with local numbering

VecSetValuesLocal(Vec gvec,int ncols, int
localcolumns, Scalar *values, );
MatSetValuesLocal(Mat gmat, ...);
MatSetValuesLocalBlocked(Mat gmat, ...);
intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

170

Sample Function Evaluation

int FormFunction(SNES snes, Vec Xglobal, Vec Fglobal, void *ptr)

{
AppCtx *user = (AppCtx *) ptr;
Predefined Scatter Context
double x1, x2, f1, f2, *x, *f ;
int
*edges = user->edges;
VecScatterBegin(user->scatter, Xglobal, user->Xlocal,
SCATTER_FORWARD, INSERT_VALUES);
VecScatterEnd(user->scatter, Xglobal, user->Xlocal, SCATTER_FORWARD,
INSERT_VALUES);
VecGetArray(Xlocal,&X); VecGetArray(Flocal,&F);
for (i=0; i < user->nlocal; i++) {
x1 = X[edges[2*i]]; x2 = X[edges[2*i+1]]; /* then compute f1, f2 */
F[edges[2*i]] += f1; F[edges[2*i+1]] += f2;
}
VecRestoreArray(Xlocal,&X); VecRestoreArray(Flocal,&F);

VecScatterBegin(user->scatter, user->Flocal, Fglobal, SCATTER_REVERSE,

INSERT_VALUES);
VecScatterEnd(user->scatter, user->Flocal, Fglobal, SCATTER_REVERSE,
INSERT_VALUES);
return 0;

intermediate
intermediate

data
datalayout:
layout:
vector
vectorscatters
scatters

Communication and Physical171

Discretization
Communication
Geometric
Data

Data Structure Ghost Point

Creation
Data Structures

stencil
[implicit]

DACreate( )

DA
AO

Ghost Point
Updates

DAGlobalToLocal( )

structured mesh
elements
edges
vertices

VecScatter
VecScatterCreate( ) AO

Loops over
I,J,K
indices

VecScatter( )

unstructured mesh

Local
Numerical
Computation

Loops over
entities

beginner
beginner intermediate
intermediate

data
datalayout
layout

172

Unstructured Meshes: Example

Programs
Location:
petsc/src/snes/examples/tutorials/ex10d/ex10.c
2 process example (PETSc requires that a separate tool
be used to divide the unstructured mesh among the
processes).

173

Matrix Partitioning and Coloring

Intermediate
intermediate

Partitioner object creation and use

Sparse finite differences for Jacobian
computation (using colorings)

tutorial outline:
data objects:
matrix partitioning
and coloring

174

Partitioning for Load Balancing

MatPartitioning - object for managing the
partitioning of meshes, vectors, matrices, etc.
Interface to parallel partitioners, i.e., the adjacency
matrix, is stored in parallel
MatPartitioningCreate(
MPI_Comm comm
MatPartitioning *matpart);

intermediate
intermediate

data
dataobjects:
objects:
matrices:
matrices:
partitioning
partitioning

175

Setting Partitioning Software

MatPartitioningSetType(
MatPartitioning matpart,
MatPartitioningType
MATPARTITIONING_PARMETIS);

PETSc interface could support other packages

(e.g., Pjostle, just no one has written wrapper
code yet)

intermediate
intermediate

data
dataobjects:
objects:
matrices:
matrices:
partitioning
partitioning

176

Setting Partitioning Information

MatPartitioningSetAdjacency(
MatPartitioning matpart,
Mat adjacency_matrix);

MatPartitioningSetVertexWeights(
MatPartitioning matpart,
int *weights);

MatPartitioningSetFromOptions(
MatPartitioning matpart);

intermediate
intermediate

data
dataobjects:
objects:
matrices:
matrices:
partitioning
partitioning

177

Partitioning
MatPartitioningApply(
MatPartitioning matpart,
IS* processforeachlocalnode);

MatPartitioningDestroy(
MatPartitioning matpart);

intermediate
intermediate

data
dataobjects:
objects:
matrices:
matrices:
partitioning
partitioning

178

Constructing Adjacency Matrix

MatCreateMPIAdj( (Uses CSR input format)

MPI_Comm comm,
int numberlocalrows,
int numbercolums,
int rowindices[ ],
int columnindices[ ],
int values[ ],
Mat *adj)

Other input formats possible, e.g., via

MatSetValues( )

intermediate
intermediate

data
dataobjects:
objects:
matrices:
matrices:
partitioning
partitioning

179

Finite Difference Approximation

of Sparse Jacobians
Two-stage process:
1 Compute a coloring of the Jacobian (e.g., determine
columns of the Jacobian that may be computed using a
single function evaluation)
2 Construct a MatFDColoring object that uses the
coloring information to compute the Jacobian
example: driven cavity model:
petsc/src/snes/examples/tutorials/ex8.c

intermediate
intermediate

data
dataobjects:
objects:
matrices:
matrices:
coloring
coloring

180

Coloring for Structured Meshes

DAGetColoring (
DA da,
ISColorType ctype,
ISColoring *coloringinfo );

DAGetMatrix(
DA da,
MatType mtype,
Mat *sparsematrix );

For structured meshes (using DAs) we provide parallel

colorings

intermediate
intermediate

data
dataobjects:
objects:
matrices:
matrices:
coloring
coloring

181

Coloring for Unstructured

Meshes
MatGetColoring (
Mat A,
MatColoringType

MATCOLORING_NATURAL
MATCOLORING_SL
MATCOLORING_LF
MATCOLORING_ID

ISColoring *coloringinfo)

Automatic coloring of sparse matrices currently

implemented only for sequential matrices
If using a local function evaluation, the sequential
coloring is enough
data
dataobjects:
objects:
intermediate
intermediate

matrices:
matrices:
coloring
coloring

182

Actual Computation of Jacobians

MatFDColoringCreate (
Mat J,
ISColoring coloringinfo,
MatFDColoring *matfd)

MatFDColoringSetFunction(
MatFDColoring matfd,
int (*f)(void),void *fctx)

MatFDColoringSetFromOptions(
MatFDColoring)
intermediate
intermediate

data
dataobjects:
objects:
matrices:
matrices:
coloring
coloring

183

Computation of Jacobians within

SNES
User calls: SNESSetJacobian(

SNES snes,
Mat A ,
Mat J,
SNESDefaultComputeJacobianColor( ),
fdcoloring);

where SNESSetJacobian() actually

uses . MatFDColoringApply(

Mat J,
MatFDColoring coloring,
Vec x1,
MatStructure *flag,
void *sctx)

to form the Jacobian

intermediate
intermediate

data
dataobjects:
objects:
matrices:
matrices:
coloring
coloring

184

Driven Cavity Model

Example code: petsc/src/snes/examples/tutorials/ex19.c
Solution Components
Velocity-vorticity formulation,
with flow driven by lid and/or
bouyancy
Finite difference discretization
with 4 DoF per mesh point

velocity: u

velocity: v

[u,v,,T]

vorticity: temperature: T
1

beginner
beginner intermediate
intermediate

solvers:
solvers:
nonlinear
nonlinear

185

Driven Cavity Program

Part A: Parallel data layout
Part B: Nonlinear solver creation, setup, and usage
Part C: Nonlinear function evaluation
ghost point updates
local function computation

Part D: Jacobian evaluation

default colored finite differencing approximation

Experimentation
1

beginner
beginner intermediate
intermediate

solvers:
solvers:
nonlinear
nonlinear

186

Driven Cavity Solution Approach

A
B

Main Routine

Nonlinear Solvers (SNES)

Solve
F(u) = 0

Linear Solvers (SLES)

PC
Application
Initialization

PETSc

KSP
Function
Evaluation

C
User code

Jacobian
Evaluation

PostProcessing

D
PETSc code

solvers:
solvers:
nonlinear
nonlinear

187

Driven Cavity:
Running the program (1)
Matrix-free Jacobian approximation with no preconditioning
(via -snes_mf) does not use explicit Jacobian
evaluation
1 process: (thermally-driven flow)
mpirun -np 1 ex19 -snes_mf -snes_monitor -grashof
1000.0 -lidvelocity 0.0

2 processes, view DA (and pausing for mouse input):

mpirun -np 2 ex19 -snes_mf -snes_monitor
-da_view_draw -draw_pause -1

View contour plots of converging iterates

mpirun ex19 -snes_mf -snes_monitor -snes_vecmonitor
beginner
beginner

solvers:
solvers:
nonlinear
nonlinear

188

Driven Cavity:
Running the program (2)
Use MatFDColoring for sparse finite difference Jacobian
approximation; view SNES options used at runtime
mpirun ex8 -snes_view -mat_view_info

Set trust region Newton method instead of default line search

mpirun ex8 -snes_type tr -snes_view -snes_monitor

Set transpose-free QMR as Krylov method; set relative KSP

convergence tolerance to be .01
mpirun ex8 -ksp_type tfqmr -ksp_rtol .01 -snes_monitor

intermediate
intermediate

solvers:
solvers:
nonlinear
nonlinear

189

Debugging and Error Handling

beginner
beginner

Automatic generation of tracebacks

beginner
beginner

Detecting memory corruption and leaks

developer
developer

Optional user-defined error handlers

tutorial
tutorialoutline:
outline:
debugging
debuggingand
anderrors
errors

190

Debugging
Support for parallel debugging
-start_in_debugger [gdb,dbx,noxterm]
-on_error_attach_debugger
[gb,dbx,noxterm]
-on_error_abort
-debugger_nodes 0,1
-display machinename:0.0

When debugging, it is often useful to place

a breakpoint in the function PetscError( ).
beginner
beginner

debugging
debuggingand
anderrors
errors

Sample Error Traceback

191

Breakdown in ILU factorization due to a zero pivot

beginner
beginner

debugging
debuggingand
anderrors
errors

192

Sample Memory Corruption Error

beginner
beginner

debugging
debuggingand
anderrors
errors

193

Sample Out-of-Memory Error

beginner
beginner

debugging
debuggingand
anderrors
errors

194

Sample Floating Point Error

beginner
beginner

debugging
debuggingand
anderrors
errors

195

Performance Requires
Managing Memory
Real systems have many levels of memory
Programming models try to hide memory hierarchy
Except Cregister

Simplest model: Two levels of memory

Divide at largest (relative) latency gap
Processes have their own memory
Managing a processes memory is known (if unsolved) problem

Exactly matches the distributed memory model

intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

196

Sparse Matrix-Vector Product

Common operation for optimal (in floating-point
operations) solution of linear systems
Sample code:
for row=0,n-1
m
= i[row+1] - i[row];
sum = 0;
for k=0,m-1
sum += *a++ * x[*j++];
y[row] = sum;

Data structures are a[nnz], j[nnz], i[n], x[n], y[n]

intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

197

Simple Performance Analysis

Memory motion:
nnz (sizeof(double) + sizeof(int)) +
n (2*sizeof(double) + sizeof(int))
Perfect cache (never load same data twice)

Computation
nnz multiply-add (MA)

Roughly 12 bytes per MA

Typical WS node can move -4 bytes/MA
Maximum performance is 4-33% of peak

intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

198

More Performance Analysis

Instruction Counts:
nnz (2*load-double + load-int + mult-add) +
n (load-int + store-double)

Roughly 4 instructions per MA

Maximum performance is 25% of peak (33% if MA
overlaps one load/store)
Changing matrix data structure (e.g., exploit small block
structure) allows reuse of data in register, eliminating some
loads (x and j)
Implementation improvements (tricks) cannot improve on
these limits
intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

199

Alternative Building Blocks

Performance of sparse matrix - multi-vector multiply:
Format
AIJ
AIJ
BAIJ
BAIJ

Number
of Vectors
1
4
1
4

Mflops
Ideal
Achieved
49
45
182
120
64
55
236
175

Results from 250 MHz R10000 (500 MF/sec peak)

BAIJ is a block AIJ with blocksize of 4
Multiple right-hand sides can be solved in nearly the same
time as a single RHS
intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

200

Matrix Memory Pre-allocation

PETSc sparse matrices are dynamic data
structures. Can add additional nonzeros freely
Dynamically adding many nonzeros
requires additional memory allocations
requires copies
can kill performance

Memory pre-allocation provides the freedom of

dynamic data structures plus good performance
intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

201

Indicating Expected Nonzeros

Sequential Sparse Matrices
MatCreateSeqAIJ(., int *nnz,Mat *A)
nnz[0] - expected number of nonzeros in row 0
nnz[1] - expected number of nonzeros in row 1
row 0
row 1
row 2
row 3
sample nonzero pattern

intermediate
intermediate

another sample
nonzero pattern
profiling
profilingand
and
performance
performancetuning
tuning

202

Symbolic Computation of Matrix

Nonzero Structure
Write code that forms the nonzero structure of the
matrix
loop over the grid for finite differences
loop over the elements for finite elements
etc.

Then create matrix via MatCreateSeqAIJ(),

MatCreateMPIAIJ(), ...

intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

203

Parallel Sparse Matrices

Each process locally owns a submatrix of
contiguously numbered global rows.
Each submatrix consists of diagonal and offdiagonal parts.
proc 0
proc 1
proc 2
proc 3
proc 4
intermediate
intermediate

diagonal portions
off-diagonal portions

} proc 3: locally owned rows

profiling
profilingand
and
performance
performancetuning
tuning

204

Indicating Expected Nonzeros

Parallel Sparse Matrices
MatCreateMPIAIJ(., int d_nz, int *d_nnz,int
o_nz,
int *o_nnz,Mat *A)

d_nnz[ ] - expected number of nonzeros per

row in diagonal portion of local submatrix
o_nnz[ ] - expected number of nonzeros per
row in off-diagonal portion of local submatrix
intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

205

Verifying Predictions
Use runtime option: -log_info
Output:
[proc #] Matrix size: % d X % d; storage space: % d
unneeded, % d used
[proc #] Number of mallocs during MatSetValues( ) is
%d

intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

206

Application Code Optimizations

Coding techniques to improve
performance of user code on cachebased RISC architectures

intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

207

Cache-based CPUs
memory

slow
intermediate
intermediate

cache

floating
registers point
units

fast
profiling
profilingand
and
performance
performancetuning
tuning

208

Variable Interleaving
real u(maxn), v(maxn)

u(i) = u(i) + a*u(i-n)

v(i) = v(i) + b*v(i-n)

Consider direct-mapped cache:

V V V
U V
U V
UV
UV
U
VU
U
VU
V
UV
V
UV
V
UV
V
UV
U
UU U
Doubles number of cache misses
Halves performance

n-way associative caches defeated by n+1 components

Many problems have at least four to five components per point
intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

209

Techniques:
Interleaving Field Variables
Not

double u[ ], double v[ ] ,double p[ ], ...

Not

double precision fields(largesize,nc)

Instead
double precision fields(nc,largesize)
intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

210

Techniques:
Reordering to Reuse Cache Data
Sort objects so that repeated use occurs together
e.g., sort edges by the first vertex they contain

Reorder to reduce matrix bandwidth

e.g., RCM ordering

intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

211

Effects on Performance: Sample

Incompressible Euler code
Unstructured grid
Edge-based discretization (i.e., computational
loops are over edges)
Block size 4
Results collated by Dinesh Kaushik (ODU)

intermediate
intermediate

profiling
profilingand
and
performance
performancetuning
tuning

212

Effects on Performance:
Sample Results on IBM SP
120
100
80
Time (seconds)

60
40

intermediate
intermediate

All

I nterlaced
reordered

I nterlaced
blocking

I nterlaced

Basic

20

profiling
profilingand
and
performance
performancetuning
tuning

213

Other Features of PETSc

Summary
New features
Interfacing with other packages
Extensibility issues
References

tutorial
tutorialoutline:
outline:
conclusion
conclusion

214

Summary
Creating data objects
Setting algorithmic options for linear, nonlinear
and ODE solvers
Using callbacks to set up the problems for
nonlinear and ODE solvers
Managing data layout and ghost point
communication
Evaluating parallel functions and Jacobians
Consistent profiling and error handling

216
Multigrid Structured Mesh Support:

DMMG: New Simple Interface

General multigrid support
PC framework wraps MG for use as preconditioner
See MGSetXXX(), MGGetXXX()
Can access via -pc_type mg

User provides coarse grid solver, smoothers, and

interpolation/restriction operators

DMMG - simple MG interface for structured meshes

User provides
Local function evaluation
[Optional] local Jacobian evaluation

Proc 10

Proc 0 Proc 1

217
Multigrid Structured Mesh Support:

Sample Function Computation

int Function(DALocalInfo *info,double **u,double **f,AppCtx
*user)

lambda = user->param;
hx
= 1.0/(info->mx-1);
hy
= 1.0/(info->my-1);
for (j=info->ys; j<info->ys+info->ym; j++) {
for (i=info->xs; i<info->xs+info->xm; i++) {
f[j][i] = u[j][i] u[j-1][i] .
}
}

218
Multigrid Structured Mesh Support:

Sample Jacobian Computation

int Jacobian (DALocalInfo *info,double **u,Mat J,AppCtx *user)
MatStencil mrow,mcols[5];
double
v[5];

for (j=info->ys; j<info->ys+info->ym; j++) {

row.j = j;
for (i=info->xs; i<info->xs+info->xm; i++) {
v[0] = ;
col[0].j = j - 1; col[0].i = i;
v[1] = ;
col[1].j = j;
col[1].i = i-1;
v[2] = ;
col[2].j = j;
col[2].i = i;
v[3] = ;
col[3].j = j;
col[3].i = i+1;
v[4] = ;
col[4].j = j + 1; col[4].i = i;
MatSetValuesStencil(jac,1,&row,5,col,v,INSERT_VALUES);
}
}

Multigrid Structured Mesh Support:

219

Nonlinear Example
2-dim nonlinear problem with 4 degrees of freedom per mesh point
Function() and Jacobian() are user-provided functions

DMMG dmmg;
DMMGCreate(comm,nlevels,user,&dmmg)
DACreate2d(comm,DA_NONPERIODIC,DA_STENCIL_STAR,4,
4,PETSC_DECIDE,PETSC_DECIDE,4,1,0,0,&da)
DMMGSetDM(dmmg,da)
DMMGSetSNESLocal(dmmg,Function,Jacobian,0,0)
DMMGSolve(dmmg)
solution = DMMGGetx(damg)

All standard SNES, SLES, PC and MG options apply.

Multigrid Structured Mesh Support:

220

Jacobian via Automatic

Differentiation
Collaboration with P. Hovland and B. Norris (see
http://www.mcs.anl.gov/autodiff)
Additional alternatives
Compute sparse Jacobian explicitly using AD
DMMGSetSNESLocal(dmmg,Function,0,ad_Function,0)
PETSc + ADIC automatically generate ad_Function

Provide a matrix-free application of the Jacobian using AD

DMMGSetSNESLocal(dmmg,Function, 0,0,admf_Function)
PETSc + ADIC automatically generate admf_Function

Similar situation for Fortran and ADIFOR

221

Using PETSc with Other

Packages
Linear algebra solvers

Optimization software

AMG

TAO

BlockSolve95

Veltisto

DSCPACK

Mesh and discretization tools

hypre

Overture

ILUTP

SAMRAI

LUSOL
SPAI
SPOOLES
SuperLU, SuperLU_Dist

SUMAA3d

ODE solvers
PVODE

Others
Matlab
ParMETIS

222
Using PETSc with Other Packages:

Linear Solvers
Interface Approach
Based on interfacing at the matrix level, where external linear solvers typically use
a variant of compressed sparse row matrix storage

Usage
Install PETSc indicating presence of any optional external packages in the file
petsc/bmake/$PETSC_ARCH/base.site, e.g.,
PETSC_HAVE_SPAI = -DPETSC_HAVE_SPAI
SPAI_INCLUDE = -I/home/username/software/spai_3.0/include
SPAI_LIB = /home/username/software/spai_3.0/lib/$
{PETSC_ARCH}/libspai.a

Set preconditioners via the usual approach

Procedural interface: PCSetType(pc,spai)
Runtime option: -pc_type spai

Set preconditioner-specific options via the usual approach, e.g.,

PCSPAISetEpsilon(), PCSPAISetVerbose(), etc.
-pc_spai_epsilon <eps> -pc_spai_verbose etc.

223
Using PETSc with Other Packages:

Linear Solvers
AMG
Algebraic multigrid code by J. Ruge, K. Steuben, and R. Hempel
(GMD)
http://www.mgnet.org/mgnet-codes-gmd.html
PETSc interface by D. Lahaye (K.U.Leuven), uses MatSeqAIJ

BlockSolve95

Parallel, sparse ILU(0) for symmetric nonzero structure and ICC(0)

M. Jones (Virginia Tech.) and P. Plassmann (Penn State Univ.)
http://www.mcs.anl.gov/BlockSolve95
PETSc interface uses MatMPIRowbs

ILUTP
Drop tolerance ILU by Y. Saad (Univ. of Minnesota), in SPARSKIT
http://www.cs.umn.edu/~saad/
PETSc interface uses MatSeqAIJ

224
Using PETSc with Other Packages:

Linear Solvers (cont.)

LUSOL

Sparse LU, part of MINOS

M. Saunders (Stanford Univ)
http://www.sbsi-sol-optimize.com
PETSc interface by T. Munson (ANL), uses MatSeqAIJ

SPAI

Sparse approximate inverse code by S. Barnhard (NASA Ames) and M. Grote (ETH Zurich)
http://www.sam.math.ethz.ch/~grote/spai
PETSc interface converts from any matrix format to SPAI matrix

SuperLU

Parallel, sparse LU
J. Demmel, J. Gilbert, (U.C. Berkeley) and X. Li (NERSC)
http://www.nersc.gov/~xiaoye/SuperLU
PETSc interface uses MatSeqAIJ
Currently only sequential interface supported; parallel interface under development

225
Using PETSc with Other Packages:

TAO Optimization Software

TAO - Toolkit for Advanced Optimization
Software for large-scale optimization problems
S. Benson, L. McInnes, and J. Mor
http://www.mcs.anl.gov/tao

Initial TAO design uses PETSc for

Low-level system infrastructure - managing portability
Parallel linear algebra tools (SLES)
Veltisto (library for PDE-constrained optimization by G. Biros, see
http://www.cs.nyu.edu/~biros/veltisto) uses a similar interface approach

TAO is evolving toward

CCA-compliant component-based design (see http://www.cca-forum.org)
Support for ESI interfaces to various linear algebra libraries (see
http://z.ca.sandia.gov/esi)

226

TAO Interface
TAO_SOLVER
Vec
vectors */
ApplicationCtx

tao;
x, g;
usercontext;

/* optimization solver */
/* solution and gradient
/* user-defined context */

TaoInitialize();
/* Initialize Application -- Create variable and gradient vectors
x and g */ ...
TaoCreate(MPI_COMM_WORLD,tao_lmvm,&tao);
TaoSetFunctionGradient(tao,x,g, FctGrad,
(void*)&usercontext);
TaoSolve(tao);
/* Finalize application -- Destroy vectors x and g */ ...
TaoDestroy(tao);
TaoFinalize();

Similar Fortran interface, e.g., call TaoCreate(...)

software
software
interfacing:
interfacing:
TAO
TAO

227
Using PETSc with Other Packages:

PVODE ODE Integrators

PVODE

Parallel, robust, variable-order stiff and non-stiff ODE integrators

A. Hindmarsh et al. (LLNL)

http://www.llnl.gov/CASC/PVODE

L. Xu developed PVODE/PETSc interface

Interface Approach

PVODE

ODE integrator evolves field variables in time

vector holds field variables
preconditioner placeholder

PETSc

Usage

TSCreate(MPI_Comm,TS_NONLINEAR,&ts)
TSSetType(ts,TS_PVODE)

.. regular TS functions

TSPVODESetType(ts,PVODE_ADAMS)

. other PVODE options

TSSetFromOptions(ts) accepts PVODE options

ODE integrator placeholder

vector
sparse matrix and preconditioner

Using PETSc with Other Packages: 228

Mesh Management and

Discretization
SUMAA3d
Scalable Unstructured Mesh Algorithms and Applications
L. Freitag (ANL), M. Jones (VA Tech), P. Plassmann (Penn State)

http://www.mcs.anl.gov/sumaa3d

L. Freitag and M. Jones developed SUMAA3d/PETSc interface

SAMRAI
Structured adaptive mesh refinement
R. Hornung, S. Kohn (LLNL)

http://www.llnl.gov/CASC/SAMRAI

SAMRAI team developed SAMRAI/PETSc interface

Overture
Structured composite meshes and discretizations
D. Brown, W. Henshaw, D. Quinlan (LLNL)

http://www.llnl.gov/CASC/Overture

K. Buschelman and Overture team developed Overture/PETSc interfaces

229
Using PETSc with Other Packages:

Matlab

Matlab

http://www.mathworks.com

Interface Approach

PETSc socket interface to Matlab

PETSc interface to MatlabEngine

Sends matrices and vectors to interactive Matlab session

MatlabEngine Matlab library that allows C/Fortran programmers to use Matlab
functions in programs
PetscMatlabEngine unwraps PETSc vectors and matrices so that
MatlabEngine can understand them

Usage

PetscMatlabEngineCreate(MPI_Comm,machinename,
PetscMatlabEngine eng)
PetscMatlabEnginePut(eng,PetscObject obj)
Vector
Matrix
PetscMatlabEngineEvaluate(eng,R = QR(A);)
PetscMatlabEngineGet(eng,PetscObject obj)

230
Using PETSc with Other Packages:

ParMETIS Graph Partitioning

ParMETIS
Parallel graph partitioning
G. Karypis (Univ. of Minnesota)
http://www.cs.umn.edu/~karypis/metis/parmetis

Interface Approach
Use PETSc MatPartitioning() interface and MPIAIJ or MPIAdj matrix
formats
Usage

MatPartitioningCreate(MPI_Comm,MatPartitioning ctx)
MatPartitioningSetAdjacency(ctx,matrix)
Optional MatPartitioningSetVertexWeights(ctx,weights)
MatPartitioningSetFromOptions(ctx)
MatPartitioningApply(ctx,IS *partitioning)

231

Recent Additions
Hypre (www.llnl.gov/casc/hypre) via PCHYPRE, includes
EUCLID (parallel ILU(k) by David Hysom)
BoomerAMG
PILUT

DSCPACK, a Domain-Separator Cholesky Package for

solving sparse spd problems, by Padma Raghavan
SPOOLES (SParse Object Oriented Linear Equations
Solver), by Cleve Ashcraft
Both SuperLU and SuperLU_Dist sequential and parallel
direct sparse solvers, by Xiaoye Li et al.

232

Extensibility Issues
Most PETSc objects are designed to allow one to
drop in a new implementation with a new set of
data structures (similar to implementing a new
class in C++).
Heavily commented example codes include
Krylov methods: petsc/src/sles/ksp/impls/cg
preconditioners: petsc/src/sles/pc/impls/jacobi

Feel free to discuss more details with us in person.

233

Caveats Revisited
Developing parallel, non-trivial PDE solvers that
deliver high performance is still difficult, and
requires months (or even years) of concentrated
effort.
PETSc is a toolkit that can ease these difficulties
and reduce the development time, but it is not a
black-box PDE solver nor a silver bullet.
Users are invited to interact directly with us
regarding correctness and performance issues by
writing to [email protected].

234

References
Documentation: http://www.mcs.anl.gov/petsc/docs

PETSc Users manual

Manual pages
Many hyperlinked examples
FAQ, Troubleshooting info, installation info, etc.

Publications: http://www.mcs.anl.gov/petsc/publications
Research and publications that make use PETSc

MPI Information: http://www.mpi-forum.org

Using MPI (2nd Edition), by Gropp, Lusk, and Skjellum
Domain Decomposition, by Smith, Bjorstad, and Gropp