Parallel Distributed Computing using Python
Lisandro Dalcin
[email protected]
Joint work with
Pablo Kler Rodrigo Paz
Mario Storti Jorge D’Elı́a
Consejo Nacional de Investigaciones Cientı́ficas y Técnicas (CONICET)
Instituto de Desarrollo Tecnológico para la Industria Quı́mica (INTEC)
Centro Internacional de Métodos Computacionales en Ingenierı́a (CIMEC)
http://www.cimec.org.ar
HPCLatAm 2011
Córdoba, Argentina
September 1, 2011
Outline
Overview
MPI for Python
PETSc for Python
Applications
Overview
MPI for Python
PETSc for Python
Applications
Motivation
◮ Apply numerical methods in complex problems of medium to
large scale in science and engineering
◮ multiphysic nature
◮ strong/loose coupling
◮ multiple interacting scales
◮ Ease the access to computing resources in distributed memory
architectures (clusters, supercomputers)
◮ beginners
◮ scientists and engineers
◮ experienced software developers
Objectives
◮ Develop extension modules for the Python programming
language providing access to MPI and PETSc libraries
◮ message passing
◮ parallel linear algebra
◮ linear and nonlinear solvers
◮ Perform computer-based simulations
◮ problems modeled by partial differential equations (PDEs)
◮ problems related to computational fluid mechanics (CFD)
Why Python?
◮ very clear, readable syntax
◮ natural expression of procedural code
◮ very high level dynamic data types
◮ intuitive object orientation
◮ exception-based error handling
◮ full modularity, hierarchical packages
◮ comprehensive standard library
◮ extensible with C and C++
◮ embeddable within applications
Python for Scientific Computing
◮ Scientific computing (and particularly HPC) has been
traditionally dominated by C, C++, y Fortran
◮ High level and general purpouse computing environments
(Maple, Mathematica, MATLAB) got popular since the 90’s
◮ Python is becoming increasingly popular in the scientific
community since the 2000’s
◮ Key feature: easy to extend with C, C++, Fortran
◮ NumPy ◮ Cython
◮ SciPy ◮ SWIG
◮ SymPy ◮ F2Py
What is MPI?
Message Passing Interface
http://www.mpi-forum.org
◮ Standardized message passing system
◮ platform-independent (POSIX, Windows)
◮ many implementations and vendors
◮ MPICH2, Open MPI
◮ HP, Intel, Oracle, Microsoft
◮ Specifies semantics of a set of library routines
◮ No special compiler support
◮ Language-neutral (C, C++, Fortran 90)
◮ Standard versions (backward-compatible)
◮ MPI-1 (1994, 2008)
◮ MPI-2 (1996, 2009)
◮ MPI-3 (under development)
What is PETSc?
Portable, Extensible Toolkit for Scientific Computation
http://www.mcs.anl.gov/petsc
◮ PETSc is a suite of algorithms and data structures
for the numerical solution of
◮ problems in science and engineering
◮ based on partial differential equations models
◮ discretized with finite differences/volumes/elements
◮ leading to large scale applications
◮ PETSc employs the MPI standard for parallelism
◮ PETSc has an OO design, it is implemented in C,
can be used from C++ , provides a Fortran 90 interface
Overview
MPI for Python
PETSc for Python
Applications
MPI for Python (mpi4py)
◮ Python bindings for MPI
◮ API based on the standard MPI-2 C++ bindings
◮ Supports all MPI features
◮ targeted to MPI-2 implementations
◮ also works with MPI-1 implementations
[mpi4py] Implementation
Implemented with Cython
◮ Code base far easier to write, maintain, and extend
◮ Faster than other solutions (mixed Python and C codes)
◮ A pythonic API that runs at C speed !
[mpi4py] Implementation – Cython [1]
cdef import from "mpi.h":
ctypedef void* MPI_Comm
MPI_Comm MPI_COMM_NULL
MPI_Comm MPI_COMM_SELF
MPI_Comm MPI_COMM_WORLD
int MPI_Comm_size(MPI_Comm,int*)
int MPI_Comm_rank(MPI_Comm,int*)
cdef inline int CHKERR(int ierr) except -1:
if ierr != 0:
raise RuntimeError("MPI error code %d" % ierr)
return 0
[mpi4py] Implementation – Cython [2]
cdef class Comm:
cdef MPI_Comm ob_mpi
...
def Get_size(self):
cdef int size
CHKERR( MPI_Comm_size(self.ob_mpi, &size) )
return size
def Get_rank(self):
cdef int rank
CHKERR( MPI_Comm_rank(self.ob_mpi, &rank) )
return rank
...
cdef inline Comm NewComm(MPI_Comm comm_c):
cdef Comm comm_py = Comm()
comm_py.ob_mpi = comm_c
return comm_py
COMM_NULL = NewComm(MPI_COMM_NULL)
COMM_SELF = NewComm(MPI_COMM_SELF)
COMM_WORLD = NewComm(MPI_COMM_WORLD)
[mpi4py] Features – MPI-1
◮ Process groups and communication domains
◮ intracommunicators
◮ intercommunicators
◮ Point to point communication
◮ blocking (send/recv)
◮ nonblocking (isend/irecv + test/wait)
◮ Collective operations
◮ Synchronization (barrier)
◮ Communication (broadcast, scatter/gather)
◮ Global reductions (reduce, scan)
[mpi4py] Features – MPI-2
◮ Extended collective operations.
◮ Dynamic process management (spawn, connect/accept)
◮ Parallel I/O (read/write)
◮ One sided operations, aka RMA (put/get/accumulate)
[mpi4py] Features – Communicating of Python objects
◮ High level and very convenient, based in pickle serialization
◮ Can be slow for large data (CPU and memory consuming)
At the sending side . . .
comm.send(object) −→ pickle.dump() −→ MPI Send()
At the receiving side . . .
object = comm.recv() ←− pickle.load() ←− MPI Recv()
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
if rank == 0:
msg1 = [77, 3.14, 2+3j, "abc", (1,2,3,4)]
elif rank == 1:
msg1 = {"A": [2,"x",3], "B": (2.17,1+3j)}
wt = MPI.Wtime()
if rank == 0:
comm.send(msg1, 1, tag=0)
msg2 = comm.recv(None, 1, tag=7)
elif rank == 1:
msg2 = comm.recv(None, 0, tag=0)
comm.send(msg1, 0, tag=7)
wt = MPI.Wtime() - wt
[mpi4py] Features – Communicating array data
◮ Lower level, slightly more verbose
◮ Very fast, almost C speed (for messages above 5-10 KB)
At the sending side . . .
message = [object, (count, displ), datatype]
comm.Send(message)−→ MPI Send()
At the receiving side . . .
message = [object, (count, displ), datatype]
comm.Recv(message)←− MPI Recv()
from mpi4py import MPI
import numpy as np
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
array1 = np.arange(2**16, dtype=np.float64)
array2 = np.empty(2**16, dtype=np.float64)
wt = MPI.Wtime()
if rank == 0:
comm.Send([array1, MPI.DOUBLE], 1, tag=0)
comm.Recv([array2, MPI.DOUBLE], 1, tag=7)
elif rank == 1:
comm.Recv([array2, MPI.DOUBLE], 0, tag=0)
comm.Send([array1, MPI.DOUBLE], 0, tag=7)
wt = MPI.Wtime() - wt
Point to Point Throughput – Gigabit Ethernet
PingPong
120
Pickle
100 Buffer
C
Throughput [MiB/s]
80
60
40
20
0
100 101 102 103 104 105 106 107
Array Size [Bytes]
Point to Point Throughput – Shared Memory
PingPong
4500
4000
Pickle
Buffer
3500 C
Throughput [MiB/s]
3000
2500
2000
1500
1000
500
0
100 101 102 103 104 105 106 107
Array Size [Bytes]
Overview
MPI for Python
PETSc for Python
Applications
PETSc for Python (petsc4py)
◮ Python bindings for PETSc
◮ Implemented with Cython
◮ Supports most important PETSc features
◮ Pythonic API that better match PETSc’s OO design
◮ class hierarchies, methods, properties
◮ automatic object lifetime management
◮ exception-based error handling
[petsc4py] Features – PETSc components
◮ Index Sets: permutations, indexing, renumbering
◮ Vectors: sequential and distributed
◮ Matrices: sequential and distributed, sparse and dense
◮ Linear Solvers: Krylov subspace methods
◮ Preconditioners: sparse direct solvers, multigrid
◮ Nonlinear Solvers: line search, trust region, matrix-free
◮ Timesteppers: time-dependent, linear and nonlinear PDE’s
Main Routine
Timestepping Solvers (TS)
Nonlinear Solvers (SNES)
Linear Solvers (KSP)
PETSc
Preconditioners (PC)
Application Function Jacobian
Postprocessing
Initialization Evaluation Evaluation
[petsc4py] Vectors (Vec) – CG Method
def cg(A, b, x, imax=50, eps=1e-6):
cg (A, x, b, imax , ǫ) : """
i ⇐0 A, b, x : matrix, rhs, solution
imax : maximum iterations
r ⇐ b − Ax eps : relative tolerance
d ⇐r """
# allocate work vectors
T
δ0 ⇐ r r r = b.duplicate()
d = b.duplicate()
δ ⇐ δ0
q = b.duplicate()
while i < imax and # initialization
2 i = 0
δ > δ0 ǫ do : A.mult(x, r)
q ⇐ Ad r.aypx(-1, b)
r.copy(d)
δ delta_0 = r.dot(r)
α⇐
dT q delta = delta_0
# enter iteration loop
x ⇐ x + αd
while (i < imax and
r ⇐ r − αq delta > delta_0 * eps**2):
δold ⇐ δ A.mult(d, q)
alpha = delta / d.dot(q)
T x.axpy(+alpha, d)
δ ⇐r r
r.axpy(-alpha, q)
δ delta_old = delta
β ⇐
δold delta = r.dot(r)
beta = delta / delta_old
d ⇐ r + βd
d.aypx(beta, r)
i ⇐i +1 i = i + 1
return i, delta**0.5
[petsc4py] Matrices (Mat) [1]
from petsc4py import PETSc
# grid size and spacing
m, n = 32, 32
hx = 1.0/(m-1)
hy = 1.0/(n-1)
# create sparse matrix
A = PETSc.Mat()
A.create(PETSc.COMM_WORLD)
A.setSizes([m*n, m*n])
A.setType(’aij’) # sparse
# precompute values for setting
# diagonal and non-diagonal entries
diagv = 2.0/hx**2 + 2.0/hy**2
offdx = -1.0/hx**2
offdy = -1.0/hy**2
[petsc4py] Matrices (Mat) [2]
# loop over owned block of rows on this
# processor and insert entry values
Istart, Iend = A.getOwnershipRange()
for I in range(Istart, Iend) :
A[I,I] = diagv
i = I//n # map row number to
j = I - i*n # grid coordinates
if i> 0 : J = I-n; A[I,J] = offdx
if i< m-1: J = I+n; A[I,J] = offdx
if j> 0 : J = I-1; A[I,J] = offdy
if j< n-1: J = I+1; A[I,J] = offdy
# communicate off-processor values
# and setup internal data structures
# for performing parallel operations
A.assemblyBegin()
A.assemblyEnd()
[petsc4py] Linear Solvers (KSP+PC)
# create linear solver,
ksp = PETSc.KSP()
ksp.create(PETSc.COMM_WORLD)
# use conjugate gradients method
ksp.setType(’cg’)
# and incomplete Cholesky
ksp.getPC().setType(’icc’)
# obtain sol & rhs vectors
x, b = A.getVecs()
x.set(0)
b.set(1)
# and next solve
ksp.setOperators(A)
ksp.setFromOptions()
ksp.solve(b, x)
[petsc4py] Interoperability
Support for wrapping other PETSc-based C/C++/F90 codes
◮ using Cython (cimport statement)
◮ using SWIG (typemaps provided)
◮ using F2Py (fortran attribute)
[petsc4py] Interoperability – SWIG
%module MyApp
%include petsc4py/petsc4py.i
%{
#include "MyApp.h"
%}
class Nonlinear {
Nonlinear(MPI_Comm comm,
const char datafile[]);
Vec createVec();
Vec createMat();
void evalFunction(SNES snes, Vec X, Vec F);
bool evalJacobian(SNES snes, Vec X, Mat J, Mat P);
};
[petsc4py] Interoperability – SWIG
from petsc4py import PETSc
import MyApp
comm = PETSc.COMM_WORLD
app = MyApp.Nonlinear(comm, "example.dat")
X = app.crateVec()
F = app.crateVec()
J = app.crateMat()
snes = PETSc.SNES().create(comm)
snes.setFunction(app.evalFunction, F)
snes.setFunction(app.evalJacobian, J)
snes.setFromOptions()
snes.solve(None, X)
Overview
MPI for Python
PETSc for Python
Applications
Microfluidics (µ-TAS)
Microfluidics (µ-TAS)
mpi4py
◮ Development: http://mpi4py.googlecode.com
◮ Mailing List: [email protected]
◮ Chat: [email protected]
petsc4py
◮ Development: http://petsc4py.googlecode.com
◮ Mailing List: [email protected]
◮ Chat: [email protected]
Thanks!
