Managing HPC Software

Complexity with Spack

SC24 Tutorial
November 18, 2024

Todd Gamblin, Gregory Becker, Alec Scott,

Phil Sakievich, and Luke Peyralans

The most recent version of these slides can be found at:

https://spack-tutorial.readthedocs.io
Todd Gamblin Alec Scott
Greg Becker
LLNL LLNL
LLNL

Phil Sakievich Luke Peyralans

Sandia National Laboratories
University of Oregon
 Tutorial Materials

Find these slides and associated scripts here:

spack-tutorial.rtfd.io
We also have a chat room on Spack slack.
You can join here:

slack.spack.io
Join the #tutorial channel!
You can ask questions here after the conference is over.
Over 3,400 people can help you on Slack!

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 3

LLNL-PRES-806064
 Claim a VM instance at: bit.ly/spack-vms

If you’re in the spack2 column,

your login and password are
both spack2

ssh [email protected]

Claim a login by putting your name in the Google Sheet

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 4
LLNL-PRES-806064
 Agenda (approximate)

Morning Afternoon
Intro 8:30 am Software Stacks 1:30 pm
Basics Packaging
Concepts Break 3:00 pm
Break 10:00 am Developer Workflows 3:30 pm
Environments 10:30 am Mirrors & Binary Caches
Configuration Scripting
Lunch 12:00 pm End 5:00 pm

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 5

LLNL-PRES-806064
 We build codes from hundreds of small, complex pieces

Just when we’re starting to solve the problem of how to create software using reusable parts, it
founders on the nuts-and-bolts problems outside the software itself.
P. DuBois & T. Epperly. Why Johnny Can’t Build. Scientific Programming. Sep/Oct 2003.

▪ Component-based software development dates back to the 60’s

— M.D. McIlroy, Mass Produced Software Components. NATO SE Conf., 1968

▪ Pros are well known:

— Teams can and must reuse each others’ work
— Teams write less code, meet deliverables faster Build-time incompatibility; fail fast

▪ Cons:
— Teams must ensure that components work together
— Integration burden increases with each additional library
— Integration must be repeated with each update to components
— Components must be vetted!

▪ Managing changes over time is becoming intractable Appears to work; subtle errors later

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 6

LLNL-PRES-806064
 Modern scientific codes rely on icebergs of dependency libraries
71 packages
188 dependencies
MFEM: LBANN: Neural Nets for HPC
Higher-order finite elements
31 packages,
69 dependencies

r-condop:
R Genome Data Analysis Tools
179 packages,
527 dependencies
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 7
LLNL-PRES-806064
 ECP’s E4S stack is even larger than these codes

— Red boxes are the packages in it (about 100)

— Blue boxes are what else you need to build it (about 600)
— It’s infeasible to build and integrate all of this manually

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 8

LLNL-PRES-806064
 Modern software integrates open source
and internal packages
LLNL ARES

30 12 71
▪ Most modern software uses tons of open source

▪ We cannot replace all these OSS components with our own

— How do we put them all together effectively?
— Do you have to integrate this stuff by hand?

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 9

LLNL-PRES-806064
 Some fairly common (but questionable) assumptions
made by package managers (conda, pip, apt, etc.)

▪ 1:1 relationship between source code and binary (per platform)

— Good for reproducibility (e.g., Debian)
— Bad for performance optimization

▪ Binaries should be as portable as possible

— What most distributions do
— Again, bad for performance

▪ Toolchain is the same across the ecosystem

— One compiler, one set of runtime libraries
— Or, no compiler (for interpreted languages)

LLNL-PRES-806064
Outside these boundaries, Materials:
users are
Join #tutorial on Slack: slack.spack.io typically on their own
spack-tutorial.readthedocs.io 10
 High Performance Computing (HPC)
violates many of these assumptions
▪ Code is typically distributed as source Some Supercomputers
— With exception of vendor libraries, compilers

▪ Often build many variants of the same package

— Developers’ builds may be very different Summit Fugaku
— Many first-time builds when machines are new Oak Ridge National Lab RIKEN
Power9 / NVIDIA Fujitsu/ARM a64fx
▪ Code is optimized for the processor and GPU
— Must make effective use of the hardware Aurora
— Can make 10-100x perf difference NERSC-9
Perlmutter
Lawrence Berkeley Argonne National Lab
▪ Rely heavily on system packages Intel Xeon / Xe
— Need to use optimized libraries that come with machines National Lab
AMD Zen / NVIDIA
— Need to use host GPU libraries and network

▪ Multi-language
— C, C++, Fortran, Python, others
Lawrence Livermore
all in the same ecosystem National Lab
Oak Ridge National Lab
AMD Zen / Radeon AMD Zen / Radeon

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 11

LLNL-PRES-806064
 What about containers?

▪ Containers provide a great way to reproduce and distribute an

already-built software stack

▪ Someone needs to build the container!

— This isn’t trivial
— Containerized applications still have hundreds of dependencies

▪ Using the OS package manager inside a container is insufficient

— Most binaries are built unoptimized
— Generic binaries, not optimized for specific architectures

▪ HPC containers may need to be rebuilt to support many

different hosts, anyway.
— Not clear that we can ever build one container for all facilities
— Containers likely won’t solve the N-platforms problem in HPC

We need something more flexible to build the containers

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 12
LLNL-PRES-806064
 Spack enables Software distribution for HPC
• Spack automates the build and installation of scientific software

• Packages are parameterized, so that users can easily tweak and tune configuration
No installation required: clone and go
$ git clone https://github.com/spack/spack
$ spack install hdf5

Simple syntax enables complex installs

$ spack install [email protected] $ spack install [email protected] cppflags="-O3 –g3"
$ spack install [email protected] %[email protected] $ spack install [email protected] target=haswell
$ spack install [email protected] +threadssafe $ spack install [email protected] +mpi ^[email protected]
github.com/spack/spack

• Ease of use of mainstream tools, with flexibility needed for HPC

• In addition to CLI, Spack also:

• Generates (but does not require) modules
• Allows conda/virtualenv-like environments
• Provides many devops features (CI, container generation, more)

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 13

LLNL-PRES-806064
 What’s a package manager?

▪ Spack is a package manager

• Manages package installation
— Does not a replace Cmake/Autotools Package • Manages dependency relationships
— Packages built by Spack can have any Manager • May drive package-level build systems
build system they want

▪ Spack manages dependencies High Level • Cmake, Autotools

— Drives package-level build systems
Build • Handle library abstractions
— Ensures consistent builds • Generate Makefiles, etc.
System
▪ Determining magic configure lines
takes time Low Level • Make, Ninja
— Spack is a cache of recipes Build • Handles dependencies among
commands in a single build
System

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 14

LLNL-PRES-806064
 Who can use Spack?

People who want to use or distribute software for HPC!

1. End Users of HPC Software
— Install and run HPC applications and tools

2. HPC Application Teams

— Manage third-party dependency libraries

3. Package Developers
— People who want to package their own software for distribution

4. User support teams at HPC Centers

— People who deploy software for users at large HPC sites

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 15

LLNL-PRES-806064
 Spack is a Linux Foundation Project!

▪ What does that mean?

— Project has a legal 501(c)(6) non-profit company

— This is a neutral legal entity

• Can be in legal agreements (e.g. for distributing binaries)
• Can get discounts on, e.g., Slack!

— Project will have a Technical Steering Committee (TSC)

• Plan is to make the main developer meetings more public
• Also have official steering committee meetings
• Main charter is written (mostly boilerplate)
• Working on initial GOVERNANCE.md, initial TSC members

— Trademark (Spack name, logo) assigned to Linux Foundation

— Project resources owned by Linux Foundation

• spack.io website
• GitHub Organization

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 16

LLNL-PRES-806064
 Spack is part of the High Performance Software Foundation

▪ Join the BOF tomorrow!

— November 19
— 5:15pm – 6:45pm
— Room B309

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 17

LLNL-PRES-806064
 Fall
2016 2020

2024

2023 Spring Spring

2018 2021

Fall Fall
2018 2021

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 18

LLNL-PRES-806064
 Spack sustains the HPC software ecosystem
with the help of many contributors
Over 8,400 software packages
Over 1,460 contributors

2023 aggregate documentation user counts from GA4

(note: yearly user counts are almost certainly too large)

Contributors continue to grow worldwide!

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 19
LLNL-PRES-806064
 Spack’s widespread adoption has made it a de facto standard,
drawing contribution and collaboration from vendors
▪ AWS is investing significantly in cloud credits for Spack
— Supporting highly scalable cloud CI system with ~250k+/year in credits
— Integrating Spack with ParallelCluster product
— Joint Spack tutorial with AWS drew 125+ participants

▪ Google is using Spack in their HPC Toolkit cloud cluster product

— List packages to deploy; automatically built and cached in cluster deployment

▪ AMD has contributed ROCm packages and compiler support

— 55+ PRs mostly from AMD, also others
— ROCm, HIP, aocc packages are all in Spack now

▪ HPE/Cray is allowing us to do CI in the cloud for the Cray PE environment

— Looking at tighter Spack integration with Cray PE

▪ Intel contributing OneApi support and licenses for our build farm

▪ NVIDIA contributing NVHPC compiler support and other features

▪ Fujitsu and RIKEN have contributed a huge number of packages for ARM/a64fx support on Fugaku

▪ ARM and Linaro members contributing ARM support

— 400+ pull requests for ARM support from various companies

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 20

LLNL-PRES-806064
 One month of Spack development is pretty busy!

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 21

LLNL-PRES-806064
 Spack v0.23.0 is out!

This is the last major release before Spack v1.0 in June!

Highlights:
1. Language Runtimes: depends_on("c"|"cxx"|"fortran")
2. ABI splicing: Build binaries with mpich, deploy w/mvapich2, etc.
3. Broader variant propagation: ++shared, build_system==Release
4. Query by namespace with spack find namespace=myrepo
5. UI Improvements
• More concise output for spack spec
• Explore concretized environments with spack find -c
• spack -C <env> to use just an environment’s configuration
6. New commands: spack env track/untrack
7. ML binary stacks for Grace-Hopper
github.com/spack/spack
8. Devtools binary stack for macos
Full release notes: https://github.com/spack/spack/releases/tag/v0.23.0

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 22

LLNL-PRES-806064
 Spack is not the only HPC/AI/data science
package manager out there
1. “Functional” Package Managers
— Nix https://nixos.org
— Guix https://hpc.guix.info

2. Build-from-source Package Managers

— Homebrew, LinuxBrew https://brew.sh
— MacPorts https://www.macports.org
— Gentoo https://gentoo.org

Other tools in the HPC Space:

▪ Easybuild
— An installation tool for HPC https://easybuild.io
— Focused on HPC system administrators – different package model from Spack
— Relies on a fixed software stack – harder to tweak recipes for experimentation

▪ Conda / Mamba / Pixi https://conda.io

— Very popular binary package ecosystem for data science https://mamba.readthedocs.io
— Not targeted at HPC; generally has unoptimized binaries https://prefix.dev

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 23

LLNL-PRES-806064
 Claim a VM instance at: bit.ly/spack-vms

If you’re in the spack2 column,

your login and password are
both spack2

ssh [email protected]

Claim a login by putting your name in the Google Sheet

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 24
LLNL-PRES-806064
 Hands-on Time: Spack Basics

Follow script at spack-tutorial.readthedocs.io

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 25
 Core Spack Concepts

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 26
 Most existing tools do not support combinatorial versioning

▪ Traditional binary package managers

— RPM, yum, APT, yast, etc.
— Designed to manage a single stack.
— Install one version of each package in a single prefix (/usr).
— Seamless upgrades to a stable, well tested stack

▪ Port systems
— BSD Ports, portage, Macports, Homebrew, Gentoo, etc.
— Minimal support for builds parameterized by compilers, dependency versions.

▪ Virtual Machines and Linux Containers (Docker)

— Containers allow users to build environments for different applications.
— Does not solve the build problem (someone has to build the image)
— Performance, security, and upgrade issues prevent widespread HPC deployment.

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 27
 Spack provides a spec syntax to describe customized package
configurations
$ spack install mpileaks unconstrained
$ spack install [email protected] @ custom version
$ spack install [email protected] %[email protected] % custom compiler
$ spack install [email protected] %[email protected] +threads +/- build option
$ spack install [email protected] cppflags="-O3 –g3" set compiler flags
$ spack install [email protected] target=cascadelake set target microarchitecture
$ spack install [email protected] ^[email protected] %[email protected] ^ dependency constraints

▪ Each expression is a spec for a particular configuration

— Each clause adds a constraint to the spec
— Constraints are optional – specify only what you need.
— Customize install on the command line!

▪ Spec syntax is recursive

— Full control over the combinatorial build space

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 28
 Spack packages are parameterized using the spec syntax
Python DSL defines many ways to build
from spack import *
Base package
class Kripke(CMakePackage): (CMake support)
"""Kripke is a simple, scalable, 3D Sn deterministic particle transport mini-app."""

homepage = "https://computation.llnl.gov/projects/co-design/kripke" Metadata at the class level

url = "https://computation.llnl.gov/projects/co-design/download/kripke-openmp-1.1.tar.gz"

version(‘1.2.3’, sha256='3f7f2eef0d1ba5825780d626741eb0b3f026a096048d7ec4794d2a7dfbe2b8a6’)
version(‘1.2.2’, sha256='eaf9ddf562416974157b34d00c3a1c880fc5296fce2aa2efa039a86e0976f3a3’) Versions
version('1.1’, sha256='232d74072fc7b848fa2adc8a1bc839ae8fb5f96d50224186601f55554a25f64a’)

variant('mpi', default=True, description='Build with MPI.’) Variants (build options)

variant('openmp', default=True, description='Build with OpenMP enabled.’)
Dependencies
depends_on('mpi', when='+mpi’)
depends_on('[email protected]:', type='build’) (same spec syntax)
def cmake_args(self):
return [
'-DENABLE_OPENMP=%s’ % ('+openmp’ in self.spec),
'-DENABLE_MPI=%s' % ('+mpi’ in self.spec), Install logic
]
in instance methods
def install(self, spec, prefix):
mkdirp(prefix.bin)
install('../spack-build/kripke', prefix.bin) Don’t typically need install() for
CMakePackage, but we can work
One package.py file per software project! around codes that don’t have it.

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 29
 Conditional variants simplify packages
CudaPackage: a mix-in for packages that use CUDA

cuda is a variant (build option)

cuda_arch is only present

if cuda is enabled

dependency on cuda, but only

if cuda is enabled

constraints on cuda version

compiler support for x86_64

and ppc64le

There is a lot of expressive power in the Spack package DSL.

LLNL-PRES-806064
Join #tutorial on Slack: spackpm.herokuapp.com Materials: spack-tutorial.readthedocs.io 30
 Spack Specs can constrain versions of dependencies

$ spack install mpileaks %[email protected] ^[email protected]

▪ Spack ensures one configuration of each library per DAG

— Ensures ABI consistency.
— User does not need to know DAG structure; only the dependency names.

▪ Spack can ensure that builds use the same compiler, or you can mix
— Working on ensuring ABI compatibility when compilers are mixed.

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 31
 Spack handles ABI-incompatible, versioned interfaces like MPI

▪ mpi is a virtual dependency

▪ Install the same package built with two different MPI implementations:

$ spack install mpileaks ^[email protected] $ spack install mpileaks ^[email protected]:

▪ Let Spack choose MPI implementation, as long as it provides MPI 2 interface:

$ spack install mpileaks ^mpi@2

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 32
 Concretization fills in missing configuration details
when the user is not explicit.
mpileaks ^[email protected]+debug ^[email protected] User input: abstract spec with some constraints
spec.yaml
Normalize

spec:
- mpileaks:
arch: linux-x86_64
compiler:
name: gcc
version: 4.9.2
dependencies:
adept-utils: kszrtkpbzac3ss2ixcjkcorlaybnptp4
callpath: bah5f4h4d2n47mgycej2mtrnrivvxy77
mpich: aa4ar6ifj23yijqmdabeakpejcli72t3
hash: 33hjjhxi7p6gyzn5ptgyes7sghyprujh
variants: {}
version: '1.0'
- adept-utils:
arch: linux-x86_64
compiler:
Concretize Store name: gcc
version: 4.9.2
dependencies:
boost: teesjv7ehpe5ksspjim5dk43a7qnowlq
mpich: aa4ar6ifj23yijqmdabeakpejcli72t3
hash: kszrtkpbzac3ss2ixcjkcorlaybnptp4
variants: {}
version: 1.0.1
- boost:
arch: linux-x86_64
compiler:
name: gcc
version: 4.9.2
dependencies: {}
hash: teesjv7ehpe5ksspjim5dk43a7qnowlq
variants: {}
version: 1.59.0
...

Abstract, normalized spec Concrete spec is fully constrained Detailed provenance is stored
with some dependencies. and can be passed to install. with the installed package

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 33
 Hashing allows us to handle combinatorial complexity

Dependency DAG ▪ Each unique dependency graph is a

unique configuration.

▪ Each configuration in a unique directory.

— Multiple configurations of the same
package can coexist.
Installation Layout
opt Hash ▪ Hash of entire directed acyclic graph
└── spack
├── darwin-mojave-skylake
(DAG) is appended to each prefix.
│ └── clang-10.0.0-apple
├── bzip2-1.0.8-hc4sm4vuzpm4znmvrfzri4ow2mkphe2e
│
│
bzip2-1.0.8-hc4sm4vuzpm4znmvrfzri4ow2mkphe2e
├── python-3.7.6-daqqpssxb6qbfrztsezkmhus3xoflbsy
python-3.7.6-daqqpssxb6qbfrztsezkmhus3xoflbsy ▪ Installed packages automatically find
│
│
├── sqlite-3.30.1-u64v26igxvxyn23hysmklfums6tgjv5r
sqlite-3.30.1-u64v26igxvxyn23hysmklfums6tgjv5r
├── xz-5.2.4-u5eawkvaoc7vonabe6nndkcfwuv233cj
xz-5.2.4-u5eawkvaoc7vonabe6nndkcfwuv233cj
dependencies
│ └── zlib-1.2.11-x46q4wm46ay4pltriijbgizxjrhbaka6
zlib-1.2.11-x46q4wm46ay4pltriijbgizxjrhbaka6 — Spack embeds RPATHs in binaries.
├── darwin-mojave-x86_64 — No need to use modules or set
│ └── clang-10.0.0-apple
│ └── coreutils-8.29-pl2kcytejqcys5dzecfrtjqxfdssvnob
coreutils-8.29-pl2kcytejqcys5dzecfrtjqxfdssvnob LD_LIBRARY_PATH
— Things work the way you built them

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 34
 An isolated compilation environment allows Spack
to easily swap compilers
Spack ▪ Forked build process isolates environment for each build.
do_install()
Process Uses compiler wrappers to:
— Add include, lib, and RPATH flags
— Ensure that dependencies are found automatically
Install dep1 Install dep2 … Install package
— Load Cray modules (use right compiler/system deps)

icc icpc ifort

Fork
Build Set up environment
Process
CC = spack/env/spack-cc SPACK_CC = /opt/ic-15.1/bin/icc
Compiler wrappers
CXX = spack/env/spack-c++ SPACK_CXX = /opt/ic-15.1/bin/icpc (spack-cc, spack-c++, spack-f77, spack-f90)
F77 = spack/env/spack-f77 SPACK_F77 = /opt/ic-15.1/bin/ifort
FC = spack/env/spack-f90 SPACK_FC = /opt/ic-15.1/bin/ifort -I /dep1-prefix/include
-L /dep1-prefix/lib
PKG_CONFIG_PATH = ... PATH = spack/env:$PATH -Wl,-rpath=/dep1-prefix/lib
CMAKE_PREFIX_PATH = ...
LIBRARY_PATH = ...

install() configure make make install

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 35
 We can configure Spack to build with external software

mpileaks ^[email protected]+debug
^openmpi ^[email protected]

packages.yaml
packages:
mpi:
buildable: Fal se
paths:
[email protected] %[email protected] arch=linux-rhel6-ppc64:
/path/to/external /gcc/openmpi-2.0.0
[email protected] %gcc@ 4.7.3 arch=linux-rhel6-ppc64:
/path/to/external /gcc/openmpi-1.10.3
...

/path/to/external/gcc/openmpi-2.0.0

Users register external packages in a

configuration file (more on these later). Spack prunes the DAG when adding external packages.

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 36
 Spack package repositories allow stacks to be layered

LLNL MARBL multi-physics

application

marbl llnl.wci.mapp

blast miranda

MARBL Application Packages

Common internal
$ spack repo create /path/to/my_repo packages
leos llnl.wci

$ spack repo add my_repo

$ spack repo list
==> 2 package repositories. Open Source Spack axom
boost raja
my_repo /path/to/my_repo packages
builtin spack/var/spack/repos/builtin builtin

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 37
 Spack mirrors Original source
on internet
▪ Spack allows you to define mirrors:
OCI Registry
▪ Directories in the filesystem
▪ On a web server
▪ In an S3 bucket S3 Bucket

▪ Mirrors are archives of fetched tarballs, repositories,

and other resources needed to build Shared FS
▪ Can also contain binary packages

▪ By default, Spack maintains a mirror in

var/spack/cache of everything you’ve fetched so far.

▪ You can host mirrors internal to your site Spack

▪ See the documentation for more details Local cache users

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 38
 The concretizer includes information from Dependency solving
packages, configuration, and CLI is NP-hard
Contributors
package.py repository
• new versions
• new dependencies
• new constraints

concretizer
spack yaml default config
developers packages.yaml
admins, yaml local preferences config
users packages.yaml

users yaml local environment config

spack.yaml Concrete spec is
fully constrained
Command line constraints and can be built.
users
spack install [email protected] +debug

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 39
 We use logic programming to simplify package solving

▪ New concretizer leverages Clingo (see potassco.org)

▪ Clingo is an Answer Set Programming (ASP) solver

— ASP looks like Prolog; leverages SAT solvers for speed/correctness
— ASP program has 2 parts:
1. Large list of facts generated from our package repositories and config
2. Small logic program (~800 lines)
– includes constraints and optimization criteria

▪ New algorithm on the Spack side is conceptually simpler:

— Generate facts for all possible dependencies, send to logic program
— Optimization criteria express preferences more clearly
— Build a DAG from the results

▪ New concretizer solves many specs that old concretizer can’t

— Backtracking is a huge win – many issues resolved
— Conditional logic that was complicated before is now much easier
Some facts for the HDF5 package
LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 40
 --fresh only reuses builds if hashes match

1. Resolve metadata
▪ Hash matches are very
sensitive to small changes
2. Create per-node hashes

cwx4qwk4bkamf4gjrglmxfu3bhasyt74 ▪ In many cases, a satisfying

qo2af23r2npatxdtna3fmwkeennywixp
cached or already installed
k2yumgxwq6ijubivfpbjpmrrbzyqcoot
spec can be missed
4xxvh5ldm7gm32ngtixcm2odaer3cvvb

74mwnxgn6nujehpyyalhwizwojwn5zga
Package ▪ Nix, Spack, Guix, Conan,
??
cache and others reuse this way
6zvh4ueem6f5yrcfugh67k2hrtxbgbcs
3. Query for exact hash match

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 41
 --reuse (now the default) is more aggressive

• --reuse tells the solver about all the installed packages!

• Add constraints for all installed packages, with their hash as the associated ID:

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 42
 Telling the solver to minimize builds is
surprisingly simple in ASP
1. Allow the solver to choose a hash for any package:

2. Choosing a hash means we impose its constraints:

3. Define a build as something without a hash:

4. Minimize builds!

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 43
 With and without --reuse optimization

Pure hash-based reuse: all misses With reuse: 16 packages were reusable

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 44
 Use `spack spec` to see the results of concretization
$ spack spec mpileaks
Input spec
------------------------------
mpileaks

Concretized
------------------------------
[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected]+atomic+chrono+date_time~debug+filesystem~graph
~icu_support+iostreams+locale+log+math~mpi+multithreaded+program_options
~python+random +regex+serialization+shared+signals+singlethreaded+system
+test+thread+timer+wave arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected]~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm~verbs+vt arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected]+sigsegv arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected]~stat_dysect arch=darwin-elcapitan-x86_64
^libdwarf@20160507%[email protected] arch=darwin-elcapitan-x86_64
^[email protected]%[email protected] arch=darwin-elcapitan-x86_64

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 45
 Spack environments enable users to build customized stacks
from an abstract description
Simple spack.yaml file

Concretize Install

spack.yaml file spack.lock describes

describes Pacakge installations
exact versions installed
requirements

▪ spack.yaml describes project requirements Concrete spack.lock file (generated)

▪ spack.lock describes exactly what versions/configurations

were installed, allows them to be reproduced.

▪ Can be used to maintain configuration of a software stack.

— Can easily version an environment in a repository

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 46
 Environments,
spack.yaml and spack.lock

Follow script at spack-tutorial.readthedocs.io

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 47
 Hands-on Time: Configuration

Follow script at spack-tutorial.readthedocs.io

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 48
We’ll resume at:
16:30 CEST
Find the slides and associated scripts here:

spack-tutorial.rtfd.io

Remember to join Spack slack so you can get help later!

slack.spack.io
Join the #tutorial channel!
 Hands-on Time:
Stacks
Follow script at spack-tutorial.readthedocs.io

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 50
 Spack packages are parameterized using the spec syntax
Python DSL defines many ways to build
from spack import *
Base package
class Kripke(CMakePackage): (CMake support)
"""Kripke is a simple, scalable, 3D Sn deterministic particle transport mini-app."""

homepage = "https://computation.llnl.gov/projects/co-design/kripke" Metadata at the class level

url = "https://computation.llnl.gov/projects/co-design/download/kripke-openmp-1.1.tar.gz"

version(‘1.2.3’, sha256='3f7f2eef0d1ba5825780d626741eb0b3f026a096048d7ec4794d2a7dfbe2b8a6’)
version(‘1.2.2’, sha256='eaf9ddf562416974157b34d00c3a1c880fc5296fce2aa2efa039a86e0976f3a3’) Versions
version('1.1’, sha256='232d74072fc7b848fa2adc8a1bc839ae8fb5f96d50224186601f55554a25f64a’)

variant('mpi', default=True, description='Build with MPI.’) Variants (build options)

variant('openmp', default=True, description='Build with OpenMP enabled.’)
Dependencies
depends_on('mpi', when='+mpi’)
depends_on('[email protected]:', type='build’) (same spec syntax)
def cmake_args(self):
return [
'-DENABLE_OPENMP=%s’ % ('+openmp’ in self.spec),
'-DENABLE_MPI=%s' % ('+mpi’ in self.spec), Install logic
]
in instance methods
def install(self, spec, prefix):
mkdirp(prefix.bin)
install('../spack-build/kripke', prefix.bin) Don’t typically need install() for
CMakePackage, but we can work
One package.py file per software project! around codes that don’t have it.

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 51
 Hands-on Time: Creating Packages

Follow script at spack-tutorial.readthedocs.io

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 52
 Get a VM here →
We’ll resume at:
10:00 PT
Find the slides and associated scripts here:

spack-tutorial.rtfd.io

Remember to join Spack slack so you can get help later!

slack.spack.io
Join the #tutorial channel!
 Hands-on Time:
Binary Caches and Mirrors
Follow script at spack-tutorial.readthedocs.io

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 54
 Hands-on Time:
Developer Workflows
Follow script at spack-tutorial.readthedocs.io

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 55
 Hands-on Time:
Scripting
Follow script at spack-tutorial.readthedocs.io

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 56
 More Features
and the Road Ahead

LLNL-PRES-806064
Join #tutorial on Slack: spackpm.herokuapp.com Materials: spack-tutorial.readthedocs.io 57
 Environments have enabled us to add build many features to
support developer workflows

spack external find class Libsigsegv(A utot oolsPackage, G NUMir ro rPackage):

"""G NU lib sigsegv is a librar y fo r handling page faults in u ser mo de."""

Automatically find and configure external packages on the system # ... spack package con tents ...

extr a_install_tests = ‘tes ts /.libs ’

def test(s elf):

data_dir =s elf.test_suit e.cur ren t_test_dat a_dir
s moke_tes t_c = dat a_d ir.join(‘s moke_tes t.c’)

s elf.r un_test(
'cc’, [
'-I%s' % s elf.pr efix.include,
'-L%s' % s elf.pr efix.lib, '-ls igs egv’,
s moke_tes t_c,

spack test ]
'-o', 'sm oke_test '

pur pos e='check linking’)

package.py spack.yaml configuration Packages know how to run their own test suites s elf.r un_test(
‘s moke_tes t’, [], dat a_dir.join('s moke_tes t.out ’),
pur pos e=‘run built smoke tes t’)

s elf.r un_test('sigs egv1': ['Test pass ed’], pur pos e='check s igs egv1o utput ’)
s elf.r un_test('sigs egv2': ['Test pass ed’], pur pos e='check s igs egv2o utput ’)

package.py

spack ci
Automatically generate parallel build pipelines
(more on this later)

.gitlab-ci.yml CI pipeline spack containerize

Turn environments into container build recipes
spack.yaml

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 58
 Spack environments are the foundation of Spack CI

▪ spack ci enables any environment to be turned

into a build pipeline

▪ Pipeline generates a .gitlab-ci.yml file from

spack.lock

▪ Pipelines can be used just to build, or to

generate relocatable binary packages
— Binary packages can be used to keep the same
Parallel GitLab build pipeline
build from running twice
spack.yaml
▪ Same repository used for spack.yaml can
generate pipelines for project

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 59
 We’ve made a lot of progress on compiler dependencies

▪ Compiler runtime libraries represented in the graph

— C++, Fortran runtimes

▪ libc is now represented in dependency graphs on Linux

— No more need to rely on OS tag for compatibility information

▪ Reuse binaries without compiler needing to be configured locally

▪ Improved buildcache hit rate using libraries for compatibility

▪ Compiler dependency solver is working

— Needs a bit more performance tuning!

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 60
 Compilers can now model their own runtimes

▪ New method, runtime_constraints,

for injecting runtimes into graphs

▪ Currently supported for gcc,

intel-oneapi
— still working on others

▪ Allows solver to take libstdc++,

fortran rutime compatibility into
account.

▪ Example:
— Intel compilers now (correctly)
depend on gcc-runtime

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 61
 Packages now declare the languages they depend on
▪ Languages are almost virtuals
— HDF5 package depends on cxx and fortran
— Handled specially internally, until compilers are
nodes

▪ Imply a compiler and compiler package can

specify runtime libraries to inject

▪ Allows solver to mix compilers correctly

— Runtimes are unified like other nodes
— Package authors can model toolchain properties
— probably not for most package authors, but very
powerful

▪ TBD:
— Other runtimes like clang libraries and OpenMP
— Compilers as nodes in the graph

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 62
 We’ve also added libc as a dependency
▪ libc is a virtual
— glibc and musl packages are providers
— (nearly) every graph has libc in it, via
the compiler
— Can be external or built by Spack

▪ We are not building libc for every stack in

Spack
— Automatically detect system libc version
— Add a node to the graph to be used for
binary compatibility

▪ No longer using OS tags for buildcaches

— Now use libc for this
— many more buildcache hits

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 63
 Libc modeling makes for a much better buildcache
experience
▪ Currently on develop (emacs 100% from binary):

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 64
 We’ve made a lot of progress on compiler dependencies

▪ Compiler runtime libraries represented in the graph

— C++, Fortran runtimes

▪ libc is now represented in dependency graphs on Linux

— No more need to rely on OS tag for compatibility information

▪ Reuse binaries without compiler needing to be configured locally

▪ Improved buildcache hit rate using libraries for compatibility

▪ Compiler dependency solver is working

— Needs a bit more performance tuning!

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 65
 What’s needed for Spack v1.0?

Big things we’ve wanted for 1.0 are:

— New concretizer
— production CI Done!
— production public build cache
— Compiler dependencies Try v1.0.0-alpha1!
— Buildcache hardening
— Separate package repository
June 2025!
• Needs a stable package API

LLNL-PRES-837654
We are looking forward to a 1.0 release in June!
github.com/spack/spack 66
 But wait! There’s more!

Join us during SC24 Join us after SC24

— Join us and 3,400+ others on Spack slack
Tuesday
— Contribute packages, docs, and features on GitHub
HPSF BOF
— Continue the tutorial at spack-tutorial.rtfd.io
5:15pm – 6:45pm
B309 Star us on GitHub!
slack.spack.io github.com/spack/spack
Thursday
@spackpm.bsky.social
Spack BOF
12:15pm – 1:15pm @[email protected]
B310 @spackpm

We hope to make distributing & using HPC software easy!

Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 67
LLNL-PRES-806064
Disclaimer
This document was prepared as an account of work sponsored by an agency of the United States government. Neither
the United States government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any
warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or
usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe
privately owned rights. Reference herein to any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or
favoring by the United States government or Lawrence Livermore National Security, LLC. The views and opinions of
authors expressed herein do not necessarily state or reflect those of the United States government or Lawrence
Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.
Roadmap:
Separate concretization of build dependencies
spack install pkg1 %intel
▪ We want to:
— Build build dependencies with the "easy"
compilers 1
B L
— Build rest of DAG (the link/run dependencies)
with the fancy compiler B
2 5
▪ 2 approaches to modify concretization:
B L B L
1. Separate solves
• Solve run and link dependencies first
• Solve for build dependencies separately
• May restrict possible solutions (build → 3 4 6 7
run env constraints) R
2. Separate models R
• Allow a bigger space of packages in the solve “Easy” compiler
• Solve all runtime environments together 8
• May explode (even more) combinatorially Fancy compiler
B: build L: link R: run
LLNL-PRES-837654
github.com/spack/spack 69
 Roadmap:
Compilers as dependencies
▪ Need separate concretization of build dependencies
to make this work Compiler-imposed
1
— Model compiler as build dep (not unified) dependency
L
2
— Runtimes as link deps (unified) B
B
— Ensure compatibility between runtimes when using
multiple compilers together
intel@16 intel@17
L
L
▪ We need deeper modeling of compilers to handle R
R
compiler interoperability [email protected]
— libstdc++, libc++ compatibility
gcc@xxx
— Compilers that depend on compilers
— Linking executables with multiple compilers
libstdc++
▪ Packages that depend on languages
— Depend on cxx@2011, cxx@2017, fortran@1995, etc Compilers and runtime libs fully modeled
— Depend on [email protected], other compiler features as dependencies
— Model languages, openmp, cuda, etc. as virtuals

LLNL-PRES-837654
github.com/spack/spack 70
 Hands-on Time:
Modules
Follow script at spack-tutorial.readthedocs.io

LLNL-PRES-806064
Join #tutorial on Slack: slack.spack.io Materials: spack-tutorial.readthedocs.io 71