Faster Container Image Distribution on

a Variety of Tools with Lazy Pulling

Kohei Tokunaga, NTT Corporation
Tao Peng, Ant Group
Pull is time-consuming

● Large images speeds down cold start of containers

● Build takes long if base images are large

● Not all images are minimizable e.g. language runtimes, frameworks, ..

pulling packages accounts for 76% of container start time,

but only 6.4% of that data is read [Harter et al. 2016]
[Harter et al. 2016] Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau. "Slacker: Fast Distribution with Lazy Docker Containers".
14th USENIX Conference on File and Storage Technologies (FAST ’16). February 22–25, 2016, Santa Clara, CA, USA
Problem on the current OCI image

container can’t start before the all layers become locally available

Image is a set of tar (+compression) layers

bin/bash
● Non seekable
bin/ls
• Need to scan the entire blob even for
etc/passwd extracting single file entry
etc/group
usr/bin/apt
● No parallel extraction
• Need to scan sequentially
・・・
OCI-alternative accelerated images and lazy pulling

Lazy pulling:
Starting up containers without waiting for the pull completion

● eStargz
• Lazy pullable format with prefetch optimization + content verification
• Proposed as a backward-compatible extension to OCI Image Spec

● Nydus
• Lazy pullable format with prefetch, chunk dedup and e2e data integrity
• Compatible with OCI Distribution Spec and Artifacts Spec
• Proposed as OCI “v2” format (incompatible to current OCI Image Spec)
 eStargz
Kohei Tokunaga, NTT Corporation
eStargz: Standard-compatible lazy pulling
● 100% OCI-compatible
• Lazy pullable from standard registries (ghcr.io, docker.io, …etc)
• Even legacy (lazy-pulling-agnostic) runtime can run eStargz

● Usable with a variety of tools (tracker https://github.com/containerd/stargz-snapshotter/issues/258)

• Kubernetes, k3s, containerd, CRI-O, Podman, BuildKit, Kaniko, ko, buildpacks.io, go-containerregistry…

● Performance optimization and content verification

• Important files can be prefetched to avoid NW overhead
• Each chunk comes with checksum

Standard Container Registry

Image builders/convertes eStargz-aware runtimes
legacy
eStargz
legacy
ctr-remote
nerdctl
Kaniko
Lazy pull BuildKit
etc… creates etc…
eStargz
still runnable on
eStargz-agnostic tools (e.g. Docker)
eStargz image layer format
● Discussed in Stargz Snapshotter of containerd: https://github.com/containerd/stargz-snapshotter

● Compatible to gzip = usable as a valid OCI/Docker image layer

● Based on stargz by CRFS (https://github.com/google/crfs)

• eStargz comes with performance optimization and content verification
stargz eStargz

gzip member bin/bash bin/bash Prioritized files

per regular file/chunk bin/ls entrypoint.sh Prefetched by a single Range Request
can be extracted per-file bin/ls
・・・
using HTTP Range Request Files fetched on demand
entrypoint.sh ・・・
Downloaded in background as well
usr/bin/apt usr/bin/apt
・・・ ・・・ TOC(Table Of Contents) and footer
Files metadata, payload offset, etc…
For more details: https://github.com/containerd/stargz-snapshotter/blob/master/docs/stargz-estargz.md
Benchmarking result

[sec]

● Method based on Hello Bench [Harter, et al. 2016]

Waits for prefetch completion ● Takes 95 percentile of 100 operations
during create ● Host: EC2 Oregon (m5.2xlarge, Ubuntu 20.04)
● Registry: GitHub Container Registry (ghcr.io)
● Runtime: containerd
● Stargz Snapshotter commit: 7f45f7438617728dd06bc9853afb5e42c1d3d5a3
Benchmarking result

● Benchmarking spec is the same as the previous slide

Lazy pulling on Kubernetes
● CRI runtimes + plugins (discussed later) enable lazy pulling on Kubernetes
• Containerd + Stargz Snapshotter
• CRI-O + Stargz Store
● Real-world use-case at CERN for speeding up analysis pipeline [1] (13x faster pull for 5GB image)
[1] Ricardo Rocha & Spyridon Trigazis, CERN. “Speeding Up Analysis Pipelines with Remote Container Images”. KubeCon+CloudNativeCon 2020 NA. https://sched.co/ekDj

● k3s supports lazy pulling of eStargz (merged to the main, will be included in k3s v1.22)
$ k3s server --snapshotter=stargz

● Lazy-pull-enabled KinD image is available on ghcr.io/stargz-containers repo

$ kind create cluster --name estargz-demo --image ghcr.io/stargz-containers/estargz-kind-node:0.8.0

Containerd Node CRI-O Node

Container Registry
kubelet kubelet

eStargz

gRPC API exposed on dir

Lazy pull
Stargz
Stargz Store
Snapshotter

external process external process

Lazy pulling on containerd
● Stargz Snapshotter plugin : https://github.com/containerd/stargz-snapshotter
• Implements remote snapshotter plugin interface

● nerdctl (Docker-compatible CLI for containerd) supports lazy pulling

• https://github.com/containerd/nerdctl

$ nerdctl --snapshotter=stargz run ghcr.io/stargz-containers/python:3.9-esgz

$ nerdctl --snapshotter=stargz compose -f docker-compose.stargz.yaml up

Node
Container Registry

container
eStargz
proc

Stargz
Lazy pull Snapshotter

Mounts rootfs as FUSE

Fetches files/chunks on demand
Lazy pulling on Podman/CRI-O

● Stargz Store plugin : https://github.com/containerd/stargz-snapshotter

• Developped in Stargz Snapshotter project
• Implements Additional Layer Store plugin
• Podman >= v3.3.0, CRI-O >= v1.22.0
$ podman run ghcr.io/stargz-containers/python:3.9-esgz

Node
Container Registry
container
Podman, CRI-O
eStargz
proc
Stargz Store

Lazy pull

Provides layers Uses layers for rootfs

(mounted as FUSE)
Lazy pulling on BuildKit
● Experimentally supports lazy pulling of base images since v0.8.0
● Can shorten the time of build that require pull
• e.g. on temporary (and fresh) CI instances
$ docker buildx create --use --name lazy-builder \
--driver docker-container --driver-opt image=moby/buildkit:master \
--buildkitd-flags '--oci-worker-snapshotter=stargz'
$ docker buildx inspect --bootstrap lazy-builder
$ docker buildx build .

Container Registry Build on node

eStargz FROM ghcr.io/stargz-containers/golang:1.15.3-esgz as dev

COPY ./hello.go /hello.go

Lazy pull RUN go build -o hello /hello.go COPY and RUN without waiting for the pull completion

golang:1.15.3-esgz Fetch files/chunks on demand

• /usr/local/go/bin/go
• /usr/local/go/src/fmt/…
etc...

More details at blog: https://medium.com/nttlabs/buildkit-lazypull-66c37690963f

Building eStargz
BuildKit: https://github.com/moby/buildkit
● Building eStargz supported on the main branch, hope to come in v0.10.x
● eStargz is supported by Buildx or standalone BuildKit (buildctl + buildkitd)
● Docs: https://github.com/moby/buildkit/blob/master/docs/stargz-estargz.md#creating-stargzestargz-images
$ docker buildx build \
-o type=registry,name=ktokunaga/hello:esgz,oci-mediatypes=true,compression=estargz .

Kaniko: https://github.com/GoogleContainerTools/kaniko
● Image builder runnable in containers and Kubernetes
● Requires GGCR_EXPERIMENT_ESTARGZ=1
● Base images need to be eStargz

$ docker run --rm -e GGCR_EXPERIMENT_ESTARGZ=1 \

-v /tmp/context:/workspace -v ~/.docker/config.json:/kaniko/.docker/config.json:ro \
gcr.io/kaniko-project/executor:v1.6.0 --destination "ghcr.io/ktock/sample:esgz"
Converting an image into eStargz
ctr-remote: https://github.com/containerd/stargz-snapshotter/blob/v0.7.0/docs/ctr-remote.md
● CLI for containerd provided by Stargz Snapshotter project
● Supports prefetch optimization for eStargz
$ ctr-remote image optimize --oci ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz

nerdctl: https://github.com/containerd/nerdctl/blob/v0.11.1/docs/stargz.md
● Docker-compatible CLI for containerd
● Can be combined with nerdctl build command
$ nerdctl image convert --estargz --oci ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz

go-containerregistry and crane CLI : https://github.com/google/go-containerregistry

● Library and CLI to interact with registries
● requires GGCR_EXPERIMENT_ESTARGZ=1
● downstream tools (e.g.Kaniko, ko and buildpacks.io) supports eStargz creation as well
$ go install github.com/google/go-containerregistry/cmd/crane@latest
$ GGCR_EXPERIMENT_ESTARGZ=1 crane optimize ghcr.io/ktock/foo:1 ghcr.io/ktock/foo:1-esgz
Discussion toward OCIv1 extension
https://github.com/opencontainers/image-spec/issues/815

● eStargz is now widely usable on tools in the community

● Proposed to OCI Image Spec (discussion is on-going)

● Proposed as backward-compatible extensions

• Extension for the current gzip layer ( application/vnd.oci.image.layer.v1.tar+gz )
• Additional annotation for content verification: org.opencontainers.image.toc.digest
Nydus Image Service
Tao Peng, Ant Group
Nydus: Improved Lazy Pulling and More
● Standard lazy pulling with prefetch policies
• Lazy pullable from standard registries (cloud registries, docker registry, harbor, etc)
• Semantical prefetch, hinted prefetch, background prefetch all data
● Chunk level deduplication with layered build cache
• Speed up both image conversion and downloading
● End-to-end data integrity
• Runtime data integrity check
● Reproducible Image Building
• Build environment agnostic
● Compatible with OCI Distribution Spec and Artifacts Spec
• Usable with most existing container registry implementations
● Rich container ecosystem integration
• Kubernetes, docker, containerd, buildkit, harbor, dragonfly, runc, kata-containers etc
● Resource efficient and production ready
• Large scale deployment at Ant and available via Alibaba Cloud services
FUSE/virtiofs Unified Architecture
● Native container runtime support
• One translation layer (FUSE or virtiofs) for both runc and kata containers
● Shared uncompressed local cache
• Download/uncompress once and use all the time
● Singleton mode with extremely low memory footprint
• ~5MB per instance
Registry Acceleration File System
● Merkle tree metadata layer
• self-verifiable
● Chunk-shared data layer
• enabling chunk level deduplication
OCI Spec Compatibility

● Metadata/data layers are OCI manifest layers

● New media type in the image manifest spec
● Compatible with OCI artifacts/distribution spec
● Compatible registry GC functionality
● Widely working with container registries
• cloud registries
• docker registry
• harbor
• etc.
Lazy Pulling Benchmark
Lazy Pulling with Variety of Tools
● Nydus Snapshotter plugin
• https://github.com/dragonflyoss/image-service/tree/master/contrib/nydus-sna
pshotter

● Usable for variety of container management tools

• kubernetes (kubelet)
• crictl
• nerdctl
• ctr-remote
Manual Image Conversion -- nydusify
● nydusify
• https://github.com/dragonflyoss/image-service/tree/master/contrib/nydusify

• download image from remote registry, convert it to nydus image, and push nydus
image to remote registry

• nydusify convert --source <registry>/<repo>:<tag> \

--target <registry>/<repo>:<nydus-tag>
Manual Image Conversion -- buildkit
● buildkit
• https://github.com/moby/buildkit/pull/2045

• build nydus image directly from dockerfile and push it to remote registry

• buildctl build --frontend=dockerfile.v0 \

--local context=/build/dir \
--local dockerfile=/build/DockerfileDir \
--output type=nydus,name=<registry>/<repo>:<tag>
Automatic Conversion with Harbor

● https://github.com/goharbor/community/pull/167

● New image conversion service in harbor

● Started image acceleration working group

● Image format agnostic

• nydus
• estargz
• future more format
 OCI Artifacts Manifest

● https://github.com/opencontainers/artifacts/pull/29

● Map OCI image manifest to artifact manifest, to

connect artifacts to images

● Beneficial to SBOM, Signatures, Nydus, Scan Results

etc.

● Nydus image as an artifact type

• cncf.nydus.v1-rc1
Community V2 Image Spec Requirements
● OCI community requirements on V2 Image format
• https://hackmd.io/@cyphar/ociv2-brainstorm
Recap

● OCI images are large and slow

● Image lazy pulling is important for starting containers quickly
● Prefetching is a good friend to lazy pulling
● Ecosystem adoption is crucial for new image formats

● eStargz
• Backward compatibility
• Extension to the existing OCI image spec

● Nydus
• Future looking
• Proposal to the next generation OCI image spec
eStargz Project Information

● 100% OCI-compatible image format for lazy pulling

● Subproject of CNCF graduated containerd

● github： https://github.com/containerd/stargz-snapshotter

● slack： https://slack.containerd.io/ (#containerd-dev channel at CNCF slack)

● Pre-converted images are available on ghcr.io/stargz-containers

Nydus Project Information

● Image acceleration service with various improvements

● CNCF incubator dragonfly sub-project

● github： https://github.com/dragonflyoss/image-service

● slack： https://tinyurl.com/nydus-slack

● tutorial： https://tinyurl.com/nydus-tutorial
Thank You
&
Questions!