Tutorial setup

If you have not done the prior sections, you’ll need to start the docker image:

docker run -it ghcr.io/spack/tutorial:sc25

and then set Spack up like this:

git clone --depth=2 --branch=releases/v1.1 https://github.com/spack/spack
. spack/share/spack/setup-env.sh
spack repo update builtin --tag v2025.11.0
spack tutorial -y
spack bootstrap now
spack compiler find

See the Basic Installation Tutorial for full details on setup. For more help, join us in the #tutorial channel on Slack – get an invitation at slack.spack.io

Binary Caches Tutorial

This section covers how to share Spack-built binaries across machines and users using build caches.

Spack supports a range of storage backends for build caches: an ordinary filesystem, Amazon S3, Google Cloud Storage, and any OCI-compatible container registry (Docker Hub, GitHub Packages, a local docker registry, and so on). We begin with the filesystem mirror that the tutorial container is already configured to use, and then move on to OCI registries, which carry the additional property that the same artifacts can be used as runnable container images.

julia is the running example throughout. It has a non-trivial dependency in llvm and an interactive REPL that makes it easy to confirm an installation works.

Installing julia from the build cache

The tutorial container ships a pre-populated filesystem build cache, registered in the basics section as the signed mirror named tutorial. Because that mirror is already active, julia and its dependencies are available as binaries without any further configuration.

Create an environment with a view and add julia to it:

$ mkdir ~/myenv && cd ~/myenv
$ spack env create --with-view view .
$ spack -e . add julia

Note

The terms mirror and build cache are used almost interchangeably, since every build cache is a binary mirror. Source mirrors also exist but are not covered in this tutorial.

Install the environment:

$ spack -e . install
...
[+] dkvv7m3 julia@1.12.6 /home/spack/spack/opt/spack/linux-x86_64_v3/julia-1.12.6-dkvv7m3wqj5xn2lphsc6tay8fbiruoze (3s)

Both julia and every transitive dependency, including llvm, are fetched and relocated from the tutorial mirror; nothing is built from source. Given a build cache, the concretizer prefers concrete specs for which binaries already exist.

Confirm that the executable works through the environment’s view:

$ ./view/bin/julia -e 'println(1 + 1)'
2

Note

Build caches can be used across different Linux distributions. The concretizer reuses specs that have a host-compatible libc (e.g. glibc or musl), and binaries built with gcc carry their compiler runtime libraries as a separate dependency, so users do not need to install a compiler first.

Setting up a local OCI build cache

The previous section consumed binaries from a build cache. This section covers publishing binaries to one, using an OCI container registry as the backend.

OCI registries are useful in this role because the same artifacts can serve both as a Spack build cache and as runnable container images. OCI registries in common use include Docker Hub, GitHub Container Registry (GHCR), and Amazon ECR. For this tutorial we run a registry locally, which avoids authentication:

$ docker run -d --rm -p 5000:5000 --name registry registry

This is the official registry image from Docker Hub. It serves an empty OCI registry on http://localhost:5000.

Add it as a second mirror:

$ spack -e . mirror add --unsigned my-registry oci+http://localhost:5000/buildcache

The URL has three parts:

  • oci+http://: an OCI registry over plain HTTP, without TLS. A remote registry would normally use oci:// (HTTPS).

  • localhost:5000: the registry’s host and port.

  • /buildcache: the image name under which Spack publishes all artifacts.

The environment’s spack.yaml now contains the mirror:

mirrors:
  my-registry:
    url: oci+http://localhost:5000/buildcache
    signed: false

The same configuration works against a hosted registry such as GHCR or Docker Hub by switching to an oci:// URL and supplying credentials with --oci-username and --oci-password-variable.

Pushing to the OCI build cache

Push the environment to the local registry:

$ spack -e . buildcache push --without-build-dependencies my-registry
==> Selected 62 specs to push to oci+http://localhost:5000/buildcache
==> Checking for existing specs in the buildcache
==> [1/62] Pushed gcc-runtime@11.5.0/rcxunjn: sha256:c9362f6244e3... (0.08s, 125.76 MB/s)
...
==> [62/62] Pushed julia@1.12.6/dkvv7m3: sha256:f8863558da07... (0.21s, 92.40 MB/s)
==> Uploading manifests
==> [1/62] Tagged gcc-runtime@11.5.0/rcxunjn as localhost:5000/buildcache:gcc-runtime-11.5.0-rcxunjnw4voqkt5zieeerei767e4s7py.spack
...
==> [62/62] Tagged julia@1.12.6/dkvv7m3 as localhost:5000/buildcache:julia-1.12.6-dkvv7m3wqj5xn2lphsc6tay8fbiruoze.spack

Two things about this invocation are worth noting.

The --without-build-dependencies flag is passed because julia was installed from a binary cache, so build-only dependencies like cmake are not present on this machine. Without the flag, Spack would report that those packages are not installed.

All artifacts live under the single image name localhost:5000/buildcache. Spack auto-generates one tag per spec, of the form <name>-<version>-<hash>.spack, including the package hash so that each spec resolves to a distinct tag.

Re-running the push detects that nothing needs to be uploaded:

$ spack -e . buildcache push --without-build-dependencies my-registry
==> Selected 62 specs to push to oci+http://localhost:5000/buildcache
==> Checking for existing specs in the buildcache
==> All specs are already in the buildcache. Use --force to overwrite them.

Reinstalling from the OCI build cache

So far julia could have come from either mirror. To confirm that the OCI registry works as a build cache on its own, disable the filesystem tutorial mirror by changing mirrors: to mirrors:: in spack.yaml. The trailing :: replaces, rather than extends, the mirrors inherited from Spack’s global configuration:

mirrors::
  my-registry:
    url: oci+http://localhost:5000/buildcache
    signed: false

Reinstall julia with --overwrite. Only julia is reinstalled; its dependencies remain installed and are not refetched. The default installer output does not report where a package comes from, so pass -v to surface the fetch:

$ spack -e . install -v --overwrite -y julia
[ ] dkvv7m3 julia@1.12.6 fetching from build cache (0s)
==> Fetching http://localhost:5000/v2/buildcache/blobs/sha256:f8863558da07...
==> Fetching http://localhost:5000/v2/buildcache/blobs/sha256:4af76428dc77...
[ ] dkvv7m3 julia@1.12.6 relocating (1s)
[+] dkvv7m3 julia@1.12.6 /home/spack/spack/opt/spack/linux-x86_64_v3/julia-1.12.6-dkvv7m3wqj5xn2lphsc6tay8fbiruoze (3s)

Two blobs are fetched per spec: a JSON manifest and the binary tarball. OCI registries are content-addressed, hence the sha256:... identifiers rather than human-readable filenames.

Creating runnable container images

So far the OCI registry has been used only as a Spack build cache. Since the artifacts are also valid OCI images, they can be pulled directly with docker.

Consider what happens when running an image without a base image:

$ docker run --rm localhost:5000/buildcache:julia-1.12.6-dkvv7m3wqj5xn2lphsc6tay8fbiruoze.spack julia -e 'println(1 + 1)'
exec /home/spack/spack/opt/spack/linux-x86_64_v3/julia-1.12.6-dkvv7m3.../bin/julia: no such file or directory

The run fails because the layers we pushed contain the Spack-built artifacts but not the host’s glibc, which Spack always treats as an external package. Without a base image the container has no /lib directory at all, which produces the error above.

The resolution is to push again with --base-image pointing at a minimal distribution that provides a compatible glibc:

$ spack -e . buildcache push --force --without-build-dependencies \
      --base-image ubuntu:24.04 my-registry

The base image’s libc must be at least as new as the one used at build time, otherwise the binaries fail at runtime with errors of the form version `GLIBC_2.38' not found. The distribution itself need not match. archlinux:latest, for example, ships a sufficiently recent glibc while providing a different userland (pacman instead of apt, and so on).

The image now runs:

$ docker run --rm localhost:5000/buildcache:julia-1.12.6-dkvv7m3wqj5xn2lphsc6tay8fbiruoze.spack julia -e 'println(1 + 1)'
2

In addition to glibc, the base image provides a shell and the standard utilities.

Spack environments as container images

The preceding section produced one image per package. For most uses a single image containing the full environment is more convenient.

Add a text editor to the environment, so that the image can both edit and run Julia code:

$ spack -e . install --add vim

Note

With the tutorial mirror disabled, vim is built from source. Change mirrors:: back to mirrors: first to install it from the tutorial cache; either way the build is quick.

Pass --tag to assign the environment image a human-readable name:

$ spack -e . buildcache push --without-build-dependencies \
      --base-image ubuntu:24.04 \
      --tag julia-and-vim \
      my-registry
...
==> Tagged localhost:5000/buildcache:julia-and-vim

Spack publishes each package as its own image layer. Layers are shared between image tags, so the combined image takes almost no extra storage. Unlike Docker, where each RUN line creates a layer that depends on the previous one, Spack package layers are independent and can be combined in any order.

Run the combined image:

$ docker run -it --rm localhost:5000/buildcache:julia-and-vim
root@f53920f8695a:/# vim example.jl    # write some Julia code
root@f53920f8695a:/# julia example.jl  # and run it

Both julia and vim are immediately usable because Spack writes PATH into the image’s environment, pointing at each package’s install prefix. Spack does not materialize the environment’s view inside the image; the packages live at their original Spack prefixes, identical to those on the host.

Relation to docker build workflows

In earlier versions of Spack the common practice was to generate a Dockerfile from a Spack environment using spack containerize and then build the image with docker build:

FROM <base image> AS build
COPY spack.yaml /root/env/spack.yaml
RUN spack -e /root/env install

FROM <base image>
COPY --from=build /opt/spack/opt /opt/spack/opt

This approach still works and spack containerize is still available, but it has several drawbacks:

  • If RUN spack -e /root/env install fails, Docker discards the whole layer, including any successfully built dependencies. Troubleshooting typically requires starting from scratch in a docker run session.

  • Some CI environments cannot run docker build safely — for example, when the CI script itself runs inside a container (“Docker-in-Docker”).

The OCI build cache approach decouples the three responsibilities that docker build combines: build isolation, running the build, and producing an image. spack install can be run in any environment (host, sandbox, container), and spack buildcache push then turns the result into images.

Relocation

Spack installs packages under an arbitrary prefix, typically ~/spack/opt/spack/.... This is more flexible than most package managers, but it also means that binaries contain absolute paths to machine-specific locations, which must be rewritten when the binary is reinstalled elsewhere.

Spack does this rewriting automatically when installing from a binary cache. When producing binaries that are meant to be redistributed, one constraint applies: Spack can only relocate paths in a binary if the target prefix is no longer than the prefix used at build time.

The reason is that absolute paths typically reside in the binary’s string table — a list of null-terminated strings referenced by offset. Strings can be edited in place, but they cannot grow without overwriting their neighbors.

To maximize the likelihood of successful relocation, build in a relatively long path. Spack can pad install prefixes automatically:

$ spack -e . config add config:install_tree:padded_length:256

Using build caches in CI

Build caches also speed up CI pipelines. Both GitHub Actions and GitLab CI support container registries, so the workflow described above applies directly in CI.

Spack provides a GitHub Action that configures a shared build cache:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Set up Spack
      uses: spack/setup-spack@v2
    - run: spack install python  # uses a shared build cache

See the setup-spack readme for instructions on caching additional binaries that are not in the shared build cache.

Summary

This tutorial covered:

  • installing julia from the pre-configured tutorial build cache without any source builds;

  • setting up a local OCI registry as a second build cache and pushing the environment to it;

  • reinstalling from the OCI cache to confirm that it functions as a regular Spack mirror;

  • using the same OCI artifacts as runnable container images, both per-package and as a combined environment with --tag.