Tutorial setup

If you have not done the prior sections, you’ll need to start the docker image:

docker run -it ghcr.io/spack/tutorial:sc23

and then set Spack up like this:

git clone --depth=100 --branch=releases/v0.21 https://github.com/spack/spack
. spack/share/spack/setup-env.sh
spack tutorial -y
spack bootstrap now
spack compiler find

See the Basic Installation Tutorial for full details on setup. For more help, join us in the #tutorial channel on Slack – get an invitation at slack.spack.io

Mirrors and Binary Caches Tutorial

This tutorial will guide you through the process of setting up a source mirror with a binary cache. Source mirrors and binary caches are extremely useful when using spack on a machine without internet access. Source mirrors allow you to fetch source code from a directory on your filesystem instead of accessing the outside internet, and binary caches allow you to install pre-compiled binaries to your spack installation path. Together, these two features can speed up builds when using spack within a larger development team.

We will use the filesystem for the mirrors in this tutorial, but mirrors can also be setup on web servers, in s3 buckets or even in OCI registries.

By default, Spack comes configured with a source mirror in the cloud to increase download reliability. We’ve also already set up a mirror in this tutorial for binary caches. We can see how these mirrors are configured:

$ spack mirror list
tutorial     [sb] file:///mirror
spack-public [s ] https://mirror.spack.io

Setting up a source cache mirror

When you run spack install, spack goes out to the internet to grab the source code to build your packages. This works fine on most clusters, but what do you do if the cluster in question doesn’t have access to the outside internet? This could happen for a variety of reasons. Maybe you’re building on a compute node that can’t connect to the internet, or maybe the whole cluster has been isolated from the internet.

Spack has a solution for this – setting up a source mirror. When you use a source mirror, spack checks the mirror for the source code before trying to download it from a remote location.

Building a source mirror is easy. Let’s start with the same simple environment. First, let’s build our software on a computer with external internet access.

$ cd ~
$ spack env create -d cache-env
==> Created environment in /home/spack/cache-env
==> You can activate this environment with:
==>   spack env activate /home/spack/cache-env
$ cd cache-env
$ spacktivate .
$ # for now, disable fortran support in all packages
$ spack config add "packages:all:variants: ~fortran"
$ spack add macsio+scr
==> Adding macsio+scr to environment /home/spack/cache-env
$ spack install
==> Concretized macsio+scr
[+]  4kxa45f  macsio@1.1%gcc@11.4.0~exodus~hdf5~ipo+mpi+pdb+scr~silo~szip~typhonio~zfp~zlib build_system=cmake build_type=Release generator=make patches=59479b9 arch=linux-ubuntu22.04-x86_64_v3
[+]  kd6xihn	  ^cmake@3.27.7%gcc@11.4.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-ubuntu22.04-x86_64_v3
[+]  ijsmc3j	      ^curl@8.4.0%gcc@11.4.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-ubuntu22.04-x86_64_v3
[+]  my64owh		  ^nghttp2@1.57.0%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  35j7wvr		  ^openssl@3.1.3%gcc@11.4.0~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-x86_64_v3
[+]  ct4al4u		      ^ca-certificates-mozilla@2023-05-30%gcc@11.4.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  qqlh6as	      ^ncurses@6.4%gcc@11.4.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  5xcetrv	      ^zlib-ng@2.1.4%gcc@11.4.0+compat+opt build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  znvoani	  ^gmake@4.4.1%gcc@11.4.0~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  pu27odn	  ^json-cwx@0.12%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  mnfnoa5	      ^autoconf@2.69%gcc@11.4.0 build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-ubuntu22.04-x86_64_v3
[+]  d3cncgl	      ^automake@1.16.5%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  rgag55h	      ^libtool@2.4.7%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  jnv5nut	      ^m4@1.4.19%gcc@11.4.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-ubuntu22.04-x86_64_v3
[+]  ueheij3		  ^diffutils@3.9%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  ivn4eq4		      ^libiconv@1.17%gcc@11.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  zuopqri		  ^libsigsegv@2.14%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  z4iqm7n	  ^openmpi@4.1.6%gcc@11.4.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools fabrics=none schedulers=none arch=linux-ubuntu22.04-x86_64_v3
[+]  dyrsjed	      ^hwloc@2.9.1%gcc@11.4.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  af4bqq6		  ^libpciaccess@0.17%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  b7j7735		      ^util-macros@1.19.3%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  67qoxbv		  ^libxml2@2.10.3%gcc@11.4.0+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  axxqoeq		      ^xz@5.4.1%gcc@11.4.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  mbwdizy	      ^numactl@2.0.14%gcc@11.4.0 build_system=autotools patches=4e1d78c,62fc8a8,ff37630 arch=linux-ubuntu22.04-x86_64_v3
[+]  zjwxy3z	      ^openssh@9.5p1%gcc@11.4.0+gssapi build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  tjiqlym		  ^krb5@1.20.1%gcc@11.4.0+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  lza42n2		      ^bison@3.8.2%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  cws3eft		      ^findutils@4.9.0%gcc@11.4.0 build_system=autotools patches=440b954 arch=linux-ubuntu22.04-x86_64_v3
[+]  y26lmlo		      ^gettext@0.22.3%gcc@11.4.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  vdb3ozo			  ^tar@1.34%gcc@11.4.0 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3
[+]  catlxmo			      ^pigz@2.7%gcc@11.4.0 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3
[+]  jkznmrm			      ^zstd@1.5.5%gcc@11.4.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  s4kq4wo		  ^libedit@3.1-20210216%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  hezcx7g		  ^libxcrypt@4.4.35%gcc@11.4.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-ubuntu22.04-x86_64_v3
[+]  dg34i2a	      ^perl@5.38.0%gcc@11.4.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-ubuntu22.04-x86_64_v3
[+]  ku7makq		  ^berkeley-db@18.1.40%gcc@11.4.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3
[+]  4oz3kpf		  ^bzip2@1.0.8%gcc@11.4.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  m626hzw		  ^gdbm@1.23%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  zjgtpdo	      ^pkgconf@1.9.5%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  fhkyvm3	      ^pmix@5.0.1%gcc@11.4.0~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  wpvxxka		  ^libevent@2.1.12%gcc@11.4.0+openssl build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  rbtcwee	  ^scr@2.0.0%gcc@11.4.0+dtcmp~fortran~ipo+libyogrt async_api=NONE build_system=cmake build_type=Release cache_base=/dev/shm cntl_base=/dev/shm copy_config=none file_lock=FLOCK generator=make resource_manager=SLURM scr_config=scr.conf arch=linux-ubuntu22.04-x86_64_v3
[+]  xmpw57x	      ^dtcmp@1.1.4%gcc@11.4.0+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  aexcyce		  ^lwgrp@1.0.5%gcc@11.4.0+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  wanayzr	      ^libyogrt@1.33%gcc@11.4.0~static build_system=autotools scheduler=slurm arch=linux-ubuntu22.04-x86_64_v3
[+]  ayvcsie		  ^slurm@23-02-4-1%gcc@11.4.0~cgroup~gtk~hdf5~hwloc~mariadb~nvml~pam~pmix+readline~restd~rsmi build_system=autotools sysconfdir=PREFIX/etc arch=linux-ubuntu22.04-x86_64_v3
[+]  iebre2x		      ^glib@2.78.0%gcc@11.4.0~libmount~strip build_system=meson buildtype=release default_library=shared patches=2c25d7b tracing=none arch=linux-ubuntu22.04-x86_64_v3
[+]  xnvt4js			  ^elfutils@0.189%gcc@11.4.0~debuginfod+exeprefix+nls build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  ra2t3ha			  ^libffi@3.4.4%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  bisaeec			  ^meson@1.2.2%gcc@11.4.0 build_system=python_pip patches=0f0b1bd,ae59765 arch=linux-ubuntu22.04-x86_64_v3
[+]  g5rag64			      ^py-pip@23.1.2%gcc@11.4.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  e4rc5s3			      ^py-setuptools@68.0.0%gcc@11.4.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  e7tv4cz			      ^py-wheel@0.41.2%gcc@11.4.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  7xzsee6			  ^ninja@1.11.1%gcc@11.4.0+re2c build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  cxbojc6			      ^re2c@2.2%gcc@11.4.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  bhevh4b			  ^pcre2@10.42%gcc@11.4.0~jit+multibyte build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  ybsi7tl			  ^python@3.11.6%gcc@11.4.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-ubuntu22.04-x86_64_v3
[+]  demnlle			      ^expat@2.5.0%gcc@11.4.0+libbsd build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  nrlj4tj				  ^libbsd@0.11.7%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  kg5ets2				      ^libmd@1.0.4%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  m2h5dly			      ^sqlite@3.43.2%gcc@11.4.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  thv4zdp			      ^util-linux-uuid@2.38.1%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  sfnsfpy		      ^json-c@0.16%gcc@11.4.0~ipo build_system=cmake build_type=Release generator=make arch=linux-ubuntu22.04-x86_64_v3
[+]  7b736zq		      ^lz4@1.9.4%gcc@11.4.0+pic build_system=makefile libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  efyihx2		      ^munge@0.5.15%gcc@11.4.0 build_system=autotools localstatedir=PREFIX/var arch=linux-ubuntu22.04-x86_64_v3
[+]  e4gmq7v			  ^libgcrypt@1.10.2%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  dtmcfvh			      ^libgpg-error@1.47%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  t4xydm4				  ^gawk@5.2.2%gcc@11.4.0~nls build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  2fyiqrp				      ^gmp@6.2.1%gcc@11.4.0+cxx build_system=autotools libs=shared,static patches=69ad2e2 arch=linux-ubuntu22.04-x86_64_v3
[+]  qpadvjw				      ^mpfr@4.2.0%gcc@11.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  km6pqxp					  ^texinfo@7.0.3%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  ier6vff	      ^pdsh@2.31%gcc@11.4.0+ssh+static_modules build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  475j7j3	  ^silo@4.11.1%gcc@11.4.0~fortran+fpzip+hdf5+hzip+mpi+pic+shared~silex build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  4wbogd6	      ^autoconf-archive@2023.02.20%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  7eeohwa	      ^hdf5@1.14.3%gcc@11.4.0~cxx~fortran~hl~ipo~java~map+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-ubuntu22.04-x86_64_v3
[+]  xxgqlmj	      ^readline@8.2%gcc@11.4.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu22.04-x86_64_v3

==> All of the packages are already installed

Once we’ve created and installed this environment, we can easily upload source code needed to reproduce this build to a mirror. The following command both creates the mirror and uploads the source code for the scr package included in our environment. The -d flag (short for --directory) tells spack where to place the mirrored source code files.

$ spack mirror create -d ~/mirror scr
==> Concretizing input specs
==> Adding package scr@2.0.0 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/47/471978ae0afb56a20847d3989b994fbd680d1dea21e77a5a46a964b6e3deed6b.tar.gz
==> Summary for mirror in /home/spack/mirror
==> Archive stats:
    0	 already present
    1	 added
    0	 failed to fetch.

We can configure spack to use this source mirror by adding a few lines to your spack.yaml file.

$ spack mirror add mymirror ~/mirror

Manually uploading every package in an environment can be tedious. Luckily, when run within an environment, spack mirror create with the --all flag will upload every source used to build the current environment to the specified directory.

$ spack mirror create -d ~/mirror --all
==> Adding package macsio@1.1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/a8/a86249b0f10647c0b631773db69568388094605ec1a0af149d9e61e95e6961ec.tar.gz
==> Adding package cmake@3.27.7 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/08/08f71a106036bf051f692760ef9558c0577c42ac39e96ba097e7662bd4158d8e.tar.gz
==> Adding package curl@8.4.0 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/e5/e5250581a9c032b1b6ed3cf2f9c114c811fc41881069e9892d115cc73f9e88c6.tar.bz2
==> Adding package gmake@4.4.1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/dd/dd16fb1d67bfab79a72f5e8390735c49e3e8e70b4945a15ab1f81ddb78658fb3.tar.gz
==> Adding package nghttp2@1.57.0 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/1e/1e3258453784d3b7e6cc48d0be087b168f8360b5d588c66bfeda05d07ad39ffd.tar.gz
==> Adding package pkgconf@1.9.5 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/1a/1ac1656debb27497563036f7bffc281490f83f9b8457c0d60bcfb638fb6b6171.tar.xz
==> Adding package openssl@3.1.3 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/f0/f0316a2ebd89e7f2352976445458689f80302093788c466692fb2a188b2eacf6.tar.gz
==> Adding package ca-certificates-mozilla@2023-05-30 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/5f/5fadcae90aa4ae041150f8e2d26c37d980522cdb49f923fc1e1b5eb8d74e71ad
==> Adding package perl@5.38.0 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/21/213ef58089d2f2c972ea353517dc60ec3656f050dcc027666e118b508423e517.tar.gz
==> Fetching https://mirror.spack.io/_source-cache/archive/9d/9da50e155df72bce55cb69f51f1dbb4b62d23740fb99f6178bb27f22ebdf8a46.tar.gz
==> Adding package berkeley-db@18.1.40 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/0c/0cecb2ef0c67b166de93732769abdeba0555086d51de1090df325e18ee8da9c8.tar.gz
==> Adding package bzip2@1.0.8 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/ab/ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269.tar.gz
==> Adding package diffutils@3.9 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/d8/d80d3be90a201868de83d78dad3413ad88160cc53bcc36eb9eaf7c20dbf023f1.tar.xz
==> Adding package libiconv@1.17 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/8f/8f74213b56238c85a50a5329f77e06198771e70dd9a739779f4c02f65d971313.tar.gz
==> Adding package gdbm@1.23 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/74/74b1081d21fff13ae4bd7c16e5d6e504a4c26f7cde1dca0d963a484174bbcacd.tar.gz
==> Adding package readline@8.2 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/3f/3feb7171f16a84ee82ca18a36d7b9be109a52c04f492a053331d7d1095007c35.tar.gz
==> Fetching https://mirror.spack.io/_source-cache/archive/bb/bbf97f1ec40a929edab5aa81998c1e2ef435436c597754916e6a5868f273aff7
==> Adding package ncurses@6.4 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/69/6931283d9ac87c5073f30b6290c4c75f21632bb4fc3603ac8100812bed248159.tar.gz
==> Adding package zlib-ng@2.1.4 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/a0/a0293475e6a44a3f6c045229fe50f69dc0eebc62a42405a51f19d46a5541e77a.tar.gz
==> Adding package json-cwx@0.12 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/3b/3bfae1f23eacba53ee130dbd1a6acf617af4627a9b4e4581d64b20a99b4e2b60.tar.gz
==> Adding package autoconf@2.69 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/95/954bd69b391edc12d6a4a51a2dd1476543da5c6bbf05a95b59dc0dd6fd4c2969.tar.gz
==> Fetching https://mirror.spack.io/_source-cache/archive/77/7793209b33013dc0f81208718c68440c5aae80e7a1c4b8d336e382525af791a7
==> Fetching https://mirror.spack.io/_source-cache/archive/35/35c449281546376449766f92d49fc121ca50e330e60fefcfc9be2af3253082c2
==> Fetching https://mirror.spack.io/_source-cache/archive/a4/a49dd5bac3b62daa0ff688ab4d508d71dbd2f4f8d7e2a02321926346161bf3ee
==> Adding package m4@1.4.19 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/3b/3be4a26d825ffdfda52a56fc43246456989a3630093cced3fbddf4771ee58a70.tar.gz
==> Adding package libsigsegv@2.14 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/cd/cdac3941803364cf81a908499beb79c200ead60b6b5b40cad124fd1e06caa295.tar.gz
==> Adding package automake@1.16.5 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/07/07bd24ad08a64bc17250ce09ec56e921d6343903943e99ccf63bbf0705e34605.tar.gz
==> Adding package libtool@2.4.7 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/04/04e96c2404ea70c590c546eba4202a4e12722c640016c12b9b2f1ce3d481e9a8.tar.gz
==> Adding package openmpi@4.1.6 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/f7/f740994485516deb63b5311af122c265179f5328a0d857a567b85db00b11e415.tar.bz2
==> Adding package hwloc@2.9.1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/a4/a440e2299f7451dc10a57ddbfa3f116c2a6c4be1bb97c663edd3b9c7b3b3b4cf.tar.gz
==> Adding package libpciaccess@0.17 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/bf/bf6985a77d2ecb00e2c79da3edfb26b909178ffca3f2e9d14ed0620259ab733b.tar.gz
==> Adding package util-macros@1.19.3 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/0f/0f812e6e9d2786ba8f54b960ee563c0663ddbe2434bf24ff193f5feab1f31971.tar.bz2
==> Adding package libxml2@2.10.3 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/5d/5d2cc3d78bec3dbe212a9d7fa629ada25a7da928af432c93060ff5c17ee28a9c.tar.xz
==> Fetching https://mirror.spack.io/_source-cache/archive/96/96151685cec997e1f9f3387e3626d61e6284d4d6e66e0e440c209286c03e9cc7.tar.gz
==> Adding package xz@5.4.1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/dd/dd172acb53867a68012f94c17389401b2f274a1aa5ae8f84cbfb8b7e383ea8d3.tar.bz2
==> Adding package numactl@2.0.14 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/1e/1ee27abd07ff6ba140aaf9bc6379b37825e54496e01d6f7343330cf1a4487035.tar.gz
==> Adding package openssh@9.5p1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/f0/f026e7b79ba7fb540f75182af96dc8a8f1db395f922bbc9f6ca603672686086b.tar.gz
==> Adding package krb5@1.20.1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/70/704aed49b19eb5a7178b34b2873620ec299db08752d6a8574f95d41879ab8851.tar.gz
==> Adding package bison@3.8.2 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/06/06c9e13bdf7eb24d4ceb6b59205a4f67c2c7e7213119644430fe82fbd14a0abb.tar.gz
==> Adding package gettext@0.22.3 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/b8/b838228b3f8823a6c1eddf07297197c4db13f7e1b173b9ef93f3f945a63080b6.tar.xz
==> Adding package tar@1.34 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/03/03d908cf5768cfe6b7ad588c921c6ed21acabfb2b79b788d1330453507647aed.tar.gz
==> Adding package pigz@2.7 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/d2/d2045087dae5e9482158f1f1c0f21c7d3de6f7cdc7cc5848bdabda544e69aa58.tar.gz
==> Adding package zstd@1.5.5 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/98/98e9c3d949d1b924e28e01eccb7deed865eefebf25c2f21c702e5cd5b63b85e1.tar.gz
==> Adding package findutils@4.9.0 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/a2/a2bfb8c09d436770edc59f50fa483e785b161a3b7b9d547573cb08065fd462fe.tar.xz
==> Adding package libedit@3.1-20210216 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/22/2283f741d2aab935c8c52c04b57bf952d02c2c02e651172f8ac811f77b1fc77a.tar.gz
==> Adding package libxcrypt@4.4.35 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/a8/a8c935505b55f1df0d17f8bfd59468c7c6709a1d31831b0f8e3e045ab8fd455d.tar.xz
==> Adding package pmix@5.0.1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/d4/d4371792d4ba4c791e1010100b4bf9a65500ababaf5ff25d681f938527a67d4a.tar.bz2
==> Adding package libevent@2.1.12 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/92/92e6de1be9ec176428fd2367677e61ceffc2ee1cb119035037a27d346b0403bb.tar.gz
==> Adding package scr@2.0.0 to mirror
==> Adding package dtcmp@1.1.4 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/dd/dd83d8cecd68e13b78b68e88675cc5847cde06742b7740e140b98f4a08127dd3.tar.gz
==> Adding package lwgrp@1.0.5 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/16/16b579e13b8a5218f4fe1b8715f6aafb09133a0cefbcd6b2eaf73802955dee6b.tar.gz
==> Adding package libyogrt@1.33 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/79/797d20c49cdc4f6beae8660b4f41ba7ac13f7e93a0344b47f0bdc64f780d1398.tar.gz
==> Adding package slurm@23-02-4-1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/72/7290143a71ce2797d0df3423f08396fd5c0ae4504749ff372d6860b2d6a3a1b0.tar.gz
==> Adding package glib@2.78.0 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/44/44eaab8b720877ce303c5540b657b126f12dc94972d9880b52959f43fb537b30.tar.xz
==> Fetching https://gitlab.gnome.org/GNOME/glib/-/commit/bda87264372c006c94e21ffb8ff9c50ecb3e14bd.diff
==> Adding package elfutils@0.189 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/39/39bd8f1a338e2b7cd4abc3ff11a0eddc6e690f69578a57478d8179b4148708c8.tar.bz2
==> Adding package libffi@3.4.4 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/d6/d66c56ad259a82cf2a9dfc408b32bf5da52371500b84745f7fb8b645712df676.tar.gz
==> Adding package meson@1.2.2 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/1c/1caa0ef6082e311bdca9836e7907f548b8c3f041a42ed41f0ff916b83ac7dddd.tar.gz
==> Adding package ninja@1.11.1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/31/31747ae633213f1eda3842686f83c2aa1412e0f5691d1c14dbbcc67fe7400cea.tar.gz
==> Adding package python@3.11.6 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/c0/c049bf317e877cbf9fce8c3af902436774ecef5249a29d10984ca3a37f7f4736.tgz
==> Adding package expat@2.5.0 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/6f/6f0e6e01f7b30025fa05c85fdad1e5d0ec7fd35d9f61b22f34998de11969ff67.tar.bz2
==> Adding package libbsd@0.11.7 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/9b/9baa186059ebbf25c06308e9f991fda31f7183c0f24931826d83aa6abd8a0261.tar.xz
==> Adding package libmd@1.0.4 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/f5/f51c921042e34beddeded4b75557656559cf5b1f2448033b4c1eec11c07e530f.tar.xz
==> Adding package sqlite@3.43.2 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/6d/6d422b6f62c4de2ca80d61860e3a3fb693554d2f75bb1aaca743ccc4d6f609f0.tar.gz
==> Adding package util-linux-uuid@2.38.1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/08/0820eb8eea90408047e3715424bc6be771417047f683950fecb4bdd2e2cbbc6e.tar.gz
==> Adding package re2c@2.2 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/0f/0fc45e4130a8a555d68e230d1795de0216dfe99096b61b28e67c86dfd7d86bda.tar.xz
==> Adding package py-pip@23.1.2 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/3e/3ef6ac33239e4027d9a5598a381b9d30880a1477e50039db2eac6e8a8f6d1b18
==> Adding package py-setuptools@68.0.0 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/11/11e52c67415a381d10d6b462ced9cfb97066179f0e871399e006c4ab101fc85f
==> Adding package py-wheel@0.41.2 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/75/75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8
==> Adding package pcre2@10.42 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/8d/8d36cd8cb6ea2a4c2bb358ff6411b0c788633a2a45dabbf1aeb4b701d1b5e840.tar.bz2
==> Adding package json-c@0.16 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/8e/8e45ac8f96ec7791eaf3bb7ee50e9c2100bbbc87b8d0f1d030c5ba8a0288d96b.tar.gz
==> Adding package lz4@1.9.4 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/0b/0b0e3aa07c8c063ddf40b082bdf7e37a1562bda40a0ff5272957f3e987e0e54b.tar.gz
==> Adding package munge@0.5.15 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/3f/3f979df117a34c74db8fe2835521044bdeb08e3b7d0f168ca97c3547f51da9ba.tar.xz
==> Adding package libgcrypt@1.10.2 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/3b/3b9c02a004b68c256add99701de00b383accccf37177e0d6c58289664cce0c03.tar.bz2
==> Adding package libgpg-error@1.47 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/9e/9e3c670966b96ecc746c28c2c419541e3bcb787d1a73930f5e5f5e1bcbbb9bdb.tar.bz2
==> Adding package gawk@5.2.2 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/3c/3c1fce1446b4cbee1cd273bd7ec64bc87d89f61537471cd3e05e33a965a250e9.tar.xz
==> Adding package gmp@6.2.1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/ea/eae9326beb4158c386e39a356818031bd28f3124cf915f8c5b1dc4c7a36b4d7c.tar.bz2
==> Adding package mpfr@4.2.0 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/69/691db39178e36fc460c046591e4b0f2a52c8f2b3ee6d750cc2eab25f1eaa999d.tar.bz2
==> Adding package autoconf-archive@2023.02.20 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/71/71d4048479ae28f1f5794619c3d72df9c01df49b1c628ef85fde37596dc31a33.tar.xz
==> Adding package texinfo@7.0.3 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/3c/3cc5706fb086b895e1dc2b407aade9f95a3a233ff856273e2b659b089f117683.tar.gz
==> Adding package pdsh@2.31 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/0e/0ee066ce395703285cf4f6cf00b54b7097d12457a4b1c146bc6f33d8ba73caa7.tar.gz
==> Adding package silo@4.11.1 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/49/49eddc00304aa4a19074b099559edbdcaa3532c98df32f99aa62b9ec3ea7cee2.tar.xz
==> Adding package hdf5@1.14.3 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/09/09cdb287aa7a89148c1638dd20891fdbae08102cf433ef128fd345338aa237c7.tar.gz
==> Summary for mirror in /home/spack/mirror
==> Archive stats:
    1	 already present
    75	 added
    0	 failed to fetch.

As long as spack can read from the mirror directory, spack will attempt to read source packages from the mirror instead of accessing the internet. This can be a huge boon for computers that can’t access the external internet but can access a shared filesystem. If you need to use spack on a system that is isolated from the external internet, you must bundle the whole spack mirror directory and unbundle it on the isolated system. From there, you follow the same steps to use the spack mirror as you would on any computer that can’t access the external internet.

If you need to add more sources to the mirror, you can re-run the command you used to create the mirror. For example, assume we want to add unzip to our environment.

$ spack add unzip
==> Adding unzip to environment /home/spack/cache-env
$ spack install
==> Concretized unzip
 -   uy5dhue  unzip@6.0%gcc@11.4.0 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3
[+]  znvoani	  ^gmake@4.4.1%gcc@11.4.0~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3

[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gmake-4.4.1-znvoaniggvauzxv4yobchqt4g7tyjuc7
==> Installing unzip-6.0-uy5dhuenjwg5q3qs7fmjou2qnvc6kazc [2/2]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-11.4.0-unzip-6.0-uy5dhuenjwg5q3qs7fmjou2qnvc6kazc.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/unzip-6.0/linux-ubuntu22.04-x86_64_v3-gcc-11.4.0-unzip-6.0-uy5dhuenjwg5q3qs7fmjou2qnvc6kazc.spack
==> Extracting unzip-6.0-uy5dhuenjwg5q3qs7fmjou2qnvc6kazc from binary cache
==> unzip: Successfully installed unzip-6.0-uy5dhuenjwg5q3qs7fmjou2qnvc6kazc
  Search: 0.00s.  Fetch: 0.22s.	 Install: 0.08s.  Extract: 0.05s.  Relocate: 0.01s.  Total: 0.30s
[+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/unzip-6.0-uy5dhuenjwg5q3qs7fmjou2qnvc6kazc
==> Updating view at /home/spack/cache-env/.spack-env/view

Now that we’ve added unzip, we need to update the mirror.

$ spack mirror create -d ~/mirror --all
==> Adding package unzip@6.0 to mirror
==> Fetching https://mirror.spack.io/_source-cache/archive/03/036d96991646d0449ed0aa952e4fbe21b476ce994abc276e49d30e686708bd37.tar.gz
==> Adding package gmake@4.4.1 to mirror
==> Adding package macsio@1.1 to mirror
==> Adding package cmake@3.27.7 to mirror
==> Adding package curl@8.4.0 to mirror
==> Adding package nghttp2@1.57.0 to mirror
==> Adding package pkgconf@1.9.5 to mirror
==> Adding package openssl@3.1.3 to mirror
==> Adding package ca-certificates-mozilla@2023-05-30 to mirror
==> Adding package perl@5.38.0 to mirror
==> Adding package berkeley-db@18.1.40 to mirror
==> Adding package bzip2@1.0.8 to mirror
==> Adding package diffutils@3.9 to mirror
==> Adding package libiconv@1.17 to mirror
==> Adding package gdbm@1.23 to mirror
==> Adding package readline@8.2 to mirror
==> Adding package ncurses@6.4 to mirror
==> Adding package zlib-ng@2.1.4 to mirror
==> Adding package json-cwx@0.12 to mirror
==> Adding package autoconf@2.69 to mirror
==> Adding package m4@1.4.19 to mirror
==> Adding package libsigsegv@2.14 to mirror
==> Adding package automake@1.16.5 to mirror
==> Adding package libtool@2.4.7 to mirror
==> Adding package openmpi@4.1.6 to mirror
==> Adding package hwloc@2.9.1 to mirror
==> Adding package libpciaccess@0.17 to mirror
==> Adding package util-macros@1.19.3 to mirror
==> Adding package libxml2@2.10.3 to mirror
==> Adding package xz@5.4.1 to mirror
==> Adding package numactl@2.0.14 to mirror
==> Adding package openssh@9.5p1 to mirror
==> Adding package krb5@1.20.1 to mirror
==> Adding package bison@3.8.2 to mirror
==> Adding package gettext@0.22.3 to mirror
==> Adding package tar@1.34 to mirror
==> Adding package pigz@2.7 to mirror
==> Adding package zstd@1.5.5 to mirror
==> Adding package findutils@4.9.0 to mirror
==> Adding package libedit@3.1-20210216 to mirror
==> Adding package libxcrypt@4.4.35 to mirror
==> Adding package pmix@5.0.1 to mirror
==> Adding package libevent@2.1.12 to mirror
==> Adding package scr@2.0.0 to mirror
==> Adding package dtcmp@1.1.4 to mirror
==> Adding package lwgrp@1.0.5 to mirror
==> Adding package libyogrt@1.33 to mirror
==> Adding package slurm@23-02-4-1 to mirror
==> Adding package glib@2.78.0 to mirror
==> Adding package elfutils@0.189 to mirror
==> Adding package libffi@3.4.4 to mirror
==> Adding package meson@1.2.2 to mirror
==> Adding package ninja@1.11.1 to mirror
==> Adding package python@3.11.6 to mirror
==> Adding package expat@2.5.0 to mirror
==> Adding package libbsd@0.11.7 to mirror
==> Adding package libmd@1.0.4 to mirror
==> Adding package sqlite@3.43.2 to mirror
==> Adding package util-linux-uuid@2.38.1 to mirror
==> Adding package re2c@2.2 to mirror
==> Adding package py-pip@23.1.2 to mirror
==> Adding package py-setuptools@68.0.0 to mirror
==> Adding package py-wheel@0.41.2 to mirror
==> Adding package pcre2@10.42 to mirror
==> Adding package json-c@0.16 to mirror
==> Adding package lz4@1.9.4 to mirror
==> Adding package munge@0.5.15 to mirror
==> Adding package libgcrypt@1.10.2 to mirror
==> Adding package libgpg-error@1.47 to mirror
==> Adding package gawk@5.2.2 to mirror
==> Adding package gmp@6.2.1 to mirror
==> Adding package mpfr@4.2.0 to mirror
==> Adding package autoconf-archive@2023.02.20 to mirror
==> Adding package texinfo@7.0.3 to mirror
==> Adding package pdsh@2.31 to mirror
==> Adding package silo@4.11.1 to mirror
==> Adding package hdf5@1.14.3 to mirror
==> Summary for mirror in /home/spack/mirror
==> Archive stats:
    76	 already present
    1	 added
    0	 failed to fetch.

Spack will skip uploading source code packages that are already included in the spack mirror. Mirrors can be shared across different environments, meaning one mirror can house all the source code needed to build your team’s dependencies.

Setting up a binary cache mirror

If your team is setting spack as part of their development practice, you’ll run up against the biggest disadvantage of using spack: building all your packages from scratch is slow. Recompiling the software dependencies for a large project can take hours to complete. If every developer is rebuilding their own software stack, that leads to a massive waste of computational resources and a loss of developer productivity.

Spack has two ways to mitigate this problem: chained spack instances and spack binary caches. For now, we’re going to discuss binary caches as a way of solving this issue.

A spack binary cache is made up of spack binary packages. Each spack binary package, ending with a *.spack extension, is a tarball of an installed spack package signed with a gpg signature. When you install a package from a mirror with a binary cache, spack

  • Checks to see if there is a spack binary package that exactly matches the hash of the spec you want to build.

  • If a binary package is found, spack checks to see if the signature on the binary package is trusted. If the signature isn’t trusted, or if no package was found, spack builds the package from source.

  • If the signature is trusted, then spack unzips and relocates the binary package.

For the user, using spack binary caches is transparent. This is a clear departure from systems like conda, where you need to choose separate workflows for binary and source packages. We’ve already demonstrated using spack binary caches earlier in the tutorial when we set up spack to use a binary mirror. As a reminder, we ran:

$ spack mirror add tutorial /mirror
$ spack buildcache keys --install --trust
==> Fetching file:///mirror/build_cache/_pgp/spack-public-binary-key.pub
gpg: key A8E0CA3C1C2ADA2F: 7 signatures not checked due to missing keys
gpg: key A8E0CA3C1C2ADA2F: public key "Spack Project Official Binaries <maintainers@spack.io>" imported
gpg: Total number processed: 1
gpg:		   imported: 1
gpg: no ultimately trusted keys found
gpg: inserting ownertrust of 6

Building a spack binary cache has some gotchas, but is almost as easy as building a source mirror. We’ll start by making a new environment for ourselves. Since we’re intending to publish to a binary cache, we’ll need to compile all these packages ourselves. This can take some time, so we’ll make a new environment with some packages that compile quickly.

$ cd ~
$ mkdir cache-binary
$ cd cache-binary
$ spack env create -d .
==> Created environment in /home/spack/cache-binary
==> You can activate this environment with:
==>   spack env activate /home/spack/cache-binary
$ spacktivate .
$ spack add bzip2
==> Adding bzip2 to environment /home/spack/cache-binary
$ spack add zlib
==> Adding zlib to environment /home/spack/cache-binary

Before we build anything, we need to modify the following line in our spack configuration file. Then, just to be sure, we’ll initiate a build of this environment and force ourselves to not use any cache.

$ spack config add "config:install_tree:padded_length:128"
$ spack install --no-cache
==> Concretized bzip2
 -   4oz3kpf  bzip2@1.0.8%gcc@11.4.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   ueheij3	  ^diffutils@3.9%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   znvoani	      ^gmake@4.4.1%gcc@11.4.0~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   ivn4eq4	      ^libiconv@1.17%gcc@11.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3

==> Concretized zlib
 -   gnpfvzj  zlib@1.3%gcc@11.4.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu22.04-x86_64_v3
 -   znvoani	  ^gmake@4.4.1%gcc@11.4.0~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3

==> Installing gmake-4.4.1-znvoaniggvauzxv4yobchqt4g7tyjuc7 [1/5]
==> Fetching https://mirror.spack.io/_source-cache/archive/dd/dd16fb1d67bfab79a72f5e8390735c49e3e8e70b4945a15ab1f81ddb78658fb3.tar.gz
==> No patches needed for gmake
==> gmake: Executing phase: 'install'
==> gmake: Successfully installed gmake-4.4.1-znvoaniggvauzxv4yobchqt4g7tyjuc7
  Stage: 0.16s.	 Install: 13.19s.  Post-install: 0.04s.	 Total: 13.42s
[+] /home/spack/spack/opt/spack/[padded-to-128-chars]/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gmake-4.4.1-znvoaniggvauzxv4yobchqt4g7tyjuc7
==> Installing libiconv-1.17-ivn4eq42mmvco5l3drcxjzbmpto5nfi2 [2/5]
==> Fetching https://mirror.spack.io/_source-cache/archive/8f/8f74213b56238c85a50a5329f77e06198771e70dd9a739779f4c02f65d971313.tar.gz
==> No patches needed for libiconv
==> libiconv: Executing phase: 'autoreconf'
==> libiconv: Executing phase: 'configure'
==> libiconv: Executing phase: 'build'
==> libiconv: Executing phase: 'install'
==> libiconv: Successfully installed libiconv-1.17-ivn4eq42mmvco5l3drcxjzbmpto5nfi2
  Stage: 0.25s.	 Autoreconf: 0.00s.  Configure: 16.41s.	 Build: 8.24s.	Install: 0.68s.	 Post-install: 0.07s.  Total: 25.76s
[+] /home/spack/spack/opt/spack/[padded-to-128-chars]/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/libiconv-1.17-ivn4eq42mmvco5l3drcxjzbmpto5nfi2
==> Installing zlib-1.3-gnpfvzjsqubwjx3zny4dp7hkafztp2o2 [3/5]
==> Fetching https://mirror.spack.io/_source-cache/archive/ff/ff0ba4c292013dbc27530b3a81e1f9a813cd39de01ca5e0f8bf355702efa593e.tar.gz
==> No patches needed for zlib
==> zlib: Executing phase: 'edit'
==> zlib: Executing phase: 'build'
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.3-gnpfvzjsqubwjx3zny4dp7hkafztp2o2
  Stage: 0.11s.	 Edit: 0.35s.  Build: 0.67s.  Install: 0.07s.  Post-install: 0.03s.  Total: 1.32s
[+] /home/spack/spack/opt/spack/[padded-to-128-chars]/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/zlib-1.3-gnpfvzjsqubwjx3zny4dp7hkafztp2o2
==> Installing diffutils-3.9-ueheij3wbhepmelnkti3jsglzzknb4en [4/5]
==> Fetching https://mirror.spack.io/_source-cache/archive/d8/d80d3be90a201868de83d78dad3413ad88160cc53bcc36eb9eaf7c20dbf023f1.tar.xz
==> No patches needed for diffutils
==> diffutils: Executing phase: 'autoreconf'
==> diffutils: Executing phase: 'configure'
==> diffutils: Executing phase: 'build'
==> diffutils: Executing phase: 'install'
==> diffutils: Successfully installed diffutils-3.9-ueheij3wbhepmelnkti3jsglzzknb4en
  Stage: 0.21s.	 Autoreconf: 0.00s.  Configure: 24.88s.	 Build: 1.73s.	Install: 0.55s.	 Post-install: 0.06s.  Total: 27.52s
[+] /home/spack/spack/opt/spack/[padded-to-128-chars]/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/diffutils-3.9-ueheij3wbhepmelnkti3jsglzzknb4en
==> Installing bzip2-1.0.8-4oz3kpfibmlvfizzivudj7pnzm2w7ce7 [5/5]
==> Fetching https://mirror.spack.io/_source-cache/archive/ab/ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269.tar.gz
==> Ran patch() for bzip2
==> bzip2: Executing phase: 'install'
==> bzip2: Successfully installed bzip2-1.0.8-4oz3kpfibmlvfizzivudj7pnzm2w7ce7
  Stage: 0.09s.	 Install: 1.71s.  Post-install: 0.03s.	Total: 1.88s
[+] /home/spack/spack/opt/spack/[padded-to-128-chars]/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/bzip2-1.0.8-4oz3kpfibmlvfizzivudj7pnzm2w7ce7
==> Updating view at /home/spack/cache-binary/.spack-env/view

This configuration change ensures that spack installs all our packages to a path that is at least 128 characters. We need this change because of how spack relocates packages. Relocation consists of the following steps:

  • Search all text files to replace the package builder’s path with your specific local installation path.

  • Modify all the RPATHs in your binaries to point to your specific local installation path. This uses patchelf if the new path is longer than the old path, but otherwise uses a faster implementation in python.

  • Search all binaries to replace hard-coded C strings of the package builder’s path with your specific local installation path.

Adding padding ensures that any paths hard-coded as C strings in our binaries will be large enough to hold our user’s eventual install path. We advise picking 128 because longer strings occasionally cause compilation problems with some software packages. If the user’s install path is too long, spack will give you a warning. All scripts and and RPATHs will still be properly relocated, but C strings within any binaries will not be modified. Depending on the package, this may cause problems when you try to use the software.

We also need to create a gpg key to sign all our packages. You should back up the secret and public keys to a secure place so they can be re-used in the future.

$ spack gpg create "My Name" "<my.email@my.domain.com>"
gpg: directory '/home/spack/spack/opt/spack/gpg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/spack/spack/opt/spack/gpg/openpgp-revocs.d/E51CD554EFA27AAA696F6E27CE3CE7A253F684E2.rev'
$ mkdir ~/private_gpg_backup
$ cp ~/spack/opt/spack/gpg/*.gpg ~/private_gpg_backup
$ cp ~/spack/opt/spack/gpg/pubring.* ~/mirror

With this setup done, we’re ready to fill a buildcache with installed packages. The steps are very similar to creating a source mirror. To upload all specs in the environment to a buildcache, simply use the command spack buildcache push or equivalently spack buildcache create:

$ spack buildcache push ~/mirror
==> Selected 5 specs to push to file:///home/spack/mirror
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:	2  signed:   0	trust: 0-, 0q, 0n, 0m, 0f, 2u
==> [1/5] Pushed bzip2@1.0.8/4oz3kpf
==> [2/5] Pushed diffutils@3.9/ueheij3
==> [3/5] Pushed gmake@4.4.1/znvoani
==> [4/5] Pushed libiconv@1.17/ivn4eq4
==> [5/5] Pushed zlib@1.3/gnpfvzj

Voila, done! Our spack mirror has now been augmented with a binary cache. This cache can be used on systems without external internet access, just like with a spack source mirror.

Though it’s outside the scope of this tutorial, spack mirrors and build caches can also be hosted over https://, s3://, and oci://, as well. Consult the spack documentation for more information on how to do this.

Before someone can use this binary cache, they will need to make sure that they trust all the packages listed in the binary cache. If you’re sharing files between trusted users on a filesystem, you can do this with the following command:

$ spack buildcache keys --install --trust --force
==> Fetching file:///mirror/build_cache/_pgp/spack-public-binary-key.pub
gpg: key A8E0CA3C1C2ADA2F: 7 signatures not checked due to missing keys
gpg: key A8E0CA3C1C2ADA2F: "Spack Project Official Binaries <maintainers@spack.io>" not changed
gpg: Total number processed: 1
gpg:		  unchanged: 1

Together, this means download all the keys on the binary cache and trust them. Have your users run the above command on a new spack instance before they initiate a build.

Bootstrapping Mirrors

In order to run Spack on an airgapped system, or anywhere else without internet access, you also need to install clingo as a dependency (clingo is used by spack’s concretizer). While you can always install clingo through your favorite method, we like to think Spack is your favorite install mechanism for everything, and we have convenient ways to bootstrap clingo on airgapped systems.

Spack can prepare a bootstrap mirror for either source or binary mirror for bootstrapping via the spack bootstrap mirror command. We will focus on binary bootstrapping for this tutorial.

$ spack bootstrap mirror --binary-packages ~/mirror
==> Adding "clingo-bootstrap@spack+python %gcc target=x86_64" and dependencies to the mirror at /home/spack/mirror/bootstrap_cache
==> Adding "gnupg@2.3: %gcc target=x86_64" and dependencies to the mirror at /home/spack/mirror/bootstrap_cache
==> Adding "patchelf@0.13.1: %gcc target=x86_64" and dependencies to the mirror at /home/spack/mirror/bootstrap_cache
==> Adding "gnuconfig" and dependencies to the mirror at /home/spack/mirror/bootstrap_cache
==> Adding binary packages from "https://github.com/spack/spack-bootstrap-mirrors/releases/download/v0.4/bootstrap-buildcache.tar.gz" to the mirror at /home/spack/mirror/bootstrap_cache

To register the mirror on the platform where it's supposed to be used, move "/home/spack/mirror" to its final location and run the following command(s):

  % spack bootstrap add --trust local-sources <final-path>/metadata/sources
  % spack bootstrap add --trust local-binaries <final-path>/metadata/binaries

Using the instructions printed from Spack, you can register the new bootstrap sources on the airgapped system.

Cache Summary

If you’re using spack within a development team, consider setting up source mirrors with binary caches. Source mirrors will let you replicate a spack environment on a machine without external internet access, and binary mirrors free you from the burden of recompiling everything from scratch and save you development time. Spack mirrors also provide the mechanism to bring Spack’s internal dependencies over to any machine without external internet access.