Tutorial setup

If you have not done the prior sections, you’ll need to start the docker image:

docker run -it ghcr.io/spack/tutorial:sc24

and then set Spack up like this:

git clone --depth=100 --branch=releases/v0.23 https://github.com/spack/spack
. spack/share/spack/setup-env.sh
spack tutorial -y
spack bootstrap now
spack compiler find

See the Basic Installation Tutorial for full details on setup. For more help, join us in the #tutorial channel on Slack – get an invitation at slack.spack.io

Stacks Tutorial

So far, we’ve talked about Spack environments in the context of a unified user environment. But environments in Spack have much broader capabilities. In this tutorial we will consider how to use Spack environments to manage large deployments of software.

What usually differs between a typical environment for a single user, and an environment used to manage large deployments, is that in the latter case we often have a set of packages we want to install across a wide range of MPIs, LAPACKs or compilers.

In the following we’ll mimic the creation of a software stack built onto a cross-product of different LAPACK and MPI libraries, with a compiler that is more recent than the one provided by the host system.

In the first part we’ll focus on how to properly configure and install the software we want. We’ll learn how to pin certain requirements, and how to write a cross product of specs in a compact, and expressive, way.

Then we’ll consider how the software we install might be consumed by our users, and see the two main mechanisms that Spack provides for that: views and module files.

Note

Before we start this hands-on, make sure the EDITOR environment variable is set to your preferred editor, for instance:

$ export EDITOR='emacs -nw'

Setup the compiler

The first step to build our stack is to setup the compiler we want to use later. This is, currently, an iterative process that can be done in two ways:

  1. Install the compiler first, then register it in the environment

  2. Use a second environment just for the compiler

In the following we’ll use the first approach. For people interested, an example of the latter approach can be found at this link.

Let’s start by creating an environment in a directory of our choice:

$ spack env activate --create ~/stacks
$ spack env status
==> In environment /home/spack5/stacks

Now we can add from the command line a new compiler. We’ll also disable the generation of views for the time being, as we’ll come back to this topic later in the tutorial:

$ spack add gcc@12 %gcc@11
==> Adding gcc@12%gcc@11 to environment /home/spack5/stacks
$ spack env view disable
$ spack config edit

What you should see on screen now is the following spack.yaml file:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  specs:
  - gcc@12%gcc@11
  view: false
  concretizer:
    unify: true

The next step is to concretize and install our compiler:

$ spack concretize
==> Concretized 1 spec:
[+]  jjgfqpd  gcc@12.3.0%gcc@11.4.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages='c,c++,fortran' arch=linux-ubuntu22.04-x86_64_v3
[+]  lljulvx	  ^diffutils@3.10%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  bsga3no	      ^libiconv@1.17%gcc@11.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  x6mwj3l	  ^gawk@5.3.1%gcc@11.4.0~nls build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  ea6qziv	      ^libsigsegv@2.14%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  aylebxv	      ^readline@8.2%gcc@11.4.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu22.04-x86_64_v3
[+]  hshzy76	  ^gcc-runtime@11.4.0%gcc@11.4.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[e]  a7drdl4	  ^glibc@2.35%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  srkzfjr	  ^gmake@4.4.1%gcc@11.4.0~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  5webgya	  ^gmp@6.3.0%gcc@11.4.0+cxx build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  vgucajy	      ^autoconf@2.72%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  awgfaon	      ^automake@1.16.5%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  qaspjyq	      ^m4@1.4.19%gcc@11.4.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-ubuntu22.04-x86_64_v3
[+]  tt3byem	  ^libtool@2.4.7%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  2sbkhch	      ^findutils@4.9.0%gcc@11.4.0 build_system=autotools patches=440b954 arch=linux-ubuntu22.04-x86_64_v3
[+]  qoauga4	  ^mpc@1.3.1%gcc@11.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  mdfsfjy	  ^mpfr@4.2.1%gcc@11.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  6s34ic3	      ^autoconf-archive@2023.02.20%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  cwpt5ec	  ^perl@5.40.0%gcc@11.4.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  cexlioh	      ^berkeley-db@18.1.40%gcc@11.4.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3
[+]  ewqc7cx	      ^bzip2@1.0.8%gcc@11.4.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  jj2atvs	      ^gdbm@1.23%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  ljsbwjc	  ^texinfo@7.1%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  bexdfw2	      ^gettext@0.22.5%gcc@11.4.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  wozcmyn		  ^libxml2@2.13.4%gcc@11.4.0+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  mmv6i4n		  ^tar@1.34%gcc@11.4.0 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3
[+]  pcwnu2w		      ^pigz@2.8%gcc@11.4.0 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3
[+]  2fvrfr6		  ^xz@5.4.6%gcc@11.4.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  rvg7j6b	      ^ncurses@6.5%gcc@11.4.0~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-ubuntu22.04-x86_64_v3
[+]  oplsmxr		  ^pkgconf@2.2.0%gcc@11.4.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  fhud6vq	  ^zlib-ng@2.2.1%gcc@11.4.0+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  t7h6imj	  ^zstd@1.5.6%gcc@11.4.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu22.04-x86_64_v3

$ spack install
[+] /usr (external glibc-2.35-a7drdl4tlx4bu3mzhor75pskvd3pdot6)
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-runtime-11.4.0-hshzy762rns57ibvxiyi3qqvc4nehse2
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/zstd-1.5.6-t7h6imj35ruac562vr7snzvli2h2tzup
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gmp-6.3.0-5webgyazi3rpemdmmhbip4glviaeet2l
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/zlib-ng-2.2.1-fhud6vqkh7jaesckalgqg5xxml72in6i
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/mpfr-4.2.1-mdfsfjykvyaknc5yzul4l25aip3rugoe
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/mpc-1.3.1-qoauga4bfx37v3lynpeoi4rcy2nw2ek5
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-jjgfqpdmihb2cq5ca7txia5oshruq56q

Finally, let’s register it as a new compiler in the environment:

$ spack compiler find "$(spack location -i gcc)"
==> Added 1 new compiler to /home/spack5/stacks/spack.yaml
    gcc@12.3.0
==> Compilers are defined in the following files:
    /home/spack5/.spack/linux/compilers.yaml  /home/spack5/stacks/spack.yaml

The spack location -i command returns the installation prefix for the spec being queried:

$ spack location -i gcc
/home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-jjgfqpdmihb2cq5ca7txia5oshruq56q

This might be useful in general when scripting Spack commands, as the example above shows. Listing the compilers now shows the presence of gcc@12.3.0:

$ spack compiler list
==> Available compilers
-- clang ubuntu22.04-x86_64 -------------------------------------
clang@14.0.0

-- gcc ubuntu22.04-x86_64 ---------------------------------------
gcc@12.3.0  gcc@11.4.0	gcc@10.5.0

The manifest file at this point looks like:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  specs:
  - gcc@12%gcc@11
  view: false
  concretizer:
    unify: true
  compilers:
  - compiler:
      spec: gcc@=12.3.0
      paths:
        cc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gcc
        cxx: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/g++
        f77: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
        fc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
      flags: {}
      operating_system: ubuntu22.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

We are ready to build more software with our newly installed GCC!

Install a software stack

Now that we have a compiler ready, the next objective is to build software with it. We’ll start by trying to add different versions of netlib-scalapack, linked against different MPI implementations:

$ spack add netlib-scalapack %gcc@12 ^openblas ^openmpi
==> Adding netlib-scalapack%gcc@12 ^openblas ^openmpi to environment /home/spack5/stacks
$ spack add netlib-scalapack %gcc@12 ^openblas ^mpich
==> Adding netlib-scalapack%gcc@12 ^mpich ^openblas to environment /home/spack5/stacks

If we try to concretize the environment, we’ll get an error:

$ spack concretize
==> Error: Spack concretizer internal error. Please submit a bug report and include the command, environment if applicable and the following error message.
    [netlib-scalapack%gcc@12 ^openblas ^openmpi, netlib-scalapack%gcc@12 ^mpich ^openblas, gcc@12.3.0/jjgfqpdmihb2cq5ca7txia5oshruq56q] is unsatisfiable, errors are:
    internal_error("Imposing constraint outside of imposed_nodes")
    internal_error("nodes in root condition set must be associated with root"). Couldn't concretize without changing the existing environment. If you are ok with changing it, try `spack concretize --force`. You could consider setting `concretizer:unify` to `when_possible` or `false` to allow multiple versions of some packages.

The error message is quite verbose, and admittedly complicated, but at the end it gives a useful hint:

You could consider setting `concretizer:unify` to `when_possible` or `false` to allow multiple versions of some packages.

Let’s see what that means.

Tuning concretizer options for a stack

Whenever we concretize an environment with more than one root spec, we can configure Spack to be more or less strict with duplicate nodes in the sub-DAG obtained by following link and run edges starting from the roots. We usually call this sub-DAG the root unification set.

A diagram might help to better visualize the concept:

_images/stacks-unify.svg

The image above represents the current environment, with our three root specs highlighted by a thicker dashed line. Any node that could be reached following a link or run edge is part of the root unification set. Pure build dependencies might fall outside of it.

The config option determining which nodes are allowed to be in the root unification set is concretizer:unify. Let’s check its value:

$ spack config get concretizer | grep unify
  unify: true

concretizer:unify:true means that only a single configuration for each package can be present. This value is good for single project environments, since it ensures we can construct a view of all the software, with the usual structure expected on a Unix-ish system, and without risks of collisions between installations.

Clearly, we can’t respect this requirement, since our roots already contain two different configurations of netlib-scalapack. Let’s set the value to false, and try to re-concretize:

$ spack config add concretizer:unify:false
$ spack concretize
==> Concretized 2 specs:
 -   me353po  netlib-scalapack@2.2.0%gcc@12.3.0~ipo~pic+shared build_system=cmake build_type=Release generator=make patches=072b006,1c9ce5f,244a9aa arch=linux-ubuntu22.04-x86_64_v3
 -   yempnaz	  ^cmake@3.30.5%gcc@12.3.0~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release patches=dbc3892 arch=linux-ubuntu22.04-x86_64_v3
 -   wsmx6sp	      ^curl@8.10.1%gcc@12.3.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-ubuntu22.04-x86_64_v3
 -   ippbbwu		  ^nghttp2@1.63.0%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   ioferp7		      ^diffutils@3.10%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   i5fmca4		  ^openssl@3.4.0%gcc@12.3.0~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-x86_64_v3
 -   xilbvuy		      ^ca-certificates-mozilla@2023-05-30%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   uajmsrc	      ^ncurses@6.5%gcc@12.3.0~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-ubuntu22.04-x86_64_v3
 -   i4mwsvm	      ^zlib-ng@2.2.1%gcc@12.3.0+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   w5nlv2k	  ^gcc-runtime@12.3.0%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[e]  gkoh6ax	  ^glibc@2.35%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   zoiouai	  ^gmake@4.4.1%gcc@12.3.0~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   ezes42n	  ^mpich@4.2.3%gcc@12.3.0~argobots~cuda+fortran+hwloc+hydra+libxml2+pci~rocm+romio~slurm~vci~verbs+wrapperrpath~xpmem build_system=autotools datatype-engine=auto device=ch4 netmod=ofi pmi=default arch=linux-ubuntu22.04-x86_64_v3
 -   wqmzrke	      ^findutils@4.9.0%gcc@12.3.0 build_system=autotools patches=440b954 arch=linux-ubuntu22.04-x86_64_v3
 -   2fvcggi	      ^hwloc@2.11.1%gcc@12.3.0~cairo~cuda~gl~libudev+libxml2~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
 -   pvefo63	      ^libfabric@1.22.0%gcc@12.3.0~cuda~debug~kdreg~uring build_system=autotools fabrics=sockets,tcp,udp arch=linux-ubuntu22.04-x86_64_v3
 -   ybxtd2t	      ^libpciaccess@0.17%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   vau55h7		  ^util-macros@1.20.1%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   4fawd6m	      ^libxml2@2.13.4%gcc@12.3.0+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   gdtmvyu		  ^libiconv@1.17%gcc@12.3.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
 -   ifvv7cz		  ^xz@5.4.6%gcc@12.3.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
 -   gutpiwq	      ^pkgconf@2.2.0%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   ferxasc	      ^yaksa@0.3%gcc@12.3.0~cuda~rocm build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   grpnckw		  ^m4@1.4.19%gcc@12.3.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-ubuntu22.04-x86_64_v3
 -   u2gofvq		      ^libsigsegv@2.14%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   wxt7aps		  ^python@3.13.0%gcc@12.3.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   vlatljj		      ^expat@2.6.4%gcc@12.3.0+libbsd build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   3oqufcr			  ^libbsd@0.12.2%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   cpmq5md			      ^libmd@1.0.4%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   jbitwpt		      ^gettext@0.22.5%gcc@12.3.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   j5dljnv			  ^tar@1.34%gcc@12.3.0 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3
 -   zgwe2ua			      ^pigz@2.8%gcc@12.3.0 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3
 -   yk3lkd4			      ^zstd@1.5.6%gcc@12.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
 -   4ga47g7		      ^libffi@3.4.6%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   tb7zvuf		      ^readline@8.2%gcc@12.3.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu22.04-x86_64_v3
 -   b63y5mx		      ^sqlite@3.46.0%gcc@12.3.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   64aenca		      ^util-linux-uuid@2.40.2%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   aliqr5t	  ^openblas@0.3.28%gcc@12.3.0~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile patches=d0b9276 symbol_suffix=none threads=none arch=linux-ubuntu22.04-x86_64_v3
 -   wpymlkh  netlib-scalapack@2.2.0%gcc@12.3.0~ipo~pic+shared build_system=cmake build_type=Release generator=make patches=072b006,1c9ce5f,244a9aa arch=linux-ubuntu22.04-x86_64_v3
 -   suepxfz	  ^openmpi@5.0.5%gcc@12.3.0+atomics~cuda~debug~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~lustre~memchecker~openshmem~romio+rsh~static~two_level_namespace+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-ubuntu22.04-x86_64_v3
 -   yfxrc66	      ^autoconf@2.72%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   e4u6ql7	      ^automake@1.16.5%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   nftqerg	      ^libevent@2.1.12%gcc@12.3.0+openssl build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   xfe5xho	      ^libtool@2.4.7%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   n4nt7lo	      ^numactl@2.0.18%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   3jdzvdm	      ^openssh@9.9p1%gcc@12.3.0+gssapi build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   frb66wn		  ^krb5@1.21.3%gcc@12.3.0+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   fky4vih		      ^bison@3.8.2%gcc@12.3.0~color build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   ht5niwq		  ^libedit@3.1-20240808%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   apxktfa		  ^libxcrypt@4.4.35%gcc@12.3.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-ubuntu22.04-x86_64_v3
 -   tt4vkz3	      ^perl@5.40.0%gcc@12.3.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   4mqvw3u		  ^berkeley-db@18.1.40%gcc@12.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3
 -   j3fd4z5		  ^bzip2@1.0.8%gcc@12.3.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   ttyuhau		  ^gdbm@1.23%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   t7gzekh	      ^pmix@5.0.3%gcc@12.3.0~munge~python~restful build_system=autotools arch=linux-ubuntu22.04-x86_64_v3

This time concretization succeeded. Setting concretizer:unify:false is effectively concretizing each root spec on its own, and then merging the results into the environment. This allows us to have the duplicates we need.

Note

If the environment is expected to have only a few duplicate nodes, then there’s another value we might consider:

$ spack config add concretizer:unify:when_possible

With this option Spack will try to unify the environment in an eager way, solving it in multiple rounds. The concretization at round n will contain all the specs that could not be unified at round n-1, and will consider all the specs from previous rounds for reuse.

Spec matrices

Let’s expand our stack further and consider also linking against different LAPACK providers. We could, of course, add new specs explicitly:

$ spack add netlib-scalapack %gcc@12 ^netlib-lapack ^openmpi
==> Adding netlib-scalapack%gcc@12 ^netlib-lapack ^openmpi to environment /home/spack5/stacks
$ spack add netlib-scalapack %gcc@12 ^netlib-lapack ^mpich
==> Adding netlib-scalapack%gcc@12 ^mpich ^netlib-lapack to environment /home/spack5/stacks

This way of proceeding, though, will become very tedious as soon as more software is requested. The best way to express a cross-product like this in Spack is instead through a matrix:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  specs:
  - gcc@12%gcc@11
  - matrix:
    - [netlib-scalapack]
    - [^openmpi, ^mpich]
    - [^openblas, ^netlib-lapack]
    - ["%gcc@12"]
  view: false
  concretizer:
    unify: false
  compilers:
  - compiler:
      spec: gcc@=12.3.0
      paths:
        cc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gcc
        cxx: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/g++
        f77: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
        fc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
      flags: {}
      operating_system: ubuntu22.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

Matrices will expand to the cross-product of their rows, so this matrix:

- matrix:
 - ["netlib-scalapack"]
 - ["^openmpi", "^mpich"]
 - ["^openblas", "^netlib-lapack"]
 - ["%gcc@12"]

is equivalent to this list of specs:

- "netlib-scalapack %gcc@12 ^openblas ^openmpi"
- "netlib-scalapack %gcc@12 ^openblas ^mpich"
- "netlib-scalapack %gcc@12 ^netlib-lapack ^openmpi"
- "netlib-scalapack %gcc@12 ^netlib-lapack ^mpich"

We are now ready to concretize and install the environment:

$ spack concretize
==> Concretized 2 specs:
 -   5fvq6fg  netlib-scalapack@2.2.0%gcc@12.3.0~ipo~pic+shared build_system=cmake build_type=Release generator=make patches=072b006,1c9ce5f,244a9aa arch=linux-ubuntu22.04-x86_64_v3
 -   yempnaz	  ^cmake@3.30.5%gcc@12.3.0~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release patches=dbc3892 arch=linux-ubuntu22.04-x86_64_v3
 -   wsmx6sp	      ^curl@8.10.1%gcc@12.3.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-ubuntu22.04-x86_64_v3
 -   ippbbwu		  ^nghttp2@1.63.0%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   ioferp7		      ^diffutils@3.10%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   i5fmca4		  ^openssl@3.4.0%gcc@12.3.0~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-x86_64_v3
 -   xilbvuy		      ^ca-certificates-mozilla@2023-05-30%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   uajmsrc	      ^ncurses@6.5%gcc@12.3.0~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-ubuntu22.04-x86_64_v3
 -   i4mwsvm	      ^zlib-ng@2.2.1%gcc@12.3.0+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   w5nlv2k	  ^gcc-runtime@12.3.0%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[e]  gkoh6ax	  ^glibc@2.35%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   zoiouai	  ^gmake@4.4.1%gcc@12.3.0~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   nn52q3b	  ^netlib-lapack@3.11.0%gcc@12.3.0~external-blas~ipo+lapacke+pic+shared~xblas build_system=cmake build_type=Release generator=make arch=linux-ubuntu22.04-x86_64_v3
 -   suepxfz	  ^openmpi@5.0.5%gcc@12.3.0+atomics~cuda~debug~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~lustre~memchecker~openshmem~romio+rsh~static~two_level_namespace+vt+wrapper-rpath build_system=autotools fabrics=none romio-filesystem=none schedulers=none arch=linux-ubuntu22.04-x86_64_v3
 -   yfxrc66	      ^autoconf@2.72%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   grpnckw		  ^m4@1.4.19%gcc@12.3.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-ubuntu22.04-x86_64_v3
 -   u2gofvq		      ^libsigsegv@2.14%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   e4u6ql7	      ^automake@1.16.5%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   2fvcggi	      ^hwloc@2.11.1%gcc@12.3.0~cairo~cuda~gl~libudev+libxml2~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
 -   nftqerg	      ^libevent@2.1.12%gcc@12.3.0+openssl build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   xfe5xho	      ^libtool@2.4.7%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   n4nt7lo	      ^numactl@2.0.18%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   3jdzvdm	      ^openssh@9.9p1%gcc@12.3.0+gssapi build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   frb66wn		  ^krb5@1.21.3%gcc@12.3.0+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   fky4vih		      ^bison@3.8.2%gcc@12.3.0~color build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   jbitwpt		      ^gettext@0.22.5%gcc@12.3.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   j5dljnv			  ^tar@1.34%gcc@12.3.0 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3
 -   zgwe2ua			      ^pigz@2.8%gcc@12.3.0 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3
 -   yk3lkd4			      ^zstd@1.5.6%gcc@12.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
 -   ht5niwq		  ^libedit@3.1-20240808%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   apxktfa		  ^libxcrypt@4.4.35%gcc@12.3.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-ubuntu22.04-x86_64_v3
 -   tt4vkz3	      ^perl@5.40.0%gcc@12.3.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   4mqvw3u		  ^berkeley-db@18.1.40%gcc@12.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3
 -   j3fd4z5		  ^bzip2@1.0.8%gcc@12.3.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   ttyuhau		  ^gdbm@1.23%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   tb7zvuf		      ^readline@8.2%gcc@12.3.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu22.04-x86_64_v3
 -   gutpiwq	      ^pkgconf@2.2.0%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   t7gzekh	      ^pmix@5.0.3%gcc@12.3.0~munge~python~restful build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   bx3g36n  netlib-scalapack@2.2.0%gcc@12.3.0~ipo~pic+shared build_system=cmake build_type=Release generator=make patches=072b006,1c9ce5f,244a9aa arch=linux-ubuntu22.04-x86_64_v3
 -   ezes42n	  ^mpich@4.2.3%gcc@12.3.0~argobots~cuda+fortran+hwloc+hydra+libxml2+pci~rocm+romio~slurm~vci~verbs+wrapperrpath~xpmem build_system=autotools datatype-engine=auto device=ch4 netmod=ofi pmi=default arch=linux-ubuntu22.04-x86_64_v3
 -   wqmzrke	      ^findutils@4.9.0%gcc@12.3.0 build_system=autotools patches=440b954 arch=linux-ubuntu22.04-x86_64_v3
 -   pvefo63	      ^libfabric@1.22.0%gcc@12.3.0~cuda~debug~kdreg~uring build_system=autotools fabrics=sockets,tcp,udp arch=linux-ubuntu22.04-x86_64_v3
 -   ybxtd2t	      ^libpciaccess@0.17%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   vau55h7		  ^util-macros@1.20.1%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   4fawd6m	      ^libxml2@2.13.4%gcc@12.3.0+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   gdtmvyu		  ^libiconv@1.17%gcc@12.3.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
 -   ifvv7cz		  ^xz@5.4.6%gcc@12.3.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
 -   ferxasc	      ^yaksa@0.3%gcc@12.3.0~cuda~rocm build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   wxt7aps		  ^python@3.13.0%gcc@12.3.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   vlatljj		      ^expat@2.6.4%gcc@12.3.0+libbsd build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   3oqufcr			  ^libbsd@0.12.2%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   cpmq5md			      ^libmd@1.0.4%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   4ga47g7		      ^libffi@3.4.6%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   b63y5mx		      ^sqlite@3.46.0%gcc@12.3.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   64aenca		      ^util-linux-uuid@2.40.2%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3

$ spack install
[+] /usr (external glibc-2.35-a7drdl4tlx4bu3mzhor75pskvd3pdot6)
[+] /usr (external glibc-2.35-gkoh6axllpo5u5oc3rv2vglpxcr22dbx)
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-runtime-11.4.0-hshzy762rns57ibvxiyi3qqvc4nehse2
==> Installing gcc-runtime-12.3.0-w5nlv2kp2nkiplculgdgli277r62gdzv [4/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-gcc-runtime-12.3.0-w5nlv2kp2nkiplculgdgli277r62gdzv.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gcc-runtime-12.3.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-gcc-runtime-12.3.0-w5nlv2kp2nkiplculgdgli277r62gdzv.spack
==> Extracting gcc-runtime-12.3.0-w5nlv2kp2nkiplculgdgli277r62gdzv from binary cache
==> gcc-runtime: Successfully installed gcc-runtime-12.3.0-w5nlv2kp2nkiplculgdgli277r62gdzv
  Search: 0.00s.  Fetch: 0.16s.	 Install: 0.44s.  Extract: 0.39s.  Relocate: 0.03s.  Total: 0.61s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gcc-runtime-12.3.0-w5nlv2kp2nkiplculgdgli277r62gdzv
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/zlib-ng-2.2.1-fhud6vqkh7jaesckalgqg5xxml72in6i
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/zstd-1.5.6-t7h6imj35ruac562vr7snzvli2h2tzup
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gmp-6.3.0-5webgyazi3rpemdmmhbip4glviaeet2l
==> Installing gmake-4.4.1-zoiouaigycba7zhhoubeuvjw65uoedaa [8/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-gmake-4.4.1-zoiouaigycba7zhhoubeuvjw65uoedaa.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gmake-4.4.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-gmake-4.4.1-zoiouaigycba7zhhoubeuvjw65uoedaa.spack
==> Extracting gmake-4.4.1-zoiouaigycba7zhhoubeuvjw65uoedaa from binary cache
==> gmake: Successfully installed gmake-4.4.1-zoiouaigycba7zhhoubeuvjw65uoedaa
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gmake-4.4.1-zoiouaigycba7zhhoubeuvjw65uoedaa
==> Installing ca-certificates-mozilla-2023-05-30-xilbvuysbu4g6ndln42xvvip67ay2bh4 [9/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-ca-certificates-mozilla-2023-05-30-xilbvuysbu4g6ndln42xvvip67ay2bh4.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/ca-certificates-mozilla-2023-05-30/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-ca-certificates-mozilla-2023-05-30-xilbvuysbu4g6ndln42xvvip67ay2bh4.spack
==> Extracting ca-certificates-mozilla-2023-05-30-xilbvuysbu4g6ndln42xvvip67ay2bh4 from binary cache
==> ca-certificates-mozilla: Successfully installed ca-certificates-mozilla-2023-05-30-xilbvuysbu4g6ndln42xvvip67ay2bh4
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/ca-certificates-mozilla-2023-05-30-xilbvuysbu4g6ndln42xvvip67ay2bh4
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/mpfr-4.2.1-mdfsfjykvyaknc5yzul4l25aip3rugoe
==> Installing openblas-0.3.28-aliqr5t6k2tzder6t7j6rvvabjqagkph [11/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-openblas-0.3.28-aliqr5t6k2tzder6t7j6rvvabjqagkph.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openblas-0.3.28/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-openblas-0.3.28-aliqr5t6k2tzder6t7j6rvvabjqagkph.spack
==> Extracting openblas-0.3.28-aliqr5t6k2tzder6t7j6rvvabjqagkph from binary cache
==> openblas: Successfully installed openblas-0.3.28-aliqr5t6k2tzder6t7j6rvvabjqagkph
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.99s.  Extract: 0.93s.  Relocate: 0.05s.  Total: 1.01s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openblas-0.3.28-aliqr5t6k2tzder6t7j6rvvabjqagkph
==> Installing pkgconf-2.2.0-gutpiwqutufsytqyn3neyckwem2qjxgo [12/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-pkgconf-2.2.0-gutpiwqutufsytqyn3neyckwem2qjxgo.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/pkgconf-2.2.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-pkgconf-2.2.0-gutpiwqutufsytqyn3neyckwem2qjxgo.spack
==> Extracting pkgconf-2.2.0-gutpiwqutufsytqyn3neyckwem2qjxgo from binary cache
==> pkgconf: Successfully installed pkgconf-2.2.0-gutpiwqutufsytqyn3neyckwem2qjxgo
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/pkgconf-2.2.0-gutpiwqutufsytqyn3neyckwem2qjxgo
==> Installing xz-5.4.6-ifvv7czmuxhion5acm5zlyfqkugbuaya [13/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-xz-5.4.6-ifvv7czmuxhion5acm5zlyfqkugbuaya.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/xz-5.4.6/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-xz-5.4.6-ifvv7czmuxhion5acm5zlyfqkugbuaya.spack
==> Extracting xz-5.4.6-ifvv7czmuxhion5acm5zlyfqkugbuaya from binary cache
==> xz: Successfully installed xz-5.4.6-ifvv7czmuxhion5acm5zlyfqkugbuaya
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.15s.  Extract: 0.11s.  Relocate: 0.01s.  Total: 0.15s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/xz-5.4.6-ifvv7czmuxhion5acm5zlyfqkugbuaya
==> Installing libffi-3.4.6-4ga47g7tj4b6yma6jgigk574sb5iq2rz [14/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libffi-3.4.6-4ga47g7tj4b6yma6jgigk574sb5iq2rz.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libffi-3.4.6/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libffi-3.4.6-4ga47g7tj4b6yma6jgigk574sb5iq2rz.spack
==> Extracting libffi-3.4.6-4ga47g7tj4b6yma6jgigk574sb5iq2rz from binary cache
==> libffi: Successfully installed libffi-3.4.6-4ga47g7tj4b6yma6jgigk574sb5iq2rz
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libffi-3.4.6-4ga47g7tj4b6yma6jgigk574sb5iq2rz
==> Installing libiconv-1.17-gdtmvyudykikcpn3q7izewostyxnstys [15/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libiconv-1.17-gdtmvyudykikcpn3q7izewostyxnstys.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libiconv-1.17/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libiconv-1.17-gdtmvyudykikcpn3q7izewostyxnstys.spack
==> Extracting libiconv-1.17-gdtmvyudykikcpn3q7izewostyxnstys from binary cache
==> libiconv: Successfully installed libiconv-1.17-gdtmvyudykikcpn3q7izewostyxnstys
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.11s.  Extract: 0.07s.  Relocate: 0.01s.  Total: 0.11s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libiconv-1.17-gdtmvyudykikcpn3q7izewostyxnstys
==> Installing zstd-1.5.6-yk3lkd4xdugjxxom2e3fuafzg2ppbnn6 [16/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-zstd-1.5.6-yk3lkd4xdugjxxom2e3fuafzg2ppbnn6.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/zstd-1.5.6/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-zstd-1.5.6-yk3lkd4xdugjxxom2e3fuafzg2ppbnn6.spack
==> Extracting zstd-1.5.6-yk3lkd4xdugjxxom2e3fuafzg2ppbnn6 from binary cache
==> zstd: Successfully installed zstd-1.5.6-yk3lkd4xdugjxxom2e3fuafzg2ppbnn6
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.06s.  Relocate: 0.01s.  Total: 0.10s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/zstd-1.5.6-yk3lkd4xdugjxxom2e3fuafzg2ppbnn6
==> Installing libmd-1.0.4-cpmq5mdiri557vmofyrser77ghcdaykn [17/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libmd-1.0.4-cpmq5mdiri557vmofyrser77ghcdaykn.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libmd-1.0.4/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libmd-1.0.4-cpmq5mdiri557vmofyrser77ghcdaykn.spack
==> Extracting libmd-1.0.4-cpmq5mdiri557vmofyrser77ghcdaykn from binary cache
==> libmd: Successfully installed libmd-1.0.4-cpmq5mdiri557vmofyrser77ghcdaykn
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libmd-1.0.4-cpmq5mdiri557vmofyrser77ghcdaykn
==> Installing zlib-ng-2.2.1-i4mwsvmsv2z7ecaoz3iyozp6dykzrqj7 [18/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-zlib-ng-2.2.1-i4mwsvmsv2z7ecaoz3iyozp6dykzrqj7.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/zlib-ng-2.2.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-zlib-ng-2.2.1-i4mwsvmsv2z7ecaoz3iyozp6dykzrqj7.spack
==> Extracting zlib-ng-2.2.1-i4mwsvmsv2z7ecaoz3iyozp6dykzrqj7 from binary cache
==> zlib-ng: Successfully installed zlib-ng-2.2.1-i4mwsvmsv2z7ecaoz3iyozp6dykzrqj7
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/zlib-ng-2.2.1-i4mwsvmsv2z7ecaoz3iyozp6dykzrqj7
==> Installing findutils-4.9.0-wqmzrkeupxsqrfj6er7bcitcntvdrnlk [19/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-findutils-4.9.0-wqmzrkeupxsqrfj6er7bcitcntvdrnlk.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/findutils-4.9.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-findutils-4.9.0-wqmzrkeupxsqrfj6er7bcitcntvdrnlk.spack
==> Extracting findutils-4.9.0-wqmzrkeupxsqrfj6er7bcitcntvdrnlk from binary cache
==> findutils: Successfully installed findutils-4.9.0-wqmzrkeupxsqrfj6er7bcitcntvdrnlk
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.10s.  Extract: 0.07s.  Relocate: 0.01s.  Total: 0.11s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/findutils-4.9.0-wqmzrkeupxsqrfj6er7bcitcntvdrnlk
==> Installing libsigsegv-2.14-u2gofvqdvwdpj7gwgz3mo56dctmqrvbi [20/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libsigsegv-2.14-u2gofvqdvwdpj7gwgz3mo56dctmqrvbi.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libsigsegv-2.14/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libsigsegv-2.14-u2gofvqdvwdpj7gwgz3mo56dctmqrvbi.spack
==> Extracting libsigsegv-2.14-u2gofvqdvwdpj7gwgz3mo56dctmqrvbi from binary cache
==> libsigsegv: Successfully installed libsigsegv-2.14-u2gofvqdvwdpj7gwgz3mo56dctmqrvbi
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libsigsegv-2.14-u2gofvqdvwdpj7gwgz3mo56dctmqrvbi
==> Installing util-macros-1.20.1-vau55h7rdi4nbwkqd5dj4zjvva5bg34c [21/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-util-macros-1.20.1-vau55h7rdi4nbwkqd5dj4zjvva5bg34c.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/util-macros-1.20.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-util-macros-1.20.1-vau55h7rdi4nbwkqd5dj4zjvva5bg34c.spack
==> Extracting util-macros-1.20.1-vau55h7rdi4nbwkqd5dj4zjvva5bg34c from binary cache
==> util-macros: Successfully installed util-macros-1.20.1-vau55h7rdi4nbwkqd5dj4zjvva5bg34c
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.04s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/util-macros-1.20.1-vau55h7rdi4nbwkqd5dj4zjvva5bg34c
==> Installing berkeley-db-18.1.40-4mqvw3uqymnhnxo57ot37kbznan7zydy [22/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-berkeley-db-18.1.40-4mqvw3uqymnhnxo57ot37kbznan7zydy.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/berkeley-db-18.1.40/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-berkeley-db-18.1.40-4mqvw3uqymnhnxo57ot37kbznan7zydy.spack
==> Extracting berkeley-db-18.1.40-4mqvw3uqymnhnxo57ot37kbznan7zydy from binary cache
==> berkeley-db: Successfully installed berkeley-db-18.1.40-4mqvw3uqymnhnxo57ot37kbznan7zydy
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.17s.  Extract: 0.13s.  Relocate: 0.02s.  Total: 0.17s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/berkeley-db-18.1.40-4mqvw3uqymnhnxo57ot37kbznan7zydy
==> Installing libfabric-1.22.0-pvefo63fafyyn7g6p2ccniaikw7nchi4 [23/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libfabric-1.22.0-pvefo63fafyyn7g6p2ccniaikw7nchi4.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libfabric-1.22.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libfabric-1.22.0-pvefo63fafyyn7g6p2ccniaikw7nchi4.spack
==> Extracting libfabric-1.22.0-pvefo63fafyyn7g6p2ccniaikw7nchi4 from binary cache
==> libfabric: Successfully installed libfabric-1.22.0-pvefo63fafyyn7g6p2ccniaikw7nchi4
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.11s.  Extract: 0.07s.  Relocate: 0.01s.  Total: 0.12s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libfabric-1.22.0-pvefo63fafyyn7g6p2ccniaikw7nchi4
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/mpc-1.3.1-qoauga4bfx37v3lynpeoi4rcy2nw2ek5
==> Installing util-linux-uuid-2.40.2-64aencak6bn7sijcaki5p5bgqgylmxzh [25/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-util-linux-uuid-2.40.2-64aencak6bn7sijcaki5p5bgqgylmxzh.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/util-linux-uuid-2.40.2/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-util-linux-uuid-2.40.2-64aencak6bn7sijcaki5p5bgqgylmxzh.spack
==> Extracting util-linux-uuid-2.40.2-64aencak6bn7sijcaki5p5bgqgylmxzh from binary cache
==> util-linux-uuid: Successfully installed util-linux-uuid-2.40.2-64aencak6bn7sijcaki5p5bgqgylmxzh
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.18s.  Extract: 0.15s.  Relocate: 0.01s.  Total: 0.19s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/util-linux-uuid-2.40.2-64aencak6bn7sijcaki5p5bgqgylmxzh
==> Installing ncurses-6.5-uajmsrcnwwu2a7vc5jooaihbsmywt3sq [26/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-ncurses-6.5-uajmsrcnwwu2a7vc5jooaihbsmywt3sq.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/ncurses-6.5/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-ncurses-6.5-uajmsrcnwwu2a7vc5jooaihbsmywt3sq.spack
==> Extracting ncurses-6.5-uajmsrcnwwu2a7vc5jooaihbsmywt3sq from binary cache
==> ncurses: Successfully installed ncurses-6.5-uajmsrcnwwu2a7vc5jooaihbsmywt3sq
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.63s.  Extract: 0.59s.  Relocate: 0.02s.  Total: 0.64s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/ncurses-6.5-uajmsrcnwwu2a7vc5jooaihbsmywt3sq
==> Installing diffutils-3.10-ioferp7x7e4decmvy5p74xqcbeczeuzc [27/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-diffutils-3.10-ioferp7x7e4decmvy5p74xqcbeczeuzc.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/diffutils-3.10/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-diffutils-3.10-ioferp7x7e4decmvy5p74xqcbeczeuzc.spack
==> Extracting diffutils-3.10-ioferp7x7e4decmvy5p74xqcbeczeuzc from binary cache
==> diffutils: Successfully installed diffutils-3.10-ioferp7x7e4decmvy5p74xqcbeczeuzc
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.06s.  Relocate: 0.01s.  Total: 0.10s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/diffutils-3.10-ioferp7x7e4decmvy5p74xqcbeczeuzc
==> Installing libbsd-0.12.2-3oqufcrjxa46foi3gsamsffbcmc6nhhn [28/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libbsd-0.12.2-3oqufcrjxa46foi3gsamsffbcmc6nhhn.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libbsd-0.12.2/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libbsd-0.12.2-3oqufcrjxa46foi3gsamsffbcmc6nhhn.spack
==> Extracting libbsd-0.12.2-3oqufcrjxa46foi3gsamsffbcmc6nhhn from binary cache
==> libbsd: Successfully installed libbsd-0.12.2-3oqufcrjxa46foi3gsamsffbcmc6nhhn
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.05s.  Relocate: 0.01s.  Total: 0.10s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libbsd-0.12.2-3oqufcrjxa46foi3gsamsffbcmc6nhhn
==> Installing pigz-2.8-zgwe2uay3twlpxzxwnp3sw7n5fgwysht [29/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-pigz-2.8-zgwe2uay3twlpxzxwnp3sw7n5fgwysht.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/pigz-2.8/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-pigz-2.8-zgwe2uay3twlpxzxwnp3sw7n5fgwysht.spack
==> Extracting pigz-2.8-zgwe2uay3twlpxzxwnp3sw7n5fgwysht from binary cache
==> pigz: Successfully installed pigz-2.8-zgwe2uay3twlpxzxwnp3sw7n5fgwysht
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/pigz-2.8-zgwe2uay3twlpxzxwnp3sw7n5fgwysht
==> Installing libxml2-2.13.4-4fawd6mhagkgk5lvgp67szz5gi4nhkrl [30/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libxml2-2.13.4-4fawd6mhagkgk5lvgp67szz5gi4nhkrl.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libxml2-2.13.4/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libxml2-2.13.4-4fawd6mhagkgk5lvgp67szz5gi4nhkrl.spack
==> Extracting libxml2-2.13.4-4fawd6mhagkgk5lvgp67szz5gi4nhkrl from binary cache
==> libxml2: Successfully installed libxml2-2.13.4-4fawd6mhagkgk5lvgp67szz5gi4nhkrl
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.16s.  Extract: 0.12s.  Relocate: 0.02s.  Total: 0.17s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libxml2-2.13.4-4fawd6mhagkgk5lvgp67szz5gi4nhkrl
==> Installing libpciaccess-0.17-ybxtd2t7kr5453br7gsy66phfkjp7zmf [31/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libpciaccess-0.17-ybxtd2t7kr5453br7gsy66phfkjp7zmf.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libpciaccess-0.17/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libpciaccess-0.17-ybxtd2t7kr5453br7gsy66phfkjp7zmf.spack
==> Extracting libpciaccess-0.17-ybxtd2t7kr5453br7gsy66phfkjp7zmf from binary cache
==> libpciaccess: Successfully installed libpciaccess-0.17-ybxtd2t7kr5453br7gsy66phfkjp7zmf
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libpciaccess-0.17-ybxtd2t7kr5453br7gsy66phfkjp7zmf
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-jjgfqpdmihb2cq5ca7txia5oshruq56q
==> Installing libedit-3.1-20240808-ht5niwqaryv67wtxh2x7lkchhrweybuu [33/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libedit-3.1-20240808-ht5niwqaryv67wtxh2x7lkchhrweybuu.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libedit-3.1-20240808/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libedit-3.1-20240808-ht5niwqaryv67wtxh2x7lkchhrweybuu.spack
==> Extracting libedit-3.1-20240808-ht5niwqaryv67wtxh2x7lkchhrweybuu from binary cache
==> libedit: Successfully installed libedit-3.1-20240808-ht5niwqaryv67wtxh2x7lkchhrweybuu
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.04s.  Relocate: 0.01s.  Total: 0.09s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libedit-3.1-20240808-ht5niwqaryv67wtxh2x7lkchhrweybuu
==> Installing readline-8.2-tb7zvufjv2zsr4br7chc5uextvcydde6 [34/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-readline-8.2-tb7zvufjv2zsr4br7chc5uextvcydde6.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/readline-8.2/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-readline-8.2-tb7zvufjv2zsr4br7chc5uextvcydde6.spack
==> Extracting readline-8.2-tb7zvufjv2zsr4br7chc5uextvcydde6 from binary cache
==> readline: Successfully installed readline-8.2-tb7zvufjv2zsr4br7chc5uextvcydde6
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.10s.  Extract: 0.06s.  Relocate: 0.01s.  Total: 0.11s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/readline-8.2-tb7zvufjv2zsr4br7chc5uextvcydde6
==> Installing nghttp2-1.63.0-ippbbwuh3aggwhh4dmj4e36flmx3g7ew [35/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-nghttp2-1.63.0-ippbbwuh3aggwhh4dmj4e36flmx3g7ew.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/nghttp2-1.63.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-nghttp2-1.63.0-ippbbwuh3aggwhh4dmj4e36flmx3g7ew.spack
==> Extracting nghttp2-1.63.0-ippbbwuh3aggwhh4dmj4e36flmx3g7ew from binary cache
==> nghttp2: Successfully installed nghttp2-1.63.0-ippbbwuh3aggwhh4dmj4e36flmx3g7ew
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.04s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/nghttp2-1.63.0-ippbbwuh3aggwhh4dmj4e36flmx3g7ew
==> Installing m4-1.4.19-grpnckwm3ev2kvmdgd7ld23o7t2qlp4z [36/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-m4-1.4.19-grpnckwm3ev2kvmdgd7ld23o7t2qlp4z.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/m4-1.4.19/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-m4-1.4.19-grpnckwm3ev2kvmdgd7ld23o7t2qlp4z.spack
==> Extracting m4-1.4.19-grpnckwm3ev2kvmdgd7ld23o7t2qlp4z from binary cache
==> m4: Successfully installed m4-1.4.19-grpnckwm3ev2kvmdgd7ld23o7t2qlp4z
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.04s.  Relocate: 0.01s.  Total: 0.09s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/m4-1.4.19-grpnckwm3ev2kvmdgd7ld23o7t2qlp4z
==> Installing bzip2-1.0.8-j3fd4z5i2t5bs5tqtjykksiarh623jkh [37/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-bzip2-1.0.8-j3fd4z5i2t5bs5tqtjykksiarh623jkh.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/bzip2-1.0.8/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-bzip2-1.0.8-j3fd4z5i2t5bs5tqtjykksiarh623jkh.spack
==> Extracting bzip2-1.0.8-j3fd4z5i2t5bs5tqtjykksiarh623jkh from binary cache
==> bzip2: Successfully installed bzip2-1.0.8-j3fd4z5i2t5bs5tqtjykksiarh623jkh
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/bzip2-1.0.8-j3fd4z5i2t5bs5tqtjykksiarh623jkh
==> Installing expat-2.6.4-vlatljjv2t62qt7s5paquxtvpvail4tl [38/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-expat-2.6.4-vlatljjv2t62qt7s5paquxtvpvail4tl.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/expat-2.6.4/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-expat-2.6.4-vlatljjv2t62qt7s5paquxtvpvail4tl.spack
==> Extracting expat-2.6.4-vlatljjv2t62qt7s5paquxtvpvail4tl from binary cache
==> expat: Successfully installed expat-2.6.4-vlatljjv2t62qt7s5paquxtvpvail4tl
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.03s.  Relocate: 0.02s.  Total: 0.07s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/expat-2.6.4-vlatljjv2t62qt7s5paquxtvpvail4tl
==> Installing hwloc-2.11.1-2fvcggi3gqewbovwkprpcvppiiexsyxh [39/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-hwloc-2.11.1-2fvcggi3gqewbovwkprpcvppiiexsyxh.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/hwloc-2.11.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-hwloc-2.11.1-2fvcggi3gqewbovwkprpcvppiiexsyxh.spack
==> Extracting hwloc-2.11.1-2fvcggi3gqewbovwkprpcvppiiexsyxh from binary cache
==> hwloc: Successfully installed hwloc-2.11.1-2fvcggi3gqewbovwkprpcvppiiexsyxh
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.26s.  Extract: 0.20s.  Relocate: 0.03s.  Total: 0.27s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/hwloc-2.11.1-2fvcggi3gqewbovwkprpcvppiiexsyxh
==> Installing sqlite-3.46.0-b63y5mxrvtgp35ruiertnxzrnwzoxk44 [40/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-sqlite-3.46.0-b63y5mxrvtgp35ruiertnxzrnwzoxk44.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/sqlite-3.46.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-sqlite-3.46.0-b63y5mxrvtgp35ruiertnxzrnwzoxk44.spack
==> Extracting sqlite-3.46.0-b63y5mxrvtgp35ruiertnxzrnwzoxk44 from binary cache
==> sqlite: Successfully installed sqlite-3.46.0-b63y5mxrvtgp35ruiertnxzrnwzoxk44
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.43s.  Extract: 0.38s.  Relocate: 0.03s.  Total: 0.44s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/sqlite-3.46.0-b63y5mxrvtgp35ruiertnxzrnwzoxk44
==> Installing gdbm-1.23-ttyuhau65qtwjzely7v6tkbnu33mrxeg [41/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-gdbm-1.23-ttyuhau65qtwjzely7v6tkbnu33mrxeg.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gdbm-1.23/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-gdbm-1.23-ttyuhau65qtwjzely7v6tkbnu33mrxeg.spack
==> Extracting gdbm-1.23-ttyuhau65qtwjzely7v6tkbnu33mrxeg from binary cache
==> gdbm: Successfully installed gdbm-1.23-ttyuhau65qtwjzely7v6tkbnu33mrxeg
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.05s.  Relocate: 0.02s.  Total: 0.10s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gdbm-1.23-ttyuhau65qtwjzely7v6tkbnu33mrxeg
==> Installing bison-3.8.2-fky4vihmlahehnlp653pujxj7hv6tvtz [42/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-bison-3.8.2-fky4vihmlahehnlp653pujxj7hv6tvtz.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/bison-3.8.2/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-bison-3.8.2-fky4vihmlahehnlp653pujxj7hv6tvtz.spack
==> Extracting bison-3.8.2-fky4vihmlahehnlp653pujxj7hv6tvtz from binary cache
==> bison: Successfully installed bison-3.8.2-fky4vihmlahehnlp653pujxj7hv6tvtz
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.15s.  Extract: 0.11s.  Relocate: 0.01s.  Total: 0.16s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/bison-3.8.2-fky4vihmlahehnlp653pujxj7hv6tvtz
==> Installing libtool-2.4.7-xfe5xhomituzbaon2uwsukotf6kvmc2p [43/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libtool-2.4.7-xfe5xhomituzbaon2uwsukotf6kvmc2p.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libtool-2.4.7/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libtool-2.4.7-xfe5xhomituzbaon2uwsukotf6kvmc2p.spack
==> Extracting libtool-2.4.7-xfe5xhomituzbaon2uwsukotf6kvmc2p from binary cache
==> libtool: Successfully installed libtool-2.4.7-xfe5xhomituzbaon2uwsukotf6kvmc2p
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.05s.  Relocate: 0.02s.  Total: 0.10s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libtool-2.4.7-xfe5xhomituzbaon2uwsukotf6kvmc2p
==> Installing tar-1.34-j5dljnv4mc5mviozu4ouxfl4r276ftst [44/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-tar-1.34-j5dljnv4mc5mviozu4ouxfl4r276ftst.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/tar-1.34/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-tar-1.34-j5dljnv4mc5mviozu4ouxfl4r276ftst.spack
==> Extracting tar-1.34-j5dljnv4mc5mviozu4ouxfl4r276ftst from binary cache
==> tar: Successfully installed tar-1.34-j5dljnv4mc5mviozu4ouxfl4r276ftst
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.15s.  Extract: 0.10s.  Relocate: 0.02s.  Total: 0.16s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/tar-1.34-j5dljnv4mc5mviozu4ouxfl4r276ftst
==> Installing perl-5.40.0-tt4vkz363spwljfu4mvpr4sd6eu2ibyv [45/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-perl-5.40.0-tt4vkz363spwljfu4mvpr4sd6eu2ibyv.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/perl-5.40.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-perl-5.40.0-tt4vkz363spwljfu4mvpr4sd6eu2ibyv.spack
==> Extracting perl-5.40.0-tt4vkz363spwljfu4mvpr4sd6eu2ibyv from binary cache
==> perl: Successfully installed perl-5.40.0-tt4vkz363spwljfu4mvpr4sd6eu2ibyv
  Search: 0.00s.  Fetch: 0.02s.	 Install: 1.02s.  Extract: 0.85s.  Relocate: 0.14s.  Total: 1.03s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/perl-5.40.0-tt4vkz363spwljfu4mvpr4sd6eu2ibyv
==> Installing gettext-0.22.5-jbitwptksuwvgydlf2jhljxiu4ku2z7o [46/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-gettext-0.22.5-jbitwptksuwvgydlf2jhljxiu4ku2z7o.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gettext-0.22.5/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-gettext-0.22.5-jbitwptksuwvgydlf2jhljxiu4ku2z7o.spack
==> Extracting gettext-0.22.5-jbitwptksuwvgydlf2jhljxiu4ku2z7o from binary cache
==> gettext: Successfully installed gettext-0.22.5-jbitwptksuwvgydlf2jhljxiu4ku2z7o
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.74s.  Extract: 0.67s.  Relocate: 0.05s.  Total: 0.75s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gettext-0.22.5-jbitwptksuwvgydlf2jhljxiu4ku2z7o
==> Installing openssl-3.4.0-i5fmca4xdydwelmicbx3wanqiiqrf554 [47/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-openssl-3.4.0-i5fmca4xdydwelmicbx3wanqiiqrf554.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openssl-3.4.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-openssl-3.4.0-i5fmca4xdydwelmicbx3wanqiiqrf554.spack
==> Extracting openssl-3.4.0-i5fmca4xdydwelmicbx3wanqiiqrf554 from binary cache
==> openssl: Successfully installed openssl-3.4.0-i5fmca4xdydwelmicbx3wanqiiqrf554
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.34s.  Extract: 0.29s.  Relocate: 0.02s.  Total: 0.35s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openssl-3.4.0-i5fmca4xdydwelmicbx3wanqiiqrf554
==> Installing autoconf-2.72-yfxrc66azgrsd4d5c6pifal2rlfgyaxc [48/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-autoconf-2.72-yfxrc66azgrsd4d5c6pifal2rlfgyaxc.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/autoconf-2.72/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-autoconf-2.72-yfxrc66azgrsd4d5c6pifal2rlfgyaxc.spack
==> Extracting autoconf-2.72-yfxrc66azgrsd4d5c6pifal2rlfgyaxc from binary cache
==> autoconf: Successfully installed autoconf-2.72-yfxrc66azgrsd4d5c6pifal2rlfgyaxc
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.11s.  Extract: 0.06s.  Relocate: 0.02s.  Total: 0.12s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/autoconf-2.72-yfxrc66azgrsd4d5c6pifal2rlfgyaxc
==> Installing libxcrypt-4.4.35-apxktfacu5pzihl72vkehndvgzoqrsgq [49/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libxcrypt-4.4.35-apxktfacu5pzihl72vkehndvgzoqrsgq.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libxcrypt-4.4.35/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libxcrypt-4.4.35-apxktfacu5pzihl72vkehndvgzoqrsgq.spack
==> Extracting libxcrypt-4.4.35-apxktfacu5pzihl72vkehndvgzoqrsgq from binary cache
==> libxcrypt: Successfully installed libxcrypt-4.4.35-apxktfacu5pzihl72vkehndvgzoqrsgq
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.04s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libxcrypt-4.4.35-apxktfacu5pzihl72vkehndvgzoqrsgq
==> Installing curl-8.10.1-wsmx6spfntot5xmdbbj5nmjwvfyr7ybg [50/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-curl-8.10.1-wsmx6spfntot5xmdbbj5nmjwvfyr7ybg.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/curl-8.10.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-curl-8.10.1-wsmx6spfntot5xmdbbj5nmjwvfyr7ybg.spack
==> Extracting curl-8.10.1-wsmx6spfntot5xmdbbj5nmjwvfyr7ybg from binary cache
==> curl: Successfully installed curl-8.10.1-wsmx6spfntot5xmdbbj5nmjwvfyr7ybg
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.18s.  Extract: 0.13s.  Relocate: 0.02s.  Total: 0.18s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/curl-8.10.1-wsmx6spfntot5xmdbbj5nmjwvfyr7ybg
==> Installing krb5-1.21.3-frb66wnvsav46slm5v44war7q7uqszf6 [51/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-krb5-1.21.3-frb66wnvsav46slm5v44war7q7uqszf6.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/krb5-1.21.3/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-krb5-1.21.3-frb66wnvsav46slm5v44war7q7uqszf6.spack
==> Extracting krb5-1.21.3-frb66wnvsav46slm5v44war7q7uqszf6 from binary cache
==> krb5: Successfully installed krb5-1.21.3-frb66wnvsav46slm5v44war7q7uqszf6
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.19s.  Extract: 0.11s.  Relocate: 0.04s.  Total: 0.20s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/krb5-1.21.3-frb66wnvsav46slm5v44war7q7uqszf6
==> Installing python-3.13.0-wxt7apsxjg2lmyyldxxselyr5kd72iyj [52/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-python-3.13.0-wxt7apsxjg2lmyyldxxselyr5kd72iyj.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/python-3.13.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-python-3.13.0-wxt7apsxjg2lmyyldxxselyr5kd72iyj.spack
==> Extracting python-3.13.0-wxt7apsxjg2lmyyldxxselyr5kd72iyj from binary cache
==> python: Successfully installed python-3.13.0-wxt7apsxjg2lmyyldxxselyr5kd72iyj
  Search: 0.00s.  Fetch: 0.06s.	 Install: 4.36s.  Extract: 4.18s.  Relocate: 0.14s.  Total: 4.42s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/python-3.13.0-wxt7apsxjg2lmyyldxxselyr5kd72iyj
==> Installing libevent-2.1.12-nftqerg3734tw6jlwe5nsjrg373ublgi [53/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libevent-2.1.12-nftqerg3734tw6jlwe5nsjrg373ublgi.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libevent-2.1.12/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-libevent-2.1.12-nftqerg3734tw6jlwe5nsjrg373ublgi.spack
==> Extracting libevent-2.1.12-nftqerg3734tw6jlwe5nsjrg373ublgi from binary cache
==> libevent: Successfully installed libevent-2.1.12-nftqerg3734tw6jlwe5nsjrg373ublgi
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.16s.  Extract: 0.11s.  Relocate: 0.02s.  Total: 0.17s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libevent-2.1.12-nftqerg3734tw6jlwe5nsjrg373ublgi
==> Installing automake-1.16.5-e4u6ql7bmet3komgmecyklgk5xfriide [54/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-automake-1.16.5-e4u6ql7bmet3komgmecyklgk5xfriide.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/automake-1.16.5/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-automake-1.16.5-e4u6ql7bmet3komgmecyklgk5xfriide.spack
==> Extracting automake-1.16.5-e4u6ql7bmet3komgmecyklgk5xfriide from binary cache
==> automake: Successfully installed automake-1.16.5-e4u6ql7bmet3komgmecyklgk5xfriide
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.11s.  Extract: 0.05s.  Relocate: 0.03s.  Total: 0.12s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/automake-1.16.5-e4u6ql7bmet3komgmecyklgk5xfriide
==> Installing cmake-3.30.5-yempnazpumsvpjmrny2j6nkstc7g67d3 [55/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-cmake-3.30.5-yempnazpumsvpjmrny2j6nkstc7g67d3.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/cmake-3.30.5/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-cmake-3.30.5-yempnazpumsvpjmrny2j6nkstc7g67d3.spack
==> Extracting cmake-3.30.5-yempnazpumsvpjmrny2j6nkstc7g67d3 from binary cache
==> cmake: Successfully installed cmake-3.30.5-yempnazpumsvpjmrny2j6nkstc7g67d3
  Search: 0.00s.  Fetch: 0.02s.	 Install: 1.50s.  Extract: 1.40s.  Relocate: 0.07s.  Total: 1.52s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/cmake-3.30.5-yempnazpumsvpjmrny2j6nkstc7g67d3
==> Installing openssh-9.9p1-3jdzvdmyxcyc6k2rh6p3anmik4zrloik [56/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-openssh-9.9p1-3jdzvdmyxcyc6k2rh6p3anmik4zrloik.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openssh-9.9p1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-openssh-9.9p1-3jdzvdmyxcyc6k2rh6p3anmik4zrloik.spack
==> Extracting openssh-9.9p1-3jdzvdmyxcyc6k2rh6p3anmik4zrloik from binary cache
==> openssh: Successfully installed openssh-9.9p1-3jdzvdmyxcyc6k2rh6p3anmik4zrloik
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.20s.  Extract: 0.12s.  Relocate: 0.04s.  Total: 0.21s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openssh-9.9p1-3jdzvdmyxcyc6k2rh6p3anmik4zrloik
==> Installing pmix-5.0.3-t7gzekhpropf62jjhntf7l2zjbcumqre [57/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-pmix-5.0.3-t7gzekhpropf62jjhntf7l2zjbcumqre.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/pmix-5.0.3/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-pmix-5.0.3-t7gzekhpropf62jjhntf7l2zjbcumqre.spack
==> Extracting pmix-5.0.3-t7gzekhpropf62jjhntf7l2zjbcumqre from binary cache
==> pmix: Successfully installed pmix-5.0.3-t7gzekhpropf62jjhntf7l2zjbcumqre
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.31s.  Extract: 0.24s.  Relocate: 0.04s.  Total: 0.32s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/pmix-5.0.3-t7gzekhpropf62jjhntf7l2zjbcumqre
==> Installing yaksa-0.3-ferxascsq35y5jrtdxc4m6w7vq5i5g6o [58/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-yaksa-0.3-ferxascsq35y5jrtdxc4m6w7vq5i5g6o.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/yaksa-0.3/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-yaksa-0.3-ferxascsq35y5jrtdxc4m6w7vq5i5g6o.spack
==> Extracting yaksa-0.3-ferxascsq35y5jrtdxc4m6w7vq5i5g6o from binary cache
==> yaksa: Successfully installed yaksa-0.3-ferxascsq35y5jrtdxc4m6w7vq5i5g6o
  Search: 0.00s.  Fetch: 0.02s.	 Install: 1.00s.  Extract: 0.92s.  Relocate: 0.05s.  Total: 1.02s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/yaksa-0.3-ferxascsq35y5jrtdxc4m6w7vq5i5g6o
==> Installing numactl-2.0.18-n4nt7lopsr6oxmo42tw2nqfbjwtnkzqm [59/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-numactl-2.0.18-n4nt7lopsr6oxmo42tw2nqfbjwtnkzqm.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/numactl-2.0.18/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-numactl-2.0.18-n4nt7lopsr6oxmo42tw2nqfbjwtnkzqm.spack
==> Extracting numactl-2.0.18-n4nt7lopsr6oxmo42tw2nqfbjwtnkzqm from binary cache
==> numactl: Successfully installed numactl-2.0.18-n4nt7lopsr6oxmo42tw2nqfbjwtnkzqm
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.03s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/numactl-2.0.18-n4nt7lopsr6oxmo42tw2nqfbjwtnkzqm
==> Installing netlib-lapack-3.11.0-nn52q3blozedtfiwwdeyvint2pbkjybn [60/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-netlib-lapack-3.11.0-nn52q3blozedtfiwwdeyvint2pbkjybn.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-lapack-3.11.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-netlib-lapack-3.11.0-nn52q3blozedtfiwwdeyvint2pbkjybn.spack
==> Extracting netlib-lapack-3.11.0-nn52q3blozedtfiwwdeyvint2pbkjybn from binary cache
==> netlib-lapack: Successfully installed netlib-lapack-3.11.0-nn52q3blozedtfiwwdeyvint2pbkjybn
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.24s.  Extract: 0.19s.  Relocate: 0.02s.  Total: 0.25s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-lapack-3.11.0-nn52q3blozedtfiwwdeyvint2pbkjybn
==> Installing mpich-4.2.3-ezes42nge7p2vj6tyaqg5y4dhhxzhzij [61/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-mpich-4.2.3-ezes42nge7p2vj6tyaqg5y4dhhxzhzij.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/mpich-4.2.3/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-mpich-4.2.3-ezes42nge7p2vj6tyaqg5y4dhhxzhzij.spack
==> Extracting mpich-4.2.3-ezes42nge7p2vj6tyaqg5y4dhhxzhzij from binary cache
==> mpich: Successfully installed mpich-4.2.3-ezes42nge7p2vj6tyaqg5y4dhhxzhzij
  Search: 0.00s.  Fetch: 0.01s.	 Install: 1.11s.  Extract: 0.97s.  Relocate: 0.10s.  Total: 1.12s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/mpich-4.2.3-ezes42nge7p2vj6tyaqg5y4dhhxzhzij
==> Installing openmpi-5.0.5-suepxfzmhyr4ewe3xorze6ftrvr46fna [62/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-openmpi-5.0.5-suepxfzmhyr4ewe3xorze6ftrvr46fna.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openmpi-5.0.5/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-openmpi-5.0.5-suepxfzmhyr4ewe3xorze6ftrvr46fna.spack
==> Extracting openmpi-5.0.5-suepxfzmhyr4ewe3xorze6ftrvr46fna from binary cache
==> openmpi: Successfully installed openmpi-5.0.5-suepxfzmhyr4ewe3xorze6ftrvr46fna
  Search: 0.00s.  Fetch: 0.02s.	 Install: 1.13s.  Extract: 1.04s.  Relocate: 0.05s.  Total: 1.14s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openmpi-5.0.5-suepxfzmhyr4ewe3xorze6ftrvr46fna
==> Installing netlib-scalapack-2.2.0-bx3g36nfqncvgsc63nfz7ekbzam5c2gf [63/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-netlib-scalapack-2.2.0-bx3g36nfqncvgsc63nfz7ekbzam5c2gf.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-netlib-scalapack-2.2.0-bx3g36nfqncvgsc63nfz7ekbzam5c2gf.spack
==> Extracting netlib-scalapack-2.2.0-bx3g36nfqncvgsc63nfz7ekbzam5c2gf from binary cache
==> netlib-scalapack: Successfully installed netlib-scalapack-2.2.0-bx3g36nfqncvgsc63nfz7ekbzam5c2gf
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.19s.  Extract: 0.11s.  Relocate: 0.04s.  Total: 0.20s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0-bx3g36nfqncvgsc63nfz7ekbzam5c2gf
==> Installing netlib-scalapack-2.2.0-me353pomglevvhbry6mehg7ctzcnneap [64/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-netlib-scalapack-2.2.0-me353pomglevvhbry6mehg7ctzcnneap.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-netlib-scalapack-2.2.0-me353pomglevvhbry6mehg7ctzcnneap.spack
==> Extracting netlib-scalapack-2.2.0-me353pomglevvhbry6mehg7ctzcnneap from binary cache
==> netlib-scalapack: Successfully installed netlib-scalapack-2.2.0-me353pomglevvhbry6mehg7ctzcnneap
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.20s.  Extract: 0.11s.  Relocate: 0.04s.  Total: 0.21s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0-me353pomglevvhbry6mehg7ctzcnneap
==> Installing netlib-scalapack-2.2.0-5fvq6fgjhaihmeoipzgrpf4tcxc427km [65/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-netlib-scalapack-2.2.0-5fvq6fgjhaihmeoipzgrpf4tcxc427km.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-netlib-scalapack-2.2.0-5fvq6fgjhaihmeoipzgrpf4tcxc427km.spack
==> Extracting netlib-scalapack-2.2.0-5fvq6fgjhaihmeoipzgrpf4tcxc427km from binary cache
==> netlib-scalapack: Successfully installed netlib-scalapack-2.2.0-5fvq6fgjhaihmeoipzgrpf4tcxc427km
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.20s.  Extract: 0.12s.  Relocate: 0.04s.  Total: 0.22s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0-5fvq6fgjhaihmeoipzgrpf4tcxc427km
==> Installing netlib-scalapack-2.2.0-wpymlkhhvzdj3cu3spayvjvzh5kqv4jo [66/66]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-netlib-scalapack-2.2.0-wpymlkhhvzdj3cu3spayvjvzh5kqv4jo.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-netlib-scalapack-2.2.0-wpymlkhhvzdj3cu3spayvjvzh5kqv4jo.spack
==> Extracting netlib-scalapack-2.2.0-wpymlkhhvzdj3cu3spayvjvzh5kqv4jo from binary cache
==> netlib-scalapack: Successfully installed netlib-scalapack-2.2.0-wpymlkhhvzdj3cu3spayvjvzh5kqv4jo
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.27s.  Extract: 0.12s.  Relocate: 0.10s.  Total: 0.28s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0-wpymlkhhvzdj3cu3spayvjvzh5kqv4jo

Let’s double check which specs we have installed so far:

$ spack find
==> In environment /home/spack5/stacks
==> 5 root specs
-- no arch / gcc@11 ---------------------------------------------
[+] gcc@12%gcc@11

-- no arch / gcc@12 ---------------------------------------------
[+] netlib-scalapack%gcc@12  [+] netlib-scalapack%gcc@12  [+] netlib-scalapack%gcc@12  [+] netlib-scalapack%gcc@12

-- linux-ubuntu22.04-x86_64_v3 / gcc@11.4.0 ---------------------
autoconf@2.72		     bzip2@1.0.8      gcc@12.3.0	  glibc@2.35	 libsigsegv@2.14  mpc@1.3.1    pigz@2.8	      texinfo@7.1
autoconf-archive@2023.02.20  diffutils@3.10   gcc-runtime@11.4.0  gmake@4.4.1	 libtool@2.4.7	  mpfr@4.2.1   pkgconf@2.2.0  xz@5.4.6
automake@1.16.5		     findutils@4.9.0  gdbm@1.23		  gmp@6.3.0	 libxml2@2.13.4	  ncurses@6.5  readline@8.2   zlib-ng@2.2.1
berkeley-db@18.1.40	     gawk@5.3.1	      gettext@0.22.5	  libiconv@1.17	 m4@1.4.19	  perl@5.40.0  tar@1.34	      zstd@1.5.6

-- linux-ubuntu22.04-x86_64_v3 / gcc@12.3.0 ---------------------
autoconf@2.72			    diffutils@3.10	hwloc@2.11.1	      libmd@1.0.4	 ncurses@6.5		 openblas@0.3.28  python@3.13.0		  zlib-ng@2.2.1
automake@1.16.5			    expat@2.6.4		krb5@1.21.3	      libpciaccess@0.17	 netlib-lapack@3.11.0	 openmpi@5.0.5	  readline@8.2		  zstd@1.5.6
berkeley-db@18.1.40		    findutils@4.9.0	libbsd@0.12.2	      libsigsegv@2.14	 netlib-scalapack@2.2.0	 openssh@9.9p1	  sqlite@3.46.0
bison@3.8.2			    gcc-runtime@12.3.0	libedit@3.1-20240808  libtool@2.4.7	 netlib-scalapack@2.2.0	 openssl@3.4.0	  tar@1.34
bzip2@1.0.8			    gdbm@1.23		libevent@2.1.12	      libxcrypt@4.4.35	 netlib-scalapack@2.2.0	 perl@5.40.0	  util-linux-uuid@2.40.2
ca-certificates-mozilla@2023-05-30  gettext@0.22.5	libfabric@1.22.0      libxml2@2.13.4	 netlib-scalapack@2.2.0	 pigz@2.8	  util-macros@1.20.1
cmake@3.30.5			    glibc@2.35		libffi@3.4.6	      m4@1.4.19		 nghttp2@1.63.0		 pkgconf@2.2.0	  xz@5.4.6
curl@8.10.1			    gmake@4.4.1		libiconv@1.17	      mpich@4.2.3	 numactl@2.0.18		 pmix@5.0.3	  yaksa@0.3
==> 90 installed packages
==> 0 concretized packages to be installed (show with `spack find -c`)

As we can see we have our four variations of netlib-scalapack installed.

Reusable definitions

So far, we have seen how we can use spec matrices to generate cross-product specs from rows containing a list of constraints. A common situation you will encounter with large deployments is the necessity to add multiple matrices to the list of specs, that possibly share some of those rows.

To reduce the amount of duplication needed in the manifest file, and thus the maintenance burden for people maintaining it, Spack allows to define lists of constraints under the definitions attribute, and expand them later when needed. Let’s rewrite our manifest in that sense:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [ openmpi, mpich ]
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]

  view: false
  concretizer:
    unify: false
  compilers:
  - compiler:
      spec: gcc@=12.3.0
      paths:
        cc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gcc
        cxx: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/g++
        f77: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
        fc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
      flags: {}
      operating_system: ubuntu22.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

And check that re-concretizing won’t change the environment:

$ spack concretize
==> No new specs to concretize.
$ spack find -l
==> In environment /home/spack5/stacks
==> 5 root specs
-- no arch / gcc@11 ---------------------------------------------
[+] jjgfqpd gcc@12%gcc@11

-- no arch / gcc@12 ---------------------------------------------
[+] bx3g36n netlib-scalapack%gcc@12  [+] me353po netlib-scalapack%gcc@12  [+] 5fvq6fg netlib-scalapack%gcc@12  [+] wpymlkh netlib-scalapack%gcc@12

-- linux-ubuntu22.04-x86_64_v3 / gcc@11.4.0 ---------------------
vgucajy autoconf@2.72		     lljulvx diffutils@3.10	 jj2atvs gdbm@1.23	 bsga3no libiconv@1.17	  qoauga4 mpc@1.3.1    oplsmxr pkgconf@2.2.0  fhud6vq zlib-ng@2.2.1
6s34ic3 autoconf-archive@2023.02.20  2sbkhch findutils@4.9.0	 bexdfw2 gettext@0.22.5	 ea6qziv libsigsegv@2.14  mdfsfjy mpfr@4.2.1   aylebxv readline@8.2   t7h6imj zstd@1.5.6
awgfaon automake@1.16.5		     x6mwj3l gawk@5.3.1		 a7drdl4 glibc@2.35	 tt3byem libtool@2.4.7	  rvg7j6b ncurses@6.5  mmv6i4n tar@1.34
cexlioh berkeley-db@18.1.40	     jjgfqpd gcc@12.3.0		 srkzfjr gmake@4.4.1	 wozcmyn libxml2@2.13.4	  cwpt5ec perl@5.40.0  ljsbwjc texinfo@7.1
ewqc7cx bzip2@1.0.8		     hshzy76 gcc-runtime@11.4.0	 5webgya gmp@6.3.0	 qaspjyq m4@1.4.19	  pcwnu2w pigz@2.8     2fvrfr6 xz@5.4.6

-- linux-ubuntu22.04-x86_64_v3 / gcc@12.3.0 ---------------------
yfxrc66 autoconf@2.72			    ttyuhau gdbm@1.23		  cpmq5md libmd@1.0.4		  5fvq6fg netlib-scalapack@2.2.0  wxt7aps python@3.13.0
e4u6ql7 automake@1.16.5			    jbitwpt gettext@0.22.5	  ybxtd2t libpciaccess@0.17	  bx3g36n netlib-scalapack@2.2.0  tb7zvuf readline@8.2
4mqvw3u berkeley-db@18.1.40		    gkoh6ax glibc@2.35		  u2gofvq libsigsegv@2.14	  ippbbwu nghttp2@1.63.0	  b63y5mx sqlite@3.46.0
fky4vih bison@3.8.2			    zoiouai gmake@4.4.1		  xfe5xho libtool@2.4.7		  n4nt7lo numactl@2.0.18	  j5dljnv tar@1.34
j3fd4z5 bzip2@1.0.8			    2fvcggi hwloc@2.11.1	  apxktfa libxcrypt@4.4.35	  aliqr5t openblas@0.3.28	  64aenca util-linux-uuid@2.40.2
xilbvuy ca-certificates-mozilla@2023-05-30  frb66wn krb5@1.21.3		  4fawd6m libxml2@2.13.4	  suepxfz openmpi@5.0.5		  vau55h7 util-macros@1.20.1
yempnaz cmake@3.30.5			    3oqufcr libbsd@0.12.2	  grpnckw m4@1.4.19		  3jdzvdm openssh@9.9p1		  ifvv7cz xz@5.4.6
wsmx6sp curl@8.10.1			    ht5niwq libedit@3.1-20240808  ezes42n mpich@4.2.3		  i5fmca4 openssl@3.4.0		  ferxasc yaksa@0.3
ioferp7 diffutils@3.10			    nftqerg libevent@2.1.12	  uajmsrc ncurses@6.5		  tt4vkz3 perl@5.40.0		  i4mwsvm zlib-ng@2.2.1
vlatljj expat@2.6.4			    pvefo63 libfabric@1.22.0	  nn52q3b netlib-lapack@3.11.0	  zgwe2ua pigz@2.8		  yk3lkd4 zstd@1.5.6
wqmzrke findutils@4.9.0			    4ga47g7 libffi@3.4.6	  me353po netlib-scalapack@2.2.0  gutpiwq pkgconf@2.2.0
w5nlv2k gcc-runtime@12.3.0		    gdtmvyu libiconv@1.17	  wpymlkh netlib-scalapack@2.2.0  t7gzekh pmix@5.0.3
==> 90 installed packages
==> 0 concretized packages to be installed (show with `spack find -c`)

Now we can use those definitions to add e.g. serial packages built against the LAPACK libraries. Let’s try to do that by using py-scypy as an example:

Another ability that is often useful, is that of excluding specific entries from a cross-product matrix. We can do that with the exclude keyword, in the same item as the matrix. Let’s try to remove py-scipy ^netlib-lapack from our matrix:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [ openmpi, mpich ]
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
      - "py-scipy ^netlib-lapack"
  view: false
  concretizer:
    unify: false
  compilers:
  - compiler:
      spec: gcc@=12.3.0
      paths:
        cc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gcc
        cxx: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/g++
        f77: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
        fc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
      flags: {}
      operating_system: ubuntu22.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

Let’s concretize the environment and install the specs once again:

$ spack concretize
==> Concretized 1 spec:
 -   g53zrs6  py-scipy@1.14.1%gcc@12.3.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
[+]  w5nlv2k	  ^gcc-runtime@12.3.0%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[e]  gkoh6ax	  ^glibc@2.35%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  aliqr5t	  ^openblas@0.3.28%gcc@12.3.0~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile patches=d0b9276 symbol_suffix=none threads=none arch=linux-ubuntu22.04-x86_64_v3
[+]  zoiouai	      ^gmake@4.4.1%gcc@12.3.0~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  gutpiwq	  ^pkgconf@2.2.0%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   efpxrfo	  ^py-cython@3.0.11%gcc@12.3.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   wbbopip	      ^py-setuptools@69.2.0%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   jgr2tnq	  ^py-meson-python@0.16.0%gcc@12.3.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   tp5miym	      ^meson@1.5.1%gcc@12.3.0 build_system=python_pip patches=0f0b1bd arch=linux-ubuntu22.04-x86_64_v3
 -   i5hagd3	      ^py-packaging@24.1%gcc@12.3.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   ltdymw4		  ^py-flit-core@3.9.0%gcc@12.3.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   p27mstg	      ^py-pyproject-metadata@0.7.1%gcc@12.3.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   ckezggl	  ^py-numpy@2.1.2%gcc@12.3.0 build_system=python_pip patches=873745d arch=linux-ubuntu22.04-x86_64_v3
 -   3eaudob	  ^py-pip@23.1.2%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   bdx5tpn	  ^py-pybind11@2.13.5%gcc@12.3.0+ipo build_system=cmake build_type=Release generator=ninja arch=linux-ubuntu22.04-x86_64_v3
[+]  yempnaz	      ^cmake@3.30.5%gcc@12.3.0~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release patches=dbc3892 arch=linux-ubuntu22.04-x86_64_v3
[+]  wsmx6sp		  ^curl@8.10.1%gcc@12.3.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-ubuntu22.04-x86_64_v3
[+]  ippbbwu		      ^nghttp2@1.63.0%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   myjhsx2	      ^ninja@1.12.1%gcc@12.3.0+re2c build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   qgh7e7n		  ^re2c@3.1%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   3bcgcev	  ^py-pythran@0.16.1%gcc@12.3.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   oyzng5h	      ^py-beniget@0.4.1%gcc@12.3.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   rbt72pq	      ^py-gast@0.5.4%gcc@12.3.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   ohaizhq	      ^py-ply@3.11%gcc@12.3.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   gho4mpq	  ^py-wheel@0.41.2%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   67g7nw4	  ^python@3.11.9%gcc@12.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-ubuntu22.04-x86_64_v3
[+]  j3fd4z5	      ^bzip2@1.0.8%gcc@12.3.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  ioferp7		  ^diffutils@3.10%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  vlatljj	      ^expat@2.6.4%gcc@12.3.0+libbsd build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  3oqufcr		  ^libbsd@0.12.2%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  cpmq5md		      ^libmd@1.0.4%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  ttyuhau	      ^gdbm@1.23%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  jbitwpt	      ^gettext@0.22.5%gcc@12.3.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  gdtmvyu		  ^libiconv@1.17%gcc@12.3.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  4fawd6m		  ^libxml2@2.13.4%gcc@12.3.0+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  j5dljnv		  ^tar@1.34%gcc@12.3.0 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3
[+]  zgwe2ua		      ^pigz@2.8%gcc@12.3.0 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3
[+]  yk3lkd4		      ^zstd@1.5.6%gcc@12.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  4ga47g7	      ^libffi@3.4.6%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  apxktfa	      ^libxcrypt@4.4.35%gcc@12.3.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-ubuntu22.04-x86_64_v3
[+]  tt4vkz3		  ^perl@5.40.0%gcc@12.3.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  4mqvw3u		      ^berkeley-db@18.1.40%gcc@12.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3
[+]  uajmsrc	      ^ncurses@6.5%gcc@12.3.0~symlinks+termlib abi=none build_system=autotools patches=7a351bc arch=linux-ubuntu22.04-x86_64_v3
[+]  i5fmca4	      ^openssl@3.4.0%gcc@12.3.0~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-x86_64_v3
[+]  xilbvuy		  ^ca-certificates-mozilla@2023-05-30%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  tb7zvuf	      ^readline@8.2%gcc@12.3.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu22.04-x86_64_v3
[+]  b63y5mx	      ^sqlite@3.46.0%gcc@12.3.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  64aenca	      ^util-linux-uuid@2.40.2%gcc@12.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  ifvv7cz	      ^xz@5.4.6%gcc@12.3.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+]  i4mwsvm	      ^zlib-ng@2.2.1%gcc@12.3.0+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   pt3p7pd	  ^python-venv@1.0%gcc@12.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3

$ spack install
[+] /usr (external glibc-2.35-a7drdl4tlx4bu3mzhor75pskvd3pdot6)
[+] /usr (external glibc-2.35-gkoh6axllpo5u5oc3rv2vglpxcr22dbx)
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-runtime-11.4.0-hshzy762rns57ibvxiyi3qqvc4nehse2
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gcc-runtime-12.3.0-w5nlv2kp2nkiplculgdgli277r62gdzv
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gmp-6.3.0-5webgyazi3rpemdmmhbip4glviaeet2l
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/zlib-ng-2.2.1-fhud6vqkh7jaesckalgqg5xxml72in6i
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/zstd-1.5.6-t7h6imj35ruac562vr7snzvli2h2tzup
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libiconv-1.17-gdtmvyudykikcpn3q7izewostyxnstys
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/ncurses-6.5-uajmsrcnwwu2a7vc5jooaihbsmywt3sq
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/zstd-1.5.6-yk3lkd4xdugjxxom2e3fuafzg2ppbnn6
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/util-linux-uuid-2.40.2-64aencak6bn7sijcaki5p5bgqgylmxzh
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-lapack-3.11.0-nn52q3blozedtfiwwdeyvint2pbkjybn
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libffi-3.4.6-4ga47g7tj4b6yma6jgigk574sb5iq2rz
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/zlib-ng-2.2.1-i4mwsvmsv2z7ecaoz3iyozp6dykzrqj7
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libxcrypt-4.4.35-apxktfacu5pzihl72vkehndvgzoqrsgq
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gmake-4.4.1-zoiouaigycba7zhhoubeuvjw65uoedaa
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/mpfr-4.2.1-mdfsfjykvyaknc5yzul4l25aip3rugoe
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libpciaccess-0.17-ybxtd2t7kr5453br7gsy66phfkjp7zmf
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/xz-5.4.6-ifvv7czmuxhion5acm5zlyfqkugbuaya
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/bzip2-1.0.8-j3fd4z5i2t5bs5tqtjykksiarh623jkh
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libmd-1.0.4-cpmq5mdiri557vmofyrser77ghcdaykn
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libedit-3.1-20240808-ht5niwqaryv67wtxh2x7lkchhrweybuu
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/pigz-2.8-zgwe2uay3twlpxzxwnp3sw7n5fgwysht
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/numactl-2.0.18-n4nt7lopsr6oxmo42tw2nqfbjwtnkzqm
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libfabric-1.22.0-pvefo63fafyyn7g6p2ccniaikw7nchi4
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openblas-0.3.28-aliqr5t6k2tzder6t7j6rvvabjqagkph
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/yaksa-0.3-ferxascsq35y5jrtdxc4m6w7vq5i5g6o
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openssl-3.4.0-i5fmca4xdydwelmicbx3wanqiiqrf554
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/readline-8.2-tb7zvufjv2zsr4br7chc5uextvcydde6
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/pkgconf-2.2.0-gutpiwqutufsytqyn3neyckwem2qjxgo
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/mpc-1.3.1-qoauga4bfx37v3lynpeoi4rcy2nw2ek5
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libbsd-0.12.2-3oqufcrjxa46foi3gsamsffbcmc6nhhn
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/tar-1.34-j5dljnv4mc5mviozu4ouxfl4r276ftst
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libevent-2.1.12-nftqerg3734tw6jlwe5nsjrg373ublgi
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gdbm-1.23-ttyuhau65qtwjzely7v6tkbnu33mrxeg
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/sqlite-3.46.0-b63y5mxrvtgp35ruiertnxzrnwzoxk44
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/libxml2-2.13.4-4fawd6mhagkgk5lvgp67szz5gi4nhkrl
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/nghttp2-1.63.0-ippbbwuh3aggwhh4dmj4e36flmx3g7ew
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-jjgfqpdmihb2cq5ca7txia5oshruq56q
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/expat-2.6.4-vlatljjv2t62qt7s5paquxtvpvail4tl
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/gettext-0.22.5-jbitwptksuwvgydlf2jhljxiu4ku2z7o
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/hwloc-2.11.1-2fvcggi3gqewbovwkprpcvppiiexsyxh
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/curl-8.10.1-wsmx6spfntot5xmdbbj5nmjwvfyr7ybg
==> Installing python-3.11.9-67g7nw4psfbgmbrswwsfotgmws4gxdm4 [44/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-python-3.11.9-67g7nw4psfbgmbrswwsfotgmws4gxdm4.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/python-3.11.9/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-python-3.11.9-67g7nw4psfbgmbrswwsfotgmws4gxdm4.spack
==> Extracting python-3.11.9-67g7nw4psfbgmbrswwsfotgmws4gxdm4 from binary cache
==> python: Successfully installed python-3.11.9-67g7nw4psfbgmbrswwsfotgmws4gxdm4
  Search: 0.00s.  Fetch: 0.21s.	 Install: 4.02s.  Extract: 3.85s.  Relocate: 0.13s.  Total: 4.23s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/python-3.11.9-67g7nw4psfbgmbrswwsfotgmws4gxdm4
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/krb5-1.21.3-frb66wnvsav46slm5v44war7q7uqszf6
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/pmix-5.0.3-t7gzekhpropf62jjhntf7l2zjbcumqre
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/mpich-4.2.3-ezes42nge7p2vj6tyaqg5y4dhhxzhzij
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/cmake-3.30.5-yempnazpumsvpjmrny2j6nkstc7g67d3
==> Installing re2c-3.1-qgh7e7n6c3vhnwkyxwprh3dmangpejka [49/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-re2c-3.1-qgh7e7n6c3vhnwkyxwprh3dmangpejka.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/re2c-3.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-re2c-3.1-qgh7e7n6c3vhnwkyxwprh3dmangpejka.spack
==> Extracting re2c-3.1-qgh7e7n6c3vhnwkyxwprh3dmangpejka from binary cache
==> re2c: Successfully installed re2c-3.1-qgh7e7n6c3vhnwkyxwprh3dmangpejka
  Search: 0.00s.  Fetch: 0.02s.	 Install: 1.07s.  Extract: 0.98s.  Relocate: 0.05s.  Total: 1.09s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/re2c-3.1-qgh7e7n6c3vhnwkyxwprh3dmangpejka
==> Installing python-venv-1.0-pt3p7pdrinwfcc7wsvr6pdflqcwju4v2 [50/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-python-venv-1.0-pt3p7pdrinwfcc7wsvr6pdflqcwju4v2.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/python-venv-1.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-python-venv-1.0-pt3p7pdrinwfcc7wsvr6pdflqcwju4v2.spack
==> Extracting python-venv-1.0-pt3p7pdrinwfcc7wsvr6pdflqcwju4v2 from binary cache
==> python-venv: Successfully installed python-venv-1.0-pt3p7pdrinwfcc7wsvr6pdflqcwju4v2
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.07s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/python-venv-1.0-pt3p7pdrinwfcc7wsvr6pdflqcwju4v2
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openssh-9.9p1-3jdzvdmyxcyc6k2rh6p3anmik4zrloik
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0-me353pomglevvhbry6mehg7ctzcnneap
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0-bx3g36nfqncvgsc63nfz7ekbzam5c2gf
==> Installing ninja-1.12.1-myjhsx2n7xebdh7kuhtqr32pob3dqcjc [54/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-ninja-1.12.1-myjhsx2n7xebdh7kuhtqr32pob3dqcjc.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/ninja-1.12.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-ninja-1.12.1-myjhsx2n7xebdh7kuhtqr32pob3dqcjc.spack
==> Extracting ninja-1.12.1-myjhsx2n7xebdh7kuhtqr32pob3dqcjc from binary cache
==> ninja: Successfully installed ninja-1.12.1-myjhsx2n7xebdh7kuhtqr32pob3dqcjc
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.17s.  Extract: 0.12s.  Relocate: 0.02s.  Total: 0.18s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/ninja-1.12.1-myjhsx2n7xebdh7kuhtqr32pob3dqcjc
==> Installing py-pip-23.1.2-3eaudob5lksrzhjgnu7rtpfjb37rbxrv [55/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-pip-23.1.2-3eaudob5lksrzhjgnu7rtpfjb37rbxrv.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-pip-23.1.2/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-pip-23.1.2-3eaudob5lksrzhjgnu7rtpfjb37rbxrv.spack
==> Extracting py-pip-23.1.2-3eaudob5lksrzhjgnu7rtpfjb37rbxrv from binary cache
==> py-pip: Successfully installed py-pip-23.1.2-3eaudob5lksrzhjgnu7rtpfjb37rbxrv
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.38s.  Extract: 0.33s.  Relocate: 0.01s.  Total: 0.39s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-pip-23.1.2-3eaudob5lksrzhjgnu7rtpfjb37rbxrv
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/openmpi-5.0.5-suepxfzmhyr4ewe3xorze6ftrvr46fna
==> Installing py-setuptools-69.2.0-wbbopiponk7livpbpxillqtftwp67swr [57/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-setuptools-69.2.0-wbbopiponk7livpbpxillqtftwp67swr.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-setuptools-69.2.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-setuptools-69.2.0-wbbopiponk7livpbpxillqtftwp67swr.spack
==> Extracting py-setuptools-69.2.0-wbbopiponk7livpbpxillqtftwp67swr from binary cache
==> py-setuptools: Successfully installed py-setuptools-69.2.0-wbbopiponk7livpbpxillqtftwp67swr
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.21s.  Extract: 0.16s.  Relocate: 0.01s.  Total: 0.21s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-setuptools-69.2.0-wbbopiponk7livpbpxillqtftwp67swr
==> Installing py-wheel-0.41.2-gho4mpqugaojeue6f56n3j5ls7o3nj6u [58/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-wheel-0.41.2-gho4mpqugaojeue6f56n3j5ls7o3nj6u.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-wheel-0.41.2/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-wheel-0.41.2-gho4mpqugaojeue6f56n3j5ls7o3nj6u.spack
==> Extracting py-wheel-0.41.2-gho4mpqugaojeue6f56n3j5ls7o3nj6u from binary cache
==> py-wheel: Successfully installed py-wheel-0.41.2-gho4mpqugaojeue6f56n3j5ls7o3nj6u
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.03s.  Relocate: 0.01s.  Total: 0.09s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-wheel-0.41.2-gho4mpqugaojeue6f56n3j5ls7o3nj6u
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0-wpymlkhhvzdj3cu3spayvjvzh5kqv4jo
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/netlib-scalapack-2.2.0-5fvq6fgjhaihmeoipzgrpf4tcxc427km
==> Installing py-cython-3.0.11-efpxrfoxiv26s64xj44w2nixjlsao76x [61/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-cython-3.0.11-efpxrfoxiv26s64xj44w2nixjlsao76x.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-cython-3.0.11/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-cython-3.0.11-efpxrfoxiv26s64xj44w2nixjlsao76x.spack
==> Extracting py-cython-3.0.11-efpxrfoxiv26s64xj44w2nixjlsao76x from binary cache
==> py-cython: Successfully installed py-cython-3.0.11-efpxrfoxiv26s64xj44w2nixjlsao76x
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.78s.  Extract: 0.60s.  Relocate: 0.14s.  Total: 0.80s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-cython-3.0.11-efpxrfoxiv26s64xj44w2nixjlsao76x
==> Installing py-pybind11-2.13.5-bdx5tpnvwzacodn2u4mkjzra5pb6swo2 [62/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-pybind11-2.13.5-bdx5tpnvwzacodn2u4mkjzra5pb6swo2.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-pybind11-2.13.5/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-pybind11-2.13.5-bdx5tpnvwzacodn2u4mkjzra5pb6swo2.spack
==> Extracting py-pybind11-2.13.5-bdx5tpnvwzacodn2u4mkjzra5pb6swo2 from binary cache
==> py-pybind11: Successfully installed py-pybind11-2.13.5-bdx5tpnvwzacodn2u4mkjzra5pb6swo2
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.11s.  Extract: 0.06s.  Relocate: 0.01s.  Total: 0.12s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-pybind11-2.13.5-bdx5tpnvwzacodn2u4mkjzra5pb6swo2
==> Installing py-ply-3.11-ohaizhq76arerdlkxwtclv7pz46ggxvr [63/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-ply-3.11-ohaizhq76arerdlkxwtclv7pz46ggxvr.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-ply-3.11/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-ply-3.11-ohaizhq76arerdlkxwtclv7pz46ggxvr.spack
==> Extracting py-ply-3.11-ohaizhq76arerdlkxwtclv7pz46ggxvr from binary cache
==> py-ply: Successfully installed py-ply-3.11-ohaizhq76arerdlkxwtclv7pz46ggxvr
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.03s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-ply-3.11-ohaizhq76arerdlkxwtclv7pz46ggxvr
==> Installing py-flit-core-3.9.0-ltdymw47r2na7g6joaro27ik5cy7sjd7 [64/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-flit-core-3.9.0-ltdymw47r2na7g6joaro27ik5cy7sjd7.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-flit-core-3.9.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-flit-core-3.9.0-ltdymw47r2na7g6joaro27ik5cy7sjd7.spack
==> Extracting py-flit-core-3.9.0-ltdymw47r2na7g6joaro27ik5cy7sjd7 from binary cache
==> py-flit-core: Successfully installed py-flit-core-3.9.0-ltdymw47r2na7g6joaro27ik5cy7sjd7
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.04s.  Relocate: 0.01s.  Total: 0.10s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-flit-core-3.9.0-ltdymw47r2na7g6joaro27ik5cy7sjd7
==> Installing py-gast-0.5.4-rbt72pqun5yd3airefyhyobj6eeguu5p [65/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-gast-0.5.4-rbt72pqun5yd3airefyhyobj6eeguu5p.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-gast-0.5.4/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-gast-0.5.4-rbt72pqun5yd3airefyhyobj6eeguu5p.spack
==> Extracting py-gast-0.5.4-rbt72pqun5yd3airefyhyobj6eeguu5p from binary cache
==> py-gast: Successfully installed py-gast-0.5.4-rbt72pqun5yd3airefyhyobj6eeguu5p
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-gast-0.5.4-rbt72pqun5yd3airefyhyobj6eeguu5p
==> Installing meson-1.5.1-tp5miymsewz5j34mxmsj7estjqnicoy3 [66/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-meson-1.5.1-tp5miymsewz5j34mxmsj7estjqnicoy3.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/meson-1.5.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-meson-1.5.1-tp5miymsewz5j34mxmsj7estjqnicoy3.spack
==> Extracting meson-1.5.1-tp5miymsewz5j34mxmsj7estjqnicoy3 from binary cache
==> meson: Successfully installed meson-1.5.1-tp5miymsewz5j34mxmsj7estjqnicoy3
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.24s.  Extract: 0.18s.  Relocate: 0.02s.  Total: 0.24s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/meson-1.5.1-tp5miymsewz5j34mxmsj7estjqnicoy3
==> Installing py-packaging-24.1-i5hagd3pwth3qya724o644qqfvdbb2ik [67/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-packaging-24.1-i5hagd3pwth3qya724o644qqfvdbb2ik.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-packaging-24.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-packaging-24.1-i5hagd3pwth3qya724o644qqfvdbb2ik.spack
==> Extracting py-packaging-24.1-i5hagd3pwth3qya724o644qqfvdbb2ik from binary cache
==> py-packaging: Successfully installed py-packaging-24.1-i5hagd3pwth3qya724o644qqfvdbb2ik
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.03s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-packaging-24.1-i5hagd3pwth3qya724o644qqfvdbb2ik
==> Installing py-beniget-0.4.1-oyzng5hw6kwhficlcptj5noroheevpao [68/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-beniget-0.4.1-oyzng5hw6kwhficlcptj5noroheevpao.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-beniget-0.4.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-beniget-0.4.1-oyzng5hw6kwhficlcptj5noroheevpao.spack
==> Extracting py-beniget-0.4.1-oyzng5hw6kwhficlcptj5noroheevpao from binary cache
==> py-beniget: Successfully installed py-beniget-0.4.1-oyzng5hw6kwhficlcptj5noroheevpao
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-beniget-0.4.1-oyzng5hw6kwhficlcptj5noroheevpao
==> Installing py-pyproject-metadata-0.7.1-p27mstg5gumacmd3fttlbqedsq7ycvcb [69/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-pyproject-metadata-0.7.1-p27mstg5gumacmd3fttlbqedsq7ycvcb.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-pyproject-metadata-0.7.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-pyproject-metadata-0.7.1-p27mstg5gumacmd3fttlbqedsq7ycvcb.spack
==> Extracting py-pyproject-metadata-0.7.1-p27mstg5gumacmd3fttlbqedsq7ycvcb from binary cache
==> py-pyproject-metadata: Successfully installed py-pyproject-metadata-0.7.1-p27mstg5gumacmd3fttlbqedsq7ycvcb
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-pyproject-metadata-0.7.1-p27mstg5gumacmd3fttlbqedsq7ycvcb
==> Installing py-meson-python-0.16.0-jgr2tnqohn4px4v5qy2mfupy6evzrlel [70/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-meson-python-0.16.0-jgr2tnqohn4px4v5qy2mfupy6evzrlel.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-meson-python-0.16.0/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-meson-python-0.16.0-jgr2tnqohn4px4v5qy2mfupy6evzrlel.spack
==> Extracting py-meson-python-0.16.0-jgr2tnqohn4px4v5qy2mfupy6evzrlel from binary cache
==> py-meson-python: Successfully installed py-meson-python-0.16.0-jgr2tnqohn4px4v5qy2mfupy6evzrlel
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.03s.  Relocate: 0.02s.  Total: 0.09s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-meson-python-0.16.0-jgr2tnqohn4px4v5qy2mfupy6evzrlel
==> Installing py-numpy-2.1.2-ckezggl7xcir66fx4sh65la532l3y6gq [71/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-numpy-2.1.2-ckezggl7xcir66fx4sh65la532l3y6gq.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-numpy-2.1.2/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-numpy-2.1.2-ckezggl7xcir66fx4sh65la532l3y6gq.spack
==> Extracting py-numpy-2.1.2-ckezggl7xcir66fx4sh65la532l3y6gq from binary cache
==> py-numpy: Successfully installed py-numpy-2.1.2-ckezggl7xcir66fx4sh65la532l3y6gq
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.86s.  Extract: 0.69s.  Relocate: 0.13s.  Total: 0.88s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-numpy-2.1.2-ckezggl7xcir66fx4sh65la532l3y6gq
==> Installing py-pythran-0.16.1-3bcgcevke7k5b5udyilwogzq2af4yhda [72/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-pythran-0.16.1-3bcgcevke7k5b5udyilwogzq2af4yhda.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-pythran-0.16.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-pythran-0.16.1-3bcgcevke7k5b5udyilwogzq2af4yhda.spack
==> Extracting py-pythran-0.16.1-3bcgcevke7k5b5udyilwogzq2af4yhda from binary cache
==> py-pythran: Successfully installed py-pythran-0.16.1-3bcgcevke7k5b5udyilwogzq2af4yhda
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.78s.  Extract: 0.71s.  Relocate: 0.02s.  Total: 0.79s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-pythran-0.16.1-3bcgcevke7k5b5udyilwogzq2af4yhda
==> Installing py-scipy-1.14.1-g53zrs6vomtrzwxheb7r5tmiij6u7a2j [73/73]
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-scipy-1.14.1-g53zrs6vomtrzwxheb7r5tmiij6u7a2j.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-scipy-1.14.1/linux-ubuntu22.04-x86_64_v3-gcc-12.3.0-py-scipy-1.14.1-g53zrs6vomtrzwxheb7r5tmiij6u7a2j.spack
==> Extracting py-scipy-1.14.1-g53zrs6vomtrzwxheb7r5tmiij6u7a2j from binary cache
==> py-scipy: Successfully installed py-scipy-1.14.1-g53zrs6vomtrzwxheb7r5tmiij6u7a2j
  Search: 0.00s.  Fetch: 0.03s.	 Install: 1.72s.  Extract: 1.53s.  Relocate: 0.15s.  Total: 1.75s
[+] /home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-12.3.0/py-scipy-1.14.1-g53zrs6vomtrzwxheb7r5tmiij6u7a2j

At this point the environment contains only py-scipy ^openblas. Let’s verify it:

$ spack find -ld py-scipy
==> In environment /home/spack5/stacks
==> 6 root specs
-- no arch / gcc@11 ---------------------------------------------
[+] jjgfqpd gcc@12%gcc@11


-- no arch / gcc@12 ---------------------------------------------
[+] bx3g36n netlib-scalapack%gcc@12
     -	------- mpich
     -	------- netlib-lapack

[+] me353po netlib-scalapack%gcc@12
     -	------- mpich
     -	------- openblas

[+] 5fvq6fg netlib-scalapack%gcc@12
     -	------- netlib-lapack
     -	------- openmpi

[+] wpymlkh netlib-scalapack%gcc@12
     -	------- openblas
     -	------- openmpi

[+] g53zrs6 py-scipy%gcc@12
     -	------- openblas


-- linux-ubuntu22.04-x86_64_v3 / gcc@12.3.0 ---------------------
g53zrs6 py-scipy@1.14.1
w5nlv2k	    gcc-runtime@12.3.0
gkoh6ax	    glibc@2.35
aliqr5t	    openblas@0.3.28
zoiouai		gmake@4.4.1
gutpiwq	    pkgconf@2.2.0
efpxrfo	    py-cython@3.0.11
wbbopip		py-setuptools@69.2.0
jgr2tnq	    py-meson-python@0.16.0
tp5miym		meson@1.5.1
i5hagd3		py-packaging@24.1
ltdymw4		    py-flit-core@3.9.0
p27mstg		py-pyproject-metadata@0.7.1
ckezggl	    py-numpy@2.1.2
3eaudob	    py-pip@23.1.2
bdx5tpn	    py-pybind11@2.13.5
yempnaz		cmake@3.30.5
wsmx6sp		    curl@8.10.1
ippbbwu			nghttp2@1.63.0
myjhsx2		ninja@1.12.1
qgh7e7n		    re2c@3.1
3bcgcev	    py-pythran@0.16.1
oyzng5h		py-beniget@0.4.1
rbt72pq		py-gast@0.5.4
ohaizhq		py-ply@3.11
gho4mpq	    py-wheel@0.41.2
67g7nw4	    python@3.11.9
j3fd4z5		bzip2@1.0.8
ioferp7		    diffutils@3.10
vlatljj		expat@2.6.4
3oqufcr		    libbsd@0.12.2
cpmq5md			libmd@1.0.4
ttyuhau		gdbm@1.23
jbitwpt		gettext@0.22.5
gdtmvyu		    libiconv@1.17
4fawd6m		    libxml2@2.13.4
j5dljnv		    tar@1.34
zgwe2ua			pigz@2.8
yk3lkd4			zstd@1.5.6
4ga47g7		libffi@3.4.6
apxktfa		libxcrypt@4.4.35
tt4vkz3		    perl@5.40.0
4mqvw3u			berkeley-db@18.1.40
uajmsrc		ncurses@6.5
i5fmca4		openssl@3.4.0
xilbvuy		    ca-certificates-mozilla@2023-05-30
tb7zvuf		readline@8.2
b63y5mx		sqlite@3.46.0
64aenca		util-linux-uuid@2.40.2
ifvv7cz		xz@5.4.6
i4mwsvm		zlib-ng@2.2.1
pt3p7pd	    python-venv@1.0

==> 1 installed package
==> 0 concretized packages to be installed (show with `spack find -c`)

Conditional definitions

Spec list definitions can also be conditioned on a when clause. The when clause is a python conditional that is evaluated in a restricted environment. The variables available in when clauses are:

variable name

value

platform

The spack platform name for this machine

os

The default spack os name and version string for this machine

target

The default spack target string for this machine

architecture

The default spack architecture string platform-os-target for this machine

arch

Alias for architecture

env

A dictionary representing the users environment variables

re

The python re module for regex

hostname

The hostname of this node

Let’s say we only want to limit to just use mpich, unless the SPACK_STACK_USE_OPENMPI environment variable is set. To do so we could write the following spack.yaml:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [mpich]
    - mpis: [openmpi]
      when: 'env.get("SPACK_STACK_USE_OPENMPI", "") == "1"'
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
    - "py-scipy ^netlib-lapack"
  view: false
  concretizer:
    unify: false
  compilers:
  - compiler:
      spec: gcc@=12.3.0
      paths:
        cc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gcc
        cxx: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/g++
        f77: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
        fc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
      flags: {}
      operating_system: ubuntu22.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

Different definitions of lists with the same name are concatenated, so we can define our MPI list in one place unconditionally, and then conditionally append one or more values to it.

Let’s first check what happens when we concretize and don’t set any environment variable:

$ spack concretize
==> No new specs to concretize.
$ spack find -cl netlib-scalapack
==> In environment /home/spack5/stacks
==> 4 root specs
-- no arch / gcc@11 ---------------------------------------------
[+] jjgfqpd gcc@12%gcc@11

-- no arch / gcc@12 ---------------------------------------------
[+] bx3g36n netlib-scalapack%gcc@12  [+] me353po netlib-scalapack%gcc@12  [+] g53zrs6 py-scipy%gcc@12

-- linux-ubuntu22.04-x86_64_v3 / gcc@12.3.0 ---------------------
[+]  me353po netlib-scalapack@2.2.0  [+]  bx3g36n netlib-scalapack@2.2.0
==> 2 installed packages
==> 0 concretized packages to be installed

As we expected now we are only using mpich as an MPI provider. To get openmpi back we just need to set the appropriate environment variable:

$ export SPACK_STACK_USE_OPENMPI=1
$ spack concretize
$ spack find -cl netlib-scalapack
==> In environment /home/spack5/stacks
==> 6 root specs
-- no arch / gcc@11 ---------------------------------------------
[+] jjgfqpd gcc@12%gcc@11

-- no arch / gcc@12 ---------------------------------------------
[+] bx3g36n netlib-scalapack%gcc@12  [+] me353po netlib-scalapack%gcc@12  [+] 5fvq6fg netlib-scalapack%gcc@12  [+] wpymlkh netlib-scalapack%gcc@12  [+] g53zrs6 py-scipy%gcc@12

-- linux-ubuntu22.04-x86_64_v3 / gcc@12.3.0 ---------------------
[+]  me353po netlib-scalapack@2.2.0  [+]  wpymlkh netlib-scalapack@2.2.0  [+]  5fvq6fg netlib-scalapack@2.2.0  [+]  bx3g36n netlib-scalapack@2.2.0
==> 4 installed packages
==> 0 concretized packages to be installed

There is no need to install this time, since all the specs were still in the store.

Other useful features

Sometimes it might be useful to create a local source mirror for the specs installed in an environment. If the environment is active, this is as simple as:

$ spack mirror create --all -d ./stacks-mirror

This command fetches all the tarballs for the packages in the spack.lock file, and puts them in the directory passed as argument. Later you can move this mirror to e.g. an air-gapped machine and:

$ spack mirror add <name> <stacks-mirror>

to be able to re-build the specs from sources. If instead you want to create a buildcache you can:

$ spack gpg create <name> <e-mail>
$ spack buildcache push ./mirror

In that case, don’t forget to set an appropriate value for the padding of the install tree, see how to setup relocation in our documentation.

By default, Spack installs one package at a time, using the -j option where it can. If you are installing a large environment, and have at disposal a beefy build node, you might need to start more installations in parallel to make an optimal use of the resources. This can be done by creating a depfile, when the environment is active:

$ spack env depfile -o Makefile

The result is a makefile that starts multiple Spack instances, and the resources are shared through the GNU jobserver. More information of this feature can be found in our documentation. This might cut down your build time by a fair amount, if you build frequently from sources.

Make the software stack easy to use

Now that the software stack has been installed, we need to focus on how it can be used by our customers. We’ll first see how we can configure views to project a subset of the specs we installed onto a filesystem folder with the usual Unix structure. Then we’ll have a similar discussion for module files. Which of the two approaches is better depends strongly on the use case at hand.

View descriptors

At the beginning, we configured Spack not to create a view for this stack because simple views won’t work with stacks. We’ve been concretizing multiple packages of the same name, and they would conflict if linked into the same view.

What we can do is create multiple views, using view descriptors. This would allows us to define which packages are linked into the view, and how. Let’s edit our spack.yaml file again.

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [mpich]
    - mpis: [openmpi]
      when: 'env.get("SPACK_STACK_USE_OPENMPI", "") == "1"'
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
    - "py-scipy ^netlib-lapack"
  concretizer:
    unify: false
  compilers:
  - compiler:
      spec: gcc@=12.3.0
      paths:
        cc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gcc
        cxx: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/g++
        f77: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
        fc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
      flags: {}
      operating_system: ubuntu22.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []
  view:
    default:
      root: views/default
      select: ['%gcc@12']
      exclude: [^mpich, ^netlib-lapack]
    full:
      root: views/full
      projections:
        ^mpi^lapack: '{compiler.name}-{compiler.version}/{^mpi.name}-{^mpi.version}-{^lapack.name}-{^lapack.version}'
        ^lapack: '{compiler.name}-{compiler.version}/{^lapack.name}-{^lapack.version}'
        all: '{compiler.name}-{compiler.version}/'

In the configuration above we created two views, named default and full. The default view consists of all the packages that are compiled with gcc@12, but do not depend on either mpich or netlib-lapack. As we can see, we can both include and exclude specs using constrains.

The full view contains a more complex projection, so to put each spec into an appropriate subdirectory, according to the first constraint that the spec matches. all is the default projection, and has always the lowest priority, independent of the order in which it appears. To avoid confusion, we advise to always keep it last in projections.

Let’s concretize to regenerate the views, and check their structure:

$ spack concretize
==> No new specs to concretize.
==> Updating view at /home/spack5/stacks/views/default
==> Updating view at /home/spack5/stacks/views/full
$ ls ~/stacks/views/default
bin  etc  include  lib	lib64  libexec	man  pyvenv.cfg	 sbin  share  var
$ ls ~/stacks/views/default/lib
cmake			    libevent_core.a		    libgettextlib.a	      libkdb5.so.10.0			   libmpi_usempif08.so		libopenblas.so.0	libtinfow.so.6
gettext			    libevent_core.so		    libgettextlib.so	      libkrad.so			   libmpi_usempif08.so.40	libpanel.a		libtinfow.so.6.5
krb5			    libevent_extra-2.1.so.7	    libgettextpo.a	      libkrad.so.0			   libmpi_usempif08.so.40.40.2	libpanel.so		libtinfow_g.a
libasan.so.8		    libevent_extra-2.1.so.7.0.1	    libgettextpo.so	      libkrad.so.0.0			   libmpich.so			libpanel.so.6		libtmglib.so
libasprintf.a		    libevent_extra.a		    libgettextpo.so.0	      libkrb5.so			   libmpichcxx.so		libpanel.so.6.5		libtmglib.so.3
libasprintf.so		    libevent_extra.so		    libgettextpo.so.0.5.10    libkrb5.so.3			   libmpichf90.so		libpanel_g.a		libtmglib.so.3.11.0
libasprintf.so.0	    libevent_openssl-2.1.so.7	    libgettextsrc-0.22.5.so   libkrb5.so.3.3			   libmpicxx.a			libpanelw.a		libtsan.so.2
libasprintf.so.0.0.0	    libevent_openssl-2.1.so.7.0.1   libgettextsrc.a	      libkrb5support.so			   libmpicxx.so			libpanelw.so		libubsan.so.1
libatomic.so.1		    libevent_openssl.a		    libgettextsrc.so	      libkrb5support.so.0		   libmpicxx.so.12		libpanelw.so.6		libuuid.a
libblas.so		    libevent_openssl.so		    libgfortran.so.5	      libkrb5support.so.0.1		   libmpicxx.so.12.4.3		libpanelw.so.6.5	libuuid.so
libblas.so.3		    libevent_pthreads-2.1.so.7	    libgomp.so.1	      liblapack.so			   libmpifort.a			libpanelw_g.a		libuuid.so.1
libblas.so.3.11.0	    libevent_pthreads-2.1.so.7.0.1  libgssapi_krb5.so	      liblapack.so.3			   libmpifort.so		libpciaccess.a		libuuid.so.1.3.0
libbsd-ctor.a		    libevent_pthreads.a		    libgssapi_krb5.so.2	      liblapack.so.3.11.0		   libmpifort.so.12		libpciaccess.so		libverto.so
libbsd.a		    libevent_pthreads.so	    libgssapi_krb5.so.2.2     liblapacke.so			   libmpifort.so.12.4.3		libpciaccess.so.0	libverto.so.0
libbsd.so		    libexpat.a			    libgssrpc.so	      liblapacke.so.3			   libmpl.so			libpciaccess.so.0.11.1	libverto.so.0.0
libbsd.so.0		    libexpat.so			    libgssrpc.so.4	      liblapacke.so.3.11.0		   libncurses++.a		libpmix.a		libxml2.so
libbsd.so.0.12.2	    libexpat.so.1		    libgssrpc.so.4.2	      liblsan.so.0			   libncurses++.so		libpmix.so		libxml2.so.2
libbz2.a		    libexpat.so.1.10.0		    libhistory.a	      liblzma.a				   libncurses++.so.6		libpmix.so.2		libxml2.so.2.13.4
libbz2.so		    libfabric.a			    libhistory.so	      liblzma.so			   libncurses++.so.6.5		libpmix.so.2.13.3	libyaksa.a
libbz2.so.1		    libfabric.so		    libhistory.so.8	      liblzma.so.5			   libncurses++_g.a		libprrte.so		libyaksa.so
libbz2.so.1.0		    libfabric.so.1		    libhistory.so.8.2	      liblzma.so.5.4.6			   libncurses++w.a		libprrte.so.3		libyaksa.so.0
libbz2.so.1.0.8		    libfabric.so.1.25.0		    libhwloc.a		      libmd.a				   libncurses++w.so		libprrte.so.3.0.6	libyaksa.so.0.0.0
libcblas.so		    libffi.a			    libhwloc.so		      libmd.so				   libncurses++w.so.6		libpython3.11.so	libz.a
libcblas.so.3		    libffi.so			    libhwloc.so.15	      libmd.so.0			   libncurses++w.so.6.5		libpython3.11.so.1.0	libz.so
libcblas.so.3.11.0	    libffi.so.8			    libhwloc.so.15.8.0	      libmd.so.0.0.5			   libncurses++w_g.a		libpython3.so		libz.so.1
libcharset.a		    libffi.so.8.1.4		    libiconv.a		      libmenu.a				   libncurses.a			libquadmath.so.0	libz.so.1.3.1.zlib-ng
libcharset.so		    libfmpich.so		    libiconv.so		      libmenu.so			   libncurses.so		libreadline.a		libzstd.a
libcharset.so.1		    libform.a			    libiconv.so.2	      libmenu.so.6			   libncurses.so.6		libreadline.so		libzstd.so
libcharset.so.1.0.0	    libform.so			    libiconv.so.2.6.1	      libmenu.so.6.5			   libncurses.so.6.5		libreadline.so.8	libzstd.so.1
libcom_err.so		    libform.so.6		    libintl.a		      libmenu_g.a			   libncurses_g.a		libreadline.so.8.2	libzstd.so.1.5.6
libcom_err.so.3		    libform.so.6.5		    libintl.so		      libmenuw.a			   libncursesw.a		libscalapack.so		mpi.mod
libcom_err.so.3.0	    libform_g.a			    libintl.so.8	      libmenuw.so			   libncursesw.so		libsqlite3.a		mpi_ext.mod
libcrypt.a		    libformw.a			    libintl.so.8.4.0	      libmenuw.so.6			   libncursesw.so.6		libsqlite3.so		mpi_f08.mod
libcrypt.so		    libformw.so			    libitm.so.1		      libmenuw.so.6.5			   libncursesw.so.6.5		libsqlite3.so.0		mpi_f08_callbacks.mod
libcrypt.so.2		    libformw.so.6		    libk5crypto.so	      libmenuw_g.a			   libncursesw_g.a		libsqlite3.so.0.8.6	mpi_f08_ext.mod
libcrypt.so.2.0.0	    libformw.so.6.5		    libk5crypto.so.3	      libmpi.a				   libnuma.a			libstdc++.so.6		mpi_f08_interfaces.mod
libcurses.so		    libformw_g.a		    libk5crypto.so.3.1	      libmpi.so				   libnuma.so			libtextstyle.a		mpi_f08_interfaces_callbacks.mod
libedit.a		    libgcc_s.so.1		    libkadm5clnt.so	      libmpi.so.12			   libnuma.so.1			libtextstyle.so		mpi_f08_types.mod
libedit.so		    libgdbm.a			    libkadm5clnt_mit.so	      libmpi.so.12.4.3			   libnuma.so.1.0.0		libtextstyle.so.0	mpi_types.mod
libedit.so.0		    libgdbm.so			    libkadm5clnt_mit.so.12    libmpi.so.40			   libopa.so			libtextstyle.so.0.2.1	openmpi
libedit.so.0.0.74	    libgdbm.so.6		    libkadm5clnt_mit.so.12.0  libmpi.so.40.40.5			   libopen-pal.so		libtinfo.a		pkgconfig
libevent-2.1.so.7	    libgdbm.so.6.0.0		    libkadm5srv.so	      libmpi_mpifh.so			   libopen-pal.so.80		libtinfo.so		pmpi_f08_interfaces.mod
libevent-2.1.so.7.0.1	    libgdbm_compat.a		    libkadm5srv_mit.so	      libmpi_mpifh.so.40		   libopen-pal.so.80.0.4	libtinfo.so.6		python3.11
libevent.a		    libgdbm_compat.so		    libkadm5srv_mit.so.12     libmpi_mpifh.so.40.40.0		   libopenblas-r0.3.28.a	libtinfo.so.6.5		terminfo
libevent.so		    libgdbm_compat.so.4		    libkadm5srv_mit.so.12.0   libmpi_usempi_ignore_tkr.so	   libopenblas-r0.3.28.so	libtinfo_g.a
libevent_core-2.1.so.7	    libgdbm_compat.so.4.0.0	    libkdb5.so		      libmpi_usempi_ignore_tkr.so.40	   libopenblas.a		libtinfow.a
libevent_core-2.1.so.7.0.1  libgettextlib-0.22.5.so	    libkdb5.so.10	      libmpi_usempi_ignore_tkr.so.40.40.0  libopenblas.so		libtinfow.so
$ ls ~/stacks/views/full
gcc-11.4.0  gcc-12.3.0
$ ls ~/stacks/views/full/gcc-12.3.0
bin  etc  include  lib	lib64  libexec	man  mpich-4.2.3-netlib-lapack-3.11.0  mpich-4.2.3-openblas-0.3.28  openmpi-5.0.5-netlib-lapack-3.11.0	openmpi-5.0.5-openblas-0.3.28  pyvenv.cfg  sbin	 share	var

The view descriptor also contains a link key. The default behavior, as we have seen, is to link all packages, including implicit link and run dependencies, into the view. If we set the option to “roots”, Spack links only the root packages into the view.

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [mpich]
    - mpis: [openmpi]
      when: 'env.get("SPACK_STACK_USE_OPENMPI", "") == "1"'
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
    - "py-scipy ^netlib-lapack"
  concretizer:
    unify: false
  compilers:
  - compiler:
      spec: gcc@=12.3.0
      paths:
        cc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gcc
        cxx: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/g++
        f77: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
        fc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
      flags: {}
      operating_system: ubuntu22.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []
  view:
    default:
      root: views/default
      select: ['%gcc@12']
      exclude: [^mpich, ^netlib-lapack]
      link: roots
    full:
      root: views/full
      projections:
        ^mpi^lapack: '{compiler.name}-{compiler.version}/{^mpi.name}-{^mpi.version}-{^lapack.name}-{^lapack.version}'
        ^lapack: '{compiler.name}-{compiler.version}/{^lapack.name}-{^lapack.version}'
        all: '{compiler.name}-{compiler.version}/'
$ spack concretize
==> No new specs to concretize.
==> Updating view at /home/spack5/stacks/views/default
$ ls ~/stacks/views/default
lib
$ ls ~/stacks/views/default/lib
cmake  libscalapack.so	pkgconfig  python3.11
$ ls ~/stacks/views/full
gcc-11.4.0  gcc-12.3.0

Now we see only the root libraries in the default view. The rest are hidden, but are still available in the full view. The complete documentation on view can be found here.

Module files

Module files are another very popular way to use software on HPC systems. In this section we’ll show how to configure and generate a hierarchical module structure, suitable for lmod.

A more in-depth tutorial, focused only on module files, can be found at Module Files Tutorial. There we discuss the general architecture of module file generation in Spack and we highlight differences between environment-modules and lmod that won’t be covered in this section.

So, let’s start by adding lmod to the software installed with the system compiler:

$ spack add lmod%gcc@11
$ spack concretize
$ spack install

Once that is done, let’s add the module command to our shell like this:

$ . $(spack location -i lmod)/lmod/lmod/init/bash

If everything worked out correctly you should now have the module command available in you shell:

$ module --version

Modules based on Lua: Version 8.7.18  2023-01-14 07:33 -06:00
    by Robert McLay mclay@tacc.utexas.edu

The next step is to add some basic configuration to our spack.yaml to generate module files:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [mpich]
    - mpis: [openmpi]
      when: 'env.get("SPACK_STACK_USE_OPENMPI", "") == "1"'
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
    - "py-scipy ^netlib-lapack"
  view: false
  concretizer:
    unify: false
  compilers:
  - compiler:
      spec: gcc@=12.3.0
      paths:
        cc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gcc
        cxx: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/g++
        f77: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
        fc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
      flags: {}
      operating_system: ubuntu22.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []
  modules:
    default:
      enable:
      - lmod
      roots:
        lmod: modules
      lmod:
        hierarchy:
        - mpi
        - lapack

In these few lines of additional configuration we told Spack to generate lmod module files in a subdirectory named modules, using a hierarchy comprising both lapack and mpi.

We can generate the module files and use them with the following commands:

$ spack module lmod refresh -y
$ module use $PWD/stacks/modules/linux-ubuntu22.04-x86_64/Core

Now we should be able to see the module files that have been generated:

$ module av

---------------------------------------------------------------- /home/spack5/stacks/modules/linux-ubuntu22.04-x86_64/Core ----------------------------------------------------------------
   autoconf-archive/2023.02.20-6s34ic3		 ed/1.4-p72c4zc		       gmp/6.3.0-5webgya		  lua/5.4.6-qcxkktp	    pkgconf/2.2.0-oplsmxr
   autoconf/2.72-vgucajy			 findutils/4.9.0-2sbkhch       libiconv/1.17-bsga3no		  m4/1.4.19-qaspjyq	    readline/8.2-aylebxv
   automake/1.16.5-awgfaon			 gawk/5.3.1-x6mwj3l	       libsigsegv/2.14-ea6qziv		  mpc/1.3.1-qoauga4	    tar/1.34-mmv6i4n
   bc/1.07.1-jasonju				 gcc-runtime/11.4.0-hshzy76    libtool/2.4.7-tt3byem		  mpfr/4.2.1-mdfsfjy	    tcl/8.6.12-w4gwr4e
   berkeley-db/18.1.40-cexlioh			 gcc/12.3.0-jjgfqpd	       libxcrypt/4.4.35-v64tnmr		  ncurses/6.5-rvg7j6b	    texinfo/7.1-ljsbwjc
   bzip2/1.0.8-ewqc7cx				 gdbm/1.23-jj2atvs	       libxml2/2.13.4-wozcmyn		  nghttp2/1.63.0-t2qkug7    unzip/6.0-akxwy4w
   ca-certificates-mozilla/2023-05-30-t2brqss	 gettext/0.22.5-bexdfw2	       lmod/8.7.18-saucmo6		  openssl/3.4.0-5gigqgw	    xz/5.4.6-2fvrfr6
   curl/8.10.1-fpywomo				 glibc/2.35-a7drdl4	       lua-luafilesystem/1.8.0-4lpeywj	  perl/5.40.0-cwpt5ec	    zlib-ng/2.2.1-fhud6vq
   diffutils/3.10-lljulvx			 gmake/4.4.1-srkzfjr	       lua-luaposix/36.1-hdnpce3	  pigz/2.8-pcwnu2w	    zstd/1.5.6-t7h6imj

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

The sets of modules is already usable, and the hierarchy already works. For instance we can load the gcc compiler and check that we have gcc in out path and we have a lot more modules available - all the ones compiled with gcc@12.3.0:

$ module load gcc
$ which gcc
/home/spack5/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-jjgfqpdmihb2cq5ca7txia5oshruq56q/bin/gcc
$ gcc --version
gcc (Spack GCC) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ module av

------------------------------------------------------------- /home/spack5/stacks/modules/linux-ubuntu22.04-x86_64/gcc/12.3.0 -------------------------------------------------------------
   autoconf/2.72-yfxrc66		      (D)    libbsd/0.12.2-3oqufcr		 ninja/1.12.1-myjhsx2		       py-pybind11/2.13.5-bdx5tpn
   automake/1.16.5-e4u6ql7		      (D)    libedit/3.1-20240808-ht5niwq	 numactl/2.0.18-n4nt7lo		       py-pyproject-metadata/0.7.1-p27mstg
   berkeley-db/18.1.40-4mqvw3u		      (D)    libevent/2.1.12-nftqerg		 openblas/0.3.28-aliqr5t	       py-setuptools/69.2.0-wbbopip
   bison/3.8.2-fky4vih				     libfabric/1.22.0-pvefo63		 openmpi/5.0.5-suepxfz		       py-wheel/0.41.2-gho4mpq
   bzip2/1.0.8-j3fd4z5			      (D)    libffi/3.4.6-4ga47g7		 openssh/9.9p1-3jdzvdm		       python-venv/1.0-pt3p7pd
   ca-certificates-mozilla/2023-05-30-xilbvuy (D)    libiconv/1.17-gdtmvyu	  (D)	 openssl/3.4.0-i5fmca4		       python/3.11.9-67g7nw4
   cmake/3.30.5-yempnaz				     libmd/1.0.4-cpmq5md		 perl/5.40.0-tt4vkz3		(D)    python/3.13.0-wxt7aps		   (D)
   curl/8.10.1-wsmx6sp			      (D)    libpciaccess/0.17-ybxtd2t		 pigz/2.8-zgwe2ua		(D)    re2c/3.1-qgh7e7n
   diffutils/3.10-ioferp7			     libsigsegv/2.14-u2gofvq	  (D)	 pkgconf/2.2.0-gutpiwq		       readline/8.2-tb7zvuf		   (D)
   expat/2.6.4-vlatljj				     libtool/2.4.7-xfe5xho	  (D)	 pmix/5.0.3-t7gzekh		       sqlite/3.46.0-b63y5mx
   findutils/4.9.0-wqmzrke			     libxcrypt/4.4.35-apxktfa		 py-beniget/0.4.1-oyzng5h	       tar/1.34-j5dljnv
   gcc-runtime/12.3.0-w5nlv2k		      (D)    libxml2/2.13.4-4fawd6m	  (D)	 py-cython/3.0.11-efpxrfo	       util-linux-uuid/2.40.2-64aenca
   gdbm/1.23-ttyuhau			      (D)    m4/1.4.19-grpnckw			 py-flit-core/3.9.0-ltdymw4	       util-macros/1.20.1-vau55h7
   gettext/0.22.5-jbitwpt		      (D)    meson/1.5.1-tp5miym		 py-gast/0.5.4-rbt72pq		       xz/5.4.6-ifvv7cz
   glibc/2.35-gkoh6ax			      (D)    mpich/4.2.3-ezes42n		 py-meson-python/0.16.0-jgr2tnq	       yaksa/0.3-ferxasc
   gmake/4.4.1-zoiouai			      (D)    ncurses/6.5-uajmsrc	  (D)	 py-packaging/24.1-i5hagd3	       zlib-ng/2.2.1-i4mwsvm		   (D)
   hwloc/2.11.1-2fvcggi				     netlib-lapack/3.11.0-nn52q3b	 py-pip/23.1.2-3eaudob		       zstd/1.5.6-yk3lkd4		   (D)
   krb5/1.21.3-frb66wn				     nghttp2/1.63.0-ippbbwu		 py-ply/3.11-ohaizhq

---------------------------------------------------------------- /home/spack5/stacks/modules/linux-ubuntu22.04-x86_64/Core ----------------------------------------------------------------
   autoconf-archive/2023.02.20-6s34ic3		     ed/1.4-p72c4zc		       gmp/6.3.0-5webgya	       (L)    lua/5.4.6-qcxkktp		    pkgconf/2.2.0-oplsmxr (D)
   autoconf/2.72-vgucajy			     findutils/4.9.0-2sbkhch	(D)    libiconv/1.17-bsga3no		      m4/1.4.19-qaspjyq	     (D)    readline/8.2-aylebxv
   automake/1.16.5-awgfaon			     gawk/5.3.1-x6mwj3l		       libsigsegv/2.14-ea6qziv		      mpc/1.3.1-qoauga4	     (L)    tar/1.34-mmv6i4n	  (D)
   bc/1.07.1-jasonju				     gcc-runtime/11.4.0-hshzy76 (L)    libtool/2.4.7-tt3byem		      mpfr/4.2.1-mdfsfjy     (L)    tcl/8.6.12-w4gwr4e
   berkeley-db/18.1.40-cexlioh			     gcc/12.3.0-jjgfqpd		(L)    libxcrypt/4.4.35-v64tnmr	       (D)    ncurses/6.5-rvg7j6b	    texinfo/7.1-ljsbwjc
   bzip2/1.0.8-ewqc7cx				     gdbm/1.23-jj2atvs		       libxml2/2.13.4-wozcmyn		      nghttp2/1.63.0-t2qkug7 (D)    unzip/6.0-akxwy4w
   ca-certificates-mozilla/2023-05-30-t2brqss	     gettext/0.22.5-bexdfw2	       lmod/8.7.18-saucmo6		      openssl/3.4.0-5gigqgw  (D)    xz/5.4.6-2fvrfr6	  (D)
   curl/8.10.1-fpywomo				     glibc/2.35-a7drdl4		(L)    lua-luafilesystem/1.8.0-4lpeywj	      perl/5.40.0-cwpt5ec	    zlib-ng/2.2.1-fhud6vq (L)
   diffutils/3.10-lljulvx		      (D)    gmake/4.4.1-srkzfjr	       lua-luaposix/36.1-hdnpce3	      pigz/2.8-pcwnu2w		    zstd/1.5.6-t7h6imj	  (L)

  Where:
   D:  Default Module
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


$ module unload gcc

There are a few issues though. For once, we have a lot of modules generated from dependencies of gcc that are cluttering the view, and won’t likely be needed directly by users. Then, module names contain hashes, which go against users being able to reuse the same script in similar, but not equal, environments.

Also, some of the modules might need to set custom environment variables, which are specific to the deployment aspects that don’t enter the hash - for instance a policy at the deploying site.

To address all these needs we can complicate out modules configuration a bit more:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [mpich]
    - mpis: [openmpi]
      when: 'env.get("SPACK_STACK_USE_OPENMPI", "") == "1"'
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
    - "py-scipy ^netlib-lapack"
  view: false
  concretizer:
    unify: false
  compilers:
  - compiler:
      spec: gcc@=12.3.0
      paths:
        cc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gcc
        cxx: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/g++
        f77: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
        fc: /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/gcc-12.3.0-hc3fmfiuomxfx4gnvorj3z3y2gjf2doh/bin/gfortran
      flags: {}
      operating_system: ubuntu22.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []
  modules:
    default:
      enable:
      - lmod
      roots:
        lmod: modules
      lmod:
        hierarchy:
        - mpi
        - lapack
        hash_length: 0
        include:
        - gcc
        exclude:
        - '%gcc@11.4.0'
        all:
          environment:
            set:
              '{name}_ROOT': '{prefix}'
        openmpi:
          environment:
            set:
              SLURM_MPI_TYPE: pmi2
              OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
        projections:
          all: '{name}/{version}'

Let’s regenerate the modules once again:

$ spack module lmod refresh --delete-tree -y
==> Regenerating lmod module files

Now we have a set of module files without hashes, with a correct hierarchy, and with all our custom modifications:

$ module load gcc
$ module load openmpi openblas netlib-scalapack py-scipy
$ module av

------------------------------------------------------------- /home/spack5/stacks/modules/linux-ubuntu22.04-x86_64/gcc/12.3.0 -------------------------------------------------------------
   autoconf/2.72			 gdbm/1.23		 libmd/1.0.4		 ninja/1.12.1	     py-flit-core/3.9.0		    python/3.13.0	   (D)
   automake/1.16.5			 gettext/0.22.5		 libpciaccess/0.17	 numactl/2.0.18	     py-gast/0.5.4		    re2c/3.1
   berkeley-db/18.1.40			 glibc/2.35		 libsigsegv/2.14	 openblas/0.3.28     py-meson-python/0.16.0	    readline/8.2
   bison/3.8.2				 gmake/4.4.1		 libtool/2.4.7		 openmpi/5.0.5	     py-packaging/24.1		    sqlite/3.46.0
   bzip2/1.0.8				 hwloc/2.11.1		 libxcrypt/4.4.35	 openssh/9.9p1	     py-pip/23.1.2		    tar/1.34
   ca-certificates-mozilla/2023-05-30	 krb5/1.21.3		 libxml2/2.13.4		 openssl/3.4.0	     py-ply/3.11		    util-linux-uuid/2.40.2
   cmake/3.30.5				 libbsd/0.12.2		 m4/1.4.19		 perl/5.40.0	     py-pybind11/2.13.5		    util-macros/1.20.1
   curl/8.10.1				 libedit/3.1-20240808	 meson/1.5.1		 pigz/2.8	     py-pyproject-metadata/0.7.1    xz/5.4.6
   diffutils/3.10			 libevent/2.1.12	 mpich/4.2.3		 pkgconf/2.2.0	     py-setuptools/69.2.0	    yaksa/0.3
   expat/2.6.4				 libfabric/1.22.0	 ncurses/6.5		 pmix/5.0.3	     py-wheel/0.41.2		    zlib-ng/2.2.1
   findutils/4.9.0			 libffi/3.4.6		 netlib-lapack/3.11.0	 py-beniget/0.4.1    python-venv/1.0		    zstd/1.5.6
   gcc-runtime/12.3.0			 libiconv/1.17		 nghttp2/1.63.0		 py-cython/3.0.11    python/3.11.9

---------------------------------------------------------------- /home/spack5/stacks/modules/linux-ubuntu22.04-x86_64/Core ----------------------------------------------------------------
   gcc/12.3.0 (L)

  Where:
   D:  Default Module
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


$ module load mpich

Lmod is automatically replacing "openmpi/5.0.5" with "mpich/4.2.3".


Due to MODULEPATH changes, the following have been reloaded:
  1) netlib-scalapack/2.2.0

$ module load netlib-lapack

Lmod is automatically replacing "openblas/0.3.28" with "netlib-lapack/3.11.0".


Due to MODULEPATH changes, the following have been reloaded:
  1) netlib-scalapack/2.2.0

$ module purge

This concludes the quick tour of module file generation, and the tutorial on stacks.

Summary

In this tutorial, we configured Spack to install a stack of software built on a cross-product of different MPI and LAPACK libraries. We used the spec matrix syntax to express in a compact way the specs to be installed, and spec list definitions to reuse the same matrix rows in different places. Then, we discussed how to make the software easy to use, leveraging either filesystem views or module files.