Tutorial setup

If you have not done the prior sections, you’ll need to start the docker image:

docker run -it ghcr.io/spack/tutorial:hpcic25

and then set Spack up like this:

git clone --depth=2 --branch=releases/v1.0 https://github.com/spack/spack
. spack/share/spack/setup-env.sh
spack repo update builtin --tag v2025.07.0
spack mirror add --unsigned tutorial /mirror
spack compiler find

See the Basic Installation Tutorial for full details on setup. For more help, join us in the #tutorial channel on Slack – get an invitation at slack.spack.io

Warning

The spack tutorial -y command is intended for use in a container or VM. Use with care in other environments since it replaces some configuration files in order to establish suitable settings for the tutorial.

Stacks Tutorial

We’ve talked about Spack environments in the context of a unified user environment. But environments in Spack have much broader capabilities. In this tutorial we will consider how to use Spack environments to manage large deployments of software.

What usually differs between a typical environment for a single user, and an environment used to manage large deployments, is that in the latter case we often have a set of packages we want to install across a wide range of MPIs, LAPACKs or compilers.

Below, we’ll mimic the creation of a software stack built onto a cross-product of different LAPACK and MPI libraries, with a compiler that is more recent than the one provided by the host system.

First, we’ll focus on how to properly configure and install the software we want. We’ll learn how to pin certain requirements, and how to write a cross product of specs in a compact, and expressive, way.

We’ll consider how the software we install might be consumed by our users, and see the two main mechanisms that Spack provides for that: views and module files.

Note

Before we start this hands-on, make sure the EDITOR environment variable is set to your preferred editor, for instance:

$ export EDITOR='emacs -nw'

Setup the compiler

The first step to build our stack is to setup the compiler we want to use later. This is currently an iterative process that can be done in two ways:

  1. Install the compiler first, then install the rest of the environment

  2. Install the compiler outside the environment (either in another environment or standalone)

Below, we’ll use the first approach. For people interested, an example of composing a compiler from another environment can be found at this link.

Let’s start by creating an environment in a directory of our choice:

$ spack env activate --create ~/stacks
$ spack env status
==> In environment /home/spack/stacks

We can add from the command line a new compiler. We’ll also disable the generation of views for the time being, as we’ll come back to this topic later in the tutorial:

$ spack add gcc@12 %gcc@11
==> Adding gcc@12 %gcc@11 to environment /home/spack/stacks
$ spack env view disable
$ spack config edit

What you should see on screen now is the following spack.yaml file:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  specs:
  - gcc@12%gcc@11
  view: false
  concretizer:
    unify: true

The next step is to concretize and install our compiler:

$ spack concretize
==> Concretized 1 spec:
 -   fmxnit5  gcc@12.5.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  rwnqbli	  ^compiler-wrapper@1.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  pptkbjb	  ^diffutils@3.10 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  kgw5nks	      ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  jxq6uva	  ^gawk@5.3.1~nls build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  vefpsal	      ^libsigsegv@2.14 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  skysn5p	      ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[e]  j7ykyvm	  ^gcc@11.4.0~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-ubuntu22.04-x86_64_v3
[+]  3c5xh7d	  ^gcc-runtime@11.4.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[e]  epwvtix	  ^glibc@2.35 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  5vb6e4b	  ^gmake@4.4.1~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  zx3x4nm	  ^gmp@6.3.0+cxx build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  5xgjgae	      ^autoconf@2.72 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  vorbe53	      ^automake@1.16.5 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ypq2dye	      ^m4@1.4.20+sigsegv build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  l2jtwyu	  ^libtool@2.4.7 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ga3qiej	      ^findutils@4.10.0 build_system=autotools patches:=440b954 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  mkqx7bu	  ^mpc@1.3.1 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  abjn4ye	  ^mpfr@4.2.1 build_system=autotools libs:=shared,static patches:=3ec29a6 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  u24n5ok	      ^autoconf-archive@2023.02.20 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  nsqyer7	  ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  k5bepow	      ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  vy2ggzh	      ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  5yo63cn	      ^gdbm@1.23 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  z47kihj	  ^texinfo@7.1 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  soasbhm	      ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  7mdlpgu		  ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ye5hcpf		  ^tar@1.35 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  prsofdp		      ^pigz@2.8 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  zeynpnj		  ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  u7sn2jt	      ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  xoghzad		  ^pkgconf@2.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  g7ixtqy	  ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  bmaw6fn	  ^zstd@1.5.7+programs build_system=makefile compression:=none libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0

$ spack install
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/compiler-wrapper-1.0-rwnqblirwn5aenlmsbvsieo7mzw672ee
[+] /usr (external glibc-2.35-epwvtixhoxdeufu6kbbrpaigv2p6iav4)
[+] /usr (external gcc-11.4.0-j7ykyvm5ancnn5rt7f35ggs6hwoslo42)
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gcc-runtime-11.4.0-3c5xh7dqudxmyorlsrsfmjyggqk57vnr
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gmake-4.4.1-5vb6e4beeoff4jan2t7edfjyi3hf3p7g
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/findutils-4.10.0-ga3qiejil4e74fukhmtwcq5vrvzgmwmk
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libiconv-1.18-kgw5nkswdpf7k2itwhjwegwyszj3v56s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/zlib-ng-2.2.4-g7ixtqygrfip5uzt526b565dqxs6ze3x
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libsigsegv-2.14-vefpsalwx43ol7ppom7svfbsfqv3jit4
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/bzip2-1.0.8-vy2ggzhuzo7pav5clvz6dvamexpeaadk
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/berkeley-db-18.1.40-k5bepowzfoxlz67vuctewwbxlrj5sgji
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/ncurses-6.5-u7sn2jtlvafr3vqli4m7i7ysecwkwepd
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/zstd-1.5.7-bmaw6fn2i5mbbfujq6kf7bomodh4iudq
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gmp-6.3.0-zx3x4nm3djw5jdqxcsimfpxykoxdijhj
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/xz-5.6.3-zeynpnjapluctm4lxzuwo7en5jmdzt6c
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libtool-2.4.7-l2jtwyuivfhutipzs462o4ny7ctu4f5c
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/diffutils-3.10-pptkbjbkikngjuyusssjuvn2mqzl663q
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/pigz-2.8-prsofdph5zl75k54hjwgcr34lipeyuh5
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/readline-8.2-skysn5ppmam2ipxq4zlzwjxm42kurky2
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libxml2-2.13.5-7mdlpguaingjbgxrvtgjnv3y6372tofw
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/mpfr-4.2.1-abjn4yec4cwckmwvwwtg45keufsqyhya
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/tar-1.35-ye5hcpfjk5svng6s2sjj4n6huwokazfk
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gdbm-1.23-5yo63cnvgytwmsoiwoilhfgnyaffncmz
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/mpc-1.3.1-mkqx7budcw2rpbdzo63hsoxhhszx366g
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gawk-5.3.1-jxq6uva54usdixrtzlkwa6dxfaqeaeww
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gettext-0.23.1-soasbhmurzckxvnvfcqm22jsegvmh4jv
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/perl-5.40.0-nsqyer7nqsmv3ams4gk5mubo74maob3z
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/texinfo-7.1-z47kihjvhlc67ukoxtgtju72dcnhxn2z
==> Fetching file:///mirror/blobs/sha256/21/210bde294ec907128aa196cee83edfee0dcd8338971388101e38683f7e03d13d
    [100%]  641.04 MB @	 331.1 GB/s
==> Extracting gcc-12.5.0-fmxnit5ggu45byub6wlsuienth2dkboh from binary cache
==> Wrote new spec file to /home/spack/spack/opt/spack/linux-x86_64_v3/gcc-12.5.0-fmxnit5ggu45byub6wlsuienth2dkboh/lib/gcc/x86_64-pc-linux-gnu/12.5.0/specs
==> gcc: Successfully installed gcc-12.5.0-fmxnit5ggu45byub6wlsuienth2dkboh
  Search: 0.00s.  Fetch: 1.38s.	 Install: 12.74s.  Extract: 11.41s.  Relocate: 1.28s.  Total: 14.13s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gcc-12.5.0-fmxnit5ggu45byub6wlsuienth2dkboh
==> Installing gcc-12.5.0-fmxnit5ggu45byub6wlsuienth2dkboh [29/29]

We are ready to build more software with our newly installed GCC!

Install a software stack

Now that we have a compiler ready, the next objective is to build software with it. We’ll start by trying to add different versions of netlib-scalapack, linked against different MPI implementations:

$ spack add netlib-scalapack %gcc@12 ^openblas ^openmpi
==> Adding netlib-scalapack %gcc@12 ^openblas ^openmpi to environment /home/spack/stacks
$ spack add netlib-scalapack %gcc@12 ^openblas ^mpich
==> Adding netlib-scalapack %gcc@12 ^mpich ^openblas to environment /home/spack/stacks

If we try to concretize the environment, we’ll get an error. We cannot unify an environment with two different configurations of the same package.

Tuning concretizer options for a stack

Whenever we concretize an environment with more than one root spec, we can configure Spack to be more or less strict with duplicate nodes in the sub-DAG obtained by following link and run edges starting from the roots. We usually call this sub-DAG the root unification set.

A diagram might help visualize the concept:

_images/stacks-unify.svg

The image above represents the current environment, with our three root specs highlighted by a thicker dashed line. Any node that could be reached following a link or run edge is part of the root unification set. Pure build dependencies might fall outside of it.

The config option determining which nodes are allowed to be in the root unification set is concretizer:unify. Let’s check its value:

$ spack config get concretizer | grep unify
  unify: true

concretizer:unify:true means that only a single configuration for each package can be present. This value is good for single project environments, since it ensures we can construct a view of all the software, with the usual structure expected on a Unix-ish system, and without risks of collisions between installations.

We can’t respect this requirement, as our roots already contain two different configurations of netlib-scalapack. Set the value to false, and try to re-concretize:

$ spack config add concretizer:unify:false
$ spack concretize
==> Starting concretization pool with 2 processes
==> Concretized 2 specs:
 -   tl5f6ke  netlib-scalapack@2.2.2~ipo~pic+shared build_system=cmake build_type=Release generator=make arch=linux-ubuntu22.04-x86_64_v3 %c,fortran=gcc@12.5.0
 -   yxjigtd	  ^cmake@3.31.8~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   etcxzzk	      ^curl@8.11.1~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   2v5o3ra		  ^nghttp2@1.65.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   6lfrh36		      ^diffutils@3.10 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   osxtmvl		  ^openssl@3.4.1~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  s3bxsw4		      ^ca-certificates-mozilla@2025-05-20 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   56rxnkf	      ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   6xddq3y	      ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  rwnqbli	  ^compiler-wrapper@1.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  fmxnit5	  ^gcc@12.5.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  pptkbjb	      ^diffutils@3.10 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  kgw5nks		  ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  jxq6uva	      ^gawk@5.3.1~nls build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  vefpsal		  ^libsigsegv@2.14 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  skysn5p		  ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[e]  j7ykyvm	      ^gcc@11.4.0~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-ubuntu22.04-x86_64_v3
[+]  3c5xh7d	      ^gcc-runtime@11.4.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  5vb6e4b	      ^gmake@4.4.1~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  zx3x4nm	      ^gmp@6.3.0+cxx build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  5xgjgae		  ^autoconf@2.72 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  vorbe53		  ^automake@1.16.5 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ypq2dye		  ^m4@1.4.20+sigsegv build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  l2jtwyu	      ^libtool@2.4.7 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ga3qiej		  ^findutils@4.10.0 build_system=autotools patches:=440b954 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  mkqx7bu	      ^mpc@1.3.1 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  abjn4ye	      ^mpfr@4.2.1 build_system=autotools libs:=shared,static patches:=3ec29a6 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  u24n5ok		  ^autoconf-archive@2023.02.20 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  nsqyer7	      ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  k5bepow		  ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  vy2ggzh		  ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  5yo63cn		  ^gdbm@1.23 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  z47kihj	      ^texinfo@7.1 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  soasbhm		  ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  7mdlpgu		      ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ye5hcpf		      ^tar@1.35 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  prsofdp			  ^pigz@2.8 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  zeynpnj		      ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  u7sn2jt		  ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  xoghzad		      ^pkgconf@2.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  g7ixtqy	      ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  bmaw6fn	      ^zstd@1.5.7+programs build_system=makefile compression:=none libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
 -   zubbt4y	  ^gcc-runtime@12.5.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[e]  epwvtix	  ^glibc@2.35 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   xvxobm5	  ^gmake@4.4.1~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   3bsqfey	  ^mpich@4.3.0~argobots~cuda+fortran+hwloc+hydra~level_zero+libxml2+pci~rocm+romio~slurm~vci~verbs+wrapperrpath~xpmem build_system=autotools datatype-engine=auto device=ch4 netmod=ofi pmi=default arch=linux-ubuntu22.04-x86_64_v3 %c,cxx,fortran=gcc@12.5.0
 -   vhsfohm	      ^findutils@4.10.0 build_system=autotools patches:=440b954 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   goyejxw		  ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   4do6jx7		      ^tar@1.35 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   zia4xj2			  ^pigz@2.8 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   2jhx5gu			  ^zstd@1.5.7+programs build_system=makefile compression:=none libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   3e4kpzp	      ^hwloc@2.11.1~cairo~cuda~gl~level_zero~libudev+libxml2~nvml~opencl+pci~rocm build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   atjn3er	      ^libfabric@2.2.0~cuda~debug~kdreg~level_zero~uring build_system=autotools fabrics:=sockets,tcp,udp arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   j336pb5	      ^libpciaccess@0.17 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   kbleacp		  ^util-macros@1.20.1 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   f2wxsch	      ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   m2sseei		  ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   3kvqody		  ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   qz3jy42	      ^pkgconf@2.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   5r6tuoo	      ^yaksa@0.3~cuda~level_zero~rocm build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   rpphs4i		  ^m4@1.4.20+sigsegv build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   ffctxur		      ^libsigsegv@2.14 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   musqpuq		  ^python@3.13.5+bz2+ctypes+dbm~debug+libxml2+lzma~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   lfwqiuw		      ^expat@2.7.1+libbsd build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   es3e2le			  ^libbsd@0.12.2 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   bqguhkf			      ^libmd@1.1.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   ddom6iu		      ^libffi@3.4.8 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   lxvqq5o		      ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   brglstm		      ^sqlite@3.46.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   v2nef6x		      ^util-linux-uuid@2.41 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   z3ynokz	  ^openblas@0.3.29~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-ubuntu22.04-x86_64_v3 %c,cxx,fortran=gcc@12.5.0
 -   nimjflq  netlib-scalapack@2.2.2~ipo~pic+shared build_system=cmake build_type=Release generator=make arch=linux-ubuntu22.04-x86_64_v3 %c,fortran=gcc@12.5.0
 -   qdunbui	  ^openmpi@5.0.8+atomics~cuda~debug+fortran~gpfs~internal-hwloc~internal-libevent~internal-pmix~ipv6~java~lustre~memchecker~openshmem~rocm~romio+rsh~static~two_level_namespace+vt+wrapper-rpath build_system=autotools fabrics:=none romio-filesystem:=none schedulers:=none arch=linux-ubuntu22.04-x86_64_v3 %c,cxx,fortran=gcc@12.5.0
 -   krgi7jf	      ^autoconf@2.72 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   jgeyxwq	      ^automake@1.16.5 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   alldwq4	      ^libevent@2.1.12+openssl build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   7z4rzcr	      ^libtool@2.4.7 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   naapp6h	      ^numactl@2.0.18 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   sgecnst	      ^openssh@9.9p1+gssapi build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   nm4trzq		  ^krb5@1.21.3+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   acgafis		      ^bison@3.8.2~color build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   wspzmod		  ^libedit@3.1-20240808 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   qhnn6wr		  ^libxcrypt@4.4.38~obsolete_api build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   txcbw3f	      ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   ikx4n4c		  ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   3kql5yq		  ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   cxbuzxk		  ^gdbm@1.23 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   wqd5fdu	      ^pmix@5.0.5~munge~python build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0

This time concretization succeeded. Setting concretizer:unify:false is effectively concretizing each root spec on its own, and then merging the results into the environment. This allows us to have the duplicates we need.

Note

If the environment is expected to have only a few duplicate nodes, then there’s another value we might consider:

$ spack config add concretizer:unify:when_possible

With this option Spack will try to unify the environment in an eager way, solving it in multiple rounds. The concretization at round n will contain all the specs that could not be unified at round n-1, and will consider all the specs from previous rounds for reuse.

Spec matrices

Let’s further expand our stack and consider also linking against different LAPACK providers. We could, of course, add new specs explicitly:

$ spack add netlib-scalapack %gcc@12 ^netlib-lapack ^openmpi
==> Adding netlib-scalapack %gcc@12 ^netlib-lapack ^openmpi to environment /home/spack/stacks
$ spack add netlib-scalapack %gcc@12 ^netlib-lapack ^mpich
==> Adding netlib-scalapack %gcc@12 ^mpich ^netlib-lapack to environment /home/spack/stacks

This way of proceeding, though, will become very tedious once more software is requested. The best way to express a cross-product like this in Spack is instead through a matrix:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  specs:
  - gcc@12%gcc@11
  - matrix:
    - [netlib-scalapack]
    - [^openmpi, ^mpich]
    - [^openblas, ^netlib-lapack]
    - ["%gcc@12"]
  view: false
  concretizer:
    unify: false

Matrices will expand to the cross-product of their rows, so this matrix:

- matrix:
 - ["netlib-scalapack"]
 - ["^openmpi", "^mpich"]
 - ["^openblas", "^netlib-lapack"]
 - ["%gcc@12"]

is equivalent to this list of specs:

- "netlib-scalapack %gcc@12 ^openblas ^openmpi"
- "netlib-scalapack %gcc@12 ^openblas ^mpich"
- "netlib-scalapack %gcc@12 ^netlib-lapack ^openmpi"
- "netlib-scalapack %gcc@12 ^netlib-lapack ^mpich"

We are now ready to concretize and install the environment:

$ spack concretize
==> Starting concretization pool with 2 processes
==> Concretized 2 specs:
 -   igwbxq4  netlib-scalapack@2.2.2~ipo~pic+shared build_system=cmake build_type=Release generator=make arch=linux-ubuntu22.04-x86_64_v3 %c,fortran=gcc@12.5.0
 -   yxjigtd	  ^cmake@3.31.8~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   etcxzzk	      ^curl@8.11.1~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   2v5o3ra		  ^nghttp2@1.65.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   6lfrh36		      ^diffutils@3.10 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   osxtmvl		  ^openssl@3.4.1~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  s3bxsw4		      ^ca-certificates-mozilla@2025-05-20 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   56rxnkf	      ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   6xddq3y	      ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  rwnqbli	  ^compiler-wrapper@1.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  fmxnit5	  ^gcc@12.5.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  pptkbjb	      ^diffutils@3.10 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  kgw5nks		  ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  jxq6uva	      ^gawk@5.3.1~nls build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  vefpsal		  ^libsigsegv@2.14 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  skysn5p		  ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[e]  j7ykyvm	      ^gcc@11.4.0~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-ubuntu22.04-x86_64_v3
[+]  3c5xh7d	      ^gcc-runtime@11.4.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  5vb6e4b	      ^gmake@4.4.1~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  zx3x4nm	      ^gmp@6.3.0+cxx build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  5xgjgae		  ^autoconf@2.72 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  vorbe53		  ^automake@1.16.5 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ypq2dye		  ^m4@1.4.20+sigsegv build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  l2jtwyu	      ^libtool@2.4.7 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ga3qiej		  ^findutils@4.10.0 build_system=autotools patches:=440b954 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  mkqx7bu	      ^mpc@1.3.1 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  abjn4ye	      ^mpfr@4.2.1 build_system=autotools libs:=shared,static patches:=3ec29a6 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  u24n5ok		  ^autoconf-archive@2023.02.20 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  nsqyer7	      ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  k5bepow		  ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  vy2ggzh		  ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  5yo63cn		  ^gdbm@1.23 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  z47kihj	      ^texinfo@7.1 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  soasbhm		  ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  7mdlpgu		      ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ye5hcpf		      ^tar@1.35 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  prsofdp			  ^pigz@2.8 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  zeynpnj		      ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  u7sn2jt		  ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  xoghzad		      ^pkgconf@2.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  g7ixtqy	      ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  bmaw6fn	      ^zstd@1.5.7+programs build_system=makefile compression:=none libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
 -   zubbt4y	  ^gcc-runtime@12.5.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[e]  epwvtix	  ^glibc@2.35 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   xvxobm5	  ^gmake@4.4.1~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   3bsqfey	  ^mpich@4.3.0~argobots~cuda+fortran+hwloc+hydra~level_zero+libxml2+pci~rocm+romio~slurm~vci~verbs+wrapperrpath~xpmem build_system=autotools datatype-engine=auto device=ch4 netmod=ofi pmi=default arch=linux-ubuntu22.04-x86_64_v3 %c,cxx,fortran=gcc@12.5.0
 -   vhsfohm	      ^findutils@4.10.0 build_system=autotools patches:=440b954 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   goyejxw		  ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   4do6jx7		      ^tar@1.35 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   zia4xj2			  ^pigz@2.8 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   2jhx5gu			  ^zstd@1.5.7+programs build_system=makefile compression:=none libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   3e4kpzp	      ^hwloc@2.11.1~cairo~cuda~gl~level_zero~libudev+libxml2~nvml~opencl+pci~rocm build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   atjn3er	      ^libfabric@2.2.0~cuda~debug~kdreg~level_zero~uring build_system=autotools fabrics:=sockets,tcp,udp arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   j336pb5	      ^libpciaccess@0.17 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   kbleacp		  ^util-macros@1.20.1 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   f2wxsch	      ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   m2sseei		  ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   3kvqody		  ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   qz3jy42	      ^pkgconf@2.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   5r6tuoo	      ^yaksa@0.3~cuda~level_zero~rocm build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   rpphs4i		  ^m4@1.4.20+sigsegv build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   ffctxur		      ^libsigsegv@2.14 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   musqpuq		  ^python@3.13.5+bz2+ctypes+dbm~debug+libxml2+lzma~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   lfwqiuw		      ^expat@2.7.1+libbsd build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   es3e2le			  ^libbsd@0.12.2 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   bqguhkf			      ^libmd@1.1.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   ddom6iu		      ^libffi@3.4.8 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   lxvqq5o		      ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   brglstm		      ^sqlite@3.46.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   v2nef6x		      ^util-linux-uuid@2.41 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   xx23tbf	  ^netlib-lapack@3.12.1~external-blas~ipo+lapacke+pic+shared~xblas build_system=cmake build_type=Release generator=make patches:=3059ebf,b1af8b6,e318340 arch=linux-ubuntu22.04-x86_64_v3 %c,fortran=gcc@12.5.0
 -   hk7ftpc  netlib-scalapack@2.2.2~ipo~pic+shared build_system=cmake build_type=Release generator=make arch=linux-ubuntu22.04-x86_64_v3 %c,fortran=gcc@12.5.0
 -   qdunbui	  ^openmpi@5.0.8+atomics~cuda~debug+fortran~gpfs~internal-hwloc~internal-libevent~internal-pmix~ipv6~java~lustre~memchecker~openshmem~rocm~romio+rsh~static~two_level_namespace+vt+wrapper-rpath build_system=autotools fabrics:=none romio-filesystem:=none schedulers:=none arch=linux-ubuntu22.04-x86_64_v3 %c,cxx,fortran=gcc@12.5.0
 -   krgi7jf	      ^autoconf@2.72 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
 -   jgeyxwq	      ^automake@1.16.5 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   alldwq4	      ^libevent@2.1.12+openssl build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   7z4rzcr	      ^libtool@2.4.7 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   naapp6h	      ^numactl@2.0.18 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   sgecnst	      ^openssh@9.9p1+gssapi build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   nm4trzq		  ^krb5@1.21.3+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   acgafis		      ^bison@3.8.2~color build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   wspzmod		  ^libedit@3.1-20240808 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   qhnn6wr		  ^libxcrypt@4.4.38~obsolete_api build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   txcbw3f	      ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   ikx4n4c		  ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   3kql5yq		  ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   cxbuzxk		  ^gdbm@1.23 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   wqd5fdu	      ^pmix@5.0.5~munge~python build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0

$ spack install
[+] /usr (external glibc-2.35-epwvtixhoxdeufu6kbbrpaigv2p6iav4)
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/compiler-wrapper-1.0-rwnqblirwn5aenlmsbvsieo7mzw672ee
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/ca-certificates-mozilla-2025-05-20-s3bxsw4irc4hd32422hnjg3lkj32ii4y
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gcc-runtime-11.4.0-3c5xh7dqudxmyorlsrsfmjyggqk57vnr
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/zstd-1.5.7-bmaw6fn2i5mbbfujq6kf7bomodh4iudq
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/zlib-ng-2.2.4-g7ixtqygrfip5uzt526b565dqxs6ze3x
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gmp-6.3.0-zx3x4nm3djw5jdqxcsimfpxykoxdijhj
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/mpfr-4.2.1-abjn4yec4cwckmwvwwtg45keufsqyhya
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/mpc-1.3.1-mkqx7budcw2rpbdzo63hsoxhhszx366g
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gcc-12.5.0-fmxnit5ggu45byub6wlsuienth2dkboh
==> Fetching file:///mirror/blobs/sha256/5b/5b48813384c089dc6e9bca695e8f7ab66744d263bf093d67b8ae0b81e4cc86fd
    [100%]   19.43 MB @	 371.3 GB/s
==> Extracting gcc-runtime-12.5.0-zubbt4yxn35hmaarhmzluk6cb6gkujee from binary cache
==> gcc-runtime: Successfully installed gcc-runtime-12.5.0-zubbt4yxn35hmaarhmzluk6cb6gkujee
  Search: 0.00s.  Fetch: 0.16s.	 Install: 0.50s.  Extract: 0.41s.  Relocate: 0.06s.  Total: 0.66s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gcc-runtime-12.5.0-zubbt4yxn35hmaarhmzluk6cb6gkujee
==> Installing gcc-runtime-12.5.0-zubbt4yxn35hmaarhmzluk6cb6gkujee [11/66]
==> Fetching file:///mirror/blobs/sha256/4d/4d98c8c2e00e54df4894cb554f14891465506523b5985720553f31449034724f
    [100%]  414.75 KB @	 710.6 MB/s
==> Extracting gmake-4.4.1-xvxobm5cq3cv3tykatbuyj5veywnyayy from binary cache
==> gmake: Successfully installed gmake-4.4.1-xvxobm5cq3cv3tykatbuyj5veywnyayy
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gmake-4.4.1-xvxobm5cq3cv3tykatbuyj5veywnyayy
==> Installing gmake-4.4.1-xvxobm5cq3cv3tykatbuyj5veywnyayy [12/66]
==> Fetching file:///mirror/blobs/sha256/91/91f1b22e3962258b3b807dbfb546f4c494dd331fa40c8c2a75350afd3dc823dc
    [100%]   63.42 MB @	 411.2 GB/s
==> Extracting berkeley-db-18.1.40-ikx4n4c7kofxzwerpdkc2qccur5d232p from binary cache
==> berkeley-db: Successfully installed berkeley-db-18.1.40-ikx4n4c7kofxzwerpdkc2qccur5d232p
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.11s.  Extract: 0.08s.  Relocate: 0.02s.  Total: 0.13s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/berkeley-db-18.1.40-ikx4n4c7kofxzwerpdkc2qccur5d232p
==> Installing berkeley-db-18.1.40-ikx4n4c7kofxzwerpdkc2qccur5d232p [13/66]
==> Fetching file:///mirror/blobs/sha256/e4/e43ab959670dbca332b33df5818d213b319f53285bc22f35c21f01c116193989
    [100%]  379.69 KB @	 699.1 MB/s
==> Extracting pkgconf-2.3.0-qz3jy42ylzeesnhmg3cw735reuumm35f from binary cache
==> pkgconf: Successfully installed pkgconf-2.3.0-qz3jy42ylzeesnhmg3cw735reuumm35f
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/pkgconf-2.3.0-qz3jy42ylzeesnhmg3cw735reuumm35f
==> Installing pkgconf-2.3.0-qz3jy42ylzeesnhmg3cw735reuumm35f [14/66]
==> Fetching file:///mirror/blobs/sha256/4e/4ecea5d6cb655c447ed641c44a96b89371102ec2af6ad9dcf9940bec28f8efed
    [100%]  173.24 KB @	 530.4 MB/s
==> Extracting libmd-1.1.0-bqguhkf6f3br7tzqgv5srgsaqy7v2vb4 from binary cache
==> libmd: Successfully installed libmd-1.1.0-bqguhkf6f3br7tzqgv5srgsaqy7v2vb4
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libmd-1.1.0-bqguhkf6f3br7tzqgv5srgsaqy7v2vb4
==> Installing libmd-1.1.0-bqguhkf6f3br7tzqgv5srgsaqy7v2vb4 [15/66]
==> Fetching file:///mirror/blobs/sha256/89/8985654e0bcf6bc93d3606fd79890da328bb4f0f9fc1581097b4d8092978e078
    [100%]   91.22 KB @	 326.7 MB/s
==> Extracting libffi-3.4.8-ddom6iure3h5olpwkvhm2xokpbebdb4m from binary cache
==> libffi: Successfully installed libffi-3.4.8-ddom6iure3h5olpwkvhm2xokpbebdb4m
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.04s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libffi-3.4.8-ddom6iure3h5olpwkvhm2xokpbebdb4m
==> Installing libffi-3.4.8-ddom6iure3h5olpwkvhm2xokpbebdb4m [16/66]
==> Fetching file:///mirror/blobs/sha256/e4/e47a73b839b90e49baa4adbdc515cff036c59881eda2017b4099151cd51ffb9c
    [100%]   61.25 MB @	 918.0 MB/s
==> Extracting libfabric-2.2.0-atjn3er45pppotm6ksncj4rftg4bo7q2 from binary cache
==> libfabric: Successfully installed libfabric-2.2.0-atjn3er45pppotm6ksncj4rftg4bo7q2
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.06s.  Relocate: 0.01s.  Total: 0.10s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libfabric-2.2.0-atjn3er45pppotm6ksncj4rftg4bo7q2
==> Installing libfabric-2.2.0-atjn3er45pppotm6ksncj4rftg4bo7q2 [17/66]
==> Fetching file:///mirror/blobs/sha256/07/07dacf553975f0cfc11fbe3f54cdacfe8292d5a5025dd766850f0a31c95f5bf4
    [100%]  115.08 KB @	 395.6 MB/s
==> Extracting libsigsegv-2.14-ffctxurczjrftrybtsu2rijddwxstsly from binary cache
==> libsigsegv: Successfully installed libsigsegv-2.14-ffctxurczjrftrybtsu2rijddwxstsly
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libsigsegv-2.14-ffctxurczjrftrybtsu2rijddwxstsly
==> Installing libsigsegv-2.14-ffctxurczjrftrybtsu2rijddwxstsly [18/66]
==> Fetching file:///mirror/blobs/sha256/63/63fbdb74e3f39643401343319de73e6f1790b225443ccd44582cc83f8357e5c8
    [100%]  221.46 KB @	 532.3 MB/s
==> Extracting zlib-ng-2.2.4-6xddq3ypcya5hm72cccnuct4kgbquf3c from binary cache
==> zlib-ng: Successfully installed zlib-ng-2.2.4-6xddq3ypcya5hm72cccnuct4kgbquf3c
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/zlib-ng-2.2.4-6xddq3ypcya5hm72cccnuct4kgbquf3c
==> Installing zlib-ng-2.2.4-6xddq3ypcya5hm72cccnuct4kgbquf3c [19/66]
==> Fetching file:///mirror/blobs/sha256/48/489656d4ab0db44a4dc842789e99c3257392054fee10080bb68ba4273ae83513
    [100%]   62.14 MB @	 381.1 GB/s
==> Extracting libiconv-1.18-m2sseeiaralmgv67wmrzua2dohic4pdq from binary cache
==> libiconv: Successfully installed libiconv-1.18-m2sseeiaralmgv67wmrzua2dohic4pdq
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.05s.  Relocate: 0.01s.  Total: 0.10s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libiconv-1.18-m2sseeiaralmgv67wmrzua2dohic4pdq
==> Installing libiconv-1.18-m2sseeiaralmgv67wmrzua2dohic4pdq [20/66]
==> Fetching file:///mirror/blobs/sha256/fd/fd1c9cfedc56d7f9c4632d7b54be69a8dfa5324f8b088a796a7c970348a96648
    [100%]   61.81 MB @	 391.1 GB/s
==> Extracting xz-5.6.3-3kvqodyyfedqojoooz7zowiqqxaqgclz from binary cache
==> xz: Successfully installed xz-5.6.3-3kvqodyyfedqojoooz7zowiqqxaqgclz
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.11s.  Extract: 0.07s.  Relocate: 0.01s.  Total: 0.12s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/xz-5.6.3-3kvqodyyfedqojoooz7zowiqqxaqgclz
==> Installing xz-5.6.3-3kvqodyyfedqojoooz7zowiqqxaqgclz [21/66]
==> Fetching file:///mirror/blobs/sha256/64/64ea13b178ef5ccc63a5609fe1ef5e3685320cfde43d7289bbd9cb993e7122bc
    [100%]   23.66 MB @	 431.3 GB/s
==> Extracting openblas-0.3.29-z3ynokzpyv4lr3hh6qrcc2y2ucrpmt26 from binary cache
==> openblas: Successfully installed openblas-0.3.29-z3ynokzpyv4lr3hh6qrcc2y2ucrpmt26
  Search: 0.00s.  Fetch: 0.04s.	 Install: 0.61s.  Extract: 0.54s.  Relocate: 0.04s.  Total: 0.65s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/openblas-0.3.29-z3ynokzpyv4lr3hh6qrcc2y2ucrpmt26
==> Installing openblas-0.3.29-z3ynokzpyv4lr3hh6qrcc2y2ucrpmt26 [22/66]
==> Fetching file:///mirror/blobs/sha256/14/14befd94eb9628e69c9bcd45ce4daabc7ce3a70cdcb01ec0b7353647c9f5ec2b
    [100%]   36.60 KB @	 207.2 MB/s
==> Extracting util-macros-1.20.1-kbleacprn5sclam2rjwplmc7eyhp44ji from binary cache
==> util-macros: Successfully installed util-macros-1.20.1-kbleacprn5sclam2rjwplmc7eyhp44ji
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.00s.  Total: 0.04s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/util-macros-1.20.1-kbleacprn5sclam2rjwplmc7eyhp44ji
==> Installing util-macros-1.20.1-kbleacprn5sclam2rjwplmc7eyhp44ji [23/66]
==> Fetching file:///mirror/blobs/sha256/b5/b55183adefbf4e0301ea2619c6192ebc233ce393a84c0148b67b3c9955fa2eca
    [100%]   61.53 MB @	 401.0 GB/s
==> Extracting zstd-1.5.7-2jhx5guzifg27gtix5cjlcqm6fqdfs2b from binary cache
==> zstd: Successfully installed zstd-1.5.7-2jhx5guzifg27gtix5cjlcqm6fqdfs2b
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.04s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/zstd-1.5.7-2jhx5guzifg27gtix5cjlcqm6fqdfs2b
==> Installing zstd-1.5.7-2jhx5guzifg27gtix5cjlcqm6fqdfs2b [24/66]
==> Fetching file:///mirror/blobs/sha256/88/88565abe3323af9b55348c2f3d2ed9c2721049c7736463e9d174deadcce03263
    [100%]   66.79 MB @	 411.2 GB/s
==> Extracting ncurses-6.5-56rxnkfzlb5k3dceq52it64acuui3syl from binary cache
==> ncurses: Successfully installed ncurses-6.5-56rxnkfzlb5k3dceq52it64acuui3syl
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.65s.  Extract: 0.52s.  Relocate: 0.01s.  Total: 0.67s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/ncurses-6.5-56rxnkfzlb5k3dceq52it64acuui3syl
==> Installing ncurses-6.5-56rxnkfzlb5k3dceq52it64acuui3syl [25/66]
==> Fetching file:///mirror/blobs/sha256/6f/6faef1135d30b82cd508aaaa0d033a35567640b36fcf5c918db045e3b9a02ade
    [100%]   63.78 MB @	 411.2 GB/s
==> Extracting util-linux-uuid-2.41-v2nef6xnzn36kard3s336ix4lnkb2n3l from binary cache
==> util-linux-uuid: Successfully installed util-linux-uuid-2.41-v2nef6xnzn36kard3s336ix4lnkb2n3l
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.13s.  Extract: 0.10s.  Relocate: 0.01s.  Total: 0.14s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/util-linux-uuid-2.41-v2nef6xnzn36kard3s336ix4lnkb2n3l
==> Installing util-linux-uuid-2.41-v2nef6xnzn36kard3s336ix4lnkb2n3l [26/66]
==> Fetching file:///mirror/blobs/sha256/11/11aae06c95b9bb852ad0bf4bef8024ea0266e6e7c2a948c7963a21d34ecadd08
    [100%]  445.48 KB @	 698.0 MB/s
==> Extracting libbsd-0.12.2-es3e2le2oexbmt3j4ik6rphojgz63u2a from binary cache
==> libbsd: Successfully installed libbsd-0.12.2-es3e2le2oexbmt3j4ik6rphojgz63u2a
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.05s.  Relocate: 0.01s.  Total: 0.09s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libbsd-0.12.2-es3e2le2oexbmt3j4ik6rphojgz63u2a
==> Installing libbsd-0.12.2-es3e2le2oexbmt3j4ik6rphojgz63u2a [27/66]
==> Fetching file:///mirror/blobs/sha256/8d/8d65fe4c5874881bc633cbca416a4a233382de2b5f58316b44c41a271c151a69
    [100%]   91.32 KB @	 318.9 MB/s
==> Extracting pigz-2.8-zia4xj2iduhdjieipgnta3vrquz5eprh from binary cache
==> pigz: Successfully installed pigz-2.8-zia4xj2iduhdjieipgnta3vrquz5eprh
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/pigz-2.8-zia4xj2iduhdjieipgnta3vrquz5eprh
==> Installing pigz-2.8-zia4xj2iduhdjieipgnta3vrquz5eprh [28/66]
==> Fetching file:///mirror/blobs/sha256/8c/8ceb3f7f2d7496eba70cffc5e5668abef76bc632cc400ea4596910e9d76d4612
    [100%]   61.10 MB @	 411.0 GB/s
==> Extracting diffutils-3.10-6lfrh36pe32k2syi2dkya5urzwgm67io from binary cache
==> diffutils: Successfully installed diffutils-3.10-6lfrh36pe32k2syi2dkya5urzwgm67io
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.04s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/diffutils-3.10-6lfrh36pe32k2syi2dkya5urzwgm67io
==> Installing diffutils-3.10-6lfrh36pe32k2syi2dkya5urzwgm67io [29/66]
==> Fetching file:///mirror/blobs/sha256/69/69642f5e700e5d38aef7356d7aafbe4fdb84db443c1b187a48dc9a4cc733cfb0
    [100%]   62.67 MB @	 401.2 GB/s
==> Extracting libxml2-2.13.5-f2wxschdrknghuvhaqo7ldb3na3s7bl5 from binary cache
==> libxml2: Successfully installed libxml2-2.13.5-f2wxschdrknghuvhaqo7ldb3na3s7bl5
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.12s.  Extract: 0.08s.  Relocate: 0.01s.  Total: 0.13s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libxml2-2.13.5-f2wxschdrknghuvhaqo7ldb3na3s7bl5
==> Installing libxml2-2.13.5-f2wxschdrknghuvhaqo7ldb3na3s7bl5 [30/66]
==> Fetching file:///mirror/blobs/sha256/e9/e9e0aa391b29dc1090626c3f0859ee05a762ab01167160c03492c3d9793cdbb1
    [100%]  178.61 KB @	 506.9 MB/s
==> Extracting libpciaccess-0.17-j336pb5rrnbu5btdjgjxxcbrwfc3iffa from binary cache
==> libpciaccess: Successfully installed libpciaccess-0.17-j336pb5rrnbu5btdjgjxxcbrwfc3iffa
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.04s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libpciaccess-0.17-j336pb5rrnbu5btdjgjxxcbrwfc3iffa
==> Installing libpciaccess-0.17-j336pb5rrnbu5btdjgjxxcbrwfc3iffa [31/66]
==> Fetching file:///mirror/blobs/sha256/d9/d964013bba222a2de543700e0656a17a4fd134252977e2549494e5682c216491
    [100%]  890.88 KB @	 946.9 MB/s
==> Extracting libedit-3.1-20240808-wspzmodxgcbagfmhnn3ezs3cjcpfo7if from binary cache
==> libedit: Successfully installed libedit-3.1-20240808-wspzmodxgcbagfmhnn3ezs3cjcpfo7if
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.06s.  Extract: 0.03s.  Relocate: 0.01s.  Total: 0.07s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libedit-3.1-20240808-wspzmodxgcbagfmhnn3ezs3cjcpfo7if
==> Installing libedit-3.1-20240808-wspzmodxgcbagfmhnn3ezs3cjcpfo7if [32/66]
==> Fetching file:///mirror/blobs/sha256/56/5655c37ecd72f4b425fa86f17e28387bc1357d44bec3f39cba422bea746304c0
    [100%]   61.41 MB @	 451.1 GB/s
==> Extracting readline-8.2-lxvqq5ood4lil2pjv2ofawfytyal4n37 from binary cache
==> readline: Successfully installed readline-8.2-lxvqq5ood4lil2pjv2ofawfytyal4n37
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.04s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/readline-8.2-lxvqq5ood4lil2pjv2ofawfytyal4n37
==> Installing readline-8.2-lxvqq5ood4lil2pjv2ofawfytyal4n37 [33/66]
==> Fetching file:///mirror/blobs/sha256/ad/adcc73426d45ac99e2c45a05ef2bbb0c3917e0324389fdd0a127afdca0b5f3d2
    [100%]  602.94 KB @	 809.2 MB/s
==> Extracting expat-2.7.1-lfwqiuwkitd5mxyk52v633u6soghu4cv from binary cache
==> expat: Successfully installed expat-2.7.1-lfwqiuwkitd5mxyk52v633u6soghu4cv
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/expat-2.7.1-lfwqiuwkitd5mxyk52v633u6soghu4cv
==> Installing expat-2.7.1-lfwqiuwkitd5mxyk52v633u6soghu4cv [34/66]
==> Fetching file:///mirror/blobs/sha256/79/799d044aaf5841fea021578a5aca4d4cd234f411047069101a312c0eabc732d9
    [100%]  880.78 KB @	 953.1 MB/s
==> Extracting nghttp2-1.65.0-2v5o3rauqqx5v5fytm7ogfimuzrvhomt from binary cache
==> nghttp2: Successfully installed nghttp2-1.65.0-2v5o3rauqqx5v5fytm7ogfimuzrvhomt
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.06s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.07s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/nghttp2-1.65.0-2v5o3rauqqx5v5fytm7ogfimuzrvhomt
==> Installing nghttp2-1.65.0-2v5o3rauqqx5v5fytm7ogfimuzrvhomt [35/66]
==> Fetching file:///mirror/blobs/sha256/1b/1ba627cc7a245a7b1be349d37deda65a68f289a565d37e7326e008a4f7a3ad84
    [100%]  288.61 KB @	 571.8 MB/s
==> Extracting bzip2-1.0.8-3kql5yq6ucjcp4ltisvme37ltccgn6yt from binary cache
==> bzip2: Successfully installed bzip2-1.0.8-3kql5yq6ucjcp4ltisvme37ltccgn6yt
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.05s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/bzip2-1.0.8-3kql5yq6ucjcp4ltisvme37ltccgn6yt
==> Installing bzip2-1.0.8-3kql5yq6ucjcp4ltisvme37ltccgn6yt [36/66]
==> Fetching file:///mirror/blobs/sha256/8d/8dec466f8949f252a44ca5425e911be4b74691962e66ffc978267d9c72ef0e29
    [100%]  733.71 KB @	 846.1 MB/s
==> Extracting m4-1.4.20-rpphs4iqga5izscs5jklti47p3xfrrvk from binary cache
==> m4: Successfully installed m4-1.4.20-rpphs4iqga5izscs5jklti47p3xfrrvk
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.03s.  Relocate: 0.01s.  Total: 0.07s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/m4-1.4.20-rpphs4iqga5izscs5jklti47p3xfrrvk
==> Installing m4-1.4.20-rpphs4iqga5izscs5jklti47p3xfrrvk [37/66]
==> Fetching file:///mirror/blobs/sha256/79/7977d342bb4742a1635c362ec330e77892283e9e4a55dd5d9655a1fae605814a
    [100%]   64.86 MB @	 401.3 GB/s
==> Extracting hwloc-2.11.1-3e4kpzphqrlylwwtfn6qw4mevnzoiwzm from binary cache
==> hwloc: Successfully installed hwloc-2.11.1-3e4kpzphqrlylwwtfn6qw4mevnzoiwzm
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.19s.  Extract: 0.15s.  Relocate: 0.02s.  Total: 0.21s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/hwloc-2.11.1-3e4kpzphqrlylwwtfn6qw4mevnzoiwzm
==> Installing hwloc-2.11.1-3e4kpzphqrlylwwtfn6qw4mevnzoiwzm [38/66]
==> Fetching file:///mirror/blobs/sha256/89/89889fd5ccbd632e89e1fbb162bae03b3a199d6c14e31a8237aad3fa76e03b36
    [100%]  874.82 KB @	 935.1 MB/s
==> Extracting gdbm-1.23-cxbuzxk4lqvsqfdneevy232tdn25p62d from binary cache
==> gdbm: Successfully installed gdbm-1.23-cxbuzxk4lqvsqfdneevy232tdn25p62d
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.16s.  Extract: 0.03s.  Relocate: 0.01s.  Total: 0.17s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gdbm-1.23-cxbuzxk4lqvsqfdneevy232tdn25p62d
==> Installing gdbm-1.23-cxbuzxk4lqvsqfdneevy232tdn25p62d [39/66]
==> Fetching file:///mirror/blobs/sha256/6d/6df6984ce0ded10efbb560c4b1bce44e34df6806a0e6fdf56879f710b2e3fb9a
    [100%]   10.94 MB @	 391.3 GB/s
==> Extracting sqlite-3.46.0-brglstmfyna5l3gv7v7pcldkfzx5xvgo from binary cache
==> sqlite: Successfully installed sqlite-3.46.0-brglstmfyna5l3gv7v7pcldkfzx5xvgo
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.26s.  Extract: 0.21s.  Relocate: 0.02s.  Total: 0.29s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/sqlite-3.46.0-brglstmfyna5l3gv7v7pcldkfzx5xvgo
==> Installing sqlite-3.46.0-brglstmfyna5l3gv7v7pcldkfzx5xvgo [40/66]
==> Fetching file:///mirror/blobs/sha256/fb/fba36ae4b90f177f76f8423bb27da8637d2a082af0454dbf4376648ba1346ecf
    [100%]   61.26 MB @	 949.5 MB/s
==> Extracting tar-1.35-4do6jx74chitcnc4sw3l4gdxtno44z2a from binary cache
==> tar: Successfully installed tar-1.35-4do6jx74chitcnc4sw3l4gdxtno44z2a
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.07s.  Extract: 0.03s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/tar-1.35-4do6jx74chitcnc4sw3l4gdxtno44z2a
==> Installing tar-1.35-4do6jx74chitcnc4sw3l4gdxtno44z2a [41/66]
==> Fetching file:///mirror/blobs/sha256/9b/9b3857eb8e07e850afe47747495a88c8980facafb70a2aba7c670b5f3b4affdd
    [100%]   61.88 MB @	 361.0 GB/s
==> Extracting bison-3.8.2-acgafisyw76drj4jv7w4olghftso3xfl from binary cache
==> bison: Successfully installed bison-3.8.2-acgafisyw76drj4jv7w4olghftso3xfl
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.12s.  Extract: 0.08s.  Relocate: 0.01s.  Total: 0.13s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/bison-3.8.2-acgafisyw76drj4jv7w4olghftso3xfl
==> Installing bison-3.8.2-acgafisyw76drj4jv7w4olghftso3xfl [42/66]
==> Fetching file:///mirror/blobs/sha256/02/029feea9156acadd3b85fd2943644e58e2181af989e7a64b9d7ba6bd8a301c1e
    [100%]   20.61 MB @	 401.3 GB/s
==> Extracting perl-5.40.0-txcbw3fccp6lkmksnwz7n4fnisten2i7 from binary cache
==> perl: Successfully installed perl-5.40.0-txcbw3fccp6lkmksnwz7n4fnisten2i7
  Search: 0.00s.  Fetch: 0.04s.	 Install: 1.02s.  Extract: 0.89s.  Relocate: 0.10s.  Total: 1.06s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/perl-5.40.0-txcbw3fccp6lkmksnwz7n4fnisten2i7
==> Installing perl-5.40.0-txcbw3fccp6lkmksnwz7n4fnisten2i7 [43/66]
==> Fetching file:///mirror/blobs/sha256/62/621da2e18fc82b44bf6408100f0cb285121cf7f6c80ad2b4037bc340e3d2ab1d
    [100%]   12.50 MB @	 351.3 GB/s
==> Extracting gettext-0.23.1-goyejxw4nhahj4kwk6dwcyjdsxcwtj44 from binary cache
==> gettext: Successfully installed gettext-0.23.1-goyejxw4nhahj4kwk6dwcyjdsxcwtj44
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.58s.  Extract: 0.52s.  Relocate: 0.03s.  Total: 0.61s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gettext-0.23.1-goyejxw4nhahj4kwk6dwcyjdsxcwtj44
==> Installing gettext-0.23.1-goyejxw4nhahj4kwk6dwcyjdsxcwtj44 [44/66]
==> Fetching file:///mirror/blobs/sha256/f4/f42a9bdbeb309bcbea102a0474a4ec7c05368572d2b015d4497f7086506e57b1
    [100%]   61.03 MB @	 401.0 GB/s
==> Extracting autoconf-2.72-krgi7jfdrslsejycs7pbkgb5v5c22che from binary cache
==> autoconf: Successfully installed autoconf-2.72-krgi7jfdrslsejycs7pbkgb5v5c22che
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.04s.  Relocate: 0.01s.  Total: 0.09s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/autoconf-2.72-krgi7jfdrslsejycs7pbkgb5v5c22che
==> Installing autoconf-2.72-krgi7jfdrslsejycs7pbkgb5v5c22che [45/66]
==> Fetching file:///mirror/blobs/sha256/61/619762a6e6879f5943b033a5cc4656663b724cb21caf15aca3fa81a57c456ae7
    [100%]  728.31 KB @	 870.1 MB/s
==> Extracting libxcrypt-4.4.38-qhnn6wrmssrbawoi6jhixtgtxmmegvwn from binary cache
==> libxcrypt: Successfully installed libxcrypt-4.4.38-qhnn6wrmssrbawoi6jhixtgtxmmegvwn
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.06s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.07s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libxcrypt-4.4.38-qhnn6wrmssrbawoi6jhixtgtxmmegvwn
==> Installing libxcrypt-4.4.38-qhnn6wrmssrbawoi6jhixtgtxmmegvwn [46/66]
==> Fetching file:///mirror/blobs/sha256/78/7816c6f8d8e9e0711885b450744e0e9ca7a800c9c0bb3efd374375eccfe7c9b4
    [100%]   67.43 MB @	 411.2 GB/s
==> Extracting openssl-3.4.1-osxtmvl4zgmziookkpdcpl2zw23gx4vm from binary cache
==> openssl: Successfully installed openssl-3.4.1-osxtmvl4zgmziookkpdcpl2zw23gx4vm
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.33s.  Extract: 0.17s.  Relocate: 0.02s.  Total: 0.34s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/openssl-3.4.1-osxtmvl4zgmziookkpdcpl2zw23gx4vm
==> Installing openssl-3.4.1-osxtmvl4zgmziookkpdcpl2zw23gx4vm [47/66]
==> Fetching file:///mirror/blobs/sha256/1b/1bb0c6a8b329b1e7bd30ccf651aae5560c8083280a5eaee282d96f1da638c730
    [100%]   61.44 MB @	 995.6 MB/s
==> Extracting findutils-4.10.0-vhsfohm5zwq422kcebw7hxmaxj3i333x from binary cache
==> findutils: Successfully installed findutils-4.10.0-vhsfohm5zwq422kcebw7hxmaxj3i333x
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.05s.  Relocate: 0.01s.  Total: 0.10s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/findutils-4.10.0-vhsfohm5zwq422kcebw7hxmaxj3i333x
==> Installing findutils-4.10.0-vhsfohm5zwq422kcebw7hxmaxj3i333x [48/66]
==> Fetching file:///mirror/blobs/sha256/7c/7cab32b7eebddde961214f0c2f2ac589cebf8327435d2dcf52ab8fe02c52a481
    [100%]  708.05 KB @	 792.6 MB/s
==> Extracting automake-1.16.5-jgeyxwqrov44oiau7ovffiatce35jaip from binary cache
==> automake: Successfully installed automake-1.16.5-jgeyxwqrov44oiau7ovffiatce35jaip
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.04s.  Relocate: 0.02s.  Total: 0.09s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/automake-1.16.5-jgeyxwqrov44oiau7ovffiatce35jaip
==> Installing automake-1.16.5-jgeyxwqrov44oiau7ovffiatce35jaip [49/66]
==> Fetching file:///mirror/blobs/sha256/a7/a7b32f92f99e15f3eb67dbdb955e77fb40c4cfc8c05a70eaee279aa57dfe4070
    [100%]   61.15 MB @	 888.8 MB/s
==> Extracting curl-8.11.1-etcxzzkvyw2nujigrhexfmpdtj6mkntf from binary cache
==> curl: Successfully installed curl-8.11.1-etcxzzkvyw2nujigrhexfmpdtj6mkntf
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.03s.  Relocate: 0.01s.  Total: 0.08s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/curl-8.11.1-etcxzzkvyw2nujigrhexfmpdtj6mkntf
==> Installing curl-8.11.1-etcxzzkvyw2nujigrhexfmpdtj6mkntf [50/66]
==> Fetching file:///mirror/blobs/sha256/19/19856700f1ba114fe900a16a7e438aba58b4220d5ad9b4882958f7deb7ff99dc
    [100%]   62.78 MB @	 371.2 GB/s
==> Extracting libevent-2.1.12-alldwq4ypkbua57tzrpoyai66dzu2ulj from binary cache
==> libevent: Successfully installed libevent-2.1.12-alldwq4ypkbua57tzrpoyai66dzu2ulj
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.11s.  Extract: 0.07s.  Relocate: 0.01s.  Total: 0.12s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libevent-2.1.12-alldwq4ypkbua57tzrpoyai66dzu2ulj
==> Installing libevent-2.1.12-alldwq4ypkbua57tzrpoyai66dzu2ulj [51/66]
==> Fetching file:///mirror/blobs/sha256/85/85a073ed6758ff753251ccaa8844599cb6e2b1bedefafb915575b4892c59ea62
    [100%]   90.46 MB @	 381.3 GB/s
==> Extracting python-3.13.5-musqpuq2h6hqu2tfvvomowyefd7mr3pm from binary cache
==> python: Successfully installed python-3.13.5-musqpuq2h6hqu2tfvvomowyefd7mr3pm
  Search: 0.00s.  Fetch: 0.14s.	 Install: 3.16s.  Extract: 2.92s.  Relocate: 0.12s.  Total: 3.31s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/python-3.13.5-musqpuq2h6hqu2tfvvomowyefd7mr3pm
==> Installing python-3.13.5-musqpuq2h6hqu2tfvvomowyefd7mr3pm [52/66]
==> Fetching file:///mirror/blobs/sha256/c5/c5344ebdde64a92e38444a5a03d2ca41561fb3367359a72b09de0fd5fac877a5
    [100%]  798.18 KB @	 835.3 MB/s
==> Extracting libtool-2.4.7-7z4rzcr3e72nkdhlc22aaswac23hinwh from binary cache
==> libtool: Successfully installed libtool-2.4.7-7z4rzcr3e72nkdhlc22aaswac23hinwh
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.09s.  Extract: 0.03s.  Relocate: 0.03s.  Total: 0.10s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libtool-2.4.7-7z4rzcr3e72nkdhlc22aaswac23hinwh
==> Installing libtool-2.4.7-7z4rzcr3e72nkdhlc22aaswac23hinwh [53/66]
==> Fetching file:///mirror/blobs/sha256/fe/fec202c15eae4d7826cde5c975ced008763d5ce67467de4de36a87ff6be29c2d
    [100%]   61.92 MB @	 411.1 GB/s
==> Extracting krb5-1.21.3-nm4trzqdawtrp52qm223s2zf66uyq74u from binary cache
==> krb5: Successfully installed krb5-1.21.3-nm4trzqdawtrp52qm223s2zf66uyq74u
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.14s.  Extract: 0.07s.  Relocate: 0.03s.  Total: 0.15s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/krb5-1.21.3-nm4trzqdawtrp52qm223s2zf66uyq74u
==> Installing krb5-1.21.3-nm4trzqdawtrp52qm223s2zf66uyq74u [54/66]
==> Fetching file:///mirror/blobs/sha256/0e/0ee8519b961a15a3f2431915acbc41c63d90b7b7fd09baad641bf03ee2201f9d
    [100%]   29.27 MB @	 471.3 GB/s
==> Extracting cmake-3.31.8-yxjigtdrlwn7qk6jitoyzdhbb2fcimfo from binary cache
==> cmake: Successfully installed cmake-3.31.8-yxjigtdrlwn7qk6jitoyzdhbb2fcimfo
  Search: 0.00s.  Fetch: 0.05s.	 Install: 1.17s.  Extract: 1.07s.  Relocate: 0.07s.  Total: 1.22s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/cmake-3.31.8-yxjigtdrlwn7qk6jitoyzdhbb2fcimfo
==> Installing cmake-3.31.8-yxjigtdrlwn7qk6jitoyzdhbb2fcimfo [55/66]
==> Fetching file:///mirror/blobs/sha256/6d/6dc4d1f4243f793eeab5e7da2ee075b864bf4d620f2c6987482397f3cf5a149d
    [100%]   10.52 MB @	 341.3 GB/s
==> Extracting pmix-5.0.5-wqd5fdu5gl7v5aki4et6wxw6fjc6mozw from binary cache
==> pmix: Successfully installed pmix-5.0.5-wqd5fdu5gl7v5aki4et6wxw6fjc6mozw
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.25s.  Extract: 0.19s.  Relocate: 0.02s.  Total: 0.27s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/pmix-5.0.5-wqd5fdu5gl7v5aki4et6wxw6fjc6mozw
==> Installing pmix-5.0.5-wqd5fdu5gl7v5aki4et6wxw6fjc6mozw [56/66]
==> Fetching file:///mirror/blobs/sha256/ab/ab39d5c2d8089b58c3dba234e4c5606cfd85717e85537009f83baa13b5a5c71a
    [100%]   24.86 MB @	 351.3 GB/s
==> Extracting yaksa-0.3-5r6tuoocxu2ry6dtl3pzh2j7icz4vgcj from binary cache
==> yaksa: Successfully installed yaksa-0.3-5r6tuoocxu2ry6dtl3pzh2j7icz4vgcj
  Search: 0.00s.  Fetch: 0.04s.	 Install: 0.61s.  Extract: 0.53s.  Relocate: 0.04s.  Total: 0.65s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/yaksa-0.3-5r6tuoocxu2ry6dtl3pzh2j7icz4vgcj
==> Installing yaksa-0.3-5r6tuoocxu2ry6dtl3pzh2j7icz4vgcj [57/66]
==> Fetching file:///mirror/blobs/sha256/33/330ffae1c03231f1e55163b16131b72483a2f93d1fb123bae610434963bc311c
    [100%]  386.03 KB @	 683.2 MB/s
==> Extracting numactl-2.0.18-naapp6hxkqoqcige2ghwgwru7a7sse4e from binary cache
==> numactl: Successfully installed numactl-2.0.18-naapp6hxkqoqcige2ghwgwru7a7sse4e
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.06s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/numactl-2.0.18-naapp6hxkqoqcige2ghwgwru7a7sse4e
==> Installing numactl-2.0.18-naapp6hxkqoqcige2ghwgwru7a7sse4e [58/66]
==> Fetching file:///mirror/blobs/sha256/56/568ebd3d07471af14f223eced047663de6b6999af7bb1caa7f259c9c7d8ddbb6
    [100%]   63.05 MB @	 501.2 GB/s
==> Extracting openssh-9.9p1-sgecnstwdqcrjkkrd44cz55b6tf4px3q from binary cache
==> openssh: Successfully installed openssh-9.9p1-sgecnstwdqcrjkkrd44cz55b6tf4px3q
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.25s.  Extract: 0.07s.  Relocate: 0.14s.  Total: 0.26s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/openssh-9.9p1-sgecnstwdqcrjkkrd44cz55b6tf4px3q
==> Installing openssh-9.9p1-sgecnstwdqcrjkkrd44cz55b6tf4px3q [59/66]
==> Fetching file:///mirror/blobs/sha256/21/21c7bf967c42fa027f7c2ddb07a60fc1551126b465164d3c4652b12a2f626fda
    [100%]   69.74 MB @	 381.3 GB/s
==> Extracting netlib-lapack-3.12.1-xx23tbfq7nbajble5zjiwvfpfhestp75 from binary cache
==> netlib-lapack: Successfully installed netlib-lapack-3.12.1-xx23tbfq7nbajble5zjiwvfpfhestp75
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.29s.  Extract: 0.23s.  Relocate: 0.03s.  Total: 0.31s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/netlib-lapack-3.12.1-xx23tbfq7nbajble5zjiwvfpfhestp75
==> Installing netlib-lapack-3.12.1-xx23tbfq7nbajble5zjiwvfpfhestp75 [60/66]
==> Fetching file:///mirror/blobs/sha256/d7/d7713fa744cbdbde6bf848263182c0486a9cf1afc9655a41e2201c270349c3bd
    [100%]   14.86 MB @	 391.3 GB/s
==> Extracting mpich-4.3.0-3bsqfeynbfweluzzmownj52kyu42frut from binary cache
==> mpich: Successfully installed mpich-4.3.0-3bsqfeynbfweluzzmownj52kyu42frut
  Search: 0.00s.  Fetch: 0.03s.	 Install: 0.95s.  Extract: 0.88s.  Relocate: 0.03s.  Total: 0.98s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/mpich-4.3.0-3bsqfeynbfweluzzmownj52kyu42frut
==> Installing mpich-4.3.0-3bsqfeynbfweluzzmownj52kyu42frut [61/66]
==> Fetching file:///mirror/blobs/sha256/df/df845749068438591eceb3dfd07271eb88e309dfb066f22a6b641fa7d37b9d6a
    [100%]   18.46 MB @	 381.3 GB/s
==> Extracting openmpi-5.0.8-qdunbuirtg2hfcpu75xsiopuiygv3ecp from binary cache
==> openmpi: Successfully installed openmpi-5.0.8-qdunbuirtg2hfcpu75xsiopuiygv3ecp
  Search: 0.00s.  Fetch: 0.03s.	 Install: 0.81s.  Extract: 0.74s.  Relocate: 0.03s.  Total: 0.85s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/openmpi-5.0.8-qdunbuirtg2hfcpu75xsiopuiygv3ecp
==> Installing openmpi-5.0.8-qdunbuirtg2hfcpu75xsiopuiygv3ecp [62/66]
==> Fetching file:///mirror/blobs/sha256/a5/a5348fe74d0eee33f0a005a8ee49f04a1b66d9883256c4664094dd4a80a5413c
    [100%]   62.67 MB @	 351.1 GB/s
==> Extracting netlib-scalapack-2.2.2-tl5f6ke6ovh4e3tir4qaegg6evzeo3em from binary cache
==> netlib-scalapack: Successfully installed netlib-scalapack-2.2.2-tl5f6ke6ovh4e3tir4qaegg6evzeo3em
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.23s.  Extract: 0.06s.  Relocate: 0.02s.  Total: 0.24s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/netlib-scalapack-2.2.2-tl5f6ke6ovh4e3tir4qaegg6evzeo3em
==> Installing netlib-scalapack-2.2.2-tl5f6ke6ovh4e3tir4qaegg6evzeo3em [63/66]
==> Fetching file:///mirror/blobs/sha256/5f/5fd1f33eca598c6332a74732c7dfa08c0cbb17acb4835cf5c07aa97043b8afc3
    [100%]   62.67 MB @	 622.0 MB/s
==> Extracting netlib-scalapack-2.2.2-igwbxq4vt5vbickvvyq5cm3edkxkfqja from binary cache
==> netlib-scalapack: Successfully installed netlib-scalapack-2.2.2-igwbxq4vt5vbickvvyq5cm3edkxkfqja
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.13s.  Extract: 0.06s.  Relocate: 0.02s.  Total: 0.14s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/netlib-scalapack-2.2.2-igwbxq4vt5vbickvvyq5cm3edkxkfqja
==> Installing netlib-scalapack-2.2.2-igwbxq4vt5vbickvvyq5cm3edkxkfqja [64/66]
==> Fetching file:///mirror/blobs/sha256/57/57526f6c84a37ee047e5f5f900e1f3e872915686a3d32932afaf09f40cc18965
    [100%]   62.67 MB @	 461.1 GB/s
==> Extracting netlib-scalapack-2.2.2-hk7ftpc2qcu4w27abni3ofvuofo7rf6i from binary cache
==> netlib-scalapack: Successfully installed netlib-scalapack-2.2.2-hk7ftpc2qcu4w27abni3ofvuofo7rf6i
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.13s.  Extract: 0.06s.  Relocate: 0.02s.  Total: 0.14s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/netlib-scalapack-2.2.2-hk7ftpc2qcu4w27abni3ofvuofo7rf6i
==> Installing netlib-scalapack-2.2.2-hk7ftpc2qcu4w27abni3ofvuofo7rf6i [65/66]
==> Fetching file:///mirror/blobs/sha256/12/123a749ac493ac2bf61dcfb91a9495a61af8e6ab6a61e84c01dffa7d18090fcf
    [100%]   62.67 MB @	 471.2 GB/s
==> Extracting netlib-scalapack-2.2.2-nimjflq6w6bgxbomoydbhgabzn5rrjnw from binary cache
==> netlib-scalapack: Successfully installed netlib-scalapack-2.2.2-nimjflq6w6bgxbomoydbhgabzn5rrjnw
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.13s.  Extract: 0.06s.  Relocate: 0.02s.  Total: 0.14s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/netlib-scalapack-2.2.2-nimjflq6w6bgxbomoydbhgabzn5rrjnw
==> Installing netlib-scalapack-2.2.2-nimjflq6w6bgxbomoydbhgabzn5rrjnw [66/66]

Let’s double check which specs we have installed:

$ spack find
==> In environment /home/spack/stacks
==> 5 root specs
-- no arch / no compilers ---------------------------------------
[+] gcc@12	      [+] netlib-scalapack  [+] netlib-scalapack
[+] netlib-scalapack  [+] netlib-scalapack

-- linux-ubuntu22.04-x86_64_v3 / %c,cxx,fortran=gcc@12.5.0 ------
mpich@4.3.0  openblas@0.3.29  openmpi@5.0.8

-- linux-ubuntu22.04-x86_64_v3 / %c,cxx=gcc@11.4.0 --------------
berkeley-db@18.1.40  gettext@0.23.1  m4@1.4.20	  texinfo@7.1	 zstd@1.5.7
gcc@12.5.0	     gmp@6.3.0	     ncurses@6.5  zlib-ng@2.2.4

-- linux-ubuntu22.04-x86_64_v3 / %c,cxx=gcc@12.5.0 --------------
berkeley-db@18.1.40  expat@2.7.1     libffi@3.4.8    openssh@9.9p1  zstd@1.5.7
bison@3.8.2	     gettext@0.23.1  m4@1.4.20	     openssl@3.4.1
cmake@3.31.8	     hwloc@2.11.1    ncurses@6.5     python@3.13.5
curl@8.11.1	     krb5@1.21.3     nghttp2@1.65.0  zlib-ng@2.2.4

-- linux-ubuntu22.04-x86_64_v3 / %c,fortran=gcc@12.5.0 ----------
netlib-lapack@3.12.1	netlib-scalapack@2.2.2	netlib-scalapack@2.2.2
netlib-scalapack@2.2.2	netlib-scalapack@2.2.2

-- linux-ubuntu22.04-x86_64_v3 / %c=gcc@11.4.0 ------------------
automake@1.16.5	  gawk@5.3.1	 libsigsegv@2.14  mpfr@4.2.1	 readline@8.2
bzip2@1.0.8	  gdbm@1.23	 libtool@2.4.7	  perl@5.40.0	 tar@1.35
diffutils@3.10	  gmake@4.4.1	 libxml2@2.13.5	  pigz@2.8	 xz@5.6.3
findutils@4.10.0  libiconv@1.18	 mpc@1.3.1	  pkgconf@2.3.0

-- linux-ubuntu22.04-x86_64_v3 / %c=gcc@12.5.0 ------------------
automake@1.16.5	  libedit@3.1-20240808	libtool@2.4.7	  pmix@5.0.5
bzip2@1.0.8	  libevent@2.1.12	libxcrypt@4.4.38  readline@8.2
diffutils@3.10	  libfabric@2.2.0	libxml2@2.13.5	  sqlite@3.46.0
findutils@4.10.0  libiconv@1.18		numactl@2.0.18	  tar@1.35
gdbm@1.23	  libmd@1.1.0		perl@5.40.0	  util-linux-uuid@2.41
gmake@4.4.1	  libpciaccess@0.17	pigz@2.8	  xz@5.6.3
libbsd@0.12.2	  libsigsegv@2.14	pkgconf@2.3.0	  yaksa@0.3

-- linux-ubuntu22.04-x86_64_v3 / no compilers -------------------
autoconf@2.72			    compiler-wrapper@1.0  glibc@2.35
autoconf@2.72			    gcc@11.4.0		  util-macros@1.20.1
autoconf-archive@2023.02.20	    gcc-runtime@11.4.0
ca-certificates-mozilla@2025-05-20  gcc-runtime@12.5.0
==> 91 installed packages
==> 0 concretized packages to be installed (show with `spack find -c`)

As we can see we have our four variations of netlib-scalapack installed.

Reusable definitions

So far, we have seen how we can use spec matrices to generate cross-product specs from rows containing a list of constraints. A common situation you will encounter with large deployments is the necessity to add multiple matrices to the list of specs, that possibly share some of those rows.

To reduce the amount of duplication needed in the manifest file, and thus the maintenance burden for people maintaining it, Spack allows to define lists of constraints under the definitions attribute, and expand them later when needed. Let’s rewrite our manifest accordingly:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [ openmpi, mpich ]
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]

  view: false
  concretizer:
    unify: false

Check that re-concretizing won’t change the environment:

$ spack concretize
==> No new specs to concretize.
$ spack find -l
==> In environment /home/spack/stacks
==> 5 root specs
-- no arch / no compilers ---------------------------------------
[+] fmxnit5 gcc@12	      [+] tl5f6ke netlib-scalapack  [+] nimjflq netlib-scalapack
[+] igwbxq4 netlib-scalapack  [+] hk7ftpc netlib-scalapack

-- linux-ubuntu22.04-x86_64_v3 / %c,cxx,fortran=gcc@12.5.0 ------
3bsqfey mpich@4.3.0  z3ynokz openblas@0.3.29  qdunbui openmpi@5.0.8

-- linux-ubuntu22.04-x86_64_v3 / %c,cxx=gcc@11.4.0 --------------
k5bepow berkeley-db@18.1.40  zx3x4nm gmp@6.3.0	  z47kihj texinfo@7.1
fmxnit5 gcc@12.5.0	     ypq2dye m4@1.4.20	  g7ixtqy zlib-ng@2.2.4
soasbhm gettext@0.23.1	     u7sn2jt ncurses@6.5  bmaw6fn zstd@1.5.7

-- linux-ubuntu22.04-x86_64_v3 / %c,cxx=gcc@12.5.0 --------------
ikx4n4c berkeley-db@18.1.40  3e4kpzp hwloc@2.11.1    sgecnst openssh@9.9p1
acgafis bison@3.8.2	     nm4trzq krb5@1.21.3     osxtmvl openssl@3.4.1
yxjigtd cmake@3.31.8	     ddom6iu libffi@3.4.8    musqpuq python@3.13.5
etcxzzk curl@8.11.1	     rpphs4i m4@1.4.20	     6xddq3y zlib-ng@2.2.4
lfwqiuw expat@2.7.1	     56rxnkf ncurses@6.5     2jhx5gu zstd@1.5.7
goyejxw gettext@0.23.1	     2v5o3ra nghttp2@1.65.0

-- linux-ubuntu22.04-x86_64_v3 / %c,fortran=gcc@12.5.0 ----------
xx23tbf netlib-lapack@3.12.1	hk7ftpc netlib-scalapack@2.2.2
igwbxq4 netlib-scalapack@2.2.2	nimjflq netlib-scalapack@2.2.2
tl5f6ke netlib-scalapack@2.2.2

-- linux-ubuntu22.04-x86_64_v3 / %c=gcc@11.4.0 ------------------
vorbe53 automake@1.16.5	  kgw5nks libiconv@1.18	   prsofdp pigz@2.8
vy2ggzh bzip2@1.0.8	  vefpsal libsigsegv@2.14  xoghzad pkgconf@2.3.0
pptkbjb diffutils@3.10	  l2jtwyu libtool@2.4.7	   skysn5p readline@8.2
ga3qiej findutils@4.10.0  7mdlpgu libxml2@2.13.5   ye5hcpf tar@1.35
jxq6uva gawk@5.3.1	  mkqx7bu mpc@1.3.1	   zeynpnj xz@5.6.3
5yo63cn gdbm@1.23	  abjn4ye mpfr@4.2.1
5vb6e4b gmake@4.4.1	  nsqyer7 perl@5.40.0

-- linux-ubuntu22.04-x86_64_v3 / %c=gcc@12.5.0 ------------------
jgeyxwq automake@1.16.5	      m2sseei libiconv@1.18	 qz3jy42 pkgconf@2.3.0
3kql5yq bzip2@1.0.8	      bqguhkf libmd@1.1.0	 wqd5fdu pmix@5.0.5
6lfrh36 diffutils@3.10	      j336pb5 libpciaccess@0.17	 lxvqq5o readline@8.2
vhsfohm findutils@4.10.0      ffctxur libsigsegv@2.14	 brglstm sqlite@3.46.0
cxbuzxk gdbm@1.23	      7z4rzcr libtool@2.4.7	 4do6jx7 tar@1.35
xvxobm5 gmake@4.4.1	      qhnn6wr libxcrypt@4.4.38	 v2nef6x util-linux-uuid@2.41
es3e2le libbsd@0.12.2	      f2wxsch libxml2@2.13.5	 3kvqody xz@5.6.3
wspzmod libedit@3.1-20240808  naapp6h numactl@2.0.18	 5r6tuoo yaksa@0.3
alldwq4 libevent@2.1.12	      txcbw3f perl@5.40.0
atjn3er libfabric@2.2.0	      zia4xj2 pigz@2.8

-- linux-ubuntu22.04-x86_64_v3 / no compilers -------------------
5xgjgae autoconf@2.72			    j7ykyvm gcc@11.4.0
krgi7jf autoconf@2.72			    3c5xh7d gcc-runtime@11.4.0
u24n5ok autoconf-archive@2023.02.20	    zubbt4y gcc-runtime@12.5.0
s3bxsw4 ca-certificates-mozilla@2025-05-20  epwvtix glibc@2.35
rwnqbli compiler-wrapper@1.0		    kbleacp util-macros@1.20.1
==> 91 installed packages
==> 0 concretized packages to be installed (show with `spack find -c`)

Now we can use those definitions to add e.g. serial packages built against the LAPACK libraries. Let’s try to do that by using py-scipy as an example:

Another useful ability is excluding specific entries from a cross-product matrix. We can do that with the exclude keyword, in the same item as the matrix. Try to remove py-scipy ^netlib-lapack from our matrix:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [ openmpi, mpich ]
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
      - "py-scipy ^netlib-lapack"
  view: false
  concretizer:
    unify: false

Concretize the environment and install the specs again:

$ spack concretize
==> Starting concretization
==> Concretized 1 spec:
 -   qbiffx7  py-scipy@1.16.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3 %c,cxx,fortran=gcc@12.5.0
[+]  rwnqbli	  ^compiler-wrapper@1.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  fmxnit5	  ^gcc@12.5.0~binutils+bootstrap~graphite~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  pptkbjb	      ^diffutils@3.10 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  kgw5nks		  ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  jxq6uva	      ^gawk@5.3.1~nls build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  vefpsal		  ^libsigsegv@2.14 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  skysn5p		  ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[e]  j7ykyvm	      ^gcc@11.4.0~binutils+bootstrap~graphite~nvptx~piclibs~profiled~strip build_system=autotools build_type=RelWithDebInfo languages:='c,c++,fortran' arch=linux-ubuntu22.04-x86_64_v3
[+]  3c5xh7d	      ^gcc-runtime@11.4.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  5vb6e4b	      ^gmake@4.4.1~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  zx3x4nm	      ^gmp@6.3.0+cxx build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  5xgjgae		  ^autoconf@2.72 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  vorbe53		  ^automake@1.16.5 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ypq2dye		  ^m4@1.4.20+sigsegv build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  l2jtwyu	      ^libtool@2.4.7 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ga3qiej		  ^findutils@4.10.0 build_system=autotools patches:=440b954 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  mkqx7bu	      ^mpc@1.3.1 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  abjn4ye	      ^mpfr@4.2.1 build_system=autotools libs:=shared,static patches:=3ec29a6 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  u24n5ok		  ^autoconf-archive@2023.02.20 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  nsqyer7	      ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  k5bepow		  ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  vy2ggzh		  ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  5yo63cn		  ^gdbm@1.23 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  z47kihj	      ^texinfo@7.1 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  soasbhm		  ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  7mdlpgu		      ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  ye5hcpf		      ^tar@1.35 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  prsofdp			  ^pigz@2.8 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  zeynpnj		      ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  u7sn2jt		  ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  xoghzad		      ^pkgconf@2.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@11.4.0
[+]  g7ixtqy	      ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  bmaw6fn	      ^zstd@1.5.7+programs build_system=makefile compression:=none libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@11.4.0
[+]  zubbt4y	  ^gcc-runtime@12.5.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[e]  epwvtix	  ^glibc@2.35 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+]  z3ynokz	  ^openblas@0.3.29~bignuma~consistent_fpcsr+dynamic_dispatch+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-ubuntu22.04-x86_64_v3 %c,cxx,fortran=gcc@12.5.0
[+]  xvxobm5	      ^gmake@4.4.1~guile build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  qz3jy42	  ^pkgconf@2.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   33ypsy5	  ^py-cython@3.0.12 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   hjettfg	      ^py-setuptools@80.9.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   bi3kygb	  ^py-meson-python@0.16.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
 -   tqc2ju3	      ^meson@1.7.0 build_system=python_pip patches:=0f0b1bd arch=linux-ubuntu22.04-x86_64_v3
 -   pn35lwa	      ^ninja@1.12.1+re2c build_system=generic patches:=93f4bb3 arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   cjnodvg		  ^re2c@3.1 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   4edf37k	      ^py-packaging@25.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   bw467tx		  ^py-flit-core@3.12.0 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   vlo72hf	      ^py-pyproject-metadata@0.9.1 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   7cmuxbw	  ^py-numpy@2.3.1 build_system=python_pip patches:=873745d arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   nne5hgs	  ^py-pip@25.1.1 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
 -   lgys3sh	  ^py-pybind11@2.13.6+ipo build_system=cmake build_type=Release generator=ninja arch=linux-ubuntu22.04-x86_64_v3 %cxx=gcc@12.5.0
[+]  yxjigtd	      ^cmake@3.31.8~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  etcxzzk		  ^curl@8.11.1~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs:=shared,static tls:=openssl arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  2v5o3ra		      ^nghttp2@1.65.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   y2j3wh4	  ^py-pythran@0.16.1 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3 %cxx=gcc@12.5.0
 -   yjbwas7	      ^py-beniget@0.4.1 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   xwt2et7	      ^py-gast@0.5.4 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   cg53hr3	      ^py-ply@3.11 build_system=python_pip arch=linux-ubuntu22.04-x86_64_v3
 -   wjtuyuo	  ^py-wheel@0.45.1 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  musqpuq	  ^python@3.13.5+bz2+ctypes+dbm~debug+libxml2+lzma~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  3kql5yq	      ^bzip2@1.0.8~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  6lfrh36		  ^diffutils@3.10 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  lfwqiuw	      ^expat@2.7.1+libbsd build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  es3e2le		  ^libbsd@0.12.2 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  bqguhkf		      ^libmd@1.1.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  cxbuzxk	      ^gdbm@1.23 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  goyejxw	      ^gettext@0.23.1+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  m2sseei		  ^libiconv@1.18 build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  f2wxsch		  ^libxml2@2.13.5~http+pic~python+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  4do6jx7		  ^tar@1.35 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  zia4xj2		      ^pigz@2.8 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  2jhx5gu		      ^zstd@1.5.7+programs build_system=makefile compression:=none libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  ddom6iu	      ^libffi@3.4.8 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  56rxnkf	      ^ncurses@6.5~symlinks+termlib abi=none build_system=autotools patches:=7a351bc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  osxtmvl	      ^openssl@3.4.1~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  s3bxsw4		  ^ca-certificates-mozilla@2025-05-20 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+]  txcbw3f		  ^perl@5.40.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  ikx4n4c		      ^berkeley-db@18.1.40+cxx~docs+stl build_system=autotools patches:=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
[+]  lxvqq5o	      ^readline@8.2 build_system=autotools patches:=1ea4349,24f587b,3d9885e,5911a5b,622ba38,6c8adf8,758e2ec,79572ee,a177edc,bbf97f1,c7b45ff,e0013d9,e065038 arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  brglstm	      ^sqlite@3.46.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  v2nef6x	      ^util-linux-uuid@2.41 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  3kvqody	      ^xz@5.6.3~pic build_system=autotools libs:=shared,static arch=linux-ubuntu22.04-x86_64_v3 %c=gcc@12.5.0
[+]  6xddq3y	      ^zlib-ng@2.2.4+compat+new_strategies+opt+pic+shared build_system=autotools arch=linux-ubuntu22.04-x86_64_v3 %c,cxx=gcc@12.5.0
 -   xmpetx7	  ^python-venv@1.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3

$ spack install
[+] /usr (external glibc-2.35-epwvtixhoxdeufu6kbbrpaigv2p6iav4)
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/compiler-wrapper-1.0-rwnqblirwn5aenlmsbvsieo7mzw672ee
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gcc-runtime-11.4.0-3c5xh7dqudxmyorlsrsfmjyggqk57vnr
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gcc-runtime-12.5.0-zubbt4yxn35hmaarhmzluk6cb6gkujee
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/zstd-1.5.7-bmaw6fn2i5mbbfujq6kf7bomodh4iudq
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gmp-6.3.0-zx3x4nm3djw5jdqxcsimfpxykoxdijhj
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/zlib-ng-2.2.4-g7ixtqygrfip5uzt526b565dqxs6ze3x
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/bzip2-1.0.8-3kql5yq6ucjcp4ltisvme37ltccgn6yt
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/pkgconf-2.3.0-qz3jy42ylzeesnhmg3cw735reuumm35f
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libxcrypt-4.4.38-qhnn6wrmssrbawoi6jhixtgtxmmegvwn
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libffi-3.4.8-ddom6iure3h5olpwkvhm2xokpbebdb4m
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libmd-1.1.0-bqguhkf6f3br7tzqgv5srgsaqy7v2vb4
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/openblas-0.3.29-z3ynokzpyv4lr3hh6qrcc2y2ucrpmt26
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/zlib-ng-2.2.4-6xddq3ypcya5hm72cccnuct4kgbquf3c
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libiconv-1.18-m2sseeiaralmgv67wmrzua2dohic4pdq
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libfabric-2.2.0-atjn3er45pppotm6ksncj4rftg4bo7q2
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gmake-4.4.1-xvxobm5cq3cv3tykatbuyj5veywnyayy
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/mpfr-4.2.1-abjn4yec4cwckmwvwwtg45keufsqyhya
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/openssl-3.4.1-osxtmvl4zgmziookkpdcpl2zw23gx4vm
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libbsd-0.12.2-es3e2le2oexbmt3j4ik6rphojgz63u2a
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/ncurses-6.5-56rxnkfzlb5k3dceq52it64acuui3syl
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/pigz-2.8-zia4xj2iduhdjieipgnta3vrquz5eprh
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/util-linux-uuid-2.41-v2nef6xnzn36kard3s336ix4lnkb2n3l
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/nghttp2-1.65.0-2v5o3rauqqx5v5fytm7ogfimuzrvhomt
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/yaksa-0.3-5r6tuoocxu2ry6dtl3pzh2j7icz4vgcj
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/xz-5.6.3-3kvqodyyfedqojoooz7zowiqqxaqgclz
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/numactl-2.0.18-naapp6hxkqoqcige2ghwgwru7a7sse4e
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/zstd-1.5.7-2jhx5guzifg27gtix5cjlcqm6fqdfs2b
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/netlib-lapack-3.12.1-xx23tbfq7nbajble5zjiwvfpfhestp75
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libpciaccess-0.17-j336pb5rrnbu5btdjgjxxcbrwfc3iffa
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/mpc-1.3.1-mkqx7budcw2rpbdzo63hsoxhhszx366g
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libevent-2.1.12-alldwq4ypkbua57tzrpoyai66dzu2ulj
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/expat-2.7.1-lfwqiuwkitd5mxyk52v633u6soghu4cv
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libedit-3.1-20240808-wspzmodxgcbagfmhnn3ezs3cjcpfo7if
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/readline-8.2-lxvqq5ood4lil2pjv2ofawfytyal4n37
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/curl-8.11.1-etcxzzkvyw2nujigrhexfmpdtj6mkntf
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/libxml2-2.13.5-f2wxschdrknghuvhaqo7ldb3na3s7bl5
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/tar-1.35-4do6jx74chitcnc4sw3l4gdxtno44z2a
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gcc-12.5.0-fmxnit5ggu45byub6wlsuienth2dkboh
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/cmake-3.31.8-yxjigtdrlwn7qk6jitoyzdhbb2fcimfo
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/hwloc-2.11.1-3e4kpzphqrlylwwtfn6qw4mevnzoiwzm
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/sqlite-3.46.0-brglstmfyna5l3gv7v7pcldkfzx5xvgo
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gdbm-1.23-cxbuzxk4lqvsqfdneevy232tdn25p62d
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/gettext-0.23.1-goyejxw4nhahj4kwk6dwcyjdsxcwtj44
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/mpich-4.3.0-3bsqfeynbfweluzzmownj52kyu42frut
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/pmix-5.0.5-wqd5fdu5gl7v5aki4et6wxw6fjc6mozw
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/krb5-1.21.3-nm4trzqdawtrp52qm223s2zf66uyq74u
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/python-3.13.5-musqpuq2h6hqu2tfvvomowyefd7mr3pm
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/netlib-scalapack-2.2.2-tl5f6ke6ovh4e3tir4qaegg6evzeo3em
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/netlib-scalapack-2.2.2-igwbxq4vt5vbickvvyq5cm3edkxkfqja
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/openssh-9.9p1-sgecnstwdqcrjkkrd44cz55b6tf4px3q
==> Fetching file:///mirror/blobs/sha256/36/36c785e5317d3cb8a2f5abe5d1385e71eae8ab7ab5b18c96643b5ed9dbfe4af7
    [100%]   18.66 KB @	 110.4 MB/s
==> Extracting python-venv-1.0-xmpetx7fszahnro6w43n3e2uhcqrtgv7 from binary cache
==> python-venv: Successfully installed python-venv-1.0-xmpetx7fszahnro6w43n3e2uhcqrtgv7
  Search: 0.00s.  Fetch: 0.22s.	 Install: 0.09s.  Extract: 0.04s.  Relocate: 0.01s.  Total: 0.31s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/python-venv-1.0-xmpetx7fszahnro6w43n3e2uhcqrtgv7
==> Installing python-venv-1.0-xmpetx7fszahnro6w43n3e2uhcqrtgv7 [52/73]
==> Fetching file:///mirror/blobs/sha256/5d/5d8f08e52c23f9320f312c6debbf5097fccb831994e1b4e03bc596364bb26281
    [100%]   28.09 MB @	 401.3 GB/s
==> Extracting re2c-3.1-cjnodvgptrmmuu5jg646wo3dslnaejn6 from binary cache
==> re2c: Successfully installed re2c-3.1-cjnodvgptrmmuu5jg646wo3dslnaejn6
  Search: 0.00s.  Fetch: 0.05s.	 Install: 0.64s.  Extract: 0.54s.  Relocate: 0.06s.  Total: 0.69s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/re2c-3.1-cjnodvgptrmmuu5jg646wo3dslnaejn6
==> Installing re2c-3.1-cjnodvgptrmmuu5jg646wo3dslnaejn6 [53/73]
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/openmpi-5.0.8-qdunbuirtg2hfcpu75xsiopuiygv3ecp
==> Fetching file:///mirror/blobs/sha256/ac/ac51304d9175c51b4ae4e946436d87b7506e8be8dc6b0bf3950cc578403c7642
    [100%]   63.97 MB @	 381.2 GB/s
==> Extracting py-pip-25.1.1-nne5hgsvomsk6wmsq4kiwpumufmkx5lc from binary cache
==> py-pip: Successfully installed py-pip-25.1.1-nne5hgsvomsk6wmsq4kiwpumufmkx5lc
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.26s.  Extract: 0.22s.  Relocate: 0.01s.  Total: 0.27s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-pip-25.1.1-nne5hgsvomsk6wmsq4kiwpumufmkx5lc
==> Installing py-pip-25.1.1-nne5hgsvomsk6wmsq4kiwpumufmkx5lc [55/73]
==> Fetching file:///mirror/blobs/sha256/a4/a4188297e7701a73ba456b92530748718139cbc0b8118a4cb76803a28d0fc1bf
    [100%]   62.86 MB @	 351.2 GB/s
==> Extracting ninja-1.12.1-pn35lwakeu3u6kczajxjaw5e4hxwquca from binary cache
==> ninja: Successfully installed ninja-1.12.1-pn35lwakeu3u6kczajxjaw5e4hxwquca
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.11s.  Extract: 0.06s.  Relocate: 0.01s.  Total: 0.12s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/ninja-1.12.1-pn35lwakeu3u6kczajxjaw5e4hxwquca
==> Installing ninja-1.12.1-pn35lwakeu3u6kczajxjaw5e4hxwquca [56/73]
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/netlib-scalapack-2.2.2-nimjflq6w6bgxbomoydbhgabzn5rrjnw
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/netlib-scalapack-2.2.2-hk7ftpc2qcu4w27abni3ofvuofo7rf6i
==> Fetching file:///mirror/blobs/sha256/44/44b7fcb729c325178165fcb03c56b4910db9ec2816858c82723fb2112232f008
    [100%]  191.59 KB @	 498.2 MB/s
==> Extracting py-wheel-0.45.1-wjtuyuom3i3wi4bl63yh4fbmazyvz2cb from binary cache
==> py-wheel: Successfully installed py-wheel-0.45.1-wjtuyuom3i3wi4bl63yh4fbmazyvz2cb
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.06s.  Extract: 0.02s.  Relocate: 0.01s.  Total: 0.07s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-wheel-0.45.1-wjtuyuom3i3wi4bl63yh4fbmazyvz2cb
==> Installing py-wheel-0.45.1-wjtuyuom3i3wi4bl63yh4fbmazyvz2cb [59/73]
==> Fetching file:///mirror/blobs/sha256/f4/f42a28dc69ba689661245617bdb43dd9a75569dcccaaf74433e19b0238551fcc
    [100%]   62.88 MB @	 401.2 GB/s
==> Extracting py-setuptools-80.9.0-hjettfg7x2mh7626cbb7txlly74owm3k from binary cache
==> py-setuptools: Successfully installed py-setuptools-80.9.0-hjettfg7x2mh7626cbb7txlly74owm3k
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.33s.  Extract: 0.28s.  Relocate: 0.01s.  Total: 0.34s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-setuptools-80.9.0-hjettfg7x2mh7626cbb7txlly74owm3k
==> Installing py-setuptools-80.9.0-hjettfg7x2mh7626cbb7txlly74owm3k [60/73]
==> Fetching file:///mirror/blobs/sha256/d3/d32b9859b6419a2fcc64a34ae19edc083abf4d32d7e4a8fb67ee4209b6b7e744
    [100%]  134.17 KB @	 454.2 MB/s
==> Extracting py-flit-core-3.12.0-bw467tx7wan6o76t5r6lumueuvx6b3rm from binary cache
==> py-flit-core: Successfully installed py-flit-core-3.12.0-bw467tx7wan6o76t5r6lumueuvx6b3rm
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.06s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-flit-core-3.12.0-bw467tx7wan6o76t5r6lumueuvx6b3rm
==> Installing py-flit-core-3.12.0-bw467tx7wan6o76t5r6lumueuvx6b3rm [61/73]
==> Fetching file:///mirror/blobs/sha256/7e/7e8b378e9c91ef7624638a854fe186ce0e3959c0d1ff58be878da875859c2106
    [100%]   76.11 KB @	 348.9 MB/s
==> Extracting py-gast-0.5.4-xwt2et7ii2nccnpujxeb3pzxyjhswyil from binary cache
==> py-gast: Successfully installed py-gast-0.5.4-xwt2et7ii2nccnpujxeb3pzxyjhswyil
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-gast-0.5.4-xwt2et7ii2nccnpujxeb3pzxyjhswyil
==> Installing py-gast-0.5.4-xwt2et7ii2nccnpujxeb3pzxyjhswyil [62/73]
==> Fetching file:///mirror/blobs/sha256/dd/dd678ba0c3066350d4c8eb60620a2a8b4c368b93219a4fee1e20d6d64d3a60af
    [100%]   62.83 MB @	 441.2 GB/s
==> Extracting meson-1.7.0-tqc2ju3muw3lhadzvdcna6i2gu5n2g2l from binary cache
==> meson: Successfully installed meson-1.7.0-tqc2ju3muw3lhadzvdcna6i2gu5n2g2l
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.18s.  Extract: 0.14s.  Relocate: 0.01s.  Total: 0.20s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/meson-1.7.0-tqc2ju3muw3lhadzvdcna6i2gu5n2g2l
==> Installing meson-1.7.0-tqc2ju3muw3lhadzvdcna6i2gu5n2g2l [63/73]
==> Fetching file:///mirror/blobs/sha256/6e/6ecabcd544b13794493ab19429e3ded45ae2b1d4be9019abd4f5e6b7a76966a8
    [100%]   15.69 MB @	 391.4 GB/s
==> Extracting py-cython-3.0.12-33ypsy5oiphbw5g6g2jda43po33enf7e from binary cache
==> py-cython: Successfully installed py-cython-3.0.12-33ypsy5oiphbw5g6g2jda43po33enf7e
  Search: 0.00s.  Fetch: 0.03s.	 Install: 0.56s.  Extract: 0.37s.  Relocate: 0.06s.  Total: 0.59s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-cython-3.0.12-33ypsy5oiphbw5g6g2jda43po33enf7e
==> Installing py-cython-3.0.12-33ypsy5oiphbw5g6g2jda43po33enf7e [64/73]
==> Fetching file:///mirror/blobs/sha256/67/67c8264edf6d59ff678351b1e129a82bc0e8c18a6e6c8cae89f24fc1fa69fef9
    [100%]  467.85 KB @	 742.7 MB/s
==> Extracting py-pybind11-2.13.6-lgys3shnnrwncryp5n7a23bsiurq7hmk from binary cache
==> py-pybind11: Successfully installed py-pybind11-2.13.6-lgys3shnnrwncryp5n7a23bsiurq7hmk
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.08s.  Extract: 0.03s.  Relocate: 0.01s.  Total: 0.09s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-pybind11-2.13.6-lgys3shnnrwncryp5n7a23bsiurq7hmk
==> Installing py-pybind11-2.13.6-lgys3shnnrwncryp5n7a23bsiurq7hmk [65/73]
==> Fetching file:///mirror/blobs/sha256/5a/5ae910622799e9462d456a6dd74cc11b07542c208fb7d87c0b90eb7889bb1843
    [100%]  140.77 KB @	 459.1 MB/s
==> Extracting py-ply-3.11-cg53hr3fdshg5hbq2dj3e4zwiszy77zn from binary cache
==> py-ply: Successfully installed py-ply-3.11-cg53hr3fdshg5hbq2dj3e4zwiszy77zn
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-ply-3.11-cg53hr3fdshg5hbq2dj3e4zwiszy77zn
==> Installing py-ply-3.11-cg53hr3fdshg5hbq2dj3e4zwiszy77zn [66/73]
==> Fetching file:///mirror/blobs/sha256/9a/9accc411a31cbf0b860ad651e1cada74658cb1886b9031886a9604c749b64b6d
    [100%]  172.70 KB @	 476.9 MB/s
==> Extracting py-packaging-25.0-4edf37ky2o6x4v2wjabd5ykoalf7chue from binary cache
==> py-packaging: Successfully installed py-packaging-25.0-4edf37ky2o6x4v2wjabd5ykoalf7chue
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.06s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.07s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-packaging-25.0-4edf37ky2o6x4v2wjabd5ykoalf7chue
==> Installing py-packaging-25.0-4edf37ky2o6x4v2wjabd5ykoalf7chue [67/73]
==> Fetching file:///mirror/blobs/sha256/99/995c1ec66d82bcf69b61e49b8349a677c3012e5372dff3c5afdabd9292952746
    [100%]   46.55 KB @	 221.6 MB/s
==> Extracting py-beniget-0.4.1-yjbwas7zwz7lvlaswx6tabxea3xax4wl from binary cache
==> py-beniget: Successfully installed py-beniget-0.4.1-yjbwas7zwz7lvlaswx6tabxea3xax4wl
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-beniget-0.4.1-yjbwas7zwz7lvlaswx6tabxea3xax4wl
==> Installing py-beniget-0.4.1-yjbwas7zwz7lvlaswx6tabxea3xax4wl [68/73]
==> Fetching file:///mirror/blobs/sha256/70/70799d510f685c508d56559fe904c5cec4d678db8ad57570e60513832febe82b
    [100%]   59.66 KB @	 253.5 MB/s
==> Extracting py-pyproject-metadata-0.9.1-vlo72hfjof5xxm55f7vfiwn6f4c77kyo from binary cache
==> py-pyproject-metadata: Successfully installed py-pyproject-metadata-0.9.1-vlo72hfjof5xxm55f7vfiwn6f4c77kyo
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.05s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.06s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-pyproject-metadata-0.9.1-vlo72hfjof5xxm55f7vfiwn6f4c77kyo
==> Installing py-pyproject-metadata-0.9.1-vlo72hfjof5xxm55f7vfiwn6f4c77kyo [69/73]
==> Fetching file:///mirror/blobs/sha256/4a/4a1f585a4513d07fe50b9a85146c9e829dd38f47fbc49b34a18acb3c357dde54
    [100%]   89.95 KB @	 371.0 MB/s
==> Extracting py-meson-python-0.16.0-bi3kygbxsoqpgsj7eysvot2stk5ewakg from binary cache
==> py-meson-python: Successfully installed py-meson-python-0.16.0-bi3kygbxsoqpgsj7eysvot2stk5ewakg
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.17s.  Extract: 0.01s.  Relocate: 0.01s.  Total: 0.18s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-meson-python-0.16.0-bi3kygbxsoqpgsj7eysvot2stk5ewakg
==> Installing py-meson-python-0.16.0-bi3kygbxsoqpgsj7eysvot2stk5ewakg [70/73]
==> Fetching file:///mirror/blobs/sha256/c8/c8ec2267301bcffecb0f37cbf9986bb09d1de27913ad17fa937985689f992fc4
    [100%]   12.03 MB @	 431.3 GB/s
==> Extracting py-numpy-2.3.1-7cmuxbwfgnhrytyhnzkl5xtknv2d75c4 from binary cache
==> py-numpy: Successfully installed py-numpy-2.3.1-7cmuxbwfgnhrytyhnzkl5xtknv2d75c4
  Search: 0.00s.  Fetch: 0.02s.	 Install: 0.54s.  Extract: 0.46s.  Relocate: 0.04s.  Total: 0.57s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-numpy-2.3.1-7cmuxbwfgnhrytyhnzkl5xtknv2d75c4
==> Installing py-numpy-2.3.1-7cmuxbwfgnhrytyhnzkl5xtknv2d75c4 [71/73]
==> Fetching file:///mirror/blobs/sha256/25/2582a7ace4653dfafd3183cfad886b87c88bf7dae6f9e93713bbfc160067c493
    [100%]   63.33 MB @	 441.2 GB/s
==> Extracting py-pythran-0.16.1-y2j3wh4mr4vfjpyfgkyjx2iuactz7yjo from binary cache
==> py-pythran: Successfully installed py-pythran-0.16.1-y2j3wh4mr4vfjpyfgkyjx2iuactz7yjo
  Search: 0.00s.  Fetch: 0.01s.	 Install: 0.78s.  Extract: 0.72s.  Relocate: 0.01s.  Total: 0.79s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-pythran-0.16.1-y2j3wh4mr4vfjpyfgkyjx2iuactz7yjo
==> Installing py-pythran-0.16.1-y2j3wh4mr4vfjpyfgkyjx2iuactz7yjo [72/73]
==> Fetching file:///mirror/blobs/sha256/e9/e9fb21bb4bcefb7926f1d69cec776a6ed49a1f816664d0f953a2e48b1a6810cf
    [100%]   36.41 MB @	 341.3 GB/s
==> Extracting py-scipy-1.16.0-qbiffx7j4vjyousxdaq5zu67ffhcc4hq from binary cache
==> py-scipy: Successfully installed py-scipy-1.16.0-qbiffx7j4vjyousxdaq5zu67ffhcc4hq
  Search: 0.00s.  Fetch: 0.06s.	 Install: 1.26s.  Extract: 1.10s.  Relocate: 0.12s.  Total: 1.32s
[+] /home/spack/spack/opt/spack/linux-x86_64_v3/py-scipy-1.16.0-qbiffx7j4vjyousxdaq5zu67ffhcc4hq
==> Installing py-scipy-1.16.0-qbiffx7j4vjyousxdaq5zu67ffhcc4hq [73/73]

At this point, the environment contains only py-scipy ^openblas. Verify it:

$ spack find -ld py-scipy
==> In environment /home/spack/stacks
==> 6 root specs
-- no arch / no compilers ---------------------------------------
[+] fmxnit5 gcc@12
     -	------- gcc@11

[+] igwbxq4 netlib-scalapack
     -	------- gcc@12
     -	------- mpich
     -	------- netlib-lapack

[+] tl5f6ke netlib-scalapack
     -	------- gcc@12
     -	------- mpich
     -	------- openblas

[+] hk7ftpc netlib-scalapack
     -	------- gcc@12
     -	------- netlib-lapack
     -	------- openmpi

[+] nimjflq netlib-scalapack
     -	------- gcc@12
     -	------- openblas
     -	------- openmpi

[+] qbiffx7 py-scipy
     -	------- gcc@12
     -	------- openblas


-- linux-ubuntu22.04-x86_64_v3 / %c,cxx,fortran=gcc@12.5.0 ------
qbiffx7 py-scipy@1.16.0
rwnqbli	    compiler-wrapper@1.0
fmxnit5	    gcc@12.5.0
pptkbjb		diffutils@3.10
kgw5nks		    libiconv@1.18
jxq6uva		gawk@5.3.1
vefpsal		    libsigsegv@2.14
skysn5p		    readline@8.2
j7ykyvm		gcc@11.4.0
3c5xh7d		gcc-runtime@11.4.0
5vb6e4b		gmake@4.4.1
zx3x4nm		gmp@6.3.0
5xgjgae		    autoconf@2.72
vorbe53		    automake@1.16.5
ypq2dye		    m4@1.4.20
l2jtwyu		libtool@2.4.7
ga3qiej		    findutils@4.10.0
mkqx7bu		mpc@1.3.1
abjn4ye		mpfr@4.2.1
u24n5ok		    autoconf-archive@2023.02.20
nsqyer7		perl@5.40.0
k5bepow		    berkeley-db@18.1.40
vy2ggzh		    bzip2@1.0.8
5yo63cn		    gdbm@1.23
z47kihj		texinfo@7.1
soasbhm		    gettext@0.23.1
7mdlpgu			libxml2@2.13.5
ye5hcpf			tar@1.35
prsofdp			    pigz@2.8
zeynpnj			xz@5.6.3
u7sn2jt		    ncurses@6.5
xoghzad			pkgconf@2.3.0
g7ixtqy		zlib-ng@2.2.4
bmaw6fn		zstd@1.5.7
zubbt4y	    gcc-runtime@12.5.0
epwvtix	    glibc@2.35
z3ynokz	    openblas@0.3.29
xvxobm5		gmake@4.4.1
qz3jy42	    pkgconf@2.3.0
33ypsy5	    py-cython@3.0.12
hjettfg		py-setuptools@80.9.0
bi3kygb	    py-meson-python@0.16.0
tqc2ju3		meson@1.7.0
pn35lwa		ninja@1.12.1
cjnodvg		    re2c@3.1
4edf37k		py-packaging@25.0
bw467tx		    py-flit-core@3.12.0
vlo72hf		py-pyproject-metadata@0.9.1
7cmuxbw	    py-numpy@2.3.1
nne5hgs	    py-pip@25.1.1
lgys3sh	    py-pybind11@2.13.6
yxjigtd		cmake@3.31.8
etcxzzk		    curl@8.11.1
2v5o3ra			nghttp2@1.65.0
y2j3wh4	    py-pythran@0.16.1
yjbwas7		py-beniget@0.4.1
xwt2et7		py-gast@0.5.4
cg53hr3		py-ply@3.11
wjtuyuo	    py-wheel@0.45.1
musqpuq	    python@3.13.5
3kql5yq		bzip2@1.0.8
6lfrh36		    diffutils@3.10
lfwqiuw		expat@2.7.1
es3e2le		    libbsd@0.12.2
bqguhkf			libmd@1.1.0
cxbuzxk		gdbm@1.23
goyejxw		gettext@0.23.1
m2sseei		    libiconv@1.18
f2wxsch		    libxml2@2.13.5
4do6jx7		    tar@1.35
zia4xj2			pigz@2.8
2jhx5gu			zstd@1.5.7
ddom6iu		libffi@3.4.8
56rxnkf		ncurses@6.5
osxtmvl		openssl@3.4.1
s3bxsw4		    ca-certificates-mozilla@2025-05-20
txcbw3f		    perl@5.40.0
ikx4n4c			berkeley-db@18.1.40
lxvqq5o		readline@8.2
brglstm		sqlite@3.46.0
v2nef6x		util-linux-uuid@2.41
3kvqody		xz@5.6.3
6xddq3y		zlib-ng@2.2.4
xmpetx7	    python-venv@1.0

==> 1 installed package
==> 0 concretized packages to be installed (show with `spack find -c`)

Conditional definitions

Spec list definitions can also be conditioned on a when clause. The when clause is a python conditional that is evaluated in a restricted environment. The variables available in when clauses are:

variable name

value

platform

The spack platform name for this machine

os

The default spack os name and version string for this machine

target

The default spack target string for this machine

architecture

The default spack architecture string platform-os-target for this machine

arch

Alias for architecture

env

A dictionary representing the users environment variables

re

The python re module for regex

hostname

The hostname of this node

Suppose we want to limit usage to only mpich, unless the SPACK_STACK_USE_OPENMPI environment variable is set. To do so we could write the following spack.yaml:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [mpich]
    - mpis: [openmpi]
      when: 'env.get("SPACK_STACK_USE_OPENMPI", "") == "1"'
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
    - "py-scipy ^netlib-lapack"
  view: false
  concretizer:
    unify: false

Different definitions of lists with the same name are concatenated, so we can define our MPI list in one place unconditionally, and then conditionally append one or more values to it.

First, check what happens when we concretize and don’t set any environment variable:

$ spack concretize
==> No new specs to concretize.
$ spack find -cl netlib-scalapack
==> In environment /home/spack/stacks
==> 4 root specs
-- no arch / no compilers ---------------------------------------
[+] fmxnit5 gcc@12	      [+] tl5f6ke netlib-scalapack
[+] igwbxq4 netlib-scalapack  [+] qbiffx7 py-scipy

-- linux-ubuntu22.04-x86_64_v3 / %c,fortran=gcc@12.5.0 ----------
[+]  igwbxq4 netlib-scalapack@2.2.2  [+]  tl5f6ke netlib-scalapack@2.2.2
==> 2 installed packages
==> 0 concretized packages to be installed

As we expected we are only using mpich as an MPI provider. To get openmpi back we need to set the appropriate environment variable:

$ export SPACK_STACK_USE_OPENMPI=1
$ spack concretize
$ spack find -cl netlib-scalapack
==> In environment /home/spack/stacks
==> 6 root specs
-- no arch / no compilers ---------------------------------------
[+] fmxnit5 gcc@12	      [+] tl5f6ke netlib-scalapack  [+] nimjflq netlib-scalapack
[+] igwbxq4 netlib-scalapack  [+] hk7ftpc netlib-scalapack  [+] qbiffx7 py-scipy

-- linux-ubuntu22.04-x86_64_v3 / %c,fortran=gcc@12.5.0 ----------
[+]  igwbxq4 netlib-scalapack@2.2.2  [+]  hk7ftpc netlib-scalapack@2.2.2
[+]  tl5f6ke netlib-scalapack@2.2.2  [+]  nimjflq netlib-scalapack@2.2.2
==> 4 installed packages
==> 0 concretized packages to be installed

There is no need to install this time, since all the specs were still in the store.

Other useful features

Sometimes it might be useful to create a local source mirror for the specs installed in an environment. If the environment is active, this is as simple as:

$ spack mirror create --all -d ./stacks-mirror

This command fetches all the tarballs for the packages in the spack.lock file, and puts them in the directory passed as argument. Later you can move this mirror to e.g. an air-gapped machine and:

$ spack mirror add <name> <stacks-mirror>

to be able to re-build the specs from sources. Alternatively, to create a buildcache you can:

$ spack gpg create <name> <e-mail>
$ spack buildcache push ./mirror

Don’t forget to set an appropriate value for the padding of the install tree, see how to setup relocation in our documentation.

By default, Spack installs one package at a time, using the -j option where it can. The spack install command has an optional -p option for the number of packages to build in parallel. Note that each parallel package may consume the number of threads of the -j option. You can use node resources more optimally by creating a depfile, when the environment is active:

$ spack env depfile -o Makefile

The result is a makefile that starts multiple Spack instances, and the resources are shared through the GNU jobserver. More information of this feature can be found in our documentation. This might cut down your build time by a fair amount, if you build frequently from sources. Expect this feature to be streamlined in future versions of Spack.

Make the software stack easy to use

Now that the software stack has been installed, we need to focus on how it can be used by our customers. We’ll first see how we can configure views to project a subset of the specs we installed onto a filesystem folder with the usual Unix structure. Then we’ll have a similar discussion for module files. Which of the two approaches is better depends strongly on the use case at hand.

View descriptors

At the beginning, we configured Spack not to create a view for this stack because simple views won’t work with stacks. We’ve been concretizing multiple packages of the same name, and they would conflict if linked into the same view.

What we can do is create multiple views, using view descriptors. This would allows us to define which packages are linked into the view, and how. Edit our spack.yaml file again.

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [mpich]
    - mpis: [openmpi]
      when: 'env.get("SPACK_STACK_USE_OPENMPI", "") == "1"'
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
    - "py-scipy ^netlib-lapack"
  concretizer:
    unify: false
  view:
    default:
      root: views/default
      select: ['%gcc@12']
      exclude: [^mpich, ^netlib-lapack]
    full:
      root: views/full
      projections:
        ^mpi^lapack: '{name}/{name}-{version}-{^c.name}-{^c.version}-{^mpi.name}-{^mpi.version}-{^lapack.name}-{^lapack.version}'
        ^lapack: '{name}/{name}-{version}-{^c.name}-{^c.version}-{^lapack.name}-{^lapack.version}'
        '%c': '{name}/{name}-{version}-{^c.name}-{^c.version}'
        '%cxx': '{name}/{name}-{version}-{^cxx.name}-{^cxx.version}'
        '%fortran': '{name}/{name}-{version}-{^fortran.name}-{^fortran.version}'
        all: '{name}/{name}-{version}'

In the configuration above we created two views, named default and full. The default view consists of all the packages that are compiled with gcc@12, but do not depend on either mpich or netlib-lapack. As we can see, we can both include and exclude specs using constraints.

The full view contains a more complex projection, so to put each spec into an appropriate subdirectory, according to the first constraint that the spec matches. all is the default projection, and has always the lowest priority, independent of the order in which it appears. To avoid confusion, we advise always keeping it last in projections.

Concretize to regenerate the views, and check their structure:

$ spack concretize
==> No new specs to concretize.
==> Updating view at /home/spack/stacks/views/default
==> Updating view at /home/spack/stacks/views/full
$ ls ~/stacks/views/default
bin  etc  include  lib	lib64  libexec	man  pyvenv.cfg	 sbin  share  var
$ ls ~/stacks/views/default/lib
cmake				libmenu.so
gettext				libmenu.so.6
krb5				libmenu.so.6.5
libasan.so.8			libmenu_g.a
libasprintf.a			libmenuw.a
libasprintf.so			libmenuw.so
libasprintf.so.0		libmenuw.so.6
libasprintf.so.0.0.0		libmenuw.so.6.5
libatomic.so.1			libmenuw_g.a
libblas.so			libmpi.a
libblas.so.3			libmpi.so
libblas.so.3.12.0		libmpi.so.12
libbsd-ctor.a			libmpi.so.12.5.0
libbsd.a			libmpi.so.40
libbsd.so			libmpi.so.40.40.7
libbsd.so.0			libmpi_mpifh.so
libbsd.so.0.12.2		libmpi_mpifh.so.40
libbz2.a			libmpi_mpifh.so.40.40.1
libbz2.so			libmpi_usempi_ignore_tkr.so
libbz2.so.1			libmpi_usempi_ignore_tkr.so.40
libbz2.so.1.0			libmpi_usempi_ignore_tkr.so.40.40.1
libbz2.so.1.0.8			libmpi_usempif08.so
libcblas.so			libmpi_usempif08.so.40
libcblas.so.3			libmpi_usempif08.so.40.40.3
libcblas.so.3.12.0		libmpich.so
libcharset.a			libmpichcxx.so
libcharset.so			libmpichf90.so
libcharset.so.1			libmpicxx.a
libcharset.so.1.0.0		libmpicxx.so
libcom_err.so			libmpicxx.so.12
libcom_err.so.3			libmpicxx.so.12.5.0
libcom_err.so.3.0		libmpifort.a
libcrypt.a			libmpifort.so
libcrypt.so			libmpifort.so.12
libcrypt.so.2			libmpifort.so.12.5.0
libcrypt.so.2.0.0		libmpl.so
libcurses.so			libncurses++.a
libedit.a			libncurses++.so
libedit.so			libncurses++.so.6
libedit.so.0			libncurses++.so.6.5
libedit.so.0.0.74		libncurses++_g.a
libevent-2.1.so.7		libncurses++w.a
libevent-2.1.so.7.0.1		libncurses++w.so
libevent.a			libncurses++w.so.6
libevent.so			libncurses++w.so.6.5
libevent_core-2.1.so.7		libncurses++w_g.a
libevent_core-2.1.so.7.0.1	libncurses.a
libevent_core.a			libncurses.so
libevent_core.so		libncurses.so.6
libevent_extra-2.1.so.7		libncurses.so.6.5
libevent_extra-2.1.so.7.0.1	libncurses_g.a
libevent_extra.a		libncursesw.a
libevent_extra.so		libncursesw.so
libevent_openssl-2.1.so.7	libncursesw.so.6
libevent_openssl-2.1.so.7.0.1	libncursesw.so.6.5
libevent_openssl.a		libncursesw_g.a
libevent_openssl.so		libnuma.a
libevent_pthreads-2.1.so.7	libnuma.so
libevent_pthreads-2.1.so.7.0.1	libnuma.so.1
libevent_pthreads.a		libnuma.so.1.0.0
libevent_pthreads.so		libopa.so
libexpat.a			libopen-pal.so
libexpat.so			libopen-pal.so.80
libexpat.so.1			libopen-pal.so.80.0.5
libexpat.so.1.10.2		libopenblas-r0.3.29.a
libfabric.a			libopenblas-r0.3.29.so
libfabric.so			libopenblas.a
libfabric.so.1			libopenblas.so
libfabric.so.1.28.0		libopenblas.so.0
libfmpich.so			libpanel.a
libform.a			libpanel.so
libform.so			libpanel.so.6
libform.so.6			libpanel.so.6.5
libform.so.6.5			libpanel_g.a
libform_g.a			libpanelw.a
libformw.a			libpanelw.so
libformw.so			libpanelw.so.6
libformw.so.6			libpanelw.so.6.5
libformw.so.6.5			libpanelw_g.a
libformw_g.a			libpciaccess.a
libgcc_s.so.1			libpciaccess.so
libgdbm.a			libpciaccess.so.0
libgdbm.so			libpciaccess.so.0.11.1
libgdbm.so.6			libpmix.a
libgdbm.so.6.0.0		libpmix.so
libgdbm_compat.a		libpmix.so.2
libgdbm_compat.so		libpmix.so.2.13.5
libgdbm_compat.so.4		libprrte.so
libgdbm_compat.so.4.0.0		libprrte.so.3
libgettextlib-0.23.1.so		libprrte.so.3.0.11
libgettextlib.a			libpython3.13.so
libgettextlib.so		libpython3.13.so.1.0
libgettextpo.a			libpython3.so
libgettextpo.so			libquadmath.so.0
libgettextpo.so.0		libreadline.a
libgettextpo.so.0.5.12		libreadline.so
libgettextsrc-0.23.1.so		libreadline.so.8
libgettextsrc.a			libreadline.so.8.2
libgettextsrc.so		libscalapack.so
libgfortran.so.5		libscalapack.so.2.2
libgomp.so.1			libscalapack.so.2.2.1
libgssapi_krb5.so		libsqlite3.a
libgssapi_krb5.so.2		libsqlite3.so
libgssapi_krb5.so.2.2		libsqlite3.so.0
libgssrpc.so			libsqlite3.so.0.8.6
libgssrpc.so.4			libssp.so.0
libgssrpc.so.4.2		libstdc++.so.6
libhistory.a			libtextstyle.a
libhistory.so			libtextstyle.so
libhistory.so.8			libtextstyle.so.0
libhistory.so.8.2		libtextstyle.so.0.2.3
libhwloc.a			libtinfo.a
libhwloc.so			libtinfo.so
libhwloc.so.15			libtinfo.so.6
libhwloc.so.15.8.0		libtinfo.so.6.5
libiconv.a			libtinfo_g.a
libiconv.so			libtinfow.a
libiconv.so.2			libtinfow.so
libiconv.so.2.7.0		libtinfow.so.6
libintl.a			libtinfow.so.6.5
libintl.so			libtinfow_g.a
libintl.so.8			libtmglib.so
libintl.so.8.4.2		libtmglib.so.3
libitm.so.1			libtmglib.so.3.12.0
libk5crypto.so			libtsan.so.2
libk5crypto.so.3		libubsan.so.1
libk5crypto.so.3.1		libuuid.a
libkadm5clnt.so			libuuid.so
libkadm5clnt_mit.so		libuuid.so.1
libkadm5clnt_mit.so.12		libuuid.so.1.3.0
libkadm5clnt_mit.so.12.0	libverto.so
libkadm5srv.so			libverto.so.0
libkadm5srv_mit.so		libverto.so.0.0
libkadm5srv_mit.so.12		libxml2.so
libkadm5srv_mit.so.12.0		libxml2.so.2
libkdb5.so			libxml2.so.2.13.5
libkdb5.so.10			libyaksa.a
libkdb5.so.10.0			libyaksa.so
libkrad.so			libyaksa.so.0
libkrad.so.0			libyaksa.so.0.0.0
libkrad.so.0.0			libz.a
libkrb5.so			libz.so
libkrb5.so.3			libz.so.1
libkrb5.so.3.3			libz.so.1.3.1.zlib-ng
libkrb5support.so		libzstd.a
libkrb5support.so.0		libzstd.so
libkrb5support.so.0.1		libzstd.so.1
liblapack.so			libzstd.so.1.5.7
liblapack.so.3			mpi.mod
liblapack.so.3.12.0		mpi_ext.mod
liblapacke.so			mpi_f08.mod
liblapacke.so.3			mpi_f08_callbacks.mod
liblapacke.so.3.12.0		mpi_f08_ext.mod
liblsan.so.0			mpi_f08_interfaces.mod
liblzma.a			mpi_f08_interfaces_callbacks.mod
liblzma.so			mpi_f08_types.mod
liblzma.so.5			mpi_types.mod
liblzma.so.5.6.3		openmpi
libmd.a				pkgconfig
libmd.so			pmpi_f08_interfaces.mod
libmd.so.0			python3.13
libmd.so.0.1.0			terminfo
libmenu.a
$ ls ~/stacks/views/full
bzip2	     hwloc	libiconv      mpich		openssh	  tar
expat	     krb5	libmd	      ncurses		openssl	  util-linux-uuid
gcc	     libbsd	libpciaccess  netlib-lapack	pigz	  xz
gcc-runtime  libedit	libxcrypt     netlib-scalapack	pmix	  yaksa
gdbm	     libevent	libxml2	      numactl		python	  zlib-ng
gettext	     libfabric	mpc	      openblas		readline  zstd
gmp	     libffi	mpfr	      openmpi		sqlite
$ ls ~/stacks/views/full/gcc/
gcc-12.5.0-gcc-11.4.0
$ ls ~/stacks/views/full/gcc/gcc-12.5.0-gcc-11.4.0
bin  include  lib  lib64  libexec  share

The view descriptor also contains a link key. The default behavior, as we have seen, is to link all packages, including implicit link and run dependencies, into the view. If we set the option to “roots”, Spack links only the root packages into the view.

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [mpich]
    - mpis: [openmpi]
      when: 'env.get("SPACK_STACK_USE_OPENMPI", "") == "1"'
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
    - "py-scipy ^netlib-lapack"
  concretizer:
    unify: false
  view:
    default:
      root: views/default
      select: ['%gcc@12']
      exclude: [^mpich, ^netlib-lapack]
      link: roots
    full:
      root: views/full
      projections:
        ^mpi^lapack: '{name}/{name}-{version}-{^c.name}-{^c.version}-{^mpi.name}-{^mpi.version}-{^lapack.name}-{^lapack.version}'
        ^lapack: '{name}/{name}-{version}-{^c.name}-{^c.version}-{^lapack.name}-{^lapack.version}'
        '%c': '{name}/{name}-{version}-{^c.name}-{^c.version}'
        '%cxx': '{name}/{name}-{version}-{^cxx.name}-{^cxx.version}'
        '%fortran': '{name}/{name}-{version}-{^fortran.name}-{^fortran.version}'
        all: '{name}/{name}-{version}'
$ spack concretize
==> No new specs to concretize.
==> Updating view at /home/spack/stacks/views/default
$ ls ~/stacks/views/default
lib
$ ls ~/stacks/views/default/lib
cmake  libscalapack.so	libscalapack.so.2.2  libscalapack.so.2.2.1  pkgconfig  python3.13
$ ls ~/stacks/views/full
bzip2	     hwloc	libiconv      mpich		openssh	  tar
expat	     krb5	libmd	      ncurses		openssl	  util-linux-uuid
gcc	     libbsd	libpciaccess  netlib-lapack	pigz	  xz
gcc-runtime  libedit	libxcrypt     netlib-scalapack	pmix	  yaksa
gdbm	     libevent	libxml2	      numactl		python	  zlib-ng
gettext	     libfabric	mpc	      openblas		readline  zstd
gmp	     libffi	mpfr	      openmpi		sqlite

Now we see only the root libraries in the default view. The rest are hidden, but are still available in the full view. The complete documentation on views can be found here.

Module files

Module files are another very popular way to use software on HPC systems. In this section we’ll show how to configure and generate a hierarchical module structure, suitable for lmod.

A more in-depth tutorial, focused only on module files, can be found at Module Files Tutorial. There we discuss the general architecture of module file generation in Spack and we highlight differences between environment-modules and lmod that won’t be covered in this section.

Let’s start by adding lmod to the software installed with the system compiler:

$ spack add lmod%gcc@11
$ spack concretize
$ spack install

Once that is done, let’s add the module command to our shell like this:

$ . $(spack location -i lmod)/lmod/lmod/init/bash

If everything worked out correctly you should have the module command available in your shell:

$ module --version

Modules based on Lua: Version 8.7.18  2023-01-14 07:33 -06:00
    by Robert McLay mclay@tacc.utexas.edu

The next step is to add some basic configuration to our spack.yaml to generate module files:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [mpich]
    - mpis: [openmpi]
      when: 'env.get("SPACK_STACK_USE_OPENMPI", "") == "1"'
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
    - "py-scipy ^netlib-lapack"
  view: false
  concretizer:
    unify: false
  modules:
    default:
      enable:
      - lmod
      roots:
        lmod: modules
      lmod:
        hierarchy:
        - mpi
        - lapack
        core_specs:
        - '%gcc@11'

In these few lines of additional configuration we told Spack to generate lmod module files in a subdirectory named modules, using a hierarchy comprising both lapack and mpi. We’ve also configured it to place all specs built with our system compiler into the Core designation in the lmod hierarchy.

We can generate the module files and use them with the following commands:

$ spack module lmod refresh -y
$ module use $PWD/stacks/modules/linux-ubuntu22.04-x86_64/Core

Now we should be able to see the module files that have been generated:

$ module av

---- /home/spack/stacks/modules/linux-ubuntu22.04-x86_64/Core ----
   autoconf-archive/2023.02.20-u24n5ok
   automake/1.16.5-vorbe53
   bc/1.07.1-vbae3vx
   berkeley-db/18.1.40-k5bepow
   bzip2/1.0.8-vy2ggzh
   ca-certificates-mozilla/2025-05-20-s3bxsw4
   compiler-wrapper/1.0-rwnqbli
   curl/8.11.1-7xf5wle
   diffutils/3.10-pptkbjb
   ed/1.4-iktf6j2
   findutils/4.10.0-ga3qiej
   gawk/5.3.1-jxq6uva
   gcc-runtime/11.4.0-3c5xh7d
   gcc-runtime/12.5.0-zubbt4y		      (D)
   gcc/11.4.0-j7ykyvm
   gcc/12.5.0-fmxnit5			      (D)
   gdbm/1.23-5yo63cn
   gettext/0.23.1-soasbhm
   glibc/2.35-epwvtix
   gmake/4.4.1-5vb6e4b
   gmp/6.3.0-zx3x4nm
   libiconv/1.18-kgw5nks
   libsigsegv/2.14-vefpsal
   libtool/2.4.7-l2jtwyu
   libxcrypt/4.4.38-iundcaq
   libxml2/2.13.5-7mdlpgu
   lmod/8.7.18-gegst44
   lua-luafilesystem/1.8.0-clpxqjm
   lua-luaposix/36.1-xtcb4fb
   lua/5.4.6-cvp4vnc
   m4/1.4.20-ypq2dye
   mpc/1.3.1-mkqx7bu
   mpfr/4.2.1-abjn4ye
   ncurses/6.5-u7sn2jt
   nghttp2/1.65.0-hpkxq4q
   openssl/3.4.1-kphogwy
   perl/5.40.0-nsqyer7
   pigz/2.8-prsofdp
   pkgconf/2.3.0-xoghzad
   readline/8.2-skysn5p
   tar/1.35-ye5hcpf
   tcl/8.6.12-6wkjgx5
   texinfo/7.1-z47kihj
   unzip/6.0-hvxekp5
   util-macros/1.20.1-kbleacp
   xz/5.6.3-zeynpnj
   zlib-ng/2.2.4-g7ixtqy
   zstd/1.5.7-bmaw6fn

  Where:
   D:  Default Module

If the avail list is too long
consider trying:

"module --default avail" or
"ml -d av" to just list the
default modules.
"module overview" or "ml ov"
to display the number of
modules for each name.

Use "module spider" to find
all possible modules and
extensions.
Use "module keyword key1 key2
..." to search for all
possible modules matching any
of the "keys".

The set of modules is already usable, and the hierarchy already works. For instance, we can load the gcc compiler and check that we have gcc in out path and we have a lot of modules available - all the ones compiled with gcc@12:

$ module load gcc
$ which gcc
/home/spack/spack/opt/spack/linux-x86_64_v3/gcc-12.5.0-fmxnit5ggu45byub6wlsuienth2dkboh/bin/gcc
$ gcc --version
gcc (Spack GCC) 12.5.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ module av

---- /home/spack/stacks/modules/linux-ubuntu22.04-x86_64/gcc/12.5.0 ----
   autoconf/2.72-krgi7jf
   automake/1.16.5-jgeyxwq
   berkeley-db/18.1.40-ikx4n4c
   bison/3.8.2-acgafis
   bzip2/1.0.8-3kql5yq		       (D)
   cmake/3.31.8-yxjigtd
   curl/8.11.1-etcxzzk
   diffutils/3.10-6lfrh36	       (D)
   expat/2.7.1-lfwqiuw
   findutils/4.10.0-vhsfohm	       (D)
   gdbm/1.23-cxbuzxk
   gettext/0.23.1-goyejxw
   gmake/4.4.1-xvxobm5
   hwloc/2.11.1-3e4kpzp
   krb5/1.21.3-nm4trzq
   libbsd/0.12.2-es3e2le
   libedit/3.1-20240808-wspzmod
   libevent/2.1.12-alldwq4
   libfabric/2.2.0-atjn3er
   libffi/3.4.8-ddom6iu
   libiconv/1.18-m2sseei	       (D)
   libmd/1.1.0-bqguhkf
   libpciaccess/0.17-j336pb5
   libsigsegv/2.14-ffctxur
   libtool/2.4.7-7z4rzcr	       (D)
   libxcrypt/4.4.38-qhnn6wr	       (D)
   libxml2/2.13.5-f2wxsch
   m4/1.4.20-rpphs4i
   meson/1.7.0-tqc2ju3
   mpich/4.3.0-3bsqfey
   ncurses/6.5-56rxnkf		       (D)
   netlib-lapack/3.12.1-xx23tbf
   nghttp2/1.65.0-2v5o3ra	       (D)
   ninja/1.12.1-pn35lwa
   numactl/2.0.18-naapp6h
   openblas/0.3.29-z3ynokz
   openmpi/5.0.8-qdunbui
   openssh/9.9p1-sgecnst
   openssl/3.4.1-osxtmvl	       (D)
   perl/5.40.0-txcbw3f		       (D)
   pigz/2.8-zia4xj2		       (D)
   pkgconf/2.3.0-qz3jy42
   pmix/5.0.5-wqd5fdu
   py-beniget/0.4.1-yjbwas7
   py-cython/3.0.12-33ypsy5
   py-flit-core/3.12.0-bw467tx
   py-gast/0.5.4-xwt2et7
   py-meson-python/0.16.0-bi3kygb
   py-packaging/25.0-4edf37k
   py-pip/25.1.1-nne5hgs
   py-ply/3.11-cg53hr3
   py-pybind11/2.13.6-lgys3sh
   py-pyproject-metadata/0.9.1-vlo72hf
   py-setuptools/80.9.0-hjettfg
   py-wheel/0.45.1-wjtuyuo
   python-venv/1.0-xmpetx7
   python/3.13.5-musqpuq
   re2c/3.1-cjnodvg
   readline/8.2-lxvqq5o
   sqlite/3.46.0-brglstm
   tar/1.35-4do6jx7		       (D)
   util-linux-uuid/2.41-v2nef6x
   xz/5.6.3-3kvqody		       (D)
   yaksa/0.3-5r6tuoo
   zlib-ng/2.2.4-6xddq3y	       (D)
   zstd/1.5.7-2jhx5gu		       (D)

---- /home/spack/stacks/modules/linux-ubuntu22.04-x86_64/Core ----
   autoconf-archive/2023.02.20-u24n5ok
   automake/1.16.5-vorbe53		      (D)
   bc/1.07.1-vbae3vx
   berkeley-db/18.1.40-k5bepow		      (D)
   bzip2/1.0.8-vy2ggzh
   ca-certificates-mozilla/2025-05-20-s3bxsw4
   compiler-wrapper/1.0-rwnqbli
   curl/8.11.1-7xf5wle			      (D)
   diffutils/3.10-pptkbjb
   ed/1.4-iktf6j2
   findutils/4.10.0-ga3qiej
   gawk/5.3.1-jxq6uva
   gcc-runtime/11.4.0-3c5xh7d		      (L)
   gcc-runtime/12.5.0-zubbt4y		      (D)
   gcc/11.4.0-j7ykyvm
   gcc/12.5.0-fmxnit5			      (L,D)
   gdbm/1.23-5yo63cn			      (D)
   gettext/0.23.1-soasbhm		      (D)
   glibc/2.35-epwvtix			      (L)
   gmake/4.4.1-5vb6e4b			      (D)
   gmp/6.3.0-zx3x4nm			      (L)
   libiconv/1.18-kgw5nks
   libsigsegv/2.14-vefpsal		      (D)
   libtool/2.4.7-l2jtwyu
   libxcrypt/4.4.38-iundcaq
   libxml2/2.13.5-7mdlpgu		      (D)
   lmod/8.7.18-gegst44
   lua-luafilesystem/1.8.0-clpxqjm
   lua-luaposix/36.1-xtcb4fb
   lua/5.4.6-cvp4vnc
   m4/1.4.20-ypq2dye			      (D)
   mpc/1.3.1-mkqx7bu			      (L)
   mpfr/4.2.1-abjn4ye			      (L)
   ncurses/6.5-u7sn2jt
   nghttp2/1.65.0-hpkxq4q
   openssl/3.4.1-kphogwy
   perl/5.40.0-nsqyer7
   pigz/2.8-prsofdp
   pkgconf/2.3.0-xoghzad		      (D)
   readline/8.2-skysn5p			      (D)
   tar/1.35-ye5hcpf
   tcl/8.6.12-6wkjgx5
   texinfo/7.1-z47kihj
   unzip/6.0-hvxekp5
   util-macros/1.20.1-kbleacp
   xz/5.6.3-zeynpnj
   zlib-ng/2.2.4-g7ixtqy		      (L)
   zstd/1.5.7-bmaw6fn			      (L)

  Where:
   D:  Default Module
   L:  Module is loaded

If the avail list is too long
consider trying:

"module --default avail" or
"ml -d av" to just list the
default modules.
"module overview" or "ml ov"
to display the number of
modules for each name.

Use "module spider" to find
all possible modules and
extensions.
Use "module keyword key1 key2
..." to search for all
possible modules matching any
of the "keys".


$ module unload gcc

There are a few issues though. For instance, we have a lot of modules generated from dependencies of gcc that are cluttering the view, and won’t likely be needed directly by users. Then, module names contain hashes, which prevent users from being able to reuse the same script in similar, but not equal, environments.

Also, some of the modules might need to set custom environment variables, which are specific to the deployment aspects that are not part of the hash - for instance a policy at the deploying site.

To address all these needs we can complicate our modules configuration a bit more:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  definitions:
    - mpis: [mpich]
    - mpis: [openmpi]
      when: 'env.get("SPACK_STACK_USE_OPENMPI", "") == "1"'
    - lapacks: [ openblas, netlib-lapack ]
    - compilers: [ "gcc@12" ]
    - mpi_packages: [ netlib-scalapack ]
    - serial_packages: [py-scipy]

  specs:
  - gcc@12%gcc@11
  - matrix:
    - [$mpi_packages]
    - [$^mpis]
    - [$^lapacks]
    - [$%compilers]
  - matrix:
    - [$serial_packages]
    - [$^lapacks]
    - [$%compilers]
    exclude:
    - "py-scipy ^netlib-lapack"
  view: false
  concretizer:
    unify: false
  modules:
    default:
      enable:
      - lmod
      roots:
        lmod: modules
      lmod:
        hierarchy:
        - mpi
        - lapack
        core_specs:
        - '%gcc@11'
        hash_length: 0
        include:
        - gcc
        exclude:
        - '%gcc@11.4.0'
        all:
          environment:
            set:
              '{name}_ROOT': '{prefix}'
        openmpi:
          environment:
            set:
              SLURM_MPI_TYPE: pmi2
              OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
        projections:
          all: '{name}/{version}'

Regenerate the modules again:

$ spack module lmod refresh --delete-tree -y
==> Regenerating lmod module files

Now we have a set of module files without hashes, with a correct hierarchy, and with all our custom modifications:

$ module load gcc
$ module load openmpi openblas netlib-scalapack py-scipy
$ module av

---- /home/spack/stacks/modules/linux-ubuntu22.04-x86_64/gcc/12.5.0 ----
   autoconf/2.72
   automake/1.16.5
   berkeley-db/18.1.40
   bison/3.8.2
   bzip2/1.0.8
   cmake/3.31.8
   curl/8.11.1
   diffutils/3.10
   expat/2.7.1
   findutils/4.10.0
   gdbm/1.23
   gettext/0.23.1
   gmake/4.4.1
   hwloc/2.11.1
   krb5/1.21.3
   libbsd/0.12.2
   libedit/3.1-20240808
   libevent/2.1.12
   libfabric/2.2.0
   libffi/3.4.8
   libiconv/1.18
   libmd/1.1.0
   libpciaccess/0.17
   libsigsegv/2.14
   libtool/2.4.7
   libxcrypt/4.4.38
   libxml2/2.13.5
   m4/1.4.20
   meson/1.7.0
   mpich/4.3.0
   ncurses/6.5
   netlib-lapack/3.12.1
   nghttp2/1.65.0
   ninja/1.12.1
   numactl/2.0.18
   openblas/0.3.29
   openmpi/5.0.8
   openssh/9.9p1
   openssl/3.4.1
   perl/5.40.0
   pigz/2.8
   pkgconf/2.3.0
   pmix/5.0.5
   py-beniget/0.4.1
   py-cython/3.0.12
   py-flit-core/3.12.0
   py-gast/0.5.4
   py-meson-python/0.16.0
   py-packaging/25.0
   py-pip/25.1.1
   py-ply/3.11
   py-pybind11/2.13.6
   py-pyproject-metadata/0.9.1
   py-setuptools/80.9.0
   py-wheel/0.45.1
   python-venv/1.0
   python/3.13.5
   re2c/3.1
   readline/8.2
   sqlite/3.46.0
   tar/1.35
   util-linux-uuid/2.41
   xz/5.6.3
   yaksa/0.3
   zlib-ng/2.2.4
   zstd/1.5.7

---- /home/spack/stacks/modules/linux-ubuntu22.04-x86_64/Core ----
   autoconf-archive/2023.02.20
   ca-certificates-mozilla/2025-05-20
   compiler-wrapper/1.0
   gcc-runtime/12.5.0
   gcc/11.4.0
   gcc/12.5.0			      (L,D)
   glibc/2.35			      (L)
   util-macros/1.20.1

  Where:
   D:  Default Module
   L:  Module is loaded

If the avail list is too long
consider trying:

"module --default avail" or
"ml -d av" to just list the
default modules.
"module overview" or "ml ov"
to display the number of
modules for each name.

Use "module spider" to find
all possible modules and
extensions.
Use "module keyword key1 key2
..." to search for all
possible modules matching any
of the "keys".


$ module load mpich

Lmod is automatically
replacing "openmpi/5.0.8"
with "mpich/4.3.0".


Due to MODULEPATH changes, the following have been reloaded:
  1) netlib-scalapack/2.2.2

$ module load netlib-lapack

Lmod is automatically
replacing "openblas/0.3.29"
with "netlib-lapack/3.12.1".


Due to MODULEPATH changes, the following have been reloaded:
  1) netlib-scalapack/2.2.2

$ module purge

This concludes the quick tour of module file generation, and the tutorial on stacks.

Summary

In this tutorial, we configured Spack to install a stack of software built on a cross-product of different MPI and LAPACK libraries. We used the spec matrix syntax to express in a compact way the specs to be installed, and spec list definitions to reuse the same matrix rows in different places. Then, we discussed how to make the software easy to use, leveraging either filesystem views or module files.