Tutorial setup
If you have not done the prior sections, you’ll need to set Spack up like this:
git clone https://github.com/spack/spack
. spack/share/spack/setup-env.sh
spack tutorial
See the Basic Installation Tutorial for full details on setup. For more
help join us in the #tutorial channel on Slack – get an
invitation at spackpm.herokuapp.com
Environments Tutorial¶
We’ve covered how to install, remove, and list packages with Spack using the commands:
- spack install to install packages;
- spack uninstall to remove them; and
- spack find to look at and query what is installed.
This section of the tutorial introduces Spack Environments, which allow you to work with independent groups of packages separately. The goal is to provide users with virtual environments similar to those supported by other commonly used tools (e.g., Python venv) while allowing common installations to be seamlessly shared. They are also intended to be easily shared and re-used by others and across systems.
Administering properly configured software involving lots of packages
and/or varying configuration requirements (e.g., different implementations
of mpi) for multiple projects and efforts can be overwhelming. Spack
environments allow you to readily:
- establish standard software requirements for your project(s);
- set up run environments for users;
- support your usual development environment(s);
- set up packages for CI/CD;
- reproduce builds (approximately or exactly) on other machines; and
- much more.
This tutorial introduces the basics of creating and using environments, then explains how to expand, configure, and build software in them. We will start with the command line interface, then cover editing key environment file directly. We will describe the difference between Spack-managed and independent environments, then finish with a section on reproducible builds.
Environment Basics¶
Let’s look at the output of spack find at this point in the tutorial.
$ spack find
==> 59 installed packages
-- linux-ubuntu18.04-x86_64 / clang@7.0.0 -----------------------
zlib@1.2.12
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 gawk@5.1.1 hwloc@2.7.1 libpciaccess@0.16 ncurses@6.2 pkgconf@1.8.0 texinfo@6.5 zlib@1.2.12
automake@1.16.5 gcc@8.4.0 libbsd@0.11.5 libsigsegv@2.13 numactl@2.0.14 pmix@4.1.2 trilinos@13.0.1 zstd@1.5.2
berkeley-db@18.1.40 gdbm@1.19 libedit@3.1-20210216 libtool@2.4.7 openblas@0.3.20 python@3.9.12 util-linux-uuid@2.37.4
bzip2@1.0.8 gettext@0.21 libevent@2.1.12 libxml2@2.9.13 openmpi@4.1.3 readline@8.1 util-macros@1.19.3
cmake@3.23.1 gmp@6.2.1 libfabric@1.14.1 m4@1.4.19 openssh@9.0p1 sqlite@3.38.5 xz@5.2.5
diffutils@3.8 hdf5@1.12.2 libffi@3.4.2 mpc@1.1.0 openssl@1.1.1o tar@1.34 yaksa@0.2
expat@2.4.8 hdf5@1.12.2 libiconv@1.16 mpfr@3.1.6 perl@5.34.1 tcl@8.6.12 zlib@1.2.8
findutils@4.9.0 hdf5@1.12.2 libmd@1.0.4 mpich@4.0.2 pigz@2.7 tcl@8.6.12 zlib@1.2.8
This is a complete, but cluttered list of the installed packages and
their dependencies. It contains packages built with both openmpi
and mpich, as well as multiple variants of other packages, like
hdf5 and zlib. The query mechanism we learned about with
spack find can help, but it would be nice if we could start from
a clean slate without losing what we’ve already installed.
Creating and activating environments¶
The spack env command can help. Let’s create a new environment
called myproject:
$ spack env create myproject
==> Created environment 'myproject' in /home/spack/spack/var/spack/environments/myproject
==> You can activate this environment with:
==> spack env activate myproject
An environment is like a virtualized Spack instance that you can use to aggregate package installations for a project or other purpose. It has an associated view, which is a single prefix where all packages from the environment are linked.
You can see the environments we’ve created so far using the spack env
list command:
$ spack env list
==> 1 environments
myproject
Now let’s activate our environment. You can use spack env activate
command:
$ spack env activate myproject
You can also use the spacktivate alias for short.
Note
If you use the -p option for spack env activate, Spack
will prepend the environment name to the prompt. This is a handy
way to be reminded if and which environment you are in.
Once you activate an environment, spack find only shows what is
in the current environment. We just created this environment, so it
does not contain any installed packages.
$ spack find
==> In environment myproject
==> No root specs
==> 0 installed packages
The output from spack find is now slightly different. It tells
you that you’re in the myproject environment, so there is no need
to panic when you see that none of the previously installed packages
are available. It also states that there are no root specs. We’ll
get back to what that means later.
If you only want to check what environment you are in, you can use
spack env status:
$ spack env status
==> In environment myproject
If you want to leave this environment, you can use spack env deactivate
or the despacktivate alias for short.
After deactivating, we can see everything installed in this Spack instance:
$ despacktivate
$ spack env status
==> No active environment
$ spack find
==> 59 installed packages
-- linux-ubuntu18.04-x86_64 / clang@7.0.0 -----------------------
zlib@1.2.12
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 gawk@5.1.1 hwloc@2.7.1 libpciaccess@0.16 ncurses@6.2 pkgconf@1.8.0 texinfo@6.5 zlib@1.2.12
automake@1.16.5 gcc@8.4.0 libbsd@0.11.5 libsigsegv@2.13 numactl@2.0.14 pmix@4.1.2 trilinos@13.0.1 zstd@1.5.2
berkeley-db@18.1.40 gdbm@1.19 libedit@3.1-20210216 libtool@2.4.7 openblas@0.3.20 python@3.9.12 util-linux-uuid@2.37.4
bzip2@1.0.8 gettext@0.21 libevent@2.1.12 libxml2@2.9.13 openmpi@4.1.3 readline@8.1 util-macros@1.19.3
cmake@3.23.1 gmp@6.2.1 libfabric@1.14.1 m4@1.4.19 openssh@9.0p1 sqlite@3.38.5 xz@5.2.5
diffutils@3.8 hdf5@1.12.2 libffi@3.4.2 mpc@1.1.0 openssl@1.1.1o tar@1.34 yaksa@0.2
expat@2.4.8 hdf5@1.12.2 libiconv@1.16 mpfr@3.1.6 perl@5.34.1 tcl@8.6.12 zlib@1.2.8
findutils@4.9.0 hdf5@1.12.2 libmd@1.0.4 mpich@4.0.2 pigz@2.7 tcl@8.6.12 zlib@1.2.8
Notice that we are no longer in an environment and all our packages are still installed.
Installing packages¶
Now that we understand how creation and activation work, let’s go
back to myproject and install a couple of packages; specifically,
tcl and trilinos.
$ spack env activate myproject
$ spack install tcl
==> All of the packages are already installed
==> Updating view at /home/spack/spack/var/spack/environments/myproject/.spack-env/view
$ spack install trilinos
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/pkgconf-1.8.0-i5po27g2n6zrzxnhxdy2fnpaai7xsdaw
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/berkeley-db-18.1.40-q6tqmmmitkatscuadcwn6ea32iscs55k
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libiconv-1.16-dr56jt456uiebij2mkueldmejdxftigc
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/zlib-1.2.12-fntvsj6xevbz5gyq7kfa4xg7oxnaolxs
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libsigsegv-2.13-ubezeychw4iojr3tg6wbsnvp6s3s2qwd
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/util-macros-1.19.3-vkk73ww5p7g4u6lpnay5hcclsckcihtt
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/xz-5.2.5-olptpbsse74hpreilbfo4qmlqexihzye
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/ncurses-6.2-frwzsc4jrou7efe7huyzeownqfjvsrfd
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/diffutils-3.8-k22pe4rhr7yllf5pojl75veh5gkjdglq
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/m4-1.4.19-uadbn2a64h744jto2xqy6u2wmujcww6v
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libxml2-2.9.13-4run532u4c6iy5th5gkpgm26tr4vajc4
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/readline-8.1-ls6erxwsj2ubsfzt4x5mblj6nrab5jkp
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libedit-3.1-20210216-zc56lv6gyjpfvi4tpdtmx5yb7ljfnlrb
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/bzip2-1.0.8-cyn2yamiu26ofxwgxzxbioreuwigzl6f
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libtool-2.4.7-7fhdrxxizepvvkxrfaaizhzkwx2sjzbt
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gdbm-1.19-xub4jsy5pz5qjf6vooropf2shkigy5pm
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libpciaccess-0.16-csp5gvqmwcmqsd4al5iafotfk5fqh3uf
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/perl-5.34.1-pn5mteswujdkuong6mbqaush54rdas7c
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/hwloc-2.7.1-oixja6dnsnc4mydqswdya2nyt4rlcxpa
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/autoconf-2.69-hlj3ujmrsu5gs7uhfdotozzg6ipys45b
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/openblas-0.3.20-dydujpzz2yqemiiboxdl2qb5e73wqesm
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/openssl-1.1.1o-hqlqpsnqoifap6mlkbr5fyxoaq3gynbk
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/automake-1.16.5-6vk5ehdtjzgcvug6vuwh2mmkjhg2pn2z
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/cmake-3.23.1-tulqz4nxor764kgeij23tfap4ddgpgvh
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/openssh-9.0p1-3ez3d3jtc4f7bph5suuzqebujp3gppwv
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libevent-2.1.12-hlnkgmrdcycqf65n6vrvoraq4dhddagv
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/numactl-2.0.14-wfnw532l74emh6rwx4mrvot3huyiq53t
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/pmix-4.1.2-e4fax4744gqg222snd6da55c3xq3zu34
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/openmpi-4.1.3-jfxctqwar7wb65rn7wf5mot7m4jxmsey
==> Installing trilinos-13.0.1-6t2acz7flltuohbhadoelljcn6te2hm2
==> Fetching file:///mirror/build_cache/linux-ubuntu18.04-x86_64-gcc-7.5.0-trilinos-13.0.1-6t2acz7flltuohbhadoelljcn6te2hm2.spec.json.sig
==> Fetching file:///mirror/build_cache/linux-ubuntu18.04-x86_64/gcc-7.5.0/trilinos-13.0.1/linux-ubuntu18.04-x86_64-gcc-7.5.0-trilinos-13.0.1-6t2acz7flltuohbhadoelljcn6te2hm2.spack
==> Extracting trilinos-13.0.1-6t2acz7flltuohbhadoelljcn6te2hm2 from binary cache
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/trilinos-13.0.1-6t2acz7flltuohbhadoelljcn6te2hm2
==> Updating view at /home/spack/spack/var/spack/environments/myproject/.spack-env/view
We see that tcl and all of the dependencies of trilinos are
already installed. Notice also that our environment’s view gets updated.
Now confirm the contents of the environment using spack find:
$ spack find
==> In environment myproject
==> Root specs
tcl trilinos
==> 31 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 cmake@3.23.1 libedit@3.1-20210216 libsigsegv@2.13 ncurses@6.2 openssh@9.0p1 pmix@4.1.2 util-macros@1.19.3
automake@1.16.5 diffutils@3.8 libevent@2.1.12 libtool@2.4.7 numactl@2.0.14 openssl@1.1.1o readline@8.1 xz@5.2.5
berkeley-db@18.1.40 gdbm@1.19 libiconv@1.16 libxml2@2.9.13 openblas@0.3.20 perl@5.34.1 tcl@8.6.12 zlib@1.2.12
bzip2@1.0.8 hwloc@2.7.1 libpciaccess@0.16 m4@1.4.19 openmpi@4.1.3 pkgconf@1.8.0 trilinos@13.0.1
We now see that tcl and trilinos are root specs in
our environment. That is because we explicitly asked for them to
be installed, which makes them the roots of the combined graph
of all packages in the environment. The other installed packages
are present because they are dependencies of one or both of the roots.
Using packages¶
Environments provide a convenient way for using installed packages.
Running spack env activate gives you everything in the environment
on your PATH. Otherwise, you would need to use spack load
or module load
for each package in order to set up the environment for the package (and
its dependencies).
When you install packages into an environment, they are, by default,
linked into a single prefix, or view. Activating the environment
with spack env activate results in subdirectories from the view
being added to PATH, LD_LIBRARY_PATH, CMAKE_PREFIX_PATH,
and other environment variables. This makes the environment easier to use.
Let’s try it out. We just installed tcl into our myproject
environment. Tcl includes a shell-like application called tclsh.
You can see the path to tclsh using which:
$ which tclsh
/home/spack/spack/var/spack/environments/myproject/.spack-env/view/bin/tclsh
Notice its path includes the name of our environment and a view
subdirectory.
You can now run tclsh like you would any other program that is
in your path:
$ tclsh
% echo "hello world!"
hello world!
% exit
Uninstalling packages¶
We can uninstall packages from an environment without affecting other environments. This is possible since, while Spack shares common installations, environments only link to those installations.
Let’s demonstrate this feature by creating another environment.
Suppose myproject requires trilinos but we have another
project that has it installed but no longer requires it.
Start by creating a myproject2 environment with the installed
packages hdf5+hl and trilinos. To ensure we’re leveraging
the tutorial build cache, let’s explicitly constrain hdf5’s
mpi dependency to mpich.
$ spack env create myproject2
==> Created environment 'myproject2' in /home/spack/spack/var/spack/environments/myproject2
==> You can activate this environment with:
==> spack env activate myproject2
$ spack env activate myproject2
$ spack install hdf5+hl ^mpich
==> All of the packages are already installed
==> Updating view at /home/spack/spack/var/spack/environments/myproject2/.spack-env/view
$ spack install trilinos
==> All of the packages are already installed
==> Updating view at /home/spack/spack/var/spack/environments/myproject2/.spack-env/view
$ spack find
==> In environment myproject2
==> Root specs
hdf5 +hl trilinos
==> 46 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 expat@2.4.8 libbsd@0.11.5 libmd@1.0.4 mpich@4.0.2 openssl@1.1.1o readline@8.1 xz@5.2.5
automake@1.16.5 findutils@4.9.0 libedit@3.1-20210216 libpciaccess@0.16 ncurses@6.2 perl@5.34.1 sqlite@3.38.5 yaksa@0.2
berkeley-db@18.1.40 gdbm@1.19 libevent@2.1.12 libsigsegv@2.13 numactl@2.0.14 pigz@2.7 tar@1.34 zlib@1.2.12
bzip2@1.0.8 gettext@0.21 libfabric@1.14.1 libtool@2.4.7 openblas@0.3.20 pkgconf@1.8.0 trilinos@13.0.1 zstd@1.5.2
cmake@3.23.1 hdf5@1.12.2 libffi@3.4.2 libxml2@2.9.13 openmpi@4.1.3 pmix@4.1.2 util-linux-uuid@2.37.4
diffutils@3.8 hwloc@2.7.1 libiconv@1.16 m4@1.4.19 openssh@9.0p1 python@3.9.12 util-macros@1.19.3
Notice the root specs show the specs and variants as we asked for
them on the command line. In this case, hdf5 shows we enabled the
hl variant.
Now we have two environments. The myproject environment has tcl
and trilinos while the myproject2 environment has hdf5 +hl
and trilinos.
Now let’s uninstall trilinos from myproject2 and review the
contents of the environment:
$ spack uninstall trilinos
y
==> The following packages will be uninstalled:
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
6t2acz7 trilinos@13.0.1
==> Do you want to proceed? [y/N] ==> Updating view at /home/spack/spack/var/spack/environments/myproject2/.spack-env/view
$ spack find
==> In environment myproject2
==> Root specs
hdf5 +hl
==> 38 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 diffutils@3.8 hdf5@1.12.2 libiconv@1.16 libxml2@2.9.13 perl@5.34.1 sqlite@3.38.5 yaksa@0.2
automake@1.16.5 expat@2.4.8 hwloc@2.7.1 libmd@1.0.4 m4@1.4.19 pigz@2.7 tar@1.34 zlib@1.2.12
berkeley-db@18.1.40 findutils@4.9.0 libbsd@0.11.5 libpciaccess@0.16 mpich@4.0.2 pkgconf@1.8.0 util-linux-uuid@2.37.4 zstd@1.5.2
bzip2@1.0.8 gdbm@1.19 libfabric@1.14.1 libsigsegv@2.13 ncurses@6.2 python@3.9.12 util-macros@1.19.3
cmake@3.23.1 gettext@0.21 libffi@3.4.2 libtool@2.4.7 openssl@1.1.1o readline@8.1 xz@5.2.5
The result is that the environment now has only one root spec, hdf5
+hl, and contains fewer dependencies.
However, we know trilinos is still needed for the myproject
environment. So let’s switch back to confirm that it is still installed
in that environment.
$ spack env activate myproject
$ spack find
==> In environment myproject
==> Root specs
tcl trilinos
==> 31 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 cmake@3.23.1 libedit@3.1-20210216 libsigsegv@2.13 ncurses@6.2 openssh@9.0p1 pmix@4.1.2 util-macros@1.19.3
automake@1.16.5 diffutils@3.8 libevent@2.1.12 libtool@2.4.7 numactl@2.0.14 openssl@1.1.1o readline@8.1 xz@5.2.5
berkeley-db@18.1.40 gdbm@1.19 libiconv@1.16 libxml2@2.9.13 openblas@0.3.20 perl@5.34.1 tcl@8.6.12 zlib@1.2.12
bzip2@1.0.8 hwloc@2.7.1 libpciaccess@0.16 m4@1.4.19 openmpi@4.1.3 pkgconf@1.8.0 trilinos@13.0.1
Phew! We see that myproject still has trilinos as a root
spec. Spack uses reference counting to know that trilinos is
still installed for myproject.
Note
Trilinos would only have been uninstalled by Spack if it were no longer needed by any environments or their package dependencies.
Dealing with Many Specs at Once¶
So far we have used install and uninstall for processing
individual packages. Since environments define sets of packages,
their specs can be added to the environment before they are
installed together. Specs can be added at the command line or
entered directly in the environment configuration file.
Whole environments can be installed at once by adding specs to the environment before installing them. Individual packages can still be added and removed from the environment as it evolves.
There are a couple of advantages of processing all the specs of an environment at once. First, we don’t have to write a custom installation script outside of Spack. Second, we can launch a large build of many packages in parallel by taking advantage of Spack’s install-level build parallelism.
This section focuses on two ways to add specs to the environment before installing them.
Adding specs¶
Let’s start by adding a couple of specs to our myproject
environment:
$ spack env activate myproject
$ spack add hdf5+hl ^mpich
==> Adding hdf5+hl ^mpich to environment myproject
$ spack add gmp
==> Adding gmp to environment myproject
Now let’s take a look at what happened using spack find:
==> In environment myproject
==> Root specs
gmp hdf5 +hl tcl trilinos
==> 31 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 libedit@3.1-20210216 ncurses@6.2 pmix@4.1.2
automake@1.16.5 libevent@2.1.12 numactl@2.0.14 readline@8.1
berkeley-db@18.1.40 libiconv@1.16 openblas@0.3.20 tcl@8.6.12
bzip2@1.0.8 libpciaccess@0.16 openmpi@4.1.3 trilinos@13.0.1
cmake@3.23.1 libsigsegv@2.13 openssh@9.0p1 util-macros@1.19.3
diffutils@3.8 libtool@2.4.7 openssl@1.1.1o xz@5.2.5
gdbm@1.19 libxml2@2.9.13 perl@5.34.1 zlib@1.2.12
hwloc@2.7.1 m4@1.4.19 pkgconf@1.8.0
Notice the two specs we added, hdf5 +hl and gmp, are now
listed as root specs. They are not actually installed in the
environment yet because spack add only adds roots to the
environment.
All of the yet-to-be-installed packages can be installed in an
active environment by simply running spack install with no
arguments:
$ spack install
==> Starting concretization pool with 2 processes
==> Environment concretized in 16.29 seconds.
==> Concretized tcl
[+] 6m3sqvr tcl@8.6.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] fntvsj6 ^zlib@1.2.12%gcc@7.5.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu18.04-x86_64
==> Concretized trilinos
[+] 6t2acz7 trilinos@13.0.1%gcc@7.5.0~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec~shards+shared~shylu~stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~thyra+tpetra~trilinoscouplings~wrapper~x11~zoltan~zoltan2 build_type=RelWithDebInfo cxxstd=14 gotype=long_long arch=linux-ubuntu18.04-x86_64
[+] tulqz4n ^cmake@3.23.1%gcc@7.5.0~doc+ncurses+ownlibs~qt build_type=Release arch=linux-ubuntu18.04-x86_64
[+] frwzsc4 ^ncurses@6.2%gcc@7.5.0~symlinks+termlib abi=none arch=linux-ubuntu18.04-x86_64
[+] i5po27g ^pkgconf@1.8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] hqlqpsn ^openssl@1.1.1o%gcc@7.5.0~docs~shared certs=system arch=linux-ubuntu18.04-x86_64
[+] pn5mtes ^perl@5.34.1%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] q6tqmmm ^berkeley-db@18.1.40%gcc@7.5.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu18.04-x86_64
[+] cyn2yam ^bzip2@1.0.8%gcc@7.5.0~debug~pic+shared arch=linux-ubuntu18.04-x86_64
[+] k22pe4r ^diffutils@3.8%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] dr56jt4 ^libiconv@1.16%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] xub4jsy ^gdbm@1.19%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] ls6erxw ^readline@8.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] fntvsj6 ^zlib@1.2.12%gcc@7.5.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu18.04-x86_64
[+] oixja6d ^hwloc@2.7.1%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-ubuntu18.04-x86_64
[+] csp5gvq ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 7fhdrxx ^libtool@2.4.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] uadbn2a ^m4@1.4.19%gcc@7.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu18.04-x86_64
[+] ubezeyc ^libsigsegv@2.13%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] vkk73ww ^util-macros@1.19.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4run532 ^libxml2@2.9.13%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] olptpbs ^xz@5.2.5%gcc@7.5.0~pic libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] dydujpz ^openblas@0.3.20%gcc@7.5.0~bignuma~consistent_fpcsr~ilp64+locking+pic+shared symbol_suffix=none threads=none arch=linux-ubuntu18.04-x86_64
[+] jfxctqw ^openmpi@4.1.3%gcc@7.5.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi+romio+rsh~singularity+static+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu18.04-x86_64
[+] wfnw532 ^numactl@2.0.14%gcc@7.5.0 patches=4e1d78c,62fc8a8,ff37630 arch=linux-ubuntu18.04-x86_64
[+] hlj3ujm ^autoconf@2.69%gcc@7.5.0 patches=35c4492,7793209,a49dd5b arch=linux-ubuntu18.04-x86_64
[+] 6vk5ehd ^automake@1.16.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 3ez3d3j ^openssh@9.0p1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zc56lv6 ^libedit@3.1-20210216%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] e4fax47 ^pmix@4.1.2%gcc@7.5.0~docs+pmi_backwards_compatibility~restful arch=linux-ubuntu18.04-x86_64
[+] hlnkgmr ^libevent@2.1.12%gcc@7.5.0+openssl arch=linux-ubuntu18.04-x86_64
==> Concretized hdf5+hl ^mpich
[+] e3cws2t hdf5@1.12.2%gcc@7.5.0~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_type=RelWithDebInfo arch=linux-ubuntu18.04-x86_64
[+] tulqz4n ^cmake@3.23.1%gcc@7.5.0~doc+ncurses+ownlibs~qt build_type=Release arch=linux-ubuntu18.04-x86_64
[+] frwzsc4 ^ncurses@6.2%gcc@7.5.0~symlinks+termlib abi=none arch=linux-ubuntu18.04-x86_64
[+] i5po27g ^pkgconf@1.8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] hqlqpsn ^openssl@1.1.1o%gcc@7.5.0~docs~shared certs=system arch=linux-ubuntu18.04-x86_64
[+] pn5mtes ^perl@5.34.1%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] q6tqmmm ^berkeley-db@18.1.40%gcc@7.5.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu18.04-x86_64
[+] cyn2yam ^bzip2@1.0.8%gcc@7.5.0~debug~pic+shared arch=linux-ubuntu18.04-x86_64
[+] k22pe4r ^diffutils@3.8%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] dr56jt4 ^libiconv@1.16%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] xub4jsy ^gdbm@1.19%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] ls6erxw ^readline@8.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] fntvsj6 ^zlib@1.2.12%gcc@7.5.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu18.04-x86_64
[+] merlslk ^mpich@4.0.2%gcc@7.5.0~argobots~cuda+fortran+hwloc+hydra+libxml2+pci~rocm+romio~slurm~two_level_namespace~vci~verbs+wrapperrpath datatype-engine=auto device=ch4 netmod=ofi pmi=pmi arch=linux-ubuntu18.04-x86_64
[+] bliztd7 ^findutils@4.9.0%gcc@7.5.0 patches=440b954 arch=linux-ubuntu18.04-x86_64
[+] oixja6d ^hwloc@2.7.1%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-ubuntu18.04-x86_64
[+] csp5gvq ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 7fhdrxx ^libtool@2.4.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] uadbn2a ^m4@1.4.19%gcc@7.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu18.04-x86_64
[+] ubezeyc ^libsigsegv@2.13%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] vkk73ww ^util-macros@1.19.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4run532 ^libxml2@2.9.13%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] olptpbs ^xz@5.2.5%gcc@7.5.0~pic libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] aeocsli ^libfabric@1.14.1%gcc@7.5.0~debug~kdreg fabrics=sockets,tcp,udp arch=linux-ubuntu18.04-x86_64
[+] gcyym67 ^yaksa@0.2%gcc@7.5.0~cuda~rocm arch=linux-ubuntu18.04-x86_64
[+] hlj3ujm ^autoconf@2.69%gcc@7.5.0 patches=35c4492,7793209,a49dd5b arch=linux-ubuntu18.04-x86_64
[+] 6vk5ehd ^automake@1.16.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] t3ey2ga ^python@3.9.12%gcc@7.5.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu18.04-x86_64
[+] 5grv4ow ^expat@2.4.8%gcc@7.5.0+libbsd arch=linux-ubuntu18.04-x86_64
[+] zliunm4 ^libbsd@0.11.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] llu53co ^libmd@1.0.4%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] wuasyuu ^gettext@0.21%gcc@7.5.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-ubuntu18.04-x86_64
[+] 3f62loj ^tar@1.34%gcc@7.5.0 zip=pigz arch=linux-ubuntu18.04-x86_64
[+] i74tgiq ^pigz@2.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] w4beu34 ^zstd@1.5.2%gcc@7.5.0+programs compression=none libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] tkru6n3 ^libffi@3.4.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zbjzhle ^sqlite@3.38.5%gcc@7.5.0+column_metadata+dynamic_extensions+fts~functions+rtree arch=linux-ubuntu18.04-x86_64
[+] 7fulw7l ^util-linux-uuid@2.37.4%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
==> Concretized gmp
[+] lf7yoxu gmp@6.2.1%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] hlj3ujm ^autoconf@2.69%gcc@7.5.0 patches=35c4492,7793209,a49dd5b arch=linux-ubuntu18.04-x86_64
[+] uadbn2a ^m4@1.4.19%gcc@7.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu18.04-x86_64
[+] ubezeyc ^libsigsegv@2.13%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] pn5mtes ^perl@5.34.1%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] q6tqmmm ^berkeley-db@18.1.40%gcc@7.5.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu18.04-x86_64
[+] cyn2yam ^bzip2@1.0.8%gcc@7.5.0~debug~pic+shared arch=linux-ubuntu18.04-x86_64
[+] k22pe4r ^diffutils@3.8%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] dr56jt4 ^libiconv@1.16%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] xub4jsy ^gdbm@1.19%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] ls6erxw ^readline@8.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] frwzsc4 ^ncurses@6.2%gcc@7.5.0~symlinks+termlib abi=none arch=linux-ubuntu18.04-x86_64
[+] i5po27g ^pkgconf@1.8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] fntvsj6 ^zlib@1.2.12%gcc@7.5.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu18.04-x86_64
[+] 6vk5ehd ^automake@1.16.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 7fhdrxx ^libtool@2.4.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
==> Installing environment myproject
==> All of the packages are already installed
==> Updating view at /home/spack/spack/var/spack/environments/myproject/.spack-env/view
Spack concretizes the new root specs before ensuring that all
the associated packages are installed. You can confirm this using
spack find:
$ spack find
==> In environment myproject
==> Root specs
gmp hdf5 +hl tcl trilinos
==> 48 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 expat@2.4.8 hwloc@2.7.1 libiconv@1.16 m4@1.4.19 openssh@9.0p1 python@3.9.12 util-linux-uuid@2.37.4
automake@1.16.5 findutils@4.9.0 libbsd@0.11.5 libmd@1.0.4 mpich@4.0.2 openssl@1.1.1o readline@8.1 util-macros@1.19.3
berkeley-db@18.1.40 gdbm@1.19 libedit@3.1-20210216 libpciaccess@0.16 ncurses@6.2 perl@5.34.1 sqlite@3.38.5 xz@5.2.5
bzip2@1.0.8 gettext@0.21 libevent@2.1.12 libsigsegv@2.13 numactl@2.0.14 pigz@2.7 tar@1.34 yaksa@0.2
cmake@3.23.1 gmp@6.2.1 libfabric@1.14.1 libtool@2.4.7 openblas@0.3.20 pkgconf@1.8.0 tcl@8.6.12 zlib@1.2.12
diffutils@3.8 hdf5@1.12.2 libffi@3.4.2 libxml2@2.9.13 openmpi@4.1.3 pmix@4.1.2 trilinos@13.0.1 zstd@1.5.2
Configuring specs¶
An environment is more than just a list of root specs. It includes
configuration settings that affect the way Spack behaves when the
environment is activated. So far, myproject relies on configuration
defaults that can be overridden. Here we’ll look at how to add specs
and ensure all the packages depending on mpi build with mpich.
We can customize the selection of the mpi provider using
concretization preferences
to change the behavior of the concretizer.
Let’s start by looking at the configuration of our environment using
spack config get:
$ spack config get
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
add package specs to the `specs` list
specs: [tcl, trilinos, hdf5+hl ^mpich, gmp]
view: true
concretizer:
unify: false
The output shows the special YAML configuration format that Spack
uses to store the environment configuration. Currently, the file
is just a spack: header and a list of the root specs.
Note
Before proceeding, make sure your EDITOR environment variable
is set to the path of your preferred text editor.
Let’s edit this file to prefer mpich as our mpi provider
using spack config edit.
You should now have the above file open in your editor. Change it
to include the packages:all:providers:mpi: entry below:
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
packages:
all:
providers:
mpi: [mpich]
# add package specs to the `specs` list
specs: [tcl, trilinos, hdf5+hl ^mpich, gmp]
Note
This setting only defines the default mpi provider.
You can still override the provider on the command line, e.g.,
with spack install hdf5 ^openmpi.
We introduce this here to show you how environment configuration can affect concretization. Configuration options are covered in much more detail in the configuration tutorial.
We’ve only scratched the surface here by changing the default of
the mpi provider for packages depending on mpi. There
are many other customizations you can make to an environment.
Refer to the links at the end of this section for more information.
Re-concretizing the environment¶
You may need to re-install packages in the environment after making significant changes to the configuration, such as changing virtual providers. This can be accomplished by forcing Spack to re-concretize the environment and re-install the specs.
For example, the packages installed in our myproject environment
are now out of sync with our new configuration since we already
installed part of the environment with openmpi. Suppose we want
to install everything in myproject with mpich.
Let’s run spack concretize --force --fresh to make Spack re-concretize
all the environment’s specs:
$ spack concretize --force --fresh
==> Starting concretization pool with 4 processes
==> Environment concretized in 19.00 seconds.
==> Concretized tcl
[+] 6m3sqvr tcl@8.6.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] fntvsj6 ^zlib@1.2.12%gcc@7.5.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu18.04-x86_64
==> Concretized trilinos
- otna7uz trilinos@13.0.1%gcc@7.5.0~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec~shards+shared~shylu~stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~thyra+tpetra~trilinoscouplings~wrapper~x11~zoltan~zoltan2 build_type=RelWithDebInfo cxxstd=14 gotype=long_long arch=linux-ubuntu18.04-x86_64
[+] tulqz4n ^cmake@3.23.1%gcc@7.5.0~doc+ncurses+ownlibs~qt build_type=Release arch=linux-ubuntu18.04-x86_64
[+] frwzsc4 ^ncurses@6.2%gcc@7.5.0~symlinks+termlib abi=none arch=linux-ubuntu18.04-x86_64
[+] i5po27g ^pkgconf@1.8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] hqlqpsn ^openssl@1.1.1o%gcc@7.5.0~docs~shared certs=system arch=linux-ubuntu18.04-x86_64
[+] pn5mtes ^perl@5.34.1%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] q6tqmmm ^berkeley-db@18.1.40%gcc@7.5.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu18.04-x86_64
[+] cyn2yam ^bzip2@1.0.8%gcc@7.5.0~debug~pic+shared arch=linux-ubuntu18.04-x86_64
[+] k22pe4r ^diffutils@3.8%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] dr56jt4 ^libiconv@1.16%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] xub4jsy ^gdbm@1.19%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] ls6erxw ^readline@8.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] fntvsj6 ^zlib@1.2.12%gcc@7.5.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu18.04-x86_64
[+] oixja6d ^hwloc@2.7.1%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-ubuntu18.04-x86_64
[+] csp5gvq ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 7fhdrxx ^libtool@2.4.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] uadbn2a ^m4@1.4.19%gcc@7.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu18.04-x86_64
[+] ubezeyc ^libsigsegv@2.13%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] vkk73ww ^util-macros@1.19.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4run532 ^libxml2@2.9.13%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] olptpbs ^xz@5.2.5%gcc@7.5.0~pic libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] merlslk ^mpich@4.0.2%gcc@7.5.0~argobots~cuda+fortran+hwloc+hydra+libxml2+pci~rocm+romio~slurm~two_level_namespace~vci~verbs+wrapperrpath datatype-engine=auto device=ch4 netmod=ofi pmi=pmi arch=linux-ubuntu18.04-x86_64
[+] bliztd7 ^findutils@4.9.0%gcc@7.5.0 patches=440b954 arch=linux-ubuntu18.04-x86_64
[+] aeocsli ^libfabric@1.14.1%gcc@7.5.0~debug~kdreg fabrics=sockets,tcp,udp arch=linux-ubuntu18.04-x86_64
[+] gcyym67 ^yaksa@0.2%gcc@7.5.0~cuda~rocm arch=linux-ubuntu18.04-x86_64
[+] hlj3ujm ^autoconf@2.69%gcc@7.5.0 patches=35c4492,7793209,a49dd5b arch=linux-ubuntu18.04-x86_64
[+] 6vk5ehd ^automake@1.16.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] t3ey2ga ^python@3.9.12%gcc@7.5.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu18.04-x86_64
[+] 5grv4ow ^expat@2.4.8%gcc@7.5.0+libbsd arch=linux-ubuntu18.04-x86_64
[+] zliunm4 ^libbsd@0.11.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] llu53co ^libmd@1.0.4%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] wuasyuu ^gettext@0.21%gcc@7.5.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-ubuntu18.04-x86_64
[+] 3f62loj ^tar@1.34%gcc@7.5.0 zip=pigz arch=linux-ubuntu18.04-x86_64
[+] i74tgiq ^pigz@2.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] w4beu34 ^zstd@1.5.2%gcc@7.5.0+programs compression=none libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] tkru6n3 ^libffi@3.4.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zbjzhle ^sqlite@3.38.5%gcc@7.5.0+column_metadata+dynamic_extensions+fts~functions+rtree arch=linux-ubuntu18.04-x86_64
[+] 7fulw7l ^util-linux-uuid@2.37.4%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] dydujpz ^openblas@0.3.20%gcc@7.5.0~bignuma~consistent_fpcsr~ilp64+locking+pic+shared symbol_suffix=none threads=none arch=linux-ubuntu18.04-x86_64
==> Concretized hdf5+hl ^mpich
[+] e3cws2t hdf5@1.12.2%gcc@7.5.0~cxx~fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_type=RelWithDebInfo arch=linux-ubuntu18.04-x86_64
[+] tulqz4n ^cmake@3.23.1%gcc@7.5.0~doc+ncurses+ownlibs~qt build_type=Release arch=linux-ubuntu18.04-x86_64
[+] frwzsc4 ^ncurses@6.2%gcc@7.5.0~symlinks+termlib abi=none arch=linux-ubuntu18.04-x86_64
[+] i5po27g ^pkgconf@1.8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] hqlqpsn ^openssl@1.1.1o%gcc@7.5.0~docs~shared certs=system arch=linux-ubuntu18.04-x86_64
[+] pn5mtes ^perl@5.34.1%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] q6tqmmm ^berkeley-db@18.1.40%gcc@7.5.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu18.04-x86_64
[+] cyn2yam ^bzip2@1.0.8%gcc@7.5.0~debug~pic+shared arch=linux-ubuntu18.04-x86_64
[+] k22pe4r ^diffutils@3.8%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] dr56jt4 ^libiconv@1.16%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] xub4jsy ^gdbm@1.19%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] ls6erxw ^readline@8.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] fntvsj6 ^zlib@1.2.12%gcc@7.5.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu18.04-x86_64
[+] merlslk ^mpich@4.0.2%gcc@7.5.0~argobots~cuda+fortran+hwloc+hydra+libxml2+pci~rocm+romio~slurm~two_level_namespace~vci~verbs+wrapperrpath datatype-engine=auto device=ch4 netmod=ofi pmi=pmi arch=linux-ubuntu18.04-x86_64
[+] bliztd7 ^findutils@4.9.0%gcc@7.5.0 patches=440b954 arch=linux-ubuntu18.04-x86_64
[+] oixja6d ^hwloc@2.7.1%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-ubuntu18.04-x86_64
[+] csp5gvq ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 7fhdrxx ^libtool@2.4.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] uadbn2a ^m4@1.4.19%gcc@7.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu18.04-x86_64
[+] ubezeyc ^libsigsegv@2.13%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] vkk73ww ^util-macros@1.19.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4run532 ^libxml2@2.9.13%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] olptpbs ^xz@5.2.5%gcc@7.5.0~pic libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] aeocsli ^libfabric@1.14.1%gcc@7.5.0~debug~kdreg fabrics=sockets,tcp,udp arch=linux-ubuntu18.04-x86_64
[+] gcyym67 ^yaksa@0.2%gcc@7.5.0~cuda~rocm arch=linux-ubuntu18.04-x86_64
[+] hlj3ujm ^autoconf@2.69%gcc@7.5.0 patches=35c4492,7793209,a49dd5b arch=linux-ubuntu18.04-x86_64
[+] 6vk5ehd ^automake@1.16.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] t3ey2ga ^python@3.9.12%gcc@7.5.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu18.04-x86_64
[+] 5grv4ow ^expat@2.4.8%gcc@7.5.0+libbsd arch=linux-ubuntu18.04-x86_64
[+] zliunm4 ^libbsd@0.11.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] llu53co ^libmd@1.0.4%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] wuasyuu ^gettext@0.21%gcc@7.5.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-ubuntu18.04-x86_64
[+] 3f62loj ^tar@1.34%gcc@7.5.0 zip=pigz arch=linux-ubuntu18.04-x86_64
[+] i74tgiq ^pigz@2.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] w4beu34 ^zstd@1.5.2%gcc@7.5.0+programs compression=none libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] tkru6n3 ^libffi@3.4.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zbjzhle ^sqlite@3.38.5%gcc@7.5.0+column_metadata+dynamic_extensions+fts~functions+rtree arch=linux-ubuntu18.04-x86_64
[+] 7fulw7l ^util-linux-uuid@2.37.4%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
==> Concretized gmp
[+] lf7yoxu gmp@6.2.1%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] hlj3ujm ^autoconf@2.69%gcc@7.5.0 patches=35c4492,7793209,a49dd5b arch=linux-ubuntu18.04-x86_64
[+] uadbn2a ^m4@1.4.19%gcc@7.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu18.04-x86_64
[+] ubezeyc ^libsigsegv@2.13%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] pn5mtes ^perl@5.34.1%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] q6tqmmm ^berkeley-db@18.1.40%gcc@7.5.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu18.04-x86_64
[+] cyn2yam ^bzip2@1.0.8%gcc@7.5.0~debug~pic+shared arch=linux-ubuntu18.04-x86_64
[+] k22pe4r ^diffutils@3.8%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] dr56jt4 ^libiconv@1.16%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] xub4jsy ^gdbm@1.19%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] ls6erxw ^readline@8.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] frwzsc4 ^ncurses@6.2%gcc@7.5.0~symlinks+termlib abi=none arch=linux-ubuntu18.04-x86_64
[+] i5po27g ^pkgconf@1.8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] fntvsj6 ^zlib@1.2.12%gcc@7.5.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu18.04-x86_64
[+] 6vk5ehd ^automake@1.16.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 7fhdrxx ^libtool@2.4.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
==> Updating view at /home/spack/spack/var/spack/environments/myproject/.spack-env/view
All the specs are now concrete and ready to be installed with
mpich as the MPI implementation.
Building in environments¶
Activated environments allow you to invoke any programs installed in them as if they were installed on the system. In this section we will take advantage of that feature.
Suppose you want to compile some MPI programs. We have an MPI
implementation installed in our myproject environment, so
mpicc is available in our path. We can confirm this using
which:
$ spack env status
==> In environment myproject
$ which mpicc
/home/spack/spack/var/spack/environments/myproject/.spack-env/view/bin/mpicc
As mentioned before, activating the environment sets a number of
environment variables. That includes variables like PATH,
LIBRARY_PATH, and LD_LIBRARY_PATH, which allows you to
easily find package executables and libraries installed in the environment.
Let’s look specifically at path-related environment variables using
env | grep PATH:
$ env | grep PATH=
LD_LIBRARY_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib64
PKG_CONFIG_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib/pkgconfig:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/share/pkgconfig:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib64/pkgconfig
ACLOCAL_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/share/aclocal
PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/bin:/home/spack/spack/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SPACK_LD_LIBRARY_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib64
MANPATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/share/man:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/man:.:
MODULEPATH=/home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64
CMAKE_PREFIX_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view
We can demonstrate use of these environment settings by building a really simple MPI program.
Let’s create a program called mpi-hello.c that contains the following
code:
#include <stdio.h>
#include <mpi.h>
#include <zlib.h>
int main(int argc, char **argv) {
int rank;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
printf("Hello world from rank %d\n", rank);
if (rank == 0) {
printf("zlib version: %s\n", ZLIB_VERSION);
}
MPI_Finalize();
}
This program includes headers from mpi and zlib.
It also prints out a message from each MPI rank and the
version of zlib.
Let’s build and run our program:
$ mpicc ./mpi-hello.c -I$(spack location -i zlib)/include
$ mpirun -n 4 ./a.out
Hello world from rank 3
Hello world from rank 1
Hello world from rank 2
Hello world from rank 0
zlib version: 1.2.12
Notice that we only needed to pass the include path to the
compiler.
We also see that Hello world is output for each of the ranks
and the version of zlib used to build the program is printed.
We can confirm the version of zlib used to build the program
is in our environment using spack find:
$ spack find zlib
==> In environment myproject
==> Root specs
gmp hdf5 +hl tcl trilinos
==> 1 installed package
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 ----------------
zlib@1.2.12
Note that the reported version does match that of our installation.
Reproducing builds¶
Spack environments provide users with virtual environments similar to Python venv and Conda environments). The goal is to ensure packages in one environment are kept separate from those of another. These environments can be managed by Spack or independent. In either case, their environment files can be used to reproduce builds by other users and on other machines. Since those files are key to reproducing builds, let’s start with them.
Environment files¶
There are two key files tracking the contents of environments:
spack.yaml and spack.lock. The spack.yaml file holds
the environment configuration that we previously edited through
spack config edit. The spack.lock file is automatically
generated during concretization.
The two files represent two fundamental concepts:
spack.yaml: abstract specs and configuration to install; andspack.lock: all fully concrete specs.
These files are intended to be used by developers and administrators to manage the environments in a reproducible way. We will cover their re-use later.
Note
Both environment files can be versioned in repositories, shared, and used to install the same set of software by different users and on other machines.
Managed versus independent environments¶
Environments are either Spack-managed or independent. Both types of environments are defined by their environment files. So far we have only created managed environments. This section describes their differences.
Managed environments are created using spack env create <name>.
They are automatically created in the var/spack/environments
subdirectory and can be referenced by their names.
Independent environments can be created in one of two ways. First,
the Spack environment file(s) can be placed in any directory
(other than var/spack/environments). Alternatively, you can
use spack env create -d <directory> to specify the directory
(<directory>) in which the files should reside. Independent
environments are not named.
Reviewing a managed environment¶
We created our currently active environment, myproject, earlier
using spack env create myproject so let’s mainly focus on its
environment files in this section.
Earlier, when we changed our environment’s configuration using spack
config edit, we were actually editing its spack.yaml file. We
can change to the directory containing the file using spack cd:
$ spack cd -e myproject
$ pwd
/home/spack/spack/var/spack/environments/myproject
$ ls
spack.lock spack.yaml
Notice that myproject is a subdirectory of var/spack/environments
within the Spack installation making it a managed environment.
Consequently, it can be referenced by name. It will also show up
when running spack env list:
$ spack env list
==> 2 environments
myproject myproject2
You can see that myproject is active because it is highlighted
in green.
We can also see from the listing that the environment directory
contains both of the environment files: spack.yaml and spack.lock.
This is because spack.lock was generated when we concretized
the environment.
If we cat the spack.yaml file, we’ll see the same specs and view
options previously shown by spack config get:
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
packages:
all:
providers:
mpi: [mpich]
specs: [tcl, trilinos, hdf5+hl ^mpich, gmp]
view: true
Creating an independent environment¶
Environments do not have to be created in or managed by a Spack instance. Rather, their environment files can be placed in any directory. This feature can be quite helpful for use cases such as environment-based software releases and CI/CD.
Let’s create an independent environment from scratch for a simple project:
$ cd
$ mkdir code
$ cd code
$ spack env create -d .
==> Created environment in /home/spack/code
==> You can activate this environment with:
==> spack env activate /home/spack/code
Notice that the command shows Spack created the environment, updated the view, and printed the command needed to activate it. As we can see in the activation command, since the environment is independent, it must be referenced by its directory path.
Let’s see what really happened with this command by listing the directory contents and looking at the configuration file:
$ ls
spack.yaml
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
add package specs to the `specs` list
specs: []
view: true
concretizer:
unify: false
Notice that Spack created a spack.yaml file in the code directory.
Also note that the configuration file has an empty spec list (i.e.,
[]). That list is intended to contain only the root specs of
the environment.
We can confirm that it is not a managed environment by running
spack env list:
$ spack env list
==> 2 environments
myproject myproject2
and noting that the path does not appear in the output.
Now let’s add some specs to the environment. Suppose your project
depends on trilinos and openmpi. Add these
packages to the spec list using your favorite text editor. The
dash syntax for a YAML list is used in our example. Your package
should now contain the following entries:
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs:
- trilinos
- openmpi
view: true
Now activate the environment and install the packages:
$ spack env activate .
$ spack install
==> Starting concretization pool with 2 processes
==> Environment concretized in 20.08 seconds.
==> Concretized trilinos
[+] 6t2acz7 trilinos@13.0.1%gcc@7.5.0~adelus~adios2+amesos+amesos2+anasazi+aztec~basker+belos~boost~chaco~complex~cuda~cuda_rdc~debug~dtk+epetra+epetraext~epetraextbtf~epetraextexperimental~epetraextgraphreorderings~exodus+explicit_template_instantiation~float+fortran~gtest~hdf5~hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos~mesquite~minitensor+ml+mpi+muelu~mumps~nox~openmp~panzer~phalanx~piro~python~rocm~rocm_rdc~rol~rythmos+sacado~scorec~shards+shared~shylu~stk~stokhos~stratimikos~strumpack~suite-sparse~superlu~superlu-dist~teko~tempus~thyra+tpetra~trilinoscouplings~wrapper~x11~zoltan~zoltan2 build_type=RelWithDebInfo cxxstd=14 gotype=long_long arch=linux-ubuntu18.04-x86_64
[+] tulqz4n ^cmake@3.23.1%gcc@7.5.0~doc+ncurses+ownlibs~qt build_type=Release arch=linux-ubuntu18.04-x86_64
[+] frwzsc4 ^ncurses@6.2%gcc@7.5.0~symlinks+termlib abi=none arch=linux-ubuntu18.04-x86_64
[+] i5po27g ^pkgconf@1.8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] hqlqpsn ^openssl@1.1.1o%gcc@7.5.0~docs~shared certs=system arch=linux-ubuntu18.04-x86_64
[+] pn5mtes ^perl@5.34.1%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] q6tqmmm ^berkeley-db@18.1.40%gcc@7.5.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu18.04-x86_64
[+] cyn2yam ^bzip2@1.0.8%gcc@7.5.0~debug~pic+shared arch=linux-ubuntu18.04-x86_64
[+] k22pe4r ^diffutils@3.8%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] dr56jt4 ^libiconv@1.16%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] xub4jsy ^gdbm@1.19%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] ls6erxw ^readline@8.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] fntvsj6 ^zlib@1.2.12%gcc@7.5.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu18.04-x86_64
[+] oixja6d ^hwloc@2.7.1%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-ubuntu18.04-x86_64
[+] csp5gvq ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 7fhdrxx ^libtool@2.4.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] uadbn2a ^m4@1.4.19%gcc@7.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu18.04-x86_64
[+] ubezeyc ^libsigsegv@2.13%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] vkk73ww ^util-macros@1.19.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4run532 ^libxml2@2.9.13%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] olptpbs ^xz@5.2.5%gcc@7.5.0~pic libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] dydujpz ^openblas@0.3.20%gcc@7.5.0~bignuma~consistent_fpcsr~ilp64+locking+pic+shared symbol_suffix=none threads=none arch=linux-ubuntu18.04-x86_64
[+] jfxctqw ^openmpi@4.1.3%gcc@7.5.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi+romio+rsh~singularity+static+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu18.04-x86_64
[+] wfnw532 ^numactl@2.0.14%gcc@7.5.0 patches=4e1d78c,62fc8a8,ff37630 arch=linux-ubuntu18.04-x86_64
[+] hlj3ujm ^autoconf@2.69%gcc@7.5.0 patches=35c4492,7793209,a49dd5b arch=linux-ubuntu18.04-x86_64
[+] 6vk5ehd ^automake@1.16.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 3ez3d3j ^openssh@9.0p1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zc56lv6 ^libedit@3.1-20210216%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] e4fax47 ^pmix@4.1.2%gcc@7.5.0~docs+pmi_backwards_compatibility~restful arch=linux-ubuntu18.04-x86_64
[+] hlnkgmr ^libevent@2.1.12%gcc@7.5.0+openssl arch=linux-ubuntu18.04-x86_64
==> Concretized openmpi
[+] jfxctqw openmpi@4.1.3%gcc@7.5.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi+romio+rsh~singularity+static+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu18.04-x86_64
[+] oixja6d ^hwloc@2.7.1%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=linux-ubuntu18.04-x86_64
[+] csp5gvq ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 7fhdrxx ^libtool@2.4.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] uadbn2a ^m4@1.4.19%gcc@7.5.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu18.04-x86_64
[+] ubezeyc ^libsigsegv@2.13%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] i5po27g ^pkgconf@1.8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] vkk73ww ^util-macros@1.19.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4run532 ^libxml2@2.9.13%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] dr56jt4 ^libiconv@1.16%gcc@7.5.0 libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] olptpbs ^xz@5.2.5%gcc@7.5.0~pic libs=shared,static arch=linux-ubuntu18.04-x86_64
[+] fntvsj6 ^zlib@1.2.12%gcc@7.5.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu18.04-x86_64
[+] frwzsc4 ^ncurses@6.2%gcc@7.5.0~symlinks+termlib abi=none arch=linux-ubuntu18.04-x86_64
[+] wfnw532 ^numactl@2.0.14%gcc@7.5.0 patches=4e1d78c,62fc8a8,ff37630 arch=linux-ubuntu18.04-x86_64
[+] hlj3ujm ^autoconf@2.69%gcc@7.5.0 patches=35c4492,7793209,a49dd5b arch=linux-ubuntu18.04-x86_64
[+] pn5mtes ^perl@5.34.1%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] q6tqmmm ^berkeley-db@18.1.40%gcc@7.5.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu18.04-x86_64
[+] cyn2yam ^bzip2@1.0.8%gcc@7.5.0~debug~pic+shared arch=linux-ubuntu18.04-x86_64
[+] k22pe4r ^diffutils@3.8%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] xub4jsy ^gdbm@1.19%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] ls6erxw ^readline@8.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 6vk5ehd ^automake@1.16.5%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 3ez3d3j ^openssh@9.0p1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zc56lv6 ^libedit@3.1-20210216%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] hqlqpsn ^openssl@1.1.1o%gcc@7.5.0~docs~shared certs=system arch=linux-ubuntu18.04-x86_64
[+] e4fax47 ^pmix@4.1.2%gcc@7.5.0~docs+pmi_backwards_compatibility~restful arch=linux-ubuntu18.04-x86_64
[+] hlnkgmr ^libevent@2.1.12%gcc@7.5.0+openssl arch=linux-ubuntu18.04-x86_64
==> Installing environment /home/spack/code
==> All of the packages are already installed
==> Updating view at /home/spack/code/.spack-env/view
Notice that Spack concretized the specs before installing them and their dependencies. It also updated the environment’s view. Since we already installed all these packages outside of the environment, their links were simply added to it.
Updating an installed environment¶
Spack supports tweaking an environment even after the initial specs are installed. You are free to add and remove specs just as you would outside of the environment using the command line interface as before.
For example, let’s add hdf5 and look at our file:
$ spack add hdf5@5.5.1
==> Adding hdf5@5.5.1 to environment /home/spack/code
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
add package specs to the `specs` list
specs: [trilinos, openmpi, hdf5@5.5.1]
view: true
concretizer:
unify: false
$ spack remove hdf5
==> Removing hdf5 from environment /home/spack/code
Notice that spack add added the package to our active environment and
it appears in the configuration file’s spec list.
Note
You’ll need to run spack install to install added packages
in your environment because spack add only adds it to the
configuration.
Now use spack remove to remove the spec from the configuration:
$ spack remove hdf5
==> Removing hdf5 from environment /home/spack/code
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
add package specs to the `specs` list
specs:
- trilinos
- openmpi
view: true
and we see that the spec was removed from the spec list of our environment.
Note
You can also edit the spack.yaml file directly instead of
using the spack add and spack remove commands.
Reviewing spack.lock¶
Now let’s turn our attention from the abstract to the concrete.
Our focus so far has been on the abstract environment configuration
represented by the spack.yaml file. Once that file is concretized,
Spack generates a corresponding spack.lock file representing
the full concretized state of the environment.
This file is intended to be a machine-readable representation of the
information needed to reproduce the build of an environment. As such,
it is written in json, which is less readable than yaml.
Let’s look at the top 30 lines of our current environment:
$ head -30 spack.lock
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 4,
"specfile-version": 3
},
"roots": [
{
"hash": "6t2acz7flltuohbhadoelljcn6te2hm2",
"spec": "trilinos"
},
{
"hash": "jfxctqwar7wb65rn7wf5mot7m4jxmsey",
"spec": "openmpi"
}
],
"concrete_specs": {
"6t2acz7flltuohbhadoelljcn6te2hm2": {
"name": "trilinos",
"version": "13.0.1",
"arch": {
"platform": "linux",
"platform_os": "ubuntu18.04",
"target": "x86_64"
},
"compiler": {
"name": "gcc",
"version": "7.5.0"
},
"namespace": "builtin",
While it is still readable, it consists of over 1300 lines of information representing the actual configurations for each of the environment’s packages.
Reproducing an environment¶
Now that we’ve described the contents of the environment files we can discuss how they can be used to reproduce environments. You may want to do this yourself on a different machine, or use an environment built by someone else. The process is the same in either case.
You can recreate an environment by passing either of the environment
files to spack env create. The file you choose depends on whether
you want to approximate the build using the abstract specs or an exact
build based on the concrete specs.
Using spack.yaml¶
An approximate build is created using the spack.yaml file. This
approach is relevant when we want to build the same specs on a new
platform, for example. It allows you to reproduce the environment
by preserving the abstract requirements in the file. However, the
software may actually build differently in part because the concretizer
may choose different dependencies.
Let’s use spack env create to create an abstract environment from
the file that we’ll call abstract:
$ spack env create abstract spack.yaml
==> Created environment 'abstract' in /home/spack/spack/var/spack/environments/abstract
==> You can activate this environment with:
==> spack env activate abstract
Here we see that Spack created a managed environment with the name we provided.
And, since it is a newly created environment, it does not have any
installed specs yet as we can see from calling spack find
after activating the environment:
$ spacktivate abstract
$ spack find
==> In environment abstract
==> Root specs
openmpi trilinos
==> 0 installed packages
Notice that we have the same root specs as were listed in the spack.yaml
file.
Using spack.lock¶
The spack.lock file is used for an exact reproduction of the
original build. It can replicate the build because it contains the
information for all the decisions made during concretization.
Now let’s create a concrete environment, called concrete, from
the file:
$ spack env create concrete spack.lock
==> Updating view at /home/spack/spack/var/spack/environments/concrete/.spack-env/view
==> Created environment 'concrete' in /home/spack/spack/var/spack/environments/concrete
==> You can activate this environment with:
==> spack env activate concrete
Here we see that Spack again created a managed environment with the provided name.
Since we created the environment from our spack.lock file,
not only do we get the same root specs, all of the packages are
installed in the environment as we can see from calling
spack find after activating the environment:
$ spacktivate concrete
$ spack find
==> In environment concrete
==> Root specs
openmpi trilinos
==> 30 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 libedit@3.1-20210216 ncurses@6.2 pmix@4.1.2
automake@1.16.5 libevent@2.1.12 numactl@2.0.14 readline@8.1
berkeley-db@18.1.40 libiconv@1.16 openblas@0.3.20 trilinos@13.0.1
bzip2@1.0.8 libpciaccess@0.16 openmpi@4.1.3 util-macros@1.19.3
cmake@3.23.1 libsigsegv@2.13 openssh@9.0p1 xz@5.2.5
diffutils@3.8 libtool@2.4.7 openssl@1.1.1o zlib@1.2.12
gdbm@1.19 libxml2@2.9.13 perl@5.34.1
hwloc@2.7.1 m4@1.4.19 pkgconf@1.8.0
Note
Use of spack.lock to reproduce a build (currently) requires you
to be on the same type of machine.
More information¶
This tutorial only scratches the surface of environments and what they can do. For more information, take a look at the Spack resources below.
Setting up and building environments¶
- Environments: reference docs
- Configuration tutorial: for customizing your environment
- Spack stacks tutorial: for configuring combinatorial environments (e.g., same packages across a list of compilers)
- Install-level parallel builds:
for how to launch
spack installto build your environment in parallel
Using environments¶
- Developer workflows: for developing code in an environment
- GitLab CI pipelines with Spack environments: for using environments to generate CI pipelines
- Container Images: for creating containers from environments
- Spack stacks tutorial: for managing large deployments of software
Finding examples of environments¶
- Spack Stack Catalog: for discovering environments that you can explore on GitHub