Tutorial setup
If you have not done the prior sections, you’ll need to set Spack up like this:
git clone https://github.com/spack/spack
. spack/share/spack/setup-env.sh
spack tutorial
See the Basic Installation Tutorial for full details on setup. For more
help join us in the #tutorial channel on Slack – get an
invitation at spackpm.herokuapp.com
Environments Tutorial¶
We’ve covered how to install, remove, and list packages with Spack using the commands:
- spack install to install packages;
- spack uninstall to remove them; and
- spack find to look at and query what is installed.
Customizing Spack’s installation with configuration files, like packages.yaml, was also discussed.
This section of the tutorial introduces Spack Environments, which allow you to work with independent groups of packages separately. The goal is to provide users with virtual environments similar to those supported by other commonly used tools (e.g., Python venv) while allowing common installations to be seamlessly shared. They are also intended to be easily shared and re-used by others and across systems.
Administering properly configured software involving lots of packages
and/or varying configuration requirements (e.g., different implementations
of mpi) for multiple projects and efforts can be overwhelming. Spack
environments allow you to readily:
- establish standard software requirements for your project(s);
- set up run environments for users;
- support your usual development environment(s);
- set up packages for CI/CD;
- reproduce builds (approximately or exactly) on other machines; and
- much more.
This tutorial introduces the basics of creating and using environments, then explains how to expand, configure, and build software in them. We will start with the command line interface then cover editing key environment file directly. We will describe the difference between Spack-managed and independent environments, then finish with a section on reproducible builds.
Environment Basics¶
Let’s look at the output of spack find at this point in the tutorial.
$ spack find
==> 63 installed packages
-- linux-ubuntu18.04-x86_64 / clang@6.0.0 -----------------------
zlib@1.2.11
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 hdf5@1.10.7 libxml2@2.9.10 netcdf-c@4.7.4 suite-sparse@5.8.1
autoconf-archive@2019.01.06 hdf5@1.10.7 m4@1.4.18 netlib-scalapack@2.1.0 tcl@8.6.10
automake@1.16.2 hdf5@1.10.7 matio@1.5.17 netlib-scalapack@2.1.0 tcl@8.6.10
berkeley-db@18.1.40 hdf5@1.10.7 matio@1.5.17 numactl@2.0.14 texinfo@6.5
boost@1.74.0 hwloc@1.11.11 metis@5.1.0 openblas@0.3.12 trilinos@13.0.1
bzip2@1.0.8 hwloc@2.2.0 mpc@1.1.0 openmpi@3.1.6 util-macros@1.19.1
cmake@3.18.4 hypre@2.20.0 mpfr@3.1.6 openssl@1.1.1h xz@5.2.5
diffutils@3.7 hypre@2.20.0 mpfr@4.0.2 parmetis@4.0.3 zlib@1.2.8
findutils@4.6.0 isl@0.18 mpich@3.3.2 parmetis@4.0.3 zlib@1.2.8
gcc@8.3.0 libiconv@1.16 mumps@5.3.3 patchelf@0.10 zlib@1.2.11
gdbm@1.18.1 libpciaccess@0.16 mumps@5.3.3 perl@5.32.0
glm@0.9.7.1 libsigsegv@2.12 ncurses@6.2 pkgconf@1.7.3
gmp@6.1.2 libtool@2.4.6 netcdf-c@4.7.4 readline@8.0
This is a complete, but cluttered list of the installed packages and
their dependencies. It contains packages built with both openmpi
and mpich, as well as multiple variants of other packages, like
hdf5 and zlib. The query mechanism we learned about with
spack find can help, but it would be nice if we could start from
a clean slate without losing what we’ve already installed.
Creating and activating environments¶
The spack env command can help. Let’s create a new environment
called myproject:
$ spack env create myproject
==> Updating view at /home/spack/spack/var/spack/environments/myproject/.spack-env/view
==> Created environment 'myproject' in /home/spack/spack/var/spack/environments/myproject
==> You can activate this environment with:
==> spack env activate myproject
An environment is like a virtualized spack instance that you can
use to aggregate package installations for a project or other purpose.
It has an associated view, which is a single prefix where all packages
from the environment are linked.
You can see the environments we’ve created so far using the spack env
list command:
$ spack env list
==> 1 environments
myproject
Now let’s activate our environment. You can use spack env activate
command:
$ spack env activate myproject
You can also use the spacktivate alias for short.
Note
If you use the -p option for spack env activate, Spack
will prepend the environment name to the prompt. This is a handy
way to be reminded if and which environment you are in.
Once you activate an environment, spack find only shows what is
in the current environment. We just created this environment, so it
does not contain any installed packages.
$ spack find
==> In environment myproject
==> No root specs
==> 0 installed packages
The output from spack find is now slightly different. It tells
you that you’re in the myproject environment, so there is no need
to panic when you see that none of the previously installed packages
are available. It also states that there are no root specs. We’ll
get back to what that means later.
If you only want to check what environment you are in, you can use
spack env status:
$ spack env status
==> In environment myproject
If you want to leave this environment, you can use spack env deactivate
or the despacktivate alias for short.
After deactivating, we can see everything installed in this Spack instance:
$ despacktivate # short alias for `spack env deactivate`
$ spack env status
==> No active environment
$ spack find
==> 63 installed packages
-- linux-ubuntu18.04-x86_64 / clang@6.0.0 -----------------------
zlib@1.2.11
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 hdf5@1.10.7 libxml2@2.9.10 netcdf-c@4.7.4 suite-sparse@5.8.1
autoconf-archive@2019.01.06 hdf5@1.10.7 m4@1.4.18 netlib-scalapack@2.1.0 tcl@8.6.10
automake@1.16.2 hdf5@1.10.7 matio@1.5.17 netlib-scalapack@2.1.0 tcl@8.6.10
berkeley-db@18.1.40 hdf5@1.10.7 matio@1.5.17 numactl@2.0.14 texinfo@6.5
boost@1.74.0 hwloc@1.11.11 metis@5.1.0 openblas@0.3.12 trilinos@13.0.1
bzip2@1.0.8 hwloc@2.2.0 mpc@1.1.0 openmpi@3.1.6 util-macros@1.19.1
cmake@3.18.4 hypre@2.20.0 mpfr@3.1.6 openssl@1.1.1h xz@5.2.5
diffutils@3.7 hypre@2.20.0 mpfr@4.0.2 parmetis@4.0.3 zlib@1.2.8
findutils@4.6.0 isl@0.18 mpich@3.3.2 parmetis@4.0.3 zlib@1.2.8
gcc@8.3.0 libiconv@1.16 mumps@5.3.3 patchelf@0.10 zlib@1.2.11
gdbm@1.18.1 libpciaccess@0.16 mumps@5.3.3 perl@5.32.0
glm@0.9.7.1 libsigsegv@2.12 ncurses@6.2 pkgconf@1.7.3
gmp@6.1.2 libtool@2.4.6 netcdf-c@4.7.4 readline@8.0
Notice that we are no longer in an environment and all our packages are still installed.
Installing packages¶
Now that we understand how creation and activation work, let’s go
back to myproject and install a couple of packages; specifically,
tcl and trilinos.
$ spack env activate myproject
$ spack install tcl
==> All of the packages are already installed
$ spack install trilinos
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libiconv-1.16-jearpk4xci4zc7dkrza4fufaqfkq7rfl
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/zlib-1.2.11-smoyzzo2qhzpn6mg6rd3l2p7b23enshg
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/pkgconf-1.7.3-4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/berkeley-db-18.1.40-4ihuiazsglf22f3pntq5hc4kyszqzexn
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libsigsegv-2.12-lbrx7lnfz46ukewxbhxnucmx76g23c6q
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/util-macros-1.19.1-gs6ag7ktdoiirb62t7bcagjw62szrrg2
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/xz-5.2.5-komekkmyciga3kl24edjmredhj3uyt7v
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/openblas-0.3.12-te35tjxctgjpeeuk357xghoinvwf2k7u
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/autoconf-archive-2019.01.06-bdyarrkdgse5adcjr4kxqc4jns74k7yn
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/diffutils-3.7-otkktenrrvcaf37a7zyeytmjuwsalvtr
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/ncurses-6.2-crhlefo3dv7lmsv5pf4icsy4gepkdorm
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/m4-1.4.18-mkc3u4x2p2wie6jfhuku7g5rkovcrxps
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libxml2-2.9.10-yn2r3wfhiilelyulh5toteicdtxjhw7d
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/bzip2-1.0.8-fvfpt26p5divvq6344vojyn64enojlui
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/readline-8.0-t54jzdy2jj4snltjazlm3br2urcilc6v
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libtool-2.4.6-jdxbjftheiotj6solpomva7dowrhlerl
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/boost-1.74.0-hc5atqce4emwrrkxuyfayf5k43r5cst2
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gdbm-1.18.1-4av4gywgpaspkhy3dvbb62nulqogtzbb
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/libpciaccess-0.16-bob4o5m3uku6vtdil5imasprgy775zg7
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/perl-5.32.0-zfdvt2jjuaees43ffrrtphqs2ky3o22t
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/autoconf-2.69-mm33a3ocsv3jsh2tfxc4mlab4xsurtdd
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/openssl-1.1.1h-es377uqsqougfc67jyg7yfjyyuukin52
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/automake-1.16.2-d2krmb5gweivlnztcymhklzsqbrpatt6
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/cmake-3.18.4-bltycqwh5oofai4f6o42q4uuj4w5zb3j
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gmp-6.1.2-3ol3tldmm3oqflzrqonh6p2jmhgwsr4t
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/numactl-2.0.14-wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/metis-5.1.0-edcrft4l4szpqji2kowdkzq2ofueh3dr
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/glm-0.9.7.1-n4xti2gmzctc2oskcjphmruidrna4vfg
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/mpfr-4.0.2-mpv2v7v7dasis5nbrrnmci5tc2rrca2y
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/hwloc-1.11.11-zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/suite-sparse-5.8.1-rdfmg5b5lobcgmho4yqzd7zq7wibbelk
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/openmpi-3.1.6-pmsyupw6w3gql4loaor25gfumlmvkl25
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/hdf5-1.10.7-ejanivxt2gsv2qd3so7fil7izp4m3bcd
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/hypre-2.20.0-nndket2abxbsbuoocew5m643e36waxj3
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/parmetis-4.0.3-htdsyvt5yp6vl3nxbxp3627gohe7ldup
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/netlib-scalapack-2.1.0-wsiydak4jm7sms57zcu4rmzmdmgxqb5j
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/matio-1.5.17-zoqjxgkw7h3yoxbagripnd6muniihwno
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/netcdf-c-4.7.4-yixejlymvzs5aduzazrbs37hugw7sl7g
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/mumps-5.3.3-3jmjrzw7kx44fyoltswqx7uhu33zikhm
==> Installing trilinos-13.0.1-qmjbxf2kamskoplhqlejy3esksrquni2
==> Extracting trilinos-13.0.1-qmjbxf2kamskoplhqlejy3esksrquni2 from binary cache
[+] /home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/trilinos-13.0.1-qmjbxf2kamskoplhqlejy3esksrquni2
==> Updating view at /home/spack/spack/var/spack/environments/myproject/.spack-env/view
We see that tcl and all the dependencies of trilinos are
already installed. Notice also that our environment’s view gets updated.
Now confirm the contents of the environment using spack find:
$ spack find
==> In environment myproject
==> Root specs
tcl trilinos
==> 37 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 glm@0.9.7.1 libtool@2.4.6 netcdf-c@4.7.3 perl@5.30.3 xz@5.2.5
automake@1.16.2 hdf5@1.10.6 libxml2@2.9.10 netlib-scalapack@2.1.0 pkgconf@1.7.3 zlib@1.2.11
boost@1.73.0 hwloc@1.11.11 m4@1.4.18 numactl@2.0.12 readline@8.0
bzip2@1.0.8 hypre@2.18.2 matio@1.5.13 openblas@0.3.10 suite-sparse@5.7.2
cmake@3.17.3 libiconv@1.16 metis@5.1.0 openmpi@3.1.6 tcl@8.6.8
diffutils@3.7 libpciaccess@0.13.5 mumps@5.3.3 openssl@1.1.1g trilinos@12.18.1
gdbm@1.18.1 libsigsegv@2.12 ncurses@6.2 parmetis@4.0.3 util-macros@1.19.1
We now see that tcl and trilinos are root specs in
our environment. That is because we explicitly asked for them to
be installed, which makes them the roots of the combined graph
of all packages in the environment. The other installed packages
are present because they are dependencies.
Using packages¶
Environments provide a convenient way for using installed packages.
Running spack env activate gives you everything in the environment
on your PATH. Otherwise, you would need to spack load or
module load a package to use it.
When you install packages into an environment, they are, by default,
linked into a single prefix, or view. Activating the environment
with spack env activate results in subdirectories from the view
being added to PATH, LD_LIBRARY_PATH, CMAKE_PREFIX_PATH
and other environment variables. This makes the environment easier to use.
Let’s try it out. We just installed tcl into our myproject
environment. Tcl includes a shell-like application called tclsh.
You can see the path to tclsh using which:
$ which tclsh
/home/spack/spack/var/spack/environments/myproject/.spack-env/view/bin/tclsh
Notice its path includes the name of our environment and a view
subdirectory.
You can now run tclsh like you would any other program that is
in your path:
$ tclsh
% echo "hello world!"
hello world!
% exit
Similarly, you can run Trilinos’ programs. Let’s see the path for
and run algebra:
$ which algebra
/home/spack/spack/var/spack/environments/myproject/.spack-env/view/bin/algebra
$ algebra
AAAAA LL GGGGG EEEEEEE BBBBBB RRRRRR AAAAA
AA AA LL GG GG EE BB BB RR RR AA AA
AA AA LL GG EE BB BB RR RR AA AA
AAAAAAA LL GG EEEEE BBBBBB RRRRRR AAAAAAA
AA AA LL GG GGG EE BB BB RRRRR AA AA
AA AA LL GG GG EE BB BB RR RR AA AA
AA AA LLLLLLL GGGGG EEEEEEE BBBBBB RR RR AA AA
*** algebra Version 1.47 ***
Revised 2019/01/25
AN ALGEBRAIC MANIPULATION PROGRAM
FOR POST-PROCESSING OF FINITE ELEMENT ANALYSES
EXODUS II VERSION
Run on 2021-04-16 at 07:13:10
==== Email gdsjaar@sandia.gov for support ====
+++ Copyright 2008 NTESS +++
+++ Under the terms of Contract +++
+++ DE-NA0003525 with NTESS, the +++
+++ U.S. Government retains certain rights in this software. +++
FATAL ERROR - Filenames not specified.
FATAL ERROR - Syntax is: "algebra file_in file_out"
PROGRAM ERROR - Dynamic allocation problem - email code sponsor
0LAST ERROR/RETURN CODE: 1
0* * * * * E R R O R C O D E S * * * * *
0OCCURANCES FLAG MESSAGE TEXT
7 1 SUCCESSFUL COMPLETION
0 2 UNABLE TO GET REQUESTED SPACE FROM SYSTEM
0 3 DATA MANAGER NOT INITIALIZED
0 4 DATA MANAGER WAS PREVIOUSLY INITIALIZED
10 5 NAME NOT FOUND IN DICTIONARY
0 6 NAME ALREADY EXISTS IN DICTIONARY
0 7 ILLEGAL LENGTH REQUEST
0 8 UNKNOWN DATA TYPE
0 9 DICTIONARY IS FULL
0 10 VOID TABLE IS FULL
0 11 MEMORY BLOCK TABLE IS FULL
0 12 OVERLAPPING VOIDS - INTERNAL ERROR
0 13 OVERLAPPING MEMORY BLOCKS - INTERNAL ERROR
0 14 INVALID MEMORY BLOCK - EXTENSION LIBRARY ERROR
0 15 INVALID ERROR CODE
0 16 INVALID INPUT NAME
0 17 ILLEGAL CALL WHILE IN DEFERRED MODE
0 18 NAME IS OF WRONG TYPE FOR OPERATION
algebra used 0.01 seconds of CPU time
Again, we see the executable under our environment’s view.
Uninstalling packages¶
We can uninstall packages from an environment without affecting other environments. This is possible since, while Spack shares common installations, environments only link to those installations.
Let’s demonstrate this feature by creating another environment.
Suppose myproject requires trilinos but we have another
project that has it installed but no longer requires it.
Start by creating a myproject2 environment with the installed
packages hdf5+hl and trilinos:
$ spack env create myproject2
==> Updating view at /home/spack/spack/var/spack/environments/myproject2/.spack-env/view
==> Created environment 'myproject2' in /home/spack/spack/var/spack/environments/myproject2
==> You can activate this environment with:
==> spack env activate myproject2
$ spack env activate myproject2
$ spack install hdf5+hl
==> All of the packages are already installed
$ spack install trilinos
==> All of the packages are already installed
$ spack find
==> In environment myproject2
==> Root specs
hdf5 +hl trilinos
==> 40 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 gdbm@1.18.1 libsigsegv@2.12 ncurses@6.2 perl@5.32.0
autoconf-archive@2019.01.06 glm@0.9.7.1 libtool@2.4.6 netcdf-c@4.7.4 pkgconf@1.7.3
automake@1.16.2 gmp@6.1.2 libxml2@2.9.10 netlib-scalapack@2.1.0 readline@8.0
berkeley-db@18.1.40 hdf5@1.10.7 m4@1.4.18 numactl@2.0.14 suite-sparse@5.8.1
boost@1.74.0 hwloc@1.11.11 matio@1.5.17 openblas@0.3.12 trilinos@13.0.1
bzip2@1.0.8 hypre@2.20.0 metis@5.1.0 openmpi@3.1.6 util-macros@1.19.1
cmake@3.18.4 libiconv@1.16 mpfr@4.0.2 openssl@1.1.1h xz@5.2.5
diffutils@3.7 libpciaccess@0.16 mumps@5.3.3 parmetis@4.0.3 zlib@1.2.11
Notice that root specs display exactly as we asked for them on the
command line. In this case, hdf5 shows the +hl requirement.
Now we have two environments. The myproject environment has tcl
and trilinos while the myproject2 environment has hdf5 +hl
and trilinos.
Now let’s uninstall trilinos from myproject2 and review the
contents of the environment:
$ spack uninstall trilinos
y
==> The following packages will be uninstalled:
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
qmjbxf2 trilinos@13.0.1
==> Do you want to proceed? [y/N] ==> Updating view at /home/spack/spack/var/spack/environments/myproject2/.spack-env/view
$ spack find
==> In environment myproject2
==> Root specs
hdf5 +hl
==> 21 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 hdf5@1.10.7 libsigsegv@2.12 ncurses@6.2 pkgconf@1.7.3 zlib@1.2.11
automake@1.16.2 hwloc@1.11.11 libtool@2.4.6 numactl@2.0.14 readline@8.0
berkeley-db@18.1.40 libiconv@1.16 libxml2@2.9.10 openmpi@3.1.6 util-macros@1.19.1
gdbm@1.18.1 libpciaccess@0.16 m4@1.4.18 perl@5.32.0 xz@5.2.5
The result is that the environment now has only one root spec, hdf5
+hl, and contains fewer dependencies.
However, we know trilinos is still needed for the myproject
environment. So let’s switch back to confirm that it is still installed
in that environment.
$ despacktivate
$ spack env activate myproject
$ spack find
==> In environment myproject
==> Root specs
tcl trilinos
==> 41 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 glm@0.9.7.1 libxml2@2.9.10 numactl@2.0.14 tcl@8.6.10
autoconf-archive@2019.01.06 gmp@6.1.2 m4@1.4.18 openblas@0.3.12 trilinos@13.0.1
automake@1.16.2 hdf5@1.10.7 matio@1.5.17 openmpi@3.1.6 util-macros@1.19.1
berkeley-db@18.1.40 hwloc@1.11.11 metis@5.1.0 openssl@1.1.1h xz@5.2.5
boost@1.74.0 hypre@2.20.0 mpfr@4.0.2 parmetis@4.0.3 zlib@1.2.11
bzip2@1.0.8 libiconv@1.16 mumps@5.3.3 perl@5.32.0
cmake@3.18.4 libpciaccess@0.16 ncurses@6.2 pkgconf@1.7.3
diffutils@3.7 libsigsegv@2.12 netcdf-c@4.7.4 readline@8.0
gdbm@1.18.1 libtool@2.4.6 netlib-scalapack@2.1.0 suite-sparse@5.8.1
Phew! We see that myproject still has trilinos as a root
spec. Spack uses reference counting to know that trilinos is
still installed for myproject.
Note
Trilinos would only have been uninstalled by Spack if it were no longer needed by any environments or their package dependencies.
Dealing with Many Specs at Once¶
So far we have used install and uninstall for processing
individual packages. Since environments define sets of packages,
their specs can be added to the environment before they are
installed together. Specs can be added at the command line or
entered directly in the environment configuration file.
Whole environments can be installed at once by adding specs to the environment before installing them. Individual packages can still be added and removed from the environment as it evolves.
There are a couple of advantages of processing all the specs of an environment at once. First, we don’t have to write a custom installation script outside of Spack. Second, we can launch a large build of many packages in parallel by taking advantage of Spack’s install-level build parallelism.
This section focused on two ways to add specs to the environment before installing them.
Adding specs¶
Let’s start by adding a couple of specs to our myproject
environment:
$ spack env activate myproject
$ spack add hdf5+hl
==> Adding hdf5+hl to environment myproject
==> Updating view at /home/spack/spack/var/spack/environments/myproject/.spack-env/view
$ spack add gmp
==> Adding gmp to environment myproject
==> Updating view at /home/spack/spack/var/spack/environments/myproject/.spack-env/view
Now let’s take a look at what happened using spack find:
$ spack find
==> In environment myproject
==> Root specs
gmp hdf5 +hl tcl trilinos
==> 37 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 glm@0.9.7.1 libtool@2.4.6 netcdf-c@4.7.3 perl@5.30.3 xz@5.2.5
automake@1.16.2 hdf5@1.10.6 libxml2@2.9.10 netlib-scalapack@2.1.0 pkgconf@1.7.3 zlib@1.2.11
boost@1.73.0 hwloc@1.11.11 m4@1.4.18 numactl@2.0.12 readline@8.0
bzip2@1.0.8 hypre@2.18.2 matio@1.5.13 openblas@0.3.10 suite-sparse@5.7.2
cmake@3.17.3 libiconv@1.16 metis@5.1.0 openmpi@3.1.6 tcl@8.6.8
diffutils@3.7 libpciaccess@0.13.5 mumps@5.3.3 openssl@1.1.1g trilinos@12.18.1
gdbm@1.18.1 libsigsegv@2.12 ncurses@6.2 parmetis@4.0.3 util-macros@1.19.1
Notice the two specs we added, hdf5 +hl and gmp, are now
listed as root specs. They are not actually installed in the
environment yet because spack add only adds roots to the
environment.
All of the yet-to-be-installed packages can be installed in an
active environment by simply running spack install with no
arguments:
$ spack install
==> Concretized hdf5+hl
[+] ejanivx hdf5@1.10.7%gcc@7.5.0~cxx~debug~fortran+hl~java+mpi+pic+shared~szip~threadsafe api=none arch=linux-ubuntu18.04-x86_64
[+] pmsyupw ^openmpi@3.1.6%gcc@7.5.0~atomics~cuda~cxx~cxx_exceptions+gpfs~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu18.04-x86_64
[+] zqwfzhw ^hwloc@1.11.11%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-ubuntu18.04-x86_64
[+] bob4o5m ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] jdxbjft ^libtool@2.4.6%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mkc3u4x ^m4@1.4.18%gcc@7.5.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-x86_64
[+] lbrx7ln ^libsigsegv@2.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4sh6pym ^pkgconf@1.7.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] gs6ag7k ^util-macros@1.19.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] yn2r3wf ^libxml2@2.9.10%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] jearpk4 ^libiconv@1.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] komekkm ^xz@5.2.5%gcc@7.5.0~pic arch=linux-ubuntu18.04-x86_64
[+] smoyzzo ^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-x86_64
[+] wbqbc5v ^numactl@2.0.14%gcc@7.5.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-ubuntu18.04-x86_64
[+] mm33a3o ^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zfdvt2j ^perl@5.32.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] 4ihuiaz ^berkeley-db@18.1.40%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4av4gyw ^gdbm@1.18.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] t54jzdy ^readline@8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] crhlefo ^ncurses@6.2%gcc@7.5.0~symlinks+termlib arch=linux-ubuntu18.04-x86_64
[+] d2krmb5 ^automake@1.16.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
==> Concretized gmp
[+] 3ol3tld gmp@6.1.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mm33a3o ^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mkc3u4x ^m4@1.4.18%gcc@7.5.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-x86_64
[+] lbrx7ln ^libsigsegv@2.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zfdvt2j ^perl@5.32.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] 4ihuiaz ^berkeley-db@18.1.40%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4av4gyw ^gdbm@1.18.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] t54jzdy ^readline@8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] crhlefo ^ncurses@6.2%gcc@7.5.0~symlinks+termlib arch=linux-ubuntu18.04-x86_64
[+] 4sh6pym ^pkgconf@1.7.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] d2krmb5 ^automake@1.16.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] jdxbjft ^libtool@2.4.6%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
==> Installing environment myproject
==> All of the packages are already installed
==> Updating view at /home/spack/spack/var/spack/environments/myproject/.spack-env/view
Spack concretizes the new root specs before ensuring that all
the associated packages are installed. You can confirm this using
spack find:
$ spack find
==> In environment myproject
==> Root specs
gmp hdf5 +hl tcl trilinos
==> 41 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 glm@0.9.7.1 libxml2@2.9.10 numactl@2.0.14 tcl@8.6.10
autoconf-archive@2019.01.06 gmp@6.1.2 m4@1.4.18 openblas@0.3.12 trilinos@13.0.1
automake@1.16.2 hdf5@1.10.7 matio@1.5.17 openmpi@3.1.6 util-macros@1.19.1
berkeley-db@18.1.40 hwloc@1.11.11 metis@5.1.0 openssl@1.1.1h xz@5.2.5
boost@1.74.0 hypre@2.20.0 mpfr@4.0.2 parmetis@4.0.3 zlib@1.2.11
bzip2@1.0.8 libiconv@1.16 mumps@5.3.3 perl@5.32.0
cmake@3.18.4 libpciaccess@0.16 ncurses@6.2 pkgconf@1.7.3
diffutils@3.7 libsigsegv@2.12 netcdf-c@4.7.4 readline@8.0
gdbm@1.18.1 libtool@2.4.6 netlib-scalapack@2.1.0 suite-sparse@5.8.1
Configuring specs¶
An environment is more than just a list of root specs. It includes
configuration settings that affect the way Spack behaves when the
environment is activated. So far, myproject relies on configuration
defaults that can be overridden. Here we’ll look at how to add specs
and ensure all the packages depending on mpi build with mpich.
Running spack spec shows that concretization looks the same as it
does outside the environment:
$ spack spec hypre
Input spec
--------------------------------
hypre
Concretized
--------------------------------
hypre@2.20.0%gcc@7.5.0~complex~debug~int64~internal-superlu~mixedint+mpi~openmp+shared~superlu-dist patches=6e3336b1d62155f6350dfe42b0f9ea25d4fa0af60c7e540959139deb93a26059 arch=linux-ubuntu18.04-x86_64
^openblas@0.3.12%gcc@7.5.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-ubuntu18.04-x86_64
^openmpi@3.1.6%gcc@7.5.0~atomics~cuda~cxx~cxx_exceptions+gpfs~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu18.04-x86_64
^hwloc@1.11.11%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-ubuntu18.04-x86_64
^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^libtool@2.4.6%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^m4@1.4.18%gcc@7.5.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-x86_64
^libsigsegv@2.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^pkgconf@1.7.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^util-macros@1.19.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^libxml2@2.9.10%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
^libiconv@1.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^xz@5.2.5%gcc@7.5.0~pic arch=linux-ubuntu18.04-x86_64
^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-x86_64
^numactl@2.0.14%gcc@7.5.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-ubuntu18.04-x86_64
^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^perl@5.32.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
^berkeley-db@18.1.40%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^gdbm@1.18.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^readline@8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^ncurses@6.2%gcc@7.5.0~symlinks+termlib arch=linux-ubuntu18.04-x86_64
^automake@1.16.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
We can customize the selection of the mpi provider using
concretization preferences
to change the behavior of the concretizer.
Let’s start by looking at the configuration of our environment using
spack config get:
$ spack config get
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs: [tcl, trilinos, hdf5+hl, gmp]
view: true
The output shows the special YAML configuration format that Spack
uses to store the environment configuration. Currently, the file
is just a spack: header and a list of the root specs.
Note
Before proceeding, make sure your EDITOR environment variable
is set to the path of your preferred text editor.
Let’s edit this file to prefer mpich as our mpi provider
using spack config edit.
You should now have the above file open in your editor. Change it
to include the packages:all:providers:mpi: entry below:
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
packages:
all:
providers:
mpi: [mpich]
# add package specs to the `specs` list
specs: [tcl, trilinos, hdf5, gmp]
Note
This setting only defines the default mpi provider.
You can still override the provider on the command line, e.g.,
with`spack install hdf5 ^openmpi`.
We introduce this here to show you how environment configuration can affect concretization. Configuration options are covered in much more detail in the configuration tutorial.
Let’s see the effects of this change on our package using spack spec:
$ spack spec hypre
Input spec
--------------------------------
hypre
Concretized
--------------------------------
hypre@2.20.0%gcc@7.5.0~complex~debug~int64~internal-superlu~mixedint+mpi~openmp+shared~superlu-dist patches=6e3336b1d62155f6350dfe42b0f9ea25d4fa0af60c7e540959139deb93a26059 arch=linux-ubuntu18.04-x86_64
^mpich@3.3.2%gcc@7.5.0~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch3 netmod=tcp patches=eb982de3366d48cbc55eb5e0df43373a45d9f51df208abf0835a72dc6c0b4774 pmi=pmi arch=linux-ubuntu18.04-x86_64
^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^m4@1.4.18%gcc@7.5.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-x86_64
^libsigsegv@2.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^perl@5.32.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
^berkeley-db@18.1.40%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^gdbm@1.18.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^readline@8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^ncurses@6.2%gcc@7.5.0~symlinks+termlib arch=linux-ubuntu18.04-x86_64
^pkgconf@1.7.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^automake@1.16.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^findutils@4.6.0%gcc@7.5.0 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu18.04-x86_64
^libtool@2.4.6%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^texinfo@6.5%gcc@7.5.0 patches=12f6edb0c6b270b8c8dba2ce17998c580db01182d871ee32b7b6e4129bd1d23a,1732115f651cff98989cb0215d8f64da5e0f7911ebf0c13b064920f088f2ffe1 arch=linux-ubuntu18.04-x86_64
^hwloc@2.2.0%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-ubuntu18.04-x86_64
^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^util-macros@1.19.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^libxml2@2.9.10%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
^libiconv@1.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
^xz@5.2.5%gcc@7.5.0~pic arch=linux-ubuntu18.04-x86_64
^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-x86_64
^openblas@0.3.12%gcc@7.5.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-ubuntu18.04-x86_64
Notice mpich is now the mpi dependency for our concretized
spec. We also see all of its concrete dependencies.
We’ve only scratched the surface here by changing the default of
the mpi provider for packages depending on mpi. There
are many other customizations you can make to an environment.
Refer to the links at the end of this section for more information.
Reconcretizing the environment¶
You may need to re-install packages in the environment after making significant changes to the configuration, such as changing virtual providers. This can be accomplished by forcing Spack to reconcretize the environment and re-install the specs.
For example, the packages installed in our myproject environment
are now out of sync with our new configuration since we already
installed part of the environment with openmpi. Suppose we want
to install everything in myproject with mpich.
Let’s run spack concretize --force to make Spack re-concretize
all the environment’s specs:
$ spack concretize --force
==> Concretized tcl
[+] rtz7664 tcl@8.6.10%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] smoyzzo ^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-x86_64
==> Concretized trilinos
[+] xvzpvxd trilinos@13.0.1%gcc@7.5.0~adios2~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost~cgns~chaco~complex~cuda~debug~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran+glm+gtest+hdf5~hwloc+hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos+matio~mesquite+metis~minitensor+ml+mpi+muelu+mumps+netcdf~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~shylu~stk~stratimikos~strumpack+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~wrapper~x11~xsdkflags~zlib+zoltan+zoltan2 build_type=RelWithDebInfo cuda_arch=none cxxstd=11 gotype=long arch=linux-ubuntu18.04-x86_64
[+] hc5atqc ^boost@1.74.0%gcc@7.5.0+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 visibility=hidden arch=linux-ubuntu18.04-x86_64
[+] fvfpt26 ^bzip2@1.0.8%gcc@7.5.0+shared arch=linux-ubuntu18.04-x86_64
[+] otkkten ^diffutils@3.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] jearpk4 ^libiconv@1.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] smoyzzo ^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-x86_64
[+] bltycqw ^cmake@3.18.4%gcc@7.5.0~doc+ncurses+openssl+ownlibs~qt patches=bf695e3febb222da2ed94b3beea600650e4318975da90e4a71d6f31a6d5d8c3d arch=linux-ubuntu18.04-x86_64
[+] crhlefo ^ncurses@6.2%gcc@7.5.0~symlinks+termlib arch=linux-ubuntu18.04-x86_64
[+] 4sh6pym ^pkgconf@1.7.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] es377uq ^openssl@1.1.1h%gcc@7.5.0+systemcerts arch=linux-ubuntu18.04-x86_64
[+] zfdvt2j ^perl@5.32.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] 4ihuiaz ^berkeley-db@18.1.40%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4av4gyw ^gdbm@1.18.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] t54jzdy ^readline@8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] n4xti2g ^glm@0.9.7.1%gcc@7.5.0~ipo build_type=RelWithDebInfo arch=linux-ubuntu18.04-x86_64
[+] cq4k4cv ^hdf5@1.10.7%gcc@7.5.0~cxx~debug~fortran+hl~java+mpi+pic+shared~szip~threadsafe api=none arch=linux-ubuntu18.04-x86_64
[+] vbelg5s ^mpich@3.3.2%gcc@7.5.0~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch3 netmod=tcp patches=eb982de3366d48cbc55eb5e0df43373a45d9f51df208abf0835a72dc6c0b4774 pmi=pmi arch=linux-ubuntu18.04-x86_64
[+] mm33a3o ^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mkc3u4x ^m4@1.4.18%gcc@7.5.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-x86_64
[+] lbrx7ln ^libsigsegv@2.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] d2krmb5 ^automake@1.16.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] hrv4dx5 ^findutils@4.6.0%gcc@7.5.0 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu18.04-x86_64
[+] jdxbjft ^libtool@2.4.6%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] p46ba5q ^texinfo@6.5%gcc@7.5.0 patches=12f6edb0c6b270b8c8dba2ce17998c580db01182d871ee32b7b6e4129bd1d23a,1732115f651cff98989cb0215d8f64da5e0f7911ebf0c13b064920f088f2ffe1 arch=linux-ubuntu18.04-x86_64
[+] a4cxlu7 ^hwloc@2.2.0%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-ubuntu18.04-x86_64
[+] bob4o5m ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] gs6ag7k ^util-macros@1.19.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] yn2r3wf ^libxml2@2.9.10%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] komekkm ^xz@5.2.5%gcc@7.5.0~pic arch=linux-ubuntu18.04-x86_64
[+] us6giii ^hypre@2.20.0%gcc@7.5.0~complex~debug~int64~internal-superlu~mixedint+mpi~openmp+shared~superlu-dist patches=6e3336b1d62155f6350dfe42b0f9ea25d4fa0af60c7e540959139deb93a26059 arch=linux-ubuntu18.04-x86_64
[+] te35tjx ^openblas@0.3.12%gcc@7.5.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-ubuntu18.04-x86_64
[+] usmrkqu ^matio@1.5.17%gcc@7.5.0+hdf5+shared+zlib arch=linux-ubuntu18.04-x86_64
[+] edcrft4 ^metis@5.1.0%gcc@7.5.0~gdb~int64~real64+shared build_type=Release patches=4991da938c1d3a1d3dea78e49bbebecba00273f98df2a656e38b83d55b281da1,b1225da886605ea558db7ac08dd8054742ea5afe5ed61ad4d0fe7a495b1270d2 arch=linux-ubuntu18.04-x86_64
[+] rmlof2f ^mumps@5.3.3%gcc@7.5.0+complex+double+float~int64~metis+mpi~parmetis~ptscotch~scotch+shared arch=linux-ubuntu18.04-x86_64
[+] h7suxc3 ^netlib-scalapack@2.1.0%gcc@7.5.0~ipo~pic+shared build_type=Release patches=1c9ce5fee1451a08c2de3cc87f446aeda0b818ebbce4ad0d980ddf2f2a0b2dc4,f2baedde688ffe4c20943c334f580eb298e04d6f35c86b90a1f4e8cb7ae344a2 arch=linux-ubuntu18.04-x86_64
[+] spkgtxp ^netcdf-c@4.7.4%gcc@7.5.0~dap~hdf4~jna+mpi~parallel-netcdf+pic+shared arch=linux-ubuntu18.04-x86_64
[+] i7otoak ^parmetis@4.0.3%gcc@7.5.0~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d arch=linux-ubuntu18.04-x86_64
[+] rdfmg5b ^suite-sparse@5.8.1%gcc@7.5.0~cuda~openmp+pic~tbb arch=linux-ubuntu18.04-x86_64
[+] 3ol3tld ^gmp@6.1.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mpv2v7v ^mpfr@4.0.2%gcc@7.5.0 patches=3f80b836948aa96f8d1cb9cc7f3f55973f19285482a96f9a4e1623d460bcccf0 arch=linux-ubuntu18.04-x86_64
[+] bdyarrk ^autoconf-archive@2019.01.06%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
==> Concretized hdf5+hl
[+] cq4k4cv hdf5@1.10.7%gcc@7.5.0~cxx~debug~fortran+hl~java+mpi+pic+shared~szip~threadsafe api=none arch=linux-ubuntu18.04-x86_64
[+] vbelg5s ^mpich@3.3.2%gcc@7.5.0~argobots+fortran+hwloc+hydra+libxml2+pci+romio~slurm~verbs+wrapperrpath device=ch3 netmod=tcp patches=eb982de3366d48cbc55eb5e0df43373a45d9f51df208abf0835a72dc6c0b4774 pmi=pmi arch=linux-ubuntu18.04-x86_64
[+] mm33a3o ^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mkc3u4x ^m4@1.4.18%gcc@7.5.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-x86_64
[+] lbrx7ln ^libsigsegv@2.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zfdvt2j ^perl@5.32.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] 4ihuiaz ^berkeley-db@18.1.40%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4av4gyw ^gdbm@1.18.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] t54jzdy ^readline@8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] crhlefo ^ncurses@6.2%gcc@7.5.0~symlinks+termlib arch=linux-ubuntu18.04-x86_64
[+] 4sh6pym ^pkgconf@1.7.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] d2krmb5 ^automake@1.16.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] hrv4dx5 ^findutils@4.6.0%gcc@7.5.0 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu18.04-x86_64
[+] jdxbjft ^libtool@2.4.6%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] p46ba5q ^texinfo@6.5%gcc@7.5.0 patches=12f6edb0c6b270b8c8dba2ce17998c580db01182d871ee32b7b6e4129bd1d23a,1732115f651cff98989cb0215d8f64da5e0f7911ebf0c13b064920f088f2ffe1 arch=linux-ubuntu18.04-x86_64
[+] a4cxlu7 ^hwloc@2.2.0%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-ubuntu18.04-x86_64
[+] bob4o5m ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] gs6ag7k ^util-macros@1.19.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] yn2r3wf ^libxml2@2.9.10%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] jearpk4 ^libiconv@1.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] komekkm ^xz@5.2.5%gcc@7.5.0~pic arch=linux-ubuntu18.04-x86_64
[+] smoyzzo ^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-x86_64
==> Concretized gmp
[+] 3ol3tld gmp@6.1.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mm33a3o ^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mkc3u4x ^m4@1.4.18%gcc@7.5.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-x86_64
[+] lbrx7ln ^libsigsegv@2.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zfdvt2j ^perl@5.32.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] 4ihuiaz ^berkeley-db@18.1.40%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4av4gyw ^gdbm@1.18.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] t54jzdy ^readline@8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] crhlefo ^ncurses@6.2%gcc@7.5.0~symlinks+termlib arch=linux-ubuntu18.04-x86_64
[+] 4sh6pym ^pkgconf@1.7.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] d2krmb5 ^automake@1.16.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] jdxbjft ^libtool@2.4.6%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
==> Updating view at /home/spack/spack/var/spack/environments/myproject/.spack-env/view
All the specs are now concrete and ready to be installed with
mpich as the MPI implementation. You can now re-run spack
install to finish the process.
Building in environments¶
Activated environments allow you to invoke any programs installed in them as if they were installed on the system. In this section we will take advantage of that feature.
Suppose you want to compile some MPI programs. We have an MPI
implementation installed in our myproject environment, so
mpicc is available in our path. We can confirm this using
which:
$ spack env status
==> In environment myproject
$ which mpicc
/home/spack/spack/var/spack/environments/myproject/.spack-env/view/bin/mpicc
As mentioned before, activating the environment sets a number of
environment variables. That includes variables like CPATH,
LIBRARY_PATH, and LD_LIBRARY_PATH, which allows you to
easily find package headers and libraries installed in the environment.
Let’s look specifically at path-related environment variables using
env | grep PATH:
$ env | grep PATH=
LIBRARY_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib64
C_INCLUDE_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/include
LD_LIBRARY_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib64
CPLUS_INCLUDE_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/include
PKG_CONFIG_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib/pkgconfig:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/share/pkgconfig:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib64/pkgconfig
ACLOCAL_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/share/aclocal
PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/bin:/home/spack/spack/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib
SPACK_LD_LIBRARY_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/lib64
MANPATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view/share/man:/home/spack/spack/var/spack/environments/myproject/.spack-env/view/man
MODULEPATH=/home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64
CMAKE_PREFIX_PATH=/home/spack/spack/var/spack/environments/myproject/.spack-env/view
We can demonstrate use of these environment settings by building a really simple MPI program.
Let’s create a program called mpi-hello.c that contains the following
code:
#include <stdio.h>
#include <mpi.h>
#include <zlib.h>
int main(int argc, char **argv) {
int rank;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
printf("Hello world from rank %d\n", rank);
if (rank == 0) {
printf("zlib version: %s\n", ZLIB_VERSION);
}
MPI_Finalize();
}
This program includes headers from mpi and zlib.
It also prints out a message from each MPI rank and the
version of zlib.
Let’s build and run our program:
$ mpicc ./mpi-hello.c
$ mpirun -n 4 ./a.out
Hello world from rank 3
Hello world from rank 0
zlib version: 1.2.11
Hello world from rank 1
Hello world from rank 2
Notice that we did not need to pass any special arguments
to the compiler, such as include paths or libraries. We also
see that Hello world is output for each of the ranks and
the version of zlib used to build the program is printed.
We can confirm the version of zlib used to build the program
is in our environment using spack find:
$ spack find zlib
==> In environment myproject
==> Root specs
gmp hdf5 +hl tcl trilinos
==> 1 installed package
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 ----------------
zlib@1.2.11
Note that the reported version does match that of our installation.
Reproducing builds¶
Spack environments provide users with virtual environments similar to Python venv and Conda environments). The goal is to ensure packages in one environment are kept separate from those of another. These environments can be managed by spack or independent. In either case, their environment files can be used to reproduce builds by other users and on other machines. Since those files are key to reproducing builds, let’s start with them.
Environment files¶
There are two key files tracking the contents of environments:
spack.yaml and spack.lock. The spack.yaml file holds
the environment configuration that we previously edited through
spack config edit. The spack.lock file is automatically
generated during concretization.
The two files represent two fundamental concepts:
spack.yaml: abstract specs and configuration to install; andspack.lock: all fully concrete specs.
These files are intended to be used by developers and administrators to manage the environments in a reproducible way. We will cover their re-use later.
Note
Both environment files can be versioned in repositories, shared, and used to install the same set of software by different users and on other machines.
Managed versus independent environments¶
Environments are either spack-managed or independent. Both types of environments are defined by their environment files. So far we have only created managed environments. This section describes their differences.
Managed environments are created using spack env create <name>.
They are automatically created in the var/spack/environments
subdirectory and can be referenced by their names.
Independent environments can be created in one of two ways. First,
the Spack’s environment file(s) can be placed in any directory
(other than var/spack/environments). Alternatively, you can
use spack env create -d <directory> to specify the directory
(<directory>) in which the files should reside. Independent
environments are not named.
Reviewing a managed environment¶
We created our currently active environment, myproject, earlier
using spack env create myproject so let’s mainly focus on its
environment files in this section.
Earlier, when we changed our environment’s configuration using spack
config edit, we were actually editing its spack.yaml file. We
can change to the directory containing the file using spack cd:
$ spack cd -e myproject
$ pwd
/home/spack/spack/var/spack/environments/myproject
$ ls
spack.lock spack.yaml
Notice myproject is a subdirectory of var/spack/environments
within the Spack installation making it a managed environment.
Consequently, it can be referenced by name. It will also show up
when running spack env list:
$ spack env list
==> 2 environments
myproject myproject2
You can see that myproject is active because it is highlighted
in green.
We can also see from the listing that the environment directory
contains both of the environment files: spack.yaml and spack.lock.
This is because spack.lock was generated when we concretized
the environment.
If we cat the spack.yaml file, we’ll see the same contents
shown previously by spack config get:
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs: [tcl, trilinos, hdf5+hl, gmp]
view: true
Creating an independent environment¶
Environments do not have to be created in or managed by a Spack instance. Rather, their environment files can be placed in any directory. This feature can be quite helpful for use cases such as environment-based software releases and CI/CD.
Let’s create an independent environment from scratch for a simple project:
$ cd
$ mkdir code
$ cd code
$ spack env create -d .
==> Updating view at /home/spack/code/.spack-env/view
==> Created environment in /home/spack/code
==> You can activate this environment with:
==> spack env activate /home/spack/code
Notice the command shows Spack created the environment, updated the view, and printed the command needed to activate it. As we can see in the activation command, since the environment is independent, it must be referenced by its directory path.
Let’s see what really happened with this command by listing the directory contents and looking at the configuration file:
$ ls
spack.yaml
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs: []
view: true
Notice Spack created a spack.yaml file in the code directory.
Also note that the configuration file has an empty spec list (i.e.,
[]). That list is intended to contain only the root specs of
the environment.
We can confirm that it is not a managed environment by running
spack env list:
$ spack env list
==> 2 environments
myproject myproject2
and noting that the path does not appear in the output.
Now let’s add some specs to the environment. Suppose your project
depends on boost, trilinos, and openmpi. Add these
packages to the spec list using your favorite text editor. The
dash syntax for a YAML list is used in our example. Your package
should now contain the following entries:
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs:
- boost
- trilinos
- openmpi
view: true
Now activate the environment and install the packages:
$ spack env activate .
$ spack install
==> Concretized boost
[+] hc5atqc boost@1.74.0%gcc@7.5.0+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 visibility=hidden arch=linux-ubuntu18.04-x86_64
[+] fvfpt26 ^bzip2@1.0.8%gcc@7.5.0+shared arch=linux-ubuntu18.04-x86_64
[+] otkkten ^diffutils@3.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] jearpk4 ^libiconv@1.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] smoyzzo ^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-x86_64
==> Concretized trilinos
[+] qmjbxf2 trilinos@13.0.1%gcc@7.5.0~adios2~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost~cgns~chaco~complex~cuda~debug~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran+glm+gtest+hdf5~hwloc+hypre+ifpack+ifpack2~intrepid~intrepid2~ipo~isorropia+kokkos+matio~mesquite+metis~minitensor+ml+mpi+muelu+mumps+netcdf~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~shylu~stk~stratimikos~strumpack+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~wrapper~x11~xsdkflags~zlib+zoltan+zoltan2 build_type=RelWithDebInfo cuda_arch=none cxxstd=11 gotype=long arch=linux-ubuntu18.04-x86_64
[+] hc5atqc ^boost@1.74.0%gcc@7.5.0+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 visibility=hidden arch=linux-ubuntu18.04-x86_64
[+] fvfpt26 ^bzip2@1.0.8%gcc@7.5.0+shared arch=linux-ubuntu18.04-x86_64
[+] otkkten ^diffutils@3.7%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] jearpk4 ^libiconv@1.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] smoyzzo ^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-x86_64
[+] bltycqw ^cmake@3.18.4%gcc@7.5.0~doc+ncurses+openssl+ownlibs~qt patches=bf695e3febb222da2ed94b3beea600650e4318975da90e4a71d6f31a6d5d8c3d arch=linux-ubuntu18.04-x86_64
[+] crhlefo ^ncurses@6.2%gcc@7.5.0~symlinks+termlib arch=linux-ubuntu18.04-x86_64
[+] 4sh6pym ^pkgconf@1.7.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] es377uq ^openssl@1.1.1h%gcc@7.5.0+systemcerts arch=linux-ubuntu18.04-x86_64
[+] zfdvt2j ^perl@5.32.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] 4ihuiaz ^berkeley-db@18.1.40%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4av4gyw ^gdbm@1.18.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] t54jzdy ^readline@8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] n4xti2g ^glm@0.9.7.1%gcc@7.5.0~ipo build_type=RelWithDebInfo arch=linux-ubuntu18.04-x86_64
[+] ejanivx ^hdf5@1.10.7%gcc@7.5.0~cxx~debug~fortran+hl~java+mpi+pic+shared~szip~threadsafe api=none arch=linux-ubuntu18.04-x86_64
[+] pmsyupw ^openmpi@3.1.6%gcc@7.5.0~atomics~cuda~cxx~cxx_exceptions+gpfs~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu18.04-x86_64
[+] zqwfzhw ^hwloc@1.11.11%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-ubuntu18.04-x86_64
[+] bob4o5m ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] jdxbjft ^libtool@2.4.6%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mkc3u4x ^m4@1.4.18%gcc@7.5.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-x86_64
[+] lbrx7ln ^libsigsegv@2.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] gs6ag7k ^util-macros@1.19.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] yn2r3wf ^libxml2@2.9.10%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] komekkm ^xz@5.2.5%gcc@7.5.0~pic arch=linux-ubuntu18.04-x86_64
[+] wbqbc5v ^numactl@2.0.14%gcc@7.5.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-ubuntu18.04-x86_64
[+] mm33a3o ^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] d2krmb5 ^automake@1.16.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] nndket2 ^hypre@2.20.0%gcc@7.5.0~complex~debug~int64~internal-superlu~mixedint+mpi~openmp+shared~superlu-dist patches=6e3336b1d62155f6350dfe42b0f9ea25d4fa0af60c7e540959139deb93a26059 arch=linux-ubuntu18.04-x86_64
[+] te35tjx ^openblas@0.3.12%gcc@7.5.0~consistent_fpcsr~ilp64+pic+shared threads=none arch=linux-ubuntu18.04-x86_64
[+] zoqjxgk ^matio@1.5.17%gcc@7.5.0+hdf5+shared+zlib arch=linux-ubuntu18.04-x86_64
[+] edcrft4 ^metis@5.1.0%gcc@7.5.0~gdb~int64~real64+shared build_type=Release patches=4991da938c1d3a1d3dea78e49bbebecba00273f98df2a656e38b83d55b281da1,b1225da886605ea558db7ac08dd8054742ea5afe5ed61ad4d0fe7a495b1270d2 arch=linux-ubuntu18.04-x86_64
[+] 3jmjrzw ^mumps@5.3.3%gcc@7.5.0+complex+double+float~int64~metis+mpi~parmetis~ptscotch~scotch+shared arch=linux-ubuntu18.04-x86_64
[+] wsiydak ^netlib-scalapack@2.1.0%gcc@7.5.0~ipo~pic+shared build_type=Release patches=1c9ce5fee1451a08c2de3cc87f446aeda0b818ebbce4ad0d980ddf2f2a0b2dc4,f2baedde688ffe4c20943c334f580eb298e04d6f35c86b90a1f4e8cb7ae344a2 arch=linux-ubuntu18.04-x86_64
[+] yixejly ^netcdf-c@4.7.4%gcc@7.5.0~dap~hdf4~jna+mpi~parallel-netcdf+pic+shared arch=linux-ubuntu18.04-x86_64
[+] htdsyvt ^parmetis@4.0.3%gcc@7.5.0~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d arch=linux-ubuntu18.04-x86_64
[+] rdfmg5b ^suite-sparse@5.8.1%gcc@7.5.0~cuda~openmp+pic~tbb arch=linux-ubuntu18.04-x86_64
[+] 3ol3tld ^gmp@6.1.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mpv2v7v ^mpfr@4.0.2%gcc@7.5.0 patches=3f80b836948aa96f8d1cb9cc7f3f55973f19285482a96f9a4e1623d460bcccf0 arch=linux-ubuntu18.04-x86_64
[+] bdyarrk ^autoconf-archive@2019.01.06%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
==> Concretized openmpi
[+] pmsyupw openmpi@3.1.6%gcc@7.5.0~atomics~cuda~cxx~cxx_exceptions+gpfs~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu18.04-x86_64
[+] zqwfzhw ^hwloc@1.11.11%gcc@7.5.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml+pci+shared arch=linux-ubuntu18.04-x86_64
[+] bob4o5m ^libpciaccess@0.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] jdxbjft ^libtool@2.4.6%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] mkc3u4x ^m4@1.4.18%gcc@7.5.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-ubuntu18.04-x86_64
[+] lbrx7ln ^libsigsegv@2.12%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4sh6pym ^pkgconf@1.7.3%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] gs6ag7k ^util-macros@1.19.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] yn2r3wf ^libxml2@2.9.10%gcc@7.5.0~python arch=linux-ubuntu18.04-x86_64
[+] jearpk4 ^libiconv@1.16%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] komekkm ^xz@5.2.5%gcc@7.5.0~pic arch=linux-ubuntu18.04-x86_64
[+] smoyzzo ^zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=linux-ubuntu18.04-x86_64
[+] wbqbc5v ^numactl@2.0.14%gcc@7.5.0 patches=4e1d78cbbb85de625bad28705e748856033eaafab92a66dffd383a3d7e00cc94 arch=linux-ubuntu18.04-x86_64
[+] mm33a3o ^autoconf@2.69%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] zfdvt2j ^perl@5.32.0%gcc@7.5.0+cpanm+shared+threads arch=linux-ubuntu18.04-x86_64
[+] 4ihuiaz ^berkeley-db@18.1.40%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] 4av4gyw ^gdbm@1.18.1%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] t54jzdy ^readline@8.0%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
[+] crhlefo ^ncurses@6.2%gcc@7.5.0~symlinks+termlib arch=linux-ubuntu18.04-x86_64
[+] d2krmb5 ^automake@1.16.2%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64
==> Installing environment /home/spack/code
==> All of the packages are already installed
==> Updating view at /home/spack/code/.spack-env/view
Notice Spack concretized the specs before installing them and their dependencies. It also updated the environment’s view. Since we already installed all these packages outside of the environment, their links were added to our environment.
Updating an installed environment¶
Spack supports tweaking an environment even after the initial specs are installed. You are free to add and remove specs just as you would outside of the environment using the command line interface as before.
For example, let’s add hdf5 and look at our file:
$ spack add hdf5@5.5.1
==> Adding hdf5@5.5.1 to environment /home/spack/code
==> Updating view at /home/spack/code/.spack-env/view
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs:
- boost
- trilinos
- openmpi
- hdf5@5.5.1
view: true
Notice spack add added the package to our active environment and
it appears in the configuration file’s spec list.
Note
You’ll need to run spack install to install added packages
in your environment because spack add only adds it to the
configuraton.
Now use spack remove to remove the spec from the configuration:
$ spack remove hdf5
==> Removing hdf5 from environment /home/spack/code
==> Updating view at /home/spack/code/.spack-env/view
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs: [boost, trilinos, openmpi]
view: true
and we see the spec was removed from the spec list of our environment.
Note
You can also edit the spack.yaml file directly instead of
using the spack add and spack remove commands.
Reviewing spack.lock¶
Now let’s turn our attention from the abstract to the concrete.
Our focus so far has been on the abstract environment configuration
represented by the spack.yaml file. Once that file is concretized,
Spack generates a corresponding spack.lock file representing
the full concretized state of the environment.
This file is intended to be a machine-readable representation of the
information needed to reproduce the build of an environment. As such,
it is written in json, which is less readable than yaml.
Let’s look at the top 30 lines of our current environment:
$ head -30 spack.lock
{
"_meta": {
"file-type": "spack-lockfile",
"lockfile-version": 2
},
"roots": [
{
"hash": "iwji2vty6m7hp27zbzufewdhq6pty3fx",
"spec": "boost"
},
{
"hash": "246z4kkzk6vsg2ot3rzlz5jwwimxj63g",
"spec": "trilinos"
},
{
"hash": "7o7eidaryeh72cizcbpvvjnq7ojjya6f",
"spec": "openmpi"
}
],
"concrete_specs": {
"iwji2vty6m7hp27zbzufewdhq6pty3fx": {
"boost": {
"version": "1.74.0",
"arch": {
"platform": "linux",
"platform_os": "ubuntu18.04",
"target": "x86_64"
},
"compiler": {
"name": "gcc",
While it is still readable, it consists of nearly 2000 lines of of information representing the actual configurations for each of the environment’s packages.
Reproducing an environment¶
Now that we’ve described the contents of the environment files we can discuss how they can be used to reproduce environments. You may want to do this yourself on a different machine, or use an environment built by someone else. The process is the same in either case.
You can recreate an environment by passing either of the environment
files to spack env create. The file you choose depends on whether
you want to approximate the build using the abstract specs or an exact
build based on the concrete specs.
Using spack.yaml¶
An approximate build is created using the spack.yaml file. This
approach is relevant when we want to build the same specs on a new
platform, for example. It allows you to reproduce the environment
by preserving the abstract requirements in the file. However, the
software may actually build differently in part because the concretizer
may choose different dependencies.
Let’s use spack env create to create an abstract environment from
the file that we’ll call abstract:
$ spack env create abstract spack.yaml
==> Updating view at /home/spack/spack/var/spack/environments/abstract/.spack-env/view
==> Created environment 'abstract' in /home/spack/spack/var/spack/environments/abstract
==> You can activate this environment with:
==> spack env activate abstract
Here we see that Spack created a managed environment with the name we provided.
And, since it is a newly created environment, it does not have any
installed specs yet as we can see from calling spack find
after activating the environment:
$ spacktivate abstract
$ spack find
==> In environment abstract
==> Root specs
boost openmpi trilinos
==> 0 installed packages
Notice we have the same root specs as were listed in the spack.yaml
file.
Using spack.lock¶
The spack.lock file is used for an exact reproduction of the
original build. It can replicate the build because it contains the
information for all the decisions made during concretization.
Now let’s create a concrete environment, called concrete, from
the file again after activating the environment:
$ spack env create concrete spack.lock
==> Updating view at /home/spack/spack/var/spack/environments/concrete/.spack-env/view
==> Created environment 'concrete' in /home/spack/spack/var/spack/environments/concrete
==> You can activate this environment with:
==> spack env activate concrete
Here we see that Spack again created a managed environment with the provided name.
Since we created the environment from our spack.lock file,
not only do we get the same root specs, all of the packages are
also installed in the environment as we can see from calling
spack find after activating the environment:
$ spacktivate concrete
$ spack find
==> In environment concrete
==> Root specs
boost openmpi trilinos
==> 40 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69 libiconv@1.16 openblas@0.3.12
autoconf-archive@2019.01.06 libpciaccess@0.16 openmpi@3.1.6
automake@1.16.2 libsigsegv@2.12 openssl@1.1.1h
berkeley-db@18.1.40 libtool@2.4.6 parmetis@4.0.3
boost@1.74.0 libxml2@2.9.10 perl@5.32.0
bzip2@1.0.8 m4@1.4.18 pkgconf@1.7.3
cmake@3.18.4 matio@1.5.17 readline@8.0
diffutils@3.7 metis@5.1.0 suite-sparse@5.8.1
gdbm@1.18.1 mpfr@4.0.2 trilinos@13.0.1
glm@0.9.7.1 mumps@5.3.3 util-macros@1.19.1
gmp@6.1.2 ncurses@6.2 xz@5.2.5
hdf5@1.10.7 netcdf-c@4.7.4 zlib@1.2.11
hwloc@1.11.11 netlib-scalapack@2.1.0
hypre@2.20.0 numactl@2.0.14
Note
Use of spack.lock to reproduce a build (currently) requires you
to be on the same type of machine.
More information¶
This tutorial only scratches the surface of environments and what they can do. For more information, take a look at the Spack resources below.
Setting up and building environments¶
- Environments: reference docs
- Configuration tutorial: for customizing your environment
- Spack stacks tutorial: for configuring combinatorial environments (e.g., same packages across a list of compilers)
- Install-level parallel builds:
for how to launch
spack installto build your environment in parallel
Using environments¶
- Developer workflows: for developing code in an environment
- GitLab CI pipelines with Spack environments: for using environments to generate CI pipelines
- Container Images: for creating containers from environments
- Spack stacks tutorial: for managing large deployments of software
Finding examples of environments¶
- Spack Stack Catalog: for discovering environments that you can explore on GitHub