Tutorial setup
If you have not done the prior sections, you’ll need to set Spack up like this:
git clone https://github.com/spack/spack
. spack/share/spack/setup-env.sh
spack tutorial
See the Basic Installation Tutorial for full details on setup. For more
help join us in the #tutorial
channel on Slack – get an
invitation at spackpm.herokuapp.com
Module Files Tutorial¶
This tutorial illustrates how Spack can be used to generate module files for the software that has been installed. Both hierarchical and non-hierarchical deployments will be discussed in details and we will show how to customize the content and naming of each module file.
At the end of the tutorial readers should have a clear understanding of:
- What module files are and how they are used on HPC clusters
- How Spack generates module files for the software it installs
- Which Spack commands can be used to manage module files
- How module files generated by Spack can be customized
and be confident that Spack can deal with all of the common use cases that occur when maintaining software installations on HPC clusters.
Setup for the Tutorial¶
To prepare for this tutorial we are going to install a small but representative set of software that includes different configurations of the same packages and some external packages. To keep the installations manageable, let’s start by uninstalling everything from earlier in the tutorial:
$ spack uninstall -ay
Build a module tool¶
The first thing that we need is the module tool itself. In the tutorial we will use
lmod
because it can work with both hierarchical and non-hierarchical layouts.
$ spack install lmod
Once the module tool is installed we need to have it available in the
current shell. Installation directories in Spack’s store are definitely not easy
to remember, but they can be retrieved with the spack location
command:
$ . $(spack location -i lmod)/lmod/lmod/init/bash
Now we can re-source the setup file and Spack modules will be put in our module path.
$ . share/spack/setup-env.sh
Add a new compiler¶
The second step is to build a recent compiler. On first use, Spack
scans the environment and automatically locates the compiler(s)
already available on the system. For this tutorial, however, we want
to use gcc@8.3.0
.
$ spack install gcc@8.3.0
You can get this in your environment using spack load gcc@8.3.0
:
$ spack load gcc@8.3.0
$ which gcc
/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gcc
Now, gcc
is in your PATH
, and you can add it to the list of
compilers with spack compiler add
:
$ spack compiler add
==> Added 1 new compiler to /home/spack/.spack/linux/compilers.yaml
gcc@8.3.0
==> Compilers are defined in the following files:
/home/spack/.spack/linux/compilers.yaml
$ spack compiler list
==> Available compilers
-- clang ubuntu18.04-x86_64 -------------------------------------
clang@6.0.0
-- gcc ubuntu18.04-x86_64 ---------------------------------------
gcc@8.3.0 gcc@7.5.0 gcc@6.5.0
Build the software that will be used in the tutorial¶
Finally, we will use Spack to install the packages used in the examples:
$ spack install netlib-scalapack ^openmpi ^openblas
$ spack install netlib-scalapack ^mpich ^openblas
$ spack install netlib-scalapack ^openmpi ^netlib-lapack
$ spack install netlib-scalapack ^mpich ^netlib-lapack
$ spack install py-scipy ^openblas
What are Module Files?¶
Module files are an easy way to modify your environment in a controlled
manner during a shell session. In general, they contain the information
needed to run an application or use a library. The module
command is
used to interpret and execute module files. For example, module show
tells you what a module will do when loaded:
$ module show zlib
----------------------------------------------------------------------------------------------------------------------------------------------
/home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/zlib/1.2.11-gcc-8.3.0:
----------------------------------------------------------------------------------------------------------------------------------------------
whatis("A free, general-purpose, legally unencumbered lossless data-compression library. ")
conflict("zlib")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/zlib-1.2.11-2icaxiyy5rrqdavfv7jbojdq36r6u37n/share/man")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/zlib-1.2.11-2icaxiyy5rrqdavfv7jbojdq36r6u37n/lib")
prepend_path("INCLUDE","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/zlib-1.2.11-2icaxiyy5rrqdavfv7jbojdq36r6u37n/include")
prepend_path("PKG_CONFIG_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/zlib-1.2.11-2icaxiyy5rrqdavfv7jbojdq36r6u37n/lib/pkgconfig")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/zlib-1.2.11-2icaxiyy5rrqdavfv7jbojdq36r6u37n/")
help([[A free, general-purpose, legally unencumbered lossless data-compression
library.
]])
$ echo $LD_LIBRARY_PATH
module load
will execute all of the changes shown above:
$ module load zlib
$ echo $LD_LIBRARY_PATH
/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/zlib-1.2.11-2icaxiyy5rrqdavfv7jbojdq36r6u37n/lib
and to undo the modifications, you can use module unload
:
$ module unload zlib
$ echo $LD_LIBRARY_PATH
$
Module Systems¶
There are two main module systems used in HPC, both installable by Spack.
In this tutorial we will be working with lmod
and be showing examples
with both Tcl and Lua.
Environment Modules¶
This is the original modules tool. It can be installed with Spack using the following command:
$ spack install environment-modules
It was first coded in C in the early 1990s and was later rewritten entirely in Tcl. Long stagnant, the project has been revived in the past few years by Xavier Delaruelle at CEA, and it is now very actively developed. For further details we refer to its documentation.
Lmod¶
Lmod is a module system written in Lua, originally created at the “Texas Advanced Computing Center” (TACC) by Robert McLay. You can get it with:
$ spack install lmod
as shown in the Setup for the Tutorial section. It is a drop-in replacement for Environment Modules, and it works with both Tcl and Lua module files. It is fully compatible with Environment Modules, but it also has many distinguishing features of its own. The main one is the module hierarchy, which simplifies the module UI by only showing modules built with the currently loaded compiler and/or MPI. There are also some unique safety features.
How does Spack generate module files?¶
Before we dive into the hands-on sections it’s worth explaining how module files are generated by Spack. The following diagram provides a high-level view of the process:
Modules in Spack are generated using configuration files (config.yaml
and modules.yaml
), information from Spack’s package recipes, and
Jinja2 templates. Spack comes with Jinja2, an external template engine, so you
do not need to install it yourself.
Modules vs spack load
¶
You may have noticed that we used spack load
in the
Setup for the Tutorial section above. This is a
built-in mechanism of Spack’s – it’s designed so that users on a cluster
or a laptop can quickly get a package into their path, and it understands
Spack’s spec syntax. It does not require modules, as Spack needs to
work regardless of whether modules are set up on the system.
As you might expect, you can see what is loaded via spack load
using
spack find
:
$ spack find --loaded
==> 6 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
gcc@8.3.0 gmp@6.1.2 isl@0.18 mpc@1.1.0 mpfr@3.1.6 zlib@1.2.11
Because Spack is designed to be run on HPC systems, it also generates a module file for every installed package. This allows users unfamiliar with Spack’s interface to see things through the module system they’re used to. To see this, try:
$ module avail
----------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------
autoconf-2.69-gcc-7.5.0-nwb3bf5 libiconv-1.16-gcc-7.5.0-jearpk4 ncurses-6.2-gcc-7.5.0-crhlefo
automake-1.16.2-gcc-7.5.0-wdflovn libidn2-2.1.1a-gcc-7.5.0-gcbfo3a openssl-1.1.1g-gcc-7.5.0-4dj34yw
bzip2-1.0.8-gcc-7.5.0-fvfpt26 libsigsegv-2.12-gcc-7.5.0-lbrx7ln patchelf-0.10-gcc-7.5.0-2nj3vxx
curl-7.68.0-gcc-7.5.0-e4rrsab libtool-2.4.6-gcc-7.5.0-jdxbjft pcre2-10.35-gcc-7.5.0-xyujt3h
diffutils-3.7-gcc-7.5.0-otkkten libunistring-0.9.10-gcc-7.5.0-dyq3apq perl-5.30.3-gcc-7.5.0-hyrsxn4
expat-2.2.9-gcc-7.5.0-3f3kkbl libxml2-2.9.10-gcc-7.5.0-m3l53bh pkgconf-1.7.3-gcc-7.5.0-4sh6pym
gcc-8.3.0-gcc-7.5.0-6hbkzol lmod-8.3-gcc-7.5.0-kqbszrm readline-8.0-gcc-7.5.0-t54jzdy
gdbm-1.18.1-gcc-7.5.0-4av4gyw lua-5.3.5-gcc-7.5.0-n37jedg tar-1.32-gcc-7.5.0-uwe6tb5
gettext-0.20.2-gcc-7.5.0-p4xwlyy lua-luafilesystem-1_7_0_2-gcc-7.5.0-tk7fdik tcl-8.6.8-gcc-7.5.0-aszidzn
git-2.27.0-gcc-7.5.0-pzqvpp2 lua-luaposix-33.4.0-gcc-7.5.0-sas2uzg unzip-6.0-gcc-7.5.0-2ozim4d
gmp-6.1.2-gcc-7.5.0-3ol3tld m4-1.4.18-gcc-7.5.0-mkc3u4x xz-5.2.5-gcc-7.5.0-6nxes4r
isl-0.18-gcc-7.5.0-4lkvfkv mpc-1.1.0-gcc-7.5.0-uur2dd6 zlib-1.2.11-gcc-7.5.0-smoyzzo
libbsd-0.10.0-gcc-7.5.0-u6ue7vw mpfr-3.1.6-gcc-7.5.0-nwsubsw
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
You can module load
any of these. By default, Spack generates modules
named by package-version-compiler-version-hash
, which is a bit hard
to read. We’ll show you how to customize this in the following sections.
Non-hierarchical Module Files¶
If you arrived to this point you should have an environment that looks similar to:
$ module avail
----------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------
autoconf-2.69-gcc-7.5.0-nwb3bf5 libidn2-2.1.1a-gcc-7.5.0-gcbfo3a openssl-1.1.1g-gcc-7.5.0-4dj34yw
autoconf-2.69-gcc-8.3.0-c7fxiuu libpciaccess-0.13.5-gcc-8.3.0-rqqclxg openssl-1.1.1g-gcc-8.3.0-yygx7ht
automake-1.16.2-gcc-7.5.0-wdflovn libsigsegv-2.12-gcc-7.5.0-lbrx7ln patchelf-0.10-gcc-7.5.0-2nj3vxx
automake-1.16.2-gcc-8.3.0-25eawip libsigsegv-2.12-gcc-8.3.0-oaiujfn patchelf-0.10-gcc-8.3.0-35lyjgt
bzip2-1.0.8-gcc-7.5.0-fvfpt26 libtool-2.4.6-gcc-7.5.0-jdxbjft pcre2-10.35-gcc-7.5.0-xyujt3h
bzip2-1.0.8-gcc-8.3.0-jx7bite libtool-2.4.6-gcc-8.3.0-too4ft7 perl-5.30.3-gcc-7.5.0-hyrsxn4
cmake-3.17.3-gcc-8.3.0-7otuah5 libunistring-0.9.10-gcc-7.5.0-dyq3apq perl-5.30.3-gcc-8.3.0-xmhlmqr
curl-7.68.0-gcc-7.5.0-e4rrsab libxml2-2.9.10-gcc-7.5.0-m3l53bh pkgconf-1.7.3-gcc-7.5.0-4sh6pym
diffutils-3.7-gcc-7.5.0-otkkten libxml2-2.9.10-gcc-8.3.0-u5khonr pkgconf-1.7.3-gcc-8.3.0-oo4jobp
diffutils-3.7-gcc-8.3.0-5fvw3jh lmod-8.3-gcc-7.5.0-kqbszrm py-cython-0.29.16-gcc-8.3.0-mbnkbhv
expat-2.2.9-gcc-7.5.0-3f3kkbl lua-5.3.5-gcc-7.5.0-n37jedg py-numpy-1.19.0-gcc-8.3.0-3s3aj6y
expat-2.2.9-gcc-8.3.0-xrk7vwz lua-luafilesystem-1_7_0_2-gcc-7.5.0-tk7fdik py-pybind11-2.5.0-gcc-8.3.0-mjmujjo
findutils-4.6.0-gcc-8.3.0-gxxeusv lua-luaposix-33.4.0-gcc-7.5.0-sas2uzg py-scipy-1.5.0-gcc-8.3.0-6f5vgp6
gcc-8.3.0-gcc-7.5.0-6hbkzol m4-1.4.18-gcc-7.5.0-mkc3u4x py-setuptools-46.1.3-gcc-8.3.0-4dcvntv
gdbm-1.18.1-gcc-7.5.0-4av4gyw m4-1.4.18-gcc-8.3.0-65odbgi python-3.7.7-gcc-8.3.0-4uojybm
gdbm-1.18.1-gcc-8.3.0-alz6ery mpc-1.1.0-gcc-7.5.0-uur2dd6 readline-8.0-gcc-7.5.0-t54jzdy
gettext-0.20.2-gcc-7.5.0-p4xwlyy mpfr-3.1.6-gcc-7.5.0-nwsubsw readline-8.0-gcc-8.3.0-l33t67c
gettext-0.20.2-gcc-8.3.0-euuxtvr mpich-3.3.2-gcc-8.3.0-focol2k sqlite-3.31.1-gcc-8.3.0-vz4awtk
git-2.27.0-gcc-7.5.0-pzqvpp2 ncurses-6.2-gcc-7.5.0-crhlefo tar-1.32-gcc-7.5.0-uwe6tb5
gmp-6.1.2-gcc-7.5.0-3ol3tld ncurses-6.2-gcc-8.3.0-klfkayd tar-1.32-gcc-8.3.0-fk4qp4i
hwloc-1.11.11-gcc-8.3.0-lz2i2dd netlib-lapack-3.8.0-gcc-8.3.0-kbk7dg3 tcl-8.6.8-gcc-7.5.0-aszidzn
hwloc-2.2.0-gcc-8.3.0-4otoxex netlib-scalapack-2.1.0-gcc-8.3.0-h2amar2 texinfo-6.5-gcc-8.3.0-7t62l5t
isl-0.18-gcc-7.5.0-4lkvfkv netlib-scalapack-2.1.0-gcc-8.3.0-kt5c5on unzip-6.0-gcc-7.5.0-2ozim4d
libbsd-0.10.0-gcc-7.5.0-u6ue7vw netlib-scalapack-2.1.0-gcc-8.3.0-qsewiuz util-macros-1.19.1-gcc-8.3.0-obeg5kx
libbsd-0.10.0-gcc-8.3.0-nr46ju6 netlib-scalapack-2.1.0-gcc-8.3.0-uf4ynvq xz-5.2.5-gcc-7.5.0-6nxes4r
libffi-3.3-gcc-8.3.0-lo4lv3d numactl-2.0.12-gcc-8.3.0-tusy4nf xz-5.2.5-gcc-8.3.0-72wq5od
libiconv-1.16-gcc-7.5.0-jearpk4 openblas-0.3.10-gcc-8.3.0-x32w5jn zlib-1.2.11-gcc-7.5.0-smoyzzo
libiconv-1.16-gcc-8.3.0-brkkjge openmpi-3.1.6-gcc-8.3.0-4twpso3 zlib-1.2.11-gcc-8.3.0-2icaxiy
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
The non-hierarchical module files that have been generated so far follow
Spack’s default rules for module generation.
Taking a look at the gcc
module you’ll see, for example:
$ module show gcc-8.3.0-gcc-7.5.0-6hbkzol
----------------------------------------------------------------------------------------------------------------------------------------------
/home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/gcc-8.3.0-gcc-7.5.0-6hbkzol:
----------------------------------------------------------------------------------------------------------------------------------------------
whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages. ")
prepend_path("PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/share/man")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/lib")
prepend_path("LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/lib")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/lib64")
prepend_path("LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/lib64")
prepend_path("C_INCLUDE_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/include")
prepend_path("CPLUS_INCLUDE_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/include")
prepend_path("INCLUDE","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/include")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/")
setenv("CC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gcc")
setenv("CXX","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/g++")
setenv("FC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gfortran")
setenv("F77","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gfortran")
help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C,
Fortran, Ada, and Go, as well as libraries for these languages.
]])
As expected, a few environment variables representing paths will be modified by the module file according to the default prefix inspection rules.
Filter unwanted modifications to the environment¶
Now consider the case that your site has decided that C_INCLUDE_PATH
,
CPLUS_INCLUDE_PATH
, and LIBRARY_PATH
modifications should not be
present in module files. What you can do to abide by the rules is to
create a configuration file ${SPACK_ROOT}/etc/spack/modules.yaml
with
the following content:
modules:
tcl:
all:
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
Next you should regenerate all the module files:
$ spack module tcl refresh
y
==> You are about to regenerate tcl module files for:
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
nwb3bf5 autoconf@2.69
wdflovn automake@1.16.2
fvfpt26 bzip2@1.0.8
e4rrsab curl@7.68.0
otkkten diffutils@3.7
3f3kkbl expat@2.2.9
6hbkzol gcc@8.3.0
4av4gyw gdbm@1.18.1
p4xwlyy gettext@0.20.2
pzqvpp2 git@2.27.0
3ol3tld gmp@6.1.2
4lkvfkv isl@0.18
u6ue7vw libbsd@0.10.0
jearpk4 libiconv@1.16
gcbfo3a libidn2@2.1.1a
lbrx7ln libsigsegv@2.12
jdxbjft libtool@2.4.6
dyq3apq libunistring@0.9.10
m3l53bh libxml2@2.9.10
kqbszrm lmod@8.3
n37jedg lua@5.3.5
tk7fdik lua-luafilesystem@1_7_0_2
sas2uzg lua-luaposix@33.4.0
mkc3u4x m4@1.4.18
uur2dd6 mpc@1.1.0
nwsubsw mpfr@3.1.6
crhlefo ncurses@6.2
4dj34yw openssl@1.1.1g
2nj3vxx patchelf@0.10
xyujt3h pcre2@10.35
hyrsxn4 perl@5.30.3
4sh6pym pkgconf@1.7.3
t54jzdy readline@8.0
uwe6tb5 tar@1.32
aszidzn tcl@8.6.8
2ozim4d unzip@6.0
6nxes4r xz@5.2.5
smoyzzo zlib@1.2.11
-- linux-ubuntu18.04-x86_64 / gcc@8.3.0 -------------------------
c7fxiuu autoconf@2.69
25eawip automake@1.16.2
jx7bite bzip2@1.0.8
7otuah5 cmake@3.17.3
5fvw3jh diffutils@3.7
xrk7vwz expat@2.2.9
gxxeusv findutils@4.6.0
alz6ery gdbm@1.18.1
euuxtvr gettext@0.20.2
lz2i2dd hwloc@1.11.11
4otoxex hwloc@2.2.0
nr46ju6 libbsd@0.10.0
lo4lv3d libffi@3.3
brkkjge libiconv@1.16
rqqclxg libpciaccess@0.13.5
oaiujfn libsigsegv@2.12
too4ft7 libtool@2.4.6
u5khonr libxml2@2.9.10
65odbgi m4@1.4.18
focol2k mpich@3.3.2
klfkayd ncurses@6.2
kbk7dg3 netlib-lapack@3.8.0
h2amar2 netlib-scalapack@2.1.0
uf4ynvq netlib-scalapack@2.1.0
qsewiuz netlib-scalapack@2.1.0
kt5c5on netlib-scalapack@2.1.0
tusy4nf numactl@2.0.12
x32w5jn openblas@0.3.10
4twpso3 openmpi@3.1.6
yygx7ht openssl@1.1.1g
35lyjgt patchelf@0.10
xmhlmqr perl@5.30.3
oo4jobp pkgconf@1.7.3
mbnkbhv py-cython@0.29.16
3s3aj6y py-numpy@1.19.0
mjmujjo py-pybind11@2.5.0
6f5vgp6 py-scipy@1.5.0
4dcvntv py-setuptools@46.1.3
4uojybm python@3.7.7
l33t67c readline@8.0
vz4awtk sqlite@3.31.1
fk4qp4i tar@1.32
7t62l5t texinfo@6.5
obeg5kx util-macros@1.19.1
72wq5od xz@5.2.5
2icaxiy zlib@1.2.11
==> Do you want to proceed? [y/n] ==> Regenerating tcl module files
If you take a look now at the module for gcc
you’ll see that the unwanted
paths have disappeared:
$ module show gcc-8.3.0-gcc-7.5.0-6hbkzol
----------------------------------------------------------------------------------------------------------------------------------------------
/home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/gcc-8.3.0-gcc-7.5.0-6hbkzol:
----------------------------------------------------------------------------------------------------------------------------------------------
whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages. ")
prepend_path("PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/share/man")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/lib")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/lib64")
prepend_path("INCLUDE","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/include")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/")
setenv("CC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gcc")
setenv("CXX","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/g++")
setenv("FC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gfortran")
setenv("F77","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gfortran")
help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C,
Fortran, Ada, and Go, as well as libraries for these languages.
]])
Prevent some module files from being generated¶
Another common request at many sites is to avoid exposing software that
is only needed as an intermediate step when building a newer stack.
Let’s try to prevent the generation of
module files for anything that is compiled with gcc@7.5.0
(the OS provided compiler).
To do this you should add a blacklist
keyword to ${SPACK_ROOT}/etc/spack/modules.yaml
:
modules:
tcl:
blacklist:
- '%gcc@7.5.0'
all:
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
and regenerate the module files. This time we’ll pass the option
--delete-tree
so that Spack will delete the existing module tree and
regenerate a new one, instead of overwriting the files in the existing
directory.
$ spack module tcl refresh --delete-tree -y
==> Regenerating tcl module files
$ module avail
----------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------
autoconf-2.69-gcc-8.3.0-c7fxiuu libtool-2.4.6-gcc-8.3.0-too4ft7 pkgconf-1.7.3-gcc-8.3.0-oo4jobp
automake-1.16.2-gcc-8.3.0-25eawip libxml2-2.9.10-gcc-8.3.0-u5khonr py-cython-0.29.16-gcc-8.3.0-mbnkbhv
bzip2-1.0.8-gcc-8.3.0-jx7bite m4-1.4.18-gcc-8.3.0-65odbgi py-numpy-1.19.0-gcc-8.3.0-3s3aj6y
cmake-3.17.3-gcc-8.3.0-7otuah5 mpich-3.3.2-gcc-8.3.0-focol2k py-pybind11-2.5.0-gcc-8.3.0-mjmujjo
diffutils-3.7-gcc-8.3.0-5fvw3jh ncurses-6.2-gcc-8.3.0-klfkayd py-scipy-1.5.0-gcc-8.3.0-6f5vgp6
expat-2.2.9-gcc-8.3.0-xrk7vwz netlib-lapack-3.8.0-gcc-8.3.0-kbk7dg3 py-setuptools-46.1.3-gcc-8.3.0-4dcvntv
findutils-4.6.0-gcc-8.3.0-gxxeusv netlib-scalapack-2.1.0-gcc-8.3.0-h2amar2 python-3.7.7-gcc-8.3.0-4uojybm
gdbm-1.18.1-gcc-8.3.0-alz6ery netlib-scalapack-2.1.0-gcc-8.3.0-kt5c5on readline-8.0-gcc-8.3.0-l33t67c
gettext-0.20.2-gcc-8.3.0-euuxtvr netlib-scalapack-2.1.0-gcc-8.3.0-qsewiuz sqlite-3.31.1-gcc-8.3.0-vz4awtk
hwloc-1.11.11-gcc-8.3.0-lz2i2dd netlib-scalapack-2.1.0-gcc-8.3.0-uf4ynvq tar-1.32-gcc-8.3.0-fk4qp4i
hwloc-2.2.0-gcc-8.3.0-4otoxex numactl-2.0.12-gcc-8.3.0-tusy4nf texinfo-6.5-gcc-8.3.0-7t62l5t
libbsd-0.10.0-gcc-8.3.0-nr46ju6 openblas-0.3.10-gcc-8.3.0-x32w5jn util-macros-1.19.1-gcc-8.3.0-obeg5kx
libffi-3.3-gcc-8.3.0-lo4lv3d openmpi-3.1.6-gcc-8.3.0-4twpso3 xz-5.2.5-gcc-8.3.0-72wq5od
libiconv-1.16-gcc-8.3.0-brkkjge openssl-1.1.1g-gcc-8.3.0-yygx7ht zlib-1.2.11-gcc-8.3.0-2icaxiy
libpciaccess-0.13.5-gcc-8.3.0-rqqclxg patchelf-0.10-gcc-8.3.0-35lyjgt
libsigsegv-2.12-gcc-8.3.0-oaiujfn perl-5.30.3-gcc-8.3.0-xmhlmqr
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
if you look closely you’ll see though that we went too far in
blacklisting modules: the module for gcc@8.3.0
disappeared as it was
bootstrapped with gcc@7.5.0
. To specify exceptions to the blacklist
rules you can use whitelist
:
modules:
tcl:
whitelist:
- gcc
blacklist:
- '%gcc@7.5.0'
all:
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
whitelist
rules always have precedence over blacklist
rules. If you regenerate the modules again:
$ spack module tcl refresh -y
==> Regenerating tcl module files
you’ll see that now the module for gcc@8.3.0
has reappeared:
$ module avail gcc-8.3.0-gcc-7.5.0-6hbkzol
----------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------
gcc-8.3.0-gcc-7.5.0-6hbkzol
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
An additional feature that you can leverage to unclutter the environment is to skip the generation of module files for implicitly installed packages. In this case you only need to add the following line:
modules:
tcl:
blacklist_implicits: true
whitelist:
- gcc
blacklist:
- '%gcc@7.5.0'
all:
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
to modules.yaml
and regenerate the module file tree as above.
Change module file naming¶
The next step in making module files more user-friendly is to
improve their naming scheme.
To reduce the length of the hash or remove it altogether you can
use the hash_length
keyword in the configuration file:
modules:
tcl:
hash_length: 0
whitelist:
- gcc
blacklist:
- '%gcc@7.5.0'
all:
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
If you try to regenerate the module files now you will get an error:
$ spack module tcl refresh --delete-tree -y
==> Error: Name clashes detected in module files:
file: /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/netlib-scalapack-2.1.0-gcc-8.3.0
spec: netlib-scalapack@2.1.0%gcc@8.3.0~pic+shared build_type=RelWithDebInfo patches=f2baedde688ffe4c20943c334f580eb298e04d6f35c86b90a1f4e8cb7ae344a2 arch=linux-ubuntu18.04-x86_64
spec: netlib-scalapack@2.1.0%gcc@8.3.0~pic+shared build_type=RelWithDebInfo patches=f2baedde688ffe4c20943c334f580eb298e04d6f35c86b90a1f4e8cb7ae344a2 arch=linux-ubuntu18.04-x86_64
spec: netlib-scalapack@2.1.0%gcc@8.3.0~pic+shared build_type=RelWithDebInfo patches=f2baedde688ffe4c20943c334f580eb298e04d6f35c86b90a1f4e8cb7ae344a2 arch=linux-ubuntu18.04-x86_64
spec: netlib-scalapack@2.1.0%gcc@8.3.0~pic+shared build_type=RelWithDebInfo patches=f2baedde688ffe4c20943c334f580eb298e04d6f35c86b90a1f4e8cb7ae344a2 arch=linux-ubuntu18.04-x86_64
==> Error: Operation aborted
Note
We try to check for errors up front!
In Spack we check for errors upfront whenever possible, so don’t worry about your module files: as a name clash was detected nothing has been changed on disk.
The problem here is that without the hashes the four different flavors of
netlib-scalapack
map to the same module file name. We can change how
the names are formatted to differentiate them:
modules:
tcl:
hash_length: 0
whitelist:
- gcc
blacklist:
- '%gcc@7.5.0'
all:
conflict:
- '{name}'
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
projections:
all: '{name}/{version}-{compiler.name}-{compiler.version}'
netlib-scalapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
^python^lapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'
As you can see it is possible to specify rules that apply only to a
restricted set of packages using anonymous specs
like ^mpi^lapack
. Here we declare a conflict between any two modules
with the same name, so they cannot be loaded together. We also format the
names of modules according to compiler, compiler version, and MPI version
using the spec format syntax.
This allows us to match specs by their dependencies, and format them
based on their DAGs.
$ spack module tcl refresh --delete-tree -y
==> Regenerating tcl module files
$ module avail
----------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------
autoconf/2.69-gcc-8.3.0 libsigsegv/2.12-gcc-8.3.0 perl/5.30.3-gcc-8.3.0
automake/1.16.2-gcc-8.3.0 libtool/2.4.6-gcc-8.3.0 pkgconf/1.7.3-gcc-8.3.0
bzip2/1.0.8-gcc-8.3.0 libxml2/2.9.10-gcc-8.3.0 py-cython/0.29.16-gcc-8.3.0
cmake/3.17.3-gcc-8.3.0 m4/1.4.18-gcc-8.3.0 py-numpy/1.19.0-gcc-8.3.0-openblas
diffutils/3.7-gcc-8.3.0 mpich/3.3.2-gcc-8.3.0 py-pybind11/2.5.0-gcc-8.3.0
expat/2.2.9-gcc-8.3.0 ncurses/6.2-gcc-8.3.0 py-scipy/1.5.0-gcc-8.3.0-openblas
findutils/4.6.0-gcc-8.3.0 netlib-lapack/3.8.0-gcc-8.3.0 py-setuptools/46.1.3-gcc-8.3.0
gcc/8.3.0-gcc-7.5.0 netlib-scalapack/2.1.0-gcc-8.3.0-netlib-lapack-mpich python/3.7.7-gcc-8.3.0
gdbm/1.18.1-gcc-8.3.0 netlib-scalapack/2.1.0-gcc-8.3.0-netlib-lapack-openmpi readline/8.0-gcc-8.3.0
gettext/0.20.2-gcc-8.3.0 netlib-scalapack/2.1.0-gcc-8.3.0-openblas-mpich sqlite/3.31.1-gcc-8.3.0
hwloc/1.11.11-gcc-8.3.0 netlib-scalapack/2.1.0-gcc-8.3.0-openblas-openmpi (D) tar/1.32-gcc-8.3.0
hwloc/2.2.0-gcc-8.3.0 (D) numactl/2.0.12-gcc-8.3.0 texinfo/6.5-gcc-8.3.0
libbsd/0.10.0-gcc-8.3.0 openblas/0.3.10-gcc-8.3.0 util-macros/1.19.1-gcc-8.3.0
libffi/3.3-gcc-8.3.0 openmpi/3.1.6-gcc-8.3.0 xz/5.2.5-gcc-8.3.0
libiconv/1.16-gcc-8.3.0 openssl/1.1.1g-gcc-8.3.0 zlib/1.2.11-gcc-8.3.0
libpciaccess/0.13.5-gcc-8.3.0 patchelf/0.10-gcc-8.3.0
Where:
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
Note
The conflict
directive is Tcl-specific and can’t be used in the
lmod
section of the configuration file.
Add custom environment modifications¶
At many sites it is customary to set an environment variable in a
package’s module file that points to the folder in which the package
is installed. You can achieve this with Spack by adding an
environment
directive to the configuration file:
modules:
tcl:
hash_length: 0
naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
whitelist:
- gcc
blacklist:
- '%gcc@7.5.0'
all:
conflict:
- '{name}'
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
environment:
set:
'{name}_ROOT': '{prefix}'
projections:
all: '{name}/{version}-{compiler.name}-{compiler.version}'
netlib-scalapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
^python^lapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'
Under the hood Spack uses the format()
API to substitute
tokens in either environment variable names or values. There are two caveats though:
- The set of allowed tokens in variable names is restricted to
name
,version
,compiler
,compiler.name
,compiler.version
,architecture
- Any token expanded in a variable name is made uppercase, but other than that case sensitivity is preserved
Regenerating the module files results in something like:
$ spack module tcl refresh -y
==> Regenerating tcl module files
$ module show gcc
----------------------------------------------------------------------------------------------------------------------------------------------
/home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/gcc/8.3.0-gcc-7.5.0:
----------------------------------------------------------------------------------------------------------------------------------------------
whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages. ")
conflict("gcc")
prepend_path("PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/share/man")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/lib")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/lib64")
prepend_path("INCLUDE","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/include")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/")
setenv("CC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gcc")
setenv("CXX","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/g++")
setenv("FC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gfortran")
setenv("F77","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gfortran")
setenv("'GCC_ROOT'","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl")
help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C,
Fortran, Ada, and Go, as well as libraries for these languages.
]])
As you can see, the gcc
module has the environment variable GCC_ROOT
set.
Sometimes it’s also useful to apply environment modifications selectively and target
only certain packages. You can for instance apply modifications to the
openmpi
module as follows:
modules:
tcl:
hash_length: 0
naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
whitelist:
- gcc
blacklist:
- '%gcc@7.5.0'
all:
conflict:
- '{name}'
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
environment:
set:
'{name}_ROOT': '{prefix}'
openmpi:
environment:
set:
SLURM_MPI_TYPE: pmi2
OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
projections:
all: '{name}/{version}-{compiler.name}-{compiler.version}'
netlib-scalapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
^python^lapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'
This time we will be more selective and regenerate only the openmpi
module file:
$ spack module tcl refresh -y openmpi
==> Regenerating tcl module files
$ module show openmpi
----------------------------------------------------------------------------------------------------------------------------------------------
/home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/openmpi/3.1.6-gcc-8.3.0:
----------------------------------------------------------------------------------------------------------------------------------------------
whatis("An open source Message Passing Interface implementation. ")
conflict("openmpi")
prepend_path("PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky/bin")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky/share/man")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky/lib")
prepend_path("INCLUDE","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky/include")
prepend_path("PKG_CONFIG_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky/lib/pkgconfig")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky/")
setenv("MPICC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky/bin/mpicc")
setenv("MPICXX","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky/bin/mpic++")
setenv("MPIF77","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky/bin/mpif77")
setenv("MPIF90","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky/bin/mpif90")
setenv("'OPENMPI_ROOT'","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/openmpi-3.1.6-4twpso3kn2f3z6oa527snqvn7fsridky")
setenv("SLURM_MPI_TYPE","pmi2")
setenv("OMPI_MCA_btl_openib_warn_default_git_prefix","0")
help([[An open source Message Passing Interface implementation. The Open MPI
Project is an open source Message Passing Interface implementation that
is developed and maintained by a consortium of academic, research, and
industry partners. Open MPI is therefore able to combine the expertise,
technologies, and resources from all across the High Performance
Computing community in order to build the best MPI library available.
Open MPI offers advantages for system and software vendors, application
developers and computer science researchers.
]])
Autoload dependencies¶
Spack can also generate module files that contain code to load the
dependencies automatically. You can, for instance generate python
modules that load their dependencies by adding the autoload
directive and assigning it the value direct
:
modules:
tcl:
verbose: True
hash_length: 0
naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
whitelist:
- gcc
blacklist:
- '%gcc@7.5.0'
all:
conflict:
- '{name}'
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
environment:
set:
'{name}_ROOT': '{prefix}'
gcc:
environment:
set:
CC: gcc
CXX: g++
FC: gfortran
F90: gfortran
F77: gfortran
openmpi:
environment:
set:
SLURM_MPI_TYPE: pmi2
OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
projections:
all: '{name}/{version}-{compiler.name}-{compiler.version}'
netlib-scalapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
^python^lapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'
^python:
autoload: direct
and regenerating the module files for every package that depends on python
:
$ spack module tcl refresh -y ^python
==> Regenerating tcl module files
and will contain code to autoload all the dependencies:
$ module load py-scipy
Autoloading openblas/0.3.10-gcc-8.3.0
Autoloading py-numpy/1.19.0-gcc-8.3.0-openblas
Autoloading python/3.7.7-gcc-8.3.0
Autoloading python/3.7.7-gcc-8.3.0
In case messages are unwanted during the autoload procedure, it will be
sufficient to omit the line setting verbose: True
in the configuration file above.
Hierarchical Module Files¶
So far we worked with non-hierarchical module files, i.e. with module files
that are all generated in the same root directory and don’t attempt to
dynamically modify the MODULEPATH
. This results in a flat module structure where
all the software is visible at the same time:
$ module avail
----------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------
autoconf/2.69-gcc-8.3.0 libsigsegv/2.12-gcc-8.3.0 perl/5.30.3-gcc-8.3.0
automake/1.16.2-gcc-8.3.0 libtool/2.4.6-gcc-8.3.0 pkgconf/1.7.3-gcc-8.3.0
bzip2/1.0.8-gcc-8.3.0 libxml2/2.9.10-gcc-8.3.0 py-cython/0.29.16-gcc-8.3.0
cmake/3.17.3-gcc-8.3.0 m4/1.4.18-gcc-8.3.0 py-numpy/1.19.0-gcc-8.3.0-openblas
diffutils/3.7-gcc-8.3.0 mpich/3.3.2-gcc-8.3.0 py-pybind11/2.5.0-gcc-8.3.0
expat/2.2.9-gcc-8.3.0 ncurses/6.2-gcc-8.3.0 py-scipy/1.5.0-gcc-8.3.0-openblas
findutils/4.6.0-gcc-8.3.0 netlib-lapack/3.8.0-gcc-8.3.0 py-setuptools/46.1.3-gcc-8.3.0
gcc/8.3.0-gcc-7.5.0 netlib-scalapack/2.1.0-gcc-8.3.0-netlib-lapack-mpich python/3.7.7-gcc-8.3.0
gdbm/1.18.1-gcc-8.3.0 netlib-scalapack/2.1.0-gcc-8.3.0-netlib-lapack-openmpi readline/8.0-gcc-8.3.0
gettext/0.20.2-gcc-8.3.0 netlib-scalapack/2.1.0-gcc-8.3.0-openblas-mpich sqlite/3.31.1-gcc-8.3.0
hwloc/1.11.11-gcc-8.3.0 netlib-scalapack/2.1.0-gcc-8.3.0-openblas-openmpi (D) tar/1.32-gcc-8.3.0
hwloc/2.2.0-gcc-8.3.0 (D) numactl/2.0.12-gcc-8.3.0 texinfo/6.5-gcc-8.3.0
libbsd/0.10.0-gcc-8.3.0 openblas/0.3.10-gcc-8.3.0 util-macros/1.19.1-gcc-8.3.0
libffi/3.3-gcc-8.3.0 openmpi/3.1.6-gcc-8.3.0 xz/5.2.5-gcc-8.3.0
libiconv/1.16-gcc-8.3.0 openssl/1.1.1g-gcc-8.3.0 zlib/1.2.11-gcc-8.3.0
libpciaccess/0.13.5-gcc-8.3.0 patchelf/0.10-gcc-8.3.0
Where:
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
This layout is quite simple to deploy, but you can see from the above snippet that nothing prevents users from loading incompatible sets of modules:
$ module purge
$ module load netlib-lapack openblas
$ module list
Currently Loaded Modules:
1) netlib-lapack/3.8.0-gcc-8.3.0 2) openblas/0.3.10-gcc-8.3.0
Even if conflicts
directives are carefully placed in module files, they:
- won’t enforce a consistent environment, but will just report an error
- need constant updates, for instance as soon as a new compiler or MPI library is installed
Hierarchical module files try to
overcome these shortcomings by showing at start-up only a restricted view of what is
available on the system: more specifically only the software that has been installed with
OS provided compilers. Among this software there will be other - usually more recent - compilers
that, once loaded, will prepend new directories to MODULEPATH
unlocking all the software
that was compiled with them. This “unlocking” idea can then be extended arbitrarily to
virtual dependencies, as we’ll see in the following section.
Core/Compiler/MPI¶
The most widely used hierarchy is the so called Core/Compiler/MPI
where, on top
of the compilers, different MPI libraries also unlock software linked to them.
There are just a few steps needed to adapt the modules.yaml
file we used previously:
- enable the
lmod
file generator- change the
tcl
tag tolmod
- remove the
tcl
specificconflict
directive- declare which compilers are considered
core_compilers
- remove the
mpi
related suffixes in projections (as they will be substituted by hierarchies)
After these modifications your configuration file should look like:
modules:
enable::
- lmod
lmod:
core_compilers:
- 'gcc@7.5.0'
hierarchy:
- mpi
hash_length: 0
whitelist:
- gcc
blacklist:
- '%gcc@7.5.0'
all:
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
environment:
set:
'{name}_ROOT': '{prefix}'
openmpi:
environment:
set:
SLURM_MPI_TYPE: pmi2
OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
projections:
all: '{name}/{version}'
^lapack: '{name}/{version}-{^lapack.name}'
Note
- Double colon in configuration files
- The double colon after
enable
is intentional and it serves the purpose of overriding the default list of enabled generators so that onlylmod
will be active (see Overriding entire sections for more details).
The directive core_compilers
accepts a list of compilers. Everything built
using these compilers will create a module in the Core
part of the hierarchy,
which is the entry point for hierarchical module files. It is
common practice to put the OS provided compilers in the list and only build common utilities
and other compilers with them.
If we now regenerate the module files:
$ spack module lmod refresh --delete-tree -y
==> Regenerating lmod module files
and update MODULEPATH
to point to the Core
:
$ module purge
$ module unuse $HOME/spack/share/spack/modules/linux-ubuntu18.04-x86_64
$ module use $HOME/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core
asking for the available modules will return:
$ module avail
---------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------
gcc/8.3.0
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
Unsurprisingly, the only visible module is gcc
. Loading that we’ll unlock
the Compiler
part of the hierarchy:
$ module load gcc
$ module avail
------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/gcc/8.3.0 --------------------------------------
autoconf/2.69 gdbm/1.18.1 libpciaccess/0.13.5 netlib-lapack/3.8.0 pkgconf/1.7.3 readline/8.0
automake/1.16.2 gettext/0.20.2 libsigsegv/2.12 numactl/2.0.12 py-cython/0.29.16 sqlite/3.31.1
bzip2/1.0.8 hwloc/1.11.11 libtool/2.4.6 openblas/0.3.10 py-numpy/1.19.0-openblas tar/1.32
cmake/3.17.3 hwloc/2.2.0 (D) libxml2/2.9.10 openmpi/3.1.6 py-pybind11/2.5.0 texinfo/6.5
diffutils/3.7 libbsd/0.10.0 m4/1.4.18 openssl/1.1.1g py-scipy/1.5.0-openblas util-macros/1.19.1
expat/2.2.9 libffi/3.3 mpich/3.3.2 patchelf/0.10 py-setuptools/46.1.3 xz/5.2.5
findutils/4.6.0 libiconv/1.16 ncurses/6.2 perl/5.30.3 python/3.7.7 zlib/1.2.11
---------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------
gcc/8.3.0 (L)
Where:
D: Default Module
L: Module is loaded
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
The same holds true also for the MPI
part, that you can enable by loading
either mpich
or openmpi
. Let’s start by loading mpich
:
$ module load mpich
$ module avail
--------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/mpich/3.3.2-focol2k/gcc/8.3.0 ----------------------------
netlib-scalapack/2.1.0-netlib-lapack netlib-scalapack/2.1.0-openblas (D)
------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/gcc/8.3.0 --------------------------------------
autoconf/2.69 gdbm/1.18.1 libpciaccess/0.13.5 netlib-lapack/3.8.0 pkgconf/1.7.3 readline/8.0
automake/1.16.2 gettext/0.20.2 libsigsegv/2.12 numactl/2.0.12 py-cython/0.29.16 sqlite/3.31.1
bzip2/1.0.8 hwloc/1.11.11 libtool/2.4.6 openblas/0.3.10 py-numpy/1.19.0-openblas tar/1.32
cmake/3.17.3 hwloc/2.2.0 (D) libxml2/2.9.10 openmpi/3.1.6 py-pybind11/2.5.0 texinfo/6.5
diffutils/3.7 libbsd/0.10.0 m4/1.4.18 openssl/1.1.1g py-scipy/1.5.0-openblas util-macros/1.19.1
expat/2.2.9 libffi/3.3 mpich/3.3.2 (L) patchelf/0.10 py-setuptools/46.1.3 xz/5.2.5
findutils/4.6.0 libiconv/1.16 ncurses/6.2 perl/5.30.3 python/3.7.7 zlib/1.2.11
---------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------
gcc/8.3.0 (L)
Where:
D: Default Module
L: Module is loaded
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
$ module load openblas netlib-scalapack/2.1.0-openblas
$ module list
Currently Loaded Modules:
1) gcc/8.3.0 2) mpich/3.3.2 3) openblas/0.3.10 4) netlib-scalapack/2.1.0-openblas
At this point we can showcase the improved consistency that a hierarchical layout provides over a non-hierarchical one:
$ export LMOD_AUTO_SWAP=yes
$ module load openmpi
Lmod is automatically replacing "mpich/3.3.2" with "openmpi/3.1.6".
Due to MODULEPATH changes, the following have been reloaded:
1) netlib-scalapack/2.1.0-openblas
Lmod
took care of swapping the MPI provider for us, and it also substituted the
netlib-scalapack
module to conform to the change in the MPI.
In this way we can’t accidentally pull-in two different MPI providers at the
same time or load a module file for a package linked to openmpi
when mpich
is also loaded.
Consistency for compilers and MPI is ensured by the tool.
Add LAPACK to the hierarchy¶
The hierarchy just shown is already a great improvement over non-hierarchical layouts,
but it still has an asymmetry: LAPACK
providers cover the same semantic role
as MPI
providers, but yet they are not part of the hierarchy.
To be more practical, this means that although we have gained an improved consistency in
our environment when it comes to MPI
, we still have the same problems as we had before
for LAPACK
implementations:
$ module list
Currently Loaded Modules:
1) gcc/8.3.0 2) openblas/0.3.10 3) openmpi/3.1.6 4) netlib-scalapack/2.1.0-openblas
$ module avail
-------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/openmpi/3.1.6-4twpso3/gcc/8.3.0 ---------------------------
netlib-scalapack/2.1.0-netlib-lapack netlib-scalapack/2.1.0-openblas (L,D)
------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/gcc/8.3.0 --------------------------------------
autoconf/2.69 gdbm/1.18.1 libpciaccess/0.13.5 netlib-lapack/3.8.0 pkgconf/1.7.3 readline/8.0
automake/1.16.2 gettext/0.20.2 libsigsegv/2.12 numactl/2.0.12 py-cython/0.29.16 sqlite/3.31.1
bzip2/1.0.8 hwloc/1.11.11 libtool/2.4.6 openblas/0.3.10 (L) py-numpy/1.19.0-openblas tar/1.32
cmake/3.17.3 hwloc/2.2.0 (D) libxml2/2.9.10 openmpi/3.1.6 (L) py-pybind11/2.5.0 texinfo/6.5
diffutils/3.7 libbsd/0.10.0 m4/1.4.18 openssl/1.1.1g py-scipy/1.5.0-openblas util-macros/1.19.1
expat/2.2.9 libffi/3.3 mpich/3.3.2 patchelf/0.10 py-setuptools/46.1.3 xz/5.2.5
findutils/4.6.0 libiconv/1.16 ncurses/6.2 perl/5.30.3 python/3.7.7 zlib/1.2.11
---------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------
gcc/8.3.0 (L)
Where:
D: Default Module
L: Module is loaded
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
$ module load netlib-scalapack/2.1.0-netlib-lapack
The following have been reloaded with a version change:
1) netlib-scalapack/2.1.0-openblas => netlib-scalapack/2.1.0-netlib-lapack
$ module list
Currently Loaded Modules:
1) gcc/8.3.0 2) openblas/0.3.10 3) openmpi/3.1.6 4) netlib-scalapack/2.1.0-netlib-lapack
Hierarchies that are deeper than Core
/Compiler
/MPI
are
probably still considered “unusual” or “impractical” at many sites, mainly because
module files are written manually and keeping track of the combinations
among multiple providers quickly becomes quite involved.
For instance, having both MPI
and LAPACK
in the hierarchy
means we must classify software into one of four categories:
- Software that doesn’t depend on
MPI
orLAPACK
- Software that depends only on
MPI
- Software that depends only on
LAPACK
- Software that depends on both
to decide when to show it to the user. The situation becomes more involved as the number of virtual dependencies in the hierarchy increases.
We can take advantage of the DAG that Spack maintains for the installed software and solve this combinatorial problem in a clean and automated way. In some sense Spack’s ability to manage this combinatorial complexity makes deeper hierarchies feasible.
Coming back to our example, let’s add lapack
to the hierarchy and
remove the remaining suffix projection for lapack
:
modules:
enable::
- lmod
lmod:
core_compilers:
- 'gcc@7.5.0'
hierarchy:
- mpi
- lapack
hash_length: 0
whitelist:
- gcc
blacklist:
- '%gcc@7.5.0'
all:
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
environment:
set:
'{name}_ROOT': '{prefix}'
openmpi:
environment:
set:
SLURM_MPI_TYPE: pmi2
OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
projections:
all: '{name}/{version}'
After module files have been regenerated as usual:
$ module purge
$ spack module lmod refresh --delete-tree -y
==> Regenerating lmod module files
we can see that now we have additional components in the hierarchy:
$ module load gcc
Inactive Modules:
1) netlib-scalapack/2.1.0-netlib-lapack
$ module load openblas
$ module avail
-------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/openmpi/3.1.6-4twpso3/openblas/0.3.10-x32w5jn/gcc/8.3.0 ---------------
netlib-scalapack/2.1.0
------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/openblas/0.3.10-x32w5jn/gcc/8.3.0 --------------------------
py-numpy/1.19.0 py-scipy/1.5.0
------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/gcc/8.3.0 --------------------------------------
autoconf/2.69 gdbm/1.18.1 libpciaccess/0.13.5 netlib-lapack/3.8.0 pkgconf/1.7.3 tar/1.32
automake/1.16.2 gettext/0.20.2 libsigsegv/2.12 numactl/2.0.12 py-cython/0.29.16 texinfo/6.5
bzip2/1.0.8 hwloc/1.11.11 libtool/2.4.6 openblas/0.3.10 (L) py-pybind11/2.5.0 util-macros/1.19.1
cmake/3.17.3 hwloc/2.2.0 (D) libxml2/2.9.10 openmpi/3.1.6 (L) py-setuptools/46.1.3 xz/5.2.5
diffutils/3.7 libbsd/0.10.0 m4/1.4.18 openssl/1.1.1g python/3.7.7 zlib/1.2.11
expat/2.2.9 libffi/3.3 mpich/3.3.2 patchelf/0.10 readline/8.0
findutils/4.6.0 libiconv/1.16 ncurses/6.2 perl/5.30.3 sqlite/3.31.1
---------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------
gcc/8.3.0 (L)
Where:
D: Default Module
L: Module is loaded
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
$ module load openmpi
$ module avail
-------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/openmpi/3.1.6-4twpso3/openblas/0.3.10-x32w5jn/gcc/8.3.0 ---------------
netlib-scalapack/2.1.0
------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/openblas/0.3.10-x32w5jn/gcc/8.3.0 --------------------------
py-numpy/1.19.0 py-scipy/1.5.0
------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/gcc/8.3.0 --------------------------------------
autoconf/2.69 gdbm/1.18.1 libpciaccess/0.13.5 netlib-lapack/3.8.0 pkgconf/1.7.3 tar/1.32
automake/1.16.2 gettext/0.20.2 libsigsegv/2.12 numactl/2.0.12 py-cython/0.29.16 texinfo/6.5
bzip2/1.0.8 hwloc/1.11.11 libtool/2.4.6 openblas/0.3.10 (L) py-pybind11/2.5.0 util-macros/1.19.1
cmake/3.17.3 hwloc/2.2.0 (D) libxml2/2.9.10 openmpi/3.1.6 (L) py-setuptools/46.1.3 xz/5.2.5
diffutils/3.7 libbsd/0.10.0 m4/1.4.18 openssl/1.1.1g python/3.7.7 zlib/1.2.11
expat/2.2.9 libffi/3.3 mpich/3.3.2 patchelf/0.10 readline/8.0
findutils/4.6.0 libiconv/1.16 ncurses/6.2 perl/5.30.3 sqlite/3.31.1
---------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------
gcc/8.3.0 (L)
Where:
D: Default Module
L: Module is loaded
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
Both MPI
and LAPACK
providers will now benefit from the same safety features:
$ module load py-numpy netlib-scalapack
Activating Modules:
1) netlib-scalapack/2.1.0
$ module load mpich
Lmod is automatically replacing "openmpi/3.1.6" with "mpich/3.3.2".
Due to MODULEPATH changes, the following have been reloaded:
1) netlib-scalapack/2.1.0
$ module load netlib-lapack
Lmod is automatically replacing "openblas/0.3.10" with "netlib-lapack/3.8.0".
Inactive Modules:
1) py-numpy
Due to MODULEPATH changes, the following have been reloaded:
1) netlib-scalapack/2.1.0
Because we only compiled py-numpy
with openblas
the module
is made inactive when we switch the LAPACK
provider. The user
environment is now consistent by design!
Working with Templates¶
As briefly mentioned in the introduction, Spack uses Jinja2 to generate each individual module file. This means that you have all of its flexibility and power when it comes to customizing what gets generated!
Module file templates¶
The templates that Spack uses to generate module files are stored in the
share/spack/templates/module
directory within the Spack prefix, and
they all share the same common structure. Usually, they start with a
header that identifies the type of module being generated. In the case of
hierarchical module files it’s:
-- -*- lua -*-
-- Module file created by spack (https://github.com/spack/spack) on {{ timestamp }}
--
-- {{ spec.short_spec }}
--
The statements within double curly brackets {{ ... }}
denote
expressions
that will be evaluated and substituted at module generation time.
The rest of the file is then divided into
blocks
that can be overridden or extended by users, if need be.
Control structures
, delimited by {% ... %}
,
are also permitted in the template language:
{% block environment %}
{% for command_name, cmd in environment_modifications %}
{% if command_name == 'PrependPath' %}
prepend_path("{{ cmd.name }}", "{{ cmd.value }}", "{{ cmd.separator }}")
{% elif command_name == 'AppendPath' %}
append_path("{{ cmd.name }}", "{{ cmd.value }}", "{{ cmd.separator }}")
{% elif command_name == 'RemovePath' %}
remove_path("{{ cmd.name }}", "{{ cmd.value }}", "{{ cmd.separator }}")
{% elif command_name == 'SetEnv' %}
setenv("{{ cmd.name }}", "{{ cmd.value }}")
{% elif command_name == 'UnsetEnv' %}
unsetenv("{{ cmd.name }}")
{% endif %}
{% endfor %}
{% endblock %}
The locations where Spack looks for templates are specified
in config.yaml
:
projections:
all: "${ARCHITECTURE}/${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH}"
# install_tree can include an optional padded length (int or boolean)
# default is False (do not pad)
and can be extended by users to employ custom templates, as we’ll see next.
Extend the default templates¶
Let’s assume one of our software is protected by group membership: allowed users belong to the same linux group, and access is granted at group level. Wouldn’t it be nice if people that are not yet entitled to use it could receive a helpful message at module load time that tells them who to contact in your organization to be inserted in the group?
To automate the generation of module files with such site-specific behavior
we’ll start by extending the list of locations where Spack looks for module
files. Let’s create the file ${SPACK_ROOT}/etc/spack/config.yaml
with the content:
config:
template_dirs:
- $HOME/.spack/templates
This tells Spack to also search another location when looking for template files. Next, we need to create our custom template extension in the folder listed above:
{% extends "modules/modulefile.lua" %}
{% block footer %}
-- Access is granted only to specific groups
if not isDir("{{ spec.prefix }}") then
LmodError (
"You don't have the necessary rights to run \"{{ spec.name }}\".\n\n",
"\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n"
)
end
{% endblock %}
Let’s name this file group-restricted.lua
. The line:
{% extends "modules/modulefile.lua" %}
tells Jinja2 that we are reusing the standard template for hierarchical module files. The section:
{% block footer %}
-- Access is granted only to specific groups
if not isDir("{{ spec.prefix }}") then
LmodError (
"You don't have the necessary rights to run \"{{ spec.name }}\".\n\n",
"\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n"
)
end
{% endblock %}
overrides the footer
block.
Finally, we need to add a couple of lines in modules.yaml
to tell Spack which specs
need to use the new custom template. For the sake of illustration let’s assume
it’s netlib-scalapack
:
modules:
enable::
- lmod
lmod:
core_compilers:
- 'gcc@7.5.0'
hierarchy:
- mpi
- lapack
hash_length: 0
whitelist:
- gcc
blacklist:
- '%gcc@7.5.0'
- readline
all:
filter:
environment_blacklist:
- "C_INCLUDE_PATH"
- "CPLUS_INCLUDE_PATH"
- "LIBRARY_PATH"
environment:
set:
'{name}_ROOT': '{prefix}'
openmpi:
environment:
set:
SLURM_MPI_TYPE: pmi2
OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
netlib-scalapack:
template: 'group-restricted.lua'
If we regenerate the module files one last time:
$ spack module lmod refresh -y netlib-scalapack
==> Regenerating lmod module files
we’ll find the following at the end of each netlib-scalapack
module file:
-- Access is granted only to specific groups
if not isDir("/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.3.0/netlib-scalapack-2.0.2-2p75lzqjbsnev7d2j2osgpkz7ib33oca") then
LmodError (
"You don't have the necessary rights to run \"netlib-scalapack\".\n\n",
"\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n"
)
end
and every user that doesn’t have access to the software will now be redirected to the right e-mail address where to ask for it!
Restore settings for future sections¶
For future sections of the tutorial, we will not use the gcc@8.3.0
compiler. Since it is currently the default compiler (our current
defualt is the most recent version of gcc available), we will remove
it now.
$ spack compiler rm gcc@8.3.0
This will ensure the rest of the tutorial goes smoothly for you.