Tutorial setup

If you have not done the prior sections, you’ll need to set Spack up like this:

git clone https://github.com/spack/spack
. spack/share/spack/setup-env.sh
spack tutorial

See the Basic Installation Tutorial for full details on setup. For more help join us in the #tutorial channel on Slack – get an invitation at spackpm.herokuapp.com

Module Files Tutorial

This tutorial illustrates how Spack can be used to generate module files for the software that has been installed. Both hierarchical and non-hierarchical deployments will be discussed in details and we will show how to customize the content and naming of each module file.

At the end of the tutorial readers should have a clear understanding of:

  • What module files are and how they are used on HPC clusters
  • How Spack generates module files for the software it installs
  • Which Spack commands can be used to manage module files
  • How module files generated by Spack can be customized

and be confident that Spack can deal with all of the common use cases that occur when maintaining software installations on HPC clusters.

Setup for the Tutorial

To prepare for this tutorial we are going to install a small but representative set of software that includes different configurations of the same packages and some external packages. To keep the installations manageable, let’s start by uninstalling everything from earlier in the tutorial:

$ spack uninstall -ay

Build a module tool

The first thing that we need is the module tool itself. In the tutorial we will use lmod because it can work with both hierarchical and non-hierarchical layouts.

$ spack install lmod

Once the module tool is installed we need to have it available in the current shell. Installation directories in Spack’s store are definitely not easy to remember, but they can be retrieved with the spack location command:

$ . $(spack location -i lmod)/lmod/lmod/init/bash

Now we can re-source the setup file and Spack modules will be put in our module path.

$ . share/spack/setup-env.sh

Add a new compiler

The second step is to build a recent compiler. On first use, Spack scans the environment and automatically locates the compiler(s) already available on the system. For this tutorial, however, we want to use gcc@8.4.0.

$ spack install gcc@8.4.0

You can get this in your environment using spack load gcc@8.4.0:

$ spack load gcc@8.4.0
$ which gcc
/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/gcc

Now, gcc is in your PATH. You can add it to the list of compilers with spack compiler add:

$ spack compiler add
==> Added 1 new compiler to /home/spack/.spack/linux/compilers.yaml
    gcc@8.4.0
==> Compilers are defined in the following files:
    /home/spack/.spack/linux/compilers.yaml

To check which compilers are available you can use spack compiler list:

$ spack compiler list
==> Available compilers
-- clang ubuntu18.04-x86_64 -------------------------------------
clang@7.0.0

-- gcc ubuntu18.04-x86_64 ---------------------------------------
gcc@8.4.0  gcc@7.5.0  gcc@6.5.0

Finally, when you confirmed gcc@8.4.0 is properly registered, clean the environment with spack unload:

$ spack unload --all

Build the software that will be used in the tutorial

Finally, we will use Spack to install the packages used in the examples:

$ spack install netlib-scalapack ^openmpi ^openblas
$ spack install netlib-scalapack ^mpich ^openblas
$ spack install netlib-scalapack ^openmpi ^netlib-lapack
$ spack install netlib-scalapack ^mpich ^netlib-lapack
$ spack install py-scipy ^openblas

What are Module Files?

Module files are an easy way to modify your environment in a controlled manner during a shell session. In general, they contain the information needed to run an application or use a library. The module command is used to interpret and execute module files. For example, module show tells you what a module will do when loaded:

$ module show zlib-1.2.12-gcc-8.4.0-rlo3go7
----------------------------------------------------------------------------------------------------------------------------------------------------------
   /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/zlib-1.2.12-gcc-8.4.0-rlo3go7:
----------------------------------------------------------------------------------------------------------------------------------------------------------
whatis("A free, general-purpose, legally unencumbered lossless data-compression library.")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/zlib-1.2.12-rlo3go7mpfwicumepslj4pth7z4t3a46/lib")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/zlib-1.2.12-rlo3go7mpfwicumepslj4pth7z4t3a46/share/man")
prepend_path("PKG_CONFIG_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/zlib-1.2.12-rlo3go7mpfwicumepslj4pth7z4t3a46/lib/pkgconfig")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/zlib-1.2.12-rlo3go7mpfwicumepslj4pth7z4t3a46/")
help([[A free, general-purpose, legally unencumbered lossless data-compression
library.
]])

$ echo $LD_LIBRARY_PATH

module load will execute all of the changes shown above:

$ module load zlib-1.2.12-gcc-8.4.0-rlo3go7
$ echo $LD_LIBRARY_PATH
/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/zlib-1.2.12-rlo3go7mpfwicumepslj4pth7z4t3a46/lib

and to undo the modifications, you can use module unload:

$ module unload zlib
$ echo $LD_LIBRARY_PATH

$

Module Systems

There are two main module systems used in HPC, both installable by Spack. In this tutorial we will be working with lmod and be showing examples with both Tcl and Lua.

Environment Modules

This is the original modules tool. It can be installed with Spack using the following command:

$ spack install environment-modules

It was first coded in C in the early 1990s and was later rewritten entirely in Tcl. Long stagnant, the project has been revived in the past few years by Xavier Delaruelle at CEA, and it is now very actively developed. For further details we refer to its documentation.

Lmod

Lmod is a module system written in Lua, originally created at the “Texas Advanced Computing Center” (TACC) by Robert McLay. You can get it with:

$ spack install lmod

as shown in the Setup for the Tutorial section. It is a drop-in replacement for Environment Modules, and it works with both Tcl and Lua module files. It is fully compatible with Environment Modules, but it also has many distinguishing features of its own. The main one is the module hierarchy, which simplifies the module UI by only showing modules built with the currently loaded compiler and/or MPI. There are also some unique safety features.

How does Spack generate module files?

Before we dive into the hands-on sections it’s worth explaining how module files are generated by Spack. The following diagram provides a high-level view of the process:

_images/module_file_generation.png

Modules in Spack are generated using configuration files (config.yaml and modules.yaml), information from Spack’s package recipes, and Jinja2 templates. Spack comes with Jinja2, an external template engine, so you do not need to install it yourself.

Modules vs spack load

You may have noticed that we used spack load in the Setup for the Tutorial section above. This is a built-in mechanism of Spack’s – it’s designed so that users on a cluster or a laptop can quickly get a package into their path, and it understands Spack’s spec syntax. It does not require modules, as Spack needs to work regardless of whether modules are set up on the system.

As you might expect, you can see what is loaded via spack load using spack find:

$ spack find --loaded
==> 21 loaded packages
-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
autoconf@2.69	     bzip2@1.0.8    gcc@8.4.0  libiconv@1.16	m4@1.4.19   ncurses@6.2    readline@8.1
automake@1.16.5      diffutils@3.8  gdbm@1.19  libsigsegv@2.13	mpc@1.1.0   perl@5.34.1    texinfo@6.5
berkeley-db@18.1.40  gawk@5.1.1     gmp@6.2.1  libtool@2.4.7	mpfr@3.1.6  pkgconf@1.8.0  zlib@1.2.12

Because Spack is designed to be run on HPC systems, it also generates a module file for every installed package. This allows users unfamiliar with Spack’s interface to see things through the module system they’re used to. To see this, try:

$ module avail

----------------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------------
   autoconf-2.69-gcc-7.5.0-hlj3ujm	    libiconv-1.16-gcc-7.5.0-dr56jt4		 ncurses-6.2-gcc-7.5.0-frwzsc4
   automake-1.16.5-gcc-7.5.0-6vk5ehd	    libsigsegv-2.13-gcc-7.5.0-ubezeyc		 openssl-1.1.1o-gcc-7.5.0-hqlqpsn
   berkeley-db-18.1.40-gcc-7.5.0-q6tqmmm    libtool-2.4.7-gcc-7.5.0-7fhdrxx		 perl-5.34.1-gcc-7.5.0-pn5mtes
   bzip2-1.0.8-gcc-7.5.0-cyn2yam	    lmod-8.7.2-gcc-7.5.0-meewols		 pkgconf-1.8.0-gcc-7.5.0-i5po27g
   curl-7.83.0-gcc-7.5.0-7fafhiv	    lua-5.3.5-gcc-7.5.0-w26r5kv 		 readline-8.1-gcc-7.5.0-ls6erxw
   diffutils-3.8-gcc-7.5.0-k22pe4r	    lua-luafilesystem-1_8_0-gcc-7.5.0-hgwsbzx	 tcl-8.6.12-gcc-7.5.0-6m3sqvr
   gawk-5.1.1-gcc-7.5.0-ejl2btv 	    lua-luaposix-35.0-gcc-7.5.0-kkrcozh 	 texinfo-6.5-gcc-7.5.0-ot5b2ak
   gcc-8.4.0-gcc-7.5.0-pufwwwg		    m4-1.4.19-gcc-7.5.0-uadbn2a 		 unzip-6.0-gcc-7.5.0-qos6nfh
   gdbm-1.19-gcc-7.5.0-xub4jsy		    mpc-1.1.0-gcc-7.5.0-lmcyfxq 		 zlib-1.2.12-gcc-7.5.0-fntvsj6
   gmp-6.2.1-gcc-7.5.0-lf7yoxu		    mpfr-3.1.6-gcc-7.5.0-vmql5cb

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

You can module load any of these. By default, Spack generates modules named by package-version-compiler-version-hash, which is a bit hard to read. We’ll show you how to customize this in the following sections.

Non-hierarchical Module Files

If you arrived to this point you should have an environment that looks similar to:

$ module avail

----------------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------------
   autoconf-2.69-gcc-7.5.0-hlj3ujm	     libsigsegv-2.13-gcc-7.5.0-ubezeyc		  pigz-2.7-gcc-8.4.0-4ocehhg
   autoconf-2.69-gcc-8.4.0-vtvkpes	     libsigsegv-2.13-gcc-8.4.0-uqf72xt		  pkgconf-1.8.0-gcc-7.5.0-i5po27g
   automake-1.16.5-gcc-7.5.0-6vk5ehd	     libtool-2.4.7-gcc-7.5.0-7fhdrxx		  pkgconf-1.8.0-gcc-8.4.0-ymaijtn
   automake-1.16.5-gcc-8.4.0-62t26ck	     libtool-2.4.7-gcc-8.4.0-r4hwigw		  pmix-4.1.2-gcc-8.4.0-6ufltz3
   berkeley-db-18.1.40-gcc-7.5.0-q6tqmmm     libxml2-2.9.13-gcc-8.4.0-ycmb6ld		  py-beniget-0.4.1-gcc-8.4.0-zolqn3v
   berkeley-db-18.1.40-gcc-8.4.0-johvuo3     lmod-8.7.2-gcc-7.5.0-meewols		  py-cython-0.29.30-gcc-8.4.0-zwkgtqx
   bzip2-1.0.8-gcc-7.5.0-cyn2yam	     lua-5.3.5-gcc-7.5.0-w26r5kv		  py-gast-0.5.3-gcc-8.4.0-5zmfno5
   bzip2-1.0.8-gcc-8.4.0-jd4qgnl	     lua-luafilesystem-1_8_0-gcc-7.5.0-hgwsbzx	  py-numpy-1.22.4-gcc-8.4.0-7wyqab4
   cmake-3.23.1-gcc-8.4.0-ag23oxs	     lua-luaposix-35.0-gcc-7.5.0-kkrcozh	  py-pip-21.3.1-gcc-8.4.0-hcixple
   curl-7.83.0-gcc-7.5.0-7fafhiv	     m4-1.4.19-gcc-7.5.0-uadbn2a		  py-ply-3.11-gcc-8.4.0-a6oqnnp
   diffutils-3.8-gcc-7.5.0-k22pe4r	     m4-1.4.19-gcc-8.4.0-cv6tp6u		  py-pybind11-2.8.1-gcc-8.4.0-v2ltwp2
   diffutils-3.8-gcc-8.4.0-3lphb6j	     mpc-1.1.0-gcc-7.5.0-lmcyfxq		  py-pythran-0.11.0-gcc-8.4.0-uy2pjua
   expat-2.4.8-gcc-8.4.0-gdftkwk	     mpfr-3.1.6-gcc-7.5.0-vmql5cb		  py-scipy-1.8.1-gcc-8.4.0-bk4orou
   findutils-4.9.0-gcc-8.4.0-hxs6igk	     mpich-4.0.2-gcc-8.4.0-6vy27u3		  py-setuptools-59.4.0-gcc-8.4.0-wovl75y
   gawk-5.1.1-gcc-7.5.0-ejl2btv 	     ncurses-6.2-gcc-7.5.0-frwzsc4		  py-wheel-0.37.0-gcc-8.4.0-6p3qaxv
   gcc-8.4.0-gcc-7.5.0-pufwwwg		     ncurses-6.2-gcc-8.4.0-o3zj6m3		  python-3.9.12-gcc-8.4.0-bxqohdj
   gdbm-1.19-gcc-7.5.0-xub4jsy		     netlib-lapack-3.10.1-gcc-8.4.0-tvolw2r	  readline-8.1-gcc-7.5.0-ls6erxw
   gdbm-1.19-gcc-8.4.0-eyzftkv		     netlib-scalapack-2.2.0-gcc-8.4.0-2wlv7e4	  readline-8.1-gcc-8.4.0-lety7xd
   gettext-0.21-gcc-8.4.0-yxcscic	     netlib-scalapack-2.2.0-gcc-8.4.0-ck7ppoo	  sqlite-3.38.5-gcc-8.4.0-fs4yqym
   gmp-6.2.1-gcc-7.5.0-lf7yoxu		     netlib-scalapack-2.2.0-gcc-8.4.0-tacocwd	  tar-1.34-gcc-8.4.0-auxqpps
   hwloc-2.7.1-gcc-8.4.0-27dpmmo	     netlib-scalapack-2.2.0-gcc-8.4.0-yxm6ejy	  tcl-8.6.12-gcc-7.5.0-6m3sqvr
   libbsd-0.11.5-gcc-8.4.0-v2foqu3	     ninja-1.10.2-gcc-8.4.0-3igl6tj		  texinfo-6.5-gcc-7.5.0-ot5b2ak
   libedit-3.1-20210216-gcc-8.4.0-riidlo4    numactl-2.0.14-gcc-8.4.0-a3lmunk		  unzip-6.0-gcc-7.5.0-qos6nfh
   libevent-2.1.12-gcc-8.4.0-vhrgro6	     openblas-0.3.20-gcc-8.4.0-ew72qgy		  util-linux-uuid-2.37.4-gcc-8.4.0-sgf54vj
   libfabric-1.14.1-gcc-8.4.0-sbpqoci	     openmpi-4.1.3-gcc-8.4.0-c4md4py		  util-macros-1.19.3-gcc-8.4.0-kf4wzxd
   libffi-3.4.2-gcc-8.4.0-qqvp57p	     openssh-9.0p1-gcc-8.4.0-2nsdyqm		  xz-5.2.5-gcc-8.4.0-zukecls
   libiconv-1.16-gcc-7.5.0-dr56jt4	     openssl-1.1.1o-gcc-7.5.0-hqlqpsn		  yaksa-0.2-gcc-8.4.0-vx5fw3m
   libiconv-1.16-gcc-8.4.0-nbhph73	     openssl-1.1.1o-gcc-8.4.0-bj2jnt7		  zlib-1.2.12-gcc-7.5.0-fntvsj6
   libmd-1.0.4-gcc-8.4.0-xr5qbzs	     perl-5.34.1-gcc-7.5.0-pn5mtes		  zlib-1.2.12-gcc-8.4.0-rlo3go7
   libpciaccess-0.16-gcc-8.4.0-4omfqev	     perl-5.34.1-gcc-8.4.0-yycipnr		  zstd-1.5.2-gcc-8.4.0-cdbbbci

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

The non-hierarchical module files that have been generated so far follow Spack’s default rules for module generation. Taking a look at the gcc module you’ll see, for example:

$ module show gcc-8.4.0-gcc-7.5.0-pufwwwg
----------------------------------------------------------------------------------------------------------------------------------------------------------
   /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/gcc-8.4.0-gcc-7.5.0-pufwwwg:
----------------------------------------------------------------------------------------------------------------------------------------------------------
whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages.")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/lib")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/lib64")
prepend_path("PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/share/man")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/")
setenv("CC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/gcc")
setenv("CXX","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/g++")
setenv("FC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/gfortran")
setenv("F77","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/gfortran")
help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C,
Fortran, Ada, and Go, as well as libraries for these languages.
]])

As expected, a few environment variables representing paths will be modified by the module file according to the default prefix inspection rules.

Filter unwanted modifications to the environment

Now consider the case that your site has decided that CC, CXX, FC and F77 modifications should not be present in module files. What you can do to abide by the rules is to create a configuration file ${SPACK_ROOT}/etc/spack/modules.yaml with the following content:

modules:
  default:
    tcl:
      all:
        filter:
          exclude_env_vars:
          - "CC"
          - "CXX"
          - "FC"
          - "F77"

Next you should regenerate all the module files:

$ spack module tcl refresh
y

==> You are about to regenerate tcl module files for:

-- linux-ubuntu18.04-x86_64 / gcc@7.5.0 -------------------------
hlj3ujm autoconf@2.69
6vk5ehd automake@1.16.5
q6tqmmm berkeley-db@18.1.40
cyn2yam bzip2@1.0.8
7fafhiv curl@7.83.0
k22pe4r diffutils@3.8
ejl2btv gawk@5.1.1
pufwwwg gcc@8.4.0
xub4jsy gdbm@1.19
lf7yoxu gmp@6.2.1
dr56jt4 libiconv@1.16
ubezeyc libsigsegv@2.13
7fhdrxx libtool@2.4.7
meewols lmod@8.7.2
w26r5kv lua@5.3.5
hgwsbzx lua-luafilesystem@1_8_0
kkrcozh lua-luaposix@35.0
uadbn2a m4@1.4.19
lmcyfxq mpc@1.1.0
vmql5cb mpfr@3.1.6
frwzsc4 ncurses@6.2
hqlqpsn openssl@1.1.1o
pn5mtes perl@5.34.1
i5po27g pkgconf@1.8.0
ls6erxw readline@8.1
6m3sqvr tcl@8.6.12
ot5b2ak texinfo@6.5
qos6nfh unzip@6.0
fntvsj6 zlib@1.2.12

-- linux-ubuntu18.04-x86_64 / gcc@8.4.0 -------------------------
vtvkpes autoconf@2.69
62t26ck automake@1.16.5
johvuo3 berkeley-db@18.1.40
jd4qgnl bzip2@1.0.8
ag23oxs cmake@3.23.1
3lphb6j diffutils@3.8
gdftkwk expat@2.4.8
hxs6igk findutils@4.9.0
eyzftkv gdbm@1.19
yxcscic gettext@0.21
27dpmmo hwloc@2.7.1
v2foqu3 libbsd@0.11.5
riidlo4 libedit@3.1-20210216
vhrgro6 libevent@2.1.12
sbpqoci libfabric@1.14.1
qqvp57p libffi@3.4.2
nbhph73 libiconv@1.16
xr5qbzs libmd@1.0.4
4omfqev libpciaccess@0.16
uqf72xt libsigsegv@2.13
r4hwigw libtool@2.4.7
ycmb6ld libxml2@2.9.13
cv6tp6u m4@1.4.19
6vy27u3 mpich@4.0.2
o3zj6m3 ncurses@6.2
tvolw2r netlib-lapack@3.10.1
ck7ppoo netlib-scalapack@2.2.0
tacocwd netlib-scalapack@2.2.0
yxm6ejy netlib-scalapack@2.2.0
2wlv7e4 netlib-scalapack@2.2.0
3igl6tj ninja@1.10.2
a3lmunk numactl@2.0.14
ew72qgy openblas@0.3.20
c4md4py openmpi@4.1.3
2nsdyqm openssh@9.0p1
bj2jnt7 openssl@1.1.1o
yycipnr perl@5.34.1
4ocehhg pigz@2.7
ymaijtn pkgconf@1.8.0
6ufltz3 pmix@4.1.2
zolqn3v py-beniget@0.4.1
zwkgtqx py-cython@0.29.30
5zmfno5 py-gast@0.5.3
7wyqab4 py-numpy@1.22.4
hcixple py-pip@21.3.1
a6oqnnp py-ply@3.11
v2ltwp2 py-pybind11@2.8.1
uy2pjua py-pythran@0.11.0
bk4orou py-scipy@1.8.1
wovl75y py-setuptools@59.4.0
6p3qaxv py-wheel@0.37.0
bxqohdj python@3.9.12
lety7xd readline@8.1
fs4yqym sqlite@3.38.5
auxqpps tar@1.34
sgf54vj util-linux-uuid@2.37.4
kf4wzxd util-macros@1.19.3
zukecls xz@5.2.5
vx5fw3m yaksa@0.2
rlo3go7 zlib@1.2.12
cdbbbci zstd@1.5.2

==> Do you want to proceed? [y/n] ==> Regenerating tcl module files

If you take a look now at the module for gcc you’ll see that the unwanted paths have disappeared:

$ module show gcc-8.4.0-gcc-7.5.0-pufwwwg
----------------------------------------------------------------------------------------------------------------------------------------------------------
   /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/gcc-8.4.0-gcc-7.5.0-pufwwwg:
----------------------------------------------------------------------------------------------------------------------------------------------------------
whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages.")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/lib")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/lib64")
prepend_path("PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/share/man")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/")
setenv("CC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/gcc")
setenv("CXX","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/g++")
setenv("FC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/gfortran")
setenv("F77","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/gfortran")
help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C,
Fortran, Ada, and Go, as well as libraries for these languages.
]])

Prevent some module files from being generated

Another common request at many sites is to avoid exposing software that is only needed as an intermediate step when building a newer stack. Let’s try to prevent the generation of module files for anything that is compiled with gcc@7.5.0 (the OS provided compiler).

To do this you should add the exclude keyword to ${SPACK_ROOT}/etc/spack/modules.yaml:

modules:
  default:
    tcl:
      exclude:
      -  '%gcc@7.5.0'
      all:
        filter:
          exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"

and regenerate the module files. This time we’ll pass the option --delete-tree so that Spack will delete the existing module tree and regenerate a new one, instead of overwriting the files in the existing directory.

$ spack module tcl refresh --delete-tree -y
==> Regenerating tcl module files
$ module avail

----------------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------------
   autoconf-2.69-gcc-8.4.0-vtvkpes	     libxml2-2.9.13-gcc-8.4.0-ycmb6ld		 py-gast-0.5.3-gcc-8.4.0-5zmfno5
   automake-1.16.5-gcc-8.4.0-62t26ck	     m4-1.4.19-gcc-8.4.0-cv6tp6u		 py-numpy-1.22.4-gcc-8.4.0-7wyqab4
   berkeley-db-18.1.40-gcc-8.4.0-johvuo3     mpich-4.0.2-gcc-8.4.0-6vy27u3		 py-pip-21.3.1-gcc-8.4.0-hcixple
   bzip2-1.0.8-gcc-8.4.0-jd4qgnl	     ncurses-6.2-gcc-8.4.0-o3zj6m3		 py-ply-3.11-gcc-8.4.0-a6oqnnp
   cmake-3.23.1-gcc-8.4.0-ag23oxs	     netlib-lapack-3.10.1-gcc-8.4.0-tvolw2r	 py-pybind11-2.8.1-gcc-8.4.0-v2ltwp2
   diffutils-3.8-gcc-8.4.0-3lphb6j	     netlib-scalapack-2.2.0-gcc-8.4.0-2wlv7e4	 py-pythran-0.11.0-gcc-8.4.0-uy2pjua
   expat-2.4.8-gcc-8.4.0-gdftkwk	     netlib-scalapack-2.2.0-gcc-8.4.0-ck7ppoo	 py-scipy-1.8.1-gcc-8.4.0-bk4orou
   findutils-4.9.0-gcc-8.4.0-hxs6igk	     netlib-scalapack-2.2.0-gcc-8.4.0-tacocwd	 py-setuptools-59.4.0-gcc-8.4.0-wovl75y
   gdbm-1.19-gcc-8.4.0-eyzftkv		     netlib-scalapack-2.2.0-gcc-8.4.0-yxm6ejy	 py-wheel-0.37.0-gcc-8.4.0-6p3qaxv
   gettext-0.21-gcc-8.4.0-yxcscic	     ninja-1.10.2-gcc-8.4.0-3igl6tj		 python-3.9.12-gcc-8.4.0-bxqohdj
   hwloc-2.7.1-gcc-8.4.0-27dpmmo	     numactl-2.0.14-gcc-8.4.0-a3lmunk		 readline-8.1-gcc-8.4.0-lety7xd
   libbsd-0.11.5-gcc-8.4.0-v2foqu3	     openblas-0.3.20-gcc-8.4.0-ew72qgy		 sqlite-3.38.5-gcc-8.4.0-fs4yqym
   libedit-3.1-20210216-gcc-8.4.0-riidlo4    openmpi-4.1.3-gcc-8.4.0-c4md4py		 tar-1.34-gcc-8.4.0-auxqpps
   libevent-2.1.12-gcc-8.4.0-vhrgro6	     openssh-9.0p1-gcc-8.4.0-2nsdyqm		 util-linux-uuid-2.37.4-gcc-8.4.0-sgf54vj
   libfabric-1.14.1-gcc-8.4.0-sbpqoci	     openssl-1.1.1o-gcc-8.4.0-bj2jnt7		 util-macros-1.19.3-gcc-8.4.0-kf4wzxd
   libffi-3.4.2-gcc-8.4.0-qqvp57p	     perl-5.34.1-gcc-8.4.0-yycipnr		 xz-5.2.5-gcc-8.4.0-zukecls
   libiconv-1.16-gcc-8.4.0-nbhph73	     pigz-2.7-gcc-8.4.0-4ocehhg 		 yaksa-0.2-gcc-8.4.0-vx5fw3m
   libmd-1.0.4-gcc-8.4.0-xr5qbzs	     pkgconf-1.8.0-gcc-8.4.0-ymaijtn		 zlib-1.2.12-gcc-8.4.0-rlo3go7
   libpciaccess-0.16-gcc-8.4.0-4omfqev	     pmix-4.1.2-gcc-8.4.0-6ufltz3		 zstd-1.5.2-gcc-8.4.0-cdbbbci
   libsigsegv-2.13-gcc-8.4.0-uqf72xt	     py-beniget-0.4.1-gcc-8.4.0-zolqn3v
   libtool-2.4.7-gcc-8.4.0-r4hwigw	     py-cython-0.29.30-gcc-8.4.0-zwkgtqx

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

if you look closely you’ll see though that we went too far in excluding modules: the module for gcc@8.4.0 disappeared as it was bootstrapped with gcc@7.5.0. To specify exceptions to the exclude rules you can use include:

modules:
  default:
    tcl:
      include:
      -  gcc
      exclude:
      -  '%gcc@7.5.0'
      all:
        filter:
          exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"

include rules always have precedence over exclude rules. If you regenerate the modules again:

$ spack module tcl refresh -y
==> Regenerating tcl module files

you’ll see that now the module for gcc@8.4.0 has reappeared:

$ module avail gcc-8.4.0-gcc-7.5.0-pufwwwg

----------------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------------
   gcc-8.4.0-gcc-7.5.0-pufwwwg

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

An additional feature that you can leverage to unclutter the environment is to skip the generation of module files for implicitly installed packages. In this case you only need to add the following line:

modules:
  default:
    tcl:
      exclude_implicits: true
      include:
      -  gcc
      exclude:
      -  '%gcc@7.5.0'
      all:
        filter:
          exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"

to modules.yaml and regenerate the module file tree as above.

Change module file naming

The next step in making module files more user-friendly is to improve their naming scheme. To reduce the length of the hash or remove it altogether you can use the hash_length keyword in the configuration file:

modules:
  default:
    tcl:
      hash_length: 0
      include:
      -  gcc
      exclude:
      -  '%gcc@7.5.0'
      all:
        filter:
          exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"

If you try to regenerate the module files now you will get an error:

$ spack module tcl refresh --delete-tree -y
==> Error: Name clashes detected in module files:

file: /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/netlib-scalapack-2.2.0-gcc-8.4.0
spec: netlib-scalapack@2.2.0%gcc@8.4.0~ipo~pic+shared build_type=Release patches=072b006,1c9ce5f arch=linux-ubuntu18.04-x86_64
spec: netlib-scalapack@2.2.0%gcc@8.4.0~ipo~pic+shared build_type=Release patches=072b006,1c9ce5f arch=linux-ubuntu18.04-x86_64
spec: netlib-scalapack@2.2.0%gcc@8.4.0~ipo~pic+shared build_type=Release patches=072b006,1c9ce5f arch=linux-ubuntu18.04-x86_64
spec: netlib-scalapack@2.2.0%gcc@8.4.0~ipo~pic+shared build_type=Release patches=072b006,1c9ce5f arch=linux-ubuntu18.04-x86_64

==> Error: Operation aborted

Note

We try to check for errors up front!

In Spack we check for errors upfront whenever possible, so don’t worry about your module files: as a name clash was detected nothing has been changed on disk.

The problem here is that without the hashes the four different flavors of netlib-scalapack map to the same module file name. We can change how the names are formatted to differentiate them:

modules:
  default:
    tcl:
      hash_length: 0
      include:
      -  gcc
      exclude:
      -  '%gcc@7.5.0'
      all:
        conflict:
        - '{name}'
        filter:
          exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"
      projections:
        all:               '{name}/{version}-{compiler.name}-{compiler.version}'
        netlib-scalapack:  '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
        ^python^lapack:    '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'

As you can see it is possible to specify rules that apply only to a restricted set of packages using anonymous specs like ^mpi^lapack. Here we declare a conflict between any two modules with the same name, so they cannot be loaded together. We also format the names of modules according to compiler, compiler version, and MPI version using the spec format syntax. This allows us to match specs by their dependencies, and format them based on their DAGs.

$ spack module tcl refresh --delete-tree -y
==> Regenerating tcl module files
$ module avail

----------------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------------
   autoconf/2.69-gcc-8.4.0	     libtool/2.4.7-gcc-8.4.0					   py-cython/0.29.30-gcc-8.4.0
   automake/1.16.5-gcc-8.4.0	     libxml2/2.9.13-gcc-8.4.0					   py-gast/0.5.3-gcc-8.4.0
   berkeley-db/18.1.40-gcc-8.4.0     m4/1.4.19-gcc-8.4.0					   py-numpy/1.22.4-gcc-8.4.0-openblas
   bzip2/1.0.8-gcc-8.4.0	     mpich/4.0.2-gcc-8.4.0					   py-pip/21.3.1-gcc-8.4.0
   cmake/3.23.1-gcc-8.4.0	     ncurses/6.2-gcc-8.4.0					   py-ply/3.11-gcc-8.4.0
   diffutils/3.8-gcc-8.4.0	     netlib-lapack/3.10.1-gcc-8.4.0				   py-pybind11/2.8.1-gcc-8.4.0
   expat/2.4.8-gcc-8.4.0	     netlib-scalapack/2.2.0-gcc-8.4.0-netlib-lapack-mpich	   py-pythran/0.11.0-gcc-8.4.0-openblas
   findutils/4.9.0-gcc-8.4.0	     netlib-scalapack/2.2.0-gcc-8.4.0-netlib-lapack-openmpi	   py-scipy/1.8.1-gcc-8.4.0-openblas
   gcc/8.4.0-gcc-7.5.0		     netlib-scalapack/2.2.0-gcc-8.4.0-openblas-mpich		   py-setuptools/59.4.0-gcc-8.4.0
   gdbm/1.19-gcc-8.4.0		     netlib-scalapack/2.2.0-gcc-8.4.0-openblas-openmpi	    (D)    py-wheel/0.37.0-gcc-8.4.0
   gettext/0.21-gcc-8.4.0	     ninja/1.10.2-gcc-8.4.0					   python/3.9.12-gcc-8.4.0
   hwloc/2.7.1-gcc-8.4.0	     numactl/2.0.14-gcc-8.4.0					   readline/8.1-gcc-8.4.0
   libbsd/0.11.5-gcc-8.4.0	     openblas/0.3.20-gcc-8.4.0					   sqlite/3.38.5-gcc-8.4.0
   libedit/3.1-20210216-gcc-8.4.0    openmpi/4.1.3-gcc-8.4.0					   tar/1.34-gcc-8.4.0
   libevent/2.1.12-gcc-8.4.0	     openssh/9.0p1-gcc-8.4.0					   util-linux-uuid/2.37.4-gcc-8.4.0
   libfabric/1.14.1-gcc-8.4.0	     openssl/1.1.1o-gcc-8.4.0					   util-macros/1.19.3-gcc-8.4.0
   libffi/3.4.2-gcc-8.4.0	     perl/5.34.1-gcc-8.4.0					   xz/5.2.5-gcc-8.4.0
   libiconv/1.16-gcc-8.4.0	     pigz/2.7-gcc-8.4.0 					   yaksa/0.2-gcc-8.4.0
   libmd/1.0.4-gcc-8.4.0	     pkgconf/1.8.0-gcc-8.4.0					   zlib/1.2.12-gcc-8.4.0
   libpciaccess/0.16-gcc-8.4.0	     pmix/4.1.2-gcc-8.4.0					   zstd/1.5.2-gcc-8.4.0
   libsigsegv/2.13-gcc-8.4.0	     py-beniget/0.4.1-gcc-8.4.0

  Where:
   D:  Default Module

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Note

The conflict directive is Tcl-specific and can’t be used in the lmod section of the configuration file.

Add custom environment modifications

At many sites it is customary to set an environment variable in a package’s module file that points to the folder in which the package is installed. You can achieve this with Spack by adding an environment directive to the configuration file:

modules:
  default:
    tcl:
      hash_length: 0
      naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
      include:
      -  gcc
      exclude:
      -  '%gcc@7.5.0'
      all:
        conflict:
        - '{name}'
        filter:
          exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"
        environment:
          set:
            '{name}_ROOT': '{prefix}'
      projections:
        all:               '{name}/{version}-{compiler.name}-{compiler.version}'
        netlib-scalapack:  '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
        ^python^lapack:    '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'

Under the hood Spack uses the format() API to substitute tokens in either environment variable names or values. There are two caveats though:

  • The set of allowed tokens in variable names is restricted to name, version, compiler, compiler.name, compiler.version, architecture
  • Any token expanded in a variable name is made uppercase, but other than that case sensitivity is preserved

Regenerating the module files results in something like:

$ spack module tcl refresh -y
==> Regenerating tcl module files
$ module show gcc
----------------------------------------------------------------------------------------------------------------------------------------------------------
   /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/gcc/8.4.0-gcc-7.5.0:
----------------------------------------------------------------------------------------------------------------------------------------------------------
whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages.")
conflict("gcc")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/lib")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/lib64")
prepend_path("PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/share/man")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/")
setenv("CC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/gcc")
setenv("CXX","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/g++")
setenv("FC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/gfortran")
setenv("F77","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl/bin/gfortran")
setenv("'GCC_ROOT'","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.4.0-pufwwwg4qlpjsgg6oy5yzk7hpgi5kdkl")
help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C,
Fortran, Ada, and Go, as well as libraries for these languages.
]])

As you can see, the gcc module has the environment variable GCC_ROOT set.

Sometimes it’s also useful to apply environment modifications selectively and target only certain packages. You can for instance apply modifications to the openmpi module as follows:

modules:
  default:
    tcl:
      hash_length: 0
      naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
      include:
      - gcc
      exclude:
      - '%gcc@7.5.0'
      all:
        conflict:
        - '{name}'
        filter:
          exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"
        environment:
          set:
            '{name}_ROOT': '{prefix}'
      openmpi:
        environment:
          set:
            SLURM_MPI_TYPE: pmi2
            OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
      projections:
        all:               '{name}/{version}-{compiler.name}-{compiler.version}'
        netlib-scalapack:  '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
        ^python^lapack:    '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'

This time we will be more selective and regenerate only the openmpi module file:

$ spack module tcl refresh -y openmpi
==> Regenerating tcl module files
$ module show openmpi
----------------------------------------------------------------------------------------------------------------------------------------------------------
   /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/openmpi/4.1.3-gcc-8.4.0:
----------------------------------------------------------------------------------------------------------------------------------------------------------
whatis("An open source Message Passing Interface implementation.")
conflict("openmpi")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/openmpi-4.1.3-c4md4pybhcksh7fvpi32fm4lysvoqjza/lib")
prepend_path("PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/openmpi-4.1.3-c4md4pybhcksh7fvpi32fm4lysvoqjza/bin")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/openmpi-4.1.3-c4md4pybhcksh7fvpi32fm4lysvoqjza/share/man")
prepend_path("PKG_CONFIG_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/openmpi-4.1.3-c4md4pybhcksh7fvpi32fm4lysvoqjza/lib/pkgconfig")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/openmpi-4.1.3-c4md4pybhcksh7fvpi32fm4lysvoqjza/")
setenv("MPICC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/openmpi-4.1.3-c4md4pybhcksh7fvpi32fm4lysvoqjza/bin/mpicc")
setenv("MPICXX","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/openmpi-4.1.3-c4md4pybhcksh7fvpi32fm4lysvoqjza/bin/mpic++")
setenv("MPIF77","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/openmpi-4.1.3-c4md4pybhcksh7fvpi32fm4lysvoqjza/bin/mpif77")
setenv("MPIF90","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/openmpi-4.1.3-c4md4pybhcksh7fvpi32fm4lysvoqjza/bin/mpif90")
setenv("'OPENMPI_ROOT'","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/openmpi-4.1.3-c4md4pybhcksh7fvpi32fm4lysvoqjza")
setenv("SLURM_MPI_TYPE","pmi2")
setenv("OMPI_MCA_btl_openib_warn_default_git_prefix","0")
help([[An open source Message Passing Interface implementation. The Open MPI
Project is an open source Message Passing Interface implementation that
is developed and maintained by a consortium of academic, research, and
industry partners. Open MPI is therefore able to combine the expertise,
technologies, and resources from all across the High Performance
Computing community in order to build the best MPI library available.
Open MPI offers advantages for system and software vendors, application
developers and computer science researchers.
]])

Autoload dependencies

Spack can also generate module files that contain code to load the dependencies automatically. You can, for instance generate python modules that load their dependencies by adding the autoload directive and assigning it the value direct:

modules:
  default:
    tcl:
      verbose: True
      hash_length: 0
      naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
      include:
      - gcc
      exclude:
      - '%gcc@7.5.0'
      all:
        conflict:
        - '{name}'
        filter:
          exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"
        environment:
          set:
            '{name}_ROOT': '{prefix}'
      gcc:
        environment:
          set:
            CC: gcc
            CXX: g++
            FC: gfortran
            F90: gfortran
            F77: gfortran
      openmpi:
        environment:
          set:
            SLURM_MPI_TYPE: pmi2
            OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
      projections:
        all:               '{name}/{version}-{compiler.name}-{compiler.version}'
        netlib-scalapack:  '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
        ^python^lapack:    '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'
      ^python:
        autoload:  direct

and regenerating the module files for every package that depends on python:

$ spack module tcl refresh -y ^python
==> Regenerating tcl module files

and will contain code to autoload all the dependencies:

$ module load py-scipy
Autoloading openblas/0.3.20-gcc-8.4.0
Autoloading py-numpy/1.22.4-gcc-8.4.0-openblas
Autoloading py-setuptools/59.4.0-gcc-8.4.0
Autoloading python/3.9.12-gcc-8.4.0
Autoloading python/3.9.12-gcc-8.4.0
Autoloading py-pybind11/2.8.1-gcc-8.4.0
Autoloading py-pythran/0.11.0-gcc-8.4.0-openblas
Autoloading py-beniget/0.4.1-gcc-8.4.0
Autoloading py-gast/0.5.3-gcc-8.4.0
Autoloading py-gast/0.5.3-gcc-8.4.0
Autoloading py-ply/3.11-gcc-8.4.0
Autoloading python/3.9.12-gcc-8.4.0

In case messages are unwanted during the autoload procedure, it will be sufficient to omit the line setting verbose: True in the configuration file above.

Hierarchical Module Files

So far we worked with non-hierarchical module files, i.e. with module files that are all generated in the same root directory and don’t attempt to dynamically modify the MODULEPATH. This results in a flat module structure where all the software is visible at the same time:

$ module avail

----------------------------------------------- /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64 -----------------------------------------------
   autoconf/2.69-gcc-8.4.0	     libtool/2.4.7-gcc-8.4.0					   py-cython/0.29.30-gcc-8.4.0
   automake/1.16.5-gcc-8.4.0	     libxml2/2.9.13-gcc-8.4.0					   py-gast/0.5.3-gcc-8.4.0
   berkeley-db/18.1.40-gcc-8.4.0     m4/1.4.19-gcc-8.4.0					   py-numpy/1.22.4-gcc-8.4.0-openblas
   bzip2/1.0.8-gcc-8.4.0	     mpich/4.0.2-gcc-8.4.0					   py-pip/21.3.1-gcc-8.4.0
   cmake/3.23.1-gcc-8.4.0	     ncurses/6.2-gcc-8.4.0					   py-ply/3.11-gcc-8.4.0
   diffutils/3.8-gcc-8.4.0	     netlib-lapack/3.10.1-gcc-8.4.0				   py-pybind11/2.8.1-gcc-8.4.0
   expat/2.4.8-gcc-8.4.0	     netlib-scalapack/2.2.0-gcc-8.4.0-netlib-lapack-mpich	   py-pythran/0.11.0-gcc-8.4.0-openblas
   findutils/4.9.0-gcc-8.4.0	     netlib-scalapack/2.2.0-gcc-8.4.0-netlib-lapack-openmpi	   py-scipy/1.8.1-gcc-8.4.0-openblas
   gcc/8.4.0-gcc-7.5.0		     netlib-scalapack/2.2.0-gcc-8.4.0-openblas-mpich		   py-setuptools/59.4.0-gcc-8.4.0
   gdbm/1.19-gcc-8.4.0		     netlib-scalapack/2.2.0-gcc-8.4.0-openblas-openmpi	    (D)    py-wheel/0.37.0-gcc-8.4.0
   gettext/0.21-gcc-8.4.0	     ninja/1.10.2-gcc-8.4.0					   python/3.9.12-gcc-8.4.0
   hwloc/2.7.1-gcc-8.4.0	     numactl/2.0.14-gcc-8.4.0					   readline/8.1-gcc-8.4.0
   libbsd/0.11.5-gcc-8.4.0	     openblas/0.3.20-gcc-8.4.0					   sqlite/3.38.5-gcc-8.4.0
   libedit/3.1-20210216-gcc-8.4.0    openmpi/4.1.3-gcc-8.4.0					   tar/1.34-gcc-8.4.0
   libevent/2.1.12-gcc-8.4.0	     openssh/9.0p1-gcc-8.4.0					   util-linux-uuid/2.37.4-gcc-8.4.0
   libfabric/1.14.1-gcc-8.4.0	     openssl/1.1.1o-gcc-8.4.0					   util-macros/1.19.3-gcc-8.4.0
   libffi/3.4.2-gcc-8.4.0	     perl/5.34.1-gcc-8.4.0					   xz/5.2.5-gcc-8.4.0
   libiconv/1.16-gcc-8.4.0	     pigz/2.7-gcc-8.4.0 					   yaksa/0.2-gcc-8.4.0
   libmd/1.0.4-gcc-8.4.0	     pkgconf/1.8.0-gcc-8.4.0					   zlib/1.2.12-gcc-8.4.0
   libpciaccess/0.16-gcc-8.4.0	     pmix/4.1.2-gcc-8.4.0					   zstd/1.5.2-gcc-8.4.0
   libsigsegv/2.13-gcc-8.4.0	     py-beniget/0.4.1-gcc-8.4.0

  Where:
   D:  Default Module

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

This layout is quite simple to deploy, but you can see from the above snippet that nothing prevents users from loading incompatible sets of modules:

$ module purge
$ module load netlib-lapack openblas
$ module list

Currently Loaded Modules:
  1) netlib-lapack/3.10.1-gcc-8.4.0   2) openblas/0.3.20-gcc-8.4.0

Even if conflicts directives are carefully placed in module files, they:

  • won’t enforce a consistent environment, but will just report an error
  • need constant updates, for instance as soon as a new compiler or MPI library is installed

Hierarchical module files try to overcome these shortcomings by showing at start-up only a restricted view of what is available on the system: more specifically only the software that has been installed with OS provided compilers. Among this software there will be other - usually more recent - compilers that, once loaded, will prepend new directories to MODULEPATH unlocking all the software that was compiled with them. This “unlocking” idea can then be extended arbitrarily to virtual dependencies, as we’ll see in the following section.

Core/Compiler/MPI

The most widely used hierarchy is the so called Core/Compiler/MPI where, on top of the compilers, different MPI libraries also unlock software linked to them. There are just a few steps needed to adapt the modules.yaml file we used previously:

  1. enable the lmod file generator
  2. change the tcl tag to lmod
  3. remove the tcl specific conflict directive
  4. declare which compilers are considered core_compilers
  5. remove the mpi related suffixes in projections (as they will be substituted by hierarchies)

After these modifications your configuration file should look like:

modules:
  default:
    enable::
      - lmod
    lmod:
      core_compilers:
      - 'gcc@7.5.0'
      hierarchy:
      - mpi
      hash_length: 0
      include:
      - gcc
      exclude:
      - '%gcc@7.5.0'
      all:
        filter:
          exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"
        environment:
          set:
            '{name}_ROOT': '{prefix}'
      openmpi:
        environment:
          set:
            SLURM_MPI_TYPE: pmi2
            OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
      projections:
        all:          '{name}/{version}'
        ^lapack:      '{name}/{version}-{^lapack.name}'

Note

Double colon in configuration files
The double colon after enable is intentional and it serves the purpose of overriding the default list of enabled generators so that only lmod will be active (see Overriding entire sections for more details).

The directive core_compilers accepts a list of compilers. Everything built using these compilers will create a module in the Core part of the hierarchy, which is the entry point for hierarchical module files. It is common practice to put the OS provided compilers in the list and only build common utilities and other compilers with them.

If we now regenerate the module files:

$ spack module lmod refresh --delete-tree -y
==> Regenerating lmod module files

and update MODULEPATH to point to the Core:

$ module purge
$ module unuse $HOME/spack/share/spack/modules/linux-ubuntu18.04-x86_64
$ module use $HOME/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core

asking for the available modules will return:

$ module avail

---------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------------
   gcc/8.4.0

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Unsurprisingly, the only visible module is gcc. Loading that we’ll unlock the Compiler part of the hierarchy:

$ module load gcc
$ module avail

------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/gcc/8.4.0 --------------------------------------------
   autoconf/2.69	  hwloc/2.7.1		  libtool/2.4.7 	  openssh/9.0p1 	      py-pip/21.3.1		    tar/1.34
   automake/1.16.5	  libbsd/0.11.5 	  libxml2/2.9.13	  openssl/1.1.1o	      py-ply/3.11		    util-linux-uuid/2.37.4
   berkeley-db/18.1.40	  libedit/3.1-20210216	  m4/1.4.19		  perl/5.34.1		      py-pybind11/2.8.1 	    util-macros/1.19.3
   bzip2/1.0.8		  libevent/2.1.12	  mpich/4.0.2		  pigz/2.7		      py-pythran/0.11.0-openblas    xz/5.2.5
   cmake/3.23.1 	  libfabric/1.14.1	  ncurses/6.2		  pkgconf/1.8.0 	      py-scipy/1.8.1-openblas	    yaksa/0.2
   diffutils/3.8	  libffi/3.4.2		  netlib-lapack/3.10.1	  pmix/4.1.2		      py-setuptools/59.4.0	    zlib/1.2.12
   expat/2.4.8		  libiconv/1.16 	  ninja/1.10.2		  py-beniget/0.4.1	      py-wheel/0.37.0		    zstd/1.5.2
   findutils/4.9.0	  libmd/1.0.4		  numactl/2.0.14	  py-cython/0.29.30	      python/3.9.12
   gdbm/1.19		  libpciaccess/0.16	  openblas/0.3.20	  py-gast/0.5.3 	      readline/8.1
   gettext/0.21 	  libsigsegv/2.13	  openmpi/4.1.3 	  py-numpy/1.22.4-openblas    sqlite/3.38.5

---------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------------
   gcc/8.4.0 (L)

  Where:
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

The same holds true also for the MPI part, that you can enable by loading either mpich or openmpi. Let’s start by loading mpich:

$ module load mpich
$ module avail

--------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/mpich/4.0.2-6vy27u3/gcc/8.4.0 ----------------------------------
   netlib-scalapack/2.2.0-netlib-lapack    netlib-scalapack/2.2.0-openblas (D)

------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/gcc/8.4.0 --------------------------------------------
   autoconf/2.69	      libedit/3.1-20210216	  ncurses/6.2	       (L)    py-beniget/0.4.1		    readline/8.1
   automake/1.16.5	      libevent/2.1.12		  netlib-lapack/3.10.1	      py-cython/0.29.30 	    sqlite/3.38.5
   berkeley-db/18.1.40	      libfabric/1.14.1	   (L)	  ninja/1.10.2		      py-gast/0.5.3		    tar/1.34
   bzip2/1.0.8		      libffi/3.4.2		  numactl/2.0.14	      py-numpy/1.22.4-openblas	    util-linux-uuid/2.37.4
   cmake/3.23.1 	      libiconv/1.16	   (L)	  openblas/0.3.20	      py-pip/21.3.1		    util-macros/1.19.3
   diffutils/3.8	      libmd/1.0.4		  openmpi/4.1.3 	      py-ply/3.11		    xz/5.2.5		   (L)
   expat/2.4.8		      libpciaccess/0.16    (L)	  openssh/9.0p1 	      py-pybind11/2.8.1 	    yaksa/0.2		   (L)
   findutils/4.9.0	      libsigsegv/2.13		  openssl/1.1.1o	      py-pythran/0.11.0-openblas    zlib/1.2.12 	   (L)
   gdbm/1.19		      libtool/2.4.7		  perl/5.34.1		      py-scipy/1.8.1-openblas	    zstd/1.5.2
   gettext/0.21 	      libxml2/2.9.13	   (L)	  pigz/2.7		      py-setuptools/59.4.0
   hwloc/2.7.1	       (L)    m4/1.4.19 		  pkgconf/1.8.0 	      py-wheel/0.37.0
   libbsd/0.11.5	      mpich/4.0.2	   (L)	  pmix/4.1.2		      python/3.9.12

---------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------------
   gcc/8.4.0 (L)

  Where:
   D:  Default Module
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
$ module load openblas netlib-scalapack/2.2.0-openblas
$ module list

Currently Loaded Modules:
  1) gcc/8.4.0		 4) xz/5.2.5	     7) ncurses/6.2	  10) yaksa/0.2        13) netlib-scalapack/2.2.0-openblas
  2) libpciaccess/0.16	 5) zlib/1.2.12      8) hwloc/2.7.1	  11) mpich/4.0.2
  3) libiconv/1.16	 6) libxml2/2.9.13   9) libfabric/1.14.1  12) openblas/0.3.20

At this point we can showcase the improved consistency that a hierarchical layout provides over a non-hierarchical one:

$ export LMOD_AUTO_SWAP=yes
$ module load openmpi

Lmod is automatically replacing "mpich/4.0.2" with "openmpi/4.1.3".


Due to MODULEPATH changes, the following have been reloaded:
  1) netlib-scalapack/2.2.0-openblas

Lmod took care of swapping the MPI provider for us, and it also substituted the netlib-scalapack module to conform to the change in the MPI. In this way we can’t accidentally pull-in two different MPI providers at the same time or load a module file for a package linked to openmpi when mpich is also loaded. Consistency for compilers and MPI is ensured by the tool.

Add LAPACK to the hierarchy

The hierarchy just shown is already a great improvement over non-hierarchical layouts, but it still has an asymmetry: LAPACK providers cover the same semantic role as MPI providers, but yet they are not part of the hierarchy.

To be more practical, this means that although we have gained an improved consistency in our environment when it comes to MPI, we still have the same problems as we had before for LAPACK implementations:

$ module list

Currently Loaded Modules:
  1) gcc/8.4.0		 4) libiconv/1.16   7) libxml2/2.9.13  10) numactl/2.0.14	 13) openssh/9.0p1    16) openmpi/4.1.3
  2) openblas/0.3.20	 5) xz/5.2.5	    8) ncurses/6.2     11) libedit/3.1-20210216  14) libevent/2.1.12  17) netlib-scalapack/2.2.0-openblas
  3) libpciaccess/0.16	 6) zlib/1.2.12     9) hwloc/2.7.1     12) openssl/1.1.1o	 15) pmix/4.1.2



$ module avail

-------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/openmpi/4.1.3-c4md4py/gcc/8.4.0 ---------------------------------
   netlib-scalapack/2.2.0-netlib-lapack    netlib-scalapack/2.2.0-openblas (L,D)

------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/gcc/8.4.0 --------------------------------------------
   autoconf/2.69	      libedit/3.1-20210216 (L)	  ncurses/6.2	       (L)    py-beniget/0.4.1		    readline/8.1
   automake/1.16.5	      libevent/2.1.12	   (L)	  netlib-lapack/3.10.1	      py-cython/0.29.30 	    sqlite/3.38.5
   berkeley-db/18.1.40	      libfabric/1.14.1		  ninja/1.10.2		      py-gast/0.5.3		    tar/1.34
   bzip2/1.0.8		      libffi/3.4.2		  numactl/2.0.14       (L)    py-numpy/1.22.4-openblas	    util-linux-uuid/2.37.4
   cmake/3.23.1 	      libiconv/1.16	   (L)	  openblas/0.3.20      (L)    py-pip/21.3.1		    util-macros/1.19.3
   diffutils/3.8	      libmd/1.0.4		  openmpi/4.1.3        (L)    py-ply/3.11		    xz/5.2.5		   (L)
   expat/2.4.8		      libpciaccess/0.16    (L)	  openssh/9.0p1        (L)    py-pybind11/2.8.1 	    yaksa/0.2
   findutils/4.9.0	      libsigsegv/2.13		  openssl/1.1.1o       (L)    py-pythran/0.11.0-openblas    zlib/1.2.12 	   (L)
   gdbm/1.19		      libtool/2.4.7		  perl/5.34.1		      py-scipy/1.8.1-openblas	    zstd/1.5.2
   gettext/0.21 	      libxml2/2.9.13	   (L)	  pigz/2.7		      py-setuptools/59.4.0
   hwloc/2.7.1	       (L)    m4/1.4.19 		  pkgconf/1.8.0 	      py-wheel/0.37.0
   libbsd/0.11.5	      mpich/4.0.2		  pmix/4.1.2	       (L)    python/3.9.12

---------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------------
   gcc/8.4.0 (L)

  Where:
   D:  Default Module
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


$ module load netlib-scalapack/2.2.0-netlib-lapack

The following have been reloaded with a version change:
  1) netlib-scalapack/2.2.0-openblas => netlib-scalapack/2.2.0-netlib-lapack

$ module list

Currently Loaded Modules:
  1) gcc/8.4.0		 4) libiconv/1.16   7) libxml2/2.9.13  10) numactl/2.0.14	 13) openssh/9.0p1    16) openmpi/4.1.3
  2) openblas/0.3.20	 5) xz/5.2.5	    8) ncurses/6.2     11) libedit/3.1-20210216  14) libevent/2.1.12  17) netlib-lapack/3.10.1
  3) libpciaccess/0.16	 6) zlib/1.2.12     9) hwloc/2.7.1     12) openssl/1.1.1o	 15) pmix/4.1.2       18) netlib-scalapack/2.2.0-netlib-lapack

Hierarchies that are deeper than Core/Compiler/MPI are probably still considered “unusual” or “impractical” at many sites, mainly because module files are written manually and keeping track of the combinations among multiple providers quickly becomes quite involved.

For instance, having both MPI and LAPACK in the hierarchy means we must classify software into one of four categories:

  1. Software that doesn’t depend on MPI or LAPACK
  2. Software that depends only on MPI
  3. Software that depends only on LAPACK
  4. Software that depends on both

to decide when to show it to the user. The situation becomes more involved as the number of virtual dependencies in the hierarchy increases.

We can take advantage of the DAG that Spack maintains for the installed software and solve this combinatorial problem in a clean and automated way. In some sense Spack’s ability to manage this combinatorial complexity makes deeper hierarchies feasible.

Coming back to our example, let’s add lapack to the hierarchy and remove the remaining suffix projection for lapack:

modules:
  default:
    enable::
    - lmod
    lmod:
      core_compilers:
      - 'gcc@7.5.0'
      hierarchy:
      - mpi
      - lapack
      hash_length: 0
      include:
      - gcc
      exclude:
      - '%gcc@7.5.0'
      all:
        filter:
          exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"
        environment:
          set:
            '{name}_ROOT': '{prefix}'
      openmpi:
        environment:
          set:
            SLURM_MPI_TYPE: pmi2
            OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
      projections:
        all:          '{name}/{version}'

After module files have been regenerated as usual:

$ module purge
$ spack module lmod refresh --delete-tree -y
==> Regenerating lmod module files

we can see that now we have additional components in the hierarchy:

$ module load gcc

Lmod is automatically replacing "openblas/0.3.20" with "netlib-lapack/3.10.1".


Inactive Modules:
  1) netlib-scalapack/2.2.0-netlib-lapack     2) openblas

$ module load openblas

Lmod is automatically replacing "netlib-lapack/3.10.1" with "openblas/0.3.20".


Activating Modules:
  1) openblas/0.3.20

$ module avail

-------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/openmpi/4.1.3-c4md4py/openblas/0.3.20-ew72qgy/gcc/8.4.0 ---------------------
   netlib-scalapack/2.2.0

------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/openblas/0.3.20-ew72qgy/gcc/8.4.0 --------------------------------
   py-numpy/1.22.4    py-pythran/0.11.0    py-scipy/1.8.1

------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/gcc/8.4.0 --------------------------------------------
   autoconf/2.69	  gettext/0.21		      libpciaccess/0.16    (L)	  numactl/2.0.14  (L)	 py-beniget/0.4.1	 readline/8.1
   automake/1.16.5	  hwloc/2.7.1	       (L)    libsigsegv/2.13		  openblas/0.3.20 (L)	 py-cython/0.29.30	 sqlite/3.38.5
   berkeley-db/18.1.40	  libbsd/0.11.5 	      libtool/2.4.7		  openmpi/4.1.3   (L)	 py-gast/0.5.3		 tar/1.34
   bzip2/1.0.8		  libedit/3.1-20210216 (L)    libxml2/2.9.13	   (L)	  openssh/9.0p1   (L)	 py-pip/21.3.1		 util-linux-uuid/2.37.4
   cmake/3.23.1 	  libevent/2.1.12      (L)    m4/1.4.19 		  openssl/1.1.1o  (L)	 py-ply/3.11		 util-macros/1.19.3
   diffutils/3.8	  libfabric/1.14.1	      mpich/4.0.2		  perl/5.34.1		 py-pybind11/2.8.1	 xz/5.2.5		(L)
   expat/2.4.8		  libffi/3.4.2		      ncurses/6.2	   (L)	  pigz/2.7		 py-setuptools/59.4.0	 yaksa/0.2
   findutils/4.9.0	  libiconv/1.16        (L)    netlib-lapack/3.10.1	  pkgconf/1.8.0 	 py-wheel/0.37.0	 zlib/1.2.12		(L)
   gdbm/1.19		  libmd/1.0.4		      ninja/1.10.2		  pmix/4.1.2	  (L)	 python/3.9.12		 zstd/1.5.2

---------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------------
   gcc/8.4.0 (L)

  Where:
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".


$ module load openmpi
$ module avail

-------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/openmpi/4.1.3-c4md4py/openblas/0.3.20-ew72qgy/gcc/8.4.0 ---------------------
   netlib-scalapack/2.2.0

------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/openblas/0.3.20-ew72qgy/gcc/8.4.0 --------------------------------
   py-numpy/1.22.4    py-pythran/0.11.0    py-scipy/1.8.1

------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/gcc/8.4.0 --------------------------------------------
   autoconf/2.69	  gettext/0.21		      libpciaccess/0.16    (L)	  numactl/2.0.14  (L)	 py-beniget/0.4.1	 readline/8.1
   automake/1.16.5	  hwloc/2.7.1	       (L)    libsigsegv/2.13		  openblas/0.3.20 (L)	 py-cython/0.29.30	 sqlite/3.38.5
   berkeley-db/18.1.40	  libbsd/0.11.5 	      libtool/2.4.7		  openmpi/4.1.3   (L)	 py-gast/0.5.3		 tar/1.34
   bzip2/1.0.8		  libedit/3.1-20210216 (L)    libxml2/2.9.13	   (L)	  openssh/9.0p1   (L)	 py-pip/21.3.1		 util-linux-uuid/2.37.4
   cmake/3.23.1 	  libevent/2.1.12      (L)    m4/1.4.19 		  openssl/1.1.1o  (L)	 py-ply/3.11		 util-macros/1.19.3
   diffutils/3.8	  libfabric/1.14.1	      mpich/4.0.2		  perl/5.34.1		 py-pybind11/2.8.1	 xz/5.2.5		(L)
   expat/2.4.8		  libffi/3.4.2		      ncurses/6.2	   (L)	  pigz/2.7		 py-setuptools/59.4.0	 yaksa/0.2
   findutils/4.9.0	  libiconv/1.16        (L)    netlib-lapack/3.10.1	  pkgconf/1.8.0 	 py-wheel/0.37.0	 zlib/1.2.12		(L)
   gdbm/1.19		  libmd/1.0.4		      ninja/1.10.2		  pmix/4.1.2	  (L)	 python/3.9.12		 zstd/1.5.2

---------------------------------------------- /home/spack/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core ----------------------------------------------
   gcc/8.4.0 (L)

  Where:
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

Both MPI and LAPACK providers will now benefit from the same safety features:

$ module load py-numpy netlib-scalapack

Activating Modules:
  1) netlib-scalapack/2.2.0

$ module load mpich

Lmod is automatically replacing "openmpi/4.1.3" with "mpich/4.0.2".


Due to MODULEPATH changes, the following have been reloaded:
  1) netlib-scalapack/2.2.0

$ module load netlib-lapack

Lmod is automatically replacing "openblas/0.3.20" with "netlib-lapack/3.10.1".


Inactive Modules:
  1) bzip2/1.0.8     3) gdbm/1.19	 5) libbsd/0.11.5     7) libmd/1.0.4	 9) py-numpy		    11) python/3.9.12	 13) sqlite/3.38.5    15) util-linux-uuid/2.37.4
  2) expat/2.4.8     4) gettext/0.21	 6) libffi/3.4.2      8) pigz/2.7	10) py-setuptools/59.4.0    12) readline/8.1	 14) tar/1.34	      16) zstd/1.5.2

Due to MODULEPATH changes, the following have been reloaded:
  1) netlib-scalapack/2.2.0

Because we only compiled py-numpy with openblas the module is made inactive when we switch the LAPACK provider. The user environment is now consistent by design!

Working with Templates

As briefly mentioned in the introduction, Spack uses Jinja2 to generate each individual module file. This means that you have all of its flexibility and power when it comes to customizing what gets generated!

Module file templates

The templates that Spack uses to generate module files are stored in the share/spack/templates/module directory within the Spack prefix, and they all share the same common structure. Usually, they start with a header that identifies the type of module being generated. In the case of hierarchical module files it’s:

-- -*- lua -*-
-- Module file created by spack (https://github.com/spack/spack) on {{ timestamp }}
--
-- {{ spec.short_spec }}
--

The statements within double curly brackets {{ ... }} denote expressions that will be evaluated and substituted at module generation time. The rest of the file is then divided into blocks that can be overridden or extended by users, if need be. Control structures , delimited by {% ... %}, are also permitted in the template language:

{% block environment %}
{% for command_name, cmd in environment_modifications %}
{% if command_name == 'PrependPath' %}
prepend_path("{{ cmd.name }}", "{{ cmd.value }}", "{{ cmd.separator }}")
{% elif command_name == 'AppendPath' %}
append_path("{{ cmd.name }}", "{{ cmd.value }}", "{{ cmd.separator }}")
{% elif command_name == 'RemovePath' %}
remove_path("{{ cmd.name }}", "{{ cmd.value }}", "{{ cmd.separator }}")
{% elif command_name == 'SetEnv' %}
setenv("{{ cmd.name }}", "{{ cmd.value }}")
{% elif command_name == 'UnsetEnv' %}
unsetenv("{{ cmd.name }}")
{% endif %}
{% endfor %}
{% endblock %}

The locations where Spack looks for templates are specified in config.yaml:

    projections:
      all: "${ARCHITECTURE}/${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH}"
    # install_tree can include an optional padded length (int or boolean)
    # default is False (do not pad)

and can be extended by users to employ custom templates, as we’ll see next.

Extend the default templates

Let’s assume one of our software is protected by group membership: allowed users belong to the same linux group, and access is granted at group level. Wouldn’t it be nice if people that are not yet entitled to use it could receive a helpful message at module load time that tells them who to contact in your organization to be inserted in the group?

To automate the generation of module files with such site-specific behavior we’ll start by extending the list of locations where Spack looks for module files. Let’s create the file ${SPACK_ROOT}/etc/spack/config.yaml with the content:

config:
  template_dirs:
    - $HOME/.spack/templates

This tells Spack to also search another location when looking for template files. Next, we need to create our custom template extension in the folder listed above:

{% extends "modules/modulefile.lua" %}
{% block footer %}
-- Access is granted only to specific groups
if not isDir("{{ spec.prefix }}") then
    LmodError (
        "You don't have the necessary rights to run \"{{ spec.name }}\".\n\n",
        "\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n"
    )
end
{% endblock %}

Let’s name this file group-restricted.lua. The line:

{% extends "modules/modulefile.lua" %}

tells Jinja2 that we are reusing the standard template for hierarchical module files. The section:

{% block footer %}
-- Access is granted only to specific groups
if not isDir("{{ spec.prefix }}") then
    LmodError (
        "You don't have the necessary rights to run \"{{ spec.name }}\".\n\n",
        "\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n"
    )
end
{% endblock %}

overrides the footer block. Finally, we need to add a couple of lines in modules.yaml to tell Spack which specs need to use the new custom template. For the sake of illustration let’s assume it’s netlib-scalapack:

modules:
  enable::
    - lmod
  lmod:
    core_compilers:
      - 'gcc@7.5.0'
    hierarchy:
      - mpi
      - lapack
    hash_length: 0
    include:
      - gcc
    exclude:
      - '%gcc@7.5.0'
      - readline
    all:
      filter:
        exclude_env_vars:
          - "C_INCLUDE_PATH"
          - "CPLUS_INCLUDE_PATH"
          - "LIBRARY_PATH"
      environment:
        set:
          '{name}_ROOT': '{prefix}'
    openmpi:
      environment:
        set:
          SLURM_MPI_TYPE: pmi2
          OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
    netlib-scalapack:
      template: 'group-restricted.lua'

If we regenerate the module files one last time:

$ spack module lmod refresh -y netlib-scalapack
==> Regenerating lmod module files

we’ll find the following at the end of each netlib-scalapack module file:

-- Access is granted only to specific groups
if not isDir("/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-8.4.0/netlib-scalapack-2.0.2-2p75lzqjbsnev7d2j2osgpkz7ib33oca") then
    LmodError (
        "You don't have the necessary rights to run \"netlib-scalapack\".\n\n",
        "\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n"
    )
end

and every user that doesn’t have access to the software will now be redirected to the right e-mail address where to ask for it!

Restore settings for future sections

For future sections of the tutorial, we will not use the gcc@8.4.0 compiler. Since it is currently the default compiler (our current defualt is the most recent version of gcc available), we will remove it now.

$ spack compiler rm gcc@8.4.0

This will ensure the rest of the tutorial goes smoothly for you.