.. Copyright Spack Project Developers. See COPYRIGHT file for details.

   SPDX-License-Identifier: (Apache-2.0 OR MIT)

.. include:: common/setup.rst

.. _modules-tutorial:

=====================
Module Files Tutorial
=====================

This tutorial illustrates how Spack can be used to generate module files for the software that has been installed.
Both hierarchical and non-hierarchical deployments will be discussed in detail and we will show how to customize the content and naming of each module file.

At the end of the tutorial readers should have a clear understanding of:

  * What module files are and how they are used on HPC clusters
  * How Spack generates module files for the software it installs
  * Which Spack commands can be used to manage module files
  * How module files generated by Spack can be customized

and be confident that Spack can deal with all of the common use cases that occur when maintaining software installations on HPC clusters.

.. _module_file_tutorial_prerequisites:

----------------------
Setup for the Tutorial
----------------------

To prepare for this tutorial we are going to install a small but representative set of software that includes different configurations of the same packages and some `external packages <https://spack.readthedocs.io/en/latest/build_settings.html#sec-external-packages>`_.
To keep the installations manageable, let's start by uninstalling everything from earlier in the tutorial:

.. code-block:: console

  $ spack uninstall -ay

and by enabling ``tcl`` module files, which are disabled by default since Spack v0.20:

.. code-block:: console

  $ spack config add "modules:default:enable:[tcl]"


^^^^^^^^^^^^^^^^^^^
Build a module tool
^^^^^^^^^^^^^^^^^^^

The first thing that we need is the module tool itself.
In this tutorial, we will use ``lmod`` because it can work with both hierarchical and non-hierarchical layouts.

.. code-block:: console

  $ spack install lmod

Once the module tool is installed we need to have it available in the current shell.
Installation directories in Spack's store are definitely not easy to remember, but they can be retrieved with the ``spack location`` command:

.. code-block:: console

  $ . $(spack location -i lmod)/lmod/lmod/init/bash

Now we can re-source the setup file, and Spack modules will be put in our module path.

.. code-block:: console

  $ . spack/share/spack/setup-env.sh

.. FIXME: this needs bootstrap support for ``lmod``

.. FIXME: check the docs here, update them if necessary
If you need to install Lmod or Environment Modules, you can refer to the documentation `here <https://spack.readthedocs.io/en/latest/getting_started.html#installenvironmentmodules>`_.


^^^^^^^^^^^^^^^^^^
Add a new compiler
^^^^^^^^^^^^^^^^^^

The second step is to build a recent compiler.
On first use, Spack scans the environment and automatically locates the compiler(s) already available on the system.
For this tutorial, however, we want to use ``gcc@12.3.0``.


.. code-block:: console

  $ spack install gcc@12.3.0


You can get this in your environment using ``spack load gcc@12.3.0``:

.. literalinclude:: outputs/modules/spack-load-gcc.out
   :language: console

Now, ``gcc`` is in your ``PATH``.
You can add it to the list of compilers with ``spack compiler add``:

.. literalinclude:: outputs/modules/add-compiler.out
   :language: console

To check which compilers are available you can use ``spack compiler list``:

.. literalinclude:: outputs/modules/list-compiler.out
   :language: console

Finally, when you have confirmed ``gcc@12.3.0`` is properly registered, clean the environment with ``spack unload``:

.. code-block:: console

   $ spack unload --all


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Build the software that will be used in the tutorial
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Finally, we will use Spack to install the packages used in the examples:

.. code-block:: console

   $ spack install netlib-scalapack ^openmpi ^openblas
   $ spack install netlib-scalapack ^mpich ^openblas
   $ spack install netlib-scalapack ^openmpi ^netlib-lapack
   $ spack install netlib-scalapack ^mpich ^netlib-lapack
   $ spack install py-scipy ^openblas

.. _module_file_tutorial_overview:

----------------------
What are Module Files?
----------------------

Module files are an easy way to modify your environment in a controlled manner during a shell session.
In general, they contain the information needed to run an application or use a library.
The ``module`` command is used to interpret and execute module files.
For example, ``module show`` tells you what a module will do when loaded:

.. literalinclude:: outputs/modules/what-are-modules-1.out
   :language: console


``module load`` will execute all of the changes shown above:

.. literalinclude:: outputs/modules/what-are-modules-2.out
   :language: console


and to undo the modifications, you can use ``module unload``:

.. literalinclude:: outputs/modules/what-are-modules-3.out
   :language: console


^^^^^^^^^^^^^^
Module Systems
^^^^^^^^^^^^^^

There are two main module systems used in HPC, both installable by Spack.
In this tutorial we will be working with ``lmod`` and be showing examples with both Tcl and Lua.


"""""""""""""""""""
Environment Modules
"""""""""""""""""""

This is the original modules tool.
It can be installed with Spack using the following command:

.. code-block:: console

   $ spack install environment-modules


It was first coded in C in the early 1990s and was later rewritten entirely in Tcl.
Long stagnant, the project has been revived in the past few years by Xavier Delaruelle at CEA, and it is now very actively developed.
For further details we refer to its `documentation <https://modules.readthedocs.io/>`_.

""""
Lmod
""""
Lmod is a module system written in Lua, originally created  at the "Texas Advanced Computing Center" (TACC) by Robert McLay.
You can get it with:

.. code-block:: console

   $ spack install lmod

as shown in the :ref:`module_file_tutorial_prerequisites` section.
It is a drop-in replacement for Environment Modules, and it works with *both* Tcl and Lua module files.
It is fully compatible with Environment Modules, but it also has many distinguishing features of its own.
The main one is the `module hierarchy <http://lmod.readthedocs.io/en/latest/010_user.html#module-hierarchy>`_, which simplifies the module UI by only showing modules built with the currently loaded compiler and/or MPI.
There are also some unique `safety features <http://lmod.readthedocs.io/en/latest/010_user.html#safety-features>`_.


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
How does Spack generate module files?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Before we dive into the hands-on sections, it's worth explaining how module files are generated by Spack.
The following diagram provides a high-level view of the process:


.. image:: module_file_generation.*

Modules in Spack are generated using configuration files (``config.yaml`` and ``modules.yaml``), information from Spack's package recipes, and Jinja2 templates.
Spack comes with `Jinja2 <http://jinja.pocoo.org/docs/2.9/>`_, an external template engine, so you do not need to install it yourself.

^^^^^^^^^^^^^^^^^^^^^^^^^
Modules vs ``spack load``
^^^^^^^^^^^^^^^^^^^^^^^^^

You may have noticed that we used ``spack load`` in the
:ref:`module_file_tutorial_prerequisites` section above. This is a
built-in mechanism of Spack's -- it's designed so that users on a cluster or a laptop can quickly get a package into their path, and it understands Spack's spec syntax.
It does *not* require modules, as Spack needs to work regardless of whether modules are set up on the system.

As you might expect, you can see what is loaded via ``spack load`` using ``spack find``:

.. literalinclude:: outputs/modules/show-loaded.out
   :language: console

Because Spack is designed to be run on HPC systems, it also generates a module file for every installed package.
This allows users unfamiliar with Spack's interface to see things through the module system they're used to.
To see this, try:

.. literalinclude:: outputs/modules/module-avail-1.out
   :language: console

You can ``module load`` any of these.
By default, Spack generates modules named by ``package-version-compiler-version-hash``, which is a bit hard to read.
We'll show you how to customize this in the following sections.


.. _module_file_tutorial_non_hierarchical:

-----------------------------
Non-hierarchical Module Files
-----------------------------

If you have arrived at this point, you should have an environment that looks similar to:

.. literalinclude:: outputs/modules/module-avail-2.out
   :language: console

The non-hierarchical module files that have been generated so far follow Spack's `default rules for module generation <https://spack.readthedocs.io/en/latest/module_file_support.html#modules-yaml>`_.
Taking a look at the ``gcc`` module you'll see, for example:

.. literalinclude:: outputs/modules/module-show-1.out
   :language: console

As expected, a few environment variables representing paths will be modified by the module file according to the default prefix inspection rules.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Filter unwanted modifications to the environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Now consider the case that your site has decided that ``CC``, ``CXX``, ``FC`` and ``F77`` modifications should not be present in module files.
What you can do to abide by the rules is to create a configuration file ``${SPACK_ROOT}/etc/spack/modules.yaml`` with the following content:

.. code-block:: yaml

  modules:
    default:
      tcl:
        all:
          filter:
            exclude_env_vars:
            - "CC"
            - "CXX"
            - "FC"
            - "F77"

This can be done either by editing the configuration manually or directly from the command line:

.. code-block:: console

   $ spack config add "modules:default:tcl:all:filter:exclude_env_vars:['CC', 'CXX', 'F77', 'FC']"

Next you should regenerate all the module files:

.. literalinclude:: outputs/modules/tcl-refresh-1.out
   :language: console

If you take a look now at the module for ``gcc`` you'll see that the unwanted paths have disappeared:

.. literalinclude:: outputs/modules/module-show-2.out
   :language: console

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Prevent some module files from being generated
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Another common request at many sites is to avoid exposing software that is only needed as an intermediate step when building a newer stack.
Let's try to prevent the generation of module files for anything that is compiled with ``gcc@11`` (the OS provided compiler).

To do this you should add the ``exclude`` keyword to ``${SPACK_ROOT}/etc/spack/modules.yaml``:

.. code-block:: yaml
  :emphasize-lines: 4,5

  modules:
    default:
      tcl:
        exclude:
        -  '%gcc@11'
        all:
          filter:
            exclude_env_vars:
            - "CC"
            - "CXX"
            - "FC"
            - "F77"


and regenerate the module files.
This time we'll pass the option ``--delete-tree`` so that Spack will delete the existing module tree and regenerate a new one, instead of overwriting the files in the existing directory.

.. literalinclude:: outputs/modules/tcl-refresh-2.out
   :language: console

.. literalinclude:: outputs/modules/module-avail-3.out
   :language: console


if you look closely, you'll see, though, that we went too far in excluding modules: the module for ``gcc@12.3.0`` disappeared as it was bootstrapped with ``gcc@11``.
To specify exceptions to the ``exclude`` rules you can use ``include``:

.. code-block:: yaml
  :emphasize-lines: 4,5

  modules:
    default:
      tcl:
        include:
        -  gcc
        exclude:
        -  '%gcc@11'
        all:
          filter:
            exclude_env_vars:
            - "CC"
            - "CXX"
            - "FC"
            - "F77"

``include`` rules always have precedence over ``exclude`` rules. If you regenerate the modules again:

.. literalinclude:: outputs/modules/tcl-refresh-3.out
   :language: console

you'll see that now the module for ``gcc@12.3.0`` has reappeared:

.. literalinclude:: outputs/modules/module-avail-4.out
   :language: console

An additional feature that you can leverage to unclutter the environment is to skip the generation of module files for implicitly installed packages.
In this case you only need to add the following line:

.. code-block:: yaml
  :emphasize-lines: 4

  modules:
    default:
      tcl:
        exclude_implicits: true
        include:
        -  gcc
        exclude:
        -  '%gcc@11'
        all:
          filter:
            exclude_env_vars:
            - "CC"
            - "CXX"
            - "FC"
            - "F77"

to ``modules.yaml`` and regenerate the module file tree as above.

^^^^^^^^^^^^^^^^^^^^^^^^^
Change module file naming
^^^^^^^^^^^^^^^^^^^^^^^^^

The next step in making module files more user-friendly is to improve their naming scheme.
To reduce the length of the hash or remove it altogether you can use the ``hash_length`` keyword in the configuration file:

.. code-block:: yaml
  :emphasize-lines: 4

  modules:
    default:
      tcl:
        hash_length: 0
        include:
        -  gcc
        exclude:
        -  '%gcc@11'
        all:
          filter:
            exclude_env_vars:
            - "CC"
            - "CXX"
            - "FC"
            - "F77"

If you try to regenerate the module files now you will get an error:

.. literalinclude:: outputs/modules/tcl-refresh-4.out
   :language: console

.. note::
   We try to check for errors up front!

   In Spack we check for errors upfront whenever possible, so don't worry
   about your module files: as a name clash was detected nothing has been
   changed on disk.

The problem here is that without the hashes the four different flavors of ``netlib-scalapack`` map to the same module file name.
We can change how the names are formatted to differentiate them:

.. code-block:: yaml
  :emphasize-lines: 10-11,18-21

  modules:
    default:
      tcl:
        hash_length: 0
        include:
        -  gcc
        exclude:
        -  '%gcc@11'
        all:
          conflict:
          - '{name}'
          filter:
            exclude_env_vars:
            - "CC"
            - "CXX"
            - "FC"
            - "F77"
        projections:
          all:               '{name}/{version}-{compiler.name}-{compiler.version}'
          netlib-scalapack:  '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
          ^python^lapack:    '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'

As you can see, it is possible to specify rules that apply only to a restricted set of packages using `anonymous specs <https://spack.readthedocs.io/en/latest/module_file_support.html#anonymous-specs>`_ like ``^python^lapack``.
Here we declare a conflict between any two modules with the same name, so they cannot be loaded together.
We also format the names of modules according to compiler, compiler version, and MPI provider name using the `spec format syntax <https://spack.readthedocs.io/en/latest/spack.html?highlight=spec%20format#spack.spec.Spec.format>`_.
This allows us to match specs by their dependencies, and format them based on their DAGs.

.. literalinclude:: outputs/modules/tcl-refresh-5.out
   :language: console

.. literalinclude:: outputs/modules/module-avail-5.out
   :language: console

.. note::
   The ``conflict`` directive is Tcl-specific and can't be used in the
   ``lmod`` section of the configuration file.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Add custom environment modifications
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

At many sites it is customary to set an environment variable in a package's module file that points to the folder in which the package is installed.
You can achieve this with Spack by adding an ``environment`` directive to the configuration file:

.. code-block:: yaml
  :emphasize-lines: 19-21

  modules:
    default:
      tcl:
        hash_length: 0
        naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
        include:
        -  gcc
        exclude:
        -  '%gcc@11'
        all:
          conflict:
          - '{name}'
          filter:
            exclude_env_vars:
            - "CC"
            - "CXX"
            - "FC"
            - "F77"
          environment:
            set:
              '{name}_ROOT': '{prefix}'
        projections:
          all:               '{name}/{version}-{compiler.name}-{compiler.version}'
          netlib-scalapack:  '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
          ^python^lapack:    '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'


Under the hood Spack uses the :meth:`~spack.spec.Spec.format` API to substitute tokens in either environment variable names or values.
There are two caveats though:

- The set of allowed tokens in variable names is restricted to
  ``name``, ``version``, ``compiler``, ``compiler.name``,
  ``compiler.version``, ``architecture``
- Any token expanded in a variable name is made uppercase, but other than that
  case sensitivity is preserved

Regenerating the module files results in something like:

.. literalinclude:: outputs/modules/tcl-refresh-6.out
   :language: console

.. literalinclude:: outputs/modules/module-show-3.out
   :language: console

As you can see, the ``gcc`` module has the environment variable ``GCC_ROOT`` set.

Sometimes it's also useful to apply environment modifications selectively and target only certain packages.
You can for instance apply modifications to the ``openmpi`` module as follows:

.. code-block:: yaml
  :emphasize-lines: 22-26

  modules:
    default:
      tcl:
        hash_length: 0
        naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
        include:
        - gcc
        exclude:
        - '%gcc@11'
        all:
          conflict:
          - '{name}'
          filter:
            exclude_env_vars:
            - "CC"
            - "CXX"
            - "FC"
            - "F77"
          environment:
            set:
              '{name}_ROOT': '{prefix}'
        openmpi:
          environment:
            set:
              SLURM_MPI_TYPE: pmi2
              OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
        projections:
          all:               '{name}/{version}-{compiler.name}-{compiler.version}'
          netlib-scalapack:  '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
          ^python^lapack:    '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'

This time we will be more selective and regenerate only the ``openmpi`` module file:

.. literalinclude:: outputs/modules/tcl-refresh-7.out
   :language: console

.. literalinclude:: outputs/modules/module-show-4.out
   :language: console


.. FIXME: remove this?

^^^^^^^^^^^^^^^^^^^^^
Autoload dependencies
^^^^^^^^^^^^^^^^^^^^^

Spack can also generate module files that contain code to load the dependencies automatically.
You can, for instance, generate python modules that load their dependencies by adding the ``autoload`` directive and assigning it the value ``direct``:

.. code-block:: yaml
  :emphasize-lines: 4,32,33

  modules:
    default:
      tcl:
        verbose: true
        hash_length: 0
        naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
        include:
        - gcc
        exclude:
        - '%gcc@11'
        all:
          conflict:
          - '{name}'
          filter:
            exclude_env_vars:
            - "CC"
            - "CXX"
            - "FC"
            - "F77"
          environment:
            set:
              '{name}_ROOT': '{prefix}'
        openmpi:
          environment:
            set:
              SLURM_MPI_TYPE: pmi2
              OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
        projections:
          all:               '{name}/{version}-{compiler.name}-{compiler.version}'
          netlib-scalapack:  '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}'
          ^python^lapack:    '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}'
        ^python:
          autoload:  direct

and regenerating the module files for every package that depends on ``python``:

.. literalinclude:: outputs/modules/tcl-refresh-8.out
   :language: console

and will contain code to autoload all the dependencies:

.. literalinclude:: outputs/modules/load-direct.out
   :language: console

In case messages are unwanted during the autoload procedure, it will be sufficient to omit the line setting ``verbose: true`` in the configuration file above.

-------------------------
Hierarchical Module Files
-------------------------

So far, we have worked with non-hierarchical module files, i.e., with module files that are all generated in the same root directory and don't attempt to dynamically modify the ``MODULEPATH``.
This results in a flat module structure where all the software is visible at the same time:

.. literalinclude:: outputs/modules/lmod-intro-avail.out
   :language: console

This layout is quite simple to deploy, but you can see from the above snippet that nothing prevents users from loading incompatible sets of modules:

.. literalinclude:: outputs/modules/lmod-intro-conflict.out
   :language: console

Even if ``conflicts`` directives are carefully placed in module files, they:

  - won't enforce a consistent environment, but will just report an error
  - need constant updates, for instance as soon as a new compiler or MPI library is installed

`Hierarchical module files <http://lmod.readthedocs.io/en/latest/080_hierarchy.html>`_ try to overcome these shortcomings by showing at start-up only a restricted view of what is available on the system: more specifically, only the software that has been installed with OS provided compilers.
Among this software, there will be other -- usually more recent -- compilers that, once loaded, will prepend new directories to ``MODULEPATH``, unlocking all the software that was compiled with them.
This "unlocking" idea can then be extended arbitrarily to virtual dependencies, as we'll see in the following section.

^^^^^^^^^^^^^^^^^
Core/Compiler/MPI
^^^^^^^^^^^^^^^^^

The most widely used hierarchy is the so called ``Core/Compiler/MPI`` where, on top of the compilers, different MPI libraries also unlock software linked to them.
There are just a few steps needed to adapt the ``modules.yaml`` file we used previously:

  #. enable the ``lmod`` file generator
  #. change the ``tcl`` tag to ``lmod``
  #. remove the ``tcl`` specific ``conflict`` directive
  #. declare which compilers are considered ``core_compilers``
  #. remove the ``mpi`` related suffixes in projections (as they will be substituted by hierarchies)

After these modifications your configuration file should look like:

.. code-block:: yaml
  :emphasize-lines: 3-9,29-31

  modules:
    default:
      enable::
        - lmod
      lmod:
        core_compilers:
        - 'gcc@11'
        hierarchy:
        - mpi
        hash_length: 0
        include:
        - gcc
        exclude:
        - '%gcc@11'
        all:
          filter:
            exclude_env_vars:
            - "C_INCLUDE_PATH"
            - "CPLUS_INCLUDE_PATH"
            - "LIBRARY_PATH"
          environment:
            set:
              '{name}_ROOT': '{prefix}'
        openmpi:
          environment:
            set:
              SLURM_MPI_TYPE: pmi2
              OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
        projections:
          all:          '{name}/{version}'
          ^lapack:      '{name}/{version}-{^lapack.name}'


.. note::
  Double colon in configuration files
    The double colon after ``enable`` is intentional, and it serves the
    purpose of overriding the default list of enabled generators so
    that only ``lmod`` will be active (see `Overriding entire sections <https://spack.readthedocs.io/en/latest/configuration.html#config-overrides>`_ for more
    details).

The directive ``core_compilers`` accepts a list of compilers.
Everything built using these compilers will create a module in the ``Core`` part of the hierarchy, which is the entry point for hierarchical module files.
It is common practice to put the OS provided compilers in the list and only build common utilities and other compilers with them.

If we now regenerate the module files:

.. literalinclude:: outputs/modules/lmod-refresh-1.out
   :language: console

and update ``MODULEPATH`` to point to the ``Core``:

.. code-block:: console

  $ module purge
  $ module unuse $HOME/spack/share/spack/modules/linux-ubuntu18.04-x86_64
  $ module use $HOME/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core

asking for the available modules will return:

.. literalinclude:: outputs/modules/module-avail-6.out
   :language: console

Unsurprisingly, the only visible module is ``gcc``.
Loading that we'll unlock the ``Compiler`` part of the hierarchy:

.. literalinclude:: outputs/modules/module-avail-7.out
   :language: console

The same holds true also for the ``MPI`` part, which you can enable by loading either ``mpich`` or ``openmpi``.
Let's start by loading ``mpich``:

.. literalinclude:: outputs/modules/module-avail-8.out
   :language: console

.. literalinclude:: outputs/modules/module-load-openblas-scalapack.out
   :language: console

At this point we can showcase the improved consistency that a hierarchical layout provides over a non-hierarchical one:

.. literalinclude:: outputs/modules/module-swap-mpi.out
   :language: console

``Lmod`` took care of swapping the MPI provider for us, and it also substituted the ``netlib-scalapack`` module to conform to the change in the MPI.
In this way we can't accidentally pull-in two different MPI providers at the same time or load a module file for a package linked to ``openmpi`` when ``mpich`` is also loaded.
Consistency for compilers and MPI is ensured by the tool.


^^^^^^^^^^^^^^^^^^^^^^^^^^^
Add LAPACK to the hierarchy
^^^^^^^^^^^^^^^^^^^^^^^^^^^

The hierarchy just shown is already a great improvement over non-hierarchical layouts, but it still has an asymmetry: ``LAPACK`` providers cover the same semantic role as ``MPI`` providers, but yet they are not part of the hierarchy.

To be more practical, this means that although we have gained an improved consistency in our environment when it comes to ``MPI``, we still have the same problems as we had before for ``LAPACK`` implementations:

.. literalinclude:: outputs/modules/lapack-conflict.out
   :language: console

Hierarchies that are deeper than ``Core``/``Compiler``/``MPI`` are probably still considered "unusual" or "impractical" at many sites, mainly because module files are written manually and keeping track of the combinations among multiple providers quickly becomes quite involved.

For instance, having both ``MPI`` and ``LAPACK`` in the hierarchy means we must classify software into one of four categories:

  #. Software that doesn't depend on ``MPI`` or ``LAPACK``
  #. Software that depends only on ``MPI``
  #. Software that depends only on ``LAPACK``
  #. Software that depends on both

to decide when to show it to the user.
The situation becomes more involved as the number of virtual dependencies in the hierarchy increases.

We can take advantage of the DAG that Spack maintains for the installed software and solve this combinatorial problem in a clean and automated way.
In some sense, Spack's ability to manage this combinatorial complexity makes deeper hierarchies feasible.

Coming back to our example, let's add ``lapack`` to the hierarchy and remove the remaining suffix projection for ``lapack``:

.. code-block:: yaml
  :emphasize-lines: 10

  modules:
    default:
      enable::
      - lmod
      lmod:
        core_compilers:
        - 'gcc@11'
        hierarchy:
        - mpi
        - lapack
        hash_length: 0
        include:
        - gcc
        exclude:
        - '%gcc@11'
        all:
          filter:
            exclude_env_vars:
            - "C_INCLUDE_PATH"
            - "CPLUS_INCLUDE_PATH"
            - "LIBRARY_PATH"
          environment:
            set:
              '{name}_ROOT': '{prefix}'
        openmpi:
          environment:
            set:
              SLURM_MPI_TYPE: pmi2
              OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
        projections:
          all:          '{name}/{version}'

After module files have been regenerated as usual:

.. literalinclude:: outputs/modules/lmod-refresh-2.out
   :language: console

we can see that now we have additional components in the hierarchy:

.. literalinclude:: outputs/modules/lapack-hier.out
   :language: console

Both ``MPI`` and ``LAPACK`` providers will now benefit from the same safety features:

.. literalinclude:: outputs/modules/lapack-correct.out
   :language: console

Because we only compiled ``py-numpy`` with ``openblas`` the module is made inactive when we switch the ``LAPACK`` provider.
The user environment is now consistent by design!

----------------------
Working with Templates
----------------------

As briefly mentioned in the introduction, Spack uses `Jinja2 <http://jinja.pocoo.org/docs/2.9/>`_ to generate each individual module file.
This means that you have all of its flexibility and power when it comes to customizing what gets generated!

^^^^^^^^^^^^^^^^^^^^^
Module file templates
^^^^^^^^^^^^^^^^^^^^^

The templates that Spack uses to generate module files are stored in the ``share/spack/templates/module`` directory within the Spack prefix, and they all share the same common structure.
Usually, they start with a header that identifies the type of module being generated.
In the case of hierarchical module files it's:

.. literalinclude:: _spack_root/share/spack/templates/modules/modulefile.lua
  :language: jinja
  :lines: 1-6

The statements within double curly brackets ``{{ ... }}`` denote `expressions <http://jinja.pocoo.org/docs/2.9/templates/#expressions>`_ that will be evaluated and substituted at module generation time.
The rest of the file is then divided into `blocks <http://jinja.pocoo.org/docs/2.9/templates/#template-inheritance>`_ that can be overridden or extended by users, if need be. `Control structures <http://jinja.pocoo.org/docs/2.9/templates/#list-of-control-structures>`_ , delimited by ``{% ... %}``, are also permitted in the template language:

.. literalinclude:: _spack_root/share/spack/templates/modules/modulefile.lua
  :language: jinja
  :lines: 73-87

The locations where Spack looks for templates are specified in ``config.yaml``:

.. literalinclude:: _spack_root/etc/spack/defaults/config.yaml
  :language: yaml
  :lines: 32-35

and can be extended by users to employ custom templates, as we'll see next.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Extend the default templates
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Let's assume one of our software is protected by group membership: allowed users belong to the same Linux group, and access is granted at the group level.
Wouldn't it be nice if people that are not yet entitled to use it could receive a helpful message at module load time that tells them who to contact in your organization to be inserted in the group?

To automate the generation of module files with such site-specific behavior we'll start by extending the list of locations where Spack looks for module files.
Let's create the file ``${SPACK_ROOT}/etc/spack/config.yaml`` with the content:

.. code-block:: yaml

  config:
    template_dirs:
      - $HOME/.spack/templates

This tells Spack to also search another location when looking for template files.
Next, we need to create our custom template extension in the folder listed above:

.. code-block:: jinja

  {% extends "modules/modulefile.lua" %}
  {% block footer %}
  -- Access is granted only to specific groups
  if not isDir("{{ spec.prefix }}") then
      LmodError (
          "You don't have the necessary rights to run \"{{ spec.name }}\".\n\n",
          "\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n"
      )
  end
  {% endblock %}

Let's name this file ``group-restricted.lua``. The line:

.. code-block:: jinja

  {% extends "modules/modulefile.lua" %}

tells Jinja2 that we are reusing the standard template for hierarchical module files.
The section:

.. code-block:: jinja

  {% block footer %}
  -- Access is granted only to specific groups
  if not isDir("{{ spec.prefix }}") then
      LmodError (
          "You don't have the necessary rights to run \"{{ spec.name }}\".\n\n",
          "\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n"
      )
  end
  {% endblock %}

overrides the ``footer`` block.
Finally, we need to add a couple of lines in ``modules.yaml`` to tell Spack which specs need to use the new custom template.
For the sake of illustration let's assume it's ``netlib-scalapack``:

.. code-block:: yaml
  :emphasize-lines: 30-31

  modules:
    enable::
      - lmod
    lmod:
      core_compilers:
        - 'gcc@11'
      hierarchy:
        - mpi
        - lapack
      hash_length: 0
      include:
        - gcc
      exclude:
        - '%gcc@11'
        - readline
      all:
        filter:
          exclude_env_vars:
            - "C_INCLUDE_PATH"
            - "CPLUS_INCLUDE_PATH"
            - "LIBRARY_PATH"
        environment:
          set:
            '{name}_ROOT': '{prefix}'
      openmpi:
        environment:
          set:
            SLURM_MPI_TYPE: pmi2
            OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
      netlib-scalapack:
        template: 'group-restricted.lua'

If we regenerate the module files one last time:

.. code-block:: console

  $ spack module lmod refresh -y netlib-scalapack
  ==> Regenerating lmod module files

we'll find the following at the end of each ``netlib-scalapack`` module file:

.. code-block:: lua

  -- Access is granted only to specific groups
  if not isDir("/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-12.1.0/netlib-scalapack-2.0.2-2p75lzqjbsnev7d2j2osgpkz7ib33oca") then
      LmodError (
          "You don't have the necessary rights to run \"netlib-scalapack\".\n\n",
          "\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n"
      )
  end

and every user that doesn't have access to the software will now be redirected to the right e-mail address where they can ask for it!

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Restore settings for future sections
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

For future sections of the tutorial, we will not use the ``gcc@12.3.0`` compiler.
Since it is currently the default compiler (our current default is the most recent version of gcc available), we will remove it now.

.. code-block:: console

  $ spack compiler rm gcc@12.3.0

This will ensure the rest of the tutorial goes smoothly for you.