Lmod
Perlmutter uses Lmod as its module environment to manage software.
Lmod is a Lua-based module system that helps manage the user environment
(PATH, LD_LIBRARY_PATH, etc.) through modulefiles. Lmod is an extension of
environment-modules that supports Tcl modules along
with hierarchical MODULEPATH.
About module
module is a bash function read by $LMOD_CMD which points to the Lmod command
that reads a modulefile and evaluates each modulefile using the eval command:
elvis@perlmutter> type module
module is a function
module ()
{
eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
Command Summary
| Command | Description |
|---|---|
module list |
List active modules in the user environment |
module av [module] |
List available modules in MODULEPATH |
module spider [module] |
Query all modules in MODULEPATH and any module hierarchy |
module overview [module] |
List all modules with a count of each module |
module load [module] |
Load a modulefile in the user environment |
module unload [module] |
Remove a loaded module from the user environment |
module purge |
Remove all modules from the user environment |
module swap [module1] [module2] |
Replace module1 with module2 |
module show [module] |
Show content of commands performed by loading module |
module --raw show [module] |
Show raw content of modulefile |
module help [module] |
Show help for a given module |
module whatis [module] |
A brief description of the module, generally single line |
module savelist |
List all user collections |
module save [collection] |
Save active modules in a user collection |
module describe [collection] |
Show content of user collection |
module restore [collection] |
Load modules from a collection |
module disable [collection] |
Disable a user collection |
module --config |
Show Lmod configuration |
module use [-a] [path] |
Prepend or append path to MODULEPATH |
module unuse [path] |
Remove path from MODULEPATH |
module --show_hidden av |
Show all available modules in MODULEPATH including hidden modules |
module --show_hidden spider |
Show all possible modules in MODULEPATH and module hierarchy including hidden modules |
ml
Lmod provides another convenient shortcut command for the module command
called ml
for user convenience, which mimics the module command.
Invoking the ml command without arguments is equivalent to module list; any sub-commands to module
are usable with the ml command. For example, ml avail is equivalent to module avail; ml spider
is equivalent to module spider, etc.
The syntax for loading and unloading modules with the ml command differs from the module command.
You can type ml <module> to load a module, and unload by prepending a minus sign (-): ml -<module>.
For instance if we want to load the gcc module we can run ml gcc. To remove this module from the
set of loaded modules we would run ml -gcc.
```console elvis@perlmutter> ml gcc elvis@perlmutter> ml
Currently Loaded Modules: 1) gcc/10.2.0 (c)
Where: c: Compiler ```
One can load and unload module in a single ml command: for instance if you want to unload gcc and load
cuda you could run: ml -gcc cuda or ml cuda -gcc.
Finding Modules
Lmod provides several commands to help you find modules including module avail, module spider and even
module overview. We will discuss a few of these commands here.
To see list of available modules, you can run
shell
module avail
The output of this will be quite long depending on the number of modulefiles, for instance if you want to see all gcc modules
you can run
shell
module avail gcc
The module spider command reports all modules in your system in MODULEPATH along with all module trees in the hierarchical
system. Note that module avail doesn’t show modules from all trees in the hierarchical system. If you want to know all
available software on the system, please use module spider.
The output will be a list of software entries with corresponding versions:
```console elvis@perlmutter> module spider
The following is a list of the modules and extensions currently available:
aocc: aocc/2.2.0.1
atp: atp/3.13.1
cce: cce/11.0.4
clingo: clingo/git_20210514
cmake: cmake/3.18.4
cpe: cpe/21.04
cray-ccdb: cray-ccdb/4.11.1
cray-cti: cray-cti/2.13.6
cray-dsmml: cray-dsmml/0.1.4
cray-fftw: cray-fftw/3.3.8.9
cray-hdf5: cray-hdf5/1.12.0.3
… ```
The module spider can report all versions of the software. For instance if we want to see all
gcc compilers we can run the following. In this example we have three versions of gcc (8.1.0,
9.3.0, and 10.2.0):
```console elvis@perlmutter> module spider gcc
gcc:
Versions:
gcc/8.1.0
gcc/9.3.0
gcc/10.2.0
For detailed information about a specific “gcc” package (including how to load the modules) use the module’s full name. Note that names that have a trailing (E) are extensions provided by other modules. For example:
$ module spider gcc/10.2.0
```
Lmod recently introduced a new command called module overview which displays
each module in short form with a number of module versions. If you want to filter
this output by the name of the module you can pass the name as an argument.
For example:
shell
module overview PrgEnv-cray PrgEnv-gnu
Module hierarchies
Lmod supports a module hierarchy, where sets of modules with a common dependency are “under” some required modules.
For example, the HPE-provided cray-netcdf module depends on
cray-hdf5 as its lower-level I/O support, and the cray-netcdf-hdf5parallel module
depends on cray-hdf5-parallel as parallel I/O.
In Lmod, the module avail command will show that cray-netcdf is not found, and the
output looks the same as checking a module which truly does not exist:
``` perlmutter$ module avail cray-netcdf No module(s) or extension(s) found! Use “module spider” to find all possible modules and extensions. Use “module keyword key1 key2 …” to search for all possible modules matching any of the “keys”.
perlmutter$ module avail gnuplot No module(s) or extension(s) found! …
perlmutter$ module avail aaa No module(s) or extension(s) found! … ```
Using module load is helpful here since it gives the hint that aaa is
truly not available but that cray-netcdf exists but can not be loaded directly.
It also suggests using module spider to see how to load it:
perlmutter$ module load aaa
Lmod has detected the following error: The following module(s) are unknown: "aaa"
...
perlmutter$ module load cray-netcdf
Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: "cray-netcdf"
Try: "module spider cray-netcdf" to see how to load the module(s).
Using module spider gives the available versions, and suggests to use the
spider command on a specific version, which then gives the list of all the
dependency modules you need to load first (modules on any one of the lines
given depending on your user environment choices such as the compiler version):
``` elvis@perlmutter> module spider cray-netcdf
cray-netcdf:
Versions:
cray-netcdf/4.9.0.3
cray-netcdf/4.9.0.7
cray-netcdf/4.9.0.9
cray-netcdf/4.9.0.13
Other possible modules matches:
cray-netcdf-hdf5parallel
To find other possible module matches execute: $ module -r spider ‘.cray-netcdf.‘
For detailed information about a specific “cray-netcdf” package (including how to load the modules) use the module’s full name. Note that names that have a trailing (E) are extensions provided by other modules. For example: $ module spider cray-netcdf/4.9.0.13
elvis@perlmutter> module spider cray-netcdf/4.9.0.13
cray-netcdf: cray-netcdf/4.9.0.13
You will need to load all module(s) on any one of the lines below before the "cray-netcdf/4.9.0.13" module is available to load.
aocc/4.1.0 cray-hdf5/1.14.3.1
gcc-native/12.3 cray-hdf5/1.14.3.1
gcc-native/13.2 cray-hdf5/1.14.3.1
intel-classic/2023.2.0 cray-hdf5/1.14.3.1
intel-oneapi/2023.2.0 cray-hdf5/1.14.3.1
intel/2023.2.0 cray-hdf5/1.14.3.1
intel/2024.1.0 cray-hdf5/1.14.3.1
nvhpc/22.7 cray-hdf5/1.14.3.1
nvhpc/23.1 cray-hdf5/1.14.3.1
nvhpc/23.9 cray-hdf5/1.14.3.1
nvhpc/24.5 cray-hdf5/1.14.3.1
nvhpc/25.5 cray-hdf5/1.14.3.1
nvidia/22.7 cray-hdf5/1.14.3.1
nvidia/23.1 cray-hdf5/1.14.3.1
nvidia/23.9 cray-hdf5/1.14.3.1
nvidia/24.5 cray-hdf5/1.14.3.1
nvidia/25.5 cray-hdf5/1.14.3.1
Help:
Release info: /opt/cray/pe/netcdf/4.9.0.13/release_info
elvis@perlmutter> module load nvidia/25.5 cray-hdf5/1.14.3.1 Lmod is automatically replacing “PrgEnv-gnu/8.5.0” with “PrgEnv-nvidia/8.5.0”. Lmod is automatically replacing “gcc-native/13.2” with “nvidia/25.5”.
Due to MODULEPATH changes, the following have been reloaded: 1) cray-libsci/24.07.0 2) cray-mpich/8.1.30
elvis@perlmutter> module load cray-netcdf/4.9.0.13 elvis@perlmutter> module list
Currently Loaded Modules: 1) craype-x86-milan 6) cpe/24.07 11) darshan/default 16) PrgEnv-nvidia/8.5.0 (prgenv) 2) libfabric/1.22.0 7) cudatoolkit/12.4 12) craype/2.7.32 (c) 17) nvidia/25.5 (g,c) 3) craype-network-ofi 8) craype-accel-nvidia80 13) cray-dsmml/0.3.0 18) cray-hdf5/1.14.3.1 4) xpmem/2.9.7-1.1_20250411150514__g191b5f8bea4c 9) gpu/1.0 14) cray-mpich/8.1.30 (mpi) 19) cray-netcdf/4.9.0.13 5) perftools-base/24.07.0 10) sqs/2.0 15) cray-libsci/24.07.0
Where: g: built for GPU mpi: MPI Providers prgenv: Programming Environment Modules c: Compiler ```
!!! tip
Always use module spider instead of module avail to find out how to module load.
User-Contributed Modules
Several projects provide software via their own user-maintained modules for the benefit
of NERSC users. To access these modules you must first load the contrib module, as
shown:
module load contrib
For example, after running the above command, module avail, will show two additional
locations:
``` ------ /global/common/software/m2878/perlmutter/modulefiles ------ bupc-narrow/2022.10.0 upcxx-extras/master (D) bupc/2022.10.0 upcxx-extras/2022.3.0 opencoarrays/2.10.1 upcxx/bleeding-edge upcxx-cuda/bleeding-edge upcxx/nightly upcxx-cuda/nightly upcxx/2022.9.0 upcxx-cuda/2022.9.0 upcxx/2023.3.0 (D) upcxx-cuda/2023.3.0 (D)
-------- /global/common/software/m4232pub/pm/modulefiles --------- wrf/4.4.0 wrf/4.5.0 (D) ```
The first project directory lists 13 modules provided by the m2878 (UPCXX++ and GASNet-EX) project. The second directory contains two modules provided by m4232 (WRF). Any of these modules can now be loaded as usual.
User Collections
Lmod introduced the concept of user collections,
allowing a user to reference a group of modules with a unique name. This is particularly useful if there
is a set of modules you load regularly to do a particular task. You can save the modules into a collection to
save the trouble of retyping them every time, loading them with the module restore command. Note that Lmod
can load only one user collection at a time.
To save modules in a collection, load the relevant modules and run module save. In this example,
we save our active modules into the default collection.
```console elvis@perlmutter> module purge elvis@perlmutter> module load gcc elvis@perlmutter> module list
Currently Loaded Modules: 1) gcc/10.2.0 (c)
Where: c: Compiler
elvis@perlmutter> module save Saved current collection of modules to: “default”, for system: “perlmutter” ```
Lmod will store user collection in $HOME/.lmod.d. You can view all collections using module savelist.
console
elvis@perlmutter> module savelist
Named collection list (For LMOD_SYSTEM_NAME = "perlmutter"):
1) default
You can see contents of a collection using module describe, which shows the modules that will be loaded
when restoring from the collection. In this next example, we will purge and restore from the collection,
which will simply load the gcc module:
```console elvis@perlmutter> module describe Collection “default” contains: 1) gcc elvis@perlmutter> module purge elvis@perlmutter> module list No modules loaded elvis@perlmutter> module restore Restoring modules from user’s default, for system: “perlmutter” elvis@perlmutter> module list
Currently Loaded Modules: 1) gcc/10.2.0 (c)
Where: c: Compiler ```
Showing the Contents of Modulefiles
There are several commands to reveal the contents of modulefiles: modulewhatis,
module help, module show, and module spider.
The module whatis is a single line summary of modulefile and module help is a multi-line description of the modulefile.
The module show command displays the commands executed when loading the module (module load). Shown below are commands run
in your user shell when loading PrgEnv-nvidia module.
!!! note
The output of `module show` is not the content of the modulefile. If you want to see the content of the modulefile, add
the `--raw` option: `module --raw show PrgEnv-nvidia`.
```console
elvis@perlmutter> module show PrgEnv-nvidia
/opt/cray/pe/lmod/modulefiles/core/PrgEnv-nvidia/8.0.0.lua:
family(“PrgEnv”) help([[ Use ‘module restore PrgEnv-nvidia’ to load the default nvidia programming environment.
]]) whatis(“Enables the Programming Environment using the nvidia compilers.”) setenv(“PE_ENV”,”NVIDIA”) load(“nvidia”) load(“craype”) load(“craype-x86-rome”) load(“craype-network-ofi”) load(“cray-dsmml”) load(“perftools-base”) load(“xpmem”) load(“cray-mpich”) load(“cray-libsci”) ```
Lmod Configuration for Perlmutter
We have made a few changes to Lmod configuration which are described below.
!!! note
Lmod configuration files can be found at $LMOD_CONFIG_DIR. If you want to view Lmod configuration
you can run module --config. Please see https://lmod.readthedocs.io/en/latest/090_configuring_lmod.html for
an overview of Lmod configuration values and their description.
NERSC Defaults
NERSC provides gpu and cpu modules that load recommended defaults
for compiling on GPUs and CPUs, respectively. We may update these
modules as new information or recommendations are made. You can use
module show <gpu/cpu> to see what each module does. You can only
have one of these modules loaded at a time and loading one unloads the
other. Currently, the gpu module is loaded by default.
Format Titles for Module Trees
We have set the environment LMOD_AVAIL_STYLE="grouped:system" which alters the output of module avail. Under
this setting, Lmod displays titles chosen by NERSC for each module tree, rather than the directory path as
its default behavior.
When you invoke module avail you will see a title for each module tree.
??? “module avail with grouped style”
```
elvis@perlmutter> module av
----------------------------------------------------------------- Cray Compiler/Network Dependent Packages -----------------------------------------------------------------
cray-mpich-abi/8.1.4 cray-mpich/8.1.4
--------------------------------------------------------------------- Cray Compiler Dependent Modules ----------------------------------------------------------------------
cray-hdf5/1.12.0.3
---------------------------------------------------------------------- Additional available software -----------------------------------------------------------------------
clingo/git_20210514 darshan/3.2.1 mpich/3.4.1 npe-llvm/0.1 py-cffi/1.14.3 spack/0.16.1
cmake/3.18.4 fast-mkl-amd/1.0 nccl/2.9.6 nvidia-nersc/20.11 python/3.8-anaconda-2020.11 xalt/2.10.2
cudnn/8.2.0 llvm/11.0.1 nccl/2.9.8 (D) nvidia-nersc/21.5 (D) pytorch/1.8.0
----------------------------------------------------------------------------- Lmod Modulefiles -----------------------------------------------------------------------------
lmod settarg
------------------------------------------------------------------------ Cray MPI Dependent Modules ------------------------------------------------------------------------
cray-hdf5-parallel/1.12.0.3 cray-hdf5-parallel/1.12.0.3 (D) cray-parallel-netcdf/1.12.1.3 (D) cray-parallel-netcdf/1.12.1.3
---------------------------------------------------------------------------- Cray Core Modules -----------------------------------------------------------------------------
aocc/2.2.0.1 (D) cray-dsmml/0.1.4 cray-R/4.0.3.0 (D) gcc/10.2.0 (L,D) nvidia/20.9 (D) PrgEnv-nvidia/8.0.0
atp/3.13.1 cray-jemalloc/5.1.0.4 cray-stat/4.10.1 gdb4hpc/4.12.5 papi/6.0.0.6 valgrind4hpc/2.11.1
cce/11.0.4 cray-libsci/21.04.1.1 craype/2.7.6 iobuf/2.0.10 perftools-base/21.02.0
cpe/21.04 cray-pmi-lib/6.0.10 craypkg-gen/1.3.14 nvhpc-byo-compiler/20.9 (D) PrgEnv-aocc/8.0.0
cray-ccdb/4.11.1 cray-pmi/6.0.10 cudatoolkit/20.9_11.0 (D) nvhpc-nompi/20.9 (D) PrgEnv-cray/8.0.0
cray-cti/2.13.6 cray-python/3.8.5.0 gcc/9.3.0 nvhpc/20.9 (D) PrgEnv-gnu/8.0.0
----------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/craype-targets/default -----------------------------------------------------------
craype-accel-amd-gfx908 craype-accel-nvidia80 craype-hugepages1G craype-hugepages2M craype-hugepages512M craype-network-none craype-x86-milan
craype-accel-host craype-hugepages128M craype-hugepages256M craype-hugepages32M craype-hugepages64M craype-network-ofi craype-x86-rome
craype-accel-nvidia70 craype-hugepages16M craype-hugepages2G craype-hugepages4M craype-hugepages8M craype-network-ucx
------------------------------------------------------------------------------- Cray Modules -------------------------------------------------------------------------------
cray-ucx/2.7.0-1 cudatoolkit/20.9_11.0 libfabric/1.11.0.3.66 nvhpc-byo-compiler/20.9 nvhpc-nompi/20.9 nvhpc/20.9
------------------------------------------------------------------------------ Cray Compilers ------------------------------------------------------------------------------
aocc/2.2.0.1 cray-R/4.0.3.0 gcc/8.1.0 gcc/9.3.0 gcc/10.2.0 nvidia/20.9
------------------------------------------------------------------------- NERSC-provided Software --------------------------------------------------------------------------
Default
Where:
L: Module is loaded
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
```
If you want to see the full path to module tree you can use the -s <system> option, which changes the Lmod avail
style to prefer system instead of grouped. To make this change persistent, set the environment variable
LMOD_AVAIL_STYLE="system:grouped" Shown below is an output using the system avail format.
??? “module avail with system style”
```console
elvis@perlmutter> module -s system av
----------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/comnet/gnu/8.0/ofi/1.0 -----------------------------------------------------------
cray-mpich-abi/8.1.4 cray-mpich/8.1.4
-------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/compiler/gnu/8.0 --------------------------------------------------------------
cray-hdf5/1.12.0.3
-------------------------------------------------------- /global/common/software/nersc/shasta2105/extra_modulefiles --------------------------------------------------------
clingo/git_20210514 darshan/3.2.1 mpich/3.4.1 npe-llvm/0.1 py-cffi/1.14.3 spack/0.16.1
cmake/3.18.4 fast-mkl-amd/1.0 nccl/2.9.6 nvidia-nersc/20.11 python/3.8-anaconda-2020.11 xalt/2.10.2
cudnn/8.2.0 llvm/11.0.1 nccl/2.9.8 (D) nvidia-nersc/21.5 (D) pytorch/1.8.0
------------------------------------------------------------------ /usr/share/lmod/lmod/modulefiles/Core -------------------------------------------------------------------
lmod settarg
---------------------------------------------------- /opt/cray/pe/lmod/modulefiles/mpi/nvidia/20/ofi/1.0/cray-mpich/8.0 ----------------------------------------------------
cray-hdf5-parallel/1.12.0.3 cray-parallel-netcdf/1.12.1.3 (D)
------------------------------------------------- /opt/cray/pe/lmod/modulefiles/mpi/crayclang/10.0/ofi/1.0/cray-mpich/8.0 --------------------------------------------------
cray-hdf5-parallel/1.12.0.3 (D) cray-parallel-netcdf/1.12.1.3
-------------------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/core --------------------------------------------------------------------
aocc/2.2.0.1 (D) cray-dsmml/0.1.4 cray-R/4.0.3.0 (D) gcc/10.2.0 (L,D) nvidia/20.9 (D) PrgEnv-nvidia/8.0.0
atp/3.13.1 cray-jemalloc/5.1.0.4 cray-stat/4.10.1 gdb4hpc/4.12.5 papi/6.0.0.6 valgrind4hpc/2.11.1
cce/11.0.4 cray-libsci/21.04.1.1 craype/2.7.6 iobuf/2.0.10 perftools-base/21.02.0
cpe/21.04 cray-pmi-lib/6.0.10 craypkg-gen/1.3.14 nvhpc-byo-compiler/20.9 (D) PrgEnv-aocc/8.0.0
cray-ccdb/4.11.1 cray-pmi/6.0.10 cudatoolkit/20.9_11.0 (D) nvhpc-nompi/20.9 (D) PrgEnv-cray/8.0.0
cray-cti/2.13.6 cray-python/3.8.5.0 gcc/9.3.0 nvhpc/20.9 (D) PrgEnv-gnu/8.0.0
----------------------------------------------------------- /opt/cray/pe/lmod/modulefiles/craype-targets/default -----------------------------------------------------------
craype-accel-amd-gfx908 craype-accel-nvidia80 craype-hugepages1G craype-hugepages2M craype-hugepages512M craype-network-none craype-x86-milan
craype-accel-host craype-hugepages128M craype-hugepages256M craype-hugepages32M craype-hugepages64M craype-network-ofi craype-x86-rome
craype-accel-nvidia70 craype-hugepages16M craype-hugepages2G craype-hugepages4M craype-hugepages8M craype-network-ucx
-------------------------------------------------------------------------- /opt/cray/modulefiles ---------------------------------------------------------------------------
cray-ucx/2.7.0-1 cudatoolkit/20.9_11.0 libfabric/1.11.0.3.66 nvhpc-byo-compiler/20.9 nvhpc-nompi/20.9 nvhpc/20.9
----------------------------------------------------------------------------- /opt/modulefiles -----------------------------------------------------------------------------
aocc/2.2.0.1 cray-R/4.0.3.0 gcc/8.1.0 gcc/9.3.0 gcc/10.2.0 nvidia/20.9
----------------------------------------------------------- /global/common/software/nersc/shasta2105/modulefiles -----------------------------------------------------------
Default
Where:
L: Module is loaded
D: Default Module
Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
```
Lmod Families
The Lmod family("name") Lua function enables the concept of
families to
be enforced.
The basic idea is that modules belong to a family group, and only one module
from the family group can be loaded at a given time.
The purpose is to protect users from loading conflicting modules, e.g., two MPI
libraries such as openmpi and mpich that provide the same binaries
mpicc, mpifort, mpic++.
HPE/Cray has defined several families in their software stack. The compiler
family contains the gcc, cce, aocc and nvidia modules. Lmod will
automatically swap modules when you load modules from same family group. Shown
below, we see that Lmod swaps nvidia with aocc and aocc with gcc since
Lmod permits only one module from the compiler family to be loaded at a time.
```console elvis@perlmutter> module load aocc
Lmod is automatically replacing “nvidia/20.9” with “aocc/2.2.0.1”.
elvis@perlmutter> module load gcc
Lmod is automatically replacing “aocc/2.2.0.1” with “gcc/10.2.0”.
```
A quick way to check the active families is to run module --mt, which shows the module tables state along with family names.
In the below output we see there are five families defined: PrgEnv, compiler, craype, craype_cpu, and craype_network.
console
elvis@perlmutter> module --mt
_ModuleTable_ = {
["MTversion"] = 3,
["c_rebuildTime"] = 7200.0,
["c_shortTime"] = 0.53126287460327,
depthT = {},
family = {
["PrgEnv"] = "PrgEnv-nvidia",
["compiler"] = "gcc",
["craype"] = "craype",
["craype_cpu"] = "craype-x86-rome",
["craype_network"] = "craype-network-ofi",
},
...
Lmod will set the environment variables LMOD_FAMILIY_<NAME> and LMOD_FAMILY_<NAME>_VERSION, where <name> is
the name of family that can be used to reference family.
Module Properties
We have configured module properties for a subset of modules to
help classify the software provided. If you run module list, you will see properties such as cpe, math, io,
c, dev to help you identify what type of software the module provides.
```console elvis@perlmutter> module list
Currently Loaded Modules: 1) nvidia/20.9 (g,c) 4) libfabric/1.11.0.3.66 7) perftools-base/21.02.0 (dev) 10) PrgEnv-nvidia/8.0.0 (cpe) 13) cray-pmi/6.0.10 2) craype/2.7.6 (c) 5) craype-network-ofi 8) xpmem/2.2.40-7.0.1.0_1.9__g1d7a24d.shasta (H) 11) xalt/2.10.2 14) cray-pmi-lib/6.0.10 3) craype-x86-rome 6) cray-dsmml/0.1.4 9) cray-libsci/21.04.1.1 (math) 12) darshan/3.2.1 (io) 15) Default
Where: g: built for GPU cpe: Cray Programming Environment Modules math: Mathematical libraries io: Input/output software c: Compiler dev: Development Tools and Programming Languages H: Hidden Module ```
Startup module
We have configured Lmod to provide a module that is loaded by default for all users,
called Default. This module can be loaded using module load Default. If you are unsure how to
get to the startup configuration you can do one of the following:
- Run
module restore - Run
module purge && module load Default
!!! note
If you have a default collection (a collection entitled `default`), then `module restore` will restore your
user collection instead of the system default. You can check your collections by running `module savelist` or
examining the content of the files in **$HOME/.lmod.d**
If you don’t have a default user collection, then Lmod will restore from the startup modulefile Default. This
is done by Lmod by reading the environment variable LMOD_SYSTEM_DEFAULT_MODULES which should not be changed.
```console elvis@perlmutter> module purge elvis@perlmutter> module restore Resetting modules to system default. Reseting $MODULEPATH back to system default. All extra directories will be removed from $MODULEPATH. elvis@perlmutter> ml
Currently Loaded Modules: 1) cray-mpich/8.1.4 (mpi) 7) cray-dsmml/0.1.4 13) darshan/3.2.1 (io) 2) nvidia/20.9 (g,c) 8) perftools-base/21.02.0 (dev) 14) cray-pmi/6.0.10 3) craype/2.7.6 (c) 9) xpmem/2.2.40-7.0.1.0_1.9__g1d7a24d.shasta (H) 15) cray-pmi-lib/6.0.10 4) craype-x86-rome 10) cray-libsci/21.04.1.1 (math) 16) Default 5) libfabric/1.11.0.3.66 11) PrgEnv-nvidia/8.0.0 (cpe) 6) craype-network-ofi 12) xalt/2.10.2
Where: g: built for GPU mpi: MPI Providers cpe: Cray Programming Environment Modules math: Mathematical libraries io: Input/output software c: Compiler dev: Development Tools and Programming Languages H: Hidden Module ```
Please refer to see https://lmod.readthedocs.io/en/latest/070_standard_modules.html for details related to startup modules
Useful Tips
Redirecting Module Output
Lmod will redirect output to stderr by default, which means this won’t work as you expected, since its output will not be stored in a file.
```console elvis@perlmutter> module list > active.txt
Currently Loaded Modules: 1) craype/2.7.6 (c) 5) cray-dsmml/0.1.4 9) PrgEnv-nvidia/8.0.0 (cpe) 13) cray-pmi-lib/6.0.10 2) craype-x86-rome 6) perftools-base/21.02.0 (dev) 10) xalt/2.10.2 14) Default 3) libfabric/1.11.0.3.66 7) xpmem/2.2.40-7.0.1.0_1.9__g1d7a24d.shasta (H) 11) darshan/3.2.1 (io) 15) gcc/10.2.0 (c) 4) craype-network-ofi 8) cray-libsci/21.04.1.1 (math) 12) cray-pmi/6.0.10
Where: cpe: Cray Programming Environment Modules math: Mathematical libraries io: Input/output software c: Compiler dev: Development Tools and Programming Languages H: Hidden Module ```
The --redirect option to any module command will redirect stderr to stdout so that shell commands
can capture module commands.
```console elvis@perlmutter> module –redirect list > active.txt elvis@perlmutter> cat active.txt
Currently Loaded Modules: 1) craype/2.7.6 (c) 5) cray-dsmml/0.1.4 9) PrgEnv-nvidia/8.0.0 (cpe) 13) cray-pmi-lib/6.0.10 2) craype-x86-rome 6) perftools-base/21.02.0 (dev) 10) xalt/2.10.2 14) Default 3) libfabric/1.11.0.3.66 7) xpmem/2.2.40-7.0.1.0_1.9__g1d7a24d.shasta (H) 11) darshan/3.2.1 (io) 15) gcc/10.2.0 (c) 4) craype-network-ofi 8) cray-libsci/21.04.1.1 (math) 12) cray-pmi/6.0.10
Where: cpe: Cray Programming Environment Modules math: Mathematical libraries io: Input/output software c: Compiler dev: Development Tools and Programming Languages H: Hidden Module ```
Autoswap modules of the same name
Lmod will automatically swap modules of the same name. For instance, if you load gcc/9.3.0, Lmod will remove gcc/10.2.0 from
your user environment:
```console elvis@perlmutter> module load gcc/9.3.0
The following have been reloaded with a version change: 1) gcc/10.2.0 => gcc/9.3.0 ```
Debugging Modules
If you want to debug the state of modules, any of these command options can help:
- Tracing modules:
module -T - Print Module Table:
module --mt - Debug Level:
module --debug=[1|2|3]
To debug modulefiles themselves, please see https://lmod.readthedocs.io/en/latest/160_debugging_modulefiles.html
Parsing output
Use the -t option with module avail, module spider, module list, module spider and module savelist commands
to produce parsable output.
To see loaded modules in a parsable format, the environment variable LOADEDMODULES is a colon separated
list of active modules loaded in your shell.
console
elvis@perlmutter> echo $LOADEDMODULES
gcc/10.2.0:dvs/2.12_4.0.102-7.0.1.0_8.1__g30d29e7a
Similarly, the full path to the modulefiles of active modules can be retrieved using the _LMFILES_ environment
variable. The output is a colon-separated list of modulefiles.
console
elvis@perlmutter> echo $_LMFILES_
/opt/cray/pe/lmod/modulefiles/core/gcc/10.2.0.lua:/opt/cray/modulefiles/dvs/2.12_4.0.102-7.0.1.0_8.1__g30d29e7a
Seeing Defaults for Modulefiles
The module -d avail command will report the default for every module. Lmod will load the
default module if you don’t specify the full version (i.e., module load gcc) since there is
only one default for every module name.
For example, to list the default for gcc:
```console elvis@perlmutter> module -d avail gcc
---------------------------------------------------------------------------- Cray Core Modules ----------------------------------------------------------------------------- gcc/10.2.0 (L)
Where: L: Module is loaded
Use “module spider” to find all possible modules and extensions. Use “module keyword key1 key2 …” to search for all possible modules matching any of the “keys”. ```
Hidden Modules
Hidden modules are modulefiles with a leading dot (.) in front of the version. These modules are not shown when running
module avail or module spider. In order to see these modules you need to use the --show_hidden option. This option
is also required if you want to show the content of a hidden modulefile: module --show_hidden show <name>,
module --show_hidden help <name>, or module --show_hidden whatis <name>.
A hidden module will have a symbol (H) next to the modulefile.
In example below we have dvs as a hidden module which is not reported in the output of module avail
```console elvis@perlmutter> module avail dvs No module(s) or extension(s) found! Use “module spider” to find all possible modules and extensions. Use “module keyword key1 key2 …” to search for all possible modules matching any of the “keys”.
elvis@perlmutter> module –show_hidden avail dvs
------------------------------------------------------------------------------- Cray Modules ------------------------------------------------------------------------------- dvs/2.12_4.0.102-7.0.1.0_8.1__g30d29e7a (H)
Where: H: Hidden Module
```
For Perlmutter, we have configured our own system defaults and hidden modules in the modulerc.lua file, which can be retrieved by the environment variable $LMOD_MODULERCFILE.
Managing User Collection
Lmod will store user collections in $HOME/.lmod.d, so if you want to delete a user collection,
you can simply delete the file in the $HOME/.lmod.d directory.
Shown below we have a default collection stored this in the file $HOME/.lmod.d/default.perlmutter.
console
elvis@perlmutter> ls ~/.lmod.d/
default.perlmutter
elvis@perlmutter> module -t savelist
default
If you want to delete the user collection default, you can remove the file
console
elvis@perlmutter> rm ~/.lmod.d/default.perlmutter
elvis@perlmutter> module describe default
Lmod Warning: No collection named "default" found.
Alternatively you can use module disable [collection] to disable a collection name. Lmod will
rename the collection name by appending a ~ to the collection name so that
Lmod won’t read the collection. In the below example we create a user collection named gcc and
running module disable gcc will rename the collection to gcc.perlmutter~; however Lmod doesn’t
recognize the gcc collection when running module describe gcc. To re-enable the user collection,
rename the file by removing the ~.
```console elvis@perlmutter> module purge elvis@perlmutter> module load gcc elvis@perlmutter> module save gcc Saved current collection of modules to: “gcc”, for system: “perlmutter”
elvis@perlmutter> ls ~/.lmod.d/ gcc.perlmutter elvis@perlmutter> module disable gcc Disabling gcc collection by renaming with a “~” elvis@perlmutter> ls ~/.lmod.d/ gcc.perlmutter~ elvis@perlmutter> module describe gcc Lmod Warning: No collection named “gcc” found. ```
Customize Module Defaults at User Level
Lmod has support for customizing module defaults at the user level. This is
done by creating a file $HOME/.modulerc.lua.
Let’s assume you want to use the latest PrgEnv-nvidia/8.4.0 as the default
module however the system default is PrgEnv-nvidia/8.3.3 so you can simply
type module load PrgEnv-nvidia without having to type the version. Unfortunately, you
can’t make this change at the system level since it will affect all users, however you
can make this change at the user level.
```console elvis@perlmutter> ml av PrgEnv-nvidia
------------------------------------------------------------------------------------------- Cray Core Modules -------------------------------------------------------------------------------------------- PrgEnv-nvidia/8.3.3 (D) PrgEnv-nvidia/8.4.0
Where: D: Default Module
If the avail list is too long consider trying:
“module –default avail” or “ml -d av” to just list the default modules. “module overview” or “ml ov” to display the number of modules for each name.
Use “module spider” to find all possible modules and extensions. Use “module keyword key1 key2 …” to search for all possible modules matching any of the “keys”. ```
To make this change you can add the following line in the ~/.modulerc.lua file.
lua
module_version("PrgEnv-nvidia/8.4.0","default")
Next, let’s rerun the module av command and take note of the (D) next to
the module name.
```console elvis@perlmutter> module av PrgEnv-nvidia
------------------------------------------------------------------------------------------- Cray Core Modules -------------------------------------------------------------------------------------------- PrgEnv-nvidia/8.3.3 PrgEnv-nvidia/8.4.0 (D)
Where: D: Default Module
If the avail list is too long consider trying:
“module –default avail” or “ml -d av” to just list the default modules. “module overview” or “ml ov” to display the number of modules for each name.
Use “module spider” to find all possible modules and extensions. Use “module keyword key1 key2 …” to search for all possible modules matching any of the “keys”. ```
The .modulerc.lua file can be used for further customization such as hiding modules or setting aliases.
For more details, we encourage you to check out https://lmod.readthedocs.io/en/latest/093_modulerc.html
Troubleshooting User Environment Issues
If you have issues with your user environment, please review your startup configuration files
for any module commands. Look at the files sourced as a result of your configuration.
- bash/sh users:
$HOME/.bashrc,$HOME/.bashrc_profile,$HOME/.profile - csh/tcsh shell:
$HOME/.cshrc
In particular, look for changes to the environment variable MODULEPATH or use of
module use command in your startup configuration.
!!! warning “‘module: Command not found.’ with csh”
Users of csh/tcsh may encounter the error module: Command not found.
when trying to load modules on Perlmutter. To mitigate this issue, we
suggest that csh users execute source $LMOD_PKG/init/csh prior to
running any module commands on Perlmutter.
References
Please see the references below for additional help. NERSC’s Lmod Training (the final link below) is a self-paced exercise covering the basics of Lmod and Lua modulefiles in a Docker container.
- Documentation: https://lmod.readthedocs.io/en/latest/index.html
- User Guide: https://lmod.readthedocs.io/en/latest/010_user.html
- GitHub: https://github.com/TACC/Lmod
- FAQ: https://lmod.readthedocs.io/en/latest/040_FAQ.html
- Lmod Training: https://gitlab.com/NERSC/lmod-training