-- mode: markdown; mode: visual-line; fill-column: 80 --
overview.md
Copyright (c) 2014 Sebastien Varrette www
Time-stamp: <Ven 2014-11-14 16:05 svarrette>
Overview and Underlying Tools
Environment Modules
Environment modules are a standard and well-established technology across HPC sites, to permit developing and using complex software builds with dependencies, allowing multiple versions of software stacks and combinations thereof.
Modules -- Software Environment Management, offers a standard and well-established approach common to most HPC facilities to provide complex [tweaked] software and libraries with dependencies in multiple versions to their users.
The tool in itself is used to manage environment variables such as PATH
, LD_LIBRARY_PATH
and MANPATH
, enabling the easy loading and unloading of application/library profiles and their dependencies.
| Command | Description |
|--------------------------------+-------------------------------------------------------------- |
| `module avail` | Lists all the modules which are available to be loaded |
| `module load <mod1> [mod2...]` | Load a module |
| `module unload <module>` | Unload a module |
| `module list` | List loaded modules |
| `module display <module>` | Display what a module does |
| `module purge` | Unload all modules (purge) |
| `module use <path>` | Prepend the directory to the MODULEPATH environment variable |
| `module unuse <path>` | Remove the directory from the MODULEPATH environment variable |
| | |
Note: for more information, see the reference man pages for modules and modulefile, or the official FAQ.
You can also see the following pages on the UL HPC website:
At the heart of environment modules interaction resides the following components:
- the
MODULEPATH
environment variable, which defined the list of searched directories for modulefiles modulefile
(see an example) associated to each available software.
Example of layout:
$> pwd
/path/to//modulefiles/hopper/myzlib
$> ls -al
total 640
drwxrwsr-x 2 xxx xxx 131072 2013-03-24 11:14 .
drwxrwsr-x 11 xxx xxx 131072 2013-03-24 11:16 ..
-rw-rw-r-- 1 xxx xxx 1656 2013-03-24 11:14 1.2.7
$> cat 1.2.7
#%Module1.0
##
## Required internal variables
set name zlib
set version 1.2.7
set root /usr/common/usg/$name/$version
[...]
Lmod
Lmod is a Lua based module system that easily handles the MODULEPATH
Hierarchical problem.
Lmod is a new implementation of Environment Modules that easily handles the MODULEPATH Hierarchical problem. It is drop-in replacement for TCL/C modules and reads TCL modulefiles directly. In particular, Lmod add many interesting features on top of the traditional implementation focusing on an easier interaction (search, load etc.) for the users.
Easybuild
Easybuild is a software build and installation framework that allows you to manage (scientific) software on High Performance Computing (HPC) systems in an efficient way. It is motivated by the need for a tool that combines the following features:
- a flexible framework for building/installing (scientific) software
- fully automates software builds
- full support and setup of Environment Modules or Lmod, eventually with a hierarchical layout
- divert from the standard
configure / make / make install
with custom procedures - allows for easily reproducing previous builds
- keep the software build recipes/specifications simple and human-readable
- supports co-existence of versions/builds via dedicated installation prefix and module files
- enables sharing with the HPC community
- automagic dependency resolution
- retain logs for traceability of the build processes
For all these reasons, Easybuild has been selected as the reference middleware to handle the building and the installation of the software provided via the modules environment.
On top of the above features, the objective is also to permit to HPC users and newbies to easily build a non-provided software (instead of an admin ;)) on top of the provided software/toolchain stack.
In particular, instead of digging through some installation documentation, fighting with some --prefix
issue to handle our classical rtfm suggestion (i.e. "we have no time now so install it in your homedir"), it might be as simple as:
# Search for an existing Easyconfigs file for
$> eb --search mysoft
[...]
* $CFGS1/mysoft-3.5-goolf-1.4.10-dmpar.eb
* $CFGS1/mysoft-3.5-ictce-4.1.13-dmpar.eb
[...]
# install it
$> eb mysoft-3.5-goolf-1.4.10-dmpar.eb --robot
Once the installation has succeeded, modules will be available for mysoft
and all of its dependencies with the classical module load mysoft
.