Module System Tutorial
The HPCC has a large amount of software installed in order to support its diverse users. This can include multiple versions of the same software. The module system exists to manage all of this by making software available to users and preventing version conflicts.
In this tutorial, you'll learn how to use the module system to:
- See which modules are currently loaded
- Search for available software versions
- Check requirements for particular modules
- Loading modules
- Saving currently loaded modules to easily reload
For the purposes of this tutorial, we'll be trying to load version 4.1.0 of the R interpreter.
Viewing Currently Loaded Modules
Several modules are already loaded by default once you log on to a development node. These include several commonly used packages such as Python, MATLAB, and the GNU compiler.
module list to see all currently available modules. The output will look like the following:
1 2 3 4 5 6 7 8 9
Unfortunately, R isn't loaded by default. We'll have to figure out what's needed to load it ourselves.
Searching for Available Modules
module command accepts a variety of "sub-commands". The example we used above,
list, is an example of a sub-command.
What other sub-commands are available? Run
module by itself to find out.
You'll see a long list of available sub-commands printed to the screen. Scroll up until you see the portion on listing and searching:
1 2 3 4 5 6 7 8 9 10 11 12 13
Here we see the sub-command
list, which we've already encountered. We'll cover
keyword in other documentation. For now, let's focus on the
spider sub-command is the most useful way to search through available modules. Its name isn't obvious, but you can think of sending a spider to walk through a tangled web of modules to find the ones matching your request.
The list of sub-commands from
module shows four options for the
module spider sub-command:
|All possible modules
|All versions of that module
|All modules containing
|Details about a module version
Searching by Module Name
For this tutorial we want to search for the versions of the R interpreter. Within the module system on the HPCC, "R" with a capital R is the formal name of the module.
module spider R. An abbreviated output is reproduced below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
We see a full list of all available R versions as well as some other helpful information:
- Other possible module names we may have been searching for
- How to search for all modules containing the string "R"
- How to get detailed information on a specific version
The first two points reference searching by
string in the table above, rather than searching by
module. The third point references the
What's the difference between searching by
module and searching by
string? As mentioned at the start of this section, "R" is the formal name of the module. The module system tries to be case insensitive, but it can have odd results.
module spider r with a lowercase r and you'll see we've executed a
string search, returning all modules that include the letter R!
This is probably more information than you would like. Press
q to quit and return to the terminal.
That said, searching by
string is powerful if you don't know the precise name of the module you are looking for.
Details on a Module Version
All modules listed by
module spider follow the
name/version format. We can use this format to get more information on a specific version.
We're interested in using R 4.1.0, so run
module spider R/4.1.0.
You'll again see a long list of output. The portion we are interested in is this:
1 2 3
This section tells us the required dependencies for our module. A module may have multiple possible sets of dependencies; for instance, different combinations of compilers and MPI libraries. Each different set will be listed on a new line.
For R 4.1.0 however there is only one possible dependency: version 8.3.0 of the GCC module and version 3.1.4 of the Open MPI module.
Now that we understand the dependencies, let's move on to loading our module.
As we saw in the last section, when checking module details, R 4.1.0 requires the GCC module with version 8.3.0 and the Open MPI module with version 3.1.4.
Let's check the output of
module list again:
1 2 3 4 5 6 7 8 9
Currently we have
OpenMPI/2.1.2 loaded. Notice
GCCcore is not the same as the
Both GCC core and Open MPI configure several compilers. Their many versions are dependencies for many modules. Since we already have several modules loaded by default which may or may not require both GCC core 6.4.0 and Open MPI 2.1.2, it's best to start with a clean module environment.
module purge to remove all currently loaded modules. You can confirm all modules have been unloaded with
What happens if we just try to load R directly? Run
module load R/4.1.0 and you should see the following error:
1 2 3
We cannot load a module without first loading its dependencies! The error message gives us a helpful reminder about how to learn what those requirements are, but we already know that R 4.1.0 requires
module load GCC/8.3.0 OpenMPI/3.1.4 R/4.1.0 to load R along with its dependency.
You will not see any output after this command. Run
module list if you would like to confirm it worked.
Saving and Restoring Loaded Modules
We often use the same pieces of software over and over on the HPCC. Remembering all the modules we need every time we log in can be a hassle.
Thankfully, the module system lets us save and restore different configurations.
Saving a Configuration
Now that we only have GCC, OpenMPI, and R version 4.1.0 loaded, let's save a configuration to easily access these modules later. We'll name it
module save R-example. You should see the following confirmation message:
Restoring a Configuration
Log out of the HPCC and log back in to a development node. This will reset your loaded modules to the default; run
module list to confirm this is the case.
module savelist to see our stored configurations. Confirm that you see
module restore R-example. The existing modules will automatically be purged and your desired modules loaded!
You should now understand the basics of the module system. For a refresher on searching for modules, see Searching software modules.