This is an old revision of the document!
The Modules package is supplied on the system to set up the users's environment variables to run a choice of the needed programs and versions. The most important of these variables are
$PATH, telling the system where to find executable files such as
$LD_LIBRARY_PATH, telling the system where to find shared libraries that an executable calls. You can manipulate the environment variables yourself instead of calling modules, but there is no advantage in doing so.
The four critical module commands are
list (currently loaded),
avail (all available),
purge. The combination purge/load clears all loaded modules and sets new ones:
$ module list Currently Loaded Modulefiles: 1) intel/14.0.3 2) impi/5.0.0 $ module avail -------- /share/apps/modules/Modules/3.2.10/modulefiles ------------------------------------------------ dot gcc/4.9.1 impi/5.1.1 mkl/14.0.3 module-info mvapich2/2.1 openmpi/1.8.6 perl/5 gcc/4.7.2 impi/5.0.0 intel/14.0.3 module-git modules null openmpi/1.8.8 use.own $ module purge $ module load intel/14.0.3 mvapich2/2.1 $ module list Currently Loaded Modulefiles: 1) intel/14.0.3 2) impi/5.0.0 $ module purge $ module load intel/14.0.3 mkl/14.0.3 mvapich2/2.1 $ module list Currently Loaded Modulefiles: 1) intel/14.0.3 2) mkl/14.0.3. 2) mvapich2/2.1
.bashrc is a file in your home directory that also sets up your user environment, either by
module load … or directly setting environment variables. Here is a basic template
.bashrc used to set up new accounts. If you don't want to use these modules, comment out the
module load … statements in
.bashrc and relogin. If a
module load … is issued with a program but no version as shown below, the system-defined default version of that program module is used.
module list will show the actual versions loaded.
$ cat ~/.bashrc . /etc/profile.d/env-modules.sh ulimit -s unlimited 2>/dev/null ulimit -l unlimited 2>/dev/null #if using goto/mkl blas with mpi these should be set to 1 unless you want hybrid mpi/openmp export GOTO_NUM_THREADS=1 export OMP_NUM_THREADS=1 export MKL_NUM_THREADS=1 #enter your modules here module load intel module load openmpi module load mkl [ -z "$PS1" ] && return PS1='`/bin/hostname -s`:`whoami`:`echo $PWD | sed "s=$HOME=="`$ ' alias ls='ls --color=auto' module list
.bashrc is sourced at the beginning of each interactive job. There is a similar file
.bash_profile sourced at the beginning of each non-interactive job. In our setup, we source
.bash_profile so that the files are effectively the same, thus reducing the maintenance effort. Interactive or batch is determined in
[ -z “$PS1” ] && return which drops out of the loop on batch runs, so commands following that are for interactive sessions only, like setting the value of the prompt
$PS1. Commands towards the top of ``.bashrc`` are for both interactive and batch.
Here are some recommended module/.bashrc setups for different cases:
Your run only one program, or all the programs you run use the same modules, or use different modules that don't conflict
module load … in
[ -z…. The same environment will be loaded from
.bashrc for every interactive session, batch job, and MPI program if any. Many modules, for instance
python, can be assumed to not conflict, though many combinations have not been tested. Modules that definitely do conflict are MPI modules, only one may be safely used at a time, and multiple versions of the same program, like
gcc/4.9.1. Multiple compilers, such as
intel usually don't conflict, but the compiler module is used to load the MPI module, so multiple compiler modules should be loaded in order (1) compiler module you want to use with MPI (2) MPI module (3) additional compiler module.
You use different (conflicting) modules for different programs, but only run single-node batch jobs
If you put
module load … in your
.bashrc, every interactive session and batch and MPI thread will load that, which won't be good if it's the wrong MPI version. Delete or comment out (add leading “#” to) the
module load … in
.bashrc. Put the relevant
module load … in your batch scripts after the
#PBS statements, for example:
#PBS ... #PBS -l node=1:ppn=12 module load R openmpi cd $PBS_O_WORKDIR R < myprog.R --nosave --vanilla
In this case, for interactive sessions such as compiling, type the relevant
module load … in your session before work.
You use different (conflicting) modules for different programs, and also run multi-node MPI jobs
This is the most difficult case and also common. The first two solutions won't always work. If a module is set in a batch script using multiple nodes, it definitely applies to the MPI threads running in the first or “master” compute node (usually the lowest numbered assigned node in our batch configuration) but does not necessarily apply to the “slave” compute nodes, depending how different MPI versions issue remote threads. Multiple nodes imply MPI is being used, and the solution varies by MPI type:
If you set the ``impi`` module at the top of your batch file, paths will be passed to slave nodes and a multiple-node job will run. If you set other environment variables in the batch file, they won't be passed to slave nodes and must be set either (a) in .bashrc or (b) set in the
mpirun statement, for instance the number of MKL threads below. Notice no equals sign in the assignment statement.
#PBS ... #PBS -l node=2:ppn=12 module load intel/14.0.3 impi/5.0.0 cd $PBS_O_WORKDIR mpirun -np 6 -machinefile $PBS_NODEFILE -genv MKL_NUM_THREADS 4 ./xhpl >logfile
MVAPICH2 is similar to Intel MPI in that program paths will correctly pass to slave nodes, but other environment variables won't, and have to be set in
.bashrc or deliberately passed by
mpirun. Syntax varies by one letter, and also has no equals sign.
#PBS ... #PBS -l node=2:ppn=12 module load intel/14.0.3 mvapich2/2.1 cd $PBS_O_WORKDIR mpirun -np 6 -machinefile $PBS_NODEFILE -env MKL_NUM_THREADS 4 ./xhpl >logfile
Open MPI in the latest versions will correctly pass program paths except
$LD_LIBRARY_PATH to slave compute nodes, so the
-x LD_LIBRARY_PATH below is required, and here an optional environment variable
MKL_NUM_THREADS is also set. Again syntax is slightly different, no equals and no value means pass the existing value, and =value means set and pass the value. This only works reliably in our setup for Open MPI version at least 1.8.8. Earlier versions need the
openmpi module set in
.bashrc. Recent Open MPI versions are much faster, so programs using early versions should be recompiled anyway. Small version changes like openmpi/1.8.6 to openmpi/1.8.8 will usually work without recompiling.
#PBS ... #PBS -l node=2:ppn=12 module load intel/14.0.3 openmpi/1.8.8 cd $PBS_O_WORKDIR mpirun -np 6 -machinefile $PBS_NODEFILE -x LD_LIBRARY_PATH -x MKL_NUM_THREADS=4 ./xhpl >logfile