User Tools

Site Tools


modules

This is an old revision of the document!


Environment Modules and .bashrc

The Modules package is supplied on the system to set up the users's environment variables to run a choice of the needed programs and versions. The most important of these variables are $PATH, telling the system where to find executable files such as matlab or mpirun, and $LD_LIBRARY_PATH, telling the system where to find shared libraries that an executable calls. You can manipulate the environment variables yourself instead of calling modules, but there is no advantage in doing so.

The four critical module commands are list (currently loaded),avail (all available), load, and purge. The combination purge/load clears all loaded modules and sets new ones:

$ module list
Currently Loaded Modulefiles:
  1) intel/14.0.3   2) impi/5.0.0
$ module avail
-------- /share/apps/modules/Modules/3.2.10/modulefiles ------------------------------------------------
dot           gcc/4.9.1     impi/5.1.1    mkl/14.0.3    module-info   mvapich2/2.1  openmpi/1.8.6 perl/5
gcc/4.7.2     impi/5.0.0    intel/14.0.3  module-git    modules       null          openmpi/1.8.8 use.own
$ module purge
$ module load intel/14.0.3 mvapich2/2.1
$ module list
Currently Loaded Modulefiles:
  1) intel/14.0.3   2) impi/5.0.0
$ module purge
$ module load intel/14.0.3 mkl/14.0.3 mvapich2/2.1
$ module list
Currently Loaded Modulefiles:
  1) intel/14.0.3   2) mkl/14.0.3. 2) mvapich2/2.1

.bashrc is a file in your home directory that also sets up your user environment, either by module load … or directly setting environment variables. Here is a basic template .bashrc used to set up new accounts. If you don't want to use these modules, comment out the module load … statements in .bashrc and relogin. If a module load … is issued with a program but no version as shown below, the system-defined default version of that program module is used. module list will show the actual versions loaded.

$ cat ~/.bashrc
. /etc/profile.d/env-modules.sh
ulimit -s unlimited 2>/dev/null
ulimit -l unlimited 2>/dev/null 
#if using goto/mkl blas with mpi these should be set to 1 unless you want hybrid mpi/openmp
export GOTO_NUM_THREADS=1
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
#enter your modules here
module load intel
module load openmpi
module load mkl
[ -z "$PS1" ] && return
  PS1='`/bin/hostname -s`:`whoami`:`echo $PWD | sed "s=$HOME=="`$ '
  alias ls='ls --color=auto'
  module list

.bashrc is sourced at the beginning of each interactive job. There is a similar file .bash_profile sourced at the beginning of each non-interactive job. In our setup, we source .bashrc from .bash_profile so that the files are effectively the same, thus reducing the maintenance effort. Interactive or batch is determined in .bashrc by [ -z “$PS1” ] && return which drops out of the loop on batch runs, so commands following that are for interactive sessions only, like setting the value of the prompt $PS1. Commands towards the top of ``.bashrc`` are for both interactive and batch.

Here are some recommended module/.bashrc setups for different cases:

Your run only one program, or all the programs you run use the same modules, or use different modules that don't conflict

Put the module load … in .bashrc, above [ -z…. The same environment will be loaded from .bashrc for every interactive session, batch job, and MPI program if any. Many modules, for instance R and matlab and python, can be assumed to not conflict, though many combinations have not been tested. Modules that definitely do conflict are MPI modules, only one may be safely used at a time, and multiple versions of the same program, like gcc/4.7.1 and gcc/4.9.1.

You use different modules for different programs, but only run single-node batch jobs

If you put module load … in your .bashrc, every interactive session and batch and MPI thread will load that, which won't be good if it's the wrong MPI version. Delete or comment out (add leading “#” to) the module load … in .bashrc. Put the relevant module load … in your batch scripts after the #PBS statements, for example:

#PBS ...
#PBS -l node=1:ppn=12
module load R openmpi
cd $PBS_O_WORKDIR
R < myprog.R --nosave --vanilla

In this case, for interactive sessions such as compiling, type the relevant module load … in your session before work.

You use different modules for different programs, and also run multi-node jobs

This is the most difficult case and also common. The first two solutions won't always work. If a module is set in a batch script using multiple nodes, it definitely applies to the MPI threads running in the first or “master” compute node (always the lowest numbered assigned node in our batch configuration) but does not necessarily apply to the “slave” compute nodes, depending how different MPI versions issue remote threads. Multiple nodes imply MPI is being used, and the solution varies by MPI type:

* Intel MPI *

If you set the ``impi`` module at the top of your batch file, paths will be passed to slave nodes and a multiple-node job will run. If you want to set other environment variables, they won't be passed to slave nodes and must be set either (a) in .bashrc or (b) set in the mpirun statement, for instance the number of MKL threads below. Notice no equals sign in the assignment statement.

#PBS ...
#PBS -l node=2:ppn=12
module load intel/14.0.3 impi/5.0.0
cd $PBS_O_WORKDIR
mpirun -np 6 -machinefile $PBS_NODEFILE -genv MKL_NUM_THREADS 4  ./xhpl >logfile

* MVAPICH2 *

MVAPICH2 is similar to Intel MPI in that program paths will correctly pass to slave nodes, but other environment variables won't. Syntax varies by one letter and also no equals sign.

#PBS ...
#PBS -l node=2:ppn=12
module load intel/14.0.3 mvapich2/2.1
cd $PBS_O_WORKDIR
mpirun -np 6 -machinefile $PBS_NODEFILE -env MKL_NUM_THREADS 4  ./xhpl >logfile

* Open MPI *

Open MPI in the latest versions will correctly pass program paths except $LD_LIBRARY_PATH to slave compute nodes, so the -x LD_LIBRARY_PATH is required, and an optional variable MKL_NUM_THREADS is also set. Again syntax is slightly different, no equals and no value means pass the existing value, and =value means set and pass the value. This only works for Open MPI version 1.8.8 +. Earlier versions need the module set in .bashrc. Recent Open MPI versions are much faster, so programs using early versions should be recompiled anyway.

#PBS ...
#PBS -l node=2:ppn=12
module load intel/14.0.3 openmpi/1.8.8
cd $PBS_O_WORKDIR
mpirun -np 6 -machinefile $PBS_NODEFILE -x LD_LIBRARY_PATH -x MKL_NUM_THREADS=4  ./xhpl >logfile
modules.1447263498.txt.gz · Last modified: 2015/11/11 17:38 by root