Arkansas High Performace Computing Center [hpcwiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====Application Software====

Locating and using software has been made a little more complicated by some then reasonable decisions made 50 years ago for Unix: ''/usr/local'' for applications, environment variables ''$PATH'' to find an executable, ''$LD\_LIBRARY\_PATH'' to find dynamic link libraries, the file ''~/.cshrc'' to set up these variables.  These environment variables continue today in Linux and Mac, while Windows combines the two PATH variables.  Also important were software packages intended to be used as infrastructure for complete applications, and which simplifed development by not needing to be copied into the code of every project..  These "shared libraries" such as MPI and FFTW were specified by source code interfaces.  At the time, nearly everyone used one computer and one compiler, so a source interface corresponded directly to one binary interface.   Today there are many types of computers and compilers, and modern commercial applications often have a defined binary interface or "ABI" to avoid compatibility issues.

Today on HPC systems, the MPI implementation must be heavily customized for each site and its network fabric. Almost all multi-node programs depend on MPI, and there are three popular implementations of MPI (Open MPI, MVAPICH2, and Intel).  There are about six compilers that are reasonably popular (GNU/gcc, Intel proprietary, Intel open LLVM, NVidia/PGI, AMD LLVM,  open LLVM). MVAPICH2 and Intel MPI are binary compatible (thus a defacto ABI).  Probably all the LLVM implementations are reasonably binary compatible (with each other, not with Open MPI vs MVAPICH2), so there are about 8 to 12 binary versions of MPI, not considering updates for each that come out 2 or 3 times a year, and not considering the approximate doubling of GPU vs non-GPU support.  As will be seen in the demonstrations below,  many of the software packaging programs (OpenHPC, Spack, conda) use their own version of MPI.  MPI over shared memory is pretty well standardized, so we can usually expect this to work within one node.   Depending on how the applications were compiled and linked, sometimes we can intercept MPI library calls with our site-compiled MPI that can run on multiple nodes.
 
With thousands of applications (most of which have multiple versions), it's obviously impractical just from name collisions to put every executable in ''/usr/local/bin''. It's also impractical (unless you use only one application) to semipermanently set up these variables in ''~/.bashrc'' or ''~/.cshrc''. There are several ways to handle this.

===Modules===

Almost all HPC centers use "modules" software to help manage versioning. This was originally [[http://modules.sourcforge.net|Environment Modules]] and at most centers, including this one, has been replaced by an upward compatible rewrite [[https://lmod.readthedocs.io/en/latest/|Lmod]].  The primary use is to manage ''$PATH'',''$L\D_LIBRARY\_PATH'', and other environment variables over a large number of applications.  Unfortunately the name is easily confused with the unrelated packaged programs in modular languages such as Python and R:[[https://docs.python.org/3/tutorial/modules.html|Python modules]].

Module command syntax for most uses is relatively simple: ''load/unload'' to invoke/remove a module, ''purge'' to unload all modules, ''list'' to show loaded modules,  ''help'', and ''spider'' for searching.  We share some examples below for our three sources of software and module definitions "modulefiles".  There is a complete list of modulefiles in the text file ''/share/apps/modulelist'' which can be grepped.

==Locally written modulefiles==

There are currently about 660 locally written modulefiles, some of which have some "smart" capability to select from multiple software builds compiled for the computer loading the module. This shows the first 5:

<code>
$ grep "/share/apps/modulefiles" /share/apps/modulelist | head -5
/share/apps/modulefiles/ abinit/8.0.8b
/share/apps/modulefiles/ abinit/8.4.4
/share/apps/modulefiles/ abinit/8.6.1
/share/apps/modulefiles/ abinit/8.6.1-QFDcc
/share/apps/modulefiles/ abinit/8.6.1-qFD-trestles
</code>

==OpenHPC modulefiles==

At this writing we have 548 module files from the OpenHPC distribution. It concentrates on mathematical software.  Many of the packages are a bit dated, but when we update to Rocky 8 Linux we will be able to install some newer packages.

You can run particular packages like so. For an example, we'll try to find the newest version of ''petsc'' and the MPI and compiler package that it needs.  We then load those three and it auto-loads some prerequisites.  There is quite a bit of unnecessary duplication, as for instance, gnu7 and gnu8 binaries are very similar, so it wasn't really necessary to duplicate every module file.

<code>
$ grep petsc /share/apps/modulelist

/opt/ohpc/pub/moduledeps/ gnu7-impi/petsc/3.9.1
/opt/ohpc/pub/moduledeps/ gnu7-mpich/petsc/3.9.1
/opt/ohpc/pub/moduledeps/ gnu7-mvapich2/petsc/3.9.1
/opt/ohpc/pub/moduledeps/ gnu7-openmpi3/petsc/3.9.1
/opt/ohpc/pub/moduledeps/ gnu7-openmpi/petsc/3.8.3
/opt/ohpc/pub/moduledeps/ gnu8-impi/petsc/3.12.0
/opt/ohpc/pub/moduledeps/ gnu8-mpich/petsc/3.12.0
/opt/ohpc/pub/moduledeps/ gnu8-mvapich2/petsc/3.12.0
/opt/ohpc/pub/moduledeps/ gnu8-openmpi3/petsc/3.12.0
/opt/ohpc/pub/moduledeps/ gnu-impi/petsc/3.8.3
/opt/ohpc/pub/moduledeps/ gnu-mpich/petsc/3.8.3
/opt/ohpc/pub/moduledeps/ gnu-mvapich2/petsc/3.8.3
/opt/ohpc/pub/moduledeps/ gnu-openmpi/petsc/3.8.3
/opt/ohpc/pub/moduledeps/ intel-impi/petsc/3.12.0
/opt/ohpc/pub/moduledeps/ intel-mpich/petsc/3.12.0
/opt/ohpc/pub/moduledeps/ intel-mvapich2/petsc/3.12.0
/opt/ohpc/pub/moduledeps/ intel-openmpi3/petsc/3.12.0
/opt/ohpc/pub/moduledeps/ intel-openmpi/petsc/3.8.3
/share/apps/modulefiles/ petsc/3.10.5
/share/apps/modulefiles/ petsc/3.11.3
/share/apps/modulefiles/ petsc/3.14.2
/share/apps/modulefiles/ petsc/3.16.4
/share/apps/modulefiles/ petsc/3.8.58

$ module purge
$ module load ohpc gnu8 gnu8/openmpi3 gnu8-openmpi3/petsc

Lmod is automatically replacing "openmpi3/3.1.4" with "gnu8/openmpi3/3.1.4".

$ module list

Currently Loaded Modules:
  1) autotools   4) gnu8/8.3.0                   7) openblas/3.20-noomp
  2) prun/1.3    5) gnu8/openmpi3/3.1.4          8) gnu8-openmpi3/scalapack/2.0.2
  3) ohpc        6) gnu8-openmpi3/phdf5/1.10.5   9) gnu8-openmpi3/petsc/3.1
</code>

It is likely that this program will only work on one node since it links back to its internal MPI which is not customized for our network.  However, it should suffice for a test- or education-sized ''petsc'' run, while ''petsc'' and its prerequisites take some hours to install manually.

==Spack modulefiles==

There are currently 114 programs auto-installed using [[https://spack.readthedocs.io/en/latest/package_list.html|Spack]].  There are over 6600 Spack applications, so there is a lot of room for growth.  The build process is highly automated but may need to be duplicated for every node type in the cluster, which will take some time.  We have begun mostly with bioinformatics programs that will have fewer MPI complicataions.

<code>
$ grep spack /share/apps/modulelist | head -5
/share/apps/spackmodulefiles/ gcc-11.2.1/SKYLAKEX/abinit/9.6.1
/share/apps/spackmodulefiles/ gcc-11.2.1/SKYLAKEX/abyss/2.3.1
/share/apps/spackmodulefiles/ gcc-11.2.1/SKYLAKEX/atompaw/4.2.0.1
/share/apps/spackmodulefiles/ gcc-11.2.1/SKYLAKEX/bamtools/2.5.2
/share/apps/spackmodulefiles/ gcc-11.2.1/SKYLAKEX/bcftools/1.14
</code>

For testing we'll try to run abyss-pe in parallel similarly to  [[https://hpc.nih.gov/apps/abyss.html|Biowulf-abyss-example]].  The Spack module turns out to reference, but not to install, ''mpirun'' so we will add some local modules.  At this time, all our local Spack software is compiled with gcc/11.2.1 and openmpi/4.1.4 (though not our versions) so we will use those modules.  This does work, though we have not tested with multiple nodes.  Testing with our manually installed abyss 2.0.2 shows that multiple nodes doesn't work well anyway.
<code>
$ module load gcc-11.2.1/SKYLAKEX/abyss/2.3.1
$ abyss-pe np=32 j=8 k=25 n=10 in='*fq' name=OutputPrefix

mpirun -np 32 ABYSS-P -k25 -q3    --coverage-hist=coverage.hist -s OutputPrefix-bubbles.fa  -o OutputPrefix-1.fa *fq 
bash: mpirun: command not found
make: *** [OutputPrefix-1.fa] Error 127

$ module load gcc/11.2.1 openmpi/4.1.4
$ time abyss-pe np=32 j=8 k=25 n=10 in='*fq' name=OutputPrefix

/share/apps/mpi/openmpi-4.1.4/cuda/gcc/bin/mpirun -np 32 ABYSS-P -k25 -q3    --coverage-hist=coverage.hist -s OutputPrefix-bubbles.fa  -o OutputPrefix-1.fa *fq 
ABySS 2.3.1
ABYSS-P -k25 -q3 --coverage-hist=coverage.hist -s OutputPrefi-bubbles.fa -o OutputPrefi-1.fa Thalassiosira-weissflogii_AJA159-02_0ppt_r8_BS-440_trimmed_filtered_1.fq Thalassiosira-weissflogii_AJA159-02_0ppt_r8_BS-440_trimmed_filtered_2.fq
Running on 32 processors
4: Running on host c1411
.etc
...
abyss-fac   OutputPrefi-unitigs.fa OutputPrefi-contigs.fa OutputPrefi-scaffolds.fa |tee OutputPrefi-stats.tab
n	n:500	L50	min	N75	N50	N25	E-size	max	sum	name
211760	17866	4775	500	936	1470	2362	1912	17668	22.72e6	OutputPrefi-unitigs.fa
204672	15838	3757	500	1051	1927	3072	2431	17668	23.38e6	OutputPrefi-contigs.fa
204045	15400	3447	500	1057	2025	3400	2642	17668	23.38e6	OutputPrefi-scaffolds.fa
ln -sf OutputPrefi-stats.tab OutputPrefi-stats
tr '\t' , <OutputPrefi-stats.tab >OutputPrefi-stats.csv
abyss-tabtomd OutputPrefi-stats.tab >OutputPrefi-stats.md

real	36m33.072s
user	759m31.204s
sys	3m11.375s

</code>
This produces similar output and runs a bit faster than our manually installed abyss/2.0.2. 
As with OpenHPC, this can be an easy way to try out a program without a lot of installation time.

==EasyBuild==

[[https://docs.easybuild.io/en/latest/version-specific/Supported_software.html|EasyBuild]] is conceptually similar to Spack and has about 2700 applications.  We are beginning testing on it.

==conda==

[[conda|conda]] was originally a way to customize Python but has evolved to a package manager for other applications. It is popular in bioinformatics, which frequently requires very specific versioning of programs that are used together but not compiled together. It lacks an easy way to document what environments are currently installed by conda.  We are working on that. Here is an example of creating your own conda environment.  Run 

<code>
-bash-4.2$ /share/apps/bin/setupconda.sh
-bash-4.2$ ls -al .conda
total 16
drwxrwxr-x 4 ddn1 ddn1 4096 Sep 30 10:06 .
drwxr-xr-x 7 ddn1 ddn1 4096 Sep 30 10:05 ..
-rw-rw-r-- 1 ddn1 ddn1    0 Sep 30 10:10 environments.txt
drwxrwxr-x 2 ddn1 ddn1 4096 Sep 30 10:07 envs
drwxrwxr-x 3 ddn1 ddn1 4096 Sep 30 10:07 pkgs
-bash-4.2$ module load gcc/8.3.1 mkl/19.0.5 python/3.10-anaconda
-bash-4.2$ source /share/apps/bin/conda-3.10.sh;conda create -n testenv
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/ddn1/.conda/envs/teste

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate testenv
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Retrieving notices: ...working... done
(base) -bash-4.2$ conda activate testenv
(testenv) -bash-4.2$ conda install pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/ddn1/.conda/envs/testenv

  added / updated specs:
    - pytorch


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _libgcc_mutex-0.1          |             main           3 KB
    _openmp_mutex-5.1          |            1_gnu          21 KB
    blas-1.0                   |              mkl           6 KB
    bzip2-1.0.8                |       h7b6447c_0          78 KB
    ca-certificates-2022.07.19 |       h06a4308_0         124 KB
    certifi-2022.9.14          |  py310h06a4308_0         155 KB
    cffi-1.15.1                |  py310h74dc2b5_0         407 KB
    future-0.18.2              |  py310h06a4308_1         647 KB
    intel-openmp-2021.4.0      |    h06a4308_3561         4.2 MB
    ld_impl_linux-64-2.38      |       h1181459_1         654 KB
    libffi-3.3                 |       he6710b0_2          50 KB
    libgcc-ng-11.2.0           |       h1234567_1         5.3 MB
    libgomp-11.2.0             |       h1234567_1         474 KB
    libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB
    libuuid-1.0.3              |       h7f8727e_2          17 KB
    mkl-2021.4.0               |     h06a4308_640       142.6 MB
    mkl-service-2.4.0          |  py310h7f8727e_0         177 KB
    mkl_fft-1.3.1              |  py310hd6ae3a3_0         567 KB
    mkl_random-1.2.2           |  py310h00e6091_0        1009 KB
    ncurses-6.3                |       h5eee18b_3         781 KB
    ninja-1.10.2               |       h06a4308_5           8 KB
    ninja-base-1.10.2          |       hd09550d_5         109 KB
    numpy-1.23.1               |  py310h1794996_0          11 KB
    numpy-base-1.23.1          |  py310hcba007f_0        14.3 MB
    openssl-1.1.1q             |       h7f8727e_0         2.5 MB
    pip-22.1.2                 |  py310h06a4308_0         2.5 MB
    pycparser-2.21             |     pyhd3eb1b0_0          94 KB
    python-3.10.4              |       h12debd9_0        24.2 MB
    pytorch-1.10.2             |cpu_py310h6894f24_0        43.9 MB
    readline-8.1.2             |       h7f8727e_1         354 KB
    setuptools-63.4.1          |  py310h06a4308_0         1.1 MB
    six-1.16.0                 |     pyhd3eb1b0_1          18 KB
    sqlite-3.39.3              |       h5082296_0         1.1 MB
    tk-8.6.12                  |       h1ccaba5_0         3.0 MB
    typing-extensions-4.3.0    |  py310h06a4308_0           9 KB
    typing_extensions-4.3.0    |  py310h06a4308_0          42 KB
    tzdata-2022c               |       h04d1e81_0         107 KB
    wheel-0.37.1               |     pyhd3eb1b0_0          33 KB
    xz-5.2.6                   |       h5eee18b_0         394 KB
    zlib-1.2.12                |       h5eee18b_3         103 KB
    ------------------------------------------------------------
                                           Total:       255.7 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
  blas               pkgs/main/linux-64::blas-1.0-mkl
  bzip2              pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0
  ca-certificates    pkgs/main/linux-64::ca-certificates-2022.07.19-h06a4308_0
  certifi            pkgs/main/linux-64::certifi-2022.9.14-py310h06a4308_0
  cffi               pkgs/main/linux-64::cffi-1.15.1-py310h74dc2b5_0
  future             pkgs/main/linux-64::future-0.18.2-py310h06a4308_1
  intel-openmp       pkgs/main/linux-64::intel-openmp-2021.4.0-h06a4308_3561
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1
  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1
  libuuid            pkgs/main/linux-64::libuuid-1.0.3-h7f8727e_2
  mkl                pkgs/main/linux-64::mkl-2021.4.0-h06a4308_640
  mkl-service        pkgs/main/linux-64::mkl-service-2.4.0-py310h7f8727e_0
  mkl_fft            pkgs/main/linux-64::mkl_fft-1.3.1-py310hd6ae3a3_0
  mkl_random         pkgs/main/linux-64::mkl_random-1.2.2-py310h00e6091_0
  ncurses            pkgs/main/linux-64::ncurses-6.3-h5eee18b_3
  ninja              pkgs/main/linux-64::ninja-1.10.2-h06a4308_5
  ninja-base         pkgs/main/linux-64::ninja-base-1.10.2-hd09550d_5
  numpy              pkgs/main/linux-64::numpy-1.23.1-py310h1794996_0
  numpy-base         pkgs/main/linux-64::numpy-base-1.23.1-py310hcba007f_0
  openssl            pkgs/main/linux-64::openssl-1.1.1q-h7f8727e_0
  pip                pkgs/main/linux-64::pip-22.1.2-py310h06a4308_0
  pycparser          pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0
  python             pkgs/main/linux-64::python-3.10.4-h12debd9_0
  pytorch            pkgs/main/linux-64::pytorch-1.10.2-cpu_py310h6894f24_0
  readline           pkgs/main/linux-64::readline-8.1.2-h7f8727e_1
  setuptools         pkgs/main/linux-64::setuptools-63.4.1-py310h06a4308_0
  six                pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1
  sqlite             pkgs/main/linux-64::sqlite-3.39.3-h5082296_0
  tk                 pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0
  typing-extensions  pkgs/main/linux-64::typing-extensions-4.3.0-py310h06a4308_0
  typing_extensions  pkgs/main/linux-64::typing_extensions-4.3.0-py310h06a4308_0
  tzdata             pkgs/main/noarch::tzdata-2022c-h04d1e81_0
  wheel              pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0
  xz                 pkgs/main/linux-64::xz-5.2.6-h5eee18b_0
  zlib               pkgs/main/linux-64::zlib-1.2.12-h5eee18b_3


Proceed ([y]/n)? y


Downloading and Extracting Packages
ninja-1.10.2         | 8 KB      | ###################################################################################### | 100% 
python-3.10.4        | 24.2 MB   | ###################################################################################### | 100% 
libgomp-11.2.0       | 474 KB    | ###################################################################################### | 100% 
xz-5.2.6             | 394 KB    | ###################################################################################### | 100% 
numpy-1.23.1         | 11 KB     | ###################################################################################### | 100% 
typing-extensions-4. | 9 KB      | ###################################################################################### | 100% 
bzip2-1.0.8          | 78 KB     | ###################################################################################### | 100% 
ninja-base-1.10.2    | 109 KB    | ###################################################################################### | 100% 
wheel-0.37.1         | 33 KB     | ###################################################################################### | 100% 
setuptools-63.4.1    | 1.1 MB    | ###################################################################################### | 100% 
blas-1.0             | 6 KB      | ###################################################################################### | 100% 
pycparser-2.21       | 94 KB     | ###################################################################################### | 100% 
libffi-3.3           | 50 KB     | ###################################################################################### | 100% 
future-0.18.2        | 647 KB    | ###################################################################################### | 100% 
mkl-service-2.4.0    | 177 KB    | ###################################################################################### | 100% 
libgcc-ng-11.2.0     | 5.3 MB    | ###################################################################################### | 100% 
libstdcxx-ng-11.2.0  | 4.7 MB    | ###################################################################################### | 100% 
cffi-1.15.1          | 407 KB    | ###################################################################################### | 100% 
intel-openmp-2021.4. | 4.2 MB    | ###################################################################################### | 100% 
six-1.16.0           | 18 KB     | ###################################################################################### | 100% 
pip-22.1.2           | 2.5 MB    | ###################################################################################### | 100% 
ncurses-6.3          | 781 KB    | ###################################################################################### | 100% 
libuuid-1.0.3        | 17 KB     | ###################################################################################### | 100% 
readline-8.1.2       | 354 KB    | ###################################################################################### | 100% 
mkl-2021.4.0         | 142.6 MB  | ###################################################################################### | 100% 
sqlite-3.39.3        | 1.1 MB    | ###################################################################################### | 100% 
mkl_fft-1.3.1        | 567 KB    | ###################################################################################### | 100% 
tzdata-2022c         | 107 KB    | ###################################################################################### | 100% 
_libgcc_mutex-0.1    | 3 KB      | ###################################################################################### | 100% 
typing_extensions-4. | 42 KB     | ###################################################################################### | 100% 
_openmp_mutex-5.1    | 21 KB     | ###################################################################################### | 100% 
tk-8.6.12            | 3.0 MB    | ###################################################################################### | 100% 
mkl_random-1.2.2     | 1009 KB   | ###################################################################################### | 100% 
ld_impl_linux-64-2.3 | 654 KB    | ###################################################################################### | 100% 
zlib-1.2.12          | 103 KB    | ###################################################################################### | 100% 
ca-certificates-2022 | 124 KB    | ###################################################################################### | 100% 
numpy-base-1.23.1    | 14.3 MB   | ###################################################################################### | 100% 
openssl-1.1.1q       | 2.5 MB    | ###################################################################################### | 100% 
certifi-2022.9.14    | 155 KB    | ###################################################################################### | 100% 
pytorch-1.10.2       | 43.9 MB   | ###################################################################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Retrieving notices: ...working... done
(testenv) -bash-4.2$ cat .conda/environments.txt
/home/ddn1/.conda/envs/testenv
(testenv) -bash-4.2$ python
Python 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as  np
>>> exit()
(testenv) -bash-4.2$ 
</code>


===Compiler-MPI Recommendations===

Most parallel programs require the selection of a compiler and an MPI version.  We usually recommend the following compiler versions (select only one, usually, with exceptions noted below):

<code>
$ module load gcc/11.2.1    
#synonym gnu also works, latest gnu compiler from "Centos 7 Development Tools", enables gcc/g++/gfortran

$ module load intel/21.2.0
#synonym intelcompiler also works, both intel proprietary icc/icpc/ifort and intel llvm icx/icpx/ifx

$ module load nvhpc/22.7
#synonym PGI also works, Nvidia/PGI compiler equally nvc/nvc++/nvfortran and pgcc/pgc++/pgf77/pgf90/pgf95/pgfortran

$ module load aocc/3.0
#AMD llvm compiler clang/clang++/flang
</code>

If you don't load any modules, there are some very old compilers built into Centos and in the default path /usr/bin:
<code>
$ gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)

$ clang -v
clang version 3.4.2 (tags/RELEASE_34/dot2-final)
Target: x86_64-redhat-linux-gnu
Thread model: posix
Found candidate GCC installation: /bin/../lib/gcc/x86_64-redhat-linux/4.8.2
Found candidate GCC installation: /bin/../lib/gcc/x86_64-redhat-linux/4.8.5
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.2
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.5
Selected GCC installation: /bin/../lib/gcc/x86_64-redhat-linux/4.8.5
</code>

These compilers aren't recommended for compiling, but probably suffice for running non-parallel applications compiled with newer gcc, as they are likely binary compatible.

We recommend the following MPI versions.  Definitely select only one (at runtime mvapich2 and impi should be equivalent, but not at compile time):

<code>
openmpi/4.1.4
#with gcc, intel, nvhpc

mvapich2/2.3.7
#with gcc, intel

impi/17.0.7
#with gcc, intel
</code>

In combination we recommend (as compiler, then mpi in order so that the correct libraries are loaded).

<code>
$ module load { gcc/11.2.1 | intel/21.2.0 | nvhpc/22.7 } openmpi/4.1.4

$ module load { gcc/11.2.1 | intel/21.2.0 } mvapich2/2.3.7

$ module load { gcc/11.2.1 | intel/21.2.0 } impi/17.0.7
</code>

There are a couple of situations where you would want multiple compilers loaded (But first compiler-MPI version that is ''module load''ed will determine the MPI code that is loaded).

(1) Most c++ compilers use the gnu c++ include libraries. For a program (''LAMMPS'' is one) that uses a lot of relatively recent c++, you will want a recent gcc to provide those libraries.

This works with the Intel proprietary ''icpc'' compiler

<code>
$ module load intel/17.0.7 openmpi/4.1.4 gcc/11.2.1
</code>

If you don't add the third module, ''icpc'' will use the libraries from the default Centos ''g++ 4.8.5'' which is quite old and probably can't compile LAMMPS at all.

(2) ''llvm'' compilers (aocc/3.0.0 and intel/21.2.0 icx) try to auto-find ''g++'' libraries but don't do it quite correctly.

<code>
$ module load aocc/3.0.0
$ clang++ -v
AMD clang version 12.0.0 (CLANG: AOCC_3.0.0-Build#78 2020_12_10) (based on LLVM Mirror.Version.12.0.0)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/AMD/aocc-compiler-3.0.0/bin
Found candidate GCC installation: /opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7
Found candidate GCC installation: /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8
Found candidate GCC installation: /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.2
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.5
Selected GCC installation: /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9
</code>

so it picks devtoolset-9 libraries in spite of devtoolset-10 and 11 being available

<code>
$ ls /opt/rh
devtoolset-10  devtoolset-11  devtoolset-3  devtoolset-7  devtoolset-8  devtoolset-9  
</code>

if devtoolset-9 (gcc/9.3.1) is new enough, then that's ok.

(3) Sometimes Intel MKL will link back to the Intel compiler when using Intel OMP instead of GNU OMP. This should work:

<code>
$ module load gcc/11.2.1 mkl/20.0.4 openmpi/4.1.4 intel/17.0.7
</code>
Arkansas High Performace Computing Center [hpcwiki]

User Tools

Site Tools

Page Tools
