Locating and using software has been made a little more complicated by some then reasonable decisions made 50 years ago for Unix: /usr/local for applications, environment variables $PATH to find an executable, $LD_LIBRARY_PATH to find dynamic link libraries, the file ~/.cshrc to set up these variables. These environment variables continue today in Linux and Mac, while Windows combines the two PATH variables. Also important were software packages intended to be used as infrastructure for complete applications, and which simplifed development by not needing to be copied into the code of every project.. These “shared libraries” such as MPI and FFTW were specified by source code interfaces. At the time, nearly everyone used one computer and one compiler, so a source interface corresponded directly to one binary interface. Today there are many types of computers and compilers, and modern commercial applications often have a defined binary interface or “ABI” to avoid compatibility issues.
Today on HPC systems, the MPI implementation must be heavily customized for each site and its network fabric. Almost all multi-node programs depend on MPI, and there are three popular implementations of MPI (Open MPI, MVAPICH2, and Intel). There are about six compilers that are reasonably popular (GNU/gcc, Intel proprietary, Intel open LLVM, NVidia/PGI, AMD LLVM, open LLVM). MVAPICH2 and Intel MPI are binary compatible (thus a defacto ABI). Probably all the LLVM implementations are reasonably binary compatible (with each other, not with Open MPI vs MVAPICH2), so there are about 8 to 12 binary versions of MPI, not considering updates for each that come out 2 or 3 times a year, and not considering the approximate doubling of GPU vs non-GPU support. As will be seen in the demonstrations below, many of the software packaging programs (OpenHPC, Spack, conda) use their own version of MPI. MPI over shared memory is pretty well standardized, so we can usually expect this to work within one node. Depending on how the applications were compiled and linked, sometimes we can intercept MPI library calls with our site-compiled MPI that can run on multiple nodes.
With thousands of applications (most of which have multiple versions), it's obviously impractical just from name collisions to put every executable in /usr/local/bin. It's also impractical (unless you use only one application) to semipermanently set up these variables in ~/.bashrc or ~/.cshrc. There are several ways to handle this.
Almost all HPC centers use “modules” software to help manage versioning. This was originally Environment Modules and at most centers, including this one, has been replaced by an upward compatible rewrite Lmod. The primary use is to manage $PATH,$L\D_LIBRARY_PATH, and other environment variables over a large number of applications. Unfortunately the name is easily confused with the unrelated packaged programs in modular languages such as Python and R:Python modules.
Module command syntax for most uses is relatively simple: load/unload to invoke/remove a module, purge to unload all modules, list to show loaded modules, help, and spider for searching. We share some examples below for our three sources of software and module definitions “modulefiles”. There is a complete list of modulefiles in the text file /share/apps/modulelist which can be grepped.
There are currently about 660 locally written modulefiles, some of which have some “smart” capability to select from multiple software builds compiled for the computer loading the module. This shows the first 5:
$ grep "/share/apps/modulefiles" /share/apps/modulelist | head -5 /share/apps/modulefiles/ abinit/8.0.8b /share/apps/modulefiles/ abinit/8.4.4 /share/apps/modulefiles/ abinit/8.6.1 /share/apps/modulefiles/ abinit/8.6.1-QFDcc /share/apps/modulefiles/ abinit/8.6.1-qFD-trestles
At this writing we have 548 module files from the OpenHPC distribution. It concentrates on mathematical software. Many of the packages are a bit dated, but when we update to Rocky 8 Linux we will be able to install some newer packages.
You can run particular packages like so. For an example, we'll try to find the newest version of petsc and the MPI and compiler package that it needs. We then load those three and it auto-loads some prerequisites. There is quite a bit of unnecessary duplication, as for instance, gnu7 and gnu8 binaries are very similar, so it wasn't really necessary to duplicate every module file.
$ grep petsc /share/apps/modulelist /opt/ohpc/pub/moduledeps/ gnu7-impi/petsc/3.9.1 /opt/ohpc/pub/moduledeps/ gnu7-mpich/petsc/3.9.1 /opt/ohpc/pub/moduledeps/ gnu7-mvapich2/petsc/3.9.1 /opt/ohpc/pub/moduledeps/ gnu7-openmpi3/petsc/3.9.1 /opt/ohpc/pub/moduledeps/ gnu7-openmpi/petsc/3.8.3 /opt/ohpc/pub/moduledeps/ gnu8-impi/petsc/3.12.0 /opt/ohpc/pub/moduledeps/ gnu8-mpich/petsc/3.12.0 /opt/ohpc/pub/moduledeps/ gnu8-mvapich2/petsc/3.12.0 /opt/ohpc/pub/moduledeps/ gnu8-openmpi3/petsc/3.12.0 /opt/ohpc/pub/moduledeps/ gnu-impi/petsc/3.8.3 /opt/ohpc/pub/moduledeps/ gnu-mpich/petsc/3.8.3 /opt/ohpc/pub/moduledeps/ gnu-mvapich2/petsc/3.8.3 /opt/ohpc/pub/moduledeps/ gnu-openmpi/petsc/3.8.3 /opt/ohpc/pub/moduledeps/ intel-impi/petsc/3.12.0 /opt/ohpc/pub/moduledeps/ intel-mpich/petsc/3.12.0 /opt/ohpc/pub/moduledeps/ intel-mvapich2/petsc/3.12.0 /opt/ohpc/pub/moduledeps/ intel-openmpi3/petsc/3.12.0 /opt/ohpc/pub/moduledeps/ intel-openmpi/petsc/3.8.3 /share/apps/modulefiles/ petsc/3.10.5 /share/apps/modulefiles/ petsc/3.11.3 /share/apps/modulefiles/ petsc/3.14.2 /share/apps/modulefiles/ petsc/3.16.4 /share/apps/modulefiles/ petsc/3.8.58 $ module purge $ module load ohpc gnu8 gnu8/openmpi3 gnu8-openmpi3/petsc Lmod is automatically replacing "openmpi3/3.1.4" with "gnu8/openmpi3/3.1.4". $ module list Currently Loaded Modules: 1) autotools 4) gnu8/8.3.0 7) openblas/3.20-noomp 2) prun/1.3 5) gnu8/openmpi3/3.1.4 8) gnu8-openmpi3/scalapack/2.0.2 3) ohpc 6) gnu8-openmpi3/phdf5/1.10.5 9) gnu8-openmpi3/petsc/3.1
It is likely that this program will only work on one node since it links back to its internal MPI which is not customized for our network. However, it should suffice for a test- or education-sized petsc run, while petsc and its prerequisites take some hours to install manually.
There are currently 114 programs auto-installed using Spack. There are over 6600 Spack applications, so there is a lot of room for growth. The build process is highly automated but may need to be duplicated for every node type in the cluster, which will take some time. We have begun mostly with bioinformatics programs that will have fewer MPI complicataions.
$ grep spack /share/apps/modulelist | head -5 /share/apps/spackmodulefiles/ gcc-11.2.1/SKYLAKEX/abinit/9.6.1 /share/apps/spackmodulefiles/ gcc-11.2.1/SKYLAKEX/abyss/2.3.1 /share/apps/spackmodulefiles/ gcc-11.2.1/SKYLAKEX/atompaw/4.2.0.1 /share/apps/spackmodulefiles/ gcc-11.2.1/SKYLAKEX/bamtools/2.5.2 /share/apps/spackmodulefiles/ gcc-11.2.1/SKYLAKEX/bcftools/1.14
For testing we'll try to run abyss-pe in parallel similarly to Biowulf-abyss-example. The Spack module turns out to reference, but not to install, mpirun so we will add some local modules. At this time, all our local Spack software is compiled with gcc/11.2.1 and openmpi/4.1.4 (though not our versions) so we will use those modules. This does work, though we have not tested with multiple nodes. Testing with our manually installed abyss 2.0.2 shows that multiple nodes doesn't work well anyway.
$ module load gcc-11.2.1/SKYLAKEX/abyss/2.3.1 $ abyss-pe np=32 j=8 k=25 n=10 in='*fq' name=OutputPrefix mpirun -np 32 ABYSS-P -k25 -q3 --coverage-hist=coverage.hist -s OutputPrefix-bubbles.fa -o OutputPrefix-1.fa *fq bash: mpirun: command not found make: *** [OutputPrefix-1.fa] Error 127 $ module load gcc/11.2.1 openmpi/4.1.4 $ time abyss-pe np=32 j=8 k=25 n=10 in='*fq' name=OutputPrefix /share/apps/mpi/openmpi-4.1.4/cuda/gcc/bin/mpirun -np 32 ABYSS-P -k25 -q3 --coverage-hist=coverage.hist -s OutputPrefix-bubbles.fa -o OutputPrefix-1.fa *fq ABySS 2.3.1 ABYSS-P -k25 -q3 --coverage-hist=coverage.hist -s OutputPrefi-bubbles.fa -o OutputPrefi-1.fa Thalassiosira-weissflogii_AJA159-02_0ppt_r8_BS-440_trimmed_filtered_1.fq Thalassiosira-weissflogii_AJA159-02_0ppt_r8_BS-440_trimmed_filtered_2.fq Running on 32 processors 4: Running on host c1411 .etc ... abyss-fac OutputPrefi-unitigs.fa OutputPrefi-contigs.fa OutputPrefi-scaffolds.fa |tee OutputPrefi-stats.tab n n:500 L50 min N75 N50 N25 E-size max sum name 211760 17866 4775 500 936 1470 2362 1912 17668 22.72e6 OutputPrefi-unitigs.fa 204672 15838 3757 500 1051 1927 3072 2431 17668 23.38e6 OutputPrefi-contigs.fa 204045 15400 3447 500 1057 2025 3400 2642 17668 23.38e6 OutputPrefi-scaffolds.fa ln -sf OutputPrefi-stats.tab OutputPrefi-stats tr '\t' , <OutputPrefi-stats.tab >OutputPrefi-stats.csv abyss-tabtomd OutputPrefi-stats.tab >OutputPrefi-stats.md real 36m33.072s user 759m31.204s sys 3m11.375s
This produces similar output and runs a bit faster than our manually installed abyss/2.0.2. As with OpenHPC, this can be an easy way to try out a program without a lot of installation time.
EasyBuild is conceptually similar to Spack and has about 2700 applications. We are beginning testing on it.
conda was originally a way to customize Python but has evolved to a package manager for other applications. It is popular in bioinformatics, which frequently requires very specific versioning of programs that are used together but not compiled together. It lacks an easy way to document what environments are currently installed by conda. We are working on that. Here is an example of creating your own conda environment. Run
-bash-4.2$ /share/apps/bin/setupconda.sh
-bash-4.2$ ls -al .conda
total 16
drwxrwxr-x 4 ddn1 ddn1 4096 Sep 30 10:06 .
drwxr-xr-x 7 ddn1 ddn1 4096 Sep 30 10:05 ..
-rw-rw-r-- 1 ddn1 ddn1 0 Sep 30 10:10 environments.txt
drwxrwxr-x 2 ddn1 ddn1 4096 Sep 30 10:07 envs
drwxrwxr-x 3 ddn1 ddn1 4096 Sep 30 10:07 pkgs
-bash-4.2$ module load gcc/8.3.1 mkl/19.0.5 python/3.10-anaconda
-bash-4.2$ source /share/apps/bin/conda-3.10.sh;conda create -n testenv
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /home/ddn1/.conda/envs/teste
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate testenv
#
# To deactivate an active environment, use
#
# $ conda deactivate
Retrieving notices: ...working... done
(base) -bash-4.2$ conda activate testenv
(testenv) -bash-4.2$ conda install pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /home/ddn1/.conda/envs/testenv
added / updated specs:
- pytorch
The following packages will be downloaded:
package | build
---------------------------|-----------------
_libgcc_mutex-0.1 | main 3 KB
_openmp_mutex-5.1 | 1_gnu 21 KB
blas-1.0 | mkl 6 KB
bzip2-1.0.8 | h7b6447c_0 78 KB
ca-certificates-2022.07.19 | h06a4308_0 124 KB
certifi-2022.9.14 | py310h06a4308_0 155 KB
cffi-1.15.1 | py310h74dc2b5_0 407 KB
future-0.18.2 | py310h06a4308_1 647 KB
intel-openmp-2021.4.0 | h06a4308_3561 4.2 MB
ld_impl_linux-64-2.38 | h1181459_1 654 KB
libffi-3.3 | he6710b0_2 50 KB
libgcc-ng-11.2.0 | h1234567_1 5.3 MB
libgomp-11.2.0 | h1234567_1 474 KB
libstdcxx-ng-11.2.0 | h1234567_1 4.7 MB
libuuid-1.0.3 | h7f8727e_2 17 KB
mkl-2021.4.0 | h06a4308_640 142.6 MB
mkl-service-2.4.0 | py310h7f8727e_0 177 KB
mkl_fft-1.3.1 | py310hd6ae3a3_0 567 KB
mkl_random-1.2.2 | py310h00e6091_0 1009 KB
ncurses-6.3 | h5eee18b_3 781 KB
ninja-1.10.2 | h06a4308_5 8 KB
ninja-base-1.10.2 | hd09550d_5 109 KB
numpy-1.23.1 | py310h1794996_0 11 KB
numpy-base-1.23.1 | py310hcba007f_0 14.3 MB
openssl-1.1.1q | h7f8727e_0 2.5 MB
pip-22.1.2 | py310h06a4308_0 2.5 MB
pycparser-2.21 | pyhd3eb1b0_0 94 KB
python-3.10.4 | h12debd9_0 24.2 MB
pytorch-1.10.2 |cpu_py310h6894f24_0 43.9 MB
readline-8.1.2 | h7f8727e_1 354 KB
setuptools-63.4.1 | py310h06a4308_0 1.1 MB
six-1.16.0 | pyhd3eb1b0_1 18 KB
sqlite-3.39.3 | h5082296_0 1.1 MB
tk-8.6.12 | h1ccaba5_0 3.0 MB
typing-extensions-4.3.0 | py310h06a4308_0 9 KB
typing_extensions-4.3.0 | py310h06a4308_0 42 KB
tzdata-2022c | h04d1e81_0 107 KB
wheel-0.37.1 | pyhd3eb1b0_0 33 KB
xz-5.2.6 | h5eee18b_0 394 KB
zlib-1.2.12 | h5eee18b_3 103 KB
------------------------------------------------------------
Total: 255.7 MB
The following NEW packages will be INSTALLED:
_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
_openmp_mutex pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
blas pkgs/main/linux-64::blas-1.0-mkl
bzip2 pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0
ca-certificates pkgs/main/linux-64::ca-certificates-2022.07.19-h06a4308_0
certifi pkgs/main/linux-64::certifi-2022.9.14-py310h06a4308_0
cffi pkgs/main/linux-64::cffi-1.15.1-py310h74dc2b5_0
future pkgs/main/linux-64::future-0.18.2-py310h06a4308_1
intel-openmp pkgs/main/linux-64::intel-openmp-2021.4.0-h06a4308_3561
ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1
libffi pkgs/main/linux-64::libffi-3.3-he6710b0_2
libgcc-ng pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1
libgomp pkgs/main/linux-64::libgomp-11.2.0-h1234567_1
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1
libuuid pkgs/main/linux-64::libuuid-1.0.3-h7f8727e_2
mkl pkgs/main/linux-64::mkl-2021.4.0-h06a4308_640
mkl-service pkgs/main/linux-64::mkl-service-2.4.0-py310h7f8727e_0
mkl_fft pkgs/main/linux-64::mkl_fft-1.3.1-py310hd6ae3a3_0
mkl_random pkgs/main/linux-64::mkl_random-1.2.2-py310h00e6091_0
ncurses pkgs/main/linux-64::ncurses-6.3-h5eee18b_3
ninja pkgs/main/linux-64::ninja-1.10.2-h06a4308_5
ninja-base pkgs/main/linux-64::ninja-base-1.10.2-hd09550d_5
numpy pkgs/main/linux-64::numpy-1.23.1-py310h1794996_0
numpy-base pkgs/main/linux-64::numpy-base-1.23.1-py310hcba007f_0
openssl pkgs/main/linux-64::openssl-1.1.1q-h7f8727e_0
pip pkgs/main/linux-64::pip-22.1.2-py310h06a4308_0
pycparser pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0
python pkgs/main/linux-64::python-3.10.4-h12debd9_0
pytorch pkgs/main/linux-64::pytorch-1.10.2-cpu_py310h6894f24_0
readline pkgs/main/linux-64::readline-8.1.2-h7f8727e_1
setuptools pkgs/main/linux-64::setuptools-63.4.1-py310h06a4308_0
six pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1
sqlite pkgs/main/linux-64::sqlite-3.39.3-h5082296_0
tk pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0
typing-extensions pkgs/main/linux-64::typing-extensions-4.3.0-py310h06a4308_0
typing_extensions pkgs/main/linux-64::typing_extensions-4.3.0-py310h06a4308_0
tzdata pkgs/main/noarch::tzdata-2022c-h04d1e81_0
wheel pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0
xz pkgs/main/linux-64::xz-5.2.6-h5eee18b_0
zlib pkgs/main/linux-64::zlib-1.2.12-h5eee18b_3
Proceed ([y]/n)? y
Downloading and Extracting Packages
ninja-1.10.2 | 8 KB | ###################################################################################### | 100%
python-3.10.4 | 24.2 MB | ###################################################################################### | 100%
libgomp-11.2.0 | 474 KB | ###################################################################################### | 100%
xz-5.2.6 | 394 KB | ###################################################################################### | 100%
numpy-1.23.1 | 11 KB | ###################################################################################### | 100%
typing-extensions-4. | 9 KB | ###################################################################################### | 100%
bzip2-1.0.8 | 78 KB | ###################################################################################### | 100%
ninja-base-1.10.2 | 109 KB | ###################################################################################### | 100%
wheel-0.37.1 | 33 KB | ###################################################################################### | 100%
setuptools-63.4.1 | 1.1 MB | ###################################################################################### | 100%
blas-1.0 | 6 KB | ###################################################################################### | 100%
pycparser-2.21 | 94 KB | ###################################################################################### | 100%
libffi-3.3 | 50 KB | ###################################################################################### | 100%
future-0.18.2 | 647 KB | ###################################################################################### | 100%
mkl-service-2.4.0 | 177 KB | ###################################################################################### | 100%
libgcc-ng-11.2.0 | 5.3 MB | ###################################################################################### | 100%
libstdcxx-ng-11.2.0 | 4.7 MB | ###################################################################################### | 100%
cffi-1.15.1 | 407 KB | ###################################################################################### | 100%
intel-openmp-2021.4. | 4.2 MB | ###################################################################################### | 100%
six-1.16.0 | 18 KB | ###################################################################################### | 100%
pip-22.1.2 | 2.5 MB | ###################################################################################### | 100%
ncurses-6.3 | 781 KB | ###################################################################################### | 100%
libuuid-1.0.3 | 17 KB | ###################################################################################### | 100%
readline-8.1.2 | 354 KB | ###################################################################################### | 100%
mkl-2021.4.0 | 142.6 MB | ###################################################################################### | 100%
sqlite-3.39.3 | 1.1 MB | ###################################################################################### | 100%
mkl_fft-1.3.1 | 567 KB | ###################################################################################### | 100%
tzdata-2022c | 107 KB | ###################################################################################### | 100%
_libgcc_mutex-0.1 | 3 KB | ###################################################################################### | 100%
typing_extensions-4. | 42 KB | ###################################################################################### | 100%
_openmp_mutex-5.1 | 21 KB | ###################################################################################### | 100%
tk-8.6.12 | 3.0 MB | ###################################################################################### | 100%
mkl_random-1.2.2 | 1009 KB | ###################################################################################### | 100%
ld_impl_linux-64-2.3 | 654 KB | ###################################################################################### | 100%
zlib-1.2.12 | 103 KB | ###################################################################################### | 100%
ca-certificates-2022 | 124 KB | ###################################################################################### | 100%
numpy-base-1.23.1 | 14.3 MB | ###################################################################################### | 100%
openssl-1.1.1q | 2.5 MB | ###################################################################################### | 100%
certifi-2022.9.14 | 155 KB | ###################################################################################### | 100%
pytorch-1.10.2 | 43.9 MB | ###################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Retrieving notices: ...working... done
(testenv) -bash-4.2$ cat .conda/environments.txt
/home/ddn1/.conda/envs/testenv
(testenv) -bash-4.2$ python
Python 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> exit()
(testenv) -bash-4.2$
Most parallel programs require the selection of a compiler and an MPI version. We usually recommend the following compiler versions (select only one, usually, with exceptions noted below):
$ module load gcc/11.2.1 #synonym gnu also works, latest gnu compiler from "Centos 7 Development Tools", enables gcc/g++/gfortran $ module load intel/21.2.0 #synonym intelcompiler also works, both intel proprietary icc/icpc/ifort and intel llvm icx/icpx/ifx $ module load nvhpc/22.7 #synonym PGI also works, Nvidia/PGI compiler equally nvc/nvc++/nvfortran and pgcc/pgc++/pgf77/pgf90/pgf95/pgfortran $ module load aocc/3.0 #AMD llvm compiler clang/clang++/flang
If you don't load any modules, there are some very old compilers built into Centos and in the default path /usr/bin:
$ gcc --version gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44) $ clang -v clang version 3.4.2 (tags/RELEASE_34/dot2-final) Target: x86_64-redhat-linux-gnu Thread model: posix Found candidate GCC installation: /bin/../lib/gcc/x86_64-redhat-linux/4.8.2 Found candidate GCC installation: /bin/../lib/gcc/x86_64-redhat-linux/4.8.5 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.2 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.5 Selected GCC installation: /bin/../lib/gcc/x86_64-redhat-linux/4.8.5
These compilers aren't recommended for compiling, but probably suffice for running non-parallel applications compiled with newer gcc, as they are likely binary compatible.
We recommend the following MPI versions. Definitely select only one (at runtime mvapich2 and impi should be equivalent, but not at compile time):
openmpi/4.1.4 #with gcc, intel, nvhpc mvapich2/2.3.7 #with gcc, intel impi/17.0.7 #with gcc, intel
In combination we recommend (as compiler, then mpi in order so that the correct libraries are loaded).
$ module load { gcc/11.2.1 | intel/21.2.0 | nvhpc/22.7 } openmpi/4.1.4
$ module load { gcc/11.2.1 | intel/21.2.0 } mvapich2/2.3.7
$ module load { gcc/11.2.1 | intel/21.2.0 } impi/17.0.7
There are a couple of situations where you would want multiple compilers loaded (But first compiler-MPI version that is module loaded will determine the MPI code that is loaded).
(1) Most c++ compilers use the gnu c++ include libraries. For a program (LAMMPS is one) that uses a lot of relatively recent c++, you will want a recent gcc to provide those libraries.
This works with the Intel proprietary icpc compiler
$ module load intel/17.0.7 openmpi/4.1.4 gcc/11.2.1
If you don't add the third module, icpc will use the libraries from the default Centos g++ 4.8.5 which is quite old and probably can't compile LAMMPS at all.
(2) llvm compilers (aocc/3.0.0 and intel/21.2.0 icx) try to auto-find g++ libraries but don't do it quite correctly.
$ module load aocc/3.0.0 $ clang++ -v AMD clang version 12.0.0 (CLANG: AOCC_3.0.0-Build#78 2020_12_10) (based on LLVM Mirror.Version.12.0.0) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/AMD/aocc-compiler-3.0.0/bin Found candidate GCC installation: /opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7 Found candidate GCC installation: /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8 Found candidate GCC installation: /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.2 Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.8.5 Selected GCC installation: /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9
so it picks devtoolset-9 libraries in spite of devtoolset-10 and 11 being available
$ ls /opt/rh devtoolset-10 devtoolset-11 devtoolset-3 devtoolset-7 devtoolset-8 devtoolset-9
if devtoolset-9 (gcc/9.3.1) is new enough, then that's ok.
(3) Sometimes Intel MKL will link back to the Intel compiler when using Intel OMP instead of GNU OMP. This should work:
$ module load gcc/11.2.1 mkl/20.0.4 openmpi/4.1.4 intel/17.0.7