| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
sapplication_software [2022/09/27 23:35]
root |
sapplication_software [2022/09/30 15:20] (current)
root |
| Locating and using software has been made a little more complicated by some then reasonable decisions made 50 years ago for Unix: ''/usr/local'' for applications, environment variables ''$PATH'' to find an executable, ''$LD\_LIBRARY\_PATH'' to find dynamic link libraries, the file ''~/.cshrc'' to set up these variables. These environment variables continue today in Linux and Mac, while Windows combines the two PATH variables. Also important were software packages intended to be used as infrastructure for complete applications, and which simplifed development by not needing to be copied into the code of every project.. These "shared libraries" such as MPI and FFTW were specified by source code interfaces. At the time, nearly everyone used one computer and one compiler, so a source interface corresponded directly to one binary interface. Today there are many types of computers and compilers, and modern commercial applications often have a defined binary interface or "ABI" to avoid compatibility issues. | Locating and using software has been made a little more complicated by some then reasonable decisions made 50 years ago for Unix: ''/usr/local'' for applications, environment variables ''$PATH'' to find an executable, ''$LD\_LIBRARY\_PATH'' to find dynamic link libraries, the file ''~/.cshrc'' to set up these variables. These environment variables continue today in Linux and Mac, while Windows combines the two PATH variables. Also important were software packages intended to be used as infrastructure for complete applications, and which simplifed development by not needing to be copied into the code of every project.. These "shared libraries" such as MPI and FFTW were specified by source code interfaces. At the time, nearly everyone used one computer and one compiler, so a source interface corresponded directly to one binary interface. Today there are many types of computers and compilers, and modern commercial applications often have a defined binary interface or "ABI" to avoid compatibility issues. |
| |
| Today on HPC systems, the MPI implementation must be heavily customized for each site and its network fabric. Almost all multi-node programs depend on MPI, and there are three popular implementations of MPI (Open MPI, MVAPICH2, and Intel). There are about six compilers that are reasonably popular (GNU/gcc, Intel proprietary, Intel open LLVM, NVidia/PGI, AMD LLVM, open LLVM). MVAPICH2 and Intel MPI are binary compatible (thus a defacto ABI). Probably all the LLVM implementations are reasonably binary compatible (with each other, not with Open MPI vs MVAPICH2), so there are about 8 to 12 binary versions of MPI, not considering updates for each that come out 2 or 3 times a year. As will be seen in the demonstrations below, many of the software packaging programs (OpenHPC, Spack, conda) use their own version of MPI. Usually we can expect this to only work within one node using shared memory. Depending on how the applications were compiled and linked, sometimes we can intercept MPI library calls with our site-compiled MPI that can run on multiple nodes. | Today on HPC systems, the MPI implementation must be heavily customized for each site and its network fabric. Almost all multi-node programs depend on MPI, and there are three popular implementations of MPI (Open MPI, MVAPICH2, and Intel). There are about six compilers that are reasonably popular (GNU/gcc, Intel proprietary, Intel open LLVM, NVidia/PGI, AMD LLVM, open LLVM). MVAPICH2 and Intel MPI are binary compatible (thus a defacto ABI). Probably all the LLVM implementations are reasonably binary compatible (with each other, not with Open MPI vs MVAPICH2), so there are about 8 to 12 binary versions of MPI, not considering updates for each that come out 2 or 3 times a year, and not considering the approximate doubling of GPU vs non-GPU support. As will be seen in the demonstrations below, many of the software packaging programs (OpenHPC, Spack, conda) use their own version of MPI. MPI over shared memory is pretty well standardized, so we can usually expect this to work within one node. Depending on how the applications were compiled and linked, sometimes we can intercept MPI library calls with our site-compiled MPI that can run on multiple nodes. |
| | |
| With thousands of applications (most of which have multiple versions), it's obviously impractical just from name collisions to put every executable in ''/usr/local/bin''. It's also impractical (unless you use only one application) to semipermanently set up these variables in ''~/.bashrc'' or ''~/.cshrc''. There are several ways to handle this. | With thousands of applications (most of which have multiple versions), it's obviously impractical just from name collisions to put every executable in ''/usr/local/bin''. It's also impractical (unless you use only one application) to semipermanently set up these variables in ''~/.bashrc'' or ''~/.cshrc''. There are several ways to handle this. |
| At this writing we have 548 module files from the OpenHPC distribution. It concentrates on mathematical software. Many of the packages are a bit dated, but when we update to Rocky 8 Linux we will be able to install some newer packages. | At this writing we have 548 module files from the OpenHPC distribution. It concentrates on mathematical software. Many of the packages are a bit dated, but when we update to Rocky 8 Linux we will be able to install some newer packages. |
| |
| You can run particular packages like so. For an example, we'll try to find the newest version of ''petsc'' and the MPI and compiler package that it needs. We then load those three and it auto-loads some prerequisites. There is quite a bit of unnecessary duplication, as for instance, gnu 7 and gnu 8 binaries are very similar, so it wasn't really necessary to duplicate every module file. | You can run particular packages like so. For an example, we'll try to find the newest version of ''petsc'' and the MPI and compiler package that it needs. We then load those three and it auto-loads some prerequisites. There is quite a bit of unnecessary duplication, as for instance, gnu7 and gnu8 binaries are very similar, so it wasn't really necessary to duplicate every module file. |
| |
| <code> | <code> |
| $ grep petsc /share/apps/modulelist | $ grep petsc /share/apps/modulelist |
| | |
| /opt/ohpc/pub/moduledeps/ gnu7-impi/petsc/3.9.1 | /opt/ohpc/pub/moduledeps/ gnu7-impi/petsc/3.9.1 |
| /opt/ohpc/pub/moduledeps/ gnu7-mpich/petsc/3.9.1 | /opt/ohpc/pub/moduledeps/ gnu7-mpich/petsc/3.9.1 |
| /share/apps/modulefiles/ petsc/3.8.58 | /share/apps/modulefiles/ petsc/3.8.58 |
| |
| $ grep "gnu8/openmpi3/" /share/apps/modulelist | $ module purge |
| /opt/ohpc/pub/moduledeps/ gnu8/openmpi3/3.1.4 | $ module load ohpc gnu8 gnu8/openmpi3 gnu8-openmpi3/petsc |
| |
| $ grep "gnu8/8" /share/apps/modulelist | Lmod is automatically replacing "openmpi3/3.1.4" with "gnu8/openmpi3/3.1.4". |
| /opt/ohpc/pub/modulefiles/ gnu8/8.3.0 | |
| |
| $ module load gnu8/8.3.0 gnu8/openmpi3/3.1.4 gnu8-openmpi3/petsc/3.12.0 | |
| $ module list | $ module list |
| Currently Laded Modules: | |
| 1) gnu8/8.3.0 2) gnu8/openmpi3/3.1.4 3) gnu8-openmpi3/phdf5/1.10.5 4) openblas/3.20-noomp 5) gnu8-openmpi3/scalapack/2.0.2 6) gnu8-openmpi3/petsc/3.12.0 | Currently Loaded Modules: |
| | 1) autotools 4) gnu8/8.3.0 7) openblas/3.20-noomp |
| | 2) prun/1.3 5) gnu8/openmpi3/3.1.4 8) gnu8-openmpi3/scalapack/2.0.2 |
| | 3) ohpc 6) gnu8-openmpi3/phdf5/1.10.5 9) gnu8-openmpi3/petsc/3.1 |
| </code> | </code> |
| |
| It is likely that this will only work on one node since it links back to its internal MPI which is not customized for our network. However, it should suffice for a test- or education-sized ''petsc'' run, while ''petsc'' and its prerequisites take some hours to install manually. | It is likely that this program will only work on one node since it links back to its internal MPI which is not customized for our network. However, it should suffice for a test- or education-sized ''petsc'' run, while ''petsc'' and its prerequisites take some hours to install manually. |
| |
| ==Spack modulefiles== | ==Spack modulefiles== |
| </code> | </code> |
| |
| As an example, we'll try to run abyss-pe in parallel as in [[https://hpc.nih.gov/apps/abyss.html|Biowulf-abyss]]. The module turns out to call, but not to load, ''mpirun'' so we will add some local modules. At this time, all the local Spack software is compiled with gcc/11.2.1 and openmpi/4.1.4 (though not our versions) so we will use those modules. This does work, though we have not tested with multiple nodes, as it is not clear which version of MPI is actually being linked to. | For testing we'll try to run abyss-pe in parallel similarly to [[https://hpc.nih.gov/apps/abyss.html|Biowulf-abyss-example]]. The Spack module turns out to reference, but not to install, ''mpirun'' so we will add some local modules. At this time, all our local Spack software is compiled with gcc/11.2.1 and openmpi/4.1.4 (though not our versions) so we will use those modules. This does work, though we have not tested with multiple nodes. Testing with our manually installed abyss 2.0.2 shows that multiple nodes doesn't work well anyway. |
| <code> | <code> |
| $ module load gcc-11.2.1/SKYLAKEX/abyss/2.3.1 | $ module load gcc-11.2.1/SKYLAKEX/abyss/2.3.1 |
| |
| [[https://docs.easybuild.io/en/latest/version-specific/Supported_software.html|EasyBuild]] is conceptually similar to Spack and has about 2700 applications. We are beginning testing on it. | [[https://docs.easybuild.io/en/latest/version-specific/Supported_software.html|EasyBuild]] is conceptually similar to Spack and has about 2700 applications. We are beginning testing on it. |
| | |
| | ==conda== |
| | |
| | [[conda|conda]] was originally a way to customize Python but has evolved to a package manager for other applications. It is popular in bioinformatics, which frequently requires very specific versioning of programs that are used together but not compiled together. It lacks an easy way to document what environments are currently installed by conda. We are working on that. Here is an example of creating your own conda environment. Run |
| | |
| | <code> |
| | -bash-4.2$ /share/apps/bin/setupconda.sh |
| | -bash-4.2$ ls -al .conda |
| | total 16 |
| | drwxrwxr-x 4 ddn1 ddn1 4096 Sep 30 10:06 . |
| | drwxr-xr-x 7 ddn1 ddn1 4096 Sep 30 10:05 .. |
| | -rw-rw-r-- 1 ddn1 ddn1 0 Sep 30 10:10 environments.txt |
| | drwxrwxr-x 2 ddn1 ddn1 4096 Sep 30 10:07 envs |
| | drwxrwxr-x 3 ddn1 ddn1 4096 Sep 30 10:07 pkgs |
| | -bash-4.2$ module load gcc/8.3.1 mkl/19.0.5 python/3.10-anaconda |
| | -bash-4.2$ source /share/apps/bin/conda-3.10.sh;conda create -n testenv |
| | Collecting package metadata (current_repodata.json): done |
| | Solving environment: done |
| | |
| | ## Package Plan ## |
| | |
| | environment location: /home/ddn1/.conda/envs/teste |
| | |
| | Proceed ([y]/n)? y |
| | |
| | Preparing transaction: done |
| | Verifying transaction: done |
| | Executing transaction: done |
| | # |
| | # To activate this environment, use |
| | # |
| | # $ conda activate testenv |
| | # |
| | # To deactivate an active environment, use |
| | # |
| | # $ conda deactivate |
| | |
| | Retrieving notices: ...working... done |
| | (base) -bash-4.2$ conda activate testenv |
| | (testenv) -bash-4.2$ conda install pytorch |
| | Collecting package metadata (current_repodata.json): done |
| | Solving environment: done |
| | |
| | ## Package Plan ## |
| | |
| | environment location: /home/ddn1/.conda/envs/testenv |
| | |
| | added / updated specs: |
| | - pytorch |
| | |
| | |
| | The following packages will be downloaded: |
| | |
| | package | build |
| | ---------------------------|----------------- |
| | _libgcc_mutex-0.1 | main 3 KB |
| | _openmp_mutex-5.1 | 1_gnu 21 KB |
| | blas-1.0 | mkl 6 KB |
| | bzip2-1.0.8 | h7b6447c_0 78 KB |
| | ca-certificates-2022.07.19 | h06a4308_0 124 KB |
| | certifi-2022.9.14 | py310h06a4308_0 155 KB |
| | cffi-1.15.1 | py310h74dc2b5_0 407 KB |
| | future-0.18.2 | py310h06a4308_1 647 KB |
| | intel-openmp-2021.4.0 | h06a4308_3561 4.2 MB |
| | ld_impl_linux-64-2.38 | h1181459_1 654 KB |
| | libffi-3.3 | he6710b0_2 50 KB |
| | libgcc-ng-11.2.0 | h1234567_1 5.3 MB |
| | libgomp-11.2.0 | h1234567_1 474 KB |
| | libstdcxx-ng-11.2.0 | h1234567_1 4.7 MB |
| | libuuid-1.0.3 | h7f8727e_2 17 KB |
| | mkl-2021.4.0 | h06a4308_640 142.6 MB |
| | mkl-service-2.4.0 | py310h7f8727e_0 177 KB |
| | mkl_fft-1.3.1 | py310hd6ae3a3_0 567 KB |
| | mkl_random-1.2.2 | py310h00e6091_0 1009 KB |
| | ncurses-6.3 | h5eee18b_3 781 KB |
| | ninja-1.10.2 | h06a4308_5 8 KB |
| | ninja-base-1.10.2 | hd09550d_5 109 KB |
| | numpy-1.23.1 | py310h1794996_0 11 KB |
| | numpy-base-1.23.1 | py310hcba007f_0 14.3 MB |
| | openssl-1.1.1q | h7f8727e_0 2.5 MB |
| | pip-22.1.2 | py310h06a4308_0 2.5 MB |
| | pycparser-2.21 | pyhd3eb1b0_0 94 KB |
| | python-3.10.4 | h12debd9_0 24.2 MB |
| | pytorch-1.10.2 |cpu_py310h6894f24_0 43.9 MB |
| | readline-8.1.2 | h7f8727e_1 354 KB |
| | setuptools-63.4.1 | py310h06a4308_0 1.1 MB |
| | six-1.16.0 | pyhd3eb1b0_1 18 KB |
| | sqlite-3.39.3 | h5082296_0 1.1 MB |
| | tk-8.6.12 | h1ccaba5_0 3.0 MB |
| | typing-extensions-4.3.0 | py310h06a4308_0 9 KB |
| | typing_extensions-4.3.0 | py310h06a4308_0 42 KB |
| | tzdata-2022c | h04d1e81_0 107 KB |
| | wheel-0.37.1 | pyhd3eb1b0_0 33 KB |
| | xz-5.2.6 | h5eee18b_0 394 KB |
| | zlib-1.2.12 | h5eee18b_3 103 KB |
| | ------------------------------------------------------------ |
| | Total: 255.7 MB |
| | |
| | The following NEW packages will be INSTALLED: |
| | |
| | _libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main |
| | _openmp_mutex pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu |
| | blas pkgs/main/linux-64::blas-1.0-mkl |
| | bzip2 pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0 |
| | ca-certificates pkgs/main/linux-64::ca-certificates-2022.07.19-h06a4308_0 |
| | certifi pkgs/main/linux-64::certifi-2022.9.14-py310h06a4308_0 |
| | cffi pkgs/main/linux-64::cffi-1.15.1-py310h74dc2b5_0 |
| | future pkgs/main/linux-64::future-0.18.2-py310h06a4308_1 |
| | intel-openmp pkgs/main/linux-64::intel-openmp-2021.4.0-h06a4308_3561 |
| | ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1 |
| | libffi pkgs/main/linux-64::libffi-3.3-he6710b0_2 |
| | libgcc-ng pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1 |
| | libgomp pkgs/main/linux-64::libgomp-11.2.0-h1234567_1 |
| | libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1 |
| | libuuid pkgs/main/linux-64::libuuid-1.0.3-h7f8727e_2 |
| | mkl pkgs/main/linux-64::mkl-2021.4.0-h06a4308_640 |
| | mkl-service pkgs/main/linux-64::mkl-service-2.4.0-py310h7f8727e_0 |
| | mkl_fft pkgs/main/linux-64::mkl_fft-1.3.1-py310hd6ae3a3_0 |
| | mkl_random pkgs/main/linux-64::mkl_random-1.2.2-py310h00e6091_0 |
| | ncurses pkgs/main/linux-64::ncurses-6.3-h5eee18b_3 |
| | ninja pkgs/main/linux-64::ninja-1.10.2-h06a4308_5 |
| | ninja-base pkgs/main/linux-64::ninja-base-1.10.2-hd09550d_5 |
| | numpy pkgs/main/linux-64::numpy-1.23.1-py310h1794996_0 |
| | numpy-base pkgs/main/linux-64::numpy-base-1.23.1-py310hcba007f_0 |
| | openssl pkgs/main/linux-64::openssl-1.1.1q-h7f8727e_0 |
| | pip pkgs/main/linux-64::pip-22.1.2-py310h06a4308_0 |
| | pycparser pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0 |
| | python pkgs/main/linux-64::python-3.10.4-h12debd9_0 |
| | pytorch pkgs/main/linux-64::pytorch-1.10.2-cpu_py310h6894f24_0 |
| | readline pkgs/main/linux-64::readline-8.1.2-h7f8727e_1 |
| | setuptools pkgs/main/linux-64::setuptools-63.4.1-py310h06a4308_0 |
| | six pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1 |
| | sqlite pkgs/main/linux-64::sqlite-3.39.3-h5082296_0 |
| | tk pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0 |
| | typing-extensions pkgs/main/linux-64::typing-extensions-4.3.0-py310h06a4308_0 |
| | typing_extensions pkgs/main/linux-64::typing_extensions-4.3.0-py310h06a4308_0 |
| | tzdata pkgs/main/noarch::tzdata-2022c-h04d1e81_0 |
| | wheel pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0 |
| | xz pkgs/main/linux-64::xz-5.2.6-h5eee18b_0 |
| | zlib pkgs/main/linux-64::zlib-1.2.12-h5eee18b_3 |
| | |
| | |
| | Proceed ([y]/n)? y |
| | |
| | |
| | Downloading and Extracting Packages |
| | ninja-1.10.2 | 8 KB | ###################################################################################### | 100% |
| | python-3.10.4 | 24.2 MB | ###################################################################################### | 100% |
| | libgomp-11.2.0 | 474 KB | ###################################################################################### | 100% |
| | xz-5.2.6 | 394 KB | ###################################################################################### | 100% |
| | numpy-1.23.1 | 11 KB | ###################################################################################### | 100% |
| | typing-extensions-4. | 9 KB | ###################################################################################### | 100% |
| | bzip2-1.0.8 | 78 KB | ###################################################################################### | 100% |
| | ninja-base-1.10.2 | 109 KB | ###################################################################################### | 100% |
| | wheel-0.37.1 | 33 KB | ###################################################################################### | 100% |
| | setuptools-63.4.1 | 1.1 MB | ###################################################################################### | 100% |
| | blas-1.0 | 6 KB | ###################################################################################### | 100% |
| | pycparser-2.21 | 94 KB | ###################################################################################### | 100% |
| | libffi-3.3 | 50 KB | ###################################################################################### | 100% |
| | future-0.18.2 | 647 KB | ###################################################################################### | 100% |
| | mkl-service-2.4.0 | 177 KB | ###################################################################################### | 100% |
| | libgcc-ng-11.2.0 | 5.3 MB | ###################################################################################### | 100% |
| | libstdcxx-ng-11.2.0 | 4.7 MB | ###################################################################################### | 100% |
| | cffi-1.15.1 | 407 KB | ###################################################################################### | 100% |
| | intel-openmp-2021.4. | 4.2 MB | ###################################################################################### | 100% |
| | six-1.16.0 | 18 KB | ###################################################################################### | 100% |
| | pip-22.1.2 | 2.5 MB | ###################################################################################### | 100% |
| | ncurses-6.3 | 781 KB | ###################################################################################### | 100% |
| | libuuid-1.0.3 | 17 KB | ###################################################################################### | 100% |
| | readline-8.1.2 | 354 KB | ###################################################################################### | 100% |
| | mkl-2021.4.0 | 142.6 MB | ###################################################################################### | 100% |
| | sqlite-3.39.3 | 1.1 MB | ###################################################################################### | 100% |
| | mkl_fft-1.3.1 | 567 KB | ###################################################################################### | 100% |
| | tzdata-2022c | 107 KB | ###################################################################################### | 100% |
| | _libgcc_mutex-0.1 | 3 KB | ###################################################################################### | 100% |
| | typing_extensions-4. | 42 KB | ###################################################################################### | 100% |
| | _openmp_mutex-5.1 | 21 KB | ###################################################################################### | 100% |
| | tk-8.6.12 | 3.0 MB | ###################################################################################### | 100% |
| | mkl_random-1.2.2 | 1009 KB | ###################################################################################### | 100% |
| | ld_impl_linux-64-2.3 | 654 KB | ###################################################################################### | 100% |
| | zlib-1.2.12 | 103 KB | ###################################################################################### | 100% |
| | ca-certificates-2022 | 124 KB | ###################################################################################### | 100% |
| | numpy-base-1.23.1 | 14.3 MB | ###################################################################################### | 100% |
| | openssl-1.1.1q | 2.5 MB | ###################################################################################### | 100% |
| | certifi-2022.9.14 | 155 KB | ###################################################################################### | 100% |
| | pytorch-1.10.2 | 43.9 MB | ###################################################################################### | 100% |
| | Preparing transaction: done |
| | Verifying transaction: done |
| | Executing transaction: done |
| | Retrieving notices: ...working... done |
| | (testenv) -bash-4.2$ cat .conda/environments.txt |
| | /home/ddn1/.conda/envs/testenv |
| | (testenv) -bash-4.2$ python |
| | Python 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] on linux |
| | Type "help", "copyright", "credits" or "license" for more information. |
| | >>> import numpy as np |
| | >>> exit() |
| | (testenv) -bash-4.2$ |
| | </code> |
| | |
| |
| ===Compiler-MPI Recommendations=== | ===Compiler-MPI Recommendations=== |
| </code> | </code> |
| |
| If you don't load any modules, there are some very old compilers built into Centos: | If you don't load any modules, there are some very old compilers built into Centos and in the default path /usr/bin: |
| <code> | <code> |
| $ gcc --version | $ gcc --version |
