| Both sides previous revision
Previous revision
Next revision
|
Previous revision
Last revision
Both sides next revision
|
python [2020/02/06 21:35]
root |
python [2021/12/14 18:25]
root |
| AHPCC maintains a number of Python versions: Anaconda, Intel/Anaconda, and compiled from source. | AHPCC maintains a number of Python versions: Anaconda, Intel/Anaconda, and compiled from source. |
| |
| Terminology node: Python modules are prewritten programs in Python. [[ environment_modules ]] are scripts on the system that set up different versions of software. | Terminology: Python modules are prewritten programs in Python. [[ environment_modules ]] are scripts on the system that set up different versions of software. |
| |
| [[ https://www.anaconda.com/distribution/ | Anaconda Python ]] is what we recommend for most purposes as it has wide compatibility and easy installation with Python modules. Recommended [[ environment_modules ]] are python/2.7.16-anaconda and 3.7.3-anaconda. | [[ https://www.anaconda.com/distribution/ | Anaconda Python ]] is what we recommend for most purposes as it has wide compatibility and easy installation with Python modules. Recommended [[ environment_modules ]] are python/2.7.16-anaconda and 3.7.3-anaconda. |
| [[ https://software.intel.com/en-us/distribution-for-python|Intel Python ]] is a version of Anaconda optimized for speed and is a good choice for general Python code that doesn't need a great number of Python modules. Recommended [[ environment_modules ]] are python/2.7.16-intel and 3.6.8-intel. | [[ https://software.intel.com/en-us/distribution-for-python|Intel Python ]] is a version of Anaconda optimized for speed and is a good choice for general Python code that doesn't need a great number of Python modules. Recommended [[ environment_modules ]] are python/2.7.16-intel and 3.6.8-intel. |
| |
| [[ https://www.python.org/downloads/source/ | Python compiled from source ]] will sometimes install Python modules that won't install in Anaconda. Recommended [[ environment_modules ]] are python/2.7.15 and 3.6.0. | [[ https://www.python.org/downloads/source/ | Python compiled from source ]] will sometimes install Python modules that won't install in Anaconda. Recommended [[ environment_modules ]] are python/2.7.15 and 3.6.0. |
| |
| If you type ''python'' at the shell without loading any [[ environment_modules ]], you will get Centos system Python, which is quite old but needs to be in the default path for some system programs to run. | If you type ''python'' at the shell without loading any [[ environment_modules ]], you will get Centos system python2 2.7.5, which is quite old and unsuitable for all but the simplest uses, but needs to be in the default path for some system programs to run. Default system python3 is 3.7.3 with very few python modules. These system versions are not modified in order to avoid breaking system utilities. |
| |
| ==Custom Python Setups== | ==Custom Python Setups== |
| You can install and use custom python environments using ''conda'' for Anaconda Python and using ''pip'' for Anaconda and compiled. ''conda'' is more convenient. | You can install and use custom python environments using ''conda'' for Anaconda and Intel Python and using ''pip'' for Anaconda and compiled. ''conda'' is more convenient. |
| |
| ==conda== | ==conda== |
| ''conda'' is a utility program for Anaconda Python, see [[ https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html| getting-started]]. It is most commonly used as ''conda list'' to show the Python modules in the current environment. | ''conda'' is a utility program for Anaconda Python, see [[ https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html| getting-started]]. It is most commonly used as ''conda list'' to show the Python modules in the current environment. ''conda'' was originally designed for Python but can also manage installations of other programs such as ''R''. |
| |
| Some examples of conda commmands are shown below. This shows environments or subversions that have different Python module configurations. If unspecified you will be using the starred "base" environment. ''conda info'' also finds a base environment for Python2 (not usable here in Python3). There are two custom environments that have been defined by the system (''tf_gpu'' and ''theano-env''), and the user running this doesn't have any yet. | Some examples of conda commmands are shown below. This shows environments or subversions that have different Python module configurations. If unspecified you will be using the starred "base" environment. ''conda info'' also finds a base environment for Python2 (not usable here in Python3). There are two custom environments that have been defined by the system (''tf_gpu'' and ''theano-env''), and the user running this doesn't have any environments yet. |
| <code> | <code> |
| $ module load python/3.7.3-anaconda | $ module load python/3.7.3-anaconda |
| $ source conda-3.7.3.sh | $ source /share/apps/bin/conda-3.7.3.sh |
| (base) $ conda info --envs | (base) $ conda info --envs |
| # conda environments: | # conda environments: |
| IMPORTANT: You may need to close and restart your shell after running 'conda init'. | IMPORTANT: You may need to close and restart your shell after running 'conda init'. |
| </code> | </code> |
| If you then do ''conda activate bash'' (once) you will see that conda has added some bash code to your ''~/.bashrc'' initialization script. This is acceptable if all the python you ever run is only Anaconda Python2 or only Anaconda Python3, and in that case you can just leave the modification alone. If you ever use other versions of Python, it is likely to cause problems. In that case, you can edit ''~/.bashrc'' and save the new code section at the end (delimited by comments conda_initialize) as a separate script file such as ''conda-3.7.3.sh'': | If you then do ''conda activate bash'' (once) you will see that conda has added some bash code to your ''~/.bashrc'' initialization script. This is acceptable if all the python you ever run is only Anaconda Python2 or only Anaconda Python3, and in that case you can just leave the modification alone. If you ever use other versions of Python, it is likely to cause problems. In that case, you can edit ''~/.bashrc'' and save the new code section at the end (delimited by comments conda_initialize) as a separate script file which we have saved as ''/share/apps/bin/conda-3.7.3.sh'': |
| <code> | <code> |
| $ cat conda-3.7.3.sh | $ cat /share/apps/bin/conda-3.7.3.sh |
| # >>> conda initialize >>> | # >>> conda initialize >>> |
| # !! Contents within this block are managed by 'conda init' !! | # !! Contents within this block are managed by 'conda init' !! |
| <code> | <code> |
| $ module load python/3.7.3-anaconda | $ module load python/3.7.3-anaconda |
| $ source conda-3.7.3.sh | $ source /share/apps/bin/conda-3.7.3.sh |
| (base) $ | (base) $ |
| </code> | </code> |
| |
| ==intel== | ==intel== |
| The Intel/Anaconda commands are a bit different [[ https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda | using-intel ]] but have similar functions. | The Intel/Anaconda commands are a bit different [[ https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda | using-intel ]] but have similar functions. Intel Python is designed a bit better to not overwrite your ''~/.bashrc''. |
| <code> | <code> |
| $ conda info --envs | $ conda info --envs |
| </code> | </code> |
| |
| The environment module for compiled Python sets the environment variable PYTHONUSERBASE to a subdirectory of your home called python{major version} such as ''~/python3.6'' to which you can add your own user modules. If necessary you can change the variable after loading the module. To install user modules, first create the directory | The environment module for compiled Python sets the environment variable ''$PYTHONUSERBASE'' to a subdirectory of your home called python{major version} such as ''~/python3.6'' to which you can add your own user modules. If necessary you can change the variable after loading the module. To install user modules, first create the directory |
| <code> | <code> |
| cd ~ | cd ~ |
| peppercorn (0.5) | peppercorn (0.5) |
| </code> | </code> |
| These local Python modules will then be automatically substituted whenever you run Python 3.6.x, and you can control the local modules with ''pip install'',''pip update'', and ''pip remove''. | These local Python modules will then be automatically substituted whenever you run Python 3.6.x, and you can control the local modules with ''pip install'', ''pip update'', and ''pip remove''. If you don't want this substitution, redefine ''$PYTHONUSERBASE''. |
