| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
python [2020/02/06 21:31]
root |
python [2022/02/18 20:21] (current)
root old revision restored (2020/12/21 19:29) |
| AHPCC maintains a number of Python versions: Anaconda, Intel/Anaconda, and compiled from source. | AHPCC maintains a number of Python versions: Anaconda, Intel/Anaconda, and compiled from source. |
| |
| Terminology node: Python modules are prewritten programs in Python. [[ environment_modules ]] are scripts on the system that set up different versions of software. | Terminology: Python modules are prewritten programs in Python. [[ environment_modules ]] are scripts on the system that set up different versions of software. |
| |
| [[ https://www.anaconda.com/distribution/ | Anaconda Python ]] is what we recommend for most purposes as it has wide compatibility and easy installation with Python modules. Recommended [[ environment_modules ]] are python/2.7.16-anaconda and 3.7.3-anaconda. | [[ https://www.anaconda.com/distribution/ | Anaconda Python ]] is what we recommend for most purposes as it has wide compatibility and easy installation with Python modules. Recommended [[ environment_modules ]] are python/2.7.16-anaconda and 3.7.3-anaconda. |
| [[ https://software.intel.com/en-us/distribution-for-python|Intel Python ]] is a version of Anaconda optimized for speed and is a good choice for general Python code that doesn't need a great number of Python modules. Recommended [[ environment_modules ]] are python/2.7.16-intel and 3.6.8-intel. | [[ https://software.intel.com/en-us/distribution-for-python|Intel Python ]] is a version of Anaconda optimized for speed and is a good choice for general Python code that doesn't need a great number of Python modules. Recommended [[ environment_modules ]] are python/2.7.16-intel and 3.6.8-intel. |
| |
| [[ https://www.python.org/downloads/source/ | Python compiled from source ]] will sometimes install Python modules that won't install in Anaconda. Recommended [[ environment_modules ]] are python/2.7.15 and 3.6.0. | [[ https://www.python.org/downloads/source/ | Python compiled from source ]] will sometimes install Python modules that won't install in Anaconda. Recommended [[ environment_modules ]] are python/2.7.15 and 3.6.0. |
| |
| If you type ''python'' at the shell without loading any [[ environment_modules ]], you will get Centos system Python, which is quite old but needs to be in the default path for some system programs to run. | If you type ''python'' at the shell without loading any [[ environment_modules ]], you will get Centos system Python2, which is quite old and unsuitable for all but the simplest uses, but needs to be in the default path for some system programs to run. There is no default Python3. |
| |
| ==Custom Python Setups== | ==Custom Python Setups== |
| You can install and use custom python environments using ''conda'' for Anaconda Python and using ''pip'' for Anaconda and compiled. ''conda'' is more convenient. | You can install and use custom python environments using ''conda'' for Anaconda and Intel Python and using ''pip'' for Anaconda and compiled. ''conda'' is more convenient. |
| |
| ==conda== | ==conda== |
| ''conda'' is a utility program for Anaconda Python, see [[ https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html| getting-started]]. It is most commonly used as ''conda list'' to show the Python modules in the current environment. | ''conda'' is a utility program for Anaconda Python, see [[ https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html| getting-started]]. It is most commonly used as ''conda list'' to show the Python modules in the current environment. |
| |
| Some examples of conda commmands are shown below. This shows environments or subversions that have different Python module configurations. If unspecified you will be using the starred "base" environment. ''conda info'' also finds a base environment for Python2 (not usable here in Python3). There are two custom environments that have been defined by the system (''tf_gpu'' and ''theano-env''), and the user running this doesn't have any yet. | Some examples of conda commmands are shown below. This shows environments or subversions that have different Python module configurations. If unspecified you will be using the starred "base" environment. ''conda info'' also finds a base environment for Python2 (not usable here in Python3). There are two custom environments that have been defined by the system (''tf_gpu'' and ''theano-env''), and the user running this doesn't have any environments yet. |
| <code> | <code> |
| $ module load python/3.7.3-anaconda | $ module load python/3.7.3-anaconda |
| $ source conda-3.7.3.sh | $ source /share/apps/bin/conda-3.7.3.sh |
| (base) $ conda info --envs | (base) $ conda info --envs |
| # conda environments: | # conda environments: |
| etc. | etc. |
| </code> | </code> |
| If this enviroment ''tf_gpu'' had been your own enviroment, you could also do commands such as ''conda install'', ''conda remove'', and ''conda update''. See [[ https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html|manage-environments ]] for ''conda create'' to create your own environments, normally stored under your home ''~/.conda/envs''. | If this enviroment ''tf_gpu'' had been your own enviroment, you could also do commands such as ''conda install'', ''conda remove'', and ''conda update''. See [[ https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html|manage-environments ]] for ''conda create'' to create your own environments, normally stored under your home in ''~/.conda/envs/''. |
| | |
| | The first time you run ''conda activate'' you should get a message like: |
| |
| The first time you run ''conda activate'' you will normally get a message like: | |
| <code> | <code> |
| CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. | CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. |
| IMPORTANT: You may need to close and restart your shell after running 'conda init'. | IMPORTANT: You may need to close and restart your shell after running 'conda init'. |
| </code> | </code> |
| If you then do ''conda activate bash'' (once) you will see that conda has added some bash code to your ''~/.bashrc'' initialization script. This is acceptable if all the python you ever run is only Anaconda Python2 or only Anaconda Python3, and in that case you can just leave the modification alone. If you ever use other versions of Python, it is likely to cause problems. In that case, you can edit ''~/.bashrc'' and save the new code section at the end as a separate script file such as ''conda-3.7.3.sh'': | |
| | If you then do ''conda activate bash'' (once) you will see that conda has added some bash code to your ''~/.bashrc'' initialization script. This is acceptable if all the python you ever run is only Anaconda Python2 or only Anaconda Python3, and in that case you can just leave the modification alone. If you ever use other versions of Python, it is likely to cause problems. In that case, you can edit ''~/.bashrc'' and save the new code section at the end (delimited by comments conda_initialize) as a separate script file which we have saved as ''/share/apps/bin/conda-3.7.3.sh'': |
| <code> | <code> |
| $ cat conda-3.7.3.sh | $ cat /share/apps/bin/conda-3.7.3.sh |
| # >>> conda initialize >>> | # >>> conda initialize >>> |
| # !! Contents within this block are managed by 'conda init' !! | # !! Contents within this block are managed by 'conda init' !! |
| <code> | <code> |
| $ module load python/3.7.3-anaconda | $ module load python/3.7.3-anaconda |
| $ source conda-3.7.3.sh | $ source /share/apps/bin/conda-3.7.3.sh |
| (base) $ | (base) $ |
| </code> | </code> |
| |
| ==intel== | ==intel== |
| The Intel/Anaconda commands are a bit different [[ https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda | using-intel ]] but have similar functions. | The Intel/Anaconda commands are a bit different [[ https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda | using-intel ]] but have similar functions. Intel Python is designed a bit better to not overwrite your ''~/.bashrc''. |
| <code> | <code> |
| $ conda info --envs | $ conda info --envs |
| </code> | </code> |
| |
| Compiled python has ''pip'', a less-convenient way to add your own environment. The environment module sets the environment variable PYTHONUSERBASE to a subdirectory of your home called python{major version} such as ''~/python3.6'' to which you can add your own user modules. If necessary you can change the variable after loading the module. To install user modules, first create the directory | The environment module for compiled Python sets the environment variable ''$PYTHONUSERBASE'' to a subdirectory of your home called python{major version} such as ''~/python3.6'' to which you can add your own user modules. If necessary you can change the variable after loading the module. To install user modules, first create the directory |
| <code> | <code> |
| cd ~ | cd ~ |
| </code> | </code> |
| To add a module not in the pip list, | To add a module not in the pip list, |
| | |
| <code> | <code> |
| pip install peppercorn --user | pip install peppercorn --user |
| </code> | </code> |
| | |
| To add a module with a different version than the pip list: | To add a module with a different version than the pip list: |
| | |
| <code> | <code> |
| pip install lxml --user --ignore-installed | pip install lxml --user --ignore-installed |
| </code> | </code> |
| | |
| To show your local modules, which will be used preferentially to the system modules: | To show your local modules, which will be used preferentially to the system modules: |
| | |
| <code> | <code> |
| $ pip list --user | $ pip list --user |
| peppercorn (0.5) | peppercorn (0.5) |
| </code> | </code> |
| These local Python modules will then be automatically substituted whenever you run Python 3.6.x. | These local Python modules will then be automatically substituted whenever you run Python 3.6.x, and you can control the local modules with ''pip install'', ''pip update'', and ''pip remove''. If you don't want this substitution, redefine ''$PYTHONUSERBASE''. |
