==== Python ====
AHPCC maintains a number of Python versions: Anaconda, Intel/Anaconda, and compiled from source.
Terminology: Python modules are prewritten programs in Python. [[ environment_modules ]] are scripts on the system that set up different versions of software.
[[ https://www.anaconda.com/distribution/ | Anaconda Python ]] is what we recommend for most purposes as it has wide compatibility and easy installation with Python modules. Recommended [[ environment_modules ]] are python/2.7.16-anaconda and 3.7.3-anaconda.
[[ https://software.intel.com/en-us/distribution-for-python|Intel Python ]] is a version of Anaconda optimized for speed and is a good choice for general Python code that doesn't need a great number of Python modules. Recommended [[ environment_modules ]] are python/2.7.16-intel and 3.6.8-intel.
[[ https://www.python.org/downloads/source/ | Python compiled from source ]] will sometimes install Python modules that won't install in Anaconda. Recommended [[ environment_modules ]] are python/2.7.15 and 3.6.0.
If you type ''python'' at the shell without loading any [[ environment_modules ]], you will get Centos system Python2, which is quite old and unsuitable for all but the simplest uses, but needs to be in the default path for some system programs to run. There is no default Python3.
==Custom Python Setups==
You can install and use custom python environments using ''conda'' for Anaconda and Intel Python and using ''pip'' for Anaconda and compiled. ''conda'' is more convenient.
==conda==
''conda'' is a utility program for Anaconda Python, see [[ https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html| getting-started]]. It is most commonly used as ''conda list'' to show the Python modules in the current environment.
Some examples of conda commmands are shown below. This shows environments or subversions that have different Python module configurations. If unspecified you will be using the starred "base" environment. ''conda info'' also finds a base environment for Python2 (not usable here in Python3). There are two custom environments that have been defined by the system (''tf_gpu'' and ''theano-env''), and the user running this doesn't have any environments yet.
$ module load python/3.7.3-anaconda
$ source /share/apps/bin/conda-3.7.3.sh
(base) $ conda info --envs
# conda environments:
#
/scrfs/apps/python/anaconda2-2.7.16
base * /share/apps/python/anaconda3-python-3.7.3
tf_gpu /share/apps/python/anaconda3-python-3.7.3/envs/tf_gpu
theano-env /share/apps/python/anaconda3-python-3.7.3/envs/theano-env
(base) $ conda activate tf_gpu
(tf_gpu) $ conda list
# packages in environment at /share/apps/python/anaconda3-python-3.7.3/envs/tf_gpu:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_tflow_select 2.1.0 gpu
etc.
If this enviroment ''tf_gpu'' had been your own enviroment, you could also do commands such as ''conda install'', ''conda remove'', and ''conda update''. See [[ https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html|manage-environments ]] for ''conda create'' to create your own environments, normally stored under your home in ''~/.conda/envs/''.
The first time you run ''conda activate'' you should get a message like:
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
If you then do ''conda activate bash'' (once) you will see that conda has added some bash code to your ''~/.bashrc'' initialization script. This is acceptable if all the python you ever run is only Anaconda Python2 or only Anaconda Python3, and in that case you can just leave the modification alone. If you ever use other versions of Python, it is likely to cause problems. In that case, you can edit ''~/.bashrc'' and save the new code section at the end (delimited by comments conda_initialize) as a separate script file which we have saved as ''/share/apps/bin/conda-3.7.3.sh'':
$ cat /share/apps/bin/conda-3.7.3.sh
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/share/apps/python/anaconda3-python-3.7.3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/share/apps/python/anaconda3-python-3.7.3/etc/profile.d/conda.sh" ]; then
. "/share/apps/python/anaconda3-python-3.7.3/etc/profile.d/conda.sh"
else
export PATH="/share/apps/python/anaconda3-python-3.7.3/bin:$PATH"
fi
fi
#unset __conda_setup
# <<< conda initialize <<<
and delete this same code from ''~/.bashrc''. Then to start Anaconda Python and use ''conda'' you can as shown above add one step:
$ module load python/3.7.3-anaconda
$ source /share/apps/bin/conda-3.7.3.sh
(base) $
This is necessary as [[ environment_modules ]] can't do the source or "." command in ''conda-3.7.3.sh''. If you also use Anaconda Python 2.7.16 you'll need a similar script for it by the same process.
==intel==
The Intel/Anaconda commands are a bit different [[ https://software.intel.com/en-us/articles/using-intel-distribution-for-python-with-anaconda | using-intel ]] but have similar functions. Intel Python is designed a bit better to not overwrite your ''~/.bashrc''.
$ conda info --envs
# conda environments:
#
root * /scrfs/apps/intel/parallel_studio_xe_2019_update_4/intelpython3
$ conda create -n idp
Fetching package metadata ...............
Solving package specifications:
Package plan for installation in environment /home/rfeynman/.conda/envs/idp:
Proceed ([y]/n)? y
#
# To activate this environment, use:
# > source activate idp
#
# To deactivate an active environment, use:
# > source deactivate
#
$ conda info --envs
# conda environments:
#
idp /home/rfeynman/.conda/envs/idp
root * /scrfs/apps/intel/parallel_studio_xe_2019_update_4/intelpython3
$ source activate idp
(idp) $
==pip==
''pip'' is a similar utility program for compiled Python, and also works on Anaconda Python. We don't recommend its use on Anaconda Python unless ''conda'' won't do what you want.
Here is a compiled version of Python specifically to run ''phonopy'' so it has very few Python modules.
$ module load python/2.7.15b
$ pip list
Package Version
----------------------------- ------------
backports.functools-lru-cache 1.5
cycler 0.10.0
h5py 2.8.0
kiwisolver 1.0.1
matplotlib 2.2.3
numpy 1.14.5
phono3py 1.14.3.post1
phonopy 1.14.2
pip 18.1
pyparsing 2.3.0
python-dateutil 2.7.5
pytz 2018.7
PyYAML 3.13
setuptools 39.0.1
six 1.11.0
subprocess32 3.5.3
$
The environment module for compiled Python sets the environment variable ''$PYTHONUSERBASE'' to a subdirectory of your home called python{major version} such as ''~/python3.6'' to which you can add your own user modules. If necessary you can change the variable after loading the module. To install user modules, first create the directory
cd ~
mkdir -p python3.6
To add a module not in the pip list,
pip install peppercorn --user
To add a module with a different version than the pip list:
pip install lxml --user --ignore-installed
To show your local modules, which will be used preferentially to the system modules:
$ pip list --user
lxml (3.6.0)
peppercorn (0.5)
These local Python modules will then be automatically substituted whenever you run Python 3.6.x, and you can control the local modules with ''pip install'', ''pip update'', and ''pip remove''. If you don't want this substitution, redefine ''$PYTHONUSERBASE''.