Arkansas High Performace Computing Center [hpcwiki]

AHPCC maintains a number of Python versions: Anaconda, Intel/Anaconda, and compiled from source.

Terminology: Python modules are prewritten programs in Python. environment_modules are scripts on the system that set up different versions of software.

Anaconda Python is what we recommend for most purposes as it has wide compatibility and easy installation with Python modules. Recommended environment_modules are python/2.7.16-anaconda and 3.7.3-anaconda.

Intel Python is a version of Anaconda optimized for speed and is a good choice for general Python code that doesn't need a great number of Python modules. Recommended environment_modules are python/2.7.16-intel and 3.6.8-intel.
Python compiled from source will sometimes install Python modules that won't install in Anaconda. Recommended environment_modules are python/2.7.15 and 3.6.0.

If you type python at the shell without loading any environment_modules, you will get Centos system Python2, which is quite old and unsuitable for all but the simplest uses, but needs to be in the default path for some system programs to run. There is no default Python3.

You can install and use custom python environments using conda for Anaconda and Intel Python and using pip for Anaconda and compiled. conda is more convenient.

conda is a utility program for Anaconda Python, see getting-started. It is most commonly used as conda list to show the Python modules in the current environment.

Some examples of conda commmands are shown below. This shows environments or subversions that have different Python module configurations. If unspecified you will be using the starred “base” environment. conda info also finds a base environment for Python2 (not usable here in Python3). There are two custom environments that have been defined by the system (tf_gpu and theano-env), and the user running this doesn't have any environments yet.

$ module load python/3.7.3-anaconda
$ source /share/apps/bin/conda-3.7.3.sh
(base) $ conda info --envs
# conda environments:
#
                         /scrfs/apps/python/anaconda2-2.7.16
base                  *  /share/apps/python/anaconda3-python-3.7.3
tf_gpu                   /share/apps/python/anaconda3-python-3.7.3/envs/tf_gpu
theano-env               /share/apps/python/anaconda3-python-3.7.3/envs/theano-env

(base) $ conda activate tf_gpu
(tf_gpu) $ conda list
# packages in environment at /share/apps/python/anaconda3-python-3.7.3/envs/tf_gpu:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_tflow_select             2.1.0                       gpu  
etc.

If this enviroment tf_gpu had been your own enviroment, you could also do commands such as conda install, conda remove, and conda update. See manage-environments for conda create to create your own environments, normally stored under your home in ~/.conda/envs/.

The first time you run conda activate you should get a message like:

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
    $ conda init <SHELL_NAME>
Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.

If you then do conda activate bash (once) you will see that conda has added some bash code to your ~/.bashrc initialization script. This is acceptable if all the python you ever run is only Anaconda Python2 or only Anaconda Python3, and in that case you can just leave the modification alone. If you ever use other versions of Python, it is likely to cause problems. In that case, you can edit ~/.bashrc and save the new code section at the end (delimited by comments conda_initialize) as a separate script file which we have saved as /share/apps/bin/conda-3.7.3.sh:

$ cat /share/apps/bin/conda-3.7.3.sh
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/share/apps/python/anaconda3-python-3.7.3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/share/apps/python/anaconda3-python-3.7.3/etc/profile.d/conda.sh" ]; then
        . "/share/apps/python/anaconda3-python-3.7.3/etc/profile.d/conda.sh"
    else
        export PATH="/share/apps/python/anaconda3-python-3.7.3/bin:$PATH"
    fi
fi
#unset __conda_setup
# <<< conda initialize <<<

and delete this same code from ~/.bashrc. Then to start Anaconda Python and use conda you can as shown above add one step:

$ module load python/3.7.3-anaconda
$ source /share/apps/bin/conda-3.7.3.sh
(base) $

This is necessary as environment_modules can't do the source or “.” command in conda-3.7.3.sh. If you also use Anaconda Python 2.7.16 you'll need a similar script for it by the same process.

The Intel/Anaconda commands are a bit different using-intel but have similar functions. Intel Python is designed a bit better to not overwrite your ~/.bashrc.

$ conda info --envs
# conda environments:
#
root                  *  /scrfs/apps/intel/parallel_studio_xe_2019_update_4/intelpython3

$ conda create -n idp
Fetching package metadata ...............
Solving package specifications: 
Package plan for installation in environment /home/rfeynman/.conda/envs/idp:

Proceed ([y]/n)? y

#
# To activate this environment, use:
# > source activate idp
#
# To deactivate an active environment, use:
# > source deactivate
#

$ conda info --envs
# conda environments:
#
idp                      /home/rfeynman/.conda/envs/idp
root                  *  /scrfs/apps/intel/parallel_studio_xe_2019_update_4/intelpython3

$ source activate idp
(idp) $ 

pip is a similar utility program for compiled Python, and also works on Anaconda Python. We don't recommend its use on Anaconda Python unless conda won't do what you want.

Here is a compiled version of Python specifically to run phonopy so it has very few Python modules.

$ module load python/2.7.15b
$ pip list
Package                       Version     
----------------------------- ------------
backports.functools-lru-cache 1.5         
cycler                        0.10.0      
h5py                          2.8.0       
kiwisolver                    1.0.1       
matplotlib                    2.2.3       
numpy                         1.14.5      
phono3py                      1.14.3.post1
phonopy                       1.14.2      
pip                           18.1        
pyparsing                     2.3.0       
python-dateutil               2.7.5       
pytz                          2018.7      
PyYAML                        3.13        
setuptools                    39.0.1      
six                           1.11.0      
subprocess32                  3.5.3       
$

The environment module for compiled Python sets the environment variable $PYTHONUSERBASE to a subdirectory of your home called python{major version} such as ~/python3.6 to which you can add your own user modules. If necessary you can change the variable after loading the module. To install user modules, first create the directory

cd ~
mkdir -p python3.6

To add a module not in the pip list,

pip install peppercorn --user

To add a module with a different version than the pip list:

pip install lxml --user --ignore-installed

To show your local modules, which will be used preferentially to the system modules:

$ pip list --user
lxml (3.6.0)
peppercorn (0.5)

These local Python modules will then be automatically substituted whenever you run Python 3.6.x, and you can control the local modules with pip install, pip update, and pip remove. If you don't want this substitution, redefine $PYTHONUSERBASE.