Arkansas High Performace Computing Center [hpcwiki]

AHPCC maintains a number of Python versions: Anaconda, Intel/Anaconda, and compiled from source.

Terminology: Python modules are prewritten programs in Python. environment_modules are scripts on the system that set up different versions of software.

Anaconda Python is what we recommend for most purposes as it has wide compatibility and easy installation with Python modules. Recommended environment_modules are python/2.7.16-anaconda and 3.7.3-anaconda.

Intel Python is a version of Anaconda optimized for speed and is a good choice for general Python code that doesn't need a great number of Python modules. Recommended environment_modules are python/2.7.16-intel and 3.6.8-intel.
Python compiled from source will sometimes install Python modules that won't install in Anaconda. Recommended environment_modules are python/2.7.15 and 3.6.0.

If you type python at the shell without loading any environment_modules, you will get Centos system Python, which is quite old and unsuitable for all but the simplest uses, but needs to be in the default path for some system programs to run.

You can install and use custom python environments using conda for Anaconda Python and using pip for Anaconda and compiled. conda is more convenient.

conda is a utility program for Anaconda Python, see getting-started. It is most commonly used as conda list to show the Python modules in the current environment.

Some examples of conda commmands are shown below. This shows environments or subversions that have different Python module configurations. If unspecified you will be using the starred “base” environment. conda info also finds a base environment for Python2 (not usable here in Python3). There are two custom environments that have been defined by the system (tfgpu and theano-env), and the user running this doesn't have any environments yet. <code> $ module load python/3.7.3-anaconda $ source conda-3.7.3.sh (base) $ conda info –envs # conda environments: # /scrfs/apps/python/anaconda2-2.7.16 base * /share/apps/python/anaconda3-python-3.7.3 tfgpu /share/apps/python/anaconda3-python-3.7.3/envs/tfgpu theano-env /share/apps/python/anaconda3-python-3.7.3/envs/theano-env (base) $ conda activate tfgpu (tfgpu) $ conda list # packages in environment at /share/apps/python/anaconda3-python-3.7.3/envs/tfgpu: # # Name Version Build Channel libgccmutex 0.1 main
tflowselect 2.1.0 gpu
etc. </code> If this enviroment tf_gpu had been your own enviroment, you could also do commands such as conda install, conda remove, and conda update. See manage-environments for conda create to create your own environments, normally stored under your home in ~/.conda/envs/. The first time you run conda activate you should get a message like: <code> CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. To initialize your shell, run $ conda init <SHELLNAME> Currently supported shells are: - bash - fish - tcsh - xonsh - zsh - powershell See 'conda init –help' for more information and options. IMPORTANT: You may need to close and restart your shell after running 'conda init'. </code> If you then do conda activate bash (once) you will see that conda has added some bash code to your ~/.bashrc initialization script. This is acceptable if all the python you ever run is only Anaconda Python2 or only Anaconda Python3, and in that case you can just leave the modification alone. If you ever use other versions of Python, it is likely to cause problems. In that case, you can edit ~/.bashrc and save the new code section at the end (delimited by comments condainitialize) as a separate script file such as conda-3.7.3.sh: <code> $ cat conda-3.7.3.sh # »> conda initialize »> # !! Contents within this block are managed by 'conda init' !! condasetup=“$('/share/apps/python/anaconda3-python-3.7.3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)” if [ $? -eq 0 ]; then eval “$condasetup” else if [ -f “/share/apps/python/anaconda3-python-3.7.3/etc/profile.d/conda.sh” ]; then . “/share/apps/python/anaconda3-python-3.7.3/etc/profile.d/conda.sh” else export PATH=“/share/apps/python/anaconda3-python-3.7.3/bin:$PATH” fi fi #unset __condasetup # «< conda initialize «< </code> and delete this same code from ~/.bashrc. Then to start Anaconda Python and use conda you can as shown above add one step: <code> $ module load python/3.7.3-anaconda $ source conda-3.7.3.sh (base) $ </code> This is necessary as environment_modules can't do the source or “.” command in conda-3.7.3.sh. If you also use Anaconda Python 2.7.16 you'll need a similar script for it by the same process. ==intel== The Intel/Anaconda commands are a bit different using-intel but have similar functions. Intel Python is designed a bit better to not overwrite your ~/.bashrc. <code> $ conda info –envs # conda environments: # root * /scrfs/apps/intel/parallelstudioxe2019update_4/intelpython3 $ conda create -n idp Fetching package metadata …………… Solving package specifications: Package plan for installation in environment /home/rfeynman/.conda/envs/idp: Proceed ([y]/n)? y # # To activate this environment, use: # > source activate idp # # To deactivate an active environment, use: # > source deactivate # $ conda info –envs # conda environments: # idp /home/rfeynman/.conda/envs/idp root * /scrfs/apps/intel/parallelstudioxe2019update_4/intelpython3 $ source activate idp (idp) $ </code> ==pip== pip is a similar utility program for compiled Python, and also works on Anaconda Python. We don't recommend its use on Anaconda Python unless conda won't do what you want. Here is a compiled version of Python specifically to run phonopy so it has very few Python modules. <code> $ module load python/2.7.15b $ pip list Package Version
—————————– ———— backports.functools-lru-cache 1.5
cycler 0.10.0
h5py 2.8.0
kiwisolver 1.0.1
matplotlib 2.2.3
numpy 1.14.5
phono3py 1.14.3.post1 phonopy 1.14.2
pip 18.1
pyparsing 2.3.0
python-dateutil 2.7.5
pytz 2018.7
PyYAML 3.13
setuptools 39.0.1
six 1.11.0
subprocess32 3.5.3
$ </code> The environment module for compiled Python sets the environment variable PYTHONUSERBASE to a subdirectory of your home called python{major version} such as ~/python3.6 to which you can add your own user modules. If necessary you can change the variable after loading the module. To install user modules, first create the directory <code> cd ~ mkdir -p python3.6 </code> To add a module not in the pip list, <code> pip install peppercorn –user </code> To add a module with a different version than the pip list: <code> pip install lxml –user –ignore-installed </code> To show your local modules, which will be used preferentially to the system modules: <code> $ pip list –user lxml (3.6.0) peppercorn (0.5) </code> These local Python modules will then be automatically substituted whenever you run Python 3.6.x, and you can control the local modules with pip install,pip update, and pip remove. If you don't want this done, redefine $PYTHONUSERBASE.