==== Python Versions ====
AHPCC maintains Python versionsf rom source 2.7.5, 2.7.11, and 3.5.2. 2.7.11 is most used.
[[https://www.continuum.io/?|Anaconda Python]] 2.7.13 is installed on razor.
[[https://software.intel.com/en-us/intel-distribution-for-python|Intel Python]] 2.7.13 is installed on razor.
"python" at the prompt (with no python enviroment modules loaded) defaults to /usr/bin/python, which is the centos rpm installation of Python 2.6 with very few Python modules (installable code used with "import" from Python source). Use environment modules ("module" command from Linux) to load the compiled version of Python 2.7 or Python 3. Please contact hpc-support to request install/upgrade of Python modules, or you can install a private library with PYTHONUSERBASE, virtualenv, or anaconda (see below). You can also install a full version of Python in your user space, but it is quite large and takes a long time to compile.
==trestles and razor python/2.7.11==
(optional) module load gcc/4.9.1 mkl/16.0.1 mvapich2/2.2
mkl is needed for scipy/numpy, and mpi is needed for mpi4py. gcc is probably only needed to install python modules.
module load python/2.7.11
==pip installed modules (2.7.11)==
trestles razor
-------- -----
alabaster (0.7.7) alabaster (0.7.7)
avro (1.8.1) avro (1.8.1)
Babel (2.2.0) Babel (2.2.0)
backports-abc (0.4) | backports.typing (1.1)
backports.shutil-get-terminal-size (1.0.0) | biopython (1.66)
backports.ssl-match-hostname (3.5.0.1) <
biopython (1.68) <
bx-python (0.7.3) <
bz2file (0.98) bz2file (0.98)
CacheControl (0.11.7) | click (6.7)
certifi (2016.2.28) <
cloudpickle (0.2.1) cloudpickle (0.2.1)
cogent (1.5.3) cogent (1.5.3)
> configobj (5.0.6)
configparser (3.3.0.post2) configparser (3.3.0.post2)
ConsensusCore (1.0.2) ConsensusCore (1.0.2)
ConsensusCore2 (2.0.5) | cram (0.7)
contextlib2 (0.5.4) <
coverage (4.2) <
cycler (0.10.0) cycler (0.10.0)
Cython (0.25.2) | Cython (0.24.1)
dammit (0.2.5) dammit (0.2.5)
dask (0.9.0) | decorator (4.0.9)
decorator (4.0.10) <
deepTools (2.2.4) <
DendroPy (4.2.0) <
Django (1.9.2) Django (1.9.2)
docutils (0.12) docutils (0.12)
doit (0.29.0) doit (0.29.0)
enum34 (1.1.6) <
fastcluster (1.1.22) <
ficus (0.1) ficus (0.1)
funcsigs (1.0.2) funcsigs (1.0.2)
functools32 (3.2.3.post2) functools32 (3.2.3.post2)
future (0.16.0) <
futures (3.0.5) <
GenomicConsensus (2.1.0) GenomicConsensus (2.1.0)
> globus-cli (0.4.6.0)
> globus-sdk (0.4.6)
h5py (2.6.0) h5py (2.6.0)
HTSeq (0.6.1) HTSeq (0.6.1)
image (1.4.4) image (1.4.4)
ipykernel (4.3.1) <
ipython (5.1.0) <
ipython-genutils (0.1.0) <
iso8601 (0.1.11) iso8601 (0.1.11)
Jinja2 (2.8) Jinja2 (2.8)
jsonschema (2.5.1) jsonschema (2.5.1)
jupyter-client (4.2.2) <
jupyter-core (4.1.0) <
Keras (2.0.0) Keras (2.0.0)
khmer (2.0) khmer (2.0)
lockfile (0.12.2) | Mako (1.0.4)
MarkupSafe (0.23) MarkupSafe (0.23)
matplotlib (1.5.3) | matplotlib (1.5.1)
mock (2.0.0) mock (2.0.0)
mpi4py (2.0.0) mpi4py (2.0.0)
mpmath (0.19) | mxnet (0.7.0)
MySQL-python (1.2.5) MySQL-python (1.2.5)
natsort (5.0.1) <
networkx (1.11) networkx (1.11)
nose (1.3.7) | ngs (1.0)
> nose (1.3.4)
nose-capturestderr (1.0) nose-capturestderr (1.0)
numexpr (2.6.1) | numexpr (2.5)
numpy (1.13.0.dev0+95e935f) | numpy (1.11.3)
numpydoc (0.6.0) | pandas (0.17.1)
pandas (0.19.2) | passlib (1.6.5)
> PasteDeploy (1.5.2)
path.py (8.1.2) path.py (8.1.2)
pathlib2 (2.2.0) <
patsy (0.4.1) patsy (0.4.1)
pbalign (0.3.0) pbalign (0.3.0)
pbcommand (0.4.14) | pbcommand (0.4.13)
pbcore (1.2.12) | pbcore (1.2.11)
pbr (1.10.0) pbr (1.10.0)
pbtestdata (0.2.1) | Pillow (3.1.1)
pexpect (4.2.1) <
phonopy (1.11.2.80) <
phylo-utils (0.0.5) <
pickleshare (0.7.4) <
Pillow (3.2.0) <
pip (9.0.1) pip (9.0.1)
progressbar-latest (2.4) | protobuf (3.1.0.post1)
prompt-toolkit (1.0.9) | Pygments (2.1.1)
protobuf (3.0.0b2) | pygpu (0.2.1)
ptyprocess (0.5.1) <
pyBigWig (0.2.8) <
pyfasta (0.5.2) <
Pygments (2.1.3) <
pyinotify (0.9.6) pyinotify (0.9.6)
pylab (0.1.3) | pyparsing (2.1.0)
pylibcurl (0.7.12) <
pyparsing (2.1.10) <
pypng (0.0.18) pypng (0.0.18)
pysam (0.9.0) | pyramid (1.7.3)
> pyramid-services (0.4)
> pysam (0.9.1.4)
pysqlite (2.8.1) pysqlite (2.8.1)
python-dateutil (2.6.0) | python-dateutil (2.4.2)
python-qt (0.50) <
pythonbrew (1.3.6) pythonbrew (1.3.6)
pytz (2016.10) | pytz (2016.3)
PyYAML (3.12) | PyYAML (3.11)
pyzmq (15.2.0) | repoze.lru (0.6)
requests (2.12.4) | requests (2.11.1)
scandir (1.4) | scikit-learn (0.17.1)
scikit-bio (0.4.2) | scipy (0.15.1)
scikit-image (0.12.3) <
scikit-learn (0.18.1) <
scipy (0.18.1) <
screed (0.9) screed (0.9)
seaborn (0.7.0) <
setuptools (32.3.1) setuptools (32.3.1)
simplegeneric (0.8.1) simplegeneric (0.8.1)
singledispatch (3.4.0.3) <
six (1.10.0) six (1.10.0)
sklearn (0.0) sklearn (0.0)
snap (1.2-dev-centos6.6-x64-py2.7) snap (1.2-dev-centos6.6-x64-py2.7)
snowballstemmer (1.2.1) snowballstemmer (1.2.1)
Sphinx (1.3.5) Sphinx (1.3.5)
sphinx-rtd-theme (0.1.9) sphinx-rtd-theme (0.1.9)
SQLAlchemy (1.0.11) SQLAlchemy (1.0.11)
statsmodels (0.8.0) statsmodels (0.8.0)
sympy (1.0) <
tables (3.2.0) tables (3.2.0)
tensorflow (0.10.0rc0) | tensorflow (0.12.1)
> tet (0.3)
TEToolkit (1.5.0) TEToolkit (1.5.0)
toolz (0.7.4) | Theano (0.8.2)
tornado (4.3) | translationstring (1.3)
traitlets (4.3.1) | typing (3.5.2.2)
tree-distance (1.0.6) <
treeCl (0.1.29) <
urllib3 (1.14) urllib3 (1.14)
wcwidth (0.1.7) | venusian (1.0)
virtualenv (15.1.0) virtualenv (15.0.1)
> WebOb (1.6.1)
wget (3.2) wget (3.2)
wheel (0.29.0) wheel (0.29.0)
xmlbuilder (1.0) <
yolk3k (0.8.8) yolk3k (0.8.8)
| zope.deprecation (4.1.2)
| zope.interface (4.3.2)
==razor python/2.7.5==
(optional) module load gcc/4.6.3 mkl/16.0.1 impi/4.0.3
(optional for khmer) gcc/5.2.1 mkl/16.0.1 impi/4.0.3
(note) MKL version changed to 16.0.1 to avoid segfaults in numpy
module load python/2.7.5
==trestles python/2.7.5==
is the rpm python27 version so has no prerequisites
module load python/2.7.5
==pip installed modules (2.7.5)==
trestles razor
-------- -----
Babel (0.9.6) | agalma (0.4.0)
bx-python (0.7.3) | alabaster (0.7.7)
> anvio (1.2.2)
> Babel (2.2.0)
> backports.ssl-match-hostname (3.4.0.2)
> bcbio-gff (0.6.2)
> biolite (0.4.0)
> biom-format (2.1.5)
> biopython (1.64)
> bottle (0.12.9)
> burrito (0.9.1)
> burrito-fillings (0.1.1)
> bx-python (0.7.1)
> bz2file (0.98)
> certifi (2016.2.28)
> cffi (1.5.2)
> click (6.3)
> cloudpickle (0.2.1)
> cogent (1.5.3)
> configparser (3.3.0.post2)
> coverage (3.6)
> cryptography (1.3.1)
cycler (0.10.0) cycler (0.10.0)
Cython (0.24) | Cython (0.21.1)
> dammit (0.2.5)
> decorator (4.0.9)
deepTools (2.2.4) deepTools (2.2.4)
docutils (0.11) | DendroPy (3.12.0)
HTSeq (0.6.1p1) | Django (1.8.3)
Jinja2 (2.6) | docutils (0.12)
MarkupSafe (0.11) | doit (0.29.0)
> emperor (0.9.51)
> enum34 (1.1.3)
> ete2 (2.3.10)
> ficus (0.1)
> fisher (0.1.4)
> future (0.15.2)
> gdata (2.0.18)
> gurobipy (4.0.0)
> h5py (2.6.0)
> hcluster (0.2.0)
> HTSeq (0.6.1)
> idna (2.1)
> image (1.4.0)
> ipaddress (1.0.16)
> ipython (4.1.1)
> ipython-genutils (0.1.0)
> Jinja2 (2.8)
> khmer (2.0)
> lxml (3.6.0)
> MarkupSafe (0.23)
matplotlib (1.5.1) matplotlib (1.5.1)
nose (1.3.0) | mock (1.0.1)
> mpi4py (1.3.1)
> natsort (3.5.6)
> ndg-httpsclient (0.4.0)
> networkx (1.9)
> nose (1.3.4)
> nose-capturestderr (1.0)
> Numeric (24.2)
> numexpr (2.5)
numpy (1.11.0) numpy (1.11.0)
numpydoc (0.6.0) | pandas (0.17.1)
pip (8.1.2) | path.py (8.1.2)
protobuf (3.0.0b2) | peppercorn (0.5)
> pexpect (4.0.1)
> pickleshare (0.6)
> Pillow (2.8.1)
> pip (8.1.1)
> Pmw (1.3)
> ptyprocess (0.5.1)
> PuLP (1.5.4)
> pyasn1 (0.1.9)
pyBigWig (0.2.8) pyBigWig (0.2.8)
Pygments (1.5) | pycparser (2.14)
pyparsing (2.1.4) | pyfasta (0.5.2)
> PyFoam (0.6.2)
> Pygments (2.1.1)
> pyinotify (0.9.6)
> pynast (1.2.2)
> pyOpenSSL (16.0.0)
> pyparsing (1.5.7)
> pyqi (0.3.2)
pysam (0.9.0) pysam (0.9.0)
python-dateutil (2.5.3) | pysqlite (2.6.3)
pytz (2016.4) | python-dateutil (2.4.2)
> pytz (2016.3)
> PyYAML (3.11)
> qcli (0.1.1)
> qiime (1.9.1)
> qiime-default-reference (0.1.3)
> scikit-bio (0.2.3)
> scikit-learn (0.17.1)
> scikits.statsmodels (0.3.1)
scipy (0.17.1) scipy (0.17.1)
setuptools (0.9.8) | screed (0.9)
simplejson (3.2.0) | setuptools (20.7.0)
> simplegeneric (0.8.1)
six (1.10.0) six (1.10.0)
Sphinx (1.1.3) | snap (1.1-dev-redenterprise-x64-py2.7)
SQLAlchemy (0.7.9) | snowballstemmer (1.2.1)
tensorflow (0.9.0rc0) | spanki (0.5.0, /share/apps/Python/2.7.5/lib/...
virtualenv (1.10.1) | Sphinx (1.3.5)
Werkzeug (0.8.3) | sphinx-rtd-theme (0.1.9)
wheel (0.29.0) | tornado (4.0.2)
> traitlets (4.1.0)
> urllib3 (1.14)
> virtualenv (15.0.1)
> wget (2.2)
> yolk3k (0.8.8
==PYTHONUSERBASE==
The Python-from-source modules set the environment variable
PYTHONUSERBASE=~/python2.7
or
PYTHONUSERBASE=~/python3.5
so you can add your own user modules. If necessary you can change the variable after loading the module. To install user modules, first create the directory
cd ~
mkdir -p python2.7
To add a module not in the pip list,
pip install peppercorn --user
To add a module with a different version than the pip list:
pip install lxml --user --ignore-installed
To show your local modules, which will be used preferentially to the system modules:
$ pip list --user
lxml (3.6.0)
peppercorn (0.5)
==virtualenv==
virtualenv is useful for a custom environment or where there is a module incompatibility with the system version. Create and activate like the following, install the modules you want, and activate in the batch job before using. This example uses the system installation as a base.
tres-l1:rfeynman:$ module load gcc/4.9.1 mkl/16.0.1 mvapich2/2.2 python/2.7.11
tres-l1:rfeynman:$ which python
/share/apps/python/2.7.11/bin/python
tres-l1:rfeynman:$ virtualenv -p /share/apps/python/2.7.11 /storage/rfeynman/virtualenv
Running virtualenv with interpreter /share/apps/python/2.7.11/bin/python
PYTHONHOME is set. You *must* activate the virtualenv before using it
New python executable in /storage/rfeynman/virtualenv/bin/python
Installing setuptools, pip, wheel...done.
tres-l1:rfeynman:$ export VIRTUALENVWRAPPER_PYTHON=/share/apps/python/2.7.11
tres-l1:rfeynman:$ source /storage/rfeynman/virtualenv/bin/activate
(virtualenv) tres-l1:rfeynman:$ which python
/storage/rfeynman/virtualenv/bin/python
(virtualenv) tres-l1:rfeynman:$ pip list
appdirs (1.4.3)
packaging (16.8)
pip (9.0.1)
pyparsing (2.2.0)
setuptools (34.3.2)
six (1.10.0)
wheel (0.29.0)
==Anaconda Python==
Anaconda is a prepackaged distribution of Python. Private installations can be made through conda environments similar to virtualenv environments. Anaconda and Intel python can also use pip --user, but all python2.7 versions will install to the same user directory /home/$USER/python2.7 .
see [[https://conda.io/docs/using/cheatsheet.html]]
$ module load python/2.7.13-anaconda (or python/3.6.0-anaconda )
$ python -V
Python 2.7.13 :: Anaconda 4.3.1 (64-bit)
$ conda list
# packages in environment at /share/apps/python/Anaconda2:
#
_license 1.1 py27_1
alabaster 0.7.9 py27_0
anaconda 4.3.1 np111py27_0
etc.
$ conda create --prefix ~/bio-env biopython
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/rfeynman/bio-env:
The following NEW packages will be INSTALLED:
biopython: 1.68-np112py27_0
mkl: 2017.0.1-0
numpy: 1.12.1-py27_0
openssl: 1.0.2k-1
pip: 9.0.1-py27_1
python: 2.7.13-0
readline: 6.2-2
setuptools: 27.2.0-py27_0
sqlite: 3.13.0-0
tk: 8.5.18-0
wheel: 0.29.0-py27_0
zlib: 1.2.8-3
Proceed ([y]/n)? y
$
...etc...
$ source activate /home/rfeynman/bio-env
==Intel Python==
Intel supplies another version of Anaconda python, supposedly optimized for runtime speed on Intel processors. It has about 130 conda modules installed.
$ module load python/2.7.13-intel
$ source activate root
(root) $ conda list
# packages in environment at /share/apps/intelpython2:
#
backports 1.0 py27_intel_6 [intel] intel
backports_abc 0.5 py27_intel_0 [intel] intel
bleach 1.5.0 py27_intel_0 [intel] intel
boost 1.61.0 py27_intel_3 [intel] intel
bzip2 1.0.6 intel_10 [intel] intel
caffe 1.0.0rc3 py27_intel_4 [intel] intel
etc.
==2.7.11 source build, python and snap.py==
An example for modules that need configure customization, or won't install from pip.
module load gcc/4.9.1
module load mkl/16.0.1
module load mvapich2/2.2
export LD_RUN_PATH=$MKL_ROOT/lib/intel64:$LD_LIBRARY_PATH
cd install_dir
tar -Jxf Python-2.7.11.tar.zx
cd Python-2.7.11
./configure --prefix=/share/apps/python/2.7.11 --with-ensurepip --with-threads --disable-ipv6
make && make install
cd ..
wget https://snap.stanford.edu/snappy/release/snap-1.2-2.4-centos6.5-x64-py2.6.tar.gz
tar zxf snap*gz
cd snap*6
python setup.py install
==custom numpy build with MKL/FFTW3==
These libraries greatly improve numpy/scipy performance.
$ cd $PYTHONPATH/lib/python2.7/site-packages
$ tar zxf numpy-1.11.0.tar.gz
$ cd numpy-1.11.0
$ tail site.cfg
[fftw]
libraries = fftw3
include_dirs = /share/apps/fftw/fftw-3.3.3/gcc-4.6.3/include
library_dirs = /share/apps/fftw/fftw-3.3.3/gcc-4.6.3/lib
[mkl]
include_dirs = /share/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/include
library_dirs = /share/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64:\
/share/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64:/usr/lib64
lapack_libs = mkl_lapack95_ilp64
mkl_libs = mkl_gf_ilp64, mkl_blas95_ilp64, mkl_def, mkl_gnu_thread, mkl_core, iomp5, pthread
$ export LD_RUN_PATH=/share/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64:\
/share/apps/fftw/fftw-3.3.3/gcc-4.6.3/lib:\
/share/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64
$ python setup.py develop build config build_ext -j 4
$ pip install -e .
==Multithreaded MKL with python example==
This is a matrix multiplication of two square matrices O(4096) through (1) default python 2.6 with numpy and multithreaded Atlas blas from RPM (number of threads set at compile time and not controllable) (2) to (4) numpy and MKL with unset,1, and 16 MKL threads respectively. Outputs size, time, GFlopRate. You can see MKL_NUM_THREADS unset acts as if it is set to the number of cores in the computer, and that MKL is around 6 times faster than Atlas on this problem with maximum threading.
$ module purge
$ python test.numpy.py
4096, 6.697931, 20.51711
$ module load gcc/4.9.1 mkl/16.0.1 mvapich2/2.2 python/2.7.11
$ python test.numpy.py
4096, 1.135037, 121.0729
$ export MKL_NUM_THREADS=1
$ python test.numpy.py
4096, 16.43035, 8.363921
$ export MKL_NUM_THREADS=16
$ python test.numpy.py
4096, 1.130565, 121.5518