==== Python Versions ====
AHPCC maintains Python versionsf rom source 2.7.5, 2.7.11, and 3.5.2. 2.7.11 is most used. Anaconda Python 2.7.13 is installed on razor. Intel Python 2.7.13 is installed on razor.
“python” at the prompt (with no python enviroment modules loaded) defaults to /usr/bin/python, which is the centos rpm installation of Python 2.6 with very few Python modules (installable code used with “import” from Python source). Use environment modules (“module” command from Linux) to load the compiled version of Python 2.7 or Python 3. Please contact hpc-support to request install/upgrade of Python modules, or you can install a private library with PYTHONUSERBASE, virtualenv, or anaconda (see below). You can also install a full version of Python in your user space, but it is quite large and takes a long time to compile.
trestles and razor python/2.7.11
(optional) module load gcc/4.9.1 mkl/16.0.1 mvapich2/2.2
mkl is needed for scipy/numpy, and mpi is needed for mpi4py. gcc is probably only needed to install python modules.
module load python/2.7.11
pip installed modules (2.7.11)
trestles razor -------- ----- alabaster (0.7.7) alabaster (0.7.7) avro (1.8.1) avro (1.8.1) Babel (2.2.0) Babel (2.2.0) backports-abc (0.4) | backports.typing (1.1) backports.shutil-get-terminal-size (1.0.0) | biopython (1.66) backports.ssl-match-hostname (3.5.0.1) < biopython (1.68) < bx-python (0.7.3) < bz2file (0.98) bz2file (0.98) CacheControl (0.11.7) | click (6.7) certifi (2016.2.28) < cloudpickle (0.2.1) cloudpickle (0.2.1) cogent (1.5.3) cogent (1.5.3) > configobj (5.0.6) configparser (3.3.0.post2) configparser (3.3.0.post2) ConsensusCore (1.0.2) ConsensusCore (1.0.2) ConsensusCore2 (2.0.5) | cram (0.7) contextlib2 (0.5.4) < coverage (4.2) < cycler (0.10.0) cycler (0.10.0) Cython (0.25.2) | Cython (0.24.1) dammit (0.2.5) dammit (0.2.5) dask (0.9.0) | decorator (4.0.9) decorator (4.0.10) < deepTools (2.2.4) < DendroPy (4.2.0) < Django (1.9.2) Django (1.9.2) docutils (0.12) docutils (0.12) doit (0.29.0) doit (0.29.0) enum34 (1.1.6) < fastcluster (1.1.22) < ficus (0.1) ficus (0.1) funcsigs (1.0.2) funcsigs (1.0.2) functools32 (3.2.3.post2) functools32 (3.2.3.post2) future (0.16.0) < futures (3.0.5) < GenomicConsensus (2.1.0) GenomicConsensus (2.1.0) > globus-cli (0.4.6.0) > globus-sdk (0.4.6) h5py (2.6.0) h5py (2.6.0) HTSeq (0.6.1) HTSeq (0.6.1) image (1.4.4) image (1.4.4) ipykernel (4.3.1) < ipython (5.1.0) < ipython-genutils (0.1.0) < iso8601 (0.1.11) iso8601 (0.1.11) Jinja2 (2.8) Jinja2 (2.8) jsonschema (2.5.1) jsonschema (2.5.1) jupyter-client (4.2.2) < jupyter-core (4.1.0) < Keras (2.0.0) Keras (2.0.0) khmer (2.0) khmer (2.0) lockfile (0.12.2) | Mako (1.0.4) MarkupSafe (0.23) MarkupSafe (0.23) matplotlib (1.5.3) | matplotlib (1.5.1) mock (2.0.0) mock (2.0.0) mpi4py (2.0.0) mpi4py (2.0.0) mpmath (0.19) | mxnet (0.7.0) MySQL-python (1.2.5) MySQL-python (1.2.5) natsort (5.0.1) < networkx (1.11) networkx (1.11) nose (1.3.7) | ngs (1.0) > nose (1.3.4) nose-capturestderr (1.0) nose-capturestderr (1.0) numexpr (2.6.1) | numexpr (2.5) numpy (1.13.0.dev0+95e935f) | numpy (1.11.3) numpydoc (0.6.0) | pandas (0.17.1) pandas (0.19.2) | passlib (1.6.5) > PasteDeploy (1.5.2) path.py (8.1.2) path.py (8.1.2) pathlib2 (2.2.0) < patsy (0.4.1) patsy (0.4.1) pbalign (0.3.0) pbalign (0.3.0) pbcommand (0.4.14) | pbcommand (0.4.13) pbcore (1.2.12) | pbcore (1.2.11) pbr (1.10.0) pbr (1.10.0) pbtestdata (0.2.1) | Pillow (3.1.1) pexpect (4.2.1) < phonopy (1.11.2.80) < phylo-utils (0.0.5) < pickleshare (0.7.4) < Pillow (3.2.0) < pip (9.0.1) pip (9.0.1) progressbar-latest (2.4) | protobuf (3.1.0.post1) prompt-toolkit (1.0.9) | Pygments (2.1.1) protobuf (3.0.0b2) | pygpu (0.2.1) ptyprocess (0.5.1) < pyBigWig (0.2.8) < pyfasta (0.5.2) < Pygments (2.1.3) < pyinotify (0.9.6) pyinotify (0.9.6) pylab (0.1.3) | pyparsing (2.1.0) pylibcurl (0.7.12) < pyparsing (2.1.10) < pypng (0.0.18) pypng (0.0.18) pysam (0.9.0) | pyramid (1.7.3) > pyramid-services (0.4) > pysam (0.9.1.4) pysqlite (2.8.1) pysqlite (2.8.1) python-dateutil (2.6.0) | python-dateutil (2.4.2) python-qt (0.50) < pythonbrew (1.3.6) pythonbrew (1.3.6) pytz (2016.10) | pytz (2016.3) PyYAML (3.12) | PyYAML (3.11) pyzmq (15.2.0) | repoze.lru (0.6) requests (2.12.4) | requests (2.11.1) scandir (1.4) | scikit-learn (0.17.1) scikit-bio (0.4.2) | scipy (0.15.1) scikit-image (0.12.3) < scikit-learn (0.18.1) < scipy (0.18.1) < screed (0.9) screed (0.9) seaborn (0.7.0) < setuptools (32.3.1) setuptools (32.3.1) simplegeneric (0.8.1) simplegeneric (0.8.1) singledispatch (3.4.0.3) < six (1.10.0) six (1.10.0) sklearn (0.0) sklearn (0.0) snap (1.2-dev-centos6.6-x64-py2.7) snap (1.2-dev-centos6.6-x64-py2.7) snowballstemmer (1.2.1) snowballstemmer (1.2.1) Sphinx (1.3.5) Sphinx (1.3.5) sphinx-rtd-theme (0.1.9) sphinx-rtd-theme (0.1.9) SQLAlchemy (1.0.11) SQLAlchemy (1.0.11) statsmodels (0.8.0) statsmodels (0.8.0) sympy (1.0) < tables (3.2.0) tables (3.2.0) tensorflow (0.10.0rc0) | tensorflow (0.12.1) > tet (0.3) TEToolkit (1.5.0) TEToolkit (1.5.0) toolz (0.7.4) | Theano (0.8.2) tornado (4.3) | translationstring (1.3) traitlets (4.3.1) | typing (3.5.2.2) tree-distance (1.0.6) < treeCl (0.1.29) < urllib3 (1.14) urllib3 (1.14) wcwidth (0.1.7) | venusian (1.0) virtualenv (15.1.0) virtualenv (15.0.1) > WebOb (1.6.1) wget (3.2) wget (3.2) wheel (0.29.0) wheel (0.29.0) xmlbuilder (1.0) < yolk3k (0.8.8) yolk3k (0.8.8) | zope.deprecation (4.1.2) | zope.interface (4.3.2)
razor python/2.7.5
(optional) module load gcc/4.6.3 mkl/16.0.1 impi/4.0.3
(optional for khmer) gcc/5.2.1 mkl/16.0.1 impi/4.0.3
(note) MKL version changed to 16.0.1 to avoid segfaults in numpy
module load python/2.7.5
trestles python/2.7.5
is the rpm python27 version so has no prerequisites
module load python/2.7.5
pip installed modules (2.7.5)
trestles razor -------- ----- Babel (0.9.6) | agalma (0.4.0) bx-python (0.7.3) | alabaster (0.7.7) > anvio (1.2.2) > Babel (2.2.0) > backports.ssl-match-hostname (3.4.0.2) > bcbio-gff (0.6.2) > biolite (0.4.0) > biom-format (2.1.5) > biopython (1.64) > bottle (0.12.9) > burrito (0.9.1) > burrito-fillings (0.1.1) > bx-python (0.7.1) > bz2file (0.98) > certifi (2016.2.28) > cffi (1.5.2) > click (6.3) > cloudpickle (0.2.1) > cogent (1.5.3) > configparser (3.3.0.post2) > coverage (3.6) > cryptography (1.3.1) cycler (0.10.0) cycler (0.10.0) Cython (0.24) | Cython (0.21.1) > dammit (0.2.5) > decorator (4.0.9) deepTools (2.2.4) deepTools (2.2.4) docutils (0.11) | DendroPy (3.12.0) HTSeq (0.6.1p1) | Django (1.8.3) Jinja2 (2.6) | docutils (0.12) MarkupSafe (0.11) | doit (0.29.0) > emperor (0.9.51) > enum34 (1.1.3) > ete2 (2.3.10) > ficus (0.1) > fisher (0.1.4) > future (0.15.2) > gdata (2.0.18) > gurobipy (4.0.0) > h5py (2.6.0) > hcluster (0.2.0) > HTSeq (0.6.1) > idna (2.1) > image (1.4.0) > ipaddress (1.0.16) > ipython (4.1.1) > ipython-genutils (0.1.0) > Jinja2 (2.8) > khmer (2.0) > lxml (3.6.0) > MarkupSafe (0.23) matplotlib (1.5.1) matplotlib (1.5.1) nose (1.3.0) | mock (1.0.1) > mpi4py (1.3.1) > natsort (3.5.6) > ndg-httpsclient (0.4.0) > networkx (1.9) > nose (1.3.4) > nose-capturestderr (1.0) > Numeric (24.2) > numexpr (2.5) numpy (1.11.0) numpy (1.11.0) numpydoc (0.6.0) | pandas (0.17.1) pip (8.1.2) | path.py (8.1.2) protobuf (3.0.0b2) | peppercorn (0.5) > pexpect (4.0.1) > pickleshare (0.6) > Pillow (2.8.1) > pip (8.1.1) > Pmw (1.3) > ptyprocess (0.5.1) > PuLP (1.5.4) > pyasn1 (0.1.9) pyBigWig (0.2.8) pyBigWig (0.2.8) Pygments (1.5) | pycparser (2.14) pyparsing (2.1.4) | pyfasta (0.5.2) > PyFoam (0.6.2) > Pygments (2.1.1) > pyinotify (0.9.6) > pynast (1.2.2) > pyOpenSSL (16.0.0) > pyparsing (1.5.7) > pyqi (0.3.2) pysam (0.9.0) pysam (0.9.0) python-dateutil (2.5.3) | pysqlite (2.6.3) pytz (2016.4) | python-dateutil (2.4.2) > pytz (2016.3) > PyYAML (3.11) > qcli (0.1.1) > qiime (1.9.1) > qiime-default-reference (0.1.3) > scikit-bio (0.2.3) > scikit-learn (0.17.1) > scikits.statsmodels (0.3.1) scipy (0.17.1) scipy (0.17.1) setuptools (0.9.8) | screed (0.9) simplejson (3.2.0) | setuptools (20.7.0) > simplegeneric (0.8.1) six (1.10.0) six (1.10.0) Sphinx (1.1.3) | snap (1.1-dev-redenterprise-x64-py2.7) SQLAlchemy (0.7.9) | snowballstemmer (1.2.1) tensorflow (0.9.0rc0) | spanki (0.5.0, /share/apps/Python/2.7.5/lib/... virtualenv (1.10.1) | Sphinx (1.3.5) Werkzeug (0.8.3) | sphinx-rtd-theme (0.1.9) wheel (0.29.0) | tornado (4.0.2) > traitlets (4.1.0) > urllib3 (1.14) > virtualenv (15.0.1) > wget (2.2) > yolk3k (0.8.8
PYTHONUSERBASE
The Python-from-source modules set the environment variable
PYTHONUSERBASE=~/python2.7 or PYTHONUSERBASE=~/python3.5
so you can add your own user modules. If necessary you can change the variable after loading the module. To install user modules, first create the directory
cd ~ mkdir -p python2.7
To add a module not in the pip list,
pip install peppercorn --user
To add a module with a different version than the pip list:
pip install lxml --user --ignore-installed
To show your local modules, which will be used preferentially to the system modules:
$ pip list --user lxml (3.6.0) peppercorn (0.5)
virtualenv
virtualenv is useful for a custom environment or where there is a module incompatibility with the system version. Create and activate like the following, install the modules you want, and activate in the batch job before using. This example uses the system installation as a base.
tres-l1:rfeynman:$ module load gcc/4.9.1 mkl/16.0.1 mvapich2/2.2 python/2.7.11 tres-l1:rfeynman:$ which python /share/apps/python/2.7.11/bin/python tres-l1:rfeynman:$ virtualenv -p /share/apps/python/2.7.11 /storage/rfeynman/virtualenv Running virtualenv with interpreter /share/apps/python/2.7.11/bin/python PYTHONHOME is set. You *must* activate the virtualenv before using it New python executable in /storage/rfeynman/virtualenv/bin/python Installing setuptools, pip, wheel...done. tres-l1:rfeynman:$ export VIRTUALENVWRAPPER_PYTHON=/share/apps/python/2.7.11 tres-l1:rfeynman:$ source /storage/rfeynman/virtualenv/bin/activate (virtualenv) tres-l1:rfeynman:$ which python /storage/rfeynman/virtualenv/bin/python (virtualenv) tres-l1:rfeynman:$ pip list appdirs (1.4.3) packaging (16.8) pip (9.0.1) pyparsing (2.2.0) setuptools (34.3.2) six (1.10.0) wheel (0.29.0)
Anaconda Python
Anaconda is a prepackaged distribution of Python. Private installations can be made through conda environments similar to virtualenv environments. Anaconda and Intel python can also use pip –user, but all python2.7 versions will install to the same user directory /home/$USER/python2.7 .
see https://conda.io/docs/using/cheatsheet.html
$ module load python/2.7.13-anaconda (or python/3.6.0-anaconda )
$ python -V
Python 2.7.13 :: Anaconda 4.3.1 (64-bit)
$ conda list
# packages in environment at /share/apps/python/Anaconda2:
#
_license 1.1 py27_1
alabaster 0.7.9 py27_0
anaconda 4.3.1 np111py27_0
etc.
$ conda create --prefix ~/bio-env biopython
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /home/rfeynman/bio-env:
The following NEW packages will be INSTALLED:
biopython: 1.68-np112py27_0
mkl: 2017.0.1-0
numpy: 1.12.1-py27_0
openssl: 1.0.2k-1
pip: 9.0.1-py27_1
python: 2.7.13-0
readline: 6.2-2
setuptools: 27.2.0-py27_0
sqlite: 3.13.0-0
tk: 8.5.18-0
wheel: 0.29.0-py27_0
zlib: 1.2.8-3
Proceed ([y]/n)? y
$
...etc...
$ source activate /home/rfeynman/bio-env
Intel Python
Intel supplies another version of Anaconda python, supposedly optimized for runtime speed on Intel processors. It has about 130 conda modules installed.
$ module load python/2.7.13-intel $ source activate root (root) $ conda list # packages in environment at /share/apps/intelpython2: # backports 1.0 py27_intel_6 [intel] intel backports_abc 0.5 py27_intel_0 [intel] intel bleach 1.5.0 py27_intel_0 [intel] intel boost 1.61.0 py27_intel_3 [intel] intel bzip2 1.0.6 intel_10 [intel] intel caffe 1.0.0rc3 py27_intel_4 [intel] intel etc.
2.7.11 source build, python and snap.py
An example for modules that need configure customization, or won't install from pip.
module load gcc/4.9.1 module load mkl/16.0.1 module load mvapich2/2.2 export LD_RUN_PATH=$MKL_ROOT/lib/intel64:$LD_LIBRARY_PATH cd install_dir tar -Jxf Python-2.7.11.tar.zx cd Python-2.7.11 ./configure --prefix=/share/apps/python/2.7.11 --with-ensurepip --with-threads --disable-ipv6 make && make install cd .. wget https://snap.stanford.edu/snappy/release/snap-1.2-2.4-centos6.5-x64-py2.6.tar.gz tar zxf snap*gz cd snap*6 python setup.py install
custom numpy build with MKL/FFTW3
These libraries greatly improve numpy/scipy performance.
$ cd $PYTHONPATH/lib/python2.7/site-packages $ tar zxf numpy-1.11.0.tar.gz $ cd numpy-1.11.0 $ tail site.cfg [fftw] libraries = fftw3 include_dirs = /share/apps/fftw/fftw-3.3.3/gcc-4.6.3/include library_dirs = /share/apps/fftw/fftw-3.3.3/gcc-4.6.3/lib [mkl] include_dirs = /share/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/include library_dirs = /share/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64:\ /share/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64:/usr/lib64 lapack_libs = mkl_lapack95_ilp64 mkl_libs = mkl_gf_ilp64, mkl_blas95_ilp64, mkl_def, mkl_gnu_thread, mkl_core, iomp5, pthread $ export LD_RUN_PATH=/share/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64:\ /share/apps/fftw/fftw-3.3.3/gcc-4.6.3/lib:\ /share/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64 $ python setup.py develop build config build_ext -j 4 $ pip install -e .
Multithreaded MKL with python example
This is a matrix multiplication of two square matrices O(4096) through (1) default python 2.6 with numpy and multithreaded Atlas blas from RPM (number of threads set at compile time and not controllable) (2) to (4) numpy and MKL with unset,1, and 16 MKL threads respectively. Outputs size, time, GFlopRate. You can see MKLNUMTHREADS unset acts as if it is set to the number of cores in the computer, and that MKL is around 6 times faster than Atlas on this problem with maximum threading.
$ module purge $ python test.numpy.py 4096, 6.697931, 20.51711 $ module load gcc/4.9.1 mkl/16.0.1 mvapich2/2.2 python/2.7.11 $ python test.numpy.py 4096, 1.135037, 121.0729 $ export MKL_NUM_THREADS=1 $ python test.numpy.py 4096, 16.43035, 8.363921 $ export MKL_NUM_THREADS=16 $ python test.numpy.py 4096, 1.130565, 121.5518
