User Tools

Site Tools


python
==== Python Versions ====

AHPCC maintains Python versionsf rom source 2.7.5, 2.7.11, and 3.5.2. 2.7.11 is most used. Anaconda Python 2.7.13 is installed on razor. Intel Python 2.7.13 is installed on razor.

“python” at the prompt (with no python enviroment modules loaded) defaults to /usr/bin/python, which is the centos rpm installation of Python 2.6 with very few Python modules (installable code used with “import” from Python source). Use environment modules (“module” command from Linux) to load the compiled version of Python 2.7 or Python 3. Please contact hpc-support to request install/upgrade of Python modules, or you can install a private library with PYTHONUSERBASE, virtualenv, or anaconda (see below). You can also install a full version of Python in your user space, but it is quite large and takes a long time to compile.

trestles and razor python/2.7.11

(optional) module load gcc/4.9.1 mkl/16.0.1 mvapich2/2.2

mkl is needed for scipy/numpy, and mpi is needed for mpi4py. gcc is probably only needed to install python modules.

module load python/2.7.11

pip installed modules (2.7.11)
trestles				        razor
--------				        -----
alabaster (0.7.7)				alabaster (0.7.7)
avro (1.8.1)					avro (1.8.1)
Babel (2.2.0)					Babel (2.2.0)
backports-abc (0.4)			      |	backports.typing (1.1)
backports.shutil-get-terminal-size (1.0.0)    |	biopython (1.66)
backports.ssl-match-hostname (3.5.0.1)	      <
biopython (1.68)			      <
bx-python (0.7.3)			      <
bz2file (0.98)					bz2file (0.98)
CacheControl (0.11.7)			      |	click (6.7)
certifi (2016.2.28)			      <
cloudpickle (0.2.1)				cloudpickle (0.2.1)
cogent (1.5.3)					cogent (1.5.3)
					      >	configobj (5.0.6)
configparser (3.3.0.post2)			configparser (3.3.0.post2)
ConsensusCore (1.0.2)				ConsensusCore (1.0.2)
ConsensusCore2 (2.0.5)			      |	cram (0.7)
contextlib2 (0.5.4)			      <
coverage (4.2)				      <
cycler (0.10.0)					cycler (0.10.0)
Cython (0.25.2)				      |	Cython (0.24.1)
dammit (0.2.5)					dammit (0.2.5)
dask (0.9.0)				      |	decorator (4.0.9)
decorator (4.0.10)			      <
deepTools (2.2.4)			      <
DendroPy (4.2.0)			      <
Django (1.9.2)					Django (1.9.2)
docutils (0.12)					docutils (0.12)
doit (0.29.0)					doit (0.29.0)
enum34 (1.1.6)				      <
fastcluster (1.1.22)			      <
ficus (0.1)					ficus (0.1)
funcsigs (1.0.2)				funcsigs (1.0.2)
functools32 (3.2.3.post2)			functools32 (3.2.3.post2)
future (0.16.0)				      <
futures (3.0.5)				      <
GenomicConsensus (2.1.0)			GenomicConsensus (2.1.0)
					      >	globus-cli (0.4.6.0)
					      >	globus-sdk (0.4.6)
h5py (2.6.0)					h5py (2.6.0)
HTSeq (0.6.1)					HTSeq (0.6.1)
image (1.4.4)					image (1.4.4)
ipykernel (4.3.1)			      <
ipython (5.1.0)				      <
ipython-genutils (0.1.0)		      <
iso8601 (0.1.11)				iso8601 (0.1.11)
Jinja2 (2.8)					Jinja2 (2.8)
jsonschema (2.5.1)				jsonschema (2.5.1)
jupyter-client (4.2.2)			      <
jupyter-core (4.1.0)			      <
Keras (2.0.0)					Keras (2.0.0)
khmer (2.0)					khmer (2.0)
lockfile (0.12.2)			      |	Mako (1.0.4)
MarkupSafe (0.23)				MarkupSafe (0.23)
matplotlib (1.5.3)			      |	matplotlib (1.5.1)
mock (2.0.0)					mock (2.0.0)
mpi4py (2.0.0)					mpi4py (2.0.0)
mpmath (0.19)				      |	mxnet (0.7.0)
MySQL-python (1.2.5)				MySQL-python (1.2.5)
natsort (5.0.1)				      <
networkx (1.11)					networkx (1.11)
nose (1.3.7)				      |	ngs (1.0)
					      >	nose (1.3.4)
nose-capturestderr (1.0)			nose-capturestderr (1.0)
numexpr (2.6.1)				      |	numexpr (2.5)
numpy (1.13.0.dev0+95e935f)		      |	numpy (1.11.3)
numpydoc (0.6.0)			      |	pandas (0.17.1)
pandas (0.19.2)				      |	passlib (1.6.5)
					      >	PasteDeploy (1.5.2)
path.py (8.1.2)					path.py (8.1.2)
pathlib2 (2.2.0)			      <
patsy (0.4.1)					patsy (0.4.1)
pbalign (0.3.0)					pbalign (0.3.0)
pbcommand (0.4.14)			      |	pbcommand (0.4.13)
pbcore (1.2.12)				      |	pbcore (1.2.11)
pbr (1.10.0)					pbr (1.10.0)
pbtestdata (0.2.1)			      |	Pillow (3.1.1)
pexpect (4.2.1)				      <
phonopy (1.11.2.80)			      <
phylo-utils (0.0.5)			      <
pickleshare (0.7.4)			      <
Pillow (3.2.0)				      <
pip (9.0.1)					pip (9.0.1)
progressbar-latest (2.4)		      |	protobuf (3.1.0.post1)
prompt-toolkit (1.0.9)			      |	Pygments (2.1.1)
protobuf (3.0.0b2)			      |	pygpu (0.2.1)
ptyprocess (0.5.1)			      <
pyBigWig (0.2.8)			      <
pyfasta (0.5.2)				      <
Pygments (2.1.3)			      <
pyinotify (0.9.6)				pyinotify (0.9.6)
pylab (0.1.3)				      |	pyparsing (2.1.0)
pylibcurl (0.7.12)			      <
pyparsing (2.1.10)			      <
pypng (0.0.18)					pypng (0.0.18)
pysam (0.9.0)				      |	pyramid (1.7.3)
					      >	pyramid-services (0.4)
					      >	pysam (0.9.1.4)
pysqlite (2.8.1)				pysqlite (2.8.1)
python-dateutil (2.6.0)			      |	python-dateutil (2.4.2)
python-qt (0.50)			      <
pythonbrew (1.3.6)				pythonbrew (1.3.6)
pytz (2016.10)				      |	pytz (2016.3)
PyYAML (3.12)				      |	PyYAML (3.11)
pyzmq (15.2.0)				      |	repoze.lru (0.6)
requests (2.12.4)			      |	requests (2.11.1)
scandir (1.4)				      |	scikit-learn (0.17.1)
scikit-bio (0.4.2)			      |	scipy (0.15.1)
scikit-image (0.12.3)			      <
scikit-learn (0.18.1)			      <
scipy (0.18.1)				      <
screed (0.9)					screed (0.9)
seaborn (0.7.0)				      <
setuptools (32.3.1)				setuptools (32.3.1)
simplegeneric (0.8.1)				simplegeneric (0.8.1)
singledispatch (3.4.0.3)		      <
six (1.10.0)					six (1.10.0)
sklearn (0.0)					sklearn (0.0)
snap (1.2-dev-centos6.6-x64-py2.7)		snap (1.2-dev-centos6.6-x64-py2.7)
snowballstemmer (1.2.1)				snowballstemmer (1.2.1)
Sphinx (1.3.5)					Sphinx (1.3.5)
sphinx-rtd-theme (0.1.9)			sphinx-rtd-theme (0.1.9)
SQLAlchemy (1.0.11)				SQLAlchemy (1.0.11)
statsmodels (0.8.0)				statsmodels (0.8.0)
sympy (1.0)				      <
tables (3.2.0)					tables (3.2.0)
tensorflow (0.10.0rc0)			      |	tensorflow (0.12.1)
					      >	tet (0.3)
TEToolkit (1.5.0)				TEToolkit (1.5.0)
toolz (0.7.4)				      |	Theano (0.8.2)
tornado (4.3)				      |	translationstring (1.3)
traitlets (4.3.1)			      |	typing (3.5.2.2)
tree-distance (1.0.6)			      <
treeCl (0.1.29)				      <
urllib3 (1.14)					urllib3 (1.14)
wcwidth (0.1.7)				      |	venusian (1.0)
virtualenv (15.1.0)			        virtualenv (15.0.1)
					      >	WebOb (1.6.1)
wget (3.2)					wget (3.2)
wheel (0.29.0)					wheel (0.29.0)
xmlbuilder (1.0)			      <
yolk3k (0.8.8)					yolk3k (0.8.8)
					      |	zope.deprecation (4.1.2)
					      |	zope.interface (4.3.2)
razor python/2.7.5

(optional) module load gcc/4.6.3 mkl/16.0.1 impi/4.0.3

(optional for khmer) gcc/5.2.1 mkl/16.0.1 impi/4.0.3

(note) MKL version changed to 16.0.1 to avoid segfaults in numpy

module load python/2.7.5

trestles python/2.7.5

is the rpm python27 version so has no prerequisites

module load python/2.7.5

pip installed modules (2.7.5)
trestles				razor
--------				-----
Babel (0.9.6)			      |	agalma (0.4.0)
bx-python (0.7.3)		      |	alabaster (0.7.7)
				      >	anvio (1.2.2)
				      >	Babel (2.2.0)
				      >	backports.ssl-match-hostname (3.4.0.2)
				      >	bcbio-gff (0.6.2)
				      >	biolite (0.4.0)
				      >	biom-format (2.1.5)
				      >	biopython (1.64)
				      >	bottle (0.12.9)
				      >	burrito (0.9.1)
				      >	burrito-fillings (0.1.1)
				      >	bx-python (0.7.1)
				      >	bz2file (0.98)
				      >	certifi (2016.2.28)
				      >	cffi (1.5.2)
				      >	click (6.3)
				      >	cloudpickle (0.2.1)
				      >	cogent (1.5.3)
				      >	configparser (3.3.0.post2)
				      >	coverage (3.6)
				      >	cryptography (1.3.1)
cycler (0.10.0)				cycler (0.10.0)
Cython (0.24)			      |	Cython (0.21.1)
				      >	dammit (0.2.5)
				      >	decorator (4.0.9)
deepTools (2.2.4)			deepTools (2.2.4)
docutils (0.11)			      |	DendroPy (3.12.0)
HTSeq (0.6.1p1)			      |	Django (1.8.3)
Jinja2 (2.6)			      |	docutils (0.12)
MarkupSafe (0.11)		      |	doit (0.29.0)
				      >	emperor (0.9.51)
				      >	enum34 (1.1.3)
				      >	ete2 (2.3.10)
				      >	ficus (0.1)
				      >	fisher (0.1.4)
				      >	future (0.15.2)
				      >	gdata (2.0.18)
				      >	gurobipy (4.0.0)
				      >	h5py (2.6.0)
				      >	hcluster (0.2.0)
				      >	HTSeq (0.6.1)
				      >	idna (2.1)
				      >	image (1.4.0)
				      >	ipaddress (1.0.16)
				      >	ipython (4.1.1)
				      >	ipython-genutils (0.1.0)
				      >	Jinja2 (2.8)
				      >	khmer (2.0)
				      >	lxml (3.6.0)
				      >	MarkupSafe (0.23)
matplotlib (1.5.1)			matplotlib (1.5.1)
nose (1.3.0)			      |	mock (1.0.1)
				      >	mpi4py (1.3.1)
				      >	natsort (3.5.6)
				      >	ndg-httpsclient (0.4.0)
				      >	networkx (1.9)
				      >	nose (1.3.4)
				      >	nose-capturestderr (1.0)
				      >	Numeric (24.2)
				      >	numexpr (2.5)
numpy (1.11.0)				numpy (1.11.0)
numpydoc (0.6.0)		      |	pandas (0.17.1)
pip (8.1.2)			      |	path.py (8.1.2)
protobuf (3.0.0b2)		      |	peppercorn (0.5)
				      >	pexpect (4.0.1)
				      >	pickleshare (0.6)
				      >	Pillow (2.8.1)
				      >	pip (8.1.1)
				      >	Pmw (1.3)
				      >	ptyprocess (0.5.1)
				      >	PuLP (1.5.4)
				      >	pyasn1 (0.1.9)
pyBigWig (0.2.8)			pyBigWig (0.2.8)
Pygments (1.5)			      |	pycparser (2.14)
pyparsing (2.1.4)		      |	pyfasta (0.5.2)
				      >	PyFoam (0.6.2)
				      >	Pygments (2.1.1)
				      >	pyinotify (0.9.6)
				      >	pynast (1.2.2)
				      >	pyOpenSSL (16.0.0)
				      >	pyparsing (1.5.7)
				      >	pyqi (0.3.2)
pysam (0.9.0)				pysam (0.9.0)
python-dateutil (2.5.3)		      |	pysqlite (2.6.3)
pytz (2016.4)			      |	python-dateutil (2.4.2)
				      >	pytz (2016.3)
				      >	PyYAML (3.11)
				      >	qcli (0.1.1)
				      >	qiime (1.9.1)
				      >	qiime-default-reference (0.1.3)
				      >	scikit-bio (0.2.3)
				      >	scikit-learn (0.17.1)
				      >	scikits.statsmodels (0.3.1)
scipy (0.17.1)				scipy (0.17.1)
setuptools (0.9.8)		      |	screed (0.9)
simplejson (3.2.0)		      |	setuptools (20.7.0)
				      >	simplegeneric (0.8.1)
six (1.10.0)				six (1.10.0)
Sphinx (1.1.3)			      |	snap (1.1-dev-redenterprise-x64-py2.7)
SQLAlchemy (0.7.9)		      |	snowballstemmer (1.2.1)
tensorflow (0.9.0rc0)		      |	spanki (0.5.0, /share/apps/Python/2.7.5/lib/...
virtualenv (1.10.1)		      |	Sphinx (1.3.5)
Werkzeug (0.8.3)		      |	sphinx-rtd-theme (0.1.9)
wheel (0.29.0)			      |	tornado (4.0.2)
				      >	traitlets (4.1.0)
				      >	urllib3 (1.14)
				      >	virtualenv (15.0.1)
				      >	wget (2.2)
				      >	yolk3k (0.8.8
PYTHONUSERBASE

The Python-from-source modules set the environment variable

PYTHONUSERBASE=~/python2.7
or
PYTHONUSERBASE=~/python3.5

so you can add your own user modules. If necessary you can change the variable after loading the module. To install user modules, first create the directory

cd ~
mkdir -p python2.7

To add a module not in the pip list,

pip install peppercorn --user

To add a module with a different version than the pip list:

pip install lxml --user --ignore-installed

To show your local modules, which will be used preferentially to the system modules:

$ pip list --user
lxml (3.6.0)
peppercorn (0.5)
virtualenv

virtualenv is useful for a custom environment or where there is a module incompatibility with the system version. Create and activate like the following, install the modules you want, and activate in the batch job before using. This example uses the system installation as a base.

tres-l1:rfeynman:$ module load gcc/4.9.1 mkl/16.0.1 mvapich2/2.2 python/2.7.11
tres-l1:rfeynman:$ which python
/share/apps/python/2.7.11/bin/python
tres-l1:rfeynman:$ virtualenv -p /share/apps/python/2.7.11 /storage/rfeynman/virtualenv
Running virtualenv with interpreter /share/apps/python/2.7.11/bin/python
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in /storage/rfeynman/virtualenv/bin/python
Installing setuptools, pip, wheel...done.
tres-l1:rfeynman:$ export VIRTUALENVWRAPPER_PYTHON=/share/apps/python/2.7.11
tres-l1:rfeynman:$ source /storage/rfeynman/virtualenv/bin/activate
(virtualenv) tres-l1:rfeynman:$ which python
/storage/rfeynman/virtualenv/bin/python
(virtualenv) tres-l1:rfeynman:$ pip list
appdirs (1.4.3)
packaging (16.8)
pip (9.0.1)
pyparsing (2.2.0)
setuptools (34.3.2)
six (1.10.0)
wheel (0.29.0)
Anaconda Python

Anaconda is a prepackaged distribution of Python. Private installations can be made through conda environments similar to virtualenv environments. Anaconda and Intel python can also use pip –user, but all python2.7 versions will install to the same user directory /home/$USER/python2.7 .

see https://conda.io/docs/using/cheatsheet.html

$ module load python/2.7.13-anaconda (or python/3.6.0-anaconda )
$ python -V
Python 2.7.13 :: Anaconda 4.3.1 (64-bit)
$ conda list
# packages in environment at /share/apps/python/Anaconda2:
#
_license                  1.1                      py27_1  
alabaster                 0.7.9                    py27_0  
anaconda                  4.3.1               np111py27_0  
etc.
$ conda create --prefix ~/bio-env biopython
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /home/rfeynman/bio-env:

The following NEW packages will be INSTALLED:

    biopython:  1.68-np112py27_0
    mkl:        2017.0.1-0      
    numpy:      1.12.1-py27_0   
    openssl:    1.0.2k-1        
    pip:        9.0.1-py27_1    
    python:     2.7.13-0        
    readline:   6.2-2           
    setuptools: 27.2.0-py27_0   
    sqlite:     3.13.0-0        
    tk:         8.5.18-0        
    wheel:      0.29.0-py27_0   
    zlib:       1.2.8-3         

Proceed ([y]/n)? y
$ 
...etc...
$ source activate /home/rfeynman/bio-env
Intel Python

Intel supplies another version of Anaconda python, supposedly optimized for runtime speed on Intel processors. It has about 130 conda modules installed.

$ module load python/2.7.13-intel
$ source activate root
(root) $ conda list
# packages in environment at /share/apps/intelpython2:
#
backports                 1.0                py27_intel_6  [intel]  intel
backports_abc             0.5                py27_intel_0  [intel]  intel
bleach                    1.5.0              py27_intel_0  [intel]  intel
boost                     1.61.0             py27_intel_3  [intel]  intel
bzip2                     1.0.6                  intel_10  [intel]  intel
caffe                     1.0.0rc3           py27_intel_4  [intel]  intel
etc.
2.7.11 source build, python and snap.py

An example for modules that need configure customization, or won't install from pip.

module load gcc/4.9.1
module load mkl/16.0.1
module load mvapich2/2.2
export LD_RUN_PATH=$MKL_ROOT/lib/intel64:$LD_LIBRARY_PATH
cd install_dir
tar -Jxf Python-2.7.11.tar.zx
cd Python-2.7.11
./configure --prefix=/share/apps/python/2.7.11 --with-ensurepip --with-threads --disable-ipv6
make && make install
cd ..
wget https://snap.stanford.edu/snappy/release/snap-1.2-2.4-centos6.5-x64-py2.6.tar.gz
tar zxf snap*gz
cd snap*6
python setup.py install
custom numpy build with MKL/FFTW3

These libraries greatly improve numpy/scipy performance.

$ cd $PYTHONPATH/lib/python2.7/site-packages
$ tar zxf numpy-1.11.0.tar.gz
$ cd numpy-1.11.0
$ tail site.cfg
[fftw]
libraries = fftw3
include_dirs = /share/apps/fftw/fftw-3.3.3/gcc-4.6.3/include
library_dirs = /share/apps/fftw/fftw-3.3.3/gcc-4.6.3/lib
[mkl]
include_dirs = /share/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/include
library_dirs = /share/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64:\
/share/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64:/usr/lib64
lapack_libs = mkl_lapack95_ilp64
mkl_libs = mkl_gf_ilp64, mkl_blas95_ilp64, mkl_def, mkl_gnu_thread, mkl_core, iomp5, pthread
$ export LD_RUN_PATH=/share/apps/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64:\
/share/apps/fftw/fftw-3.3.3/gcc-4.6.3/lib:\
/share/apps/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64
$ python setup.py develop build config build_ext -j 4 
$ pip install -e .
Multithreaded MKL with python example

This is a matrix multiplication of two square matrices O(4096) through (1) default python 2.6 with numpy and multithreaded Atlas blas from RPM (number of threads set at compile time and not controllable) (2) to (4) numpy and MKL with unset,1, and 16 MKL threads respectively. Outputs size, time, GFlopRate. You can see MKL_NUM_THREADS unset acts as if it is set to the number of cores in the computer, and that MKL is around 6 times faster than Atlas on this problem with maximum threading.

$ module purge
$ python test.numpy.py
4096,  6.697931,  20.51711
$ module load gcc/4.9.1 mkl/16.0.1 mvapich2/2.2 python/2.7.11
$ python test.numpy.py
4096,  1.135037,  121.0729
$ export MKL_NUM_THREADS=1
$ python test.numpy.py
4096,  16.43035,  8.363921
$ export MKL_NUM_THREADS=16
$ python test.numpy.py
4096,  1.130565,  121.5518
python.txt · Last modified: 2017/06/07 21:40 by root