Using Python in HPCC with virtualenv

Python applications usually use packages and modules that require specific versions of libraries. This means one installed application may conflict with another application due to using the same library but with different versions. It is difficult to meet the requirements of every application by one global Python installation. To resolve this issue, users can create an isolated virtual environment with a particular version of Python in a self-contained directory of their home or research space. Any package and the dependent libraries installed inside the directory can be available only through the virtual environment. Different applications can then use different virtual environments to avoid any conflict.

Create and use virtual environments

To create python virtual environments, please make sure your preferred version of Python is loaded. It is also a good idea to create a directory of the python version to store different environments and their applications:

[UserName@dev-intel18 ~]$ module list Python

Currently Loaded Modules Matching: Python
  1) Python/3.6.4

[UserName@dev-intel18 ~]$ which python
/opt/software/Python/3.6.4-foss-2018a/bin/python

[UserName@dev-intel18 ~]$ mkdir Python3.6.4
[UserName@dev-intel18 ~]$ cd Python3.6.4
[UserName@dev-intel18 Python3.6.4]$

Currently, two common tools can be used to create Python virtual environments. Please use only one of them:

venv is available for Python 3.3 and later by default. The application pip and setuptools should be ready to use in HPCC system. A virtual environment can be created by running "python3 -m venv <DIR>", where <DIR> is the directory of the created environment. Following is an example of the command, and the directory tutorial is created for the virtual environment.
```
[UserName@dev-intel18 Python3.6.4]$ python3 -m venv tutorial  
[UserName@dev-intel18 Python3.6.4]$ ls tutorial  
bin  include  lib  lib64  pyvenv.cfg
```

virtualenv supports all Python versions. By default, HPCC system has pip, setuptools and wheel installed and available. Similarly to venv, a virtual environment can be created by executing virtualenv <DIR>, where applications of the virtual environment are installed in tutorial directory.

    [UserName@dev-intel18 Python3.6.4]$ virtualenv tutorial
    Using base prefix '/opt/software/Python/3.6.4-foss-2018a'
    New python executable in /mnt/home/UserName/Python3.6.4/tutorial/bin/python
    Installing setuptools, pip, wheel...done.
    [UserName@dev-intel18 Python3.6.4]$ ls tutorial
    bin  include  lib  lib64  pip-selfcheck.json

Please use one of them to create the virtual environment.

The created tutorial environment can now be used by sourcing the script file activate under the bin directory:

[UserName@dev-intel18 Python3.6.4]$ source tutorial/bin/activate
(tutorial) [UserName@dev-intel18 Python3.6.4]$

where the name inside the parentheses (tutorial) in front of the prompt line shows the current Python environment. To leave the environment, just run "deactivate":

(tutorial) [UserName@dev-intel18 Python3.6.4]$ deactivate
[UserName@dev-intel18 Python3.6.4]$

and the parentheses disappear. Any time you want to use the tutorial environment. Simply source the file again: source ~/Python3.6.4/tutorial/bin/activate and the environment is back. More information can be found about venv or virtualenv.

Install packages from PyPI using pip

The most common usage of pip is to install python packages from the Python Package Index with a requirement specifier. Users can also check other usages with pip. Below, some of the common usage scenarios are introduced.

To install the latest version of a python package, users can run pip install <Package Name>, for example, using sympy as <Package Name>:

(tutorial) [UserName@dev-intel18 Python3.6.4]$ pip install "sympy"
Collecting sympy
Using cached https://files.pythonhosted.org/packages/21/21/f4105795ca7f35c541d82c5b06be684dd2f5cb4f508fb487cd7aea4de776/sympy-1.4-py2.py3-none-any.whl
Collecting mpmath>=0.19 (from sympy)
Installing collected packages: mpmath, sympy
Successfully installed mpmath-1.1.0 sympy-1.4

To install a specific version of a python package, please run pip install <Package Name>==<Version Number>. For example, install numpy with <version number> as 1.16.2:

(tutorial) [UserName@dev-intel18 Python3.6.4]$ pip install "numpy==1.16.2"
Collecting numpy==1.16.2
Downloading https://files.pythonhosted.org/packages/35/d5/4f8410ac303e690144f0a0603c4b8fd3b986feb2749c435f7cdbb288f17e/numpy-1.16.2-cp36-cp36m-manylinux1_x86_64.whl (17.3MB)
    |████████████████████████████████| 17.3MB 10.5MB/s
Installing collected packages: numpy
Successfully installed numpy-1.16.2

To upgrade an already installed package to the latest from PyPI, users can run pip install --upgrade. For example, upgrade the installed package numpy:

(tutorial) [UserName@dev-intel18 Python3.6.4]$ pip install --upgrade numpy
Collecting numpy
Using cached https://files.pythonhosted.org/packages/e5/e6/c3fdc53aed9fa19d6ff3abf97dfad768ae3afce1b7431f7500000816bda5/numpy-1.17.2-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: numpy
Found existing installation: numpy 1.16.2
    Uninstalling numpy-1.16.2:
    Successfully uninstalled numpy-1.16.2
Successfully installed numpy-1.17.2

With pip, you can also list all installed packages and their versions with the command pip freeze:

(tutorial) [UserName@dev-intel18 Python3.6.4]$ pip freeze
mpmath==1.1.0
numpy==1.17.2
sympy==1.4

For more detail, see the pip docs, which includes a complete Reference Guide.

PYTHONPATH environment variable

You can use the environment variable PYTHONPATH to include the packages already installed by the same python version in other directories. By adding the site-packages paths to PYTHONPATH environment variable and separating them with the ":" symbol:

(tutorial) [UserName@dev-intel18 Python3.6.4]$ export PYTHONPATH=~/Python3.6.4/tutorial/lib/python3.6/site-packages:/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages
(tutorial) [UserName@dev-intel18 Python3.6.4]$ pip freeze
absl-py==0.5.0
alabaster==0.7.12
appdirs==1.4.3
artemis==0.1.4
...
nose==1.3.7
numpy==1.17.2
numpydoc==0.8.0
...
suspenders==0.2.6
sympy==1.4
tensorboard==1.10.0
tensorflow==1.10.1
...
virtualenv==15.1.0
wcwidth==0.1.7
Werkzeug==0.14.1
xopen==0.3.5

all packages inside the paths are now ready to use. Please make sure the site-packages path of the current environment is set the first in PYTHONPATH variable. If a package is installed in more than one path (possibly with different versions), the package of the first path showing in the variable (PYTHONPATH) will be used.