Migrating Python Virtual Environments

As part of ICER's OS upgrade, the available versions and locations of Python on the system have changed. Since Python virtual environments link to Python on the system, virtual environments will need to be reinstallled to work properly on the new system.

To migrate virtual environments, please follow the steps below.

Save your current virtual environment

First, activate your virtual environment in a backwards compatibility container. In this guide, we will use an example named project:

ssh user@hpcc.msu.edu
ssh dev-amd20
old_os
# In directory where "project" virtual env exists
source project/bin/activate

Then, save the packages installed via pip to a requirements.txt file.

pip freeze > requirements.txt

Create a new virtual environment

Now, exit the container back into a development node running the new operating system, and, if you have specific version requirements, select the version of Python you'd like to use from the module system. If you have the default modules loaded, this will be Python/3.11.3.

# Exit backwards compatibility container
exit
# Load newer version of Python than default (optional)
module purge
module load Python/3.11.5-GCCcore-13.2.0

Then use this python to create a new virtual environment. To avoid confusion with the old environment and to maintain backwards compatibility if necessary, use a new name.

python -m venv project_ubuntu

Install the packages in your old virtual environment

Activate the new virtual environment, and use pip to reinstall the packages from the old environment.

source project_ubuntu/bin/activate
# The next step may take a while
pip install -r requirements.txt

The new environment is ready to use. If the reinstallation fails, see the Troubleshooting section below.

Update old batch scripts

Assuming this virtual environment was used in a batch script, you now need to update it for the new system. Suppose we have the following script, my_script.sb.

#/bin/bash --login

#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=500MB

source project/bin/activate
python do_some_work.py

Copy the script for the new system.

cp my_script.sb my_script_ubuntu.sb

Then, change the way the virtual environment is activated to match our new virtual environment.

#/bin/bash --login

#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=500MB

# Make sure we have the right version of Python loaded
module purge
module load Python/3.11.5-GCCcore-13.2.0

# Use the new virtual environment
source project_ubuntu/bin/activate
python do_some_work.py

To test, make sure you log into an Ubuntu development node, and submit the job using

sbatch my_script_ubuntu.sb

Troubleshooting

Using an older version

Moving from older versions of Python to newer versions can cause problems when reinstalling packages. If you run into issues, try an older version of Python on the new operating system, e.g., when creating the new virtual environment, use

module purge
module load Python/3.10.4-GCCcore-11.3.0
python -m venv project_ubuntu
source project_ubuntu/bin/activate
pip install -r requirements.txt

However, even the older versions on the new operating system are still relatively new (we don't recommend using anything older than Python/3.10.4, as the older versions are deprecated). If you need an even older version, we recommend you create a Conda environment. See our instructions for Installing Conda.

Using Conda

Not for reinstalling Conda environments

This section explains how to reinstall a Python virtual environment into a Conda environment. For instructions to migrate Conda environments, please see our Conda specific documentation.

After installing Conda, you can create an environment with the same version of Python from your virtual environment, and install the packages via pip.

module purge
module load Conda/3
# Use version of Python from old venv, can get using "python --version"
conda create -n project_ubuntu python=3.6.3 pip
conda activate project_ubuntu
pip install -r requirements.txt

To use this Conda environment in the future, use

module purge
module load Conda/3
conda activate project_ubuntu

Note that when using Conda, we recommend using the Conda package manager to install packages via conda install instead of using pip. However, for migrating virtual environments like this, using pip is fine, so long as you do not use conda install to install more packages into the same environment afterward.

Using pre-installed packages

Also note that the module system already includes a few Python "bundles" that have popular packages pre-installed. These include Python-bundle-PyPI and Scipy-bundle. Use the module spider command to see if these can meet your needs.

Use backwards compatibility containers

If all else fails you can use our backwards compatibility scripts (which have limitations). For more information, please see our backwards compatibility documentation.