Migrating Conda Environments

As part of ICER's OS upgrade, all software has changed. While Conda does a good job at isolating software in environments and installing most things from scratch, it is possible that you may need to reinstall your environments on the new system.

Optional

In limited testing, we have seen that Conda environments created on the current system appear to work on the new system without intervention. However, if you notice issues or your environments become nonfunctional, please follow the steps below.

Save your current environments

First, start a backwards compatibility container and get a list of your environments.

ssh user@hpcc.msu.edu
ssh dev-amd20
old_os
module load Conda/3
conda env list

Select the environments you would like to transfer over. In this guide, we will use an example named project. Use conda env export to get the list of packages in the environment.

conda env export -n project > project_environment.yml

Reinstall Conda on the new operating system

Optional

In limited testing, ICER has not seen a need to reinstall Conda on the new operating system. However, if you are having issues, this would be a good troubleshooting step to take.

Log into a development node running the new operating system, and follow our instructions for Installing Conda.

In the step where you are asked to install Miniforge3 in your home directory, make sure to choose a different location if you'd like to maintain backwards compatibility, e.g., /mnt/home/<user>/miniforge3_ubuntu. You can add the following to the .bashrc file in your home directory to make sure that the new version of Conda is only used on Ubuntu nodes.

if [[ $(source /etc/os-release && echo $ID) == "ubuntu" ]]; then
    export CONDA3PATH=/mnt/home/<user>/miniforge3_ubuntu
fi

Create a new virtual environment

Exit the container back into a development node running the new operating system, and reinstall your environments. To avoid confusion with the old environment, you can use a new name.

# Exit backwards compatibility container
exit 

# Load Conda module
module purge
module load Conda/3

# Reinstall environment (may take a while)
conda env create -f project_environment.yml -n project_ubuntu

The new environment is ready to use.

Update old batch scripts

Assuming this Conda environment was used in a batch script, you now need to update it for the new system. Suppose we have the following script, my_script.sb.

#/bin/bash --login

#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=500MB

module purge
module load Conda/3

conda activate project
python do_some_work.py

Copy the script for the new system.

cp my_script.sb my_script_ubuntu.sb

Then, change the environment name to match the new Conda environment.

#/bin/bash --login

#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=2
#SBATCH --mem-per-cpu=500MB

module purge
module load Conda/3

conda activate project_ubuntu
python do_some_work.py

To test, make sure you log into an Ubuntu development node, and submit the job using

sbatch my_script_ubuntu.sb