(2022-09-23) Lab Notebook: Singularity --- Using an alternate OS

Warning

This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.

Lab Notebook --- Using an alternate OS via Singularity (2022-09-23)

Occasionally, I run into a peice of software which needs a newer version of GLIBC (> 2.17) to be installed. Rather than try to upgrade GLIBC (which seems complicated and can go very poorly, see this thread for a general idea), its alot easier to run a different OS image in a container and run the software there. For this, we will use Singularity, which functions similarly to Docker, but is available on HPCC because unlike Docker it doesn't give you root privledges on the host system. Futhremore, its alot easier to work with than a virtual box (used them in the past, they can be fun but a lot of overhead + issues sharing files). Finally, you can run environment managers like Conda with Singularity to layer package management on top of the alternate OS, or do things like work with a downgraded python, all without affecting any system level install.

Below, I will go through obtaining a CentOS8 image which has an updated GLIBC, building an overlay, and installing conda in that overlay, and building a conda environment for pymesh (version 0.3) as an example of this process

Note

This is very brief and direct explanation of installing something in a Singularity image. A more general explanation of Singularity images, overlay, etc will be added in the future (probably as another Notebook)

Getting a CentOS8 image

You can pull a Docker built image directly using Singularity. The code below will grab a CentOS8 image with a new GLIBC which is needed from the pymesh 0.3 version on conda.

# This works for now but
# Probably want to update/find different OS since centos8 hit end of life at the end of 2021, probably centOS9
# As of current, most similar OS with GLIBC > 2.17
singularity pull centos8.sif docker://centos:8.3.2011

I keep my images in a folder called "singularity_pull_images" so, below replace "../singularity_pull_images" with whever your images are kept

Pymesh 0.3 in CentOS8 Container

Now what we have an image, we can make an 'overlay' which is basically a 'filesystem inside a singe file'. This both cuts down on the number of files used on your home directory AND help silo off environments/installs/etc that are associated with a particular Singularity image (although this last step isn't really necessary, but it helps). Also, you generally have more acess to the filesystem in the overlay, meaning you can install things outside of your home directory in the overlay.

For now, we will make an overlay, install conda in that overlay and then use the overlay to contain all the files for pymesh and its dependencies

Setup Singularity Container

# The following commands are in powertools, which should be loaded by default; otherwise you will need to run 'module load powertools'
overlay_build 5 overlay.img
overlay_install_conda overlay.img https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh ../singularity_pull_images/centos8.sif 

# Now we access our overlay within our CentOS8 image
overlay_start overlay.img ../singularity_pull_images/centos8.sif 

# After this, your prompt should be 'Singularity>'

Install Pymesh within the container

Now that we are inside of our container, we can install pymesh. Using the CentOS8 image, we shouldn't get a GLIBC error, but we still need an older version of python (CentOS8 uses 3.9, pymesh needs 3.6). So, we will make conda enviroment with an older python (which is alot easier that trying to downgrade the system python of the Singularity image)

# Need an older version of python for pymesh 0.3
conda create -n pymesh2 python=3.6 pymesh2=0.3    

# Now, activate our new environmet
conda init
source ~/.bashrc                                  # Need to manually update shell post conda init
conda activate pymesh2

# After this your prompt should be '(pymesh2) Singularity>'

Check install using nose

Now, within our conda environment, within our Singualrity container, lets install the nose package (unit testing in python) and run the test from the pymesh github (https://github.com/PyMesh/PyMesh)

# From within the pymesh2 environment
conda install nose
python -c "import pymesh; pymesh.test()"

# You should get a couple warning and a couple S (skipped) tests, pretty sure this is fine

Endnote

This solution might seem (overly) complicated, because at the end we are running a conda environment inside of Singularity container. However, this allows us to use an older python version (via conda) and an alternative OS/GLIBC (via Singularity), either of which is an issue which is difficult to solve (downgrade python/upgrade GLIBC) and which it is not difficult to really screw things up by attempting