(2022-09-23) Lab Notebook: Singularity --- Using an alternate OS
This is as a Lab Notebook which describes how to solve a specific problem at a specific time. Please keep this in mind as you read and use the content. Please pay close attention to the date, version information and other details.
Lab Notebook --- Using an alternate OS via Singularity (2022-09-23)
Occasionally, I run into a peice of software which needs a newer version of GLIBC (> 2.17) to be installed. Rather than try to upgrade GLIBC (which seems complicated and can go very poorly, see this thread for a general idea), its alot easier to run a different OS image in a container and run the software there. For this, we will use Singularity, which functions similarly to Docker, but is available on HPCC because unlike Docker it doesn't give you root privledges on the host system. Futhremore, its alot easier to work with than a virtual box (used them in the past, they can be fun but a lot of overhead + issues sharing files). Finally, you can run environment managers like Conda with Singularity to layer package management on top of the alternate OS, or do things like work with a downgraded python, all without affecting any system level install.
Below, I will go through obtaining a CentOS8 image which has an updated GLIBC, building an overlay, and installing conda in that overlay, and building a conda environment for pymesh (version 0.3) as an example of this process
This is very brief and direct explanation of installing something in a Singularity image. A more general explanation of Singularity images, overlay, etc will be added in the future (probably as another Notebook)
Getting a CentOS8 image
You can pull a Docker built image directly using Singularity. The code below will grab a CentOS8 image with a new GLIBC which is needed from the pymesh 0.3 version on conda.
1 2 3 4
I keep my images in a folder called "singularity_pull_images" so, below replace "../singularity_pull_images" with whever your images are kept
Pymesh 0.3 in CentOS8 Container
Now what we have an image, we can make an 'overlay' which is basically a 'filesystem inside a singe file'. This both cuts down on the number of files used on your home directory AND help silo off environments/installs/etc that are associated with a particular Singularity image (although this last step isn't really necessary, but it helps). Also, you generally have more acess to the filesystem in the overlay, meaning you can install things outside of your home directory in the overlay.
For now, we will make an overlay, install conda in that overlay and then use the overlay to contain all the files for pymesh and its dependencies
Setup Singularity Container
1 2 3 4 5 6 7 8
Install Pymesh within the container
Now that we are inside of our container, we can install pymesh. Using the CentOS8 image, we shouldn't get a GLIBC error, but we still need an older version of python (CentOS8 uses 3.9, pymesh needs 3.6). So, we will make conda enviroment with an older python (which is alot easier that trying to downgrade the system python of the Singularity image)
1 2 3 4 5 6 7 8 9
Check install using nose
Now, within our conda environment, within our Singualrity container, lets install the nose package (unit testing in python) and run the test from the pymesh github (https://github.com/PyMesh/PyMesh)
1 2 3 4 5
This solution might seem (overly) complicated, because at the end we are running a conda environment inside of Singularity container. However, this allows us to use an older python version (via conda) and an alternative OS/GLIBC (via Singularity), either of which is an issue which is difficult to solve (downgrade python/upgrade GLIBC) and which it is not difficult to really screw things up by attempting