Skip to content

Singularity: I. Introduction

Singularity is installed on our HPCC. However, you may want to develop your own containers first on a local machine.

Many HPC centers including MSU HPCC do not allow Docker containers through Docker. However, Singularity is compatible with Docker, and you can use Docker containers through Singularity. There are a few distinct difference between Docker and Singularity.

Docker:

  • Inside a Docker image, the user's privilege is escalated to root on the host system. This privilege is not supported by most HPCC including MSU HPCC. It means that Docker will not be installed on our system.

Singularity:

  • User has root privileges if elevated with "sudo" when a container runs.
  • Can run and modify Docker images and containers These key difference make Singularity be installed on most HPCC. In addition, virtually all Docker containers can be run through Singularity, users can effectively run Docker on MSU HPCC.

### Installation Singularity exits as two major version, 2 and 3. Current version on MSU HPCC is 3.5.3. Therefore, in this tutorial, I will use version 3.

To Install Singularity on your local machine, click here: https://www.sylabs.io/guides/3.0/user-guideinstallation.html#installation

  • The version of singularity on MSU HPCC is currently 3.4.1. The official documentation for this version is at https://www.sylabs.io/guides/3.4/user-guide/index.html. All singularity commands are built into the system such as 'singularity shell' and 'singularity exec', which means you can invoke these commands directly from the command line.

Check installation

When you install Singularity on your local machine, then you can check the installation with

1
2
3
$ singularity pull shub://vsoch/hello-world
INFO:    Downloading shub image
59.75 MiB / 59.75 MiB      [========================================================================================] 100.00% 10.46 MiB/s 5s

In the above example, I used the Singularity Hub “unique resource identifier,” or uri, "shub://" which tells the software to run an image from Singularity Hub. To get help, you can use the help command which gives a general overview of Singularity options and subcommands as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ singularity --help

Linux container platform optimized for High Performance Computing (HPC) and
Enterprise Performance Computing (EPC)

Usage:
  singularity [global options...]

Description:
  Singularity containers provide an application virtualization layer enabling
  mobility of compute via both application and environment portability. With
  Singularity one is capable of building a root file system that runs on any
  other Linux system where Singularity is installed.

Options:
  -d, --debug     print debugging information (highest verbosity)
  -h, --help      help for singularity
  --nocolor   print without color output (default False)
  -q, --quiet     suppress normal output
  -s, --silent    only print errors
  -v, --verbose   print additional information
      --version   version for singularity
 ...

You can use the help command if you want to see the information about subcommands. For example, to see the pull command help,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ singularity help pullPull an image from a URI

Usage:
singularity pull [pull options...] [output file] <URI>

Description:
The 'pull' command allows you to download or build a container from a given
URI. Supported URIs include:

library: Pull an image from the currently configured library
library://user/collection/container[:tag]

docker: Pull an image from Docker Hub
docker://user/image:tag

shub: Pull an image from Singularity Hub
shub://user/image:tag

oras: Pull a SIF image from a supporting OCI registry
oras://registry/namespace/image:tag

http, https: Pull an image using the http(s?) protocol
https://library.sylabs.io/v1/imagefile/library/default/alpine:latest
...

Downloading pre-built images

I already downloaded a pre-built image "hello-world" from shub, one of the registries, using pull command. This is the easiest way to use Singularity.

You can use the pull command to download pre-built images from a number of Container Registries, here we’ll be focusing on the Singularity-Hub or DockerHub. The following are some of container registries.

  • library - images hosted on Sylabs Cloud
  • shub - images hosted on Singularity Hub
  • docker - images hosted on Docker Hub
  • localimage - images saved on your machine
  • yum - yum based systems such as CentOS and Scientific Linux
  • debootstrap - apt based systems such as Debian and Ubuntu
  • arch - Arch Linux
  • busybox - BusyBox
  • zypper - zypper based systems such as Suse and OpenSuse

Pulling an images from Sylabs cloud library

In this example, I will pull a base Alpine container from Sylabs cloud:

1
2
3
$ singularity pull library://sylabsed/linux/alpine
INFO:    Downloading library image
 2.08 MiB / 2.08 MiB [===========================================================================================] 100.00% 4.74 MiB/s 0s

You can rename the container using the –name flag:

1
2
3
$ singularity pull --name my_alpine.sif library://sylabsed/linux/alpine
INFO:    Downloading library image
 2.08 MiB / 2.08 MiB [===========================================================================================] 100.00% 9.65 MiB/s 0s

The above example will save the image as "my_alpine.sif"

Pulling an images from Docker hub

This example pulls an Alpine image from Docker hub

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$ singularity pull docker://alpine
INFO:    Converting OCI blobs to SIF format
 INFO:    Starting build...
Getting image source signatures
Copying blob df20fa9351a1 done
Copying config 0f5f445df8 done
Writing manifest to image destination
Storing signatures
2020/08/20 15:53:52  info unpack layer: sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c
INFO:    Creating SIF file...
INFO:    Build complete: alpine_latest.sif

interact with images

You can interact with images with shell, exec, and run commands. To learn how to interact with images, let's first pull an image "lolcow_latest.sif" from the libray.

1
$ singularity pull library://sylabsed/examples/lolcow

shell

The shell command allows you to spawn a new shell within your container and interact with it as if it is a virtual machine.

1
2
$singularity shell lolcow_latest.sif
  Singularity>

The change in prompt indicates that you have entered the container. Once inside of a container, you are the same user as you are on the host system.

1
2
Singularity>whoami
choiyj

To exit from a container, type exit.

1
2
Singularity>exit
$

exec The exec command allows you to execute a custom command within a container by specifying the image file. For instance, to execute the cowsay program within the lolcow_latest.sif container:

1
2
3
4
5
6
7
8
9
$ singularity exec lolcow_latest.sif cowsay container camp rocks
______________________
< container camp rocks >
 ----------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

You can also use shell command to run the program in the container.

1
2
3
4
5
6
7
8
9
Singularity>  cowsay container camp rocks
 ______________________
< container camp rocks >
 ----------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

run Singularity containers contain runscripts. These are user defined scripts which define the actions of a container when user runs it. The runscript can be performed with the run command, or simply by calling the container as though it were an executable.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
$ singularity run lolcow_latest.sif
 _________________________________________
/ You're ugly and your mother dresses you \
\ funny.                                  /
 -----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Cache setting

By default, Singularity uses a temporary directory to save Docker files as tarballs:

1
2
3
4
5
6
7
8
$ ls ~/.singularity
cache/  docker/  metadata/
$ ls .singularity/docker/
sha256:0efe07335a049e6afcd757db2d17ba37a12b717eb807acb03ddf3cd756b9fc2a.tar.gz
sha256:43a98c1873995475a895f3d79f405232ef5230076b3f610c949c2e8341743af7.tar.gz
sha256:6b1bb01b3a3b72463ae8ac5666d57b28f1a21d5256271910ac8df841aa04ecd1.tar.gz
sha256:c2ca09a1934b951505ecc4d6b2e4ab7f9bf27bcdfb8999d0181deca74daf7683.tar.gz
sha256:d6c3619d2153ffdefa4a9c19f15c5d566ce271b397a84537baa9ee45b24178f2.tar.gz

You can change these by theses cache directories by specifying the location on your localhost as following:

1
2
3
$ mkdir -p $SCRATCH/singularity_tmp
$ mkdir -p $SCRATCH/singularity_scratch
$ SINGULARITY_TMPDIR=$SCRATCH/singularity_scratch SINGULARITY_CACHEDIR=$SCRATCH/singularity_tmp singularity --debug pull --name ubuntu-tmpdir.sif docker://ubuntu

Creating writable containers with --sandbox options

If you want to build a container within a writable directory (called a sandbox), you can do that with --sandbox option. We’ve already covered how you can pull an existing container from Docker Hub, but we can also build a Singularity container from docker using the build command:

1
$ singularity build --sandbox ubuntu-latest/  docker://ubuntu

With --sandbox option, you can changes and write files within the directory, but those changes will not remain when you finished using the container. To make those changes persistent, use --writable option when you start a container.

1
$ singularity shell --writable ubuntu-latest/