Singularity Introduction
Note
This tutorial is adapted from the Container Camp Tutorial produced and copyrighted by CyVerse.
Singularity allows users to run software inside of containers. Another popular container system is Docker, which is interoperable with Singularity. Singularity provides many features that make it well suited for HPC applications, and therefore, it is the container system that is installed on our HPCC.
Installation
To install Singularity on your local machine, please see the installation instructions in the documentation. Singularity can only be run natively on Linux, but if you need to run it locally on a Windows or Mac computer, you can use a virtual machine provided by the creators of Singularity. See these alternative instructions for details.
Note
As of December 2024, the version of Singularity on the MSU HPCC is currently
4.1.2. For any questions that this tutorial or the Advanced
Topics page do not answer, please consult
the official documentation for this
version. All
Singularity commands are built into the system such as singularity shell
and singularity exec
, which means you can invoke these commands directly
from the command line.
Check installation
If you would like to check your installation of Singularity on your local machine or the installation on the HPCC, you can run
$ singularity pull shub://vsoch/hello-world
INFO: Downloading shub image
59.75 MiB / 59.75 MiB [========================================================================================] 100.00% 10.46 MiB/s 5s
$ singularity run hello-world_latest.sif
RaawwWWWWWRRRR!! Avocado!
In the above example, we used the Singularity Hub "unique resource identifier,"
or URI, "shub://" which tells the software to run an image from Singularity
Hub. To get help, you can use the --help
flag which gives a general overview
of Singularity options and subcommands as follows:
$ singularity --help
Linux container platform optimized for High Performance Computing (HPC) and
Enterprise Performance Computing (EPC)
Usage:
singularity [global options...]
Description:
Singularity containers provide an application virtualization layer enabling
mobility of compute via both application and environment portability. With
Singularity one is capable of building a root file system that runs on any
other Linux system where Singularity is installed.
Options:
-c, --config string specify a configuration file (for root or
unprivileged installation only) (default
"/etc/singularity/singularity.conf")
-d, --debug print debugging information (highest verbosity)
-h, --help help for singularity
--nocolor print without color output (default False)
-q, --quiet suppress normal output
-s, --silent only print errors
-v, --verbose print additional information
--version version for singularity
Available Commands:
build Build a Singularity image
cache Manage the local cache
capability Manage Linux capabilities for users and groups
completion Generate the autocompletion script for the specified shell
config Manage various singularity configuration (root user only)
...
You can use the help
command if you want to see the information about
subcommands. For example, to see the pull
command help,
$ singularity help pull
Pull an image from a URI
Usage:
singularity pull [pull options...] [output file] <URI>
Description:
The 'pull' command allows you to download or build a container from a given
URI. Supported URIs include:
library: Pull an image from the currently configured library
library://user/collection/container[:tag]
docker: Pull a Docker/OCI image from Docker Hub, or another OCI registry.
docker://user/image:tag
shub: Pull an image from Singularity Hub
shub://user/image:tag
oras: Pull an image from an OCI registry that supports ORAS / OCI artifacts.
oras://registry/namespace/image:tag
http, https: Pull an image using the http(s?) protocol
https://example.com/containers/mycontainer.sif
By default, images from a library URI will be pulled in the same format they
were uploaded. If the --oci flag is specified then the pull is required
to result in an OCI-SIF image.
By default, images pulled from docker and other oci URIs will be converted
into a singularity native SIF image. If the --oci flag is specified then they
will be encapsulated in an OCI-SIF image.
Images pulled from a shub/oras/http/https URI are always directly downloaded,
in the same format as they were uploaded.
...
Downloading pre-built images
We already downloaded a pre-built image "hello-world" from shub, one of the registries, using pull command. This is the easiest way to use Singularity.
You can use the pull
command to download pre-built images from a number of
Container Registries. Here we’ll be focusing on the Singularity Hub or Docker
Hub. The following are some of container registries.
- library - images hosted on Sylabs Cloud
- shub - images hosted on the (archived) Singularity Hub
- docker - images hosted on Docker Hub
Pulling an images from Sylabs cloud library
In this example, I will pull a base Alpine container from Sylabs cloud:
$ singularity pull library://alpine
INFO: Downloading library image
2.08 MiB / 2.08 MiB [===========================================================================================] 100.00% 4.74 MiB/s 0s
You can rename the container using the --name
flag:
$ singularity pull --name my_alpine.sif library://alpine
INFO: Downloading library image
2.08 MiB / 2.08 MiB [===========================================================================================] 100.00% 9.65 MiB/s 0s
The above example will save the image in the current directory as my_alpine.sif
Pulling an image from Docker hub
Many programs are available as Docker containers pre-built, and many of those are available on Docker Hub. For more details on using Docker containers with Singularity, see our section on Migrating from Docker to Singularity.
Here is a quick example of pulling an Alpine Docker container for use with Singularity.
$ singularity pull docker://alpine
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob df20fa9351a1 done
Copying config 0f5f445df8 done
Writing manifest to image destination
Storing signatures
2020/08/20 15:53:52 info unpack layer: sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c
INFO: Creating SIF file...
INFO: Build complete: alpine_latest.sif
Interacting with images
You can interact with images via the shell
, exec
, and run
commands. To
learn how to interact with images, let's first pull an image
lolcow_latest.sif
from the library.
singularity pull library://lolcow
shell
The shell
command allows you to spawn a new shell within your container and
interact with it as if it is a virtual machine.
$ singularity shell lolcow_latest.sif
Singularity>
The change in prompt indicates that you have entered the container. Once inside of a container, you are the same user as you are on the host system.
Singularity> whoami
choiyj
You will also have access to a few directories that you can access outside the
container, most notably, the directory you ran the container from and your home
directory. Try running pwd
and ls ~
inside the container to verify this.
Anything you write to these directories will stay around after you are done with the container. This is a significant difference from Docker, where the default is to close off the files in the container and you need to use "bind mounts" to manually connect file spaces.
To exit from a container, type exit
.
Singularity> exit
$
exec
The exec
command allows you to execute a custom command within a container by
specifying the image file. For instance, to execute the cowsay
program within
the lolcow_latest.sif
container:
$ singularity exec lolcow_latest.sif cowsay container camp rocks
______________________
< container camp rocks >
----------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
You can also use shell
command to run the program in the container.
$ singularity shell lolcow_latest.sif
Singularity> cowsay container camp rocks
______________________
< container camp rocks >
----------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
run
Singularity containers contain runscripts. These are predefined scripts which
define the actions of a container when user runs it. The runscript can be
performed with the run
command, or simply by calling the container as though
it were an executable.
$ singularity run lolcow_latest.sif
_________________________________________
/ You're ugly and your mother dresses you \
\ funny. /
-----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Submitting a Singularity job
In general, running Singularity commands is the same as running any kind of program when you prepare your SLURM script.
In this case, you put your Singularity commands (singularity exec <image>.sif
<command>
) right after the sbatch
directive lines you use to specify your
job resources. If the program needs to use multiple threads/cores on a node,
say 8, you would request 8 cores using #SBATCH --cpus-per-task=8
as you would
do with any other sbatch
script.
For situations where you are using MPI within the container (e.g., you would like to split your code over multiple nodes) or would like to use GPU resources, please see Using Singularity with MPI and GPUs on the Advanced Topics page.