Containers and Images

Overview

Teaching: 20 min
Exercises: 5 min

Questions

• How to pull Apptainer images from the libraries?

• How to run commands inside the containers?

Objectives

• Learn to search and pull images from the Sylabs Singularity library and Docker Hub.

• Interact with the containers using the command-line interface.

Apptainer is a Module at MSI

MSI provides a modulefile for Singularity and Apptainer. Currently we provide this module under both naming conventions, so you can load it by running either of the equivalent commands:

module load apptainer

or

module load singularity

But it is already available?

You may notice that the singularity and apptainer commands are available before you load the module. This is because due to some peculiarities with how this software interacts with the permissions model on our network storage, we are not always able to use a regular module-based deployment method. However, the module-based deployment is the intended workflow and you should get used to loading the module before using Apptainer.

The Apptainer Command Line Interface

Apptainer provides a command-line interface (CLI) to interact with the containers. You can search, build or run containers in a single line.

You can check the version of the Apptainer or Singularity command you are using with the --version option:

singularity --version
# This works for both Singularity and Apptainer, which installs a link named `singularity` to maintain compatibility.
# In the future you may need to use `apptainer --version`

For this training we recommend Apptainer >= 1.0 or Singularity >= 3.5. Older versions may not have some of the features or behave differently. If you need to install or upgrade Apptainer/Singularity please refer to the Setup section. When asking for support please remember to include the version of Apptainer or Singularity being used, as in the output of the above commands.

You can check the available options and subcommands using --help:

apptainer --help

Downloading Images

Apptainer can store, search and retrieve images in registries. Images built by other users can be accessible using the CLI, can be pulled down, and become containers at runtime.

Sylabs, the developers of one Singularity flavor, hosts a public image registry, the Singularity Container Library where many user built images are available.

Configuring the Sylab Registry

The Linux Foundation flavor, Apptainer, does not point by default to the Sylab registry as previous versions did. MSI configures its Apptainer installation to provide the Sylab registry, but if you are running on other resources you may need to add it yourself. You can do so by running these commands (documented here):

apptainer remote add --no-login SylabsCloud cloud.sycloud.io

INFO:    Remote "SylabsCloud" added.

apptainer remote use SylabsCloud

INFO:    Remote "SylabsCloud" now in use.

You can see the currently configured registries by running:

apptainer remote list

Cloud Services Endpoints
========================

NAME           URI                  ACTIVE  GLOBAL  EXCLUSIVE
DefaultRemote  cloud.apptainer.org  NO      YES     NO
SylabsCloud    cloud.sycloud.io     YES     NO      NO
...

Once you have confirmed (or setup) a working registry you can use search and pull. The command search lists containers of interest and shows information about groups and collections. For example:

# this command can take around a minute to complete
apptainer search centos7

No users found for 'centos7'

Found 1 collections for 'centos7'
        library://shahzebmsiddiqui/easybuild-centos7

Found 15 containers for 'centos7'
        library://gmk/default/centos7-devel
                Tags: latest
...

What if the search times out?

It is fairly common for the no-login access to the Sylabs cloud to take a while to respond, and possible time out before the search can complete. In these cases, you can get at the same information in a web browser by searching their library website

Downloading an image from the Container Library is pretty straightforward:

apptainer pull library://gmk/default/centos7-devel

and the image is stored locally as a .sif file (centos7-devel_latest.sif, in this case).

Docker Images

Fortunately, Apptainer is also compatible with Docker images. There are many more registries with Docker images. Docker Hub is one of the largest libraries available, and any image hosted on the hub can be easily downloaded with the docker:// URL as reference:

apptainer pull docker://centos:centos7

Running Containers

There are several ways to interact with images and start containers. Here we will review how to initialize a shell environment and how to execute directly a command.

Initializing a shell and exiting it

The shell command initializes a new interactive shell inside the container.

apptainer shell centos7-devel_latest.sif

Apptainer>

In this case, the container works as a lightweight virtual machine in which you can execute commands. Remember, inside the container you have the same user and permissions.

Apptainer> id

uid=1001(myuser) gid=1001(myuser) groups=1001(myuser),500(myothergroup)

Now quit the container by typing

Apptainer> exit

or hitting Ctrl + D. Note that when exiting from the Apptainer image all the running processes are killed (stopped). Changes saved into bound directories are preserved. By default anything else in the container is lost (we’ll see later about writable images).

Bound directories

When an outside directory is accessible also inside Apptainer we say it is bound, or bind mounted. The path to access it may differ but anything you do to its content outside is visible inside and vice-versa. By default, Apptainer binds the home of the user, /tmp and $PWD into the container. This means your files at hostname:~/ are accessible inside the container. You can specify additional bind mounts using the --bind option. For example, let’s say /scratch.global is available in the host, and you would like to have access to global scratch inside the container (here, host refers to the computer/server that you are running apptainer on). Then let’s do

apptainer shell --bind /scratch.global:/mnt centos7-devel_latest.sif

Here, the colon : separates the path to the directory on the host (/scratch.global/) from the mounting point (/mnt/) inside of the container. More information on binding is provided later.

Let’s check that this works:

Apptainer> ls /mnt/$USER
bin                        etc                  SITECONF           slc7_aarch64_gcc530
bootstrap.sh               external             slc5_amd64_gcc434  slc7_aarch64_gcc700
...

URLs as input

Each of the different commands to set a container from a local .sif also accepts the URL of the image as input. For example, starting a shell with Scientific Linux 6 is as easy as

apptainer shell docker://sl:6

2020/12/17 21:42:46  info unpack layer: sha256:e0a6b33502f39d76f7c70213fa5b91688a46c2217ad9ba7a4d1690d33c6675ef
INFO:    Creating SIF file...
Apptainer>

Executing commands

The command exec starts the container from a specified image and executes a command inside it. Let’s use the official Docker image of ROOT to start ROOT inside a container:

apptainer exec docker://rootproject/root root -b

INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...

   ------------------------------------------------------------------
  | Welcome to ROOT 6.22/06                        https://root.cern |
  | (c) 1995-2020, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Nov 27 2020, 15:14:08                 |
  | From tags/v6-22-06@v6-22-06                                      |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'       |
   ------------------------------------------------------------------

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
root [0]

And just like that, ROOT can be used in any laptop, large-scale cluster or grid system with Apptainer available.

Execute Python with PyROOT available

Using the official Docker image of ROOT, start a Python session with PyROOT available.

Solution

apptainer exec docker://rootproject/root python3

INFO:    Using cached SIF image
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
>>> import ROOT
>>> # Now you can work with PyROOT, creating a histogram for example
>>> h = ROOT.TH1F("myHistogram", "myTitle", 50, -10, 10)

Key Points

• Use apptainer --version or singularity --version to know what you are using and to communicate it if asking for support

• A container can be started from a local .sif or directly with the URL of the image.

• Apptainer is also compatible with Docker images, providing access to the large collection of images hosted by Docker Hub.

• Get a shell inside of your container with apptainer shell <path/URL to image>

• Execute a command inside of your container with apptainer exec <path/URL> <command>

• Bind outside directories with --bind