This lesson is in the early stages of development (Alpha version)

previous episode

Reproducible computational environments using containers: Introduction to Apptainer/Singularity

lesson home

Apptainer instances

Overview

Teaching: 60 min
Exercises: 10 min
Questions

  • How can I keep my container running in the background?

  • What are the use cases for instances?

Objectives

  • Run containers in a detached mode to keep services up.

  • Deploy instances via definition files.

Note: This won’t work on MSI until the rollout of Rocky8 in May, 2024

As we have studied in previous chapters, commands such as run and shell allocate Apptainer containers in the foreground, stopping any process running inside the container after logout. This behavior suits the use case of containers for executing interactive commands in a well-defined environment, but there are cases when running processes in the background is convenient. For example, when a web application like a Jupyter notebook is deployed inside a container, it is desired to keep the container up while it waits for connections from the web browser.

Apptainer provides the concept of instances to deploy services in the background. While Docker is a common choice of tool for setting services, Apptainer has the advantage of working without requiring any special permissions (like when you are working in a cluster provided by your university/laboratory). In this chapter we will learn the basics about their capabilities and some use cases as examples.

Instances from image files

To start an instance, Apptainer provides the command instance. To exemplify, let’s pull the CentOS image used in previous chapters

apptainer pull docker://centos:centos7

The image must be started in the following way:

apptainer instance start centos_centos7.sif mycentos7

In this example, the .sif is the image downloaded from Dockerhub, and mycentos7 is the name that we have assigned to the instance. Instead of opening a shell session or executing a command, the container is running in the background.

Confirm that the instance is running using the instance list command

apptainer instance list

INSTANCE NAME    PID      IP    IMAGE
mycentos7        10782          /home/myuser/centos_centos7.sif

To interact with the instance, the commands exec and shell are available. The instance must be referred as instance://name. For example, to open a shell inside the CentOS instance:

apptainer shell instance://mycentos7

Remember that exiting the shell instance will not stop the container. For doing so, use instance stop:

apptainer instance stop mycentos7

You can confirm the instance doesn’t exist with instance list.

Instances with bind paths

When starting an instance, the same options for bind directories between the host and the container as running an interactive session are available. For example, if you want a directory mounted inside the instance, use the --bind option:

apptainer instance start --bind /home/user/mydata:/data centos_centos7.sif mycentos7

binding the directory mydata/ from the host as /data inside the instance.

A web server as an instance

One of the main purposes of the Apptainer instances is deploying services with customized environments. Before moving to more complex use cases, let’s start with a basic example: a web service showing a HTML with a message.

Let’s write a basic index.html file as:

<!DOCTYPE html>
<html>
<head>
<title>My awesome service</title>
</head>
<body>
<h1>Hello world!</h1>
<p>If you see this page, my awesome service is up and running.</p>
</body>
</html>

If you are not familiar with HTML take a quick look at the HTML Tutorial, but it is not mandatory. What really matters is having a minimal webpage that our server will show.

Now, let’s prepare a basic web server using Python http.server. Create a definition file, saved as basicServer.def, which contains:

Bootstrap: docker
From: ubuntu:20.04

%post
    apt-get update -y
    apt-get install -y python3.9

%files
    index.html /tmp/index.html

%startscript
   cd /tmp
   python3.9 -m http.server 8850

If you recall the chapter about definition files, this definition file will pull the official Ubuntu image from Dockerhub, and will install Python3.9. In addition, it copies index.html in /tmp inside the container. When the instance starts, commands specified on %startscript are executed. On this example, http.server will be executed, serving a page in the port 8850 (you can use any other port if 8850 is busy with another service).

Let’s build an image from the definition. Remember that building images requires either superuser permissions or using the flag --fakeroot as

apptainer build --fakeroot basicServer.sif basicServer.def

Now, let’s start an instance named myWebService with the image that we just built

apptainer instance start --no-mount tmp basicServer.sif myWebService

Reminder from the previous chapter: with --no-mount tmp we are asking Apptainer to NOT bind /tmp from the host to the instance (it is mounted by default), we use instead an isolated /tmp inside the instance where index.html has been copied.

You can confirm in the terminal that the web service is up using curl as

curl http://localhost:8850

<!DOCTYPE html>
<html>
<head>
<title>Welcome to my service!</title>
</head>
<body>
<h1>Hello world!</h1>
<p>If you see this page, my awesome service is up and running.</p>
</body>
</html>

If you are executing Apptainer locally, try to open http://localhost:8850. This approach will also work if you are running Apptainer in an Open OnDemand desktop session, provided you also run the browser inside the session.

Remember to stop the instance once you are done.

Serving a Jupyter notebook with custom environment

As an example of the capabilities of instances as services, let’s extend our definition file to deploy a Jupyter notebook server with a customized environment.

What if we provide a Jupyter notebook ready to use ROOT? If you remember our example from the definition files chapter, at this point it must be almost straightforward:

Bootstrap: docker
From: ubuntu:20.04

%post
    apt-get update -y
    apt-get install -y python3
    apt-get install -y python3-pip
    pip install notebook

    apt-get install wget -y
    export DEBIAN_FRONTEND=noninteractive
    apt-get install dpkg-dev cmake g++ gcc binutils libx11-dev libxpm-dev \
    libxft-dev libxext-dev python libssl-dev libgsl0-dev libtiff-dev -y
    cd /opt
    wget https://root.cern/download/root_v6.22.06.Linux-ubuntu20-x86_64-gcc9.3.tar.gz
    tar -xzvf root_v6.22.06.Linux-ubuntu20-x86_64-gcc9.3.tar.gz

%environment
    export PATH=/opt/root/bin:$PATH
    export LD_LIBRARY_PATH=/opt/root/lib:$LD_LIBRARY_PATH
    export PYTHONPATH=/opt/root/lib

%startscript
   jupyter notebook --port 8850

Save the definition file as jupyterWithROOT.def, and let’s build an image called jupyterWithROOT.sif

apptainer build --fakeroot jupyterWithROOT.sif jupyterWithROOT.def

Now, start an instance named mynotebook with our brand-new image

apptainer instance start jupyterWithROOT.sif mynotebook

and confirm that the instance is up

apptainer instance list

INSTANCE NAME    PID      IP    IMAGE
mynotebook       10720          /home/myuser/jupyterWithROOT.sif

If you go to http://localhost:8850 (with SSH tunneling if needed), you will find out that for security reasons the Jupyter webapp will ask for an access token. Fortunately, you can get the token listing the URL of active servers using the jupyter notebook list command. To execute the command inside the instance, use sigularity exec:

apptainer exec instance://notebook jupyter notebook list

Currently running servers:
http://localhost:8850/?token=12asldc9b2084f9b664b39a6246022312bc9c605b :: /home/myHome

Open the URL with the token, and you will be able to see the Jupyter interface. Try to open a new notebook and write in a cell to confirm that ROOT is available:

import ROOT
# Now you can work with PyROOT, creating a histogram for example
h = ROOT.TH1F("myHistogram", "myTitle", 50, -10, 10)
h.FillRandom("gaus", 10000)

c = ROOT.TCanvas("myCanvasName","The Canvas Title",800,600)
h.Draw()
c.Draw()

The bottom line: with any Jupyter notebook that you write, you can provide an Apptainer image that will set the environment required to execute the cells. It doesn’t matter if yourself or someone else comes in one, five, ten years, your code will work independently of the software available in your computer as far as Apptainer is available!

A Jupyter notebook with Uproot available

Can you setup a Jupyter notebook server with Uproot available in Apptainer?

Hint: Uproot can be installed using pip.

Solution

Bootstrap: docker
From: ubuntu:20.04

%post
   apt-get update -y
   apt-get install -y python3
   apt-get install -y python3-pip
   pip install notebook
   pip install uproot

%startscript
  jupyter notebook --port 8850

Confirm that Uproot is available opening a notebook and executing in a cell

import uproot
print(uproot.__doc__)

Key Points

  • Instances allow to setup services via Apptainer images or definition files.

  • Code provided in Jupyter notebooks can be accompanied by a Apptainer image with the environment needed for its execution, ensuring the reproducibility of the results.

previous episode

lesson home