Skip to content
Docker Resources

Docker Resources

Per-cluster resources

Type Name pattern Example (sind create cluster dev)
Network <realm>-<cluster>-net sind-dev-net
Controller <realm>-<cluster>-controller sind-dev-controller
Backup controller <realm>-<cluster>-controller-backup sind-dev-controller-backup
Db <realm>-<cluster>-db sind-dev-db
Api <realm>-<cluster>-api sind-dev-api
Submitter <realm>-<cluster>-submitter sind-dev-submitter
Worker <realm>-<cluster>-worker-<N> sind-dev-worker-0
Config volume <realm>-<cluster>-config sind-dev-config
Munge volume (not with identity clientIds) <realm>-<cluster>-munge sind-dev-munge
Data volume <realm>-<cluster>-data sind-dev-data
State volume (backup controller only) <realm>-<cluster>-state sind-dev-state
Home volume (users only) <realm>-<cluster>-home sind-dev-home
Config helper (temporary, managed clusters only) <realm>-<cluster>-config-helper sind-dev-config-helper
Munge helper (temporary, not with identity clientIds) <realm>-<cluster>-munge-helper sind-dev-munge-helper
Journal helper (temporary, monitoring only) <realm>-<cluster>-journal-helper sind-dev-journal-helper

The helpers mount the config and munge volumes while sind create cluster writes the Slurm configuration and the munge key into them, and the realm’s journal volume while it makes an empty journal directory for each node of a monitored cluster, using the controller’s image, and are removed once they are done. They carry the sind.realm and sind.cluster labels, so sind delete cluster removes one that an interrupted create left behind (docker ps -a).

With identity clientIds there is no munge volume: the authentication secret, slurm.key, is on the config volume (see Slurm Configuration).

The default realm is sind and the default cluster name is default, resulting in prefixes like sind-default-*. See Realms for custom realm naming.

Global resources (mesh)

Type Name pattern Example Image
Mesh network <realm>-mesh sind-mesh —
DNS container <realm>-dns sind-dns coredns/coredns:1.15.0
SSH container <realm>-ssh sind-ssh sind’s default node image (runs sleep infinity)
SSH volume <realm>-ssh-config sind-ssh-config — (keys written once, through the SSH container before it first starts)

The mesh images do not follow defaults.image: the SSH relay runs sind’s default node image, which has the ssh client it needs (custom node images need not), and CoreDNS comes from Docker Hub. sind get mesh shows the images the mesh containers run. --pull pulls the mesh images when sind create cluster creates the mesh containers. An existing mesh keeps its containers and images until the realm’s last cluster is deleted.

Monitoring stack

A realm with a monitored cluster also has:

Type Name pattern Example Image
Grafana container <realm>-grafana sind-grafana grafana/grafana:13.3.1
Prometheus container <realm>-prometheus sind-prometheus prom/prometheus:v3.16.0
Loki container <realm>-loki sind-loki grafana/loki:3.7.9
Alloy container <realm>-alloy sind-alloy grafana/alloy:v1.21.3
Data volumes <realm>-<service> sind-grafana, sind-prometheus, sind-loki, sind-alloy —
Journal volume <realm>-journal sind-journal — (a directory <cluster>/<node> per monitored node)

The stack runs on the mesh network only. Grafana, Prometheus and Loki have fixed addresses just below the DNS container’s and the DNS names grafana.<realm>.sind, prometheus.<realm>.sind and loki.<realm>.sind; Alloy has neither. Grafana, Prometheus and Loki are capped at 1 GB of memory, Alloy at 512 MB. Alloy runs as root in the host’s cgroup namespace, with the host’s /sys/fs/cgroup mounted read-only and the journal volume at /var/log/sind, so that it reads the nodes’ cgroups and journals; it needs no Docker socket.

The images do not follow defaults.image. sind pins each of them and pulls them with the node images when a monitored sind create cluster creates the stack. An existing stack keeps its containers and images until the realm’s last cluster is deleted, or until a newer sind replaces the containers, keeping the volumes, because their images, configuration or dashboards changed. Deleting the realm’s last cluster removes the stack’s containers, then the mesh, then the stack’s volumes and the journal volume.

Realm lock

Type Name pattern Example Image
Configuration-only network <realm>-lock sind-lock —

While a sind command creates or deletes clusters or workers, connects or disconnects containers, or powers nodes on, it holds the realm’s lock on the Docker daemon: this network, which it creates when it takes the lock and removes when it releases it (see Advisory locking). Docker refuses a second network of the same name, so only one command at a time holds it. A configuration-only network (docker network create --config-only) has the driver null: it takes no subnet from the address pools, has no bridge device, and no container joins it.

Besides sind.realm, its labels name the command that holds it (sind.lock.command, sind.lock.host, sind.lock.pid), let sind tell whether a holder on the same host still runs (sind.lock.boot-id, sind.lock.pid-ns), and mark one acquisition (sind.lock.token). sind get networks does not list it.

Volume mounts

Volume Mount point Controller Db Api Worker Submitter
<realm>-<cluster>-config /etc/slurm rw ro ro ro ro
<realm>-<cluster>-munge /etc/munge ro ro ro ro ro (not with identity clientIds)
<realm>-<cluster>-data /data rw rw rw rw rw
<realm>-<cluster>-state /var/spool/slurmctld rw (backup controller pairs only) — — — —
<realm>-<cluster>-home /home rw rw rw rw rw (users only)
cvmfs plugin volume or host /cvmfs /cvmfs ro ro ro ro ro (storage.cvmfs only)
<realm>-journal, directory <cluster>/<node> /var/log/journal rw rw rw rw rw (monitoring only)
tmpfs /tmp configurable configurable configurable configurable configurable
tmpfs /run exec,mode=755,size=64m exec,mode=755,size=64m exec,mode=755,size=64m exec,mode=755,size=64m exec,mode=755,size=64m
tmpfs /run/lock — — — — —
shm /dev/shm half the memory half the memory half the memory half the memory half the memory

Files in the tmpfs mounts and /dev/shm count against the node’s memory limit, as do its services. Nodes get no swap: --memory-swap equals --memory.

Unmanaged clusters use the same mounts; their config volume starts empty.

SELinux relabeling (:z) is not used because containers run with --security-opt label=disable. This avoids expensive recursive relabeling of bind-mounted host directories, and lets systemd manage its cgroups without host policy changes. On a Docker daemon with SELinux labelling enabled, the nodes therefore run unconfined (spc_t instead of container_t): SELinux does not separate them from the host. A securityOpt such as label=type:... cannot change that, as Docker gives disable precedence.

-v sind-dev-config:/etc/slurm:rw      # controller
-v sind-dev-config:/etc/slurm:ro      # all others
-v sind-dev-munge:/etc/munge:ro       # all nodes, not with identity clientIds
-v sind-dev-data:/data:rw             # all nodes
-v sind-dev-state:/var/spool/slurmctld:rw  # both controllers of a backup pair
-v sind-dev-home:/home:rw             # all nodes, with users
--tmpfs /tmp:rw,nosuid,nodev,size=256m  # configurable size

Host path storage

With dataStorage.type: hostPath, or a hostPath without type, the data volume is replaced with a bind mount. If the config sets neither type nor hostPath, the --data flag of sind create cluster decides; its default, ., bind-mounts the working directory:

--mount type=bind,source=/path/on/host,target=/data

The directory must exist on the Docker host: unlike -v, --mount fails on a missing source instead of creating it as an empty, root-owned directory.

Journal

On a monitored cluster, every node mounts its own directory of the realm’s journal volume, where journald writes the node’s persistent journal and Alloy reads it:

--mount type=volume,source=sind-journal,target=/var/log/journal,volume-subpath=dev/worker-0

The node’s entrypoint caps that journal at 64 MB with the journald drop-in /etc/systemd/journald.conf.d/zz-sind-monitoring.conf.

CVMFS

With storage.cvmfs: true, every node mounts CVMFS read-only at /cvmfs, from the cvmfs Docker volume plugin when it is installed and enabled, otherwise from the Docker host’s /cvmfs as a slave mount, so repositories the host’s autofs mounts on demand appear in the nodes:

--mount type=volume,volume-driver=cvmfs,source=cvmfs,target=/cvmfs,readonly     # volume plugin
--mount type=bind,source=/cvmfs,target=/cvmfs,readonly,bind-propagation=rslave  # host /cvmfs

All clusters share the plugin’s cvmfs volume; sind does not create or remove it. See Using CVMFS.

Container labels

sind applies labels to containers for filtering and metadata:

Label Example Description
sind.realm sind Realm namespace
sind.cluster dev Cluster name
sind.role worker Node role
sind.managed true Whether sind manages Slurm on the node: false for unmanaged workers, db and api nodes and for every node of an unmanaged cluster. Nodes created before this label existed count as managed.
sind.slurm.version 25.11.9 Slurm version, empty for an unmanaged cluster
sind.data.hostpath /home/user/project Resolved data mount host path, empty with the data volume
sind.data.mountpath /shared Data mount point (storage.dataStorage.mountPath, /data by default)
sind.cvmfs hostPath How the node mounts CVMFS, with storage.cvmfs: volume (plugin) or hostPath (host /cvmfs); empty without it
sind.users alice:1000:1000 bob:2001:3000 The cluster users, as space-separated name:uid:gid entries; empty without users
sind.groups alice:1000 hpc:3000:carol The cluster groups, private groups included, as space-separated name:gid entries with :member+member... for supplementary members; empty without users and groups
sind.identity clientIds The identity mode: local, nssSlurm or clientIds
sind.monitoring true Whether the node’s cluster is monitored: true or false. For listings and filters only: sind tells a monitored node by its mount of the realm’s journal volume, which an image cannot forge

Every node container gets all of these labels, empty where there is nothing to record. Docker merges the image’s labels into the container’s, so a label sind left out could otherwise come from the node image.

A cluster’s network and volumes carry sind.realm and sind.cluster; the mesh network, the SSH volume, the monitoring stack’s volumes and the realm lock carry sind.realm only. The monitoring stack’s containers carry no sind.realm, so that they do not count as a cluster of the realm, only the Compose labels and sind.observability.version, a fingerprint of their images, configuration and dashboards. sind get networks and sind get volumes find resources by these labels, leaving out the realm lock. sind delete cluster --all finds clusters by them and by the labels of their containers, which also covers clusters made before sind labelled networks and volumes (v0.9.0).

Every node container, the mesh’s DNS and SSH containers, the monitoring stack’s containers, and every network and volume but the realm lock also carry Docker Compose labels, so Compose-aware tools group them: the project is <realm>-<cluster> (<realm>-mesh for the mesh), the service is the node’s role (dns, ssh or the monitoring stack’s grafana, prometheus, loki and alloy in the mesh), the container number is 1, or N+1 for worker-N and 2 for controller-backup, and networks and volumes name themselves net, mesh, the volume type, ssh-config, the monitoring service or journal.

Last updated on