Docker Resources
Per-cluster resources
| Type | Name pattern | Example (sind create cluster dev) |
|---|---|---|
| Network | <realm>-<cluster>-net |
sind-dev-net |
| Controller | <realm>-<cluster>-controller |
sind-dev-controller |
| Backup controller | <realm>-<cluster>-controller-backup |
sind-dev-controller-backup |
| Db | <realm>-<cluster>-db |
sind-dev-db |
| Api | <realm>-<cluster>-api |
sind-dev-api |
| Submitter | <realm>-<cluster>-submitter |
sind-dev-submitter |
| Worker | <realm>-<cluster>-worker-<N> |
sind-dev-worker-0 |
| Config volume | <realm>-<cluster>-config |
sind-dev-config |
Munge volume (not with identity clientIds) |
<realm>-<cluster>-munge |
sind-dev-munge |
| Data volume | <realm>-<cluster>-data |
sind-dev-data |
| State volume (backup controller only) | <realm>-<cluster>-state |
sind-dev-state |
Home volume (users only) |
<realm>-<cluster>-home |
sind-dev-home |
| Config helper (temporary, managed clusters only) | <realm>-<cluster>-config-helper |
sind-dev-config-helper |
Munge helper (temporary, not with identity clientIds) |
<realm>-<cluster>-munge-helper |
sind-dev-munge-helper |
Journal helper (temporary, monitoring only) |
<realm>-<cluster>-journal-helper |
sind-dev-journal-helper |
The helpers mount the config and munge volumes while sind create cluster writes the Slurm configuration and the munge key into them, and the realm’s journal volume while it makes an empty journal directory for each node of a monitored cluster, using the controller’s image, and are removed once they are done. They carry the sind.realm and sind.cluster labels, so sind delete cluster removes one that an interrupted create left behind (docker ps -a).
With identity clientIds there is no munge volume: the authentication secret, slurm.key, is on the config volume (see Slurm Configuration).
The default realm is sind and the default cluster name is default, resulting in prefixes like sind-default-*. See Realms for custom realm naming.
Global resources (mesh)
| Type | Name pattern | Example | Image |
|---|---|---|---|
| Mesh network | <realm>-mesh |
sind-mesh |
— |
| DNS container | <realm>-dns |
sind-dns |
coredns/coredns:1.15.0 |
| SSH container | <realm>-ssh |
sind-ssh |
sind’s default node image (runs sleep infinity) |
| SSH volume | <realm>-ssh-config |
sind-ssh-config |
— (keys written once, through the SSH container before it first starts) |
The mesh images do not follow defaults.image: the SSH relay runs sind’s default node image, which has the ssh client it needs (custom node images need not), and CoreDNS comes from Docker Hub. sind get mesh shows the images the mesh containers run. --pull pulls the mesh images when sind create cluster creates the mesh containers. An existing mesh keeps its containers and images until the realm’s last cluster is deleted.
Monitoring stack
A realm with a monitored cluster also has:
| Type | Name pattern | Example | Image |
|---|---|---|---|
| Grafana container | <realm>-grafana |
sind-grafana |
grafana/grafana:13.3.1 |
| Prometheus container | <realm>-prometheus |
sind-prometheus |
prom/prometheus:v3.16.0 |
| Loki container | <realm>-loki |
sind-loki |
grafana/loki:3.7.9 |
| Alloy container | <realm>-alloy |
sind-alloy |
grafana/alloy:v1.21.3 |
| Data volumes | <realm>-<service> |
sind-grafana, sind-prometheus, sind-loki, sind-alloy |
— |
| Journal volume | <realm>-journal |
sind-journal |
— (a directory <cluster>/<node> per monitored node) |
The stack runs on the mesh network only. Grafana, Prometheus and Loki have fixed addresses just below the DNS container’s and the DNS names grafana.<realm>.sind, prometheus.<realm>.sind and loki.<realm>.sind; Alloy has neither. Grafana, Prometheus and Loki are capped at 1 GB of memory, Alloy at 512 MB. Alloy runs as root in the host’s cgroup namespace, with the host’s /sys/fs/cgroup mounted read-only and the journal volume at /var/log/sind, so that it reads the nodes’ cgroups and journals; it needs no Docker socket.
The images do not follow defaults.image. sind pins each of them and pulls them with the node images when a monitored sind create cluster creates the stack. An existing stack keeps its containers and images until the realm’s last cluster is deleted, or until a newer sind replaces the containers, keeping the volumes, because their images, configuration or dashboards changed. Deleting the realm’s last cluster removes the stack’s containers, then the mesh, then the stack’s volumes and the journal volume.
Realm lock
| Type | Name pattern | Example | Image |
|---|---|---|---|
| Configuration-only network | <realm>-lock |
sind-lock |
— |
While a sind command creates or deletes clusters or workers, connects or disconnects containers, or powers nodes on, it holds the realm’s lock on the Docker daemon: this network, which it creates when it takes the lock and removes when it releases it (see Advisory locking). Docker refuses a second network of the same name, so only one command at a time holds it. A configuration-only network (docker network create --config-only) has the driver null: it takes no subnet from the address pools, has no bridge device, and no container joins it.
Besides sind.realm, its labels name the command that holds it (sind.lock.command, sind.lock.host, sind.lock.pid), let sind tell whether a holder on the same host still runs (sind.lock.boot-id, sind.lock.pid-ns), and mark one acquisition (sind.lock.token). sind get networks does not list it.
Volume mounts
| Volume | Mount point | Controller | Db | Api | Worker | Submitter |
|---|---|---|---|---|---|---|
<realm>-<cluster>-config |
/etc/slurm |
rw | ro | ro | ro | ro |
<realm>-<cluster>-munge |
/etc/munge |
ro | ro | ro | ro | ro (not with identity clientIds) |
<realm>-<cluster>-data |
/data |
rw | rw | rw | rw | rw |
<realm>-<cluster>-state |
/var/spool/slurmctld |
rw (backup controller pairs only) | — | — | — | — |
<realm>-<cluster>-home |
/home |
rw | rw | rw | rw | rw (users only) |
cvmfs plugin volume or host /cvmfs |
/cvmfs |
ro | ro | ro | ro | ro (storage.cvmfs only) |
<realm>-journal, directory <cluster>/<node> |
/var/log/journal |
rw | rw | rw | rw | rw (monitoring only) |
| tmpfs | /tmp |
configurable | configurable | configurable | configurable | configurable |
| tmpfs | /run |
exec,mode=755,size=64m | exec,mode=755,size=64m | exec,mode=755,size=64m | exec,mode=755,size=64m | exec,mode=755,size=64m |
| tmpfs | /run/lock |
— | — | — | — | — |
| shm | /dev/shm |
half the memory | half the memory | half the memory | half the memory | half the memory |
Files in the tmpfs mounts and /dev/shm count against the node’s memory limit, as do its services. Nodes get no swap: --memory-swap equals --memory.
Unmanaged clusters use the same mounts; their config volume starts empty.
SELinux relabeling (:z) is not used because containers run with --security-opt label=disable. This avoids expensive recursive relabeling of bind-mounted host directories, and lets systemd manage its cgroups without host policy changes. On a Docker daemon with SELinux labelling enabled, the nodes therefore run unconfined (spc_t instead of container_t): SELinux does not separate them from the host. A securityOpt such as label=type:... cannot change that, as Docker gives disable precedence.
-v sind-dev-config:/etc/slurm:rw # controller
-v sind-dev-config:/etc/slurm:ro # all others
-v sind-dev-munge:/etc/munge:ro # all nodes, not with identity clientIds
-v sind-dev-data:/data:rw # all nodes
-v sind-dev-state:/var/spool/slurmctld:rw # both controllers of a backup pair
-v sind-dev-home:/home:rw # all nodes, with users
--tmpfs /tmp:rw,nosuid,nodev,size=256m # configurable sizeHost path storage
With dataStorage.type: hostPath, or a hostPath without type, the data volume is replaced with a bind mount. If the config sets neither type nor hostPath, the --data flag of sind create cluster decides; its default, ., bind-mounts the working directory:
--mount type=bind,source=/path/on/host,target=/dataThe directory must exist on the Docker host: unlike -v, --mount fails on a missing source instead of creating it as an empty, root-owned directory.
Journal
On a monitored cluster, every node mounts its own directory of the realm’s journal volume, where journald writes the node’s persistent journal and Alloy reads it:
--mount type=volume,source=sind-journal,target=/var/log/journal,volume-subpath=dev/worker-0The node’s entrypoint caps that journal at 64 MB with the journald drop-in /etc/systemd/journald.conf.d/zz-sind-monitoring.conf.
CVMFS
With storage.cvmfs: true, every node mounts CVMFS read-only at /cvmfs, from the cvmfs Docker volume plugin when it is installed and enabled, otherwise from the Docker host’s /cvmfs as a slave mount, so repositories the host’s autofs mounts on demand appear in the nodes:
--mount type=volume,volume-driver=cvmfs,source=cvmfs,target=/cvmfs,readonly # volume plugin
--mount type=bind,source=/cvmfs,target=/cvmfs,readonly,bind-propagation=rslave # host /cvmfsAll clusters share the plugin’s cvmfs volume; sind does not create or remove it. See Using CVMFS.
Container labels
sind applies labels to containers for filtering and metadata:
| Label | Example | Description |
|---|---|---|
sind.realm |
sind |
Realm namespace |
sind.cluster |
dev |
Cluster name |
sind.role |
worker |
Node role |
sind.managed |
true |
Whether sind manages Slurm on the node: false for unmanaged workers, db and api nodes and for every node of an unmanaged cluster. Nodes created before this label existed count as managed. |
sind.slurm.version |
25.11.9 |
Slurm version, empty for an unmanaged cluster |
sind.data.hostpath |
/home/user/project |
Resolved data mount host path, empty with the data volume |
sind.data.mountpath |
/shared |
Data mount point (storage.dataStorage.mountPath, /data by default) |
sind.cvmfs |
hostPath |
How the node mounts CVMFS, with storage.cvmfs: volume (plugin) or hostPath (host /cvmfs); empty without it |
sind.users |
alice:1000:1000 bob:2001:3000 |
The cluster users, as space-separated name:uid:gid entries; empty without users |
sind.groups |
alice:1000 hpc:3000:carol |
The cluster groups, private groups included, as space-separated name:gid entries with :member+member... for supplementary members; empty without users and groups |
sind.identity |
clientIds |
The identity mode: local, nssSlurm or clientIds |
sind.monitoring |
true |
Whether the node’s cluster is monitored: true or false. For listings and filters only: sind tells a monitored node by its mount of the realm’s journal volume, which an image cannot forge |
Every node container gets all of these labels, empty where there is nothing to record. Docker merges the image’s labels into the container’s, so a label sind left out could otherwise come from the node image.
A cluster’s network and volumes carry sind.realm and sind.cluster; the mesh network, the SSH volume, the monitoring stack’s volumes and the realm lock carry sind.realm only. The monitoring stack’s containers carry no sind.realm, so that they do not count as a cluster of the realm, only the Compose labels and sind.observability.version, a fingerprint of their images, configuration and dashboards. sind get networks and sind get volumes find resources by these labels, leaving out the realm lock. sind delete cluster --all finds clusters by them and by the labels of their containers, which also covers clusters made before sind labelled networks and volumes (v0.9.0).
Every node container, the mesh’s DNS and SSH containers, the monitoring stack’s containers, and every network and volume but the realm lock also carry Docker Compose labels, so Compose-aware tools group them: the project is <realm>-<cluster> (<realm>-mesh for the mesh), the service is the node’s role (dns, ssh or the monitoring stack’s grafana, prometheus, loki and alloy in the mesh), the container number is 1, or N+1 for worker-N and 2 for controller-backup, and networks and volumes name themselves net, mesh, the volume type, ssh-config, the monitoring service or journal.