Design Overview
One-shot, not reconciling
While the cluster configuration resembles a Kubernetes manifest, sind is not a reconciling controller. The configuration is a one-shot input:
sind create clusterinterprets the manifest once to generate the cluster- sind does not watch or reconcile cluster state
- sind does not automatically repair drift or failures
sind provides commands for inspection (get), modification (create/delete worker), and simulation (power), but these are imperative operations, not declarative state management.
This is intentional: sind is a development and testing tool, not a production cluster controller.
Container-per-node
Each Slurm node runs as a separate Docker container with systemd as PID 1. This provides a realistic environment where munge, sshd, and Slurm daemons run as they would on bare metal.
Containers require specific security options for systemd:
--security-opt writable-cgroups=true— allows systemd to manage cgroups--cgroupns=private— private cgroup namespace--security-opt label=disable— lets systemd manage its cgroups, and the nodes use host bind mounts, on Docker daemons with SELinux labelling enabled, without host policy changes. The nodes then run unconfined (spc_tinstead ofcontainer_t), so SELinux does not separate them from the host--tmpfs /run:exec,mode=755,size=64m— systemd runtime directory, sized so that a volatile journal cannot grow into the node’s memory--tmpfs /run/lock— systemd lock files--entrypoint /bin/shwith a short script — PID 1 moves itself intoinit.scope, enables every available cgroup controller in the root cgroup’scgroup.subtree_control, then execs/sbin/init
docker exec puts its process in the container’s root cgroup unless that cgroup has controllers enabled, and with a process there systemd can no longer enable them (cgroup v2’s no internal processes rule). Enabling them before systemd starts keeps a docker exec of sind’s from racing systemd’s boot, so Delegate=yes daemons such as slurmd always get the memory and cpu controllers.
Concurrency
Creating and deleting clusters and workers, deleting a realm, and connecting and disconnecting containers acquire a per-realm lock to serialize concurrent modifications, and so do power on, reboot and cycle, which start a stopped mesh and rewrite DNS records. Read-only operations and the other power commands do not take it. The lock is a file lock (flock) in the user’s state directory, which orders that user’s commands, and a configuration-only network <realm>-lock on the Docker daemon, which orders every client of the daemon: other users, CI jobs that share its socket. A lock that a killed command left on the same host is taken over; one from another host or container is removed by hand. Different realms operate independently — see Realms.
Creation flow
┌ PreflightCheck → createResources ───────────────────┐
├ resolveInfra (DNS IP ║ SSH key ║ Slurm version) ─────┤
├ DetectCVMFS (storage.cvmfs only) ────────────────────┼→ setupNodes
└ CheckNsdelegate ─────────────────────────────────────┘
│
registerNodes ║ enableSlurm → createSlurmAccounts ║ createHomes ║ ensureMonitoring
│
registerStackDNS (monitoring only)
│
*ClustercreateResourcescreates the cluster network and connects the SSH relay and the realm’s external containers to it, the config volume and its Slurm configuration (managed clusters only; with a managed api node alsojwt_hs256.key), the munge volume and key (with identityclientIds,slurm.keyon the config volume instead), the data volume (unless the data is a host path), for a backup controller pair the state volume, withusersthe home volume and, on a monitored cluster, the realm’s journal volume<realm>-journal, if it does not exist yet, with an empty directory<cluster>/<node>for each node, all in parallel.resolveInfrastarts the mesh DNS and SSH relay if they are stopped, and looks up the mesh DNS IP, the SSH public key and, for managed clusters, the Slurm version of the controller’s image, while the resources are created. On a monitored cluster the Slurm version comes from the check of the images for the endpoints that the monitoring stack uses, which fails the creation, before any node exists, for an image that cannot serve them.DetectCVMFS(storage.cvmfsonly) picks how the nodes mount CVMFS, also while the resources are created: thecvmfsDocker volume plugin if one is enabled, otherwise a bind mount of the host’s/cvmfs, first tried in a throwaway container of the controller’s image.CheckNsdelegatereads/proc/self/mountsin a throwaway container of the controller’s image, also while the resources are created, and fails the creation, before any node container exists, when the cgroup2 mount lacksnsdelegate.nsdelegateis an option of the whole cgroup2 hierarchy, which the kernel lists in every cgroup2 mount, so the container shows the option of the kernel the Docker daemon runs on.- Each distinct node image the daemon does not have, or with
--pulleach one, is pulled once, concurrently, before the helper containers, the version check, the CVMFS check, the nsdelegate check and the nodes run it; none of these containers is created with--pull always, and no node’sdocker createpulls. On a monitored cluster the monitoring stack’s images are pulled with the node images. A new mesh’s DNS and relay containers, whichEnsureMeshcreates before, get their images the same way: each is pulled, when the daemon does not have it or with--pull, in the row of its part of the mesh step, before itsdocker create. - The helper containers that write the Slurm configuration and the munge key copy each file in with its final mode, then hand the secrets to their owner with
chown;docker rm -fstops them. setupNodescreates, waits for, and sets up every node: nss_slurm on managed workers with identitynssSlurmorclientIds, the check for slurmrestd and its systemd drop-in on a managed api node, the cluster users and groups where the identity mode puts them, SSH and host keys. The setup is onedocker execper node, a script that stops at its first failing step; the error names the step and shows what it wrote.enableSlurm(managed clusters only) first starts mariadb, the accounting database and slurmdbd on a managed db node, then slurmctld and slurmd, slurmrestd on a managed api node, andsackdon the submitter with identityclientIds. On a monitored cluster it then waits until slurmctld serves the HTTP endpoints that the monitoring stack uses.createSlurmAccounts(accountsonly) waits until slurmdbd lists the cluster, then creates the Slurm accounts, the users’ associations and the coordinators withsacctmgr -ioncontroller.createHomescreates the users’ home directories on the shared home volume, once, oncontroller(usersonly).ensureMonitoring(monitoringonly) creates, starts or upgrades the realm’s monitoring stack, waits until it is ready and adds the new nodes to it. When the concurrent steps are done,registerStackDNSadds the stack’s names to the mesh DNS.- If any step fails,
sind create clusterremoves what it created, the mesh only if no other cluster uses it, and a monitoring stack it created only if no other cluster of the realm is monitored.
The creation reports these phases as the steps of its progress display: mesh (EnsureMesh), pull images, preflight (the four branches above), nodes (setupNodes, a target for each node), mesh registration (registerNodes), slurm (enableSlurm, a target for each managed node with a Slurm daemon), accounts (createSlurmAccounts), home directories (createHomes), monitoring (ensureMonitoring) and, after a failure, rollback.
Each node is created, monitored, and probed in a single pipeline — no barrier between node creation and readiness checking. Early-starting nodes begin probing while later nodes are still being created.
Mesh registration (batch DNS ║ known_hosts), Slurm enablement, the home directories and, on a monitored cluster, the monitoring stack run concurrently after all nodes are ready. Slurm resolves the nodes’ short hostnames on the cluster network, which the nodes join with gateway priority ahead of the mesh (see Networking), so it does not wait for the mesh DNS records.
Readiness probes
sind waits for each node to become ready before returning success. Probes are accelerated by two event sources:
- Docker events — a single
docker eventsstream watches all cluster containers for start/die events - Systemd D-Bus monitors — per-node
busctl monitor --watch-bind=yesstreams watch for unit state changes (e.g., sshd.service becoming active or slurmd.service failing). They run until the command ends, through the waits for the Slurm daemons and the accounts
When an event of the node arrives, its probes re-evaluate immediately instead of waiting for the next poll tick; the events queued by then go with it, so a burst of unit changes during boot costs one probe round. Each node’s wait receives only its own container’s events. If event sources are unavailable, sind falls back to poll-only mode.
Each round runs a node’s checks in order and stops at the first that fails. A check that has passed is not run again in later rounds of the same wait, until an event says that what it checks may have changed: an event of its systemd unit for the sshd, munge and Slurm daemon checks (a failed munge or Slurm daemon unit then ends the wait), any other event of the container for every check. After a full event buffer, or once a monitor has stopped, and in poll-only mode, the passed checks run again. While systemd boots, a round then costs one docker exec instead of a docker inspect and an exec.
| Check | Description |
|---|---|
| Container running | Docker container in running state |
| systemd ready | systemctl is-system-running returns running or degraded |
| sshd listening | Port 22 accepting connections |
| munge ready | munge service active (not with identity clientIds, which masks munge) |
| slurmctld ready | scontrol ping reports this controller UP (controllers of managed clusters; each controller of a backup pair is checked for its own host); while it does not, a failed slurmctld unit ends the wait |
| slurmctld HTTP | GET /metrics/nodes on port 6817 answers 2xx on controller, GET /livez on controller-backup, whose standby slurmctld redirects the metrics (monitored clusters only, after slurmctld ready) |
| slurmd ready | slurmd service active (managed workers only) |
| sackd ready | sackd service active (the submitter of a managed cluster with identity clientIds) |
| slurmrestd ready | slurmrestd service active and port 6820 accepting connections (managed api node) |
| slurmdbd ready | slurmdbd service active (managed db nodes). mariadb is started before it with systemctl enable --now, which returns once the unit is active |
| cluster registered | sacctmgr show cluster on controller lists the cluster (accounts only, before the accounts are created) |
A unit that has failed does not recover on its own: a failed munge, slurmctld, slurmd, sackd, slurmrestd or slurmdbd unit fails sind create cluster and sind create worker at once, with the tail of the unit’s journal. So does a container that exits.
With a managed db node, mariadb and slurmdbd are started and slurmdbd must be ready before slurmctld and slurmd are enabled.
Docker CLI, not SDK
sind interacts with Docker by shelling out to the docker CLI rather than using the Docker SDK. This approach, proven by kind, provides:
- Simpler maintenance and fewer dependencies
- Wider compatibility across Docker versions
- No tight coupling to Docker daemon internals
The docker package wraps command execution in a thin abstraction layer with proper output handling and error reporting. It runs at most 16 docker commands at once, so a large cluster does not fork hundreds of docker processes while its nodes boot; every node still boots at once, and long-lived streams such as docker events do not count.
Dual use
sind is designed for use as both:
- CLI tool — standalone command-line interface
- Go library — embeddable package for wrapper tools and integrations
The CLI command structure is reflected in the library API, allowing programmatic access to all sind operations.