CI/CD
GitHub Action
The sind-action GitHub Action installs sind and creates clusters in your workflow. sind runs on standard GitHub-hosted Ubuntu runners, x64 (ubuntu-latest) and ARM64 (ubuntu-26.04-arm), with the runner’s Docker daemon — no privileged containers or custom runner images required.
sind itself needs no sudo, but the runner must allow enough inotify instances. The Ubuntu 26.04 images, which ubuntu-latest and ubuntu-26.04-arm run, allow 128 per user. About 10 nodes on the runner, in one cluster or in several, use them up, and the next node containers exit with code 255 (see Container exits with code 255). sind-action v2.3.0 and later (the v2 tag) raise fs.inotify.max_user_instances to 8192 with the runner’s passwordless sudo before they create the clusters. On a runner without passwordless sudo, the action only warns, so raise the limit on that host yourself. Without sind-action, or with an older release of it, run sudo sysctl -w fs.inotify.max_user_instances=8192 first.
See the sind-action documentation for inputs, outputs, cluster definitions, and examples including parallel job isolation via realms.
Other CI systems
sind works in any CI environment that provides a rootful Docker daemon. Install the binary from a GitHub release, check its build provenance before running it, and run sind doctor to verify prerequisites:
curl -fsSL -o sind-linux-amd64 \
"https://github.com/GSI-HPC/sind/releases/latest/download/sind-linux-amd64"
gh attestation verify sind-linux-amd64 --repo GSI-HPC/sind \
--signer-workflow GSI-HPC/sind/.github/workflows/release.yml
install -m 755 sind-linux-amd64 ./sind
# 10 or more nodes on this host: sudo sysctl -w fs.inotify.max_user_instances=8192
./sind doctor
./sind create cluster --config cluster.ymlOn 64-bit ARM, download sind-linux-arm64 instead. gh attestation verify needs the GitHub CLI 2.102.0 or later and a token in GH_TOKEN; see Installation for the checksum alternative.
sind requires Docker Engine 28.0+, running rootful and without userns-remap, and a Linux host with cgroupv2 and nsdelegate. Most modern CI runners meet these requirements out of the box. Rootless Docker and userns-remap do not work: Docker refuses the writable cgroups of sind’s nodes there, which sind doctor and sind create cluster report before anything is pulled. A host without nsdelegate fails sind doctor, and sind create cluster before it creates a node.
10 or more nodes on the Docker host, counted over all its clusters and jobs, also need fs.inotify.max_user_instances of at least 1024 (see Container exits with code 255). The kernel’s default is 128. Raising the limit takes root on the Docker host: run sudo sysctl -w fs.inotify.max_user_instances=8192 before sind create cluster, or set it in /etc/sysctl.d/ on a self-hosted runner. An unprivileged job container cannot change it. sind doctor only warns below 1024 and exits 0. A job on the Docker host can fail there instead with ./sind doctor -o json | jq -e '.[] | select(.name == "inotify") | .status == "ok"'.
sind runs on the Docker host. A job whose DOCKER_HOST points at a daemon elsewhere, such as a Docker-in-Docker service, gets a Docker host warning from sind doctor, which does not fail it; --data paths then name directories on that daemon’s host.
Jobs that share a Docker daemon, such as jobs on one self-hosted runner or jobs that mount the host’s Docker socket, should each use their own realm (SIND_REALM). sind’s realm lock keeps jobs in one realm from breaking each other’s mesh, but a job’s sind delete cluster --all would delete the other jobs’ clusters too. A sind command killed while it held the lock, as by a job timeout, leaves the lock network <realm>-lock behind; a later sind command of the job, such as the cleanup step’s, finds that the killed command no longer runs and removes it, as long as it runs on the same runner and in the same job container (see Realms). Each realm’s mesh and each cluster take one network from the daemon’s default address pools, which a stock daemon fills at about 30 networks; a runner that hosts many such jobs at once needs smaller pools (see Limits).
Progress in CI logs
A CI log is not a terminal, so sind shows no progress there unless asked: create, delete and power print nothing on success, and their standard error holds what it did before sind had a progress display, so sind-action and scripts that read it see no change. Two settings add to the log (sind also appends every command’s events to its activity log in ~/.local/state/sind/, where a runner that is thrown away after the job loses it; SIND_PROGRESS_LOG= turns that off):
--progress plain, orSIND_PROGRESS=plainfor every command of the job, writes a line as each step starts and ends, as a wait with a limit starts and as a wait or a target fails, each with the time since the command started (see Plain lines).--progress-log FILE, orSIND_PROGRESS_LOG, appends every progress event to FILE as a line of JSON, readable by the job’s user alone. The commands of a job can share one file, which you can keep as an artifact of a failed run.
export SIND_PROGRESS=plain
export SIND_PROGRESS_LOG="$PWD/sind-events.jsonl"
./sind create cluster --config cluster.ymlSIND_PROGRESS=tty or counter fails no command where there is no terminal; it shows nothing. --progress tty or counter on the command line is a usage error there, exit status 2.
Data directory
By default every node mounts the job’s working directory, usually the checked-out workspace, read-write at /data (see Data mount). Files that root writes there from a node belong to root on the runner, so a later step that cleans the workspace may need sudo. --data volume, or storage.dataStorage.type: volume in the config, keeps the workspace out of the cluster.
The directory is resolved on the Docker host. When the CI job runs in a container that shares the host’s Docker socket, the job’s paths do not exist there and sind create cluster fails; use --data volume in that case.
Monitoring
A cluster with monitoring: true collects Slurm’s metrics, the nodes’ resource use and their journals into Prometheus and Loki, which a later step of the job can query by the addresses sind get monitoring -o json prints, for example to save the Slurm daemons’ errors of a failed run as an artifact (see In CI). The stack publishes no host port, so parallel jobs in their own realms each get one without colliding. Its four images come from Docker Hub and are pulled with the node images when a job creates the stack.
Untrusted changes
A cluster config is as trusted as a script with the runner’s Docker access (see Trust): do not create clusters from configs that untrusted pull requests can change in a pull_request_target or other privileged workflow.
Node failure tests
Tests that take a worker away with sind power freeze, cut or shutdown wait for slurmctld to mark it DOWN, which takes about five minutes with Slurm’s default SlurmdTimeout=300. Lower it in the cluster config to keep such a job short (see Node failure detection):
slurm:
main: |
SlurmdTimeout=30