Slurm Configuration
Generated files
sind generates a multi-file Slurm configuration and writes it to the <realm>-<cluster>-config volume. For an unmanaged cluster it writes nothing: the volume stays empty for your own configuration.
/etc/slurm/
├── slurm.conf # main config
├── sind-nodes.conf # sind-managed node definitions
├── cgroup.conf # cgroupv2 configuration
├── plugstack.conf # SPANK plugin config (always created)
├── plugstack.conf.d/ # SPANK plugin fragments (always created)
├── slurm.conf.d/ # main config fragments (if slurm.main is a map)
├── cgroup.conf.d/ # cgroup fragments (if slurm.cgroup is a map)
├── gres.conf # generic resources (if slurm.gres is set)
├── gres.conf.d/ # gres fragments (if slurm.gres is a map)
├── topology.conf # network topology (if slurm.topology is set)
├── topology.conf.d/ # topology fragments (if slurm.topology is a map)
├── slurmdbd.conf # accounting daemon config (if a managed db node exists)
├── slurmdbd.conf.d/ # slurmdbd fragments (if slurm.slurmdbd is a map)
├── slurm.key # auth/slurm key (identity clientIds only)
└── jwt_hs256.key # auth/jwt key (managed api node only)slurm.conf
sind generates this baseline (for sind create cluster dev):
# Generated by sind
ClusterName=dev
SlurmctldHost=controller
SlurmUser=slurm
StateSaveLocation=/var/spool/slurmctld
SlurmdSpoolDir=/var/spool/slurmd
PlugStackConfig=/etc/slurm/plugstack.conf
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup
MpiDefault=pmix
ReturnToService=2
DefMemPerCPU=512
include /etc/slurm/sind-nodes.confThe main section goes before the last line, the include of sind-nodes.conf: its content in string form, an include line per fragment in map form. Slurm applies NodeName=DEFAULT and PartitionName=DEFAULT lines only to the lines after them, so those in main reach sind’s nodes and partition, e.g. PartitionName=DEFAULT DefaultTime=00:30:00 gives jobs on partition all a default time limit. slurmctld reads every node before any partition, so a partition in main may name sind’s nodes, e.g. PartitionName=debug Nodes=worker-[0-1] MaxTime=30.
The first block is fixed: sind’s volumes, images and readiness checks depend on it. A backup controller adds SlurmctldHost=controller-backup to it, and SlurmctldTimeout=20 unless the main section sets it. Every other parameter is written unless the main section sets it, so a value there replaces sind’s instead of following it:
TaskPlugin=task/cgroupdoes not bind tasks to CPUs. A node’s CPUs are a CPU quota, not a set of host CPUs, so every worker sees all host CPUs, andtask/affinitywould bind the tasks of every worker, of every cluster on the host, to the same first host CPUs. To test CPU binding, setTaskPlugin=task/cgroup,task/affinityinmain; managed workers then get theSYS_NICEcapability (see Capabilities and devices).MpiDefault=pmixmakessrunlaunch through PMIx, which needs Slurm’smpi/pmixplugin, as the official images have. SetMpiDefault=noneinmainfor an image without it (see Custom image requirements).ReturnToService=2returns aDOWNworker to service when its slurmd registers with a valid configuration, e.g. aftersind power cutandsind power on.DefMemPerCPUis the smallest memory per CPU of the managed workers (memoryin MiB divided bycpus), so that every CPU of every worker can run a job that does not ask for memory. Slurm’s defaults,SelectType=select/cons_treswithSelectTypeParameters=CR_Core_Memory, make memory a consumable resource, and without a default such a job would get all of a node’s memory, one job per worker at a time. It is left out whenmainsetsDefMemPerCPUorDefMemPerNode. sind writesslurm.confonce: a worker added later with less memory per CPU runs fewer such jobs at a time than it has CPUs.- Everything else is Slurm’s default, e.g.
SlurmdTimeout=300(see Node failure detection).
With a managed db node it also contains these accounting parameters, each unless the main section sets it:
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=db
JobAcctGatherType=jobacct_gather/cgroupWith identity nssSlurm or clientIds it contains the identity parameters, clientIds only for the first three, each unless the main section sets it, in which case its value must list sind’s (validation checks it):
AuthType=auth/slurm
CredType=cred/slurm
AuthInfo=use_client_ids
LaunchParameters=enable_nss_slurmWith a managed api node it contains the JWT parameters for slurmrestd, each unless the main section sets it, with ,use_jwt_client_ids after the key under identity clientIds. An AuthAltTypes set there must list auth/jwt (validation checks it); an AuthAltParameters set there replaces sind’s, e.g. with a JWKS file:
AuthAltTypes=auth/jwt
AuthAltParameters=jwt_key=/etc/slurm/jwt_hs256.keyOn a monitored cluster it also contains this parameter, unless the main section sets it, in which case it must be metrics/openmetrics (validation checks it):
MetricsType=metrics/openmetricssind does not modify slurm.conf after initial creation.
slurmdbd.conf
Only generated for a cluster with a managed db node. It points slurmdbd at the local MariaDB (StorageHost=localhost, StorageLoc=slurm_acct_db, StorageUser=slurm, which MariaDB authenticates by unix_socket as the OS user slurm slurmdbd runs as, or by the StoragePass the slurmdbd section sets), keeps the pid file in /run/slurmdbd and the log in /var/log/slurm/slurmdbd.log, and is owned by slurm with mode 0600, as slurmdbd requires. It authenticates with AuthType=auth/munge, or with identity clientIds with AuthType=auth/slurm and AuthInfo=use_client_ids, and with a managed api node also with the JWT parameters of slurm.conf, each unless the slurmdbd section sets it (slurmdbd has no default key path, so an AuthAltParameters there needs a jwt_key= or jwks=). The slurmdbd section extends it like the other sections. The fragments of its map form, in slurmdbd.conf.d/, get the same protection, as they can hold secrets such as a StoragePass: owned by slurm with mode 0600, in a directory only slurm can read.
slurm.key
Only generated with identity clientIds: 1024 random bytes, owned by slurm with mode 0600, the key of auth/slurm and cred/slurm. Every node reads it from the config volume; there is no munge key. sind get auth-key prints it.
jwt_hs256.key
Only generated with a managed api node: 32 random bytes, owned by slurm with mode 0600, the HS256 key with which slurmctld and slurmdbd check the tokens of REST API requests (auth/jwt). slurmrestd itself does not read it. sind get auth-key --type jwt prints it. Whoever has the key can sign a token for any user, root included.
sind-nodes.conf
This file contains node and partition definitions for sind-managed nodes, e.g. for three workers with cpus: 2 and memory: 1g:
# Generated by sind
NodeName=worker-0 CPUs=2 RealMemory=1024 State=UNKNOWN
NodeName=worker-1 CPUs=2 RealMemory=1024 State=UNKNOWN
NodeName=worker-2 CPUs=2 RealMemory=1024 State=UNKNOWN
PartitionName=all Nodes=worker-0,worker-1,worker-2 Default=YESA worker’s Slurm CPUs and RealMemory are its container’s cpus and memory (in MiB). Nodes with managed: false are left out.
Partition all holds every managed worker and is the default partition. It sets nothing else, so Slurm’s defaults apply: no time limit (MaxTime=UNLIMITED, no DefaultTime) and State=UP. Change them with a PartitionName=DEFAULT line in the main section, or add partitions of your own there. If main declares a default partition of its own (Default=YES on a partition other than DEFAULT), sind writes Default=NO on all, so that yours stays the default. sind create worker checks slurm.conf and the files it includes the same way when it writes the partition line again: after sind delete worker removed it with the last managed worker, or on a cluster created without managed workers.
sind create clustergenerates the filesind create workeradds new managed nodes, replacing a definition of the same name, and removes them again when it failssind delete workerremoves managed nodes
Both worker commands rewrite only the Nodes= list of the partition line, so other edits to the file survive, but prefer NodeName=DEFAULT and PartitionName=DEFAULT lines in main. They replace the file in one step, through a temporary file in /etc/slurm, so that a command interrupted by Ctrl-C or its --wait limit never leaves it cut short, and a failed sind create worker writes back the file it found. To add node definitions of your own, put them in main or in a separate file included from slurm.conf.
cgroup.conf
sind generates a minimal cgroup.conf:
# Generated by sind
CgroupPlugin=autodetectWith sind’s proctrack/cgroup and task/cgroup, and jobacct_gather/cgroup with a db node, Slurm puts every job and step into a cgroup, which tracks its processes and measures its usage, but it constrains nothing. Constraints are opt-in through the cgroup section, and both common ones have a catch in sind:
ConstrainRAMSpace=yeslimits each job to the memory it was allocated:--mem, orDefMemPerCPUper CPU. A worker’sRealMemoryis its whole container memory limit, with no reserve for slurmd, the other daemons and/tmp, so jobs that together use all of it can make the container’s OOM killer hit the daemons. Leave room with a smallerDefMemPerCPUinmain.ConstrainCores=yesconfines each job to a cpuset of its allocated CPUs. sind limits a node’s CPUs with a CPU quota, not a cpuset, so every worker sees all host CPUs, and Slurm maps a worker’s CPU N to host CPU N: the jobs of every worker, of every cluster on the host, run on the same low-numbered host CPUs. Use it to test the setting itself, not for throughput.task/affinitybinds the same way.
plugstack.conf
Always created with an include /etc/slurm/plugstack.conf.d/* directive and an empty .conf.d/ directory. This allows SPANK plugins to be dropped in as fragment files without additional configuration.
Extending configuration via slurm sections
The slurm key in the cluster config allows extending any Slurm config file declaratively at creation time. Each section supports two forms:
String form — content appended to the config file (for slurm.conf, before the include of sind-nodes.conf):
slurm:
main: |
SelectType=select/cons_tres
cgroup: |
ConstrainRAMSpace=yesMap form — named fragments in a .conf.d/ directory, included explicitly:
slurm:
main:
scheduling: |
SchedulerType=sched/backfill
resources: |
SelectType=select/cons_tresThis creates slurm.conf.d/scheduling.conf and slurm.conf.d/resources.conf, with an explicit include directive for each fragment file in slurm.conf, before the include of sind-nodes.conf. Glob includes are not used because Slurm’s main config parser does not support them (the SPANK plugstack.conf parser does).
Standalone sections (gres, topology) are only created when configured and require enabling in slurm.conf via the main section (e.g., GresTypes=gpu).
See Cluster Configuration for the full schema.
Version discovery
While it creates the cluster network and volumes, sind discovers the Slurm version of the controller’s image by running an ephemeral container:
docker run --rm <controller image> slurmctld -V
# Output: "slurm 26.05.6"The version is stored as the sind.slurm.version label on each node container and shown in the SLURM column of sind get clusters and sind get cluster. It does not change the generated configuration. Workers added with sind create worker copy the controller’s label. Without --image they run an image the cluster already runs; with --image, sind discovers that image’s version and refuses managed workers whose version differs from the cluster’s. At cluster creation, nodes with a different image carry the controller’s version too: sind create cluster does not compare versions across images.
sind skips the discovery for unmanaged clusters, where the Slurm you provision may differ from the one in the image. Their SLURM column shows -.
When it creates a monitored cluster, sind runs no slurmctld -V: the container that checks the controller’s image for the endpoints that the monitoring stack uses reports its version too.
Post-creation customization
sind provides a working starter configuration. For additional customization after creation, users can:
- Edit config files on the controller (the config volume is writable)
- Add include files for custom configuration
- Replace the entire configuration (but
sind create workerwill fail for managed nodes withoutsind-nodes.conf)