Skip to content
Slurm Configuration

Slurm Configuration

Generated files

sind generates a multi-file Slurm configuration and writes it to the <realm>-<cluster>-config volume. For an unmanaged cluster it writes nothing: the volume stays empty for your own configuration.

/etc/slurm/
├── slurm.conf              # main config
├── sind-nodes.conf         # sind-managed node definitions
├── cgroup.conf             # cgroupv2 configuration
├── plugstack.conf          # SPANK plugin config (always created)
├── plugstack.conf.d/       # SPANK plugin fragments (always created)
├── slurm.conf.d/           # main config fragments (if slurm.main is a map)
├── cgroup.conf.d/          # cgroup fragments (if slurm.cgroup is a map)
├── gres.conf               # generic resources (if slurm.gres is set)
├── gres.conf.d/            # gres fragments (if slurm.gres is a map)
├── topology.conf           # network topology (if slurm.topology is set)
├── topology.conf.d/        # topology fragments (if slurm.topology is a map)
├── slurmdbd.conf           # accounting daemon config (if a managed db node exists)
├── slurmdbd.conf.d/        # slurmdbd fragments (if slurm.slurmdbd is a map)
├── slurm.key               # auth/slurm key (identity clientIds only)
└── jwt_hs256.key           # auth/jwt key (managed api node only)

slurm.conf

sind generates this baseline (for sind create cluster dev):

# Generated by sind
ClusterName=dev
SlurmctldHost=controller
SlurmUser=slurm
StateSaveLocation=/var/spool/slurmctld
SlurmdSpoolDir=/var/spool/slurmd
PlugStackConfig=/etc/slurm/plugstack.conf

ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup
MpiDefault=pmix
ReturnToService=2
DefMemPerCPU=512

include /etc/slurm/sind-nodes.conf

The main section goes before the last line, the include of sind-nodes.conf: its content in string form, an include line per fragment in map form. Slurm applies NodeName=DEFAULT and PartitionName=DEFAULT lines only to the lines after them, so those in main reach sind’s nodes and partition, e.g. PartitionName=DEFAULT DefaultTime=00:30:00 gives jobs on partition all a default time limit. slurmctld reads every node before any partition, so a partition in main may name sind’s nodes, e.g. PartitionName=debug Nodes=worker-[0-1] MaxTime=30.

The first block is fixed: sind’s volumes, images and readiness checks depend on it. A backup controller adds SlurmctldHost=controller-backup to it, and SlurmctldTimeout=20 unless the main section sets it. Every other parameter is written unless the main section sets it, so a value there replaces sind’s instead of following it:

  • TaskPlugin=task/cgroup does not bind tasks to CPUs. A node’s CPUs are a CPU quota, not a set of host CPUs, so every worker sees all host CPUs, and task/affinity would bind the tasks of every worker, of every cluster on the host, to the same first host CPUs. To test CPU binding, set TaskPlugin=task/cgroup,task/affinity in main; managed workers then get the SYS_NICE capability (see Capabilities and devices).
  • MpiDefault=pmix makes srun launch through PMIx, which needs Slurm’s mpi/pmix plugin, as the official images have. Set MpiDefault=none in main for an image without it (see Custom image requirements).
  • ReturnToService=2 returns a DOWN worker to service when its slurmd registers with a valid configuration, e.g. after sind power cut and sind power on.
  • DefMemPerCPU is the smallest memory per CPU of the managed workers (memory in MiB divided by cpus), so that every CPU of every worker can run a job that does not ask for memory. Slurm’s defaults, SelectType=select/cons_tres with SelectTypeParameters=CR_Core_Memory, make memory a consumable resource, and without a default such a job would get all of a node’s memory, one job per worker at a time. It is left out when main sets DefMemPerCPU or DefMemPerNode. sind writes slurm.conf once: a worker added later with less memory per CPU runs fewer such jobs at a time than it has CPUs.
  • Everything else is Slurm’s default, e.g. SlurmdTimeout=300 (see Node failure detection).

With a managed db node it also contains these accounting parameters, each unless the main section sets it:

AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=db
JobAcctGatherType=jobacct_gather/cgroup

With identity nssSlurm or clientIds it contains the identity parameters, clientIds only for the first three, each unless the main section sets it, in which case its value must list sind’s (validation checks it):

AuthType=auth/slurm
CredType=cred/slurm
AuthInfo=use_client_ids
LaunchParameters=enable_nss_slurm

With a managed api node it contains the JWT parameters for slurmrestd, each unless the main section sets it, with ,use_jwt_client_ids after the key under identity clientIds. An AuthAltTypes set there must list auth/jwt (validation checks it); an AuthAltParameters set there replaces sind’s, e.g. with a JWKS file:

AuthAltTypes=auth/jwt
AuthAltParameters=jwt_key=/etc/slurm/jwt_hs256.key

On a monitored cluster it also contains this parameter, unless the main section sets it, in which case it must be metrics/openmetrics (validation checks it):

MetricsType=metrics/openmetrics

sind does not modify slurm.conf after initial creation.

slurmdbd.conf

Only generated for a cluster with a managed db node. It points slurmdbd at the local MariaDB (StorageHost=localhost, StorageLoc=slurm_acct_db, StorageUser=slurm, which MariaDB authenticates by unix_socket as the OS user slurm slurmdbd runs as, or by the StoragePass the slurmdbd section sets), keeps the pid file in /run/slurmdbd and the log in /var/log/slurm/slurmdbd.log, and is owned by slurm with mode 0600, as slurmdbd requires. It authenticates with AuthType=auth/munge, or with identity clientIds with AuthType=auth/slurm and AuthInfo=use_client_ids, and with a managed api node also with the JWT parameters of slurm.conf, each unless the slurmdbd section sets it (slurmdbd has no default key path, so an AuthAltParameters there needs a jwt_key= or jwks=). The slurmdbd section extends it like the other sections. The fragments of its map form, in slurmdbd.conf.d/, get the same protection, as they can hold secrets such as a StoragePass: owned by slurm with mode 0600, in a directory only slurm can read.

slurm.key

Only generated with identity clientIds: 1024 random bytes, owned by slurm with mode 0600, the key of auth/slurm and cred/slurm. Every node reads it from the config volume; there is no munge key. sind get auth-key prints it.

jwt_hs256.key

Only generated with a managed api node: 32 random bytes, owned by slurm with mode 0600, the HS256 key with which slurmctld and slurmdbd check the tokens of REST API requests (auth/jwt). slurmrestd itself does not read it. sind get auth-key --type jwt prints it. Whoever has the key can sign a token for any user, root included.

sind-nodes.conf

This file contains node and partition definitions for sind-managed nodes, e.g. for three workers with cpus: 2 and memory: 1g:

# Generated by sind
NodeName=worker-0 CPUs=2 RealMemory=1024 State=UNKNOWN
NodeName=worker-1 CPUs=2 RealMemory=1024 State=UNKNOWN
NodeName=worker-2 CPUs=2 RealMemory=1024 State=UNKNOWN
PartitionName=all Nodes=worker-0,worker-1,worker-2 Default=YES

A worker’s Slurm CPUs and RealMemory are its container’s cpus and memory (in MiB). Nodes with managed: false are left out.

Partition all holds every managed worker and is the default partition. It sets nothing else, so Slurm’s defaults apply: no time limit (MaxTime=UNLIMITED, no DefaultTime) and State=UP. Change them with a PartitionName=DEFAULT line in the main section, or add partitions of your own there. If main declares a default partition of its own (Default=YES on a partition other than DEFAULT), sind writes Default=NO on all, so that yours stays the default. sind create worker checks slurm.conf and the files it includes the same way when it writes the partition line again: after sind delete worker removed it with the last managed worker, or on a cluster created without managed workers.

  • sind create cluster generates the file
  • sind create worker adds new managed nodes, replacing a definition of the same name, and removes them again when it fails
  • sind delete worker removes managed nodes

Both worker commands rewrite only the Nodes= list of the partition line, so other edits to the file survive, but prefer NodeName=DEFAULT and PartitionName=DEFAULT lines in main. They replace the file in one step, through a temporary file in /etc/slurm, so that a command interrupted by Ctrl-C or its --wait limit never leaves it cut short, and a failed sind create worker writes back the file it found. To add node definitions of your own, put them in main or in a separate file included from slurm.conf.

cgroup.conf

sind generates a minimal cgroup.conf:

# Generated by sind
CgroupPlugin=autodetect

With sind’s proctrack/cgroup and task/cgroup, and jobacct_gather/cgroup with a db node, Slurm puts every job and step into a cgroup, which tracks its processes and measures its usage, but it constrains nothing. Constraints are opt-in through the cgroup section, and both common ones have a catch in sind:

  • ConstrainRAMSpace=yes limits each job to the memory it was allocated: --mem, or DefMemPerCPU per CPU. A worker’s RealMemory is its whole container memory limit, with no reserve for slurmd, the other daemons and /tmp, so jobs that together use all of it can make the container’s OOM killer hit the daemons. Leave room with a smaller DefMemPerCPU in main.
  • ConstrainCores=yes confines each job to a cpuset of its allocated CPUs. sind limits a node’s CPUs with a CPU quota, not a cpuset, so every worker sees all host CPUs, and Slurm maps a worker’s CPU N to host CPU N: the jobs of every worker, of every cluster on the host, run on the same low-numbered host CPUs. Use it to test the setting itself, not for throughput. task/affinity binds the same way.

plugstack.conf

Always created with an include /etc/slurm/plugstack.conf.d/* directive and an empty .conf.d/ directory. This allows SPANK plugins to be dropped in as fragment files without additional configuration.

Extending configuration via slurm sections

The slurm key in the cluster config allows extending any Slurm config file declaratively at creation time. Each section supports two forms:

String form — content appended to the config file (for slurm.conf, before the include of sind-nodes.conf):

slurm:
  main: |
    SelectType=select/cons_tres
  cgroup: |
    ConstrainRAMSpace=yes

Map form — named fragments in a .conf.d/ directory, included explicitly:

slurm:
  main:
    scheduling: |
      SchedulerType=sched/backfill
    resources: |
      SelectType=select/cons_tres

This creates slurm.conf.d/scheduling.conf and slurm.conf.d/resources.conf, with an explicit include directive for each fragment file in slurm.conf, before the include of sind-nodes.conf. Glob includes are not used because Slurm’s main config parser does not support them (the SPANK plugstack.conf parser does).

Standalone sections (gres, topology) are only created when configured and require enabling in slurm.conf via the main section (e.g., GresTypes=gpu).

See Cluster Configuration for the full schema.

Version discovery

While it creates the cluster network and volumes, sind discovers the Slurm version of the controller’s image by running an ephemeral container:

docker run --rm <controller image> slurmctld -V
# Output: "slurm 26.05.6"

The version is stored as the sind.slurm.version label on each node container and shown in the SLURM column of sind get clusters and sind get cluster. It does not change the generated configuration. Workers added with sind create worker copy the controller’s label. Without --image they run an image the cluster already runs; with --image, sind discovers that image’s version and refuses managed workers whose version differs from the cluster’s. At cluster creation, nodes with a different image carry the controller’s version too: sind create cluster does not compare versions across images.

sind skips the discovery for unmanaged clusters, where the Slurm you provision may differ from the one in the image. Their SLURM column shows -.

When it creates a monitored cluster, sind runs no slurmctld -V: the container that checks the controller’s image for the endpoints that the monitoring stack uses reports its version too.

Post-creation customization

sind provides a working starter configuration. For additional customization after creation, users can:

  • Edit config files on the controller (the config volume is writable)
  • Add include files for custom configuration
  • Replace the entire configuration (but sind create worker will fail for managed nodes without sind-nodes.conf)
Last updated on