Skip to content
Controller Failover

Controller Failover

Show the steps for

sind can run Slurm’s active/passive controller setup: a primary slurmctld on controller and a backup slurmctld on controller-backup. This guide shows how to create the pair, check which controller is in control, and trigger a failover gracefully or by simulating an outage.

Create a cluster with a backup controller

kind: Cluster
name: dev
nodes:
  - role: controller
    backupController: true
  - role: worker
    count: 2
sind create cluster --config ha.yaml
  1. Press New cluster… and set the name to dev.
  2. Open More nodes below the nodes and tick backup.
  3. Press Create dev.
The New cluster page: the starts, the name and realm, the nodes by what they are for, and the plan with the nodes, what they take of this host and the command that sind ui will run The New cluster page: the starts, the name and realm, the nodes by what they are for, and the plan with the nodes, what they take of this host and the command that sind ui will run

sind then:

  • creates controller-backup next to controller, with the same image, resources, volumes, capabilities, devices and security options
  • writes both hosts to slurm.conf (SlurmctldHost=controller, then SlurmctldHost=controller-backup) and sets SlurmctldTimeout=20
  • mounts a shared state volume, <realm>-<cluster>-state, at /var/spool/slurmctld (StateSaveLocation) on both controllers
  • starts slurmctld on both; the backup waits in backup mode and takes over when the primary stops responding for SlurmctldTimeout seconds

sind create cluster waits until both controllers answer scontrol ping.

To change the takeover delay, set SlurmctldTimeout in the main section:

slurm:
  main: |
    SlurmctldTimeout=10

SlurmctldHost and StateSaveLocation are generated for the pair and cannot be set in slurm.main.

Check which controller is in control

sind get cluster adds an HA column for the pair. It shows each controller’s position, and * marks the one in control:

NODES
NAME                   ROLE        HA        IP          STATUS   SERVICES
controller.dev         controller  primary*  172.19.0.2  running  munge ✓ slurmctld ✓ sshd ✓
controller-backup.dev  controller  backup    172.19.0.3  running  munge ✓ slurmctld ✓ sshd ✓
worker-0.dev           worker      -         172.19.0.4  running  munge ✓ slurmd ✓ sshd ✓

With -o json, each controller’s health carries "ha": {"position": "primary", "in_control": true}. sind get node controller-backup.dev shows the same information for one node.

sind reads this from the heartbeat file the controller in control writes to StateSaveLocation. A controller counts as in control when it wrote the heartbeat within the last 60 seconds and its own slurmctld answers scontrol ping.

From inside the cluster, scontrol ping reports whether each slurmctld is up:

sind exec dev -- scontrol ping
Slurmctld(primary) at controller is UP
Slurmctld(backup) at controller-backup is UP

Graceful failover

Ask the backup to take over:

sind exec dev -- scontrol takeover

The backup takes control and the primary’s slurmctld stops. Running and pending jobs are kept because both controllers share the saved state. sind get cluster dev then shows backup* and slurmctld ✗ on controller.

Hand control back by starting the primary’s slurmctld again. On start, the primary tells the backup to return to backup mode and takes over:

sind ssh controller.dev -- systemctl start slurmctld

Failover by outage

Use power control to simulate the primary failing. The backup takes over once the primary has been unresponsive for SlurmctldTimeout seconds.

Scenario Command Recovery
Host crash sind power cut controller.dev sind power on controller.dev
Clean host shutdown sind power shutdown controller.dev sind power on controller.dev
Hung controller sind power freeze controller.dev sind power unfreeze controller.dev, then sind ssh controller.dev -- systemctl restart slurmctld
Daemon crash sind ssh controller.dev -- systemctl kill -s KILL slurmctld sind ssh controller.dev -- systemctl start slurmctld

When the primary’s container starts again, systemd starts its slurmctld, which reclaims control from the backup. A frozen primary resumes without knowing that the backup took over, so restart its slurmctld after unfreezing it to hand control back cleanly.

Watch the takeover and the logs of either controller:

watch sind get cluster dev
sind logs controller-backup.dev slurmctld --follow

While the backup is in control

  • sind enter and sind exec run on the controller in control when the cluster has no submitter.
  • sind create worker and sind delete worker update sind-nodes.conf and reconfigure Slurm through a running controller, so they keep working while the primary is down. With both controllers down, they refuse to add or delete managed workers.
  • sind ssh controller.dev and sind logs controller.dev still address the primary container by name.

To run a pair whose Slurm configuration you provision yourself, see Unmanaged Cluster.

Last updated on