Unmanaged Cluster
An unmanaged cluster gives you sind’s nodes (systemd containers with networking, DNS, SSH access and a shared munge key) and leaves Slurm to you. Use it to test Chef or Ansible code that provisions Slurm end to end, or to debug a configuration by running slurmctld -Dvvvvvv by hand.
Create an unmanaged cluster
Set managed: false on the controller:
kind: Cluster
name: dev
nodes:
- role: controller
managed: false
- role: submitter
- role: worker
count: 2sind create cluster --config unmanaged.yaml- Press New cluster…, set the name to
devand tick submitter. - Press defaults next to the controller. Open More: managed, capabilities, devices and tick Unmanaged.
- Press Create dev.
The flag applies to the whole cluster:
- Every node is unmanaged. A worker, db or api node with
managed: trueis rejected. slurmsections (main,cgroup,gres,topology,plugstack,slurmdbd) are rejected, because sind writes no Slurm configuration.- Identity modes other than
local(nssSlurm,clientIds) are rejected for the same reason, and so are Slurm accounts and the users’accounts,coordinatorandadminLevel, since sind runs no slurmdbd. Monitoring is rejected too, from the config key and fromsind create cluster --monitoringalike, as it needs theslurm.confsind writes:monitoring requires a managed controller: sind writes no Slurm configuration for an unmanaged cluster. Users and groups still work: every node gets the Linux accounts and, withusers, mounts the shared home volume. - A db node is a bare node too: sind starts neither MariaDB nor slurmdbd on it, so your tooling can provision accounting as well. To keep sind’s Slurm and provision only slurmdbd, use a managed cluster with
managed: falseon the db node instead. - An api node is a bare node too: sind starts no slurmrestd and sets up no JWT.
backupController: truestill addscontroller-backupand the shared state volume (see Controller pair).
What sind sets up
| Managed cluster | Unmanaged cluster | |
|---|---|---|
| Containers, networks, DNS, SSH access | ✓ | ✓ |
| munge key, munge running | ✓ | ✓ |
/etc/slurm |
generated configuration | empty |
| slurmctld and slurmd | running | not started |
| MariaDB and slurmdbd (db node) | running | not started |
| slurmrestd and its JWT key (api node) | running | not started |
Slurm version (sind.slurm.version label) |
discovered from the image | empty |
/etc/slurm is one volume shared by all nodes, like an NFS share: writable on the controllers, read-only on the other nodes. /etc/munge holds sind’s munge key and is read-only on every node. sind create cluster returns once each node runs systemd, sshd and munge.
The stock sind-node image has Slurm installed but not enabled. sind neither runs nor queries Slurm on an unmanaged cluster, so a custom image may also leave Slurm out for your provisioning to install (see Building Images).
Provision Slurm
Write the configuration to /etc/slurm on a controller, then start the daemons. By hand, a minimal setup for the cluster above looks like this:
sind ssh controller.dev -- 'cat > /etc/slurm/slurm.conf' <<'EOF'
ClusterName=dev
SlurmctldHost=controller
SlurmUser=slurm
StateSaveLocation=/var/spool/slurmctld
SlurmdSpoolDir=/var/spool/slurmd
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup
ReturnToService=2
NodeName=worker-[0-1] CPUs=1 State=UNKNOWN
PartitionName=all Nodes=ALL Default=YES MaxTime=INFINITE State=UP
EOF
sind ssh controller.dev -- 'echo CgroupPlugin=autodetect > /etc/slurm/cgroup.conf'
sind ssh controller.dev -- systemctl start slurmctld
sind ssh worker-0.dev -- systemctl start slurmd
sind ssh worker-1.dev -- systemctl start slurmd
sind exec dev -- srun -N2 hostnameChef, Ansible and similar tools reach the nodes over SSH as root. Include sind’s exported SSH configuration (see User SSH client integration) and use the node names as hosts, for example in an Ansible inventory:
[slurm_controller]
controller.dev.sind.sind
[slurm_submitter]
submitter.dev.sind.sind
[slurm_workers]
worker-0.dev.sind.sind
worker-1.dev.sind.sindKeep in mind:
- Write Slurm files on a controller. On the other nodes
/etc/slurmis read-only, and every node reads the same files. slurmdbd insists on aslurmdbd.confowned by itsSlurmUserwith mode0600. - munge is sind’s.
/etc/mungeis read-only, so recipes that manage the munge key must leave it as it is.sind get auth-key devprints the key base64-encoded if your tooling needs it. sind get cluster devreports munge and sshd only. Check Slurm itself withsind exec dev -- sinfoandsind logs controller.dev slurmctld.
Run slurmctld by hand
To debug a configuration, run slurmctld in the foreground with full verbosity instead of starting the service:
sind ssh -t controller.dev -- slurmctld -DvvvvvvStop it with Ctrl-C, edit the files under /etc/slurm on a controller, and run it again. slurmd -Dvvvvvv works the same way on a worker.
Controller pair
With backupController: true, sind creates controller-backup next to controller and mounts the shared state volume <realm>-<cluster>-state at /var/spool/slurmctld on both. Use it as StateSaveLocation, and list the controllers in this order so that your pair matches sind’s names:
SlurmctldHost=controller
SlurmctldHost=controller-backup
StateSaveLocation=/var/spool/slurmctldsind shows no HA column for an unmanaged pair, because it cannot tell which controller is in control. Ask Slurm instead with sind exec dev -- scontrol ping. Without a submitter, sind enter and sind exec target controller if it runs, otherwise controller-backup. Controller Failover shows how to trigger failovers; the power commands work the same way on an unmanaged pair.
Workers
sind create worker dev adds unmanaged workers, with or without --unmanaged: sind does not register them with Slurm or start slurmd. Add them to your configuration and start slurmd yourself. sind delete worker removes the containers and never edits /etc/slurm.