Running MPI Jobs
The default sind-node image ships with a full MPI stack (OpenMPI, PMIx, PRRTE, UCX) and Slurm configured with MpiDefault=pmix. This guide walks through compiling and running a simple MPI program across multiple nodes.
Create a multi-node cluster
Create a cluster with 3 workers:
sind create cluster --config - <<'EOF'
kind: Cluster
nodes:
- controller
- worker: 3
EOFVerify the MPI stack
Check that OpenMPI is installed and configured with PMIx and UCX:
sind exec -- ompi_info | grep -E "Open MPI:|MCA pml|with-pmix"You should see OpenMPI’s version, the UCX point-to-point layer, and --with-pmix in the configure line.
Write an MPI program
Create a hello-world program that synchronizes all ranks with a barrier:
sind exec -- sh -c 'cat > hello_mpi.c << "CSRC"
#include <mpi.h>
#include <stdio.h>
#include <unistd.h>
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
int rank, size;
char hostname[256];
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
gethostname(hostname, sizeof(hostname));
MPI_Barrier(MPI_COMM_WORLD);
printf("rank %d of %d on %s\n", rank, size, hostname);
MPI_Finalize();
return 0;
}
CSRC'Compile
The image includes mpicc, which wraps gcc with the correct MPI flags:
sind exec -- mpicc -o hello_mpi hello_mpi.cRun across all nodes
Launch one task per worker node via srun:
sind exec -- srun -N3 --ntasks-per-node=1 ./hello_mpiExpected output (order may vary):
rank 0 of 3 on worker-0
rank 1 of 3 on worker-1
rank 2 of 3 on worker-2Since MpiDefault=pmix is set in the generated slurm.conf (unless the main section sets another MpiDefault), srun uses PMIx for process launch automatically — no --mpi=pmix flag needed.
Batch submission
The same program works with sbatch:
sind exec -- sh -c 'cat > mpi_job.sh << "SCRIPT"
#!/bin/sh
#SBATCH --job-name=mpi-hello
#SBATCH --nodes=3
#SBATCH --ntasks-per-node=1
srun hello_mpi
SCRIPT'
sind exec -- sbatch --wait mpi_job.sh
sind exec -- sh -c 'cat slurm-*.out'