Skip to content
Running MPI Jobs

Running MPI Jobs

The web UI has no terminal to run jobs in. The steps on this page are for the CLI.

The default sind-node image ships with a full MPI stack (OpenMPI, PMIx, PRRTE, UCX) and Slurm configured with MpiDefault=pmix. This guide walks through compiling and running a simple MPI program across multiple nodes.

Create a multi-node cluster

Create a cluster with 3 workers:

sind create cluster --config - <<'EOF'
kind: Cluster
nodes:
  - controller
  - worker: 3
EOF

Verify the MPI stack

Check that OpenMPI is installed and configured with PMIx and UCX:

sind exec -- ompi_info | grep -E "Open MPI:|MCA pml|with-pmix"

You should see OpenMPI’s version, the UCX point-to-point layer, and --with-pmix in the configure line.

Write an MPI program

Create a hello-world program that synchronizes all ranks with a barrier:

sind exec -- sh -c 'cat > hello_mpi.c << "CSRC"
#include <mpi.h>
#include <stdio.h>
#include <unistd.h>

int main(int argc, char **argv) {
    MPI_Init(&argc, &argv);

    int rank, size;
    char hostname[256];
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    gethostname(hostname, sizeof(hostname));

    MPI_Barrier(MPI_COMM_WORLD);
    printf("rank %d of %d on %s\n", rank, size, hostname);

    MPI_Finalize();
    return 0;
}
CSRC'

Compile

The image includes mpicc, which wraps gcc with the correct MPI flags:

sind exec -- mpicc -o hello_mpi hello_mpi.c

Run across all nodes

Launch one task per worker node via srun:

sind exec -- srun -N3 --ntasks-per-node=1 ./hello_mpi

Expected output (order may vary):

rank 0 of 3 on worker-0
rank 1 of 3 on worker-1
rank 2 of 3 on worker-2

Since MpiDefault=pmix is set in the generated slurm.conf (unless the main section sets another MpiDefault), srun uses PMIx for process launch automatically — no --mpi=pmix flag needed.

Batch submission

The same program works with sbatch:

sind exec -- sh -c 'cat > mpi_job.sh << "SCRIPT"
#!/bin/sh
#SBATCH --job-name=mpi-hello
#SBATCH --nodes=3
#SBATCH --ntasks-per-node=1
srun hello_mpi
SCRIPT'

sind exec -- sbatch --wait mpi_job.sh
sind exec -- sh -c 'cat slurm-*.out'
Last updated on