Progress Display

sind create cluster, create worker, delete cluster (also --all), delete worker and the power commands show their progress on standard error while they run. The other commands (get, ssh, enter, exec, logs, doctor, version, mcp, completion) never do.

By default sind shows it only on a terminal. Where standard error is not a terminal, in a pipe, a file or a CI log, these commands print nothing on success, as they did before, and their standard error holds exactly what it did: errors, Warning: lines and the -v logs.

On a terminal, sind draws a live tree in the last rows: the command and how long it has run, each step under way with how many of its targets are done, and each target that runs, with how long it has run and what it is doing. The targets that are done fold into one row, such as ✓ worker-[0-7,11]; those that failed get a row for each error. A row the terminal has no room for is cut, as … 2 more running, and the tree takes at most six rows, or a third of a taller terminal’s rows.

When a step is done, the tree leaves one line for it above itself, such as ✓ nodes 6.3s 6 ok. When the command is done, the tree goes and only these lines stay:

• A command that succeeds leaves nothing else. Its exit status 0 says the rest.

• A command that fails, or that you interrupt, also leaves a summary line, as it does under --progress counter and plain, then the error line:

  sind: ✗ create cluster: failed in 38s: 5 ok, 1 failed, 2 canceled
  12:00:00.000 ERRO cluster dev not ready within 12s: ...
  

The tree draws nothing in the first second, so a command done by then leaves nothing on the terminal, as before. A step that ends at once leaves no line, and mesh and preflight, which sind usually finishes quickly, show only once they have run for a second, and leave a line only when they took that long or failed. Each docker command shows under the step or node it runs for once it has taken a second, such as worker-1 12.0s docker exec.

What the command writes goes above the tree, never into it: Warning: lines, the logs that -v asks for, and the summary. While another sind command holds the realm lock, the tree shows the wait below the warning, with the command it waits for when the warning names one:

Warning: waiting for another sind command in realm "sind" to finish: sind create cluster dev (pid 4242 on build-07, since 2026-10-04 10:02:03)
delete cluster · 0:03.1
  realm lock  3.1s  sind create cluster dev (pid 4242 on build-07, since 2026-10-04 10:02:03)

After Ctrl-C or SIGTERM, the command’s row says so while sind cleans up, with how many running targets will stop and how many queued ones will not start, as create cluster · interrupting · 2 running will stop · 0:41.3. The summary line then says canceled.

On a terminal, the tree, the counter and the summary line draw in colour: ok in green, failed in red, canceled in amber and running in blue. They use the terminal’s own 16 colours where it shows no more, and the marks alone when NO_COLOR is set. Where stderr is no terminal, or TERM is dumb, they write plain text, as a log wants.

The tree draws with ASCII marks (+, x, ~, >) unless LC_ALL, LC_CTYPE or LANG names a UTF-8 locale. On a terminal smaller than 40 columns or 8 rows it draws the counter’s single line instead. A sind started in the background (&, or Ctrl-Z and bg) draws nothing, so that the shell’s prompt stays as it is. sind create cluster --config - takes the tree off the terminal while it reads the configuration from standard input.

sind create cluster shows these steps:

mesh registration, slurm, home directories and monitoring run at the same time, and accounts after slurm. While a node waits, its row says which check it waits for and how much of the --wait limit it has used, 2.4s/5m ready munge. sind create worker shows pull images (for an --image the daemon does not have, or with --pull), nodes, mesh registration, slurm and rollback the same way, and on a monitored cluster monitoring before nodes.

A creation on a host that had neither the mesh nor the images, while the nodes boot:

$ sind create cluster --config dev.yaml
✓ mesh  12s  4 ok
✓ pull images  9.4s  2 ok
✓ preflight  10s
create cluster · 0:25.3
  nodes  0/6 · 6 running
    ▸ controller  2.4s/5m  ready munge
    ▸ db  2.4s/5m  ready sshd
    ▸ worker-0  2.1s  setup
    ▸ worker-1  2.4s/5m  ready systemd
    ▸ worker-2  2.4s/5m  ready systemd
    ▸ worker-3  2.4s/5m  ready sshd

and once it is done:

$ sind create cluster --config dev.yaml
✓ mesh  12s  4 ok
✓ pull images  9.4s  2 ok
✓ preflight  10s
✓ nodes  6.3s  6 ok
✓ mesh registration  0.4s
✓ slurm  6.0s  6 ok
✓ accounts  1.7s

The same creation with --wait 12s, where worker-2 does not become ready in time and the two nodes still booting are stopped. A failure row writes the node’s own name as {}, so that nodes that failed the same way share one row:

$ sind create cluster --config dev.yaml --wait 12s
✓ mesh  12s  4 ok
✓ pull images  9.4s  2 ok
✓ preflight  10s
12:00:31.605 ERRO cleaning up partial resources, please wait
✗ nodes  12s  3 ok, 1 failed, 2 canceled
  ✗ worker-2  timeout: waiting for {}: node sind-dev-{} not ready: context deadline exceeded; last probe error: probe munge: munge not ready: activating
✓ rollback  3.1s
sind: create cluster: failed in 38s: 5 ok, 1 failed, 2 canceled
12:00:34.839 ERRO cluster dev not ready within 12s: waiting for worker-2: node sind-dev-worker-2 not ready: context deadline exceeded; last probe error: probe munge: munge not ready: activating

The power commands show a step named by what they do, with a target for each node, named as you name it in a node argument (worker-0, or worker-0.dev in the cluster dev): stopping (shutdown), killing (cut), starting (on), pausing (freeze) or unpausing (unfreeze); reboot shows stopping and then starting, and cycle shows killing and then starting. A power reboot of 24 workers, at most 8 at a time, while it stops them:

$ sind power reboot worker-[0-23]
power reboot · 0:07.3
  stopping  9/24 · 8 running · 7 queued
    ▸ worker-8  4.2s  docker stop
    ▸ worker-9  4.1s  docker stop
    ▸ worker-10  3.4s  docker stop
    ▸ worker-12  1.2s  docker stop
    ▸ worker-13  1.2s  docker stop
    ▸ worker-14  1.2s  docker stop
    … 2 more running
    ✓ worker-[0-7,11]

and once it is done:

$ sind power reboot worker-[0-23]
✓ stopping  14s  24 ok
✓ starting  2.2s  24 ok

sind delete cluster --all shows the step clusters, with a target for each cluster. sind delete cluster and sind delete worker show no steps, only the docker commands that take a second or longer.

--progress chooses what sind shows, and SIND_PROGRESS does when the flag is not given:

auto draws nothing while standard output goes into a pipe, as in sind create cluster | tee out.log: whatever reads it writes to the same terminal.

--progress is a global flag: it may come before or after the subcommand, and every command checks it, also one that shows no progress. A mode it does not know, or tty or counter where standard error is not a terminal or TERM is dumb, is a usage error, exit status 2 (see Exit Status):

$ sind --progress tty create cluster 2>create.log
$ cat create.log
12:00:00.000 ERRO --progress asks for a live tree, but standard error is not a terminal; use none, or auto to draw one only where it can be

SIND_PROGRESS fails no command, since a shell profile or a CI job sets it for every command that follows: tty or counter where neither can be drawn shows nothing, and a value sind does not take shows nothing and says so in one line:

sind: SIND_PROGRESS is "plian"; it takes one of auto, tty, counter, plain, none; no progress is shown

--progress plain writes a line when a step starts and when it ends, when a wait with a limit starts, when a wait or a target fails, and every ten seconds for each step under way that counts its targets. Each line starts with the time since the command started and names what it tells of by its path from the command. Plain lines have no escape codes and need no terminal, so they suit a CI log or a file. mesh and preflight, which the tree shows only after a second, get no line, and neither do the docker commands or the lines docker writes. The same creation as above, trimmed to two nodes of each step:

$ sind create cluster --config dev.yaml --progress plain
[0:12] create cluster › pull images: start, 2 targets
[0:21] create cluster › pull images: done in 9.4s: 2 ok
[0:22] create cluster › nodes: start, 6 targets
[0:23] create cluster › nodes › worker-0 › ready: waiting 5m0s
[0:23] create cluster › nodes › controller › ready: waiting 5m0s
[0:29] create cluster › nodes: done in 6.3s: 6 ok
[0:29] create cluster › slurm: start, 6 targets
[0:29] create cluster › mesh registration: start
[0:29] create cluster › mesh registration: done in 0.4s
[0:31] create cluster › slurm › db › ready: waiting 4m52s
[0:33] create cluster › slurm › controller › ready: waiting 4m50s
[0:35] create cluster › slurm: done in 6.0s: 6 ok
[0:35] create cluster › accounts: start
[0:35] create cluster › accounts › ready: waiting 4m48s
[0:36] create cluster › accounts: done in 1.7s

The waits ready say how much of the --wait limit they have left.

--progress-log FILE, or SIND_PROGRESS_LOG when the flag is not given, appends the progress events of the command to FILE, one JSON object per line, in go-clikit’s event log format, version 1: what started and ended when, under which step or target, and how it ended and why, with each docker command by its subcommand alone and its exit code. It holds neither the command line nor the environment of the command, and of what the programs sind runs write, only what an error quotes; the realm lock wait names the command that holds the lock, its command line and host, as the Warning: line does. It works with every --progress mode, none included, and writes nothing on standard error.

• sind creates the file readable and writable by you alone (mode 0600) and appends to it, so the commands of a CI job can share one log; each command’s lines carry a run of their own.
• An existing file must be yours, have no other name (no hard link to it) and be readable or writable by nobody else: sind refuses one that is not, and for one that others can read or write says run chmod 600 FILE. A symbolic link on the way to it, and a named pipe, must be yours or root’s: sind follows no link put at the file’s name after it looked, and refuses another user’s named pipe at once instead of waiting for someone to read it. /dev/stderr, /dev/stdout and /dev/fd/N are written to where that stream goes; on the terminal the tree is drawn on, the lines go above the tree, as the -v logs do.
• A file named by --progress-log that sind cannot use is a usage error, before the command does anything. One named by SIND_PROGRESS_LOG is not: sind says why in one line and runs without a log. --progress-log "" writes no log, whatever the variable says.
• A log that cannot be written while the command runs, such as on a full disk, does not fail the command: sind says once that the log stops short.
• Only create, delete and power open the file; other commands leave it alone.

When sind is started with a W3C trace context in TRACEPARENT (and TRACESTATE), the log continues that trace. While sind shows progress or writes a log, it removes both variables from the environment of the programs it runs, such as docker; otherwise they inherit them.

Where standard error is not a terminal, nothing changes unless you ask: no tree, no summary, no event log. sind-action and scripts that read sind’s standard error see what they did before. --progress plain adds lines to a CI log, and --progress-log keeps the events as a job artifact; see CI/CD.

The MCP server never shows progress: its tools take neither flag, and it runs its tool calls with SIND_PROGRESS=none and without SIND_PROGRESS_LOG, whatever its own environment says.