Architecture Guide
Package map
cmd/sind/ CLI commands (cobra)
├── main.go Entry point
├── root.go Root command, persistent --realm, --progress, --progress-log and -v flags, TraverseChildren, the strict tree (go-clikit's cobratree)
├── context.go Dependency injection via context; --realm flag glue for cluster.ResolveRealm
├── exitcode.go Exit statuses, the words of flag value errors and the usage line of argument errors (exit 2), child exit status of ssh/exec/enter/logs
├── logging.go Logger construction from -v verbosity
├── lock.go Realm lock for mutating commands (pkg/state): inLockedRealm, and updateSSHExport after the change
├── completion.go Shell completion for cluster/node names
├── nodeargs.go Node argument parsing (go-nodeset expressions, cluster suffix)
├── sshexport.go SSH config export to ~/.local/state/sind/
├── output.go -o/--output handling (human, json; SIND_OUTPUT)
├── progress.go Progress of create, delete, connect, disconnect and power: withProgress, the flags and variables handed to go-clikit's cliprogress, the streams under a display
├── mcp.go MCP server setup (ophis): tool selection, JSON output, annotations, tool processes in a process group of their own
├── mcpstream.go sind mcp stream: HTTP server with a bearer token, forwarding to ophis
├── mcpeditor.go sind mcp <editor> enable/disable/list: register sind's MCP server with Claude Desktop, VS Code, Cursor
├── worker.go Worker create/delete commands
├── ui.go sind ui: flags only, the server is internal/ui
└── *.go One file per command group
internal/mock/ Test doubles for cmdexec.Executor
├── mock.go mock.Executor (FIFO + OnCall dispatch)
├── recorder.go mock.Recorder (mock in unit mode, OSExecutor in integration mode)
└── recording.go mock.RecordingExecutor, RecordedCall
internal/mcpconfig/ Edits the MCP servers of an editor's configuration (JSONC) in place, with a backup
internal/envvar/ The names of the SIND_* environment variables and Lookup, which takes an empty variable for one that is not set; the CLI, pkg/cluster and sind ui read them through it
internal/hostname/ DNS label check behind config.CheckName (cluster and realm names) and pkg/ssh's ssh_config host names
internal/natsort/ Natural order of names (dev2 before dev10), for every listing of pkg/cluster and the mesh DNS records
internal/detach/ The context of a write that must not be cut short (mesh DNS, known_hosts, sind-nodes.conf): it ignores the caller's cancellation and deadline, but ends with the rollback it runs under
internal/help/ The help texts (words, the actions of sind ui, its pages, how-tos) in embedded YAML: what sind help TOPIC prints and sind ui shows as hint cards, in its help pane and in its jump box; Load checks them
internal/ui/ sind ui, the web UI (doc.go has the extension API: pages, fragments, routes, services)
├── server.go The handler: go-clikit's localweb checks and sessions (headers, Host, same origin, token, CSRF) in front of the pages; ui.go: the listener of 127.0.0.1, localweb.Serve, and the single instance and the browser through localweb/launch
├── sse.go Live updates: renders the fragments of a page for go-clikit's localweb/sse
├── pg_*.go, more_*.go, run_*.go Pages: realm, clusters and nodes; mesh, monitoring, topology, doctor; runs
├── store/ Immutable snapshots and their change topics (a localweb/sse Notifier)
├── collect/ Collectors: Docker events, realms, clusters, statuses, lock, monitoring, doctor
├── activity/ Reads the activity log into runs (Tail, Reducer, Follow)
├── action/ The actions of the dialogs: forms to argv, validated against the snapshot
├── run/ Runs a dialog's command as a sind child process and stops it with SIGINT
├── demo/ --demo: made-up clusters and commands that write event logs in memory
└── web/ Embedded templates, CSS and JavaScript (htmx comes from go-clikit's localweb/htmx)
internal/testutil/ Shared test helpers
├── testutil.go ExitCode1, NoSuchContainer/Network/Volume, Image (SIND_TEST_IMAGE), TarModes, WithoutCalls
├── client.go NewClient, Realm (unit test variants)
└── client_integration.go NewClient, Realm (integration test variants)
pkg/cmdexec/ Command executor abstraction
├── exec.go Executor interface, OSExecutor (tees output to the progress span of the call)
├── output.go WithStdoutAsData: a command's stdout is data, not lines to show; WithLineParser: a callback for the lines a command writes, as PullImage reads those of docker pull; WithProcessGroup: a cancel kills the command's process group
├── terminal.go WithTerminal: a command's stdout is a pseudo-terminal, as docker pull writes its byte counts only there; TerminalLines turns what it draws there into plain lines
├── stream.go Process, Start (long-lived commands with streamed stdout)
├── exiterror.go ExitError (exit code + stderr of a failed command)
└── logging.go LoggingExecutor (TRACE-level command logging)
pkg/docker/ Docker CLI wrapper
├── client.go Client type, run/do/output/exists helpers, each docker command a hidden progress call
├── container.go Container operations
├── network.go Network operations
├── volume.go Volume operations
├── image.go Image operations
├── pull.go The work, in bytes or in layers, that PullImage reports from the status lines of docker pull
├── info.go docker info (daemon version, cgroup version, security options, OS and kernel)
├── endpoint.go Daemon endpoint the docker CLI uses (DOCKER_HOST, docker context)
├── plugin.go Volume plugin queries (CVMFS)
└── labels.go Docker Compose compatibility labels
pkg/cluster/ Cluster operations (orchestration)
├── create.go Cluster creation flow
├── delete.go Cluster deletion; DeleteAll's pool of clusters (fanout.Map)
├── get.go Listing clusters, nodes (GetNode, GetAllNodes), networks, volumes; GetSSHConfig
├── status.go Health status collection
├── diagnostics.go Low-level diagnostics helpers used by get cluster/node
├── slurminfo.go What Slurm says about a cluster, for sind ui (GetSlurmInfo: sinfo's nodes and partitions, slurm.conf and sind-nodes.conf)
├── errors.go Error sentinels for library callers (ErrClusterExists ... ErrNoMonitoring), rollback bound
├── notfound.go Cluster-not-found error naming the realms that hold the cluster
├── readiness.go The --wait limit of Create and WorkerAdd
├── ha.go Controller pair (backup controller) position and control state
├── realm.go Realm resolution (ResolveRealm: --realm > SIND_REALM > config > default)
├── clustername.go Cluster resolution (ResolveCluster: argument > SIND_CLUSTER > config > default)
├── db.go Accounting services (mariadb, slurmdbd) on the db node
├── api.go slurmrestd setup steps of the api node
├── accounts.go Slurm accounts, associations and coordinators (sacctmgr)
├── users.go Linux users and groups, their labels, home directories
├── identity.go Identity modes: nss_slurm on workers, sackd on the submitter
├── cvmfs.go CVMFS backend detection and mount arguments
├── monitoring.go Monitored clusters: image probe, journal volume and helper, the realm's stack (GetMonitoring)
├── worker.go Worker add
├── worker_remove.go Worker remove
├── power.go Power state operations; the pool of each action (fanout.Map)
├── node.go Node initialization and setup
├── setup.go In-container node setup (nss_slurm, slurmrestd, users, SSH) in one docker exec
├── discovery.go Cluster/node discovery queries, VolumeType
├── external.go External containers on the mesh: connect, disconnect, and their connection to new clusters
├── resources.go Resource creation helpers
├── types.go Shared types (Cluster, Node, State)
├── ssh.go ssh/enter/exec arg building and target selection
├── logs.go Log command arg building
├── dns.go Node DNS names and search domain
├── naming.go Resource naming conventions
├── progress.go Progress span helpers of the pools (startStep, startTargets, nodeTargets, endSpan), joinFailures, WithPanicLog
├── rollback.go The rollback step of a failed Create or WorkerAdd
└── preflight.go Pre-creation validation
pkg/config/ YAML configuration parsing and validation
pkg/doctor/ Host prerequisite checks (Docker version, the daemon's mode, where the daemon runs,
cgroupv2 and the nsdelegate probe, inotify, DNS policy); `doctor.Run` assembles and runs them
pkg/log/ Context-based structured logging (slog)
pkg/mesh/ Global infrastructure (mesh network, DNS records, SSH relay and keypair, host DNS)
pkg/monitor/ Event-driven Docker and systemd watchers for readiness
pkg/observability/ The realm's monitoring stack (Alloy, Prometheus, Loki, Grafana): containers,
configuration, Alloy config generation, dashboards
pkg/probe/ Node readiness probes
pkg/slurm/ Slurm and slurmdbd config generation, sind-nodes.conf editing, version
discovery and image probe (ProbeImage), sinfo parsing, munge key, slurm.key
and jwt_hs256.key generation, sacctmgr account commands
pkg/ssh/ SSH key injection, host key collection, ssh_config export
pkg/state/ sind's state directory and the realm lock (flock, and the <realm>-lock
network on the daemon) that library callers of Create, Delete, DeleteAll,
WorkerAdd, WorkerRemove, Connect, Disconnect, PowerOn, PowerReboot and
PowerCycle hold; Holder reads who holds itDependency flow
cmd/sind → pkg/cluster → pkg/cmdexec
→ internal/detach
→ internal/envvar
→ internal/natsort
→ pkg/config → internal/hostname
→ pkg/docker → pkg/cmdexec
→ pkg/log
→ pkg/doctor → pkg/docker
→ pkg/cmdexec
→ pkg/config
→ pkg/log
→ pkg/mesh
→ pkg/log
→ pkg/mesh → pkg/docker
→ pkg/cmdexec
→ pkg/config
→ pkg/log
→ internal/detach
→ internal/natsort
→ pkg/monitor → pkg/docker
→ pkg/cmdexec
→ pkg/log
→ pkg/observability → pkg/config
→ pkg/docker
→ pkg/log
→ pkg/mesh
→ pkg/probe
→ pkg/probe → pkg/docker
→ pkg/config
→ pkg/log
→ pkg/monitor
→ pkg/slurm → pkg/docker
→ pkg/config
→ pkg/ssh → pkg/docker
→ internal/hostname
→ pkg/state → pkg/docker
→ pkg/log
→ pkg/doctor
→ pkg/observability
→ pkg/state
→ github.com/GSI-HPC/go-clikit/cobratree
→ github.com/GSI-HPC/go-clikit/termtext
→ github.com/GSI-HPC/go-nodesetcmd/sind also imports pkg/cmdexec, pkg/config, pkg/docker, pkg/log, pkg/mesh and pkg/ssh directly, and internal/envvar, internal/help, internal/mcpconfig, internal/ui (sind ui) and internal/ui/collect (the bounds of its --monitoring-interval).
The pkg/cmdexec package provides the executor abstraction at the bottom of the stack. pkg/docker wraps Docker CLI commands and pkg/mesh uses a separate executor for system commands (resolvectl, systemctl). The pkg/cluster package orchestrates everything. pkg/doctor runs host prerequisite checks (doctor.Run) directly from cmd/sind (no cluster orchestration; cmd/sind/doctor.go only writes the result with go-clikit’s doctor package, whose Check is the type of the checks); pkg/cluster uses its nsdelegate probe in the create preflight. pkg/monitor streams Docker and systemd events for event-driven readiness. The internal/mock and internal/testutil packages are test-only and not part of the production dependency graph. cmd/sind escapes the final error line, get table cells and doctor details with the termtext package of go-clikit, writes files whole (editor configurations, the MCP stream’s token, the configs of sind ui) with its fsutil package, and takes from its cobratree package the strict command tree (cobratree.Strict: a group refuses an unknown subcommand, help an unknown topic), the usage errors that exit 2 (cobratree.Usage, cobratree.IsUsage) and the build info sind version falls back to (cobratree.ReadBuildInfo). It parses node arguments (worker-[0-3]!worker-2) with go-nodeset in nodeargs.go.
The packages that report progress, pkg/cluster, pkg/cmdexec, pkg/docker, pkg/mesh and pkg/probe, import go-clikit’s progress package, and pkg/cluster its fanout package for the pools of the power commands and DeleteAll; only cmd/sind imports progress/display, which shows the spans (see Progress spans).
Adding a new CLI command
-
Create the command file in
cmd/sind/(e.g.,mycommand.go) -
Define the cobra command with
Use,Short,Args, andRunE. AnArgscheck that fails exits 2, a usage error;RunEreturnscobratree.Usage(err)orcobratree.Usagef(...)for a flag value it rejects, andargsError(err)orargsErrorf(...)for an argument, which ends with the usage line, so that it exits 2 too -
Wire it up in
root.goviacmd.AddCommand(newMyCommand()) -
Use context helpers to get the Docker client and mesh manager:
ctx := cmd.Context() client := clientFrom(ctx) realm, err := realmFromFlag(cmd) if err != nil { return err } meshMgr := meshMgrFrom(ctx, client, realm) -
Implement the operation in
pkg/cluster/(not incmd/sind/) -
Show its progress if it changes clusters and asks nothing: wrap
RunEwithwithProgress(...), and have the operation report its work as progress spans (see Progress spans) -
Classify it for MCP in
mcpEffects(read-only, additive or destructive), or list it inmcpExcludedif it is interactive, runs until interrupted or prints a secret (cmd/sind/mcp.go); a unit test fails for an unclassified tool -
Write tests for both the CLI layer and the cluster operation
The CLI layer should be thin — argument parsing, flag handling, and output formatting. Business logic belongs in pkg/cluster/.
Adding a Docker operation
- Add the method to
pkg/docker/client.go(or the appropriate resource file) - Follow the pattern: call
c.run()orc.runWithStdin(), parse output;c.do()runs a command for its effect alone andc.output()returns its stdout. They all report the command as a hidden progress call named after its subcommand, and each waits for one of the client’sdocker.MaxConcurrentCallsslots, so a command must not wait for another docker command while it runs; long-lived streams go throughExecutor.Startand take no slot - Use strong types:
ContainerName,NetworkName,VolumeName, etc. - Write unit tests using
mock.Executor
Key patterns
Executor abstraction
All external commands go through the cmdexec.Executor interface (from pkg/cmdexec), making every operation testable, with two exceptions (see below):
type Executor interface {
Run(ctx context.Context, name string, args ...string) (stdout, stderr string, err error)
RunWithStdin(ctx context.Context, stdin io.Reader, name string, args ...string) (stdout, stderr string, err error)
Start(ctx context.Context, name string, args ...string) (*Process, error)
}pkg/docker uses an executor for Docker CLI calls. pkg/mesh uses a separate executor for system commands (resolvectl, systemctl, pkcheck). The CLI wraps the mesh executor in a LoggingExecutor to emit TRACE-level logs for system commands.
One is dockerExec in cmd/sind/ssh.go, used by ssh, enter, exec and logs. It runs docker through os/exec with the terminal’s stdin, stdout and stderr attached, because Executor only captures output, and unlike the calls through docker.Client it logs no TRACE line. Its unit tests cannot use mock.Executor: they put the test binary first on PATH as a fake docker, controlled by the SIND_TEST_DOCKER_* variables in cmd/sind/main_test.go.
The other is the browser that sind ui --open opens: go-clikit’s localweb/launch starts the desktop’s opener (xdg-open, open) with launch.Detached, in a session of its own and bound to no context, so that stopping sind ui does not take the browser with it. Its unit tests replace it through the start field of internal/ui’s Options.
Context-based dependency injection
The CLI layer injects dependencies via Go context:
ctx = withClient(ctx, client)
ctx = withMeshMgr(ctx, meshMgr)
ctx = sindlog.With(ctx, logger) // injected by PersistentPreRunECommands retrieve them with clientFrom(ctx) and meshMgrFrom(ctx, ...). The logger is injected automatically by the root command’s PersistentPreRunE based on the -v flag count; under a progress display, withProgress replaces it, and the stderr withStderr stored, with writers that go above the display.
Structured logging
The pkg/log package provides context-based logging via slog. All pkg/ code extracts the logger from context — never from slog.Default():
log := sindlog.From(ctx)
log.InfoContext(ctx, "creating cluster", "name", cfg.Name)
log.DebugContext(ctx, "waiting for node", "node", shortName)
log.Log(ctx, sindlog.LevelTrace, "docker", "cmd", strings.Join(args, " "))When no logger is in the context (library use without the CLI), From returns a no-op logger. In errgroup goroutines, use gctx (not the outer ctx) for log calls.
Progress spans
Work that takes a while reports what it does as progress spans of go-clikit’s progress package, on the context: progress.Start(ctx, kind, name, opts...) returns the span’s context, under which the spans of the work below it start, and the span, which the work ends with its error (span.End(err)). Without a Bus in the context, as for a library caller or a test that watches none, Start returns a nil span, and every method of a nil span does nothing, so the code needs no branch for it.
- A step (
progress.KindStep) is a phase of the command; a step with theFoldflag and aTotalcounts its targets (KindTarget: a node, an image). A wait (KindWait) can have a bound (progress.Timeout) and a message that says what it waits for; a call (KindCall) is one request, such as a docker command. - Announce every target of a counted step queued (
progress.Queued()) before the first of them runs, then mark each running (span.Run()) when its work starts: a pool that starts each target in its own goroutine as it takes its place breaks the counts a display draws, andprogresstest.Checkfails it. pkg/cluster/progress.gohas the helpers of the pools written by hand:startStepstarts a step,startTargetsandnodeTargetsannounce its targets, andendSpanends a span canceled (stopped) when a sibling’s failure or an interrupt stopped its work, rather than failed withsignal: killed.- A pool that tries every item and keeps going after a failure runs on go-clikit’s
fanout.Map, which announces its targets asCheckrequires, bounds them withLimit, and turns a panic in one item into that item’s error (sind panicked; this is a bug, please report it: ...) with its stack in the Bus’s panic log, or without a Bus wherecluster.WithPanicLogsays (withStderrsets the command’s stderr). The power commands andDeleteAllpassjoinFailuresas itsSummarize, which joins each item’s error worded as before, so the error text anderrors.Isdo not change. cmd/sindalone decides how the spans are shown:withProgress(progress.go) runs the command in a span of its own and makes the Bus through go-clikit’scliprogresspackage, with the display--progressasks for and the event log--progress-lognames (whichcliprogressopens privately) or, when neither it norSIND_PROGRESS_LOGis given, the activity log ($XDG_STATE_HOME/sind/activity.jsonlfromstate.PrepareActivityDir, handed tocliprogressas its default log, rotated above 8 MiB, left out silently when it cannot be used), and tears them down beforerunprints the error line.pkg/never writes to stdout or stderr: a display owns the bottom rows of the terminal, and only the writerscmd/sindhands out, cobra’s streams,stderrFrom(ctx)and the logger, go above it.
Resource naming
All resource names are derived from cluster name and realm via functions in pkg/cluster/naming.go. Never hardcode resource name prefixes.