Architecture

The kit is a set of packages that a command-line program and the library packages it calls share: a library reports what it does, and the command decides how that is shown. This document says where the packages came from, how they fit together, what runs on which goroutine, what a program fills in, and the convention for standard error that the programs using the kit follow.

How the kit came to be#

The packages were written for clusterctl, a command-line tool that administers HPC clusters and works on hundreds of hosts at a time, to show how far such a command has got. They were written in its internal/ tree, where clusterctl’s records of the decisions behind them still are: ADR 0021 on reporting progress as events of its own, and ADR 0022 on the bounded pools.

sind, which runs Slurm clusters in Docker, set out to report its progress the same way, from a library that other programs import too. So clusterctl cut what tied the packages to it: its exit codes became a hook, its name and the noun for its targets parameters, and the escaper moved out of its output package. The packages then moved here with their history, as they shipped in clusterctl v0.4.0, under paths of their own. progress/cliprogress was written here afterwards, from the glue between the command line and the progress packages that clusterctl and sind each carried (decision 21), and cobratree from the code both carried to make a cobra command tree strict and to read the build info (decision 24).

The packages#

Every arrow points down the table: nothing imports cliprogress, progresstest, mcpprogress, eventlog or fanout, only cliprogress imports display, progress imports only termtext, and termtext only golang.org/x/text. cliout and doctor import only termtext, and fsutil too, on Linux, where it names the holder of a lock. cobratree stands apart: it imports no package of the kit, and none imports it, not even cobratree/cobratreetest, which takes a cobra command. A library imports progress, and fanout when it works on many items; the command imports cliprogress, or display when it picks its displays by a rule of its own, an MCP server mcpprogress, a program that reads back the event log it or a child wrote eventlog, cliout to write its result, doctor for its doctor command, fsutil for the files it shares or keeps private, and cobratree when it is built with cobra; tests import progresstest, fanouttest for a limit, and cobratreetest for a command tree. The localweb packages stand apart as cobratree does, on the web side: they import no package of the kit but their own, and are the only ones that import net/http and html/template. go-nodeset folds the names of targets into node sets, exe[0001-0480], in the tree, the tests’ trees and the summary of a pool that failed. Decision 3 says what else the kit may require, which is nothing without a record; decisions 22, 26 and 35 let cliprogress, fsutil and localweb/launch alone import golang.org/x/sys, and decision 24 cobratree alone cobra and pflag.

Spans and the Bus#

Work is a tree of spans, one level for each kind of unit: the command, its steps, the batches of a staggered action, the targets a step works on, the calls made for a target, and the waits in between. The six kinds are an interface of the kit. A span travels in the context.Context the work is done under, so a call nests under the target it is made for without being handed down, and a library reports its work without a parameter for it.

A command makes a Bus with its sinks, puts it in its context with WithBus, and starts the command’s span. Every change to a span, a start, a run, an update, a line of output or an end, is one Event of plain data, numbered by Seq, and handed to every sink in that one order.

Span.End takes the error the work ended with. A nil error ends the span ok; an error that is progress.ErrSkipped, as errors.Is tells, such as one progress.Skip returns, ends it skipped, work left out on purpose, before any rule for classes is asked, so that a program’s rule cannot turn a skip into a failure; any other error ends it failed or canceled, as progress.Classify tells. Span.Skip is End with a skip. errors.Is looks through every error an error joins, so one that joins a skip with a failure, by errors.Join or two %w verbs, ends skipped, and the failure goes uncounted: work that gathers the errors of its parts returns a skip only when every part was skipped.

Without a Bus in the context, Start returns the context as it is and a nil *Span, whose methods do nothing. The path allocates nothing, which two tests hold it to, so a library reports its work at no cost when no command is listening.

What a span says is closed: the fields of progress.Fields, and nothing else. There is no field for an argument vector, a script, standard input, the environment or a header, so none of them can reach a display or the log. Every text in an event has been through Sanitize, which applies carriage returns the way a terminal does, escapes the rest with termtext.Escape and cuts it to its bound. The Bus calls it, so neither the work that reports a span nor a sink needs to; a program that shows text from elsewhere outside a Bus may call it too.

A pool announces every target queued before the first runs, marks each running as it takes its place, and ends it before it gives the place up, so a display knows the whole of the work from the start, never counts more running than the limit, and reaches its total however the work ends. progresstest.Check holds every source of events to this, and fanout keeps it without a line of the caller’s.

Work#

A span may also report its work: an amount in a unit, out of a size when one is known (decision 23). The span’s own Advance, SetAmount, AdvanceWriter and AdvanceReader add to an atomic amount and nothing else, so a copy that writes a hundred thousand times a second takes no lock. The Bus samples that amount: it sends it as a TypeAdvance event at most every 100 ms a span, by its own clock, the newest amount never lost, and with the span’s End. Advance and SetAmount read that clock, BusOptions.Now, on the goroutine that calls them and outside the Bus’s lock, so the clock must be safe for concurrent use. A span without a Bus costs nothing, as everything else on a nil span.

An event carries only what its span said. progress.Meter is the sink-side half: given every event in order, as Tally is, it keeps the work of each span and rolls it up the tree, an amount within a unit, a share done from the span’s own size, else from the targets it counts, short of done until their count is complete, else from the work below, where a span with a size of its own counts for everything below it, and the rate, the time left and a stall from the amounts’ times. The displays each keep a Meter and read it at the time of a frame, so the rows move between events; progresstest.Capture is sent every advance, and progresstest.Check holds them to their rules.

The event log samples on its own. It writes an advance of a span at most once a second by default (LogOptions.AdvanceEvery), by the events’ times, and counts the advances it leaves out in leftOut; they are the only gaps in seq. The displays, the Meter and Capture miss none.

Concurrency#

• Sinks run under the Bus’s lock. The Bus calls every sink’s Handle with its lock held, in Seq order. A sink updates memory and returns: it never blocks, never writes to a terminal or a file there, and never calls the Bus back. A sink that panics is taken off the Bus, whatever its type, and its stack goes to BusOptions.PanicLog. The Bus knows its sinks by where they are listed, never by comparing them, and releases its lock however a call under it ends, so neither a sink nor a clock that panics leaves it locked. The clock, BusOptions.Now, is the one thing the Bus calls outside its lock as well, from Advance and SetAmount, so it must be safe for concurrent use.
• Displays draw on their own time. Start gives a display a goroutine that draws from what Handle kept, ten times a second for the tree and the counter; a test calls Draw instead, on a clock it moves. The event log writes from a goroutine of its own too, in whole lines, and stops, rather than hold the work up, when its writer falls 8 MiB behind. mcpprogress.Notifier sends from a goroutine of its own as well, at most every 500 ms and at once as a step or a batch ends; its Close, after the Bus’s, sends what is left, so that nothing follows the answer to the call, and waits no longer than the context it is given. fsutil.Lock and launch.Acquire wait for their locks no longer than their contexts either. eventlog.Tail starts no goroutine: the program polls it from one of its own, and the Feed it gives the lines to, and a sink behind an eventlog.Forwarder, are called there, under no lock of the kit’s.
• The terminal is lent. progress.Suspend calls every sink that is a Suspender outside the lock, and returns once each display is off the terminal, so that a question can be asked there; the function it returns puts them back, even a display taken off the Bus for panicking in between. Suspensions nest.
• One order for the terminal. The command’s own writes go through Terminal.Writer, which takes the display’s region off first and lets it back only once a line has ended, so a question that waits for its answer is never drawn over. The region comes off one line a row, as it was drawn, so a line of the command’s is never cleared, even on a terminal made narrower since (decision 9). Lines from goroutines beside the command, a log line or the stack of a panic a pool recovered from, go through Terminal.Lines, which holds them, whole lines only and bounded, while a question is asked or the command has a line open. The lines a display leaves for good are written above the region, ahead of whatever the command writes after the events they tell of. A Terminal carries one display, from the constructor that makes it to its Close, after which the writers pass bytes through; making a second display on it panics.
• The pools. fanout.Each is the one loop every pool runs on: at most its limit at a time, and nothing started once the context has ended, so an interrupt stops every pool the same way. Map returns what each item came to in the order the items were given, whatever order they finished in, with the error its step ended with, and turns a panic in one item’s work, or in what it acquired and released for it, into that item’s error, since recover reaches only its own goroutine; Each and Batches recover nothing, and a panic in a batch goes up the goroutine that called Batches. Work, an acquire or a release that calls runtime.Goexit, as t.FailNow does, fails its item with an error that says which of them did, rather than pass for done, and loses nothing else that broke the item. The end of the context, an interrupt or a deadline alike, ends the items it left out, and those whose acquire or work then gave up with an error other than a skip, canceled, with the context’s error, while each outcome keeps the error it was given. Both end in the same class, so the class does not tell an interrupt from a deadline; the error text in the event log does, context canceled or context deadline exceeded, as does the error of an item the pool left out, and the context itself. A panic or a call of runtime.Goexit is a bug, and fails its item even once the context has ended. An item its acquire refused ends at once, before it gives its place up, as every target does. What an item acquired is released before its target ends, so that a panic in the release is the target’s failure; the next item can so take it and run before that target has ended. Batches runs one batch after the other, with a pause between them, and stops after one that failed; a batch whose run returns a skip ends skipped and does not stop it. A skip is progress.ErrSkipped, which the pools test with errors.Is; the error of a batch not tried, fanout.ErrNotTried, is none, since a failure left that batch out, not a purpose (decision 13).

What a program fills in#

The kit knows no program. What depends on one is a parameter or a hook:

A Bus hands on what it was told: Bus.Program, Bus.PanicLog and Bus.Classify answer for the Bus that progress.BusFrom finds in a context, so that a library that runs work of its own names the program, writes its panics and classes its errors as the Bus does. A nil Bus answers “”, nil and progress.Classify without a fallback; the defaults are the caller’s. fanout.Map takes them so: its own Program, PanicLog and Classify override the Bus’s, and without a Bus or an option of its own it falls back to “the program”, standard error and ClassTarget. The rule that tells an item canceled is the one that classes its target, so the step and its targets agree (decision 12).

What stays in the program: the names of its flags and variables, and reading them, its own, NO_COLOR, TERM, COLORTERM and the locale’s among them; whether a stream is a terminal; whether to draw in colour and in how many colours (from NO_COLOR, TERM, COLORTERM and whether standard error is a terminal); taking TRACEPARENT and TRACESTATE out of its environment; its signals and its exit codes; which commands show progress, and the span each runs in; and putting its own writers, and its logger’s, through the Terminal while a display is drawn, and back. cliprogress picks the display, and whether the terminal can show one, from what the program tells it, and makes and takes down the Bus, the display, the summary and the event log; a program that picks by a rule of its own uses display directly. The kit reads no variable, sets nothing process-wide and ends no process, so two Buses in one process, one per call of a server, share no state.

The local web server#

sind, brig and sdash serve a page to the user who started them, on the user’s own machine. Such a server acts as the user, so whoever can send it a request acts as the user too: another user of the machine, who can reach a loopback port; any page in the user’s browser, which can send requests there; and a page of another site that makes its own name resolve to 127.0.0.1. localweb keeps the three out, each by a check of its own (decision 34):

• The listener. Listen binds a loopback address and nothing else, or a Unix socket that nobody but the user can reach or replace. On TCP, PeerListener closes, on Linux, a connection that a process of another user opened, before a request is read.
• The Host. Hosts refuses a request whose Host header is not one Bound.Answers names, which is how DNS rebinding shows: such a page names its own site. localhost is answered only while the server holds its port on both 127.0.0.1 and ::1.
• The site. SameOrigin refuses what a page of another site caused, by Sec-Fetch-Site and Origin, which a browser sets and no page can change. Every port of 127.0.0.1 is the same site, so the SameSite attribute of a cookie keeps out none of the pages other programs serve there. A page lets in a navigation from anywhere (OriginPolicy.Navigate); what a page asks for itself, its fragments, its stream and its forms, is let in only from its own origin.
• The headers. Headers sets the policy that lets a page load and contact its own origin alone, and forbids framing, sniffing and the referrer to other sites.

Behind them, localweb/session lets in only the browser that opened the address the program printed: the launch token in it is exchanged for a session cookie, and a request that may change something carries the session’s CSRF token.

localweb/sse streams the live parts of a page. The program’s store notifies the topics it changes; a stream subscribes, sends every fragment of the page first, since the page may be older than a change the stream did not hear of, and then each fragment whose topic changed and whose HTML differs from what it last sent. A subscriber that falls behind keeps the set of topics it missed, so it renders no more than changed. The stream is let in from the page’s own origin alone, and ends when localweb.Serve starts to stop, so that it does not hold the stop for the grace period.

A program chains them, outermost first: Headers, Hosts, SameOrigin, session.Manager.Middleware and its handler; Serve serves the chain with every timeout set. Each check answers a refused request through a Refuser, which the program may give to write it in its own form and log the reason; the messages are an interface of the kit.

Around the server, localweb/launch keeps one per user and opens the browser on it (decision 35). A start takes the lock of Acquire, and waits for it no longer than its context; when Running finds the record of a server that still runs, whose listening socket is the user’s, the start opens that one and ends; otherwise it listens, Publishes its address and drops the lock. The Opener writes the address, token and all, into a file only the user can read, and hands that file to the desktop’s opener, as a command line is readable by every user. localweb/assets serves the page’s files under a hash of them all, so a browser keeps each for good; localweb/page renders the page and its fragments from one template set, the same blocks the stream of sse sends; and localweb/htmx holds the scripts that swap them in, checked against their releases.

Standard error#

The programs that use the kit share one convention for what goes where, so that a script that reads their streams, and a person who reads their terminal, finds the same in each:

• Standard output carries the command’s result, and nothing of its progress, whatever display is drawn.
• Standard error carries, in this order of precedence:
  • progress, drawn by a display only when the program chose one: the tree or the counter only on a terminal, plain lines wherever they are asked for;
  • notes, lines the command writes itself, such as a fallback it took or a pause it waits out, through Terminal.Writer while a display is drawn;
  • logs, lines from goroutines beside the command, through Terminal.Lines, so that none lands inside a question;
  • and last, once the display is gone, the summary a display leaves, and one line for the error, escaped with termtext, which says what failed and why.
• Without a display, standard error holds exactly what it would without the kit, byte for byte. A command done within its first second draws nothing and leaves no summary.

Text that came from elsewhere, a remote host, a container or a service, is escaped with termtext.Escape, or EscapeLines where its lines are kept, which keeps a CRLF line end as it is (decision 42), before it reaches a terminal, and there is no other escaper: a program that escapes everything with it has one place to fix when a character turns out to need escaping too. Decision 7 calls the two by their earlier names, EscapeCell and EscapeText. The forms of the escapes are stable, but the runes escaped may grow in a minor release (decision 14). termtext.Width counts the columns of text that is already escaped. Decision 8 names RuneWidth, the width of one rune, which is no longer exported: Width(string(r)) gives it.