The data layer¶

The data layer (sources and pipelines) is the heart of tui-builder: the TUI is one consumer of it and wrangl is another. This page covers how it is put together. For every field, see the reference: sources, pipelines and templating.

One map, two kinds of entry¶

Everything under data.sources: is one map. Each entry's type: picks either a leaf kind, which fetches from outside the program (http, exec, file, websocket, static, merge), or an operator, which reads other entries and transforms them (passthrough, filter, project, derive, sort, union, compose, join, cache). Components and wrangl address either kind by name, so swapping an http source for an exec one, or putting a filter in front of it, changes nothing downstream.

The split in the YAML mirrors the code: data: is read by both binaries, tui: only by tui-builder.

The model¶

Every entry, leaf or operator, satisfies one contract:

type Source interface {
    Fetch(ctx context.Context) (any, error)
    Refresh() time.Duration
}

type StreamingSource interface {
    Source
    Subscribe(ctx context.Context) (<-chan Event, error)
}

Fetch returns the value with root: already applied. Refresh reports the polling interval. Streaming sources also implement Subscribe, pushing one Event per frame until cancelled.

Two more interfaces cover sets too big to hold whole:

type WindowedSource interface {
    Source
    FetchWindow(ctx context.Context, q WindowQuery) (WindowPage, error)
}

type AnchoredSource interface {
    Source
    FetchEdge(ctx context.Context, q EdgeQuery) (EdgePage, error)
}

A source with window: is windowed: the request carries which rows, matching what filter, in what sort order, and the answer is one page. With window.cursor: it is Anchored instead: it's walked from an edge (older or newer than a cursor) rather than paged by offset. Both still implement Fetch, which returns the first page, so wrangl sees representative data. The TUI and wrangl consume sources through these interfaces and nothing else.

Lifecycle¶

Two independent properties, both inferred from the config (there is no lifecycle: field), and reported by wrangl --list and --describe:

They combine: polled (refresh: 5s) · needs params, streamed (follow) · needs params.

A windowed source is fetched a page at a time as the bound component scrolls, filters and sorts, rather than on a cadence. One with window.growing: also polls its newest end every window.follow_every (default 2s) while its while: expression holds; refresh: can't be combined with it.

wrangl¶

wrangl runs this layer on its own and writes the result to stdout, which makes every source and pipeline inspectable and scriptable without a terminal UI. See wrangl for its flags and output format.

The boundary¶

The data layer never depends on the TUI:

• internal/config, internal/datasource, internal/pipeline, internal/expr, internal/action, internal/output and cmd/wrangl must not import internal/screen, internal/build or any tuilib package.
• scripts/check-data-layer-boundary.sh walks go list -deps in CI and fails the build on a violation.

Data wrangling is the product and the TUI is one sink. wrangl stays cheap to build and test without Bubble Tea. See AGENTS.md for the rule.