Sources¶

Every entry in data.sources: has a type:. Leaf kinds fetch from outside the program and are covered here: http, exec, file, websocket, static and merge. Operator kinds transform other entries and are covered in pipelines.md. Both share one map, so a component or wrangl addresses either by name.

How a source delivers data over time follows from its fields; there is no lifecycle: setting:

A source that declares a required parameter with no default "needs params": it can only run once a caller binds them (a push binding, on_cursor, a join, or wrangl --param).

Fields shared by leaf sources¶

Validation: format is json or text; refresh, timeout and cache.ttl parse as durations.

http¶

Requests a URL and parses the response. For REST and JSON APIs, GraphQL (via a POST body), kube proxies, and streamed endpoints with follow: true.

releases:
  type: http
  url: https://api.github.com/repos/jsdrews/tui-builder/releases
  refresh: 5m
  timeout: 15s
  headers:
    Accept: application/vnd.github+json
    Authorization: "Bearer ${env.GITHUB_TOKEN}"

exec¶

Runs a command and consumes its stdout. Any CLI that prints JSON becomes a source (kubectl get -o json, gh api, scripts); with follow: true, any command that streams lines (kubectl logs -f, tail -f).

recent_commits:
  type: exec
  command: [sh, -c, "git log -20 --format='{\"sha\":\"%h\",\"subject\":\"%s\"}' | jq -s ."]
  refresh: 30s

file¶

Reads a file from disk.

people:
  type: file
  path: ./examples/fixtures/people.json
  refresh: 5s

websocket¶

Connects to a WebSocket and emits one event per frame. Always streamed.

trades:
  type: websocket
  url: wss://ws.bitstamp.net
  initial_messages:
    - '{"event":"bts:subscribe","data":{"channel":"live_trades_btcusd"}}'

static¶

Inline data in the YAML. No I/O. For fixtures, lookup tables, offline demos and building a pipeline before pointing it at a real source.

regions:
  type: static
  data:
    - {code: us-east-1, name: N. Virginia}
    - {code: eu-west-1, name: Ireland}

merge¶

Fetches several sources and concatenates their results into one list, optionally tagging each row with where it came from. Streams when every child streams; otherwise polls. Prefer the union operator in new configs; merge remains for its leaf fields (refresh, timeout).

Validation: exactly one of sources and children; tag_field only with sources; on_error is fail or skip.

Merge children¶

pods_all:
  type: merge
  children:
    - {source: pods_prod,    tags: {cluster: prod,    cluster_url: "http://localhost:8001"}}
    - {source: pods_staging, tags: {cluster: staging, cluster_url: "http://localhost:8002"}}
  on_error: skip
  refresh: 5s

Parameters¶

Typed inputs on a source, referenced as ${params.NAME} in its templated fields (url, method, body, headers, path, command, env, initial_messages). Callers supply values: a push binding's bind:, on_cursor.bind, a join lookup's on:, or wrangl --param NAME=VALUE. On an operator, values are available in its expressions as params.NAME (see pipelines.md).

The form fields (label, placeholder, options, mask, order) are accepted too but only matter where a parameter is rendered as a form field, which is in action inputs and boot prompts.

Resolution: the caller's value, else default, else an error if required, else empty. A supplied name the source doesn't declare is an error, so typos surface. A ${params.X} with no declared parameter is left literal, so it shows up in the URL rather than vanishing.

pod:
  type: http
  parameters:
    namespace: {required: true}
    name:      {required: true}
  url: http://localhost:8001/api/v1/namespaces/${params.namespace}/pods/${params.name}

paginate¶

Walks every page of an http endpoint before returning, concatenating each page's items (after root:) into one list. For sets of up to a few thousand rows; for bigger ones use window.

jobs:
  type: http
  url: ${env.AWX_HOST}/api/v2/jobs/
  root: results
  paginate: {strategy: link, next_path: next, max_pages: 50}

window¶

Fetches only the rows a table is showing, and fetches again as the user scrolls, filters or sorts, so the server answers the filter and sort over the whole set.

Supported on http (the request goes in the query string) and exec (the request is templated into the argv).

On exec, command: uses these tokens instead of the *_param fields:

An argv element whose window tokens all resolve empty is dropped, so --author=${window.filters.author} disappears when no author term is typed. Substitution is literal, so in a sh -c script the quoting around a token is what keeps it safe; plain argv elements are passed as single arguments.

Query parameters already on an http url: are kept on every request; a parameter the window also sets is overwritten, so ?q=x on the URL acts as a default search the user's filter replaces. refresh: and r refetch the window on screen in place.

Anchored windows¶

Some APIs can't jump to row N: Elasticsearch past 10,000 hits (search_after), log APIs filtered by timestamp, and any API that pages with an ?after=<token>. They're Anchored: read by walking from an edge item to the items before or after it. Setting cursor: makes a window Anchored. An eventlog bound to one opens at the newest item and walks older as you scroll up, or, with anchor: oldest, opens at the oldest and walks newer.

Each request carries the edge item's cursor, which direction to walk, and the page size:

• http: the older: or newer: map is merged into body: (a JSON object), or, for a request with no body, set as query parameters. A value that is exactly ${window.cursor} takes the cursor's JSON value (an ES sort array stays an array) and is left out of the first request, which has no cursor yet. Inside body: text, ${window.limit} is a number and ${window.search} / ${window.filters.NAME} are escaped for a JSON string. search_param and, with no body, filters: still name query parameters.
• exec: the command reads ${window.cursor} (a string cursor unquoted, anything else as JSON), ${window.dir} (older or newer) and ${window.limit}, plus the usual filter tokens.

Answer an older walk newest first, as a descending sort does; tui-builder reverses it. ${window.limit} is one more than the page size: whether the extra item comes back is how tui-builder knows that edge has more.

Items are held by cursor plus the component's key:, so two items with the same cursor (log lines stamped the same millisecond) both stay.

logs:
  type: http
  method: POST
  url: ${env.ES_URL}/app-logs/_search
  root: hits.hits
  body: '{"query": {"query_string": {"query": "${window.search}*"}}, "size": ${window.limit}}'
  window:
    cursor: sort
    older: {sort: [{"@timestamp": desc}, {_id: desc}], search_after: "${window.cursor}"}
    newer: {sort: [{"@timestamp": asc},  {_id: asc}],  search_after: "${window.cursor}"}

An Anchored window has no offsets, total or user sort, so offset_param, total_path, sort_param, sorts, sort_desc_prefix and prefetch don't apply. Where it opens is anchor:; the eventlog's start: is for paged sources and is rejected here. An eventlog or a table can bind one; a table over an Anchored source is a span table (see components.md).

Growing¶

growing: as a map names another source and a condition on its value, re-evaluated every time that source answers. That source needn't be shown anywhere: the screen fetches it when it opens, and polls it on its own refresh:, so give it one.

data:
  sources:
    job:
      type: http
      url: ${env.AWX_HOST}/api/v2/jobs/${params.id}/
      refresh: 5s
    job_events:
      type: exec
      command: [awx-events, --job, "${params.id}", --offset, "${window.offset}", --limit, "${window.limit}", --search, "${window.search}"]
      window:
        total_path: count
        growing: {source: job, while: 'status in ["pending", "waiting", "running"]'}
        follow_every: 2s

Validation:

• window: only on http and exec, with json format, not with follow or paginate.
• Only a type: table or an eventlog may bind a windowed source, and an operator can't consume one (it would see one page and present it as everything). growing: is eventlog-only, can't be combined with refresh:, and its source: must exist and its while: compile; follow_every needs growing.
• http: offset_param and limit_param are required, and at least one of search_param / filters (otherwise the filter bar has nowhere to send input). An Anchored http window can instead carry the filter as ${window.search} in body:, older: or newer:. sorts and sort_desc_prefix need sort_param.
• exec: the http-only fields are errors. command: must reference ${window.offset}, ${window.limit} and at least one filter token; every filters: entry must be referenced; command[0] can't use a window token.
• A sortable column on the bound table needs sort_param (http).
• page_size and prefetch are not negative.
• With cursor:: http needs both older: and newer:, which are http-only; exec must reference ${window.cursor}, ${window.dir} and ${window.limit}; the offset, total and sort fields are rejected.
• anchor: needs cursor:. A paged window starts where the bound component's start: says.

wrangl on a windowed source returns its first page.

books:
  type: http
  url: https://openlibrary.org/search.json?q=tolkien&fields=title,author_name
  root: docs
  window:
    prefetch: 1
    offset_param: offset
    limit_param: limit
    total_path: numFound
    search_param: q          # "hobbit"         → ?q=hobbit
    filters: {Author: author} # "author:tolkien" → ?author=tolkien

cache¶

Memoises a source's fetches, keyed on its bound parameter values: many calls with the same values within the TTL make one upstream request. For sources fed by on_cursor or join lookups, so sweeping a cursor over rows already visited doesn't refetch them. Streamed events pass through uncached, and errors aren't cached. Distinct from the type: cache operator, which holds one snapshot.

Join lookups get a cache automatically (60s TTL, 100 entries); declare cache: to change the numbers.

user_posts:
  type: http
  parameters: {user_id: {type: int, required: true}}
  url: https://api.example.com/users/${params.user_id}/posts
  cache: {ttl: 5m, size: 200}