Storage and execution — tcm-ts

39. Storage and execution

39.1 Storage

store keeps everything in .tcm/codebase.sqlite (node:sqlite):

  • definitions: canonical IR, the original source text, and the span of every reference in it;
  • dependency edges, signatures, docs;
  • names per branch, and their history: namespace nodes, causals (entries) and each branch's head, plus a log of every name change; patches and todo.

App data lives apart from the codebase in .tcm/data/: store.sqlite (tables), sessions.sqlite, jobs.sqlite. It and secrets are gitignored.

39.2 The generated cache

loader turns each component into .tcm/cache/<hash>.ts:

  • the stored source with the current names spliced into its reference spans;
  • import { h_<hash> as name } from "./<hash>.ts" for dependencies, so modules import each other by hash;
  • members in an order that has what each needs as it's initialized ready first: what it uses then, and, for a function it calls then (const yieldNow = makePrimitive(…)), what that function's body uses, and so on through the functions that calls; a function only read then (const Reference = Service) doesn't run;
  • functions carry their hash at runtime (Symbol.for("tcm.hash")), so code can name them durably (jobs do); a constant holding another definition's function (const constVoid = constUndefined) is that function, which keeps the hash it has;
  • tables carry their row schema (Symbol.for("tcm.schema")).

Node runs these files with no build step. The cache and the .tcm/codebase shim are derived data: both are rebuilt from the database, written only when their content changes. A module is rendered again only when what's in it besides its code changes: the names it's rendered with (its members' and what they refer to) or its members' docs, which its source maps point past; what each was rendered with is in .tcm/cache/rendered.json. So running something whose closure was run before, with the same names, writes nothing.

39.2.1 Pure constants are computed once

A constant whose value is computed (const table = Array.from({ length: 30 }, (_, i) => fib(i))) is evaluated once, the first time a module needing it is written, and its value is kept with its hash: its module holds the value, written as code, and no run computes it again. The original expression stays in the module, uncalled, so the constant's type is what it was (__tcm_kept(() => (…), [0,1,1,2,…])). The constants run in one process, all at once.

A constant is kept only if computing it can't come out differently, or do anything a kept value wouldn't:

  • Its code reaches nothing outside the codebase, through everything it uses: not the clock (Date.now(), new Date()), randomness, the network, files, the environment (process.env, node:os), other processes, and it writes nothing out (console). This is worked out from the code (what it could reach, called or not) and kept with each definition.
  • It uses no npm package's export that isn't pure, as read from the package's code (packages): ms("1d") is kept.
  • Nothing it uses holds state something else could change before it's computed: a top-level let, or a constant holding an object or array. Functions, types, and constants written as a number, string or boolean (or computed ones kept as one) are fine.
  • Its value can be written as code that makes it again: numbers, strings, bigints, arrays, plain objects, Map, Set, Date, RegExp, up to 256 KB. Not functions, class instances, frozen objects, or an object shared with another definition's value (a copy would be a different object).

The rest are computed as written, and that's decided once too. Values are kept per Node version.

Imported code that uses syntax Node can't strip (enums, namespaces, parameter properties) gets a // tcm: transform first line. Processes tcm spawns load .tcm/cache/tcm-transform.js, a module hook that compiles exactly those files; the tcm process registers the same hook in-process.

39.3 npm packages

npm dependencies are installed into the cache under an alias per compatible range (npm__react__19), so code on different ranges coexists, each range at the version the codebase's lock has for it. Installs use --ignore-scripts, and one version is one copy: npm's own copy of a package a peer asks for becomes a link to the alias's (packages). Only the direct package's compatible range is part of a hash, not its transitive dependencies.

39.4 Running code

tcm run <name> [args] materializes the definition's closure, writes a small runner and spawns Node on it. Functions are called with the arguments; the result is awaited and printed if it isn't undefined. What it starts can run the codebase's own commands (its packages' bins), named in <COMMAND>_BIN (TCM_BIN for tcm), as suites do. One the caller set is kept (and said): TCM_BIN=<tcm.js> tcm run <test> runs an end-to-end test against that tcm. One a tcm run further out set (it records them in TCM_COMMANDS) is the build as it was then, and is replaced. tcm run … --on <remote> does the same on another machine (sync).

built from c482a14e