libraries

tcmts.design.libraries · #koyd2zhi · md file

Libraries from another codebase: a registry with no registry

A design, partly built. How a codebase depends on part of another, as Unison's lib does, and how people would find libraries without anyone running a registry. Built so far (tcmts.docs.libraries): step 1 of the path below, closure fetch, lib mounting with tcm lib add / rm, read-only libraries, and tcm lib upgrade as one patch; step 2, several sources per library (tcm lib sources); and from step 3, what a library reaches, said when it's added and upgraded. From step 4, discovery, announcing a release as a codes.tcm.release record (tcm release --announce, tcmts.docs.releases), and adding one by its announcement (tcm lib add @alice.dev/geo@1.2.0, upgraded by the announcer's DID; the records' signatures aren't checked yet); the hub (tcm hub, tcmts.hub): its ingest (Jetstream v2, codes.tcm.*; with an API key, its archive first), its mirror job (each release checked against its author's signed record with com.atproto.sync.getRecord, its code mounted by DID) and its pages; and what people say (tcm lib attest, tcm lib trust): reviews, advisories and yanks, a collection each, shown by tcm lib and the hub; and signing in to the hub (AT Protocol OAuth from the visitor's browser), with "People I follow", "Libraries I use" and stars (codes.tcm.star, one per person per library); and tcm lib search, asking a hub.

What a registry does, and what content addressing already gives

A registry like npm does four jobs. Content addressing pulls them apart:

Storage and integrity need no trust. The only claim left is "which hash is geo 1.2", and that's a signed record, not a registry. Nobody has to publish: a release already names a point in a codebase's history, and any codebase tcm serve serves is already a source.

What a library is

• A namespace hash, usually a release's. It's pinned by that hash, not by a name or a URL.
• Sources are only places to fetch from: the author's server, a mirror, a colleague's codebase, a hub. Whichever answers is fine, since what arrives is checked.
• Mounted read-only under lib.<name>, as in Unison, in the project's root namespace. Code uses app.lib.geo.area like any definition; editing it fails with "copy it into your own namespace". What's a library isn't told by the name lib (tcm's docs use lib. for people's own code): a JSON file beside the mounts, <namespace>.libraries, records each one's namespace hash, release and source, so it's history like any file.
• Its own libraries come along by hash (lib.geo.lib.shapes). Two libraries using the same version of a third share its hashes, so it's stored once; different versions are different hashes, side by side.

tcm lib add https://alice.dev geo@1.2.0     # by release, from a source
tcm lib add origin #k3x9… --as geo           # by hash, from any source that has it
tcm lib add @alice.dev/geo@1.2.0                # by announced release (see discovery)
tcm lib sources geo add https://mirror.example   # another place to fetch it
tcm lib upgrade geo                              # to the newest release, as one patch

What tcm has, and what this adds

The pieces

1. Closure fetch. Fetch what a namespace hash reaches (names, definitions, libraries), not the history behind it. The missing, get and put routes already work by hash; this adds a walk from a namespace.
2. lib namespaces. Mounted read-only, named as the library names them, shown as lib.geo.area. Apps and tests treat them like any code: cached by hash, so a library's tests that passed anywhere don't run again.
3. Upgrades as patches. tcm lib upgrade geo maps each definition of the old version to its new one, by name and kind, and applies the mapping as one update. What still typechecks follows; what doesn't goes to todo, on the old version. The semver report comes first ("2 breaking changes: area takes a Shape now").
4. Trust signals when adding. Effects, computed from the code ("geo reaches nothing outside itself; geo.fetchTiles reaches the network"); attestations from others; the licence, read from its manifest.
5. Discovery over AT Protocol, below.

Discovery

Records

Authors announce releases as AT Protocol records in their own repositories. The repository signs them, so an author's identity is their DID, and pinning a library's DID the first time it's added replaces pinning keys.

• codes.tcm.release (built): package, version, namespace hash, history entry, hash format, the notes, sources (server URLs) and when it was made, keyed <package>:<version>. Description and licence could join it.
• codes.tcm.review, codes.tcm.advisory, codes.tcm.yank (built), in anyone's repository, about code by its namespace hash: "reviewed #x", "#x is vulnerable (high; fixed in 1.2.1)", and, from its author alone, "don't use #x".
• Moderation (spam, malware) uses AT Protocol labelers a hub subscribes to, so no hub is the authority.

Announcing is optional: tcm release --announce writes the record. Unannounced releases still work by URL or hash.

The hub

A hub is a tcm app anyone can run. It ingests the records, mirrors the code they name, and shows it.

• Ingest from Jetstream v2, filtered to codes.tcm.*: a scheduled job (every("1m", readJetstream)) reads from a stored cursor until the stream goes quiet, and writes rows and the new cursor in one transaction. every runs each slot once however many servers share the codebase, so there's one reader. Jetstream's websocket replays 36 hours; its full history is an archive that takes an API key (JETSTREAM_API_KEY), so a new hub with a key replays the archive and then the live stream (the SDK's replay), and one without starts 36 hours back.

• Tables:

  export const releases = table<HubRelease>("releases", { unique: ["uri"], index: ["did", "package", "namespace", "state"] });
  export const uses = table<Use>("uses", { index: ["uri", "namespace"] });   // the libraries each release holds, by hash
  export const people = table<Person>("people", { unique: ["did"] });
  export const cursors = table<Cursor>("cursors", { unique: ["feed"] });          // Jetstream's seq
  export const attestations = table<HubAttestation>("attestations", { unique: ["uri"], index: ["namespace", "did", "state"] });
  export const stars = table<HubStar>("stars", { unique: ["uri"], index: ["author", "package", "did", "state"] });

  A release's state is pending, mirrored or failed (with the reason). A deleted record marks it withdrawn; its code stays, since others may use it.

• Mirror as a job. A new release enqueues mirror(uri) in the same transaction: fetched from its sources with closure fetch into the hub's codebase, mounted read-only by DID (plc_<id>.geo.v1_3_0: a handle can change hands, a DID can't), re-hashed, so no source can change it. The libraries it holds go in uses, so dependents are exact.

• Check authorship ourselves. Jetstream sends records without their repository's signed proof. Before a release shows as its author's, mirror fetches it with com.atproto.sync.getRecord and checks the signed commit against the DID's key; one that fails stays failed. (If Jetstream v2 turns out to verify, this step goes.)

• What it computes, since nobody has to claim it: dependents, exactly, by hash; effects; the semver report between versions; type search across everything mirrored.

• It runs no code, as the forge doesn't.

Every hub is also a source: what it mirrors, it serves.

The site

The forge (tcmts.share) renders a codebase it's given, so the hub shows mirrored code with the forge's pages (CodePage, NamespaceView, DefinitionView, ReleasesPage, SearchPage) and adds:

• search by name, docs or type, across every library;
• a library page: versions, dependents, effects, attestations, licence, and the tcm lib add line to copy;
• an author page: their handle and DID, and their libraries;
• what changed between two versions, from the semver report.

The userinput example's network code (polite fetching, OAuth sign-in) is where reading the network and announcing start from.

How it differs

• No publishing step, and no single point of failure or capture: a source or a hub going down, or being compromised, can't change code. At worst it stops serving it.
• Nothing to resolve transitively. Hashes are exact; npm packages libraries use come with their range and lock.
• Upgrades are checked before they happen, and what breaks is left on a todo list, not in a broken build.
• Unison Share has the content addressing but one host. This keeps the addressing and drops the host.

Open questions

• Granularity. Add a whole namespace, or only what code reaches? Probably the whole namespace, bundling only what's reached, as bundles already do.
• Telling two geos apart in docs and errors: by handle, or DID.
• npm inside libraries. Two libraries locking different patches of one range: probably the newest, as today, noted in the upgrade report.
• Yanks and advisories: the author's DID, attesters you trust, or both.
• Private libraries: a token on a source, as today; announced only by hash with no sources, or not at all.
• "latest": a moving pointer, or only explicit versions? Leaning explicit, with tcm lib upgrade showing what's newer.
• What a hub mirrors: only announced releases, with a size limit?
• Test results already travel with sync, so a library page could show "tests passed at #x".
• Garbage: old versions nothing uses stay in history.

A path there

1. Local first: closure fetch, lib mounting and upgrades as patches, from another codebase or a tcm serve remote, by release or hash. Useful alone: sharing between your own codebases, and split repositories that stay one model. It needs no trust layer, since a hash is its own proof.
2. Sources and mirrors: several per library, fetched from whichever answers.
3. Trust signals: effects summaries when adding, attestations.
4. Discovery: codes.tcm.* records (under tcm.codes), tcm release --announce, tcm lib add @handle/geo, the hub and its site.