server-library

tcmts.design.server-library · #s356gx4b · md file

#tcm/server as a library

Built (all four units below have landed: #tcm/server is retired, and apps mount the server library tcm ships). Kept as the record of why it's this way; "Today" is how it was before. The API apps write their server code against (table, transaction, enqueue, every, sessions, cookies, the request) becomes a tcm library, server: versioned, released, read and added like any other. The runtime that keeps the data, runs the jobs and answers the requests stays in tcm, behind a host interface. It uses the mechanism React's library uses: the manifest's host protocol version and roles.

Today

• #tcm/server isn't code in the codebase. An import of it is an external module, tcm:server, hashed by the name of what's imported (ext tcm:server table), as an npm import is by its package.
• The runtime is one function's source: tcmts.cli.server_api.createServerApi, written as text into .tcm/cache/tcm-server.js with its declarations (tcm-server.d.ts), and everything it uses passed in as deps (AsyncLocalStorage, SQLite, the wire codec, the schema checker), so it stays self-contained.
• tcm recognises the API by that module name:
  • tableOf (tcmts.core.analyze) finds const users = table<User>("users") by the import's specifier (#tcm/server), and keeps the row type's schema (and its hash) with the definition;
  • schedules are definitions whose spans hold ext tcm:server every (appSchedules, startBackground, isSchedule);
  • placement calls any value use of tcm:server server-only ("uses #tcm/server (table)");
  • the runtime tells a deliberate error from others by instanceof the module's HttpError.

What a library gives

• The API has versions, under semver with the release checks, and a changelog, apart from tcm's own.
• It's read like any code: its definitions in the forge and the editor, its docs as doc comments, tcm view and tcm find across it.
• An app pins the API it was written against, and tcm's runtime can change underneath without moving it.
• Others can write their own against the same host interface: a test double, an in-memory variant, an adapter to another database.

The split

Three layers, of which only the middle one is new:

• What moves into the library is the API's own logic: the Table class (row checks against the schema, migration steps, indexes, find's query), sessions (their table in sessions.sqlite, through the host's handle), cookie formatting, rate-limit windows, HttpError, and the schedule a job is given (every returns { spec, fn }, marked). What needs the process stays in the host: one AsyncLocalStorage for the call, one SQLite connection per file, the lock every table call and transaction takes on the store, the job queue (jobs queued in a transaction wait for its commit), the worker that runs them by hash, and the scheduler.
• createServerApi becomes createServerHost: smaller, under the same self-contained-source rule (everything through deps), since it's still written out as text and run in the app's process. The library has no such rule: it's ordinary codebase code, compiled and bundled like any.
• One host per process, any number of API copies. Two libraries each holding a version of server both call the one host, so there's one data directory, one job queue, one store lock and one request context, as now. (Rate-limit windows are each copy's own.)
• Deliberate errors are known by a mark, not a class: the library's HttpError sets Symbol.for("tcm.httpError"), and the runtime sends an error with that mark to the caller as it sends an HttpError now. Any copy's (or anyone's) HttpError is one.
• One library, not data, jobs and http apart: they share the host's call context and a transaction spans tables and jobs, so they change together.

The host interface, version 1

What #tcm/host exports, for the library (and anyone writing one):

• call(): the current call (the request, its cookies, the Set-Cookie headers so far, whether it's secure, the cookie path, the data directory), or undefined outside one; dataDir().
• database(file): the SQLite handle for store, sessions (in the data directory, in memory in tests).
• exclusive(key, fn): run fn holding key (the store's handle), as every table call does; transaction(fn); inTransaction().
• enqueue(runAt, fn, args, opts): queue a job (a codebase function, found by hash); runJobs(opts) for tests; checkSchedule(spec).
• encode, decode (the wire codec), checkValue (the schema checker), randomId(bytes).
• withCall(call, opts, fn): run a call in its context (the runtime's, and withRequest's).
• For the runtime only: pendingCookies, startWorker, setSchedules.

The protocol version

The host and the API meet at that interface, which will change. Each side says which version it speaks:

• The library declares it in its manifest: { name: "server", npm: false, host: 1, roles: … }. A release whose host calls changed incompatibly bumps it (and is a major release, since what it can run on changed).
• tcm declares what it serves: HOST_PROTOCOLS (tcmts.cli.roles). It's one sequence for all of tcm's runtime: version 1 is the page's globals React's library reads and #tcm/host's exports above. A new runtime keeps serving old versions as long as it can.
• Checked before anything runs: the build refuses an app using a library that speaks a version this tcm doesn't serve (hostRefusal, from React's R1). There's no second check at run time: an import of an export #tcm/host doesn't have fails as the module loads.

Finding tables and schedules: by identity, not by import path

The library can be mounted anywhere (app.lib.server, vendor.server2), so tcm can't recognise its table by the path it's imported from. It recognises the definition itself:

• The library's manifest names its roles: roles: { table: "table", every: "every" }, relative names as bin and exports are. A mounted library keeps its manifest, so tcm reads, for every mount, which hash plays which role (rolesIn, as React's Stylesheet is found).
• tableOf asks whether the callee's hash plays table: analysis already resolves a #codebase/… import to a hash; the codebase's roles come to it through its host, and the row type's schema is taken from the call as now.
• Code analyzed without names keeps what it was. A table's schema is worked out where the code is analyzed, and some analysis has no names to read roles from: a dependent re-derived after what it uses changed (in a checker process, from rendered modules), and code received from another codebase (a push, a pull, tcm lib add), whose library may not be named yet. There, a definition is a table if its record was one (the re-derived definition's earlier version, or the sender's record), and its schema is still worked out here, from its code, never taken from the sender.
• Schedules are definitions whose spans reach a hash playing every, in place of a span on ext tcm:server every. They're found when an app is built or started, where the names are.
• A definition is a table wherever it's mounted, and whichever release it comes from, because the role is read from the release it came with. A table's rows are its name's and its row type's, not the table function's, so an app that moves onto the library keeps its data.

transaction and enqueue need no role: nothing in the analysis or build treats them specially.

Three alternatives, for the record:

• A brand in the type: the library's Table<T> carries a marker property, and analysis takes any call whose result has it as a table. It needs no names anywhere, but it's a second way for a library to say what's special, beside the manifest's roles.
• A list of known hashes: tcm ships the hash of every released table. Simple, but tcm must know a release before an app can use it, and a library of one's own can't play the role.
• Marker definitions: the API wraps each role in a marker call (role("table", impl)), which analysis spots. It needs no manifest, but every role's hash depends on the marker, and the marker is one more thing to keep stable.

Placement

• The host is server-only by being an external module, as #tcm/server is now: placement treats a value use of tcm:host as server-only ("uses #tcm/host (database)"). The library's functions use it, so they're server-only through their dependencies, and so is any app code calling them ("uses table, which uses #tcm/host").
• What's pure in the library runs anywhere: HttpError, the types. A browser component can import HttpError, as it can't today, since everything in #tcm/server is server-only.
• Server functions the browser calls by hash are found as now, by placement, with nothing new.

Getting it, and what happens to #tcm/server

• tcm ships the library, as React's R3 ships react: the manifest's libraries lists it, the package carries its newest release, and tcm lib add tcm server mounts it at <root>.lib.server. Upgrading tcm and running tcm lib upgrade server moves an app to the newer release, when the app is ready to.
• #tcm/server goes: an import of it is an error saying what to do (tcm lib add tcm server, then import from #codebase/<root>/lib/server). tcm's team is its only user, so there's no codemod or shim: tcm's apps (the forge, the hub), the examples and the end-to-end fixtures move in the same change, and re-hash once.
• The data format doesn't change: tables, jobs and sessions are stored as now, by the host and the library.
• No hash-format change: a definition importing the library re-hashes as any definition whose dependency changed; ext tcm:server stays something the IR can hold, and nothing new makes it.

How to get there

Each a unit of its own, landed apart:

1. This page.
2. Roles table and every: tableOf, schedules and the build find them by hash, through rolesIn, as well as by #tcm/server while it lasts. Tested with a library of a test's own whose table wraps #tcm/server's. No change for apps.
3. The host and the library: createServerHost as #tcm/host; tcmts.server (the deploy files there now move to tcmts.deploy) holding the API on it, with host: 1 and its roles; the runtime knows deliberate errors by their mark. #tcm/server is rebuilt on the host too, so apps on either share one host while both exist (the API is in two places for this one unit). Released as server@0.1.0 on main once landed.
4. Moving onto it: tcm ships server beside react (R3's mechanism); the forge, the hub, the examples and the fixtures use it; #tcm/server and its generated module are retired, and docs (server-api, server-reference, data, jobs, libraries, limitations, files) say how to get it.

Announced later, by hand, as @tcm.codes/server.