verify

tcmts.dev.verify · #vvu5ty62 · term · runs on the server

uses node:path

Everything a change to tcm must pass, run from its codebase (pnpm verify, which is pnpm tcm run tcmts.dev.verify), with the codebase's own tcm:

nothing needs fixing (tcm todo), every definition renders and re-hashes to itself (tcm verify), and tcm's own libraries (TCM_LIBRARIES) hold everything they use (tcm release check), and their newest releases, added to a fresh codebase, name everything they bring (checkLibraries);
its test definitions pass (tcm test --definitions), run again only where code they test changed;
its docs (tcmts.docs) build into the site, every link and anchor resolving;
the tcm-ts package packed from it installs, works far from the repository (with its own tools: it installs none), and passes the end-to-end tests (tcm test --suites --bin, against it, the tcm that becomes the known-good one), each run only if no run has passed with the code it runs: one a merge ran, say, holds for the package that has that code.

The four run side by side, each saying how it went as it finishes, and each of their programs (and the docs build) in a slot of the machine's (withSlot), shared with every tcm's work: the tests take theirs as tcm test does. When all of that passes, the package is the known-good tcm that pnpm tcm runs (dist/known-good); the one it replaces is dist/known-good-previous, and the one before that is kept with the older builds (dist/builds/, retireBuild).

What passed is kept, like a test's result: run again with nothing changed in the codebase (its history's head, on the same Node and platform), there's nothing to check; with a change, tcm verify and tcm test check only what it reaches, and a package packed the same as the known-good one isn't installed and tried again. One whose code is the known-good's (only its pages changed) is the known-good install with the new pages; one whose dependencies are the known-good's is unpacked beside them rather than installed from scratch. pnpm verify --fresh checks everything again, tests included.

A branch's change (a session's, before it lands) is verified on its branch: TCM_BRANCH=<b> pnpm verify (this code as the branch has it, and every tcm it runs, follow TCM_BRANCH) checks all of that there, but the known-good tcm is main's, so nothing becomes it (and a new hash format migrates nothing): that waits for the change to land on main and main to be verified.

Each branch builds its tcm in a place of its own, as each worktree has its own files: dist/candidates/ (candidateDir: the install, and the package it's from), so verifies of different branches run side by side, and only another of the same branch waits (its verify.lock there). What's shared waits its turn: making a tcm the known-good one (and moving tcm's codebase to a new hash format with it), main's alone, holds dist/verify.lock for just that. A branch's candidate goes once it passes; what verifies leave (a failed one's, kept a day to try again, or one of a branch that's gone) is taken away by the next (cleanCandidates).

import { SLOT_WAIT_LINE, withSlot } from "#codebase/tcmts/cli/budget";
import { retireBuild } from "#codebase/tcmts/cli/builds";
import { formatsOf } from "#codebase/tcmts/cli/migrate";
import { IR_VERSION } from "#codebase/tcmts/core/hash";
import {
  buildSite, cleanCandidates, codebaseDir, repository, shAsync, TCM_LIBRARIES, verifyTurns,
} from "#codebase/tcmts/dev";
import { Store } from "#codebase/tcmts/store";
import crypto from "node:crypto";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";

export async function verify(...args: string[]): Promise<void> {
  const tcm = process.env.TCM_BIN;
  if (!tcm)
    throw new Error("run this with `tcm run`, which gives it the codebase's own tcm (TCM_BIN)");
  const fresh = args.includes("--fresh");
  const branch =
    process.env.TCM_BRANCH && process.env.TCM_BRANCH !== "main"
      ? process.env.TCM_BRANCH
      : undefined;
  const codebase = codebaseDir();
  const repo = repository();
  const started = Date.now();
  const seconds = (since: number) => `${((Date.now() - since) / 1000).toFixed(1)}s`;
  const dist = path.join(repo, "dist");
  const good = path.join(dist, "known-good");
  const goodBin = path.join(good, "node_modules", "tcm-ts", "bin", "tcm.js");
  const { dir: mine, promote } = await verifyTurns(dist, branch);
  // what the known-good was verified as: the codebase's history entry, the runtime, and the package
  interface Verified {
    head: string;
    node: string;
    platform: string;
    packed: string;
    /** the package but its pages (Markdown): its code, and what it installs */
    code: string;
  }
  const record = path.join(good, "verified.json");
  const was =
    fs.existsSync(record) && fs.existsSync(goodBin)
      ? (JSON.parse(fs.readFileSync(record, "utf8")) as Verified)
      : undefined;
  const headNow = () => {
    const store = new Store(path.join(codebase, ".tcm", "codebase.sqlite"));
    try {
      return store.head();
    } finally {
      store.close();
    }
  };
  const head = headNow();
  {
    const store = new Store(path.join(codebase, ".tcm", "codebase.sqlite"));
    let branches: string[];
    try {
      branches = [...store.branches(), "main"];
    } finally {
      store.close();
    }
    const removed = cleanCandidates(dist, branches);
    if (removed.length)
      console.log(`took away what verifies left: ${removed.map((r) => `dist/${r}`).join(", ")}`);
  }
  const same = !fresh && was?.node === process.version && was.platform === process.platform;
  if (same && was.head === head) {
    console.log(
      `nothing changed since the known-good tcm was verified (${fs.readFileSync(path.join(good, "VERIFIED"), "utf8").trim()}): nothing to check`,
    );
    console.log(`all passed in ${seconds(started)}`);
    return;
  }

  let failed = false;
  // each stage runs at once, the others beside it, and says how it went when it's done
  const step = async (
    name: string,
    show: RegExp,
    run: () => Promise<{ ok: boolean; out: string }>,
  ) => {
    const t0 = Date.now();
    // (one that throws failed: it says why, and what follows a failure still happens)
    const r = await run().catch((e: unknown) => ({
      ok: false,
      out: `error: ${(e as Error)?.stack ?? e}`,
    }));
    const lines = r.out.split("\n");
    const why = lines.filter((l) => /^\s*✗ /.test(l) || /error|failed/i.test(l));
    const report = [
      `${name}: ${r.ok ? "ok" : "FAILED"} (${seconds(t0)})`,
      ...lines.filter((l) => show.test(l)).map((l) => `  ${l.trim()}`),
      // (a failure that doesn't say so in words: how its output ended)
      ...(r.ok
        ? []
        : (why.length ? why.slice(0, 30) : lines.filter((l) => l.trim()).slice(-15)).map(
            (l) => `  ${l.trim()}`,
          )),
    ];
    console.log(report.join("\n"));
    failed ||= !r.ok;
    return r.ok;
  };
  // a new hash format: this code (the codebase's, the new tcm's, as TCM_BIN runs it) can't open tcm's codebase until
  // it's migrated, so until then the codebase is checked with the known-good tcm, and what opens it in this process
  // (the docs site) waits for the migration
  const sqlite = path.join(codebase, ".tcm", "codebase.sqlite");
  const formatChange = formatsOf(sqlite).some((v) => v !== IR_VERSION);
  // a test run says how it's going as it goes: each failure at once, and every 30 seconds how many have finished
  const progress = (label: string) => {
    let passed = 0;
    let failed = 0;
    let last = "";
    const timer = setInterval(() => {
      const now = `${passed + failed} finished, ${failed} failed`;
      if (passed + failed && now !== last) console.log(`  ${label}: ${now} (${seconds(started)})`);
      last = now;
    }, 30_000);
    return {
      onLine: (l: string) => {
        if (/^\s*✓ /.test(l)) passed++;
        else if (/^\s*✗ /.test(l)) {
          failed++;
          console.log(`  ${label}: ${l.trim()}`);
        }
        // (and when it waits for the machine's suite slots, and how long it took: not how the wait goes meanwhile)
        else if (SLOT_WAIT_LINE.test(l) && !/^\s*still /.test(l))
          console.log(`  ${label}: ${l.trim()}`);
      },
      stop: () => clearInterval(timer),
    };
  };
  const run = async (...args: string[]) => {
    const p =
      args[0] === "test" ? progress(args.includes("--bin") ? "package" : "tests") : undefined;
    try {
      return await shAsync(
        process.execPath,
        [formatChange ? goodBin : tcm, ...args],
        codebase,
        {},
        p?.onLine,
      );
    } finally {
      p?.stop();
    }
  };
  const again = fresh ? ["--fresh"] : [];
  // a program of a stage's (or its work in this process) in a slot of the machine's, what it starts in that slot too
  const what = `verify of ${codebase}${branch ? ` on ${branch}` : ""}`;
  const inSlot = <T>(label: string, work: (env: Record<string, string>) => Promise<T>) =>
    withSlot(what, work, { say: (l) => console.log(`  ${label}: ${l}`) });
  const runInSlot = (label: string, ...args: string[]) =>
    inSlot(label, (env) =>
      shAsync(process.execPath, [formatChange ? goodBin : tcm, ...args], codebase, env),
    );

  const checked = step("codebase", /nothing to do|verified/, async () => {
    const [todo, identity, libraries] = await Promise.all([
      runInSlot("codebase", "todo"),
      runInSlot("codebase", "verify", ...again),
      // (tcm's own libraries hold everything they use, as their releases must)
      Promise.all([
        ...TCM_LIBRARIES.map((name) => runInSlot("codebase", "release", "check", name)),
        // (and their newest releases, added to a fresh codebase, name everything they bring)
        runInSlot("codebase", "run", "tcmts.dev.checkLibraries"),
      ]),
    ]);
    return {
      ok: todo.ok && /nothing to do/.test(todo.out) && identity.ok && libraries.every((l) => l.ok),
      out: [todo.out, identity.out, ...libraries.filter((l) => !l.ok).map((l) => l.out)].join("\n"),
    };
  });
  // (the end-to-end tests run once, against the package, below)
  const tested = step("tests", /^\d+ passed/, () =>
    run("test", "tcmts", "--definitions", ...again),
  );
  const candidate = path.join(mine, "install");
  const out = path.join(mine, "package");
  // a package is its files: one packed the same as the known-good's is that package (`code`: all but its pages)
  const fingerprint = (dir: string, code = false) => {
    const h = crypto.createHash("sha256");
    const walk = (d: string) => {
      for (const e of fs
        .readdirSync(d, { withFileTypes: true })
        .sort((a, b) => (a.name < b.name ? -1 : 1))) {
        const p = path.join(d, e.name);
        if (e.isDirectory()) walk(p);
        else if (!(code && e.name.endsWith(".md")))
          h.update(`${path.relative(dir, p)}\0`)
            .update(fs.readFileSync(p))
            .update("\0");
      }
    };
    walk(dir);
    return h.digest("hex").slice(0, 32);
  };
  const filesIn = (dir: string, keep: (rel: string) => boolean): string[] =>
    fs
      .readdirSync(dir, { recursive: true, withFileTypes: true })
      .filter((e) => e.isFile())
      .map((e) => path.relative(dir, path.join(e.parentPath, e.name)))
      .filter(keep);
  // (a copy of the known-good install shares its files where the filesystem can: APFS clones them. Main's verify
  // may be making another tcm the known-good one as this copies it, renaming it away: then it's copied again)
  const cloneGood = async () => {
    for (let tries = 1; ; tries++) {
      try {
        fs.cpSync(good, candidate, { recursive: true, mode: fs.constants.COPYFILE_FICLONE });
        return;
      } catch (e) {
        fs.rmSync(candidate, { recursive: true, force: true });
        if (tries === 3) throw e;
        await new Promise((r) => setTimeout(r, 2000));
      }
    }
  };
  // the tarball packed now, kept beside the known-good install when that's what was packed (main's alone)
  let goodTarball: (() => void) | undefined;
  let packedAs = "";
  let codeAs = "";
  let installed = false;
  const packaged = step(
    "package",
    /^(installed|unchanged|pages only|far away|\d+ passed|suites:)/,
    async () => {
      const pack = await runInSlot("package", "pack", "tcm-ts", "--out", out);
      if (!pack.ok) return pack;
      packedAs = fingerprint(out);
      codeAs = fingerprint(out, true);
      // the tarball npm would publish, packed now: installed from (unless only pages changed), and kept beside the
      // install as what's deployed (`pnpm server:deploy` ships it)
      const tarballs = fs.mkdtempSync(path.join(os.tmpdir(), "tcm pack "));
      const npmPack = await inSlot("package", (env) =>
        shAsync("npm", ["pack", "--silent", "--pack-destination", tarballs], out, env),
      );
      if (!npmPack.ok) return npmPack;
      const tgz = path.join(
        tarballs,
        fs.readdirSync(tarballs).find((f) => f.endsWith(".tgz"))!,
      );
      const keepTarball = (dir = candidate) => {
        for (const f of fs.readdirSync(dir)) if (f.endsWith(".tgz")) fs.rmSync(path.join(dir, f));
        fs.copyFileSync(tgz, path.join(dir, path.basename(tgz)));
      };
      if (same && was.packed === packedAs) {
        if (!branch) goodTarball = () => keepTarball(good);
        const tests = await run("test", "tcmts", "--suites", "--bin", `tcm=${goodBin}`, ...again);
        return {
          ok: tests.ok,
          out: `unchanged: packed the same as the known-good tcm, which is installed already\n${tests.out}`,
        };
      }
      fs.rmSync(candidate, { recursive: true, force: true });
      const bin = path.join(candidate, "node_modules", "tcm-ts", "bin", "tcm.js");
      const installedPkg = path.join(candidate, "node_modules", "tcm-ts");
      if (same && was.code === codeAs) {
        // only its pages changed: the known-good install, with the pages packed now
        await cloneGood();
        for (const rel of filesIn(installedPkg, (r) => r.endsWith(".md")))
          fs.rmSync(path.join(installedPkg, rel));
        for (const rel of filesIn(out, (r) => r.endsWith(".md"))) {
          fs.mkdirSync(path.dirname(path.join(installedPkg, rel)), { recursive: true });
          fs.copyFileSync(path.join(out, rel), path.join(installedPkg, rel));
        }
        keepTarball();
        const tests = await run("test", "tcmts", "--suites", "--bin", `tcm=${bin}`, ...again);
        installed = tests.ok;
        return {
          ok: tests.ok,
          out: `pages only: the known-good tcm's code, with the pages packed now\n${tests.out}`,
        };
      }
      const dependencies = (dir: string) =>
        fs.existsSync(path.join(dir, "package.json"))
          ? JSON.stringify(
              (
                JSON.parse(fs.readFileSync(path.join(dir, "package.json"), "utf8")) as {
                  dependencies?: unknown;
                }
              ).dependencies ?? {},
            )
          : undefined;
      let log = "";
      if (
        fs.existsSync(goodBin) &&
        dependencies(path.join(good, "node_modules", "tcm-ts")) === dependencies(out)
      ) {
        // with the known-good's dependencies (the same, at the same versions): unpacked beside them
        await cloneGood();
        fs.rmSync(installedPkg, { recursive: true, force: true });
        fs.mkdirSync(installedPkg, { recursive: true });
        const unpack = await shAsync(
          "tar",
          ["-xzf", tgz, "-C", installedPkg, "--strip-components", "1"],
          installedPkg,
        );
        if (!unpack.ok) return unpack;
        log += `installed tcm-ts, packed from the codebase, in ${path.relative(repo, candidate)} (its dependencies are the known-good's)\n`;
      } else {
        fs.mkdirSync(candidate, { recursive: true });
        const installing = await inSlot("package", async (env) => {
          for (const s of [
            () => shAsync("npm", ["init", "-y"], candidate, env),
            () =>
              shAsync(
                "npm",
                ["install", tgz, "--no-audit", "--no-fund", "--loglevel=error"],
                candidate,
                env,
              ),
          ]) {
            const r = await s();
            log += r.out;
            if (!r.ok) return false;
          }
          return true;
        });
        if (!installing) return { ok: false, out: log };
        log += `installed tcm-ts, packed from the codebase, in ${path.relative(repo, candidate)}\n`;
      }
      keepTarball();
      // far from the repository, nothing it needs is found by walking up to its node_modules; beside that, the tests
      // against it
      const farAway = () =>
        inSlot("package", async (slot) => {
          const far = fs.mkdtempSync(path.join(os.tmpdir(), "tcm far "));
          // (its own codebase: not the one this verifies, which TCM_CODEBASE names, nor on a branch it has; in this slot)
          const env = { ...slot, TCM_CODEBASE: "", TCM_BRANCH: "" };
          const init = await shAsync(process.execPath, [bin, "init"], far, env);
          fs.writeFileSync(
            path.join(far, "scratch.ts"),
            "export const platform = () => process.platform;\n",
          );
          const add = init.ok ? await shAsync(process.execPath, [bin, "add"], far, env) : init;
          if (!(init.ok && add.ok)) return { ok: false, out: `${init.out}${add.out}` };
          // its tools came with it (`shippedPackage`): it linted and formatted that with nothing installed; and a test
          // there runs with its Vitest, which the test imports
          fs.writeFileSync(
            path.join(far, "scratch.ts"),
            'import { expect } from "vitest";\nimport { platform } from "#codebase";\nexport const testPlatform = () => expect(platform()).toBe(process.platform);\n',
          );
          const addTest = await shAsync(process.execPath, [bin, "add"], far, env);
          const test = addTest.ok
            ? await shAsync(process.execPath, [bin, "test"], far, env)
            : addTest;
          if (!test.ok || !/1 passed/.test(test.out))
            return { ok: false, out: `${addTest.out}${test.out}` };
          const all = `${add.out}${addTest.out}${test.out}`;
          const installed = /installing (oxlint|oxfmt|vitest)/.exec(all);
          if (installed)
            return {
              ok: false,
              out: `far away: the packed tcm installed ${installed[1]}, which it should have with it\n${all}`,
            };
          return {
            ok: true,
            out: `far away: the packed tcm makes a codebase in ${far} and checks code against @types/node there, with its own tools and Vitest\n`,
          };
        });
      const [farOut, tests] = await Promise.all([
        farAway(),
        run("test", "tcmts", "--suites", "--bin", `tcm=${bin}`, ...again),
      ]);
      installed = farOut.ok && tests.ok;
      return { ok: installed, out: `${log}${farOut.out}${tests.out}` };
    },
  );
  // (it builds the site in this process: after the others' programs are running. The docs app's build is in its
  // slot: this process waits for it to end, so it couldn't give back a slot that build would be waiting for)
  const buildDocs = () =>
    step("docs", /pages|link/, async () => {
      const docs = await inSlot("docs", (env) =>
        buildSite(fs.mkdtempSync(path.join(os.tmpdir(), "tcm site ")), { env }),
      );
      return {
        ok: !docs.problems.length,
        out: docs.problems.length
          ? docs.problems.join("\n")
          : `${docs.pages} pages, every link and anchor resolving`,
      };
    });
  await Promise.all([checked, tested, packaged, ...(formatChange ? [] : [buildDocs()])]);

  // making the known-good tcm, which every checkout runs, is main's alone, one verify at a time (dist/verify.lock):
  // the builds before it, each in its own place, ran side by side
  const promoting = !failed && !branch ? await promote() : undefined;
  // the tcm that runs on tcm's codebase from now on: what was just verified
  let verifiedHead = head;
  // (whether this run migrated tcm's codebase: only then is there anything to put back)
  let migrated = false;
  if (!failed && installed && formatChange && !branch) {
    // tcm's codebase moves to the new format with the tcm that reads it, or neither moves. The candidate migrates it
    // (keeping it as it was beside it, and using the migrated one only if it reads back as itself); this keeps it
    // too, to put back if what's checked after (the site, from the migrated codebase) fails. The known-good tcm it
    // replaces is kept as well (dist/known-good-previous), so going back is a rename or two
    fs.copyFileSync(sqlite, path.join(dist, "codebase-before-migrate.sqlite"));
    migrated = true;
    const bin = path.join(candidate, "node_modules", "tcm-ts", "bin", "tcm.js");
    const t0 = Date.now();
    const m = await shAsync(process.execPath, [bin, "migrate"], codebase);
    console.log(
      `migrate: ${m.ok ? "ok" : "FAILED"} (${seconds(t0)})\n${m.out
        .trim()
        .split("\n")
        .map((l) => `  ${l}`)
        .join("\n")}`,
    );
    failed ||= !m.ok;
    if (!failed) await buildDocs();
    verifiedHead = headNow();
  }
  const before = path.join(dist, "codebase-before-migrate.sqlite");
  if (failed && migrated) {
    for (const f of [`${sqlite}-wal`, `${sqlite}-shm`]) fs.rmSync(f, { force: true });
    fs.copyFileSync(before, sqlite);
    for (const f of fs.readdirSync(path.join(codebase, ".tcm")))
      if (/^codebase\.v\d+(\.\d+)?\.sqlite$/.test(f)) fs.rmSync(path.join(codebase, ".tcm", f));
    console.log(
      "tcm's codebase is back as it was, in its old hash format: the known-good tcm reads it",
    );
  }
  if (!failed && !branch) {
    goodTarball?.();
    if (installed) {
      // the known-good tcm it replaces is the previous one; the one before that is kept among the older builds
      // (dist/builds/<id>, the newest few), to run again (`tcm --build <name>`, `tcm builds`)
      const previous = path.join(dist, "known-good-previous");
      retireBuild(dist, previous);
      if (fs.existsSync(good)) fs.renameSync(good, previous);
      fs.renameSync(candidate, good);
    }
    // (what it was made from: the codebase's history entry when this started; if the codebase changed while it
    // ran, what passed may not be what's there now, so nothing is kept for next time)
    fs.writeFileSync(
      path.join(good, "VERIFIED"),
      `#${verifiedHead.slice(0, 8)}, verified ${new Date().toISOString().slice(0, 16).replace("T", " ")}\n`,
    );
    if (headNow() === verifiedHead) {
      const now: Verified = {
        head: verifiedHead,
        node: process.version,
        platform: process.platform,
        packed: packedAs,
        code: codeAs,
      };
      fs.writeFileSync(record, JSON.stringify(now, null, 2) + "\n");
    } else fs.rmSync(record, { force: true });
    console.log(
      `known-good tcm: ${fs.readFileSync(path.join(good, "VERIFIED"), "utf8").trim()}; pnpm tcm and the MCP server run it`,
    );
  }
  promoting?.();
  // (a branch's candidate never becomes the known-good tcm: once it's passed, nothing's kept)
  if (!failed && branch)
    for (const d of [candidate, out]) fs.rmSync(d, { recursive: true, force: true });
  if (!failed && branch)
    console.log(
      `verified ${branch} (#${head.slice(0, 8)}): the known-good tcm is still main's; this becomes it once it lands on main and main is verified`,
    );
  console.log(`${failed ? "FAILED" : "all passed"} in ${seconds(started)}`);
  process.exitCode = failed ? 1 : 0;
}