tsus evals
An eval harness for the thesis behind TypeScript-Ultra-Strict: when a model writes the code, the .tsus version is correct more often than the plain TypeScript version.
Results live in RESULTS.md.
How it works
Each task in tasks/<id>/ has:
| File | Purpose |
|---|---|
task.json |
title, tags, and whether it is in the pilot set |
prompt.md |
the task statement, shared by both languages |
signature.ts, signature.tsus |
the API the model must implement, per language |
test.mjs |
hidden node:test tests, run against the compiled JS of either language |
reference.ts, reference.tsus |
known-good solutions (used by validate.mjs and the self-test providers) |
mutant.ts, mutant.tsus |
plausible buggy solutions the hidden tests must reject |
The two signatures express the same contract idiomatically. In .ts, absence is undefined or an optional field and failure is a thrown error class with a kind field. In .tsus, absence is Option<T> and failure is Result<T, E> with a { kind } error. lib/harness.mjs normalizes both, so one test file covers both languages and a crash (an unexpected TypeError, say) always counts as a failure.
run.mjs, for every task and language:
- Sends the model a fixed system prompt (
prompts/system-ts.md, orprompts/system-tsus.mdwith LANGUAGE.md inlined) plus the task prompt and signature. - Compiles the reply.
.ts:tscwithstrict: true, lib ES2022, no DOM or Node types..tsus:tsus check --json, thentsus build --json, using the compiler inpackages/cliat the current commit. - If the compiler rejects it, sends the diagnostics back with the previous attempt and asks for a fix, up to
--repairsrounds (default 3). - Runs the hidden tests once, on the first version that compiles. The model never sees test results.
It records, per task and language: whether the first attempt compiled and passed (pass@1), whether any attempt passed (pass after repair), the number of repair rounds, every diagnostic code the compiler raised, failing hidden tests, tokens, cost, and time. Every attempt's code and diagnostics are saved under results/<run>/attempts/.
Running it
From the repository root, after npm install:
node evals/validate.mjs # references pass, mutants fail; no model
node evals/run.mjs --provider selftest # whole pipeline incl. repair loop; no model
node evals/run.mjs --pilot # the 10 pilot tasks, both languages
node evals/run.mjs # all tasks
node evals/report.mjs # rewrite evals/RESULTS.md from evals/results/Useful flags: --tasks a,b, --langs tsus, --model <id> (default claude-sonnet-5-5, or TSUS_EVAL_MODEL), --repairs N, --concurrency N (default 4), --max-cost <usd> (stops starting new model calls past this spend), --label <text>, --tsus-bin <path> (test another compiler build).
Model access
- Claude Code CLI (default when
ANTHROPIC_API_KEYis unset): each call isclaude -p --model <id> --system-prompt <fixed prompt> --tools "" --strict-mcp-config --no-session-persistence --output-format json, run from an empty temp directory so no projectCLAUDE.mdis picked up. Cost comes from the CLI's owntotal_cost_usd. The CLI must be logged in: runclaudeonce interactively and/loginifclaude -p "hi"reports an authentication error. If you start the eval from inside a Claude Code session, the runner strips the variables that tie a childclaudeto the parent session. - Anthropic API (default when
ANTHROPIC_API_KEYis set, or--provider anthropic): needsnpm install --prefix evalsfor@anthropic-ai/sdk. Cost is computed from token usage and the price table inlib/providers.mjs. Add--effort low|medium|highto set the effort level. referenceandselftestcall no model. They exist to prove the harness works.
Keys are never printed or written to results; both paths read credentials themselves.
Rough cost: a task is about 2k to 4k input tokens (the .tsus side carries the primer) and 1k to 3k output tokens per attempt. At Sonnet 5.5 prices ($2 / $10 per million tokens) a full 30-task, two-language run with few repairs is on the order of $1 to $2; the 10-task pilot is well under $1. Use --max-cost to cap it.
Adding a task
Copy tasks/parse-config/ and keep its conventions. The prompt states the contract precisely but does not list the edge cases; the tests probe them. Every test must follow from the prompt. Then run node evals/validate.mjs <id> until the references pass and the mutants fail.
Caveats
These matter more than any number in RESULTS.md.
- The tsus checker is younger than tsc. At the time of the first runs,
tsus checkenforced the five bans and the macro syntax but did no semantic type-checking, so.tsgot more compile-time help than.tsus. The harness always uses the compiler at the current commit and records that commit in each run, so results before and after the typed checker lands are comparable side by side. - The tasks favor the thesis by design. They were picked because null, undefined, exceptions and unhandled cases cause real bugs in them, and the tasks and tests were written with Claude. A neutral task set (or tasks drawn from real bug reports) would be a fairer test.
- Prompt asymmetry. The model knows TypeScript from training and learns tsus from a primer in the system prompt. That probably hurts tsus, but it also means the tsus side gets more instructions.
- One sample per task, no fixed temperature on the CLI path. Re-run to see variance; a difference of a task or two is noise.
- Contracts differ slightly in shape.
.tsthrows typed error classes,.tsusreturnsErr. The tests treat both as the same outcome, but a model can lose a.tstest by throwing a plainError, which has no.tsusanalogue.