Rebasely Beta: data may be purged

justintypescript-ultra-strict Public

Markdown · 67 lines · 5.8 KiB · at 83c2f03

# tsus evalsAn eval harness for the thesis behind TypeScript-Ultra-Strict: when a model writes the code, the `.tsus` version is correct more often than the plain TypeScript version.Results live in [RESULTS.md](RESULTS.md).## How it worksEach task in `tasks/<id>/` has:| File | Purpose ||---|---|| `task.json` | title, tags, and whether it is in the pilot set || `prompt.md` | the task statement, shared by both languages || `signature.ts`, `signature.tsus` | the API the model must implement, per language || `test.mjs` | hidden `node:test` tests, run against the compiled JS of either language || `reference.ts`, `reference.tsus` | known-good solutions (used by `validate.mjs` and the self-test providers) || `mutant.ts`, `mutant.tsus` | plausible buggy solutions the hidden tests must reject |The two signatures express the same contract idiomatically. In `.ts`, absence is `undefined` or an optional field and failure is a thrown error class with a `kind` field. In `.tsus`, absence is `Option<T>` and failure is `Result<T, E>` with a `{ kind }` error. `lib/harness.mjs` normalizes both, so one test file covers both languages and a crash (an unexpected `TypeError`, say) always counts as a failure.`run.mjs`, for every task and language:1. Sends the model a fixed system prompt (`prompts/system-ts.md`, or `prompts/system-tsus.md` with [LANGUAGE.md](LANGUAGE.md) inlined) plus the task prompt and signature.2. Compiles the reply. `.ts`: `tsc` with `strict: true`, lib ES2022, no DOM or Node types. `.tsus`: `tsus check --json`, then `tsus build --json`, using the compiler in `packages/cli` at the current commit.3. If the compiler rejects it, sends the diagnostics back with the previous attempt and asks for a fix, up to `--repairs` rounds (default 3).4. Runs the hidden tests once, on the first version that compiles. The model never sees test results.It records, per task and language: whether the first attempt compiled and passed (pass@1), whether any attempt passed (pass after repair), the number of repair rounds, every diagnostic code the compiler raised, failing hidden tests, tokens, cost, and time. Every attempt's code and diagnostics are saved under `results/<run>/attempts/`.## Running itFrom the repository root, after `npm install`:```shnode evals/validate.mjs                      # references pass, mutants fail; no modelnode evals/run.mjs --provider selftest       # whole pipeline incl. repair loop; no modelnode evals/run.mjs --pilot                   # the 10 pilot tasks, both languagesnode evals/run.mjs                           # all tasksnode evals/report.mjs                        # rewrite evals/RESULTS.md from evals/results/```Useful flags: `--tasks a,b`, `--langs tsus`, `--model <id>` (default `claude-sonnet-5-5`, or `TSUS_EVAL_MODEL`), `--repairs N`, `--concurrency N` (default 4), `--max-cost <usd>` (stops starting new model calls past this spend), `--label <text>`, `--tsus-bin <path>` (test another compiler build).### Model access- **Claude Code CLI** (default when `ANTHROPIC_API_KEY` is unset): each call is `claude -p --model <id> --system-prompt <fixed prompt> --tools "" --strict-mcp-config --no-session-persistence --output-format json`, run from an empty temp directory so no project `CLAUDE.md` is picked up. Cost comes from the CLI's own `total_cost_usd`. The CLI must be logged in: run `claude` once interactively and `/login` if `claude -p "hi"` reports an authentication error. If you start the eval from inside a Claude Code session, the runner strips the variables that tie a child `claude` to the parent session.- **Anthropic API** (default when `ANTHROPIC_API_KEY` is set, or `--provider anthropic`): needs `npm install --prefix evals` for `@anthropic-ai/sdk`. Cost is computed from token usage and the price table in `lib/providers.mjs`. Add `--effort low|medium|high` to set the effort level.- `reference` and `selftest` call no model. They exist to prove the harness works.Keys are never printed or written to results; both paths read credentials themselves.Rough cost: a task is about 2k to 4k input tokens (the `.tsus` side carries the primer) and 1k to 3k output tokens per attempt. At Sonnet 5.5 prices ($2 / $10 per million tokens) a full 30-task, two-language run with few repairs is on the order of $1 to $2; the 10-task pilot is well under $1. Use `--max-cost` to cap it.## Adding a taskCopy `tasks/parse-config/` and keep its conventions. The prompt states the contract precisely but does not list the edge cases; the tests probe them. Every test must follow from the prompt. Then run `node evals/validate.mjs <id>` until the references pass and the mutants fail.## CaveatsThese matter more than any number in RESULTS.md.- **The tsus checker is younger than tsc.** At the time of the first runs, `tsus check` enforced the five bans and the macro syntax but did no semantic type-checking, so `.ts` got more compile-time help than `.tsus`. The harness always uses the compiler at the current commit and records that commit in each run, so results before and after the typed checker lands are comparable side by side.- **The tasks favor the thesis by design.** They were picked because null, undefined, exceptions and unhandled cases cause real bugs in them, and the tasks and tests were written with Claude. A neutral task set (or tasks drawn from real bug reports) would be a fairer test.- **Prompt asymmetry.** The model knows TypeScript from training and learns tsus from a primer in the system prompt. That probably hurts tsus, but it also means the tsus side gets more instructions.- **One sample per task, no fixed temperature** on the CLI path. Re-run to see variance; a difference of a task or two is noise.- **Contracts differ slightly in shape.** `.ts` throws typed error classes, `.tsus` returns `Err`. The tests treat both as the same outcome, but a model can lose a `.ts` test by throwing a plain `Error`, which has no `.tsus` analogue.