tsus eval results
Generated by node evals/report.mjs from the runs in evals/results/. Method and caveats are in evals/README.md.
The question: when a model writes the same small program in plain TypeScript (tsc --strict) and in TypeScript-Ultra-Strict, which version is correct more often? Each of the 30 tasks has a prompt, a signature per language, and hidden node:test tests the model never sees. The compiler's errors are fed back for up to N repair rounds; test results never are.
How to read the metrics:
- pass@1: the first attempt compiled and passed every hidden test.
- Pass after repair: some attempt within the repair budget compiled, and that version passed every hidden test.
- Compile-time catches: attempts the compiler rejected. In .tsus these include the ban rules (TSUS001 to TSUS005) and macro errors. A catch is only good news if the repaired version then passes; "tasks rescued by repair" counts those.
- Failing at runtime: the solution compiled but at least one hidden test failed. These are the bugs that would have shipped.
No model run has been recorded yet. The sections below come from the harness self-test, which uses the reference solutions instead of a model. See evals/README.md for how to run the pilot.
Harness self-test: 2026-10-17, selftest
No model was called in this run. The selftest provider feeds the hand-written reference solutions (with a planted compile error on the first attempt, to exercise the repair loop) through the full pipeline. It shows the harness and the hidden tests work end to end. It says nothing about the thesis.
Run 2026-10-17-05-27-00-selftest (evals/results/2026-10-17-05-27-00-selftest): provider selftest, model none, effort default, up to 3 repair rounds, 30 task(s), repo commit 6dae872, Node v24.17.0. Total cost $0.0000.
| Metric | .ts | .tsus |
|---|---|---|
| Tasks scored | 30 | 30 |
| pass@1 (first attempt compiles and passes every hidden test) | 0/30 (0%) | 0/30 (0%) |
| Pass after compiler-guided repair | 30/30 (100%) | 30/30 (100%) |
| Compiled on first attempt | 0/30 (0%) | 0/30 (0%) |
| Tasks where the compiler caught something | 30 | 30 |
| Compile-time catches (rejected attempts / diagnostics) | 30 / 30 | 30 / 30 |
| Tasks rescued by repair (rejected, then passed) | 30 | 30 |
| Never compiled within the repair budget | 0 | 0 |
| Average repair rounds | 1.00 | 1.00 |
| Tasks failing at runtime (hidden tests) | 0 | 0 |
| Failing hidden test cases | 0 of 369 | 0 of 369 |
| Cost | $0.0000 | $0.0000 |
| Output tokens | 0 | 0 |
| Average wall time per task (model calls only) | 0.0 s | 0.0 s |
Diagnostic codes seen in .ts attempts: TS2322 x30.
Diagnostic codes seen in .tsus attempts: TSUS002 x30.
Per-task detail
| Task | .ts | .tsus |
|---|---|---|
| bank-transfer | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| cart-total | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| csv-rows | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| date-range-merge | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| deep-get | pass after repair (2 attempts, tests 11/11) | pass after repair (2 attempts, tests 11/11) |
| feature-flag | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| free-slot | pass after repair (2 attempts, tests 11/11) | pass after repair (2 attempts, tests 11/11) |
| inventory-ledger | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| json-api-handler | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| lru-cache | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| manager-chain | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| merge-profile | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| moving-average | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| order-state-machine | pass after repair (2 attempts, tests 14/14) | pass after repair (2 attempts, tests 14/14) |
| paginate-cursor | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| parse-amount | pass after repair (2 attempts, tests 10/10) | pass after repair (2 attempts, tests 10/10) |
| parse-config | pass after repair (2 attempts, tests 11/11) | pass after repair (2 attempts, tests 11/11) |
| parse-duration | pass after repair (2 attempts, tests 11/11) | pass after repair (2 attempts, tests 11/11) |
| parse-log-line | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| query-params | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| resolve-path | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| retry-fetch | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| route-match | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| semver-compare | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| shortest-path | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| signup-validation | pass after repair (2 attempts, tests 14/14) | pass after repair (2 attempts, tests 14/14) |
| stats-summary | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| token-bucket | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |
| topo-sort | pass after repair (2 attempts, tests 13/13) | pass after repair (2 attempts, tests 13/13) |
| ttl-cache | pass after repair (2 attempts, tests 12/12) | pass after repair (2 attempts, tests 12/12) |