Rebasely Beta: data may be purged

justintypescript-ultra-strict Public

Markdown · 6.0 KiB · at f30ac63

tsus eval results

Generated by node evals/report.mjs from the runs in evals/results/. Method and caveats are in evals/README.md.

The question: when a model writes the same small program in plain TypeScript (tsc --strict) and in TypeScript-Ultra-Strict, which version is correct more often? Each of the 30 tasks has a prompt, a signature per language, and hidden node:test tests the model never sees. The compiler's errors are fed back for up to N repair rounds; test results never are.

How to read the metrics:

  • pass@1: the first attempt compiled and passed every hidden test.
  • Pass after repair: some attempt within the repair budget compiled, and that version passed every hidden test.
  • Compile-time catches: attempts the compiler rejected. In .tsus these include the ban rules (TSUS001 to TSUS005) and macro errors. A catch is only good news if the repaired version then passes; "tasks rescued by repair" counts those.
  • Failing at runtime: the solution compiled but at least one hidden test failed. These are the bugs that would have shipped.

No model run has been recorded yet. The sections below come from the harness self-test, which uses the reference solutions instead of a model. See evals/README.md for how to run the pilot.

Harness self-test: 2026-10-17, selftest

No model was called in this run. The selftest provider feeds the hand-written reference solutions (with a planted compile error on the first attempt, to exercise the repair loop) through the full pipeline. It shows the harness and the hidden tests work end to end. It says nothing about the thesis.

Run 2026-10-17-05-27-00-selftest (evals/results/2026-10-17-05-27-00-selftest): provider selftest, model none, effort default, up to 3 repair rounds, 30 task(s), repo commit 6dae872, Node v24.17.0. Total cost $0.0000.

Metric .ts .tsus
Tasks scored 30 30
pass@1 (first attempt compiles and passes every hidden test) 0/30 (0%) 0/30 (0%)
Pass after compiler-guided repair 30/30 (100%) 30/30 (100%)
Compiled on first attempt 0/30 (0%) 0/30 (0%)
Tasks where the compiler caught something 30 30
Compile-time catches (rejected attempts / diagnostics) 30 / 30 30 / 30
Tasks rescued by repair (rejected, then passed) 30 30
Never compiled within the repair budget 0 0
Average repair rounds 1.00 1.00
Tasks failing at runtime (hidden tests) 0 0
Failing hidden test cases 0 of 369 0 of 369
Cost $0.0000 $0.0000
Output tokens 0 0
Average wall time per task (model calls only) 0.0 s 0.0 s

Diagnostic codes seen in .ts attempts: TS2322 x30.

Diagnostic codes seen in .tsus attempts: TSUS002 x30.

Per-task detail
Task .ts .tsus
bank-transfer pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
cart-total pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
csv-rows pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
date-range-merge pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
deep-get pass after repair (2 attempts, tests 11/11) pass after repair (2 attempts, tests 11/11)
feature-flag pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
free-slot pass after repair (2 attempts, tests 11/11) pass after repair (2 attempts, tests 11/11)
inventory-ledger pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
json-api-handler pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
lru-cache pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
manager-chain pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
merge-profile pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
moving-average pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
order-state-machine pass after repair (2 attempts, tests 14/14) pass after repair (2 attempts, tests 14/14)
paginate-cursor pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
parse-amount pass after repair (2 attempts, tests 10/10) pass after repair (2 attempts, tests 10/10)
parse-config pass after repair (2 attempts, tests 11/11) pass after repair (2 attempts, tests 11/11)
parse-duration pass after repair (2 attempts, tests 11/11) pass after repair (2 attempts, tests 11/11)
parse-log-line pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
query-params pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
resolve-path pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
retry-fetch pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
route-match pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
semver-compare pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
shortest-path pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
signup-validation pass after repair (2 attempts, tests 14/14) pass after repair (2 attempts, tests 14/14)
stats-summary pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
token-bucket pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)
topo-sort pass after repair (2 attempts, tests 13/13) pass after repair (2 attempts, tests 13/13)
ttl-cache pass after repair (2 attempts, tests 12/12) pass after repair (2 attempts, tests 12/12)