justintsus-experiment Public
tsus-experiment
README.md
TSUS experiment: plain TypeScript against TypeScript-Ultra-Strict
Two Claude agents built the same Rebasely feature from the same brief. Agent A wrote plain strict
TypeScript. Agent B wrote TypeScript-Ultra-Strict
(TSUS), a stricter dialect with no null, undefined, as, !, any, class or throw, plus
Option, Result and an exhaustive match. A judge agent then scored both builds against a rubric
written before either agent started.
The feature: a GitHub-style search query language (is:open label:bug author:@me sort:updated) for
Rebasely's Issues and Pull requests lists. It parses the query, prints a canonical form, and turns it
into parameterized SQL for D1, scoped to the repository and organization.
The write-up is on the Rebasely blog: https://rebasely.com/blog/strict-typescript-ai-code
How it was run
- One brief, two copies.
brief/BRIEF-ts.mdandbrief/BRIEF-tsus.mddiffer only in their "Language" section (brief/BRIEF.mdis the template). Each agent worked alone in its own git worktree of Rebasely at commit903f762f, on the same model (Claude Opus 5.5), with no access to the other's work or to the tests. - 118 hidden tests (
hidden-tests/), written before the builds from the brief alone: parsing, canonical form and round trips, semantics on SQLite, dates, SQL injection and scope.run.sh <worktree>runs them against a build. - A judge agent scored each build against
brief/RUBRIC.md: hidden tests (40), code review (30), UI (20), cost (10), minus security penalties. A gap under 5 points is a tie. The judge also fuzzed both builds against each other and reviewed the page wiring for security (judge/JUDGE.md, probes injudge/probes/). - The harness recorded each agent's tokens, tool calls and wall time (
usage.json).
The judge knew which build was which: .tsus files cannot be hidden. The rubric tells it not to
reward or penalize the language itself.
Results
| A, plain TypeScript | B, TSUS | |
|---|---|---|
| Hidden tests passed | 118 / 118 | 118 / 118 |
| Judge's total (of 100) | 98.0 | 95.9 |
| Code review (of 30) | 28.5 | 27.0 |
| UI (of 20) | 19.5 | 20.0 |
| Cost (of 10) | 10.0 | 8.86 |
| Security findings | 0 | 0 |
| Tokens | 277,747 | 290,102 (+4.4%) |
| Output tokens | 35.7k | 57.7k (+61%) |
| Tool calls | 117 | 150 (+28%) |
| Wall time | 25.9 min | 25.1 min |
| Search module lines | 626 | 1,033 (+65%) |
as casts |
11 | 0 |
| New unit tests | 17 | 28 |
| TSUS diagnostics during the build | n/a | 0 in 8 check runs |
On 20,400 random inputs plus about 300 targeted probes the two builds returned the same results,
errors, canonical forms and SQLite rows. The only differences were outside the brief: B fails on a
row whose labels column is not valid JSON (A guards it with json_valid), and B throws on
parseQuery(null).
Verdict
A tie on quality, and plain TypeScript was cheaper. The TSUS build had no casts where the plain
build had 11, all of them safe. TSUS caught nothing during the build: every check run came back
clean, because the agent read the rules first and wrote inside them. In exchange it took 33 more tool
calls, 61% more output tokens and 65% more code, including a 51-line adapter between Option and
Result and the plain-TypeScript page code. The build that shipped in Rebasely is A, with B's
full-width search box.
On this feature, TSUS did not measurably beat strict TypeScript for an AI builder. Where it did help
was an earlier pilot that rewrote existing code (Rebasely's on-call watcher) in TSUS: following the
Option and match rewrite turned up two real production bugs, described in the blog post.
What is here
| Path | What |
|---|---|
brief/ |
The template brief, the two briefs the agents got, and the judge's rubric |
hidden-tests/ |
The 118 hidden tests, their SQLite fixture (helpers.mjs) and run.sh; results/ has both runs |
build-a-ts/ |
Build A's new files: apps/web/src/search/*.ts, apps/web/src/ui/search.ts, test/unit/search/ |
build-b-tsus/ |
Build B's new files: apps/web/src/search/*.tsus and api.ts, lists.ts, ui/search.ts, test/unit/search/ |
judge/JUDGE.md |
The judge's full report and scores |
judge/probes/ |
The judge's differential probes, page renderer and screenshot script, and the probe output |
judge/shots/ |
Six of the judge's 72 screenshots |
usage.json |
Tokens, tool calls and milliseconds per agent, from the harness |
The builds are excerpts: only the feature's new files, not the routes, pages and styles they changed
in the rest of Rebasely, which is not open source. So they do not run on their own here. The hidden
tests only need apps/web/src/search/api.ts of a build, and import it from SEARCH_WORKTREE;
B's .tsus files need the TSUS compiler first. The judge's scripts take the two checkouts from
WORKTREE_A and WORKTREE_B.
Links
- TypeScript-Ultra-Strict: https://rebasely.com/justin/typescript-ultra-strict
- The blog post: https://rebasely.com/blog/strict-typescript-ai-code
License
MIT, see LICENSE. The two builds are excerpts of Rebasely source, published by its owner for this experiment.