Rebasely Beta: data may be purged

justintsus-experiment Public

Markdown · 20.5 KiB · at 8e57616

Judge report: search query language, plain TypeScript (A) vs TSUS (B)

  • A, plain TypeScript: .claude/worktrees/agent-a7895120081998e14, commit 40d9e6fd
  • B, TSUS: .claude/worktrees/agent-a66151373793240e4, commit a69ca436
  • Base: 903f762f. Judged 2026-10-17.

Probe scripts, renderers and raw output are in judge/probes/. The judge took 72 screenshots; six are in judge/shots/.

Verdict

It's a tie on quality, and A is the cheaper build. Ship A, but bring over B's search box layout and B's test fix first.

The two builds behave the same everywhere I could measure:

  • The hidden tests pass 118/118 for both.
  • On 20,400 random (input, target) pairs plus about 300 targeted adversarial probes, they return the same ok, the same error codes and spans, the same canonical form and the same rows on SQLite.
  • The page glue produces the same tab hrefs, counts, pager links and label-chip links.

Only two behavioral differences turned up, and neither is in the brief's contract:

  1. Malformed labels JSON. If a stored labels value isn't valid JSON, B's label:, -label: and no:label queries fail in SQLite. A guards the column with json_valid. The base code had the same unguarded json_each that B uses.
  2. Non-string input. B's parseQuery(null) and parseQuery(undefined) throw. A coerces them with String(). The contract only covers strings.

The totals are 98.0 for A and 95.9 for B. That gap is inside the rubric's 5-point tie band, and most of it comes from cost: A used 12k fewer tokens and 33 fewer tool calls.

What TSUS bought here: zero casts in B's whole diff, against 11 in A. Every one of A's casts is safe today. The TSUS checker reported zero diagnostics in all 8 of B's check runs. It caught nothing, because the builder read the rules first and wrote inside them. Its value in this experiment was preventive, not corrective. No bug existed in A that TSUS would have stopped.

What TSUS cost:

A B B vs A
Tool calls 117 150 +28%
Total tokens 277,747 290,102 +4.4%
Output tokens 35.7k 57.7k +61%
Search-module lines 626 1,033 +65%
Calls on TSUS docs and checks 0 15

B also needed a 51-line adapter and a few awkward idioms to work around banned indexing. Wall clock was a wash: B took 25.1 min and A took 25.9 min.

Scores

Item Max A B Evidence
Hidden tests 40 40.0 40.0 118/118 each (ts.out, tsus.out)
Correctness beyond tests 8 8.0 7.5 Identical on every probe. B fails on malformed labels JSON, which A guards.
Error handling 6 5.5 5.5 A: a dead catch-all in parseQuery would hide a parser bug as a whole-input bad_value. B: an unreachable throw in ui/search.ts, and it throws on non-string input. Neither catches D1 failures, same as base. Both log and survive a RepoDO hydrate failure.
Readability and structure 6 5.5 5.0 A is compact (6 files, 486 code lines) but has 5 casts and a closure that mutates the lexer index. B has real tagged unions and exhaustive match, but it's 65% longer, with ban workarounds like parts.slice(0, 1).join("") and a for-loop over a 0-or-1-element array.
Edge cases 5 5.0 4.5 Same limits, unicode and whitespace behavior. A guards labels JSON. B guards a null closedAt and mergedAt in rows.
Tests the builder wrote 3 2.5 3.0 A: 17 new tests, and test/unit/issues.test.ts still fails. That failure predates both builds (I checked on base 903f762f), but A edited the file and left it red. B: 28 new tests, and it fixed that assertion.
Fit with the codebase 2 2.0 1.5 A keeps the one-round-trip DB.batch of base and reuses ISSUE_COLS. B makes two D1 round trips (Promise.all of two statements), duplicates ISSUE_COLS, and puts page glue that imports repoStub and log (lists.ts) in search/. B uses typed .all<R>() instead of casts.
Code review subtotal 30 28.5 27.0
UI: search runs the language 6 6 6 Same results for state, label, author, text, date and sort queries
UI: errors inline 5 5 5 Both: wavy underline plus a superscript number per span, message list, aria-invalid, no list shown, input kept
UI: syntax help 3 3 3 <details> on both lists, only that list's qualifiers, opens with Enter and without JS
UI: URL round trip 3 3 3 Canonical q in the box, tabs, pager and chips. Legacy state= and label= links work.
UI: design 3 2.5 3 A's box is 460px wide in the filter row, so the canonical query is cut off at 1280 px. B puts a full-width box above the tabs. No overflow at 390 px for either. No uppercase, gradients or em dashes in either.
UI subtotal 20 19.5 20.0
Cost: tokens (6) 6 6.00 5.74 277,747 vs 290,102
Cost: tool calls (4) 4 4.00 3.12 117 vs 150
Cost subtotal 10 10.0 8.86 Per hidden test passed: 2,354 vs 2,458 tokens
Security penalties 0 0 None found (see below)
Total 100 98.0 95.9 Tie (within 5); A is cheaper

Preparation checks

Check A B
Commits on branch 1 1
Stray files committed None None. No generated .js/.d.ts, no migration, no edits to scripts/tsus-web.mjs, tools/tsus, wrangler or CI.
npm run typecheck Pass Pass
npm run lint Pass Pass
Builder's own search tests 17/17 28/28
test/unit/issues.test.ts 2/3, red, inherited from base 3/3
TSUS constraint (B only) n/a, no .tsus used Honored. All logic is in 7 .tsus files. api.ts (51 lines) only converts Result/Option and wraps edit helpers.

Code metrics (added code under apps/web/src, comments excluded)

A B
Source files added or changed 14 15
Search module files 6 .ts 7 .tsus, api.ts, lists.ts
Search module lines (total / non-blank, non-comment) 626 / 486 1,033 / 776
Lines added in src + css / in tests 1,002 / 495 1,354 / 570
New unit tests 17 28
as casts (excluding as const) 11 0
Non-null assertions ! 0 0
any 0 0
try/catch/throw added catch-all in parseQuery, .catch on hydrate try/catch on hydrate, 1 unreachable throw in ui/search.ts

A's 11 casts are as Key, as DateOp, as RelUnit, as StateTab, as StateValue, as IssueTab | null, as "open" | "closed" | "merged" | "all", as IssueRow[], as PullIndexRow[] and 2 of as { state: string; n: number }[]. All are safe given the surrounding checks. The last four copy the base code's D1 pattern.

Extra probe results

All probes import each build's apps/web/src/search/api.ts in one Node 24 process and run SQL on the hidden-test fixture (node:sqlite). Scripts: judge-probes/probe.mjs and judge-probes/show.mjs. Raw output: judge-probes/probe.out.

Probe A B Brief
SQL injection. 16 payloads × 2 targets in text, phrases, labels, author, sort, dates and #n ('; DROP TABLE, ") OR 1=1 --, ?1 :name, NUL, U+2028). Plus a hostile ctx: repoId repo-1' OR 1=1 --, orgId with "), viewerId ' OR 1=1 --. Inert. No ctx value appears in the SQL text. Inert. No ctx value appears in the SQL text. Both match
Scope escapes. 20,400 fuzz pairs, every returned row checked against ctx repo and org, ANON included 0 leaks 0 leaks Both match
Cross-org agent slug. author:fixer: org-1's fixer is FIXER; org-2's fixer (FIXER2) authored issue 6 and PR 6 in this repo. Issues [5], pulls [2], never 6. Under orgId org-2: [] Same Both match
review-requested:fixer, -author:fixer on pulls [] and [6,1,4,3,5] Same Both match
NULL under negation. -author:@me on pulls (PR 1 authored by ME, PR 2 owned by ME, others have owner NULL) [6,4,3,5] Same Match
-author:BOB on pulls (PR 4 authored by Bob, PR 6 owned by Bob) [2,1,3,5] Same Match
-author:@me / -review-requested:@me signed out Every row in scope Same Match
author:@me signed out [] [] Match
LIKE escaping. 100% → [2], a_c → [4], % → [2], _ → [4], \ → [6], a\b → [6], a_b on pulls → [4] As listed Same Match. No wildcards leak.
ASCII-only case folding: Ü → [4], ü → [], ÜNÏCODE → [] As listed Same Match (SQLite LIKE semantics)
Unicode spans. "🐛 → unterminated 0..3. a 🐛" b → unexpected 2..5. 🐛:x → free text. A lone surrogate \uD83D is free text. UTF-16 offsets, as listed Identical Match
Whitespace kinds. NBSP, U+3000, U+2003 and U+FEFF separate terms. U+200B, U+0085 and U+180E do not (JS \s). As listed Same Match
Error ordering. -label: / -sort: → empty_value before negation_not_allowed. sort:x sort:y → two bad_value, no conflict. sort:created-asc sort:CREATED-ASC → deduplicated, no conflict. As listed Same Match
Dates (48 values × 2 fields × 2 targets). *..*, <0d, <10000d, 7d, =D, 1969-12-31, 2026-04-31, <٧d (Arabic digit), A..B with A after B, >D..E → all bad_date. <0007D → <7d. 9999-12-31 accepted. As listed Same Match
Limits. 1025 chars → too_long 0..1025 alone. 33 terms → too_many_terms from the 33rd term to the end, even when an unterminated quote follows. As listed Same Match
D1 100-bound-parameter limit, worst case. Most params per term is 2 (text on issues; author: on pulls). 32 distinct heavy terms + scope = 66 params, + LIMIT/OFFSET = 68. Counts query: 66. Hydrate IN list: at most 25. 68 max 68 max Both under 100
SQL text size, worst case 3.6 KB 4.5 KB Fine
Speed. 10k random queries, parse + serialize + toSql, median of 5 runs 13.5 ms 18.8 ms Both negligible
Worst-case 1024-char inputs (escaped quotes, 1024 ", 1023 \, 341 negations), µs per parse 7 to 37 9 to 30 Both linear, no blowup
Never throws, string inputs. 20,400 fuzz + targeted 0 throws 0 throws Match
Non-string input (outside contract): parseQuery(null) / parseQuery(undefined) ok: true (coerces) Throws TypeError Not required
toSql with now: NaN (outside contract) Binds null Binds null n/a
Malformed labels JSON row ('not json') with label:bug, -label:bug, no:label Works; the row counts as no labels SQLite "malformed JSON", which would be a 500 on the page Not in the brief. Base code behaves like B.
Round trip: parseQuery(serialize(q)) re-serializes identically and gives identical SQL, on every accepted fuzz input 0 violations 0 violations Match
Placeholders = params on every accepted fuzz input 0 violations 0 violations Match

The only behavioral differences between A and B are the last-but-one two rows: non-string input and malformed labels JSON. On every input the brief defines, the builds agree.

Security review of the page wiring

Area A B
Permission issuesPage and pullsPage still go through pageOf (prContext, repo.read) before any search. Unchanged. Same
SQL in the glue List, counts and pagination queries interpolate only fixed table and column lists plus toSql output. LIMIT ? OFFSET ? are bound. Same. The table name comes from a literal union.
Scope on counts Counts query is toSql(withState(q, "all")), so it is scoped Counts query is toSql(withState(q, null)), so it is scoped
XSS The error echo, the message list <code> and value= all go through html```, which escapes & < > " '. Payload "> foo: label:"x" is rendered inert (issues-xss.html`). Same, verified the same way
Open redirects No new redirects. Every link is searchHref(base, …) with base the repo path and a URLSearchParams-encoded q. Same
Pagination limits page clamped to 1..400 (max OFFSET 9,975). 25 + 1 rows per page. Same (MAX_PAGE 400)
RepoDO hydrate failure .catch logs pulls.list_hydrate_failed and renders from pull_index try/catch logs search.pulls_hydrate_failed and renders from pull_index
D1 failure Not caught (500), same as base listIssueViews Not caught (500), same as base
Agent slug resolution Subquery agents.org_id = ? AND slug = ? with ctx.orgId bound Same

No security findings for either build, so no penalties.

Findings by severity

High: none in either build.

Medium:

  • A, UI: The search box is 460px wide in the filter row, so at 1280 px the canonical query is cut off: is:open label:"needs review" sort:upd. For a query language whose canonical form is the point of the box, that hurts. (See A-issues-query-1280-light.png and B-issues-query-1280-light.png.)

Low:

  • B: label:, -label: and no:label throw in SQLite if any row in scope has malformed labels JSON, which makes the list page return 500. The base code had the same weakness. A's LABELS CASE guard avoids it.
  • B: parseQuery(null) and parseQuery(undefined) throw. This is outside the string contract, but A tolerates it.
  • B: Two D1 round trips per list (Promise.all of .all() calls) where base and A use one DB.batch.
  • B: Page glue (search/lists.ts, which imports repoStub and log) lives in search/ next to the pure modules, and ISSUE_COLS is duplicated instead of exported.
  • A: test/unit/issues.test.ts stays red. The failure is inherited from base, but A edited that test and didn't fix it. B did.
  • A: The catch-all try/catch in parseQuery is unreachable by design. If a bug ever made it reachable, it would report a whole-input bad_value instead of surfacing the bug.
  • A: 11 casts. All safe today, but none are checked.

Info:

  • B: Hydrated pull rows show the RepoDO's live title and state. A shows pull_index's state with the DO's comment count. Both are allowed.
  • B: Old ?label= links with an invalid label show a parse error. A silently drops the label. Both are defensible.
  • A: On an invalid query, A shows a "Nothing listed" empty state with a Clear button. B shows the error card with a "Clear the search" link and no list card. A's guidance is slightly clearer. A also renders the whole message list in the error color; B uses body text with a red heading, which reads more easily.

UI check

I rendered both builds' pages with each build's own page glue (repo/list-search.ts in A, search/lists.ts in B) against the hidden fixture through a node:sqlite D1 shim and a mocked RepoDO, then wrapped them in the real renderPage layout. The fixture had 30 extra issues so a second page exists. Script: judge-probes/render.mjs.

I screenshotted the pages with Playwright (Chromium) at 1280 and 390 px, in light and dark (judge-probes/shots.mjs). Pages covered:

  • issues: default, a query, page 2, a legacy ?state=closed&label=bug link, an invalid query, an XSS payload
  • pulls: default, a query, an invalid query
  • the help popover on both lists

What I checked:

  • Links and counts: every tab href, count, pager link and label-chip link is identical between the builds.
  • Overflow: no horizontal overflow at 390 px on any page for either build.
  • Help: <details> opens with Enter from keyboard focus, and also with JavaScript disabled.
  • Help contents: each list shows only its own qualifiers. Pulls never show label: or no:; issues never show review-requested:.
  • Error marks: the offending spans are underlined (wavy) and carry a number, so they don't rely on color alone. A adds a visually hidden "(problem n)"; B marks the echo aria-hidden and lists the terms in <code>. B also puts role="alert" on the error card and points aria-describedby at the help.
  • DESIGN.md: no uppercase text transforms, gradients or em dashes in the new UI. Inter body text and mono for queries in both.

What TSUS demonstrably caught or prevented, and what it cost

Caught during the build: nothing. All 8 tsus-web.mjs check --json runs in B's transcript returned {"ok": true, "diagnostics": []}. B's only failures came from its own unit tests (2 parse expectations, 2 SQL expectations) and the repo's design test (a raw z-index), the same kinds of failures A hit.

Prevented, by construction: unchecked narrowing. A narrows strings to unions with as after a runtime check. B has to prove the narrowing with a total match:

TypeScript
// A, parse.ts
if (!Object.hasOwn(KEYS, key)) return fail("unknown_qualifier", ...);
const k = key as Key;
...
const op = m[1] as DateOp;
return { kind: "age", op, n, unit: rel[2].toLowerCase() as RelUnit };
TypeScript
// B, parse.tsus
const keyOf = (k: string): Option<Key> =>
  match (k) {
    "is" => Some<Key>("is"),
    "label" => Some<Key>("label"),
    ...
    _ => None,
  };

A design both reached, by different routes: NULL-safe negation. A wraps every negated term generically. B makes each positive form never NULL (IS instead of = on the nullable column). Both are correct; A's is one rule, and B's needs care per term:

TypeScript
// A, sql.ts
parts.push("neg" in t && t.neg ? `NOT IFNULL((${c.sql}), 0)` : `(${c.sql})`);
TypeScript
// B, sql.tsus
User(x) => Some(frag("(pull_index.author_id = ? OR pull_index.owner_id IS ?)", [x.id, x.id])),

Where B cost more:

  1. Adapter code. api.ts (51 lines) converts at every boundary. null becomes None, Result becomes { ok }, and the edit helpers need Option wrappers:
TypeScript
// B, api.ts
export function toSql(query: SearchQuery, ctx: SqlCtx) {
  const r = sql(query, { repoId: ctx.repoId, orgId: ctx.orgId, viewer: ctx.viewerId === null ? None : Some(ctx.viewerId), now: ctx.now });
  return { where: r.where, params: [...r.params], orderBy: r.orderBy };
}
export function tabOf(query: SearchQuery): ListState | null {
  const t = tabOfQuery(query);
  return t.isSome() ? t.value : null;
}
  1. Ban workarounds. These are where plain indexing would have been clearer:
TypeScript
// B, values.tsus: parts[0] and parts[1] without indexing
const parts = v.split("..");
if (parts.length !== 2) return None;
const from = endOf(parts.slice(0, 1).join(""))?;
const to = endOf(parts.slice(1).join(""))?;
TypeScript
// B, edit.tsus: "the one element, if exactly one"
const only = states.length === 1 ? states.slice(0, 1) : [];
for (const s of only) if (!s.negated) return Some(s.state);
return None;

Compare A:

TypeScript
// A, edit.ts
const [t] = s;
return s.length === 1 && t.kind === "is" && !t.neg ? (t.value as StateTab) : null;
  1. Volume and calls:
A B
Search module lines 626 1,033 (+65%)
Output tokens 35.7k 57.7k (+61%)
Tool calls 117 150 (+33)

About 15 of B's extra calls went to TSUS itself: 7 reads of tools/tsus/AGENTS.md, the runtime .d.ts and the templates, plus 8 check runs. The rest went to a longer UI preview session and a WIP-commit-then-soft-reset. The total token gap is small (+4.4%) because input and cache reads dominate.

Where TSUS did not cost more: wall clock (25.1 vs 25.9 min) and behavior (identical). TSUS added no runtime overhead worth mentioning: 18.8 vs 13.5 ms per 10k queries, both far below a request's budget.

Ship recommendation

Ship A: same behavior, cheaper, one D1 round trip, tolerant of bad labels JSON. Before shipping, fix three things:

  1. Widen the search box so the canonical query is visible on desktop. B's full-width box above the tabs is the better layout; take it.
  2. Fix the inherited aria-current="page">Issues< assertion in test/unit/issues.test.ts (B's regex fix works).
  3. Optionally, drop the unreachable catch-all in parseQuery, or make it log.

If the team prefers B, it's equally shippable. Add the json_valid guard on labels (or accept base's behavior) and batch the two D1 statements.

Notes on the comparison

  • Hidden tests didn't separate the builds. They pass 118/118 each, and my fuzzing found no disagreement on any input the brief defines. The two independent implementations agree with each other and with the hidden tests, which is good evidence the brief was unambiguous.
  • Environment friction hit both builders. The worktree guard refused compound ln/git commands 7 times for A and 9 times for B. The pre-existing red issues.test.ts affected both equally; only B spent calls on it, and only B fixed it.
  • UI was judged from static renders through each build's own glue and the real layout, not a live wrangler dev stack with sessions. Signed-in versus signed-out differences (New issue button, sign-in prompt) weren't exercised beyond the canOpen flag. Neither build changed them.