Rebasely Beta: data may be purged

justintsus-experiment Public

Markdown · 110 lines · 7.6 KiB · at 8e57616

# Rubric: TypeScript vs TypeScript-Ultra-Strict, search query languageYou are the judge. Two builders implemented the same brief (`BRIEF.md` in this directory; builder Agot `BRIEF-ts.md`, builder B got `BRIEF-tsus.md`, identical except the "Language" section). Scoreeach build on its own against this rubric, then compare. Do not reward or penalize the languageitself: judge what each builder produced under its constraints. Give every score with the evidence(file and line, test name, command output).You are given, per builder: the worktree path and branch, and from the harness the token count andtool-call count. Run everything yourself; do not trust a builder's own report.## 0. Preparation (no points)1. In each worktree: `git log --oneline main..HEAD` and `git diff main...HEAD --stat`. Note   anything committed that should not be (generated `.js` or `.d.ts` under `apps/web/src`, a   migration, edits to `scripts/tsus-web.mjs`, `tools/tsus`, the wrangler config or CI, edits to   unrelated features). Changing the build wiring or the vendored compiler is a red flag for B and   a scope violation for either.2. `npm run typecheck` and `npm run lint` in each worktree (record pass or fail and the output).3. Run the builder's own tests (whatever they added under `test/unit/`), and record the count.## 1. Hidden tests: 40 pointsRun `hidden-tests/run.sh <worktree>` (it links the worktree's `node_modules`, runs`scripts/tsus-web.mjs build`, then all `hidden-tests/*.test.mjs`). The last line is`hidden tests: pass N / total M`. Score = 40 x N / M, rounded to one decimal.- If `run.sh` exits 2 (missing `api.ts`, or the `.tsus` build fails), the score is 0.- Do not edit the tests or the build to make them run. If a test fails because of an ambiguity in  the brief (not a bug), note it, but keep the score mechanical; the same tests run for both.- 118 tests: parsing (42), canonical form and round trips (9), semantics on SQLite (30), dates (18),  SQL safety (9, three of them covering 15 injection payloads), scope and generated inputs (10), plus the structural  checks every SQL-running test makes (placeholders equal params, fixed `orderBy`, every returned  row in ctx's repository and organization).- `reference/` holds a quick reference implementation that passes all 118; it exists only to show the  tests are satisfiable. Do not use it to grade code style.## 2. Security: automatic severe penalty (up to minus 40, applied after the total)Look for these yourself in the code (the evaluator and the page glue), not only in test failures.Each confirmed finding:| Finding | Penalty ||---|---|| A value from the query text or from ctx is written into SQL text (string concatenation or a template literal with a value), anywhere: evaluator, page glue, count queries, pagination | -40, and the build cannot win || A query (any input) can return rows outside the repository or organization, or the page runs a list query without the scope (for example the counts or "next page" query) | -40, and the build cannot win || The page lists rows without the route's existing `repo.read` decision (`prContext`/`pageOf`), or the search reaches private data another way (for example resolving an agent slug or user across organizations) | -30 || An unescaped `LIKE` pattern (`%`, `_`, `\` from the input act as wildcards) | -5 || Error messages or the page echo the raw query into HTML without escaping (XSS) | -20 || `parseQuery`, `toSql` or the page throws (a 500) on some input | -10 per distinct crash |Hidden-test failures in the SQL safety or scope groups count here too: any failure in`sql-safety.test.mjs` or `permission.test.mjs` is investigated, and if it is a real injection orscope bug it takes the penalty above in addition to the lost test points.## 3. Code review: 30 pointsRead the diff as a senior reviewer on this codebase would. Score each line item and justify it.| Item | Points | What good looks like ||---|---|---|| Correctness beyond the tests | 8 | Grammar and semantics match the brief where the tests do not reach (error positions for odd input, `-author:` NULL handling, `is:closed` on pulls including merged, slug resolution scoped to the org, the page's default `is:open`, legacy `state=`/`label=` links). Spot-check with your own inputs. || Error handling | 6 | Bad input is data end to end: typed error codes, spans, one error per bad term, no exceptions in the parser or evaluator, D1 failures on the page handled like the rest of the codebase. || Readability and structure | 6 | A reader can follow tokenizer, parser, serializer and evaluator; clear types (a real AST, a closed error union); no dead code; names and comments in the repository's style (see neighbouring files). || Edge cases | 5 | Limits, unicode offsets, whitespace kinds, empty input, quoting and escaping in both directions, pagination with a sort, counts for the tabs. || Tests the builder wrote | 3 | Useful unit tests of their own, covering what they changed. || Fit with the codebase | 2 | Follows existing patterns (D1 access, `Out`/page helpers, `DESIGN.md`), touches only what the feature needs, no new dependencies. |For builder B also check the tsus constraint was honored: parser, serializer and evaluator in`.tsus`, `api.ts` a thin adapter with no logic of its own. Logic moved into `.ts` to dodge thechecker loses up to 6 points from "Readability and structure" and "Fit". For builder A check that no`.tsus` was used.## 4. UI check: 20 pointsStart the stack in each worktree (`npm run dev:v2`, port 8790; seed a repository with a few issues,labels and pull requests in several states, signed in and signed out; the repository's test helpersand `test/helpers/seed.ts` show how). Check desktop and a 375 px phone width, light and dark.| Item | Points ||---|---|| The search box on both lists runs the language; results are right for a handful of queries you try (state, label, author, text, date, sort) | 6 || Errors are shown inline: message per error under the box, offending span marked without relying on color alone, input kept as typed, status 200, no results listed | 5 || Syntax help: present on both lists, lists only the qualifiers of that list, works by keyboard and without JavaScript | 3 || URL round trip: `?q=` holds the canonical form; reload, back, the state tabs, a label link and next page all keep the query | 3 || Design: follows `DESIGN.md` and the existing list styles (one accent, Inter 450, no stripes, gradients, emoji, uppercase kickers, em dashes in copy); fits on a phone with no horizontal scroll | 3 |## 5. Cost: 10 pointsThe harness supplies, per builder, total tokens (input plus output, cache reads counted as theharness reports them) and tool calls. Score relative to the cheaper build:- tokens: 6 x (fewer tokens / this build's tokens)- tool calls: 4 x (fewer calls / this build's calls)So the cheaper build gets the full 10 and the other gets proportionally less. Report the raw numberstoo, and the cost per hidden test passed (tokens / N).## Total and verdictTotal = hidden tests (40) + code review (30) + UI (20) + cost (10) - security penalties, out of 100.Report a table with every line item for both builds, the totals, and a verdict:- The winner is the higher total, unless it carries a "cannot win" finding.- If the totals are within 5 points, say it is a tie on quality and name the cheaper build.- Then say plainly which build to ship (or that neither is shippable and why), and list what must be  fixed before shipping it.- Note anything about the comparison itself that was unfair or noisy (a hidden test the brief left  ambiguous, an environment problem one builder hit and the other did not, a tsus toolchain bug).