examplesstorefront Public
Search the catalog with D1 full text search #4
Added.rebasely/architecture/flows/catalog-search.yaml+17−0
| @@ -0,0 +1,17 @@ | ||
| +title: Searching the catalog | ||
| +description: >- | ||
| + The merchant renames a product, and a shopper searches for it a moment later. Every product | ||
| + write updates the product's row in the search index, a separate D1 database. A search matches | ||
| + in the index, then reads the products from the catalog, so prices, stock and retired products | ||
| + are always current. | ||
| +actors: [merchant, shopper] | ||
| +steps: | ||
| + - { from: merchant, to: storefront-api, label: "PUT /admin/products/linen-apron, admin token" } | ||
| + - { from: storefront-api, to: db, label: upsert the product } | ||
| + - { from: storefront-api, to: search, label: "one batch: delete the product's row, insert its name and description" } | ||
| + - { from: storefront-api, to: merchant, label: "the product, or 503 search_index_failed to retry" } | ||
| + - { from: shopper, to: storefront-api, label: "GET /search?q=linen apr" } | ||
| + - { from: storefront-api, to: search, label: "MATCH '\"linen\" \"apr\"*', ranked by bm25, name weighted 10" } | ||
| + - { from: search, to: storefront-api, label: "product ids, best match first" } | ||
| + - { from: storefront-api, to: db, label: "read those products, active ones only" } | ||
| + - { from: storefront-api, to: shopper, label: "the products in match order, with current price and stock" } | ||
Changed.rebasely/architecture/system.yaml+13−3
| @@ -2,13 +2,14 @@ | ||
| storefront-api: | ||
| kind: worker | ||
| description: >- | ||
| - The public HTTP API (wrangler.jsonc, Worker storefront-api). Shoppers browse the catalog, fill | ||
| - carts and check out; the merchant edits products and uploads their images with the admin | ||
| - token. Its OrdersService entrypoint is how the other Workers change orders: it is the only | ||
| + The public HTTP API (wrangler.jsonc, Worker storefront-api). Shoppers browse and search the | ||
| + catalog, fill carts and check out; the merchant edits products and uploads their images with | ||
| + the admin token. Its OrdersService entrypoint is how the other Workers change orders: it is the only | ||
| Worker with the database binding. A cron trigger expires unpaid orders every five minutes. | ||
| owns: [src/**] | ||
| talks_to: | ||
| - { to: db, via: d1 } | ||
| + - { to: search, via: d1 } | ||
| - { to: images, via: r2 } | ||
| - { to: stripe, via: https } | ||
| - { to: order-events, via: queue } | ||
| @@ -37,6 +38,15 @@ | ||
| file in migrations/, applied with wrangler d1 migrations apply. | ||
| binding: DB | ||
| owns: [migrations/**] | ||
| + search: | ||
| + kind: d1 | ||
| + description: >- | ||
| + Database storefront-search: an FTS5 index of the names and descriptions of active products, | ||
| + for GET /search. Derived from the catalog in db, updated on every product write and refilled | ||
| + by POST /admin/search/rebuild, so it can be dropped and rebuilt at any time. Kept apart from | ||
| + db so search reads never queue behind checkout's writes. Schema in search/migrations/. | ||
| + binding: SEARCH | ||
| + owns: [search/**] | ||
| images: | ||
| kind: r2-bucket | ||
| description: >- | ||
Changed.rebasely/architecture/views/catalog.yaml+2−2
| @@ -1,2 +1,2 @@ | ||
| -description: What browsing the shop touches, and where product images come from. | ||
| -components: [storefront-api, db, images] | ||
| +description: What browsing and searching the shop touches, and where product images come from. | ||
| +components: [storefront-api, db, search, images] | ||
Changed.rebasely/kb/concepts/catalog.md+6−2
| @@ -6,7 +6,7 @@ | ||
| describes: | ||
| - src/catalog.ts | ||
| - migrations/0001_products.sql | ||
| -reviewed: 2026-10-13 | ||
| +reviewed: 2026-10-16 | ||
| --- | ||
| # Catalog | ||
| @@ -30,9 +30,13 @@ | ||
| A cursor rather than `?page=3` because the query starts right after the last row (`(name, id) > (?, ?)`) on the `products_by_name` index, so page 40 costs the same as page 1, and a product added or retired between two requests does not shift the next page by one. | ||
| +## Searching | ||
| + | ||
| +`GET /search` finds products by the words in their name and description; see [[search]]. The index lives in another database and only returns ids, so search results are read from this table with `activeProducts()` and have the same fields, prices and stock as the rest of the catalog. | ||
| + | ||
| ## Writing | ||
| -The merchant writes products with `PUT /admin/products/:id` and the admin token. The call is an upsert: the same request creates the product or replaces its fields, which makes it safe to retry and easy to drive from a spreadsheet export. Validation lives in `parseProductInput`: | ||
| +The merchant writes products with `PUT /admin/products/:id` and the admin token. The call is an upsert: the same request creates the product or replaces its fields, which makes it safe to retry and easy to drive from a spreadsheet export. It also updates the product's row in the search index. Validation lives in `parseProductInput`: | ||
| - the id is a lowercase slug of at most 64 characters, because it appears in URLs and image keys; | ||
| - `price_cents` is a whole number above zero; | ||
Added.rebasely/kb/concepts/search.md+51−0
| @@ -0,0 +1,51 @@ | ||
| +--- | ||
| +id: search | ||
| +type: concept | ||
| +owner: justin | ||
| +component: search | ||
| +describes: | ||
| + - src/search.ts | ||
| + - search/migrations/** | ||
| +reviewed: 2026-10-16 | ||
| +--- | ||
| +# Search | ||
| + | ||
| +`GET /search?q=linen apr` returns active products whose name or description match, best match first: | ||
| + | ||
| +``` | ||
| +GET /search?q=linen%20apr&limit=24 | ||
| +{ "query": "linen apr", "products": [{ "id": "linen-apron", "name": "Linen work apron", ... }] } | ||
| +``` | ||
| + | ||
| +`q` is 1 to 100 characters. `limit` works as on `GET /products`: 1 to 100, default 24. There is no paging; a shopper who does not find it in the first page types more. | ||
| + | ||
| +## The index | ||
| + | ||
| +The index is one FTS5 table, `products_fts`, in its own D1 database, `storefront-search` ([[0011-search-index-in-its-own-database]]). It has a row per active product with the product's id (stored, not searched), name and description. Nothing else about a product is in it. | ||
| + | ||
| +The tokenizer is `porter unicode61 remove_diacritics 2`: `mugs` finds `mug`, and `creme` finds `Crème`. | ||
| + | ||
| +## Queries | ||
| + | ||
| +`toMatch()` turns what the shopper typed into an FTS5 query. It keeps letters and digits, quotes each word, and makes the last one a prefix: | ||
| + | ||
| +| typed | MATCH | | ||
| +| --- | --- | | ||
| +| `linen apr` | `"linen" "apr"*` | | ||
| +| `apron" OR name:mug` | `"apron" "OR" "name" "mug"*` | | ||
| +| `***` | nothing; the answer is an empty list | | ||
| + | ||
| +Quoting means FTS5 operators and column filters in the input are only words, so no input can make the query fail. The prefix on the last word lets results follow someone who is still typing. Up to eight words are used. | ||
| + | ||
| +Matches are ranked with `bm25(products_fts, 0, 10, 1)`: a word in the name counts ten times as much as one in the description, so the apron comes before the tote whose description mentions an apron. | ||
| + | ||
| +## Reading the results | ||
| + | ||
| +The index only answers with ids. `activeProducts()` (`src/catalog.ts`) reads those products from the catalog with `json_each`, keeps the active ones and returns them in the index's order. Price and stock are therefore always the catalog's, including stock that checkout took a second ago, and a product retired since it was indexed is left out. | ||
| + | ||
| +## Keeping it in step | ||
| + | ||
| +- `PUT /admin/products/:id` saves the product, then calls `indexProduct()`, which deletes the product's row and inserts it again in one batch, or only deletes it when the product is inactive. If that fails, the answer is `503 search_index_failed`; the product is saved, and sending the same request again fixes the index. | ||
| +- `POST /admin/search/rebuild` (admin token) empties the index and fills it from every active product, 50 at a time, and answers `{ "indexed": n }`. Use it for a new search database, after `npm run db:seed`, or whenever results look wrong. While it runs, searches can miss products it has not reached yet. | ||
| + | ||
| +Image uploads and checkout do not touch the index: they change nothing search matches on. | ||
Changed.rebasely/kb/decisions/0001-d1-for-shop-data.md+2−1
| @@ -2,7 +2,8 @@ | ||
| id: 0001-d1-for-shop-data | ||
| type: decision | ||
| owner: justin | ||
| -status: accepted | ||
| +status: superseded | ||
| +superseded-by: 0011-search-index-in-its-own-database | ||
| reviewed: 2026-10-01 | ||
| --- | ||
| # 0001 One D1 database for the shop's data | ||
Added.rebasely/kb/decisions/0011-search-index-in-its-own-database.md+31−0
| @@ -0,0 +1,31 @@ | ||
| +--- | ||
| +id: 0011-search-index-in-its-own-database | ||
| +type: decision | ||
| +owner: justin | ||
| +status: accepted | ||
| +reviewed: 2026-10-16 | ||
| +--- | ||
| +# 0011 The search index gets its own D1 database | ||
| + | ||
| +## Context | ||
| + | ||
| +[[0001-d1-for-shop-data]] put everything in one D1 database, `storefront`, so checkout could take stock and create an order in one batch. Shoppers now need to search the catalog, and D1 has full text search through SQLite's FTS5, so the obvious place for the index is an FTS5 table next to `products`. | ||
| + | ||
| +Two things argue against that: | ||
| + | ||
| +- A D1 database runs one query at a time. Search is the busiest read the shop will have, since the search box asks on every pause in typing, and each request ranks matches with `bm25()`. In the same database, those queries wait in the same line as checkout's batches and the expiry cron's writes. | ||
| +- The index is derived data. It is rebuilt from `products` whenever the tokenizer changes, and nothing in it needs keeping. In the shop's database, changing it would mean a migration next to the ones that hold orders, and every rebuild would run its deletes and inserts on the database checkout writes to. | ||
| + | ||
| +## Decision | ||
| + | ||
| +The shop's data stays in one database, `storefront`, with everything [[0001-d1-for-shop-data]] said about migrations and batches. The search index lives in a second database, `storefront-search` (binding `SEARCH`), with its own migrations in `search/migrations/`. It holds only the text search matches on: a product's id, name and description, for active products. | ||
| + | ||
| +storefront-api writes the index right after every product write, and `POST /admin/search/rebuild` refills it from the catalog. A search finds ids in the index and reads the products from `storefront`. See [[search]]. | ||
| + | ||
| +## Consequences | ||
| + | ||
| +- Search load cannot slow down checkout, and the index can be dropped, retuned and rebuilt without touching the database that holds orders. | ||
| +- A product write is now two writes in two databases, with no transaction across them. If the index write fails, the merchant gets `503 search_index_failed` and sends the same request again; the upsert makes that safe. Anything that drifts anyway is fixed by a rebuild. | ||
| +- A stale index can only miss a product or return one that no longer matches. It can never show a wrong price or an inactive product, because results always come from the catalog. | ||
| +- Writes that bypass the API, such as `scripts/seed.sql`, do not reach the index. Run a rebuild after them. | ||
| +- Still one writer: only storefront-api binds either database ([[0006-one-writer-for-the-database]]). | ||
Changed.rebasely/kb/guides/local-dev.md+9−2
| @@ -6,7 +6,7 @@ | ||
| - wrangler.jsonc | ||
| - workers/*/wrangler.jsonc | ||
| - scripts/seed.sql | ||
| -reviewed: 2026-10-12 | ||
| +reviewed: 2026-10-16 | ||
| --- | ||
| # Running the shop locally | ||
| @@ -16,10 +16,17 @@ | ||
| ```bash | ||
| npm install | ||
| -npm run db:migrate # applies migrations/ to the local D1 | ||
| +npm run db:migrate # applies migrations/ and search/migrations/ to the local D1s | ||
| npm run db:seed # six products, one of them out of stock | ||
| echo 'ADMIN_TOKEN=dev-admin-token' > .dev.vars | ||
| npm run dev # storefront-api on localhost:8787 | ||
| +``` | ||
| + | ||
| +The seed writes straight to the database, so the search index is empty until you rebuild it: | ||
| + | ||
| +```bash | ||
| +curl -X POST localhost:8787/admin/search/rebuild -H 'authorization: Bearer dev-admin-token' | ||
| +curl 'localhost:8787/search?q=linen' | ||
| ``` | ||
| `curl localhost:8787/products` lists all six; the candle is out of stock, so adding it to a cart fails. Add `STRIPE_SECRET_KEY` (a test mode secret key) to `.dev.vars` to try checkout. | ||
Changed.rebasely/kb/guides/testing.md+3−2
| @@ -4,7 +4,7 @@ | ||
| owner: justin | ||
| describes: | ||
| - test/helpers/** | ||
| -reviewed: 2026-10-09 | ||
| +reviewed: 2026-10-16 | ||
| --- | ||
| # Running and writing tests | ||
| @@ -20,6 +20,7 @@ | ||
| `test/helpers/app.ts` builds the real fetch handler from `createApp` with a fresh environment per test: | ||
| - `DB` is `TestD1` (`test/helpers/d1.ts`), a small D1 stand-in over `node:sqlite` with every file in `migrations/` applied in order and foreign keys on. It covers `prepare`, `bind`, `first`, `all`, `run` and `batch`, and a batch is a transaction, as in D1. | ||
| +- `SEARCH` is a second `TestD1` with `search/migrations/` applied (`testSearchDb()`). Node's SQLite has FTS5, so the search tests run the real `MATCH` and `bm25()` queries. | ||
| - `IMAGES` is `TestR2` (`test/helpers/r2.ts`), an in-memory bucket whose `objects` map tests can inspect, and `IMAGES_URL` is `https://images.shop.test`. | ||
| - `ORDER_EVENTS` is `TestQueue` (`test/helpers/queue.ts`), which keeps every message in `sent`. | ||
| - Payments are `FakePayments` (`test/helpers/payments.ts`), which records every charge, fails the next one when `failNext` is set, and refuses to cancel the intents in `uncancellable`. `test/stripe-payments.test.ts` checks the real Stripe adapter against a fake `fetch`, so no test calls Stripe. | ||
| @@ -29,4 +30,4 @@ | ||
| ## Writing one | ||
| -Test behavior through the API rather than the modules, the way a client sees it. When code calls a D1 method the stand-in does not have yet, add it to `TestD1` in the same change. A new migration needs nothing: the stand-in applies whatever is in `migrations/`. See [[0001-d1-for-shop-data]] for why the SQL is shared. | ||
| +Test behavior through the API rather than the modules, the way a client sees it. When code calls a D1 method the stand-in does not have yet, add it to `TestD1` in the same change. A new migration needs nothing: the stand-ins apply whatever is in `migrations/` and `search/migrations/`. See [[0011-search-index-in-its-own-database]] for why there are two databases. | ||
Changed.rebasely/kb/index.md+3−1
| @@ -5,6 +5,7 @@ | ||
| ## Concepts | ||
| - [[catalog]]: products, prices and stock, and how the merchant edits them. | ||
| +- [[search]]: full text search over the catalog, and how the index stays in step. | ||
| - [[carts]]: what a cart holds and the limits it enforces. | ||
| - [[product-images]]: how images get into R2 and back out. | ||
| - [[checkout]]: how a cart becomes an order, and how stock and payment are handled. | ||
| @@ -14,7 +15,7 @@ | ||
| ## Decisions | ||
| -- [[0001-d1-for-shop-data]]: one D1 database for everything the shop sells. | ||
| +- [[0001-d1-for-shop-data]]: one D1 database for everything the shop sells (superseded). | ||
| - [[0002-serve-images-through-the-api]]: images were served by storefront-api (superseded). | ||
| - [[0003-money-in-whole-cents]]: prices and totals are integers. | ||
| - [[0004-take-stock-at-checkout]]: carts check stock, checkout takes it. | ||
| @@ -23,6 +24,7 @@ | ||
| - [[0007-at-least-once-with-idempotency-keys]]: repeats are expected and harmless. | ||
| - [[0008-images-from-the-bucket-domain]]: browsers load images from R2's custom domain. | ||
| - [[0009-cancel-unpaid-orders-after-30-minutes]]: unpaid orders give their stock back. | ||
| +- [[0011-search-index-in-its-own-database]]: the shop's data in one database, the search index in a second. | ||
| ## Guides | ||
ChangedREADME.md+5−1
| @@ -1,6 +1,6 @@ | ||
| # storefront | ||
| -The API behind a small online shop, running on Cloudflare Workers. Products, carts and orders live in D1, product images in R2, and payments go through Stripe. | ||
| +The API behind a small online shop, running on Cloudflare Workers. Products, carts and orders live in D1, the search index in a second D1 database, product images in R2, and payments go through Stripe. | ||
| There are three Workers: | ||
| @@ -27,6 +27,7 @@ | ||
| npm run db:seed | ||
| echo 'ADMIN_TOKEN=dev-admin-token' > .dev.vars | ||
| npm run dev | ||
| +curl -X POST localhost:8787/admin/search/rebuild -H 'authorization: Bearer dev-admin-token' | ||
| ``` | ||
| Product images are served from the bucket's custom domain, `IMAGES_URL` in `wrangler.jsonc`. Checkout needs Stripe: add `STRIPE_SECRET_KEY` to `.dev.vars` with a secret key from your Stripe account's test mode. | ||
| @@ -46,8 +47,10 @@ | ||
| | --- | --- | | ||
| | `GET /products` | Active products by name, a page at a time: `?limit=` (1 to 100, default 24) and `?after=` the previous page's `next` | | ||
| | `GET /products/:id` | One active product | | ||
| +| `GET /search` | Active products matching `?q=`, best match first, up to `?limit=` | | ||
| | `PUT /admin/products/:id` | Create or replace a product (admin token) | | ||
| | `PUT /admin/products/:id/image` | Upload the product's image (admin token) | | ||
| +| `POST /admin/search/rebuild` | Refill the search index from the catalog (admin token) | | ||
| | `POST /carts` | A new, empty cart | | ||
| | `GET /carts/:id` | A cart with its lines and totals | | ||
| | `PUT /carts/:id/items/:product` | Set a quantity, `{ "quantity": n }` | | ||
| @@ -63,6 +66,7 @@ | ||
| src/ storefront-api: router, catalog, carts, images, checkout, orders | ||
| workers/ the other Workers, each with its own wrangler.jsonc | ||
| migrations/ D1 schema, applied in order | ||
| +search/ the search index's schema (search/migrations/) | ||
| test/ node --test suites and the D1 and R2 stand-ins | ||
| .rebasely/ architecture map, knowledge base and CI workflow | ||
| ``` | ||
Changedpackage.json+1−1
| @@ -12,7 +12,7 @@ | ||
| "test": "node --disable-warning=ExperimentalWarning --test --test-concurrency=1 \"test/**/*.test.ts\"", | ||
| "typecheck": "tsc", | ||
| "dev": "npx wrangler dev", | ||
| - "db:migrate": "npx wrangler d1 migrations apply storefront --local", | ||
| + "db:migrate": "npx wrangler d1 migrations apply storefront --local && npx wrangler d1 migrations apply storefront-search --local", | ||
| "db:seed": "npx wrangler d1 execute storefront --local --file scripts/seed.sql" | ||
| }, | ||
| "devDependencies": { | ||
Addedsearch/migrations/0001_products_fts.sql+13−0
| @@ -0,0 +1,13 @@ | ||
| +-- The search index, in its own D1 database (storefront-search). One row per active product with | ||
| +-- the text shoppers search: its name and description. Everything else about a product is read | ||
| +-- from the storefront database, which stays the source of truth. The index can be emptied and | ||
| +-- rebuilt from the catalog at any time with POST /admin/search/rebuild. | ||
| +-- | ||
| +-- porter stems English words (aprons matches apron); unicode61 with remove_diacritics 2 lets | ||
| +-- creme match Crème. | ||
| +CREATE VIRTUAL TABLE products_fts USING fts5( | ||
| + product_id UNINDEXED, | ||
| + name, | ||
| + description, | ||
| + tokenize = 'porter unicode61 remove_diacritics 2' | ||
| +); | ||
Changedsrc/app.ts+20−1
| @@ -10,6 +10,7 @@ | ||
| import { getOrder } from "./orders.ts"; | ||
| import { stripePayments, type Payments } from "./payments.ts"; | ||
| import { Router } from "./router.ts"; | ||
| +import { indexProduct, parseQuery, rebuildIndex, searchProducts } from "./search.ts"; | ||
| export type Deps = { | ||
| now: () => number; | ||
| @@ -32,6 +33,12 @@ | ||
| return json(await listProducts(env.DB, images, parseLimit(q.get("limit")), q.get("after"))); | ||
| }); | ||
| + r.on("GET", "/search", async (req) => { | ||
| + const params = new URL(req.url).searchParams; | ||
| + const q = parseQuery(params.get("q")); | ||
| + return json({ query: q, products: await searchProducts(env.DB, env.SEARCH, images, q, parseLimit(params.get("limit"))) }); | ||
| + }); | ||
| + | ||
| r.on("GET", "/products/:id", async (_req, p) => { | ||
| const product = await getProduct(env.DB, p.id!, images); | ||
| if (!product || !product.active) throw new HttpError(404, "not_found", "No such product."); | ||
| @@ -41,7 +48,19 @@ | ||
| r.on("PUT", "/admin/products/:id", async (req, p) => { | ||
| requireAdmin(req, env.ADMIN_TOKEN); | ||
| const input = parseProductInput(p.id!, await readJson(req)); | ||
| - return json({ product: await upsertProduct(env.DB, input, deps.now(), images) }); | ||
| + const product = await upsertProduct(env.DB, input, deps.now(), images); | ||
| + try { | ||
| + await indexProduct(env.SEARCH, product); | ||
| + } catch (e) { | ||
| + console.error(JSON.stringify({ msg: "indexing the product failed", product: product.id, error: String(e) })); | ||
| + throw new HttpError(503, "search_index_failed", "The product was saved, but the search index was not updated. Send the same request again."); | ||
| + } | ||
| + return json({ product }); | ||
| + }); | ||
| + | ||
| + r.on("POST", "/admin/search/rebuild", async (req) => { | ||
| + requireAdmin(req, env.ADMIN_TOKEN); | ||
| + return json({ indexed: await rebuildIndex(env.DB, env.SEARCH) }); | ||
| }); | ||
| r.on("PUT", "/admin/products/:id/image", async (req, p) => { | ||
Changedsrc/catalog.ts+15−0
| @@ -90,6 +90,21 @@ | ||
| return row ? toProduct(row, images) : null; | ||
| } | ||
| +// The active products among `ids`, in the order given. Ids that are unknown or inactive are left | ||
| +// out. Search uses it to turn index hits into products with their current price and stock. | ||
| +export async function activeProducts(db: D1Database, ids: string[], images: ImageUrls): Promise<Product[]> { | ||
| + if (!ids.length) return []; | ||
| + const { results } = await db | ||
| + .prepare(`SELECT ${COLUMNS} FROM products WHERE active = 1 AND id IN (SELECT value FROM json_each(?))`) | ||
| + .bind(JSON.stringify(ids)) | ||
| + .all<Row>(); | ||
| + const byId = new Map(results.map((r) => [r.id, r])); | ||
| + return ids.flatMap((id) => { | ||
| + const row = byId.get(id); | ||
| + return row ? [toProduct(row, images)] : []; | ||
| + }); | ||
| +} | ||
| + | ||
| export type ProductInput = { | ||
| id: string; | ||
| name: string; | ||
Changedsrc/env.ts+1−0
| @@ -3,6 +3,7 @@ | ||
| export interface Env { | ||
| DB: D1Database; | ||
| + SEARCH: D1Database; // the catalog's search index (src/search.ts) | ||
| IMAGES: R2Bucket; | ||
| ORDER_EVENTS: Queue<OrderEvent>; | ||
| IMAGES_URL: string; // the bucket's custom domain, where browsers load images | ||
Addedsrc/search.ts+87−0
| @@ -0,0 +1,87 @@ | ||
| +// Catalog search: a full text index of product names and descriptions in its own D1 database | ||
| +// (SEARCH, search/migrations/). The storefront database stays the source of truth. The index | ||
| +// holds only the text search matches on, and every hit is read back from the catalog, so a | ||
| +// result always has the current price and stock and never includes an inactive product. | ||
| +// See .rebasely/kb/concepts/search.md. | ||
| + | ||
| +import { activeProducts, type Product } from "./catalog.ts"; | ||
| +import { HttpError } from "./http.ts"; | ||
| +import type { ImageUrls } from "./images.ts"; | ||
| + | ||
| +export const MAX_QUERY = 100; | ||
| +const MAX_TERMS = 8; | ||
| +const REBUILD_PAGE = 50; | ||
| + | ||
| +// Name matches count ten times as much as description matches. The first weight is for | ||
| +// product_id, which is not indexed. | ||
| +const RANK = "bm25(products_fts, 0.0, 10.0, 1.0)"; | ||
| + | ||
| +type Indexed = { id: string; name: string; description: string; active: boolean }; | ||
| + | ||
| +// Turns what a shopper typed into an FTS5 query. Each word becomes a quoted string, so FTS5 | ||
| +// syntax in the input (AND, NEAR, column filters, a stray quote) is just text, and the last word | ||
| +// matches as a prefix, so results keep up while someone is still typing. Null when nothing | ||
| +// searchable is left. | ||
| +export function toMatch(q: string): string | null { | ||
| + const words = q.normalize("NFKC").match(/[\p{L}\p{N}]+/gu)?.slice(0, MAX_TERMS) ?? []; | ||
| + if (!words.length) return null; | ||
| + return words.map((w, i) => (i === words.length - 1 ? `"${w}"*` : `"${w}"`)).join(" "); | ||
| +} | ||
| + | ||
| +export function parseQuery(value: string | null): string { | ||
| + const q = (value ?? "").trim(); | ||
| + if (!q || q.length > MAX_QUERY) throw new HttpError(400, "bad_query", `q is the text to search for, 1 to ${MAX_QUERY} characters.`); | ||
| + return q; | ||
| +} | ||
| + | ||
| +// The products matching `q`, best match first. | ||
| +export async function searchProducts(db: D1Database, search: D1Database, images: ImageUrls, q: string, limit: number): Promise<Product[]> { | ||
| + const match = toMatch(q); | ||
| + if (match === null) return []; | ||
| + const { results } = await search | ||
| + .prepare(`SELECT product_id FROM products_fts WHERE products_fts MATCH ? ORDER BY ${RANK} LIMIT ?`) | ||
| + .bind(match, limit) | ||
| + .all<{ product_id: string }>(); | ||
| + return activeProducts( | ||
| + db, | ||
| + results.map((r) => r.product_id), | ||
| + images, | ||
| + ); | ||
| +} | ||
| + | ||
| +// Brings one product's row in the index in line with the catalog: replaced when the product is | ||
| +// active, removed when it is not. Called after every write to the product. | ||
| +export async function indexProduct(search: D1Database, p: Indexed): Promise<void> { | ||
| + const remove = search.prepare("DELETE FROM products_fts WHERE product_id = ?").bind(p.id); | ||
src/search.ts line 55
| ||
| + if (!p.active) { | ||
| + await remove.run(); | ||
| + return; | ||
| + } | ||
| + await search.batch([ | ||
| + remove, | ||
| + search.prepare("INSERT INTO products_fts (product_id, name, description) VALUES (?, ?, ?)").bind(p.id, p.name, p.description), | ||
| + ]); | ||
| +} | ||
| + | ||
| +// Empties the index and fills it again from every active product in the catalog, a page at a | ||
| +// time. For a new search database, and for repairing one that has drifted. While it runs, | ||
| +// searches can miss products that have not been added back yet. | ||
| +export async function rebuildIndex(db: D1Database, search: D1Database): Promise<number> { | ||
| + await search.prepare("DELETE FROM products_fts").run(); | ||
| + let after = ""; | ||
| + let indexed = 0; | ||
| + for (;;) { | ||
| + const { results } = await db | ||
| + .prepare("SELECT id, name, description FROM products WHERE active = 1 AND id > ? ORDER BY id LIMIT ?") | ||
| + .bind(after, REBUILD_PAGE) | ||
| + .all<{ id: string; name: string; description: string }>(); | ||
| + if (!results.length) break; | ||
| + await search.batch( | ||
| + results.map((r) => search.prepare("INSERT INTO products_fts (product_id, name, description) VALUES (?, ?, ?)").bind(r.id, r.name, r.description)), | ||
| + ); | ||
| + indexed += results.length; | ||
| + after = results.at(-1)!.id; | ||
| + if (results.length < REBUILD_PAGE) break; | ||
| + } | ||
| + return indexed; | ||
| +} | ||
Changedtest/helpers/app.ts+2−2
| @@ -3,7 +3,7 @@ | ||
| import { createApp } from "../../src/app.ts"; | ||
| import type { Env } from "../../src/env.ts"; | ||
| import type { OrderEvent } from "../../src/events.ts"; | ||
| -import { testDb } from "./d1.ts"; | ||
| +import { testDb, testSearchDb } from "./d1.ts"; | ||
| import { FakePayments } from "./payments.ts"; | ||
| import { testQueue } from "./queue.ts"; | ||
| import { testBucket } from "./r2.ts"; | ||
| @@ -13,7 +13,7 @@ | ||
| export function setup() { | ||
| const events = testQueue<OrderEvent>(); | ||
| - const env: Env = { DB: testDb(), IMAGES: testBucket(), ORDER_EVENTS: events, IMAGES_URL: "https://images.shop.test", ADMIN_TOKEN, STRIPE_SECRET_KEY: "sk_test_unused" }; | ||
| + const env: Env = { DB: testDb(), SEARCH: testSearchDb(), IMAGES: testBucket(), ORDER_EVENTS: events, IMAGES_URL: "https://images.shop.test", ADMIN_TOKEN, STRIPE_SECRET_KEY: "sk_test_unused" }; | ||
| const payments = new FakePayments(); | ||
| let n = 0; | ||
| const app = createApp(env, { now: () => NOW, newId: () => `id-${++n}`, payments: () => payments }); | ||
Changedtest/helpers/d1.ts+14−6
| @@ -1,5 +1,5 @@ | ||
| -// A D1 stand-in over node:sqlite with every file in migrations/ applied, so tests run the | ||
| -// storefront's real SQL. It covers what the code calls: prepare, bind, first, all, run and batch. | ||
| +// A D1 stand-in over node:sqlite with every file in a migrations folder applied (migrations/ | ||
| +// unless told otherwise), so tests run the storefront's real SQL. It covers what the code calls: prepare, bind, first, all, run and batch. | ||
| // Foreign keys are on, as they are in D1. | ||
| import { readdirSync, readFileSync } from "node:fs"; | ||
| @@ -7,7 +7,9 @@ | ||
| import { DatabaseSync, type SQLInputValue } from "node:sqlite"; | ||
| import { fileURLToPath } from "node:url"; | ||
| -const MIGRATIONS = join(dirname(fileURLToPath(import.meta.url)), "../../migrations"); | ||
| +const ROOT = join(dirname(fileURLToPath(import.meta.url)), "../.."); | ||
| +const MIGRATIONS = join(ROOT, "migrations"); | ||
| +const SEARCH_MIGRATIONS = join(ROOT, "search/migrations"); | ||
| class Statement { | ||
| readonly db: DatabaseSync; | ||
| @@ -44,10 +46,10 @@ | ||
| export class TestD1 { | ||
| readonly sqlite = new DatabaseSync(":memory:"); | ||
| - constructor() { | ||
| + constructor(migrations = MIGRATIONS) { | ||
| this.sqlite.exec("PRAGMA foreign_keys = ON"); | ||
| - const files = readdirSync(MIGRATIONS).filter((f) => f.endsWith(".sql")).sort(); | ||
| - for (const f of files) this.sqlite.exec(readFileSync(join(MIGRATIONS, f), "utf8")); | ||
| + const files = readdirSync(migrations).filter((f) => f.endsWith(".sql")).sort(); | ||
| + for (const f of files) this.sqlite.exec(readFileSync(join(migrations, f), "utf8")); | ||
| } | ||
| prepare(sql: string): Statement { | ||
| @@ -71,4 +73,10 @@ | ||
| export function testDb(): D1Database { | ||
| return new TestD1() as unknown as D1Database; | ||
| +} | ||
| + | ||
| +// The search index database, with search/migrations/ applied. node:sqlite is built with FTS5, | ||
| +// as D1 is. | ||
| +export function testSearchDb(): D1Database { | ||
| + return new TestD1(SEARCH_MIGRATIONS) as unknown as D1Database; | ||
| } | ||
Addedtest/search.test.ts+101−0
| @@ -0,0 +1,101 @@ | ||
| +import assert from "node:assert/strict"; | ||
| +import { test } from "node:test"; | ||
| +import { toMatch } from "../src/search.ts"; | ||
| +import { ADMIN_TOKEN, setup } from "./helpers/app.ts"; | ||
| + | ||
| +const ids = (res: { body: { products: { id: string }[] } }) => res.body.products.map((p) => p.id); | ||
| + | ||
| +async function shop() { | ||
| + const s = setup(); | ||
| + await s.addProduct("linen-apron", { name: "Linen work apron", description: "Heavy linen with two front pockets and cross-back straps.", price_cents: 5200 }); | ||
| + await s.addProduct("canvas-tote", { name: "Waxed canvas tote", description: "A 20 litre tote with leather handles. Pairs well with the linen apron.", price_cents: 4800 }); | ||
| + await s.addProduct("enamel-mug", { name: "Enamel camp mug", description: "A 350 ml steel mug with a rolled rim.", price_cents: 1600 }); | ||
| + await s.addProduct("creme-soap", { name: "Crème fraîche soap", description: "Cold process, unscented.", price_cents: 900 }); | ||
| + return s; | ||
| +} | ||
| + | ||
| +test("toMatch quotes every word and matches the last one as a prefix", () => { | ||
| + assert.equal(toMatch("linen apr"), '"linen" "apr"*'); | ||
| + assert.equal(toMatch(" Mug! "), '"Mug"*'); | ||
| +}); | ||
| + | ||
| +test("toMatch turns FTS5 syntax into plain words", () => { | ||
| + assert.equal(toMatch('apron" OR name:mug NEAR(x'), '"apron" "OR" "name" "mug" "NEAR" "x"*'); | ||
| + assert.equal(toMatch("*** --- ()"), null); | ||
| +}); | ||
| + | ||
| +test("finds products by name and description, name matches first", async () => { | ||
| + const s = await shop(); | ||
| + const res = await s.call("GET", "/search?q=linen"); | ||
| + assert.equal(res.status, 200); | ||
| + assert.equal(res.body.query, "linen"); | ||
| + assert.deepEqual(ids(res), ["linen-apron", "canvas-tote"]); | ||
| +}); | ||
| + | ||
| +test("matches while the shopper is still typing, across word forms and accents", async () => { | ||
| + const s = await shop(); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=enam")), ["enamel-mug"]); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=mugs")), ["enamel-mug"]); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=creme")), ["creme-soap"]); | ||
| +}); | ||
| + | ||
| +test("results carry the catalog's current price and stock", async () => { | ||
| + const s = await shop(); | ||
| + const cartId = (await s.call("POST", "/carts")).body.cart.id; | ||
| + await s.call("PUT", `/carts/${cartId}/items/enamel-mug`, { quantity: 2 }); | ||
| + await s.call("POST", `/carts/${cartId}/checkout`, { | ||
| + email: "ada@park.test", | ||
| + ship_to: { name: "Ada Park", line1: "12 Alder St", city: "Portland", postal_code: "97205", country: "US" }, | ||
| + }); | ||
| + const [mug] = (await s.call("GET", "/search?q=mug")).body.products; | ||
| + assert.equal(mug.stock, 3); | ||
| + assert.equal(mug.price_cents, 1600); | ||
| +}); | ||
| + | ||
| +test("an edited product is found by its new text, a retired one not at all", async () => { | ||
| + const s = await shop(); | ||
| + await s.addProduct("enamel-mug", { name: "Enamel camp cup", description: "A 350 ml steel cup." }); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=mug")), []); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=cup")), ["enamel-mug"]); | ||
| + await s.addProduct("linen-apron", { name: "Linen work apron", active: false }); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=apron")), ["canvas-tote"]); | ||
| +}); | ||
| + | ||
| +test("limit caps the results and q is required", async () => { | ||
| + const s = await shop(); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=linen&limit=1")), ["linen-apron"]); | ||
| + assert.equal((await s.call("GET", "/search?q=%20")).body.error, "bad_query"); | ||
| + assert.equal((await s.call("GET", `/search?q=${"a".repeat(101)}`)).body.error, "bad_query"); | ||
| + assert.equal((await s.call("GET", "/search?q=linen&limit=0")).body.error, "bad_limit"); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=%2A%2A")), []); | ||
| +}); | ||
| + | ||
| +test("a rebuild puts every active product back in the index", async () => { | ||
| + const s = await shop(); | ||
| + // A product written straight to the catalog, as a seed script does, is not in the index yet. | ||
| + await s.env.DB.prepare( | ||
| + "INSERT INTO products (id, name, description, price_cents, stock, active, created_at, updated_at) VALUES ('wool-socks', 'Merino wool socks', '', 1800, 40, 1, 0, 0)", | ||
| + ).run(); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=socks")), []); | ||
| + | ||
| + assert.equal((await s.call("POST", "/admin/search/rebuild")).status, 401); | ||
| + const res = await s.call("POST", "/admin/search/rebuild", undefined, { authorization: `Bearer ${ADMIN_TOKEN}` }); | ||
| + assert.deepEqual(res.body, { indexed: 5 }); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=socks")), ["wool-socks"]); | ||
| + assert.deepEqual(ids(await s.call("GET", "/search?q=linen")), ["linen-apron", "canvas-tote"]); | ||
| +}); | ||
| + | ||
| +test("a failed index write is reported, and the product is still saved", async () => { | ||
| + const s = await shop(); | ||
| + const search = s.env.SEARCH; | ||
| + s.env.SEARCH = { | ||
| + prepare: (sql: string) => search.prepare(sql), | ||
| + batch: async () => { | ||
| + throw new Error("D1_ERROR: storefront-search is unavailable"); | ||
| + }, | ||
| + } as unknown as D1Database; | ||
| + const res = await s.call("PUT", "/admin/products/enamel-mug", { name: "Enamel camp cup", price_cents: 1600 }, s.admin); | ||
| + assert.equal(res.status, 503); | ||
| + assert.equal(res.body.error, "search_index_failed"); | ||
| + assert.equal((await s.call("GET", "/products/enamel-mug")).body.product.name, "Enamel camp cup"); | ||
| +}); | ||
Changedwrangler.jsonc+8−0
| @@ -12,6 +12,14 @@ | ||
| "database_name": "storefront", | ||
| "database_id": "", | ||
| "migrations_dir": "migrations" | ||
| + }, | ||
| + // The catalog's search index, apart from the shop's data so search reads never queue behind | ||
| + // checkout's writes. Create it with `npx wrangler d1 create storefront-search`. | ||
| + { | ||
| + "binding": "SEARCH", | ||
| + "database_name": "storefront-search", | ||
| + "database_id": "", | ||
| + "migrations_dir": "search/migrations" | ||
| } | ||
| ], | ||
| "r2_buckets": [{ "binding": "IMAGES", "bucket_name": "storefront-images" }], | ||
View catalog
1 component added, 1 changed, 1 connection added
- Added
- Changed
- Removed
What browsing and searching the shop touches, and where product images come from.
The view catalog at the head of this pull request, with what it adds, changes and removes from the merge base f6403bb. Select a component for what changed in it.
search
Database storefront-search: an FTS5 index of the names and descriptions of active products, for GET /search. Derived from the catalog in db, updated on every product write and refilled by POST /admin/search/rebuild, so it can be dropped and rebuilt at any time. Kept apart from db so search reads never queue behind checkout's writes. Schema in search/migrations/.
Added by this pull request.
Connections: storefront-api → search added.
Takes part in Searching the catalog.
Changes
- Changed: ~ view catalog: description
- Changed: ~ view catalog: components +search
- Changed: ~ storefront-api: description
- Added: + component search (d1)
- Added: + edge storefront-api → search (d1)
Knowledge base
- Catalog Changedconcepts/catalog.md · concept
- Search Addedconcepts/search.md · concept
- 0001 One D1 database for the shop's data Supersededdecisions/0001-d1-for-shop-data.md · decision · superseded by 0011-search-index-in-its-own-database
- 0011 The search index gets its own D1 database Addeddecisions/0011-search-index-in-its-own-database.md · decision
- Running the shop locally Changedguides/local-dev.md · guide
- Running and writing tests Changedguides/testing.md · guide
- storefront knowledge base Changedindex.md · index