feat(search): replace lunr with Pagefind and a grouped-results search modal - #2093
Closed
haranrk wants to merge 11 commits into
Closed
feat(search): replace lunr with Pagefind and a grouped-results search modal#2093haranrk wants to merge 11 commits into
haranrk wants to merge 11 commits into
Conversation
The lunr-based `search` plugin ships its entire index to the browser on every page load, before the reader has typed anything: a 3.54 MB search_index.json (841 KB gzipped) plus a 39 KB search worker, measured on a 226-page build of this site. Pagefind builds a sharded index at build time and the browser fetches only the shards a query touches. `pagefind[bin]` (a 5.4 MB wheel) rather than `pagefind[extended]` (52.7 MB): the two differ only in CJK segmentation support, which this English-only site does not use. Neither needs a Node toolchain - the wheel carries a prebuilt binary, invoked as `python -m pagefind`. pagefind-bin is pinned separately from pagefind[bin]. The extra only declares `pagefind-bin<2,>=1`, and it is the binary, not the small Python wrapper, that determines the index format, the shipped pagefind.js and the CLI surface - so an unpinned binary would drift on every `pip install`, including in CI and on deploy.
Pagefind indexes only the pages carrying `data-pagefind-body`, and within them only that element's subtree. overrides/main.html puts the attribute on <article>, which keeps the header, navigation, table of contents and footer out of the index, and keeps the 3,140 generated API-reference files under docs/api-reference/ out entirely: MkDocs copies those verbatim rather than rendering them through the template, so they never receive the attribute. Exactly one page under that tree is indexed, the hand-written api-reference/index.md landing page. The scope assertion compares Pagefind's reported page count against a count of files carrying the attribute, both measured from the build that just ran, rather than against a hardcoded number. That number is not stable across environments: CI builds 226 pages, while a local checkout with untracked planning documents under docs/superpowers/ builds 234. A hardcoded expectation would fail for every developer. --exclude-selectors drops the permalink anchor, the pymdownx.tabbed label strip and the language-support badge before indexing. Pagefind concatenates adjacent inline elements without a separator, so without this those three fuse into tokens such as "ADKPython" and "PythonTypeScriptGoJava": 219 of them across 193 of the 226 indexed pages, measured on a control build with no exclude selectors. With the selectors excluded, zero. Tab contents stay indexed, so per-language content remains searchable. --include-characters keeps ".-@#+" inside tokens, so dotted and prefixed identifiers survive tokenization. Without it the identifier `google.adk.agents` is split into three unrelated words and cannot be searched as a unit. PAGEFIND_SKIP=1 skips indexing, for `mkdocs serve` authoring loops where re-indexing on every save is not worth it. Skipping logs a warning, so a --strict build cannot silently ship dead search.
The trigger button and the dialog it opens are split across the closing </header> tag, which looks arbitrary. It is the only placement that satisfies two constraints that pull in opposite directions, and it is the least obvious decision on this branch. First, the dialog must stay outside the seven elements Material's instant navigation replaces on every client-side navigation: announce, container, header-topic, outdated, logo, skip and tabs. Anything inside those is torn out and rebuilt, taking its event listeners with it, and the dialog would stop responding after the first in-site link. Second, the dialog must stay outside .md-header. custom.css declares `[data-md-color-scheme="slate"] .md-header *`, a blanket descendant selector at specificity 0-2-0 that beats the modal's own 0-1-1 rules and flattened the excerpt text, the section links and the <mark> accent on matched terms in dark mode. That rule is load-bearing for the real header, so the dialog moves out from under it rather than being fought with higher-specificity overrides. base.html renders the header block immediately before <div class="md-container" data-md-component="container">, so including the dialog after </header> makes it a sibling of both: outside .md-header, and outside every element instant navigation replaces. The trigger is a header control and is meant to follow header colours, so it stays inside.
What the reader gains: the search payload downloaded on every page load, before any query is typed, drops from 3.58 MB to zero - a 3.54 MB search_index.json plus a 39 KB worker, measured on a 226-page build. Pagefind fetches index shards only once there is a query. What does not change: which pages are searchable. lunr indexed 226 pages, Pagefind indexes 226, and the two URL sets are identical, so nothing findable before becomes unfindable. What the reader loses, for now: Material bound its global `/`, `s` and `f` search shortcuts inside mountSearch(), which no longer runs, so those three keys go inert. A later commit in this series rebinds them to the new dialog. The `#__search` toggle input base.html renders stays, even though nothing reads it any more. Material's toggle/index.ts resolves it eagerly at module scope, alongside the drawer toggle, and its getElement() throws a ReferenceError when the selector matches nothing. Removing the input would therefore take down every Material behaviour on the page, not just search.
build-docs.yaml is `on: pull_request` only, so on its own the verifier never guards the path that actually publishes the site. publish-docs.yaml runs `mkdocs gh-deploy --force`, which builds and pushes in a single step, leaving no point at which the workflow can inspect the output before it is live. An explicit `mkdocs build --strict` in front of it produces a site/ the verifier can run against, so check_never_indexed and check_fused_tokens now guard the deploy path as well as pull requests. CONTRIBUTING.md picks up PAGEFIND_SKIP, which was until now only discoverable from a comment in mkdocs.yml.
Removing the lunr search plugin also removed Material's global keyboard shortcuts, which were bound inside mountSearch(). Rebinding them keeps existing muscle memory working rather than silently dropping three shortcuts that readers of this site may already use. Also corrects the hook docstring, which still described the lunr plugin as enabled.
✅ Deploy Preview for adk-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
The trigger advertised Ctrl+K to every visitor, but the handler in search.js has always accepted Meta as well as Control, so the shortcut worked on macOS while the label denied it. aria-keyshortcuts was wrong there for the same reason. The hint is still server-rendered as Ctrl, so the HTML is identical for every visitor and stays cacheable, and an inline script immediately after the button corrects it during parse. It cannot live in search.js: that is a plain <script src> at the end of <body>, measured finishing 6ms after first paint on a throttled connection, so Mac readers would watch Ctrl repaint to Cmd. The inline version swaps 117ms before first paint at 200KB/s with 300ms latency. The hint ships with an inline display:none that the script clears once the label is correct, which also hands responsive control back to the stylesheet so it stays hidden below 60em. With JavaScript off it stays hidden, which is right: the shortcut would not work either. This mirrors how Starlight solves it on genkit.dev, including finding the modifier through the nested <kbd> structure rather than adding an attribute for it.
Contributor
Author
|
Superseded by #2096 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the built-in
searchplugin (lunr.js) with Pagefind, plus a search modal whose markup and behaviour live in this repo.Two problems, solved together:
Relevance. The lunr index is tokenized with the bare default separator (
[\s\-]+— whitespace and hyphens only), while 60% of the index sits inside<code>. Searchingappend_event,LlmAgent,google.adk.agentsorbefore_agent_callbackcurrently returns nothing. Separately, Pagefind and lunr both concatenate adjacent inline elements, so the language badge and tab-label strip fuse into single tokens —ADKPython× 154,PythonTypeScriptGoJava× 59, across 193 of 226 pages. A reader searchingPythonmisses all of them.Interface. Material's dropdown returns a flat list of section rows. This returns results grouped by page with the matching headings nested beneath, like genkit.dev and pydantic.dev/docs (both of which also use Pagefind).
The indexed page set is unchanged
The main risk in swapping search engines is quietly changing what is searchable. It did not happen:
mainZero pages added, zero dropped. Verified by extracting both sets from CI-equivalent builds (
git archive).Measured effects
search.js+search.css)append_event,LlmAgent,google.adk.agents,before_agent_callbackThe index is sharded, so a query fetches only the shards it touches (~162 KB to initialise, ~460 KB for a full cold search). A reader who never searches pays nothing beyond the ~9 KB.
How it works
A
on_post_buildhook runs the Pagefind binary over the rendered HTML insite_dir.data-pagefind-bodyon<article>scopes the index — and because Pagefind indexes only pages carrying that attribute, it also keeps the 3,140 generated API-reference files out, matching the current lunr scope exactly. The attribute is conditional on{% if page %}, which keeps it off404.html(MkDocs renders theme templates without a page in context).--exclude-selectors ".headerlink, .tabbed-labels, .language-support-tag"fixes the token fusion; tab contents stay indexed, so per-language content remains searchable.--include-characters ".-@#+"keepsgoogle.adk.agentsfrom splitting into three unrelated words.Pagefind is used as a query API rather than a widget, so result rendering, keyboard handling and styling are ours — which is what makes the page-grouped layout and the deferred loading possible.
Notable details
</header>boundary. The dialog must sit outside the seven elements instant navigation replaces (so it survives client-side navigation) and outside.md-header, becausecustom.css's[data-md-color-scheme="slate"] .md-header *at specificity 0-2-0 beats the modal's own rules and flattens every colour in dark mode. Only the split satisfies both.target="_self"on result links is load-bearing. Material intercepts internal links by stripping query and hash and testing the bare URL against its sitemap, so?pagefind-highlight=links would be intercepted — leaving the dialog open and preventing the arrival highlighter from running. Material's handler bails on a truthytarget.__searchtoggle input stays.toggle/index.tsresolves it eagerly at module scope and throws if missing, which would take down all Material JS.Ctrl/Cmd+K, plus/,sandfrebound — those were bound inside Material'smountSearch, which no longer runs, so they would otherwise have silently stopped working. Suppressed while typing in a field.aria-activedescendant; native<dialog>for focus trapping and Escape.pagefind-highlight.js, loaded only when the URL carries the param, and palette-aware via--md-typeset-mark-color).pagefind-binis pinned explicitly becausepagefind[bin]only constrains the binary to<2.Guardrails
hooks/pagefind_index.pyfails the build on: a missing or failing indexer, zero indexable pages, a page-count mismatch, and any--exclude-selectorsentry that has stopped matching (a stale-but-valid selector would otherwise silently let fused tokens back in).scripts/verify_search_index.pyruns after the build in bothbuild-docs.yamlandpublish-docs.yaml, asserting index scope, that nothing under the generated API-reference subtrees or404.htmlis indexed, and that no fused tokens survive. Counts are always measured, never hardcoded.PAGEFIND_SKIP=1skips indexing formkdocs serveauthoring loops; it logs a warning, so--strictbuilds fail while it is set.Testing
mkdocs build --strictclean; all 10 commits build individually (bisect-safe).Ctrl+K,//s/f, Escape, grouped results, arrow-key navigation,aria-activedescendant, full-document navigation on Enter, 167 terms marked on arrival, dark mode, and header layout from 1000px to 1920px.--exclude-selectors→ 219 tokens across 193 pages; with → 0.