feat(search): replace the lunr search index with Pagefind - #2085
Closed
haranrk wants to merge 1 commit into
Closed
Conversation
✅ Deploy Preview for adk-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
haranrk
force-pushed
the
pagefind-search
branch
2 times, most recently
from
August 6, 2026 23:11
e24c46e to
6645a9c
Compare
MkDocs' built-in `search` plugin builds a single lunr.js index and the
browser downloads all of it before the first query. On this site that is
3.5 MB (843 KB gzipped) across 2,413 records, and it is fetched on every
page load whether or not the reader ever opens search.
Pagefind instead shards the index at build time and ships a WASM query
engine, so the browser fetches only the shards a query touches.
Measured against a build of the previous commit, serving both locally and
reading the server access log (worker-initiated fetches do not show up in
page-level devtools events):
before (lunr) after (Pagefind)
on page load 855 KB gzip 46 KB gzip
on first search 0 KB 378 KB gzip
total, if searching 855 KB gzip 425 KB gzip
Readers who never search pay 95% less. Readers who do pay about half.
Implementation
--------------
`hooks/pagefind_index.py` runs the indexer over the rendered HTML in
`on_post_build`, adding ~2.2s to a ~31s build. Pagefind is distributed as
a PyPI wheel with a prebuilt binary, so it installs from requirements.txt
and needs no Node toolchain in CI.
Index scope is controlled by `data-pagefind-body` on the article element,
added via a `container` block override that mirrors the theme's own. Once
that attribute exists anywhere, Pagefind indexes only pages carrying it,
which keeps the 3,130 generated Dokka/Javadoc/TypeDoc files out of the
index. Those were never in the lunr index either, so search scope is
unchanged: 232 pages in, 232 pages indexed.
The UI is Pagefind's Component UI (`<pagefind-modal>`), which as of
Pagefind 1.5.0 supersedes the older Default UI. It is themed entirely
through Material's palette: the `--pf-*` variables are mapped onto
`--md-*` on `body` rather than `:root`, because Material declares its
light palette at `:root` but its dark palette on `[data-md-color-scheme]`
on `<body>`, so mapping at `:root` would stay light after the palette
toggle. Custom properties are exempt from Pagefind's `all: initial` host
reset, so dark mode needs no JavaScript.
The components sit in the header, outside every region Material's instant
navigation swaps, so they stay mounted across client-side navigation.
Index quality
-------------
Pagefind concatenates adjacent inline elements without a separator, so
sibling elements authored without whitespace fuse into one token. Three
constructs here hit that, and `--exclude-selectors` removes them:
.headerlink "¶" permalink anchors, fused into headings on
230 of 232 pages
.tabbed-labels `<label>Python</label><label>Java</label>` from
pymdownx.tabbed, indexed as "PythonJava"
.language-support-tag the "Supported in ADK" badge, which produced
"ADKPythonTypeScriptGoJava"
Pages with fused language tokens drop from 109/232 to 5/232, and all five
remaining are genuine identifiers ("RxJava", "pdfArtifactJava"). Tab
contents stay indexed, so per-language content remains searchable.
`--include-characters ".-@#+"` keeps dotted and prefixed identifiers
intact; without it "google.adk.agents" degrades to three bare words.
Set PAGEFIND_SKIP=1 to skip indexing, e.g. for link-checking builds.
haranrk
force-pushed
the
pagefind-search
branch
from
August 7, 2026 18:58
6645a9c to
df8821c
Compare
Contributor
Author
|
Superseded by #2096 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces MkDocs' built-in lunr.js search with Pagefind.
The built-in
searchplugin builds one lunr index and the browser downloads all of it before the first query. On this site that's 3.5 MB (843 KB gzipped), fetched on every page load whether or not the reader opens search. Pagefind shards the index at build time and ships a WASM query engine, so the browser fetches only the shards a query touches.Measured impact
Built the previous commit as a baseline, served both locally, and read the server access log — Pagefind fetches its index from a Web Worker, and those requests don't appear in page-level devtools events.
Readers who never search pay 95% less. Readers who do pay about half.
One honest caveat: unlike lunr, Pagefind's cost isn't one-time — it fetches more fragments as you scroll results, so a session with many queries eventually crosses over.
How it works
hooks/pagefind_index.pyruns the indexer inon_post_build: +2.2s on a ~31s build.requirements.txtwith no Node toolchain in CI. Amanylinuxwheel exists forubuntu-latest.data-pagefind-bodyon the article, via acontainerblock override mirroring the theme's own. Once that attribute exists anywhere, Pagefind indexes only pages carrying it — which keeps the 3,130 generated Dokka/Javadoc/TypeDoc files out. Those were never in the lunr index either, so search scope is unchanged: 232 pages in, 232 indexed.<pagefind-modal>), which supersedes the older Default UI as of Pagefind 1.5.0. KeepsCmd/Ctrl+Kand adds WAI-ARIA compliance.--pf-*onto Material's--md-*onbody, not:root— Material declares its light palette at:rootbut its dark palette on[data-md-color-scheme]on<body>, so mapping at:rootwould stay light after the palette toggle. Custom properties are exempt from Pagefind'sall: initialhost reset, so dark mode needs no JavaScript.PAGEFIND_SKIP=1skips indexing, e.g. for link-checking builds.Index quality fixes
Pagefind concatenates adjacent inline elements without a separator, so siblings authored without whitespace fuse into one token. It does not fix this for free — three constructs here hit it, removed via
--exclude-selectors:.headerlink¶permalink anchors fused into headings on 230 of 232 pages.tabbed-labels<label>Python</label><label>Java</label>→PythonJava.language-support-tagADKPythonTypeScriptGoJavaPages with fused language tokens: 109/232 → 5/232, and all five remaining are genuine identifiers (
RxJava,pdfArtifactJavais a real variable in a Java sample). Tab contents stay indexed, so per-language content remains searchable — verifiedKotlin agent→ 25 results,Java streaming→ 34.--include-characters ".-@#+"keeps dotted identifiers intact; without itgoogle.adk.agentsdegrades to three bare words (133 results with it).Verification
Driven in headless Chrome against the built site:
Ctrl+Kopens, queries return results, and¶permalink anchors no longer appear in indexed headingssearch_index.jsonor the lunr workerdialogBgresolves torgb(30, 33, 41)with light text after the palette togglelocalhostbecause Material gates interception onsitemap.xml, whose URLs are absolute toadk.dev. I rebuilt with a localsite_urlto reproduce production conditions, and confirmed productionadk.devdoes use it.mkdocs build --strictpasses;mkdocs serveworks (search functions in dev too)PAGEFIND_SKIP=1, and a clear error if the dependency is missingNote for reviewers
Screenshots of the modal in light and dark mode can be attached if useful. Licensing is unchanged: Pagefind is MIT, same as mkdocs-material and lunr.