Stop Context7 serving anti-pattern samples; collapse three doc copies into one - #15
Merged
franco-zalamena-iterable merged 2 commits intoAug 27, 2026
Merged
Conversation
The published Context7 library /iterable/iterable-sdk-skill was serving eval/transcripts/android/*.baseline.sample.md as authoritative answers. Those files are hand-authored *anti-pattern* samples — a query for initializeInBackground returned setAutoPushRegistration(true) alongside a manual registerForPush() (PITFALLS #6), and jwt-auth.baseline.sample.md puts setEmail inside the init callback (PITFALLS #2). `folders` alone was not holding: index results cited sources/, iterable-android/snapshot/, SKILL.md and PITFALLS.md despite folders being ["polished"]. Add iterable-android to folders so the intended surface is declared explicitly, and exclude *.sample.md / *.baseline.md so the eval fixtures cannot be served regardless of folder scoping. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repo carried the same 14 docs three times: sources/ (raw fetch),
polished/ (transform output), and iterable-android/snapshot/ (a
byte-identical copy of polished/ that CI enforced via snapshot:verify).
Only the third was ever read by the agent. The other two existed so that
CI could check them against each other, and snapshot:verify existed to
catch the copy step someone would inevitably forget.
Now there is one copy: iterable-android/reference/, written directly by
`pnpm refresh:docs` (fetch + transform in one pass, no intermediate on
disk). 12k lines of duplicated markdown deleted.
Pipeline:
- new src/refresh.ts replaces fetch.ts + polish-layer-a.ts
- snapshot.ts deleted (nothing to keep in sync)
- validate-polished.ts -> validate-reference.ts; it now also checks that
the corpus and pipeline/config agree on which slugs exist, so a config
rename that orphans a routing-table slug fails CI
- dropped the `snippets:` frontmatter manifest, its schema block, and
recompute-manifest.ts. It was ~12KB of hashes and line counts across
the corpus whose only consumer was the validator that checked it
against itself, and the agent had to skip past it on every read.
- dropped `layer`/`polished_at` frontmatter. polished_at regenerated on
every run, which meant the refresh workflow's `git diff` change-check
ALWAYS reported changes and opened a PR with an empty slug list even
when no doc had moved. Provenance now comes from source_sha, so a
no-op refresh is a genuine no-op (verified: 0 written, 14 unchanged).
CI:
- removed check:snippets and lint:chunking. Both hardcoded exit 0, so
neither could ever fail a build, and check:snippets pulled a Kotlin
1.9.24 + Java 17 toolchain into every run to produce warnings that
were skipped locally anyway.
- deleted publish-context7.yml: 4 runs, 4 failures, POSTing to a
placeholder URL with a secret that was never provisioned. Context7
crawls on its own schedule; this only produced red Xs.
- folded validate-plugins.yml into validate.yml (it existed to avoid the
Java setup that validate.yml no longer has).
Skill: SKILL.md reads from reference/ and no longer claims Context7 is
unpublished-and-to-be-avoided. Dropped the hardcoded "known latest 3.8.0"
floor — Maven is at 3.10.1 and it sat six lines under "never trust a
number baked into this file". Removed the TODO-PHASE-3 placeholder in
.context7-library-id, which nothing referenced.
REVIEW.md's docs-refresh flow is rewritten around the question a reviewer
can actually answer — does this new guidance contradict PITFALLS.md —
instead of the mechanical copy-and-verify steps that no longer exist.
Verified: bodies of all 14 docs byte-identical before/after; refresh:docs
idempotent on a second run; validate-reference fails correctly on bad
frontmatter, an orphaned file, and a missing slug.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
franco-zalamena-iterable
deleted the
fix/context7-exclude-eval-transcripts
branch
August 27, 2026 11:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Collapse to one doc corpus; drop CI that only served CI
Two commits on
fix/context7-exclude-eval-transcripts:53842b7— stop Context7 indexing the eval sample transcripts8c9091a— collapse three doc copies intoiterable-android/reference/1. Context7 was serving anti-pattern code as authoritative
The published library
/iterable/iterable-sdk-skill(High reputation, underIterable's name) was indexing
eval/transcripts/android/*.baseline.sample.md.Those are hand-authored deliberately wrong samples — the baseline arm the eval
scores the skill against.
A
query-docsforinitializeInBackgroundreturned 3 of 5 results from thosetranscripts, including:
setAutoPushRegistration(true)alongside a manualregisterForPush()(PITFALLS Fix Cursor README install steps so the skill loads on first try. #6)setEmailinside the init callback (PITFALLS Package as a Cursor plugin (skill + Context7 MCP) #2)initializeInBackgroundoverload that binds config to the callback slot (PITFALLS #18)3.7.0as the version to depend onfolders: ["polished"]was not being honoured — results citedsources/,iterable-android/snapshot/,SKILL.mdandPITFALLS.mdtoo. So the fix addsexcludeFilespatterns rather than relying on folder scoping alone.This is the only change here that affects anyone outside the repo. Worth
re-querying the index after the crawl to confirm the exclusion took.
2. Three copies of the same 14 docs
sources/(raw fetch) →polished/(transform output) →iterable-android/snapshot/(byte-identical copy ofpolished/).The agent only ever read the third. The first two existed so CI could diff them
against each other, and
snapshot:verifyexisted to catch the copy step areviewer would forget. That's the "nest where CI requires things that are just
there for CI" — the checks were real, but what they protected wasn't.
Now: one copy at
iterable-android/reference/, written directly bypnpm refresh:docs(fetch + transform in one pass, nothing intermediate ondisk). 12,192 lines deleted, 365 added.
Two real bugs this surfaced
The refresh workflow opened an empty PR on every run.
polished_atwasregenerated on every polish, so
git diff --quiet sources/ polished/alwaysreported changes even when no doc had moved. The PR would open with an empty
slug list. Verified before the fix: re-polishing untouched docs produced
"14 files changed, 14 insertions, 14 deletions" — all timestamps. Provenance now
comes from
source_sha, and a no-op refresh is genuinely a no-op ("0 written,14 unchanged").
publish-context7.ymlhad never once succeeded — 4 runs, 4 failures,POSTing to a placeholder URL with a secret that was never provisioned, fired on
every push touching
context7.json. Deleted; Context7 crawls on its own.What else went
snippets:frontmatter manifest — ~12KB of hashes and line counts acrossthe corpus. Its only consumer was the validator that checked it against
itself, and SKILL.md had to tell the agent to skip past it on every read.
check:snippetsandlint:chunking— both hardcodedexit 0, so neithercould fail a build.
check:snippetspulled Kotlin 1.9.24 + Java 17 into everyCI run to emit warnings that were skipped locally anyway (no
kotlincon PATH).validate-plugins.yml— folded intovalidate.yml, which no longer needsthe Java setup that justified a separate workflow.
.context7-library-id—TODO-PHASE-3/iterable-sdk-skill; nothing read it.snapshot.ts,fetch.ts,polish-layer-a.ts,recompute-manifest.ts.What CI checks now
typecheck && validate:reference && validate:plugins. Every gate can fail.validate:referencegained a check the old one didn't have: the corpus andpipeline/configmust agree on which slugs exist, so a config rename thatorphans a routing-table slug now fails instead of silently producing a dead link.
SKILL.md
reference/; no longer claims Context7 is unpublished andmust-not-be-called. It doesn't send the agent to Context7 either — the docs are
already on disk, so there's nothing to gain from a network round-trip for the
same content.
3.10.1. It sat six lines under "never trust a number baked into this file".
REVIEW.md
Rewritten around the question a reviewer can actually answer — does this new
guidance contradict
PITFALLS.md? — instead of the copy-and-verify mechanicsthat no longer exist. If the docs start recommending something a pitfall warns
against, that's the thing worth catching, and no gate can catch it.
Verification
stripped; the only changes are the removed frontmatter fields)
refresh:docsidempotent on a second runvalidate:referencefails correctly on bad frontmatter, an orphaned file, anda slug listed in config but missing from disk
pnpm check:allgreen from a clean clone with--frozen-lockfileAccepted trade-offs
"does this read like Iterable's docs", not a mechanical diff.
source_shastillpins the exact upstream blob if someone needs to fetch and compare by hand.
source_refcan vary across the corpus — skip-on-unchanged means a dockeeps the ref it was last fetched at.
pipeline/config'ssource.refis thelatest refresh point. Documented in the schema.
Not done / open
eval/left alone. 519 LOC + 4 hand-authored sample transcripts, never runin CI, no committed results. It's tied to the beta value case rather than the
repo's correctness, so it's your call — I didn't want to delete the evidence
base for "does the skill help" while cleaning up plumbing. The transcripts are
now excluded from Context7 either way.
pipeline/configsdk.tagand thereforesdk_min_versionacross all 14 docs. Maven is at 3.10.1 and the corpusdocuments 3.10.0. Bumping
sdk.tagrewrites frontmatter in all 14 files, so Ileft it as its own change.
franco-zalamena-iterable/iterable-sdk-skillsis 20+ commitsbehind and missing
.claude-plugin/plugin.json; archive or reset it.🤖 Generated with Claude Code