Skip to content

Stop Context7 serving anti-pattern samples; collapse three doc copies into one - #15

Merged
franco-zalamena-iterable merged 2 commits into
mainfrom
fix/context7-exclude-eval-transcripts
Aug 27, 2026
Merged

Stop Context7 serving anti-pattern samples; collapse three doc copies into one#15
franco-zalamena-iterable merged 2 commits into
mainfrom
fix/context7-exclude-eval-transcripts

Conversation

@franco-zalamena-iterable

Copy link
Copy Markdown
Contributor

Collapse to one doc corpus; drop CI that only served CI

Two commits on fix/context7-exclude-eval-transcripts:

  1. 53842b7 — stop Context7 indexing the eval sample transcripts
  2. 8c9091a — collapse three doc copies into iterable-android/reference/

1. Context7 was serving anti-pattern code as authoritative

The published library /iterable/iterable-sdk-skill (High reputation, under
Iterable's name) was indexing eval/transcripts/android/*.baseline.sample.md.
Those are hand-authored deliberately wrong samples — the baseline arm the eval
scores the skill against.

A query-docs for initializeInBackground returned 3 of 5 results from those
transcripts, including:

folders: ["polished"] was not being honoured — results cited sources/,
iterable-android/snapshot/, SKILL.md and PITFALLS.md too. So the fix adds
excludeFiles patterns rather than relying on folder scoping alone.

This is the only change here that affects anyone outside the repo. Worth
re-querying the index after the crawl to confirm the exclusion took.

2. Three copies of the same 14 docs

sources/ (raw fetch) → polished/ (transform output) →
iterable-android/snapshot/ (byte-identical copy of polished/).

The agent only ever read the third. The first two existed so CI could diff them
against each other, and snapshot:verify existed to catch the copy step a
reviewer would forget. That's the "nest where CI requires things that are just
there for CI" — the checks were real, but what they protected wasn't.

Now: one copy at iterable-android/reference/, written directly by
pnpm refresh:docs (fetch + transform in one pass, nothing intermediate on
disk). 12,192 lines deleted, 365 added.

Two real bugs this surfaced

The refresh workflow opened an empty PR on every run. polished_at was
regenerated on every polish, so git diff --quiet sources/ polished/ always
reported changes even when no doc had moved. The PR would open with an empty
slug list. Verified before the fix: re-polishing untouched docs produced
"14 files changed, 14 insertions, 14 deletions" — all timestamps. Provenance now
comes from source_sha, and a no-op refresh is genuinely a no-op ("0 written,
14 unchanged").

publish-context7.yml had never once succeeded — 4 runs, 4 failures,
POSTing to a placeholder URL with a secret that was never provisioned, fired on
every push touching context7.json. Deleted; Context7 crawls on its own.

What else went

  • snippets: frontmatter manifest — ~12KB of hashes and line counts across
    the corpus. Its only consumer was the validator that checked it against
    itself, and SKILL.md had to tell the agent to skip past it on every read.
  • check:snippets and lint:chunking — both hardcoded exit 0, so neither
    could fail a build. check:snippets pulled Kotlin 1.9.24 + Java 17 into every
    CI run to emit warnings that were skipped locally anyway (no kotlinc on PATH).
  • validate-plugins.yml — folded into validate.yml, which no longer needs
    the Java setup that justified a separate workflow.
  • .context7-library-idTODO-PHASE-3/iterable-sdk-skill; nothing read it.
  • snapshot.ts, fetch.ts, polish-layer-a.ts, recompute-manifest.ts.

What CI checks now

typecheck && validate:reference && validate:plugins. Every gate can fail.
validate:reference gained a check the old one didn't have: the corpus and
pipeline/config must agree on which slugs exist, so a config rename that
orphans a routing-table slug now fails instead of silently producing a dead link.

SKILL.md

  • Reads from reference/; no longer claims Context7 is unpublished and
    must-not-be-called. It doesn't send the agent to Context7 either — the docs are
    already on disk, so there's nothing to gain from a network round-trip for the
    same content.
  • Dropped the hardcoded "known latest: 3.8.0" floor. Maven Central is at
    3.10.1. It sat six lines under "never trust a number baked into this file".

REVIEW.md

Rewritten around the question a reviewer can actually answer — does this new
guidance contradict PITFALLS.md?
— instead of the copy-and-verify mechanics
that no longer exist. If the docs start recommending something a pitfall warns
against, that's the thing worth catching, and no gate can catch it.

Verification

  • Bodies of all 14 docs byte-identical before/after (diffed with frontmatter
    stripped; the only changes are the removed frontmatter fields)
  • refresh:docs idempotent on a second run
  • Forced a rewrite of 2 docs: bodies reproduced exactly
  • validate:reference fails correctly on bad frontmatter, an orphaned file, and
    a slug listed in config but missing from disk
  • pnpm check:all green from a clean clone with --frozen-lockfile
  • Both workflow YAMLs parse; step sequences check out

Accepted trade-offs

  • No committed raw copy to diff the transform against. Fidelity review is now
    "does this read like Iterable's docs", not a mechanical diff. source_sha still
    pins the exact upstream blob if someone needs to fetch and compare by hand.
  • source_ref can vary across the corpus — skip-on-unchanged means a doc
    keeps the ref it was last fetched at. pipeline/config's source.ref is the
    latest refresh point. Documented in the schema.

Not done / open

  • eval/ left alone. 519 LOC + 4 hand-authored sample transcripts, never run
    in CI, no committed results. It's tied to the beta value case rather than the
    repo's correctness, so it's your call — I didn't want to delete the evidence
    base for "does the skill help" while cleaning up plumbing. The transcripts are
    now excluded from Context7 either way.
  • Version pins still say 3.7.0pipeline/config sdk.tag and therefore
    sdk_min_version across all 14 docs. Maven is at 3.10.1 and the corpus
    documents 3.10.0. Bumping sdk.tag rewrites frontmatter in all 14 files, so I
    left it as its own change.
  • The fork franco-zalamena-iterable/iterable-sdk-skills is 20+ commits
    behind and missing .claude-plugin/plugin.json; archive or reset it.

🤖 Generated with Claude Code

The published Context7 library /iterable/iterable-sdk-skill was serving
eval/transcripts/android/*.baseline.sample.md as authoritative answers.
Those files are hand-authored *anti-pattern* samples — a query for
initializeInBackground returned setAutoPushRegistration(true) alongside a
manual registerForPush() (PITFALLS #6), and jwt-auth.baseline.sample.md
puts setEmail inside the init callback (PITFALLS #2).

`folders` alone was not holding: index results cited sources/,
iterable-android/snapshot/, SKILL.md and PITFALLS.md despite folders
being ["polished"]. Add iterable-android to folders so the intended
surface is declared explicitly, and exclude *.sample.md / *.baseline.md
so the eval fixtures cannot be served regardless of folder scoping.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repo carried the same 14 docs three times: sources/ (raw fetch),
polished/ (transform output), and iterable-android/snapshot/ (a
byte-identical copy of polished/ that CI enforced via snapshot:verify).
Only the third was ever read by the agent. The other two existed so that
CI could check them against each other, and snapshot:verify existed to
catch the copy step someone would inevitably forget.

Now there is one copy: iterable-android/reference/, written directly by
`pnpm refresh:docs` (fetch + transform in one pass, no intermediate on
disk). 12k lines of duplicated markdown deleted.

Pipeline:
  - new src/refresh.ts replaces fetch.ts + polish-layer-a.ts
  - snapshot.ts deleted (nothing to keep in sync)
  - validate-polished.ts -> validate-reference.ts; it now also checks that
    the corpus and pipeline/config agree on which slugs exist, so a config
    rename that orphans a routing-table slug fails CI
  - dropped the `snippets:` frontmatter manifest, its schema block, and
    recompute-manifest.ts. It was ~12KB of hashes and line counts across
    the corpus whose only consumer was the validator that checked it
    against itself, and the agent had to skip past it on every read.
  - dropped `layer`/`polished_at` frontmatter. polished_at regenerated on
    every run, which meant the refresh workflow's `git diff` change-check
    ALWAYS reported changes and opened a PR with an empty slug list even
    when no doc had moved. Provenance now comes from source_sha, so a
    no-op refresh is a genuine no-op (verified: 0 written, 14 unchanged).

CI:
  - removed check:snippets and lint:chunking. Both hardcoded exit 0, so
    neither could ever fail a build, and check:snippets pulled a Kotlin
    1.9.24 + Java 17 toolchain into every run to produce warnings that
    were skipped locally anyway.
  - deleted publish-context7.yml: 4 runs, 4 failures, POSTing to a
    placeholder URL with a secret that was never provisioned. Context7
    crawls on its own schedule; this only produced red Xs.
  - folded validate-plugins.yml into validate.yml (it existed to avoid the
    Java setup that validate.yml no longer has).

Skill: SKILL.md reads from reference/ and no longer claims Context7 is
unpublished-and-to-be-avoided. Dropped the hardcoded "known latest 3.8.0"
floor — Maven is at 3.10.1 and it sat six lines under "never trust a
number baked into this file". Removed the TODO-PHASE-3 placeholder in
.context7-library-id, which nothing referenced.

REVIEW.md's docs-refresh flow is rewritten around the question a reviewer
can actually answer — does this new guidance contradict PITFALLS.md —
instead of the mechanical copy-and-verify steps that no longer exist.

Verified: bodies of all 14 docs byte-identical before/after; refresh:docs
idempotent on a second run; validate-reference fails correctly on bad
frontmatter, an orphaned file, and a missing slug.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@franco-zalamena-iterable
franco-zalamena-iterable merged commit 0d122a6 into main Aug 27, 2026
1 check passed
@franco-zalamena-iterable
franco-zalamena-iterable deleted the fix/context7-exclude-eval-transcripts branch August 27, 2026 11:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant