Skip to content

build: Fix build cache of page content - #1982

Open
marcleblanc2 wants to merge 4 commits into
mainfrom
marc/build/turbopack-and-contentlayer-cache
Open

build: Fix build cache of page content#1982
marcleblanc2 wants to merge 4 commits into
mainfrom
marc/build/turbopack-and-contentlayer-cache

Conversation

@marcleblanc2

@marcleblanc2 marcleblanc2 commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Every Vercel build spent 25-40s of its ~2 minutes re-rendering all 506 MDX documents, even when they didn't change, because contentlayer2's cache lived in .contentlayer/.cache and Vercel only keeps .next/cache between builds

Changes

  • pnpm run build now runs contentlayer2 as its own step (dev/build-content.mjs) with its cache under .next/cache, so a deploy re-renders only the documents that changed

  • Upstream Contentlayer2 decides "changed" by comparing each file's mtime, but Vercel builds use fresh clone, which sets every mtime to the clone time, so the restored cache never hit

  • This removes the only reason next build was still holding onto webpack, so we switched it to using Vercel's Turbopack for faster builds and better caching

  • This upgrade to Turbopack required 2 fixes in our code:

    • src/data/redirects.ts is imported instead of require()
    • contentlayer.config.ts imports the Shiki theme as JSON instead of by a relative path

Vercel build times

Step Cold build on main Warm build on this branch
contentlayer ~40s 3s
compile ~19s 4.7s
TypeScript 7s 4.5s
static pages ~37s ~27s (1512 pages since #1945)
build traces ~8s –
Build Completed 2m 49s

  • The first build after any change to this script is still cold (its restored cache holds the old hashes)

    • Every build after is warm

  • Also checked locally:

    • Touching every .mdx without changing content still hits the cache (8.5s → 2.8s)
    • Changing one file re-renders only that file
    • next start serves docs pages, /api/og, .md rewrites, redirects, sitemap, /api/releases, 404
    • Shiki output identical to webpack
    • tsc, lint and check-redirects unchanged from main

@vercel

vercel Bot commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
sourcegraph-docs Ignored Ignored Preview Sep 17, 2026 10:10am UTC

Request Review

@marcleblanc2 marcleblanc2 changed the title build: Turbopack for next build, keep the contentlayer cache between Vercel deploys build: Cache contentlayer on Vercel, switch to Turbopack Sep 17, 2026
@marcleblanc2
marcleblanc2 force-pushed the marc/build/turbopack-and-contentlayer-cache branch from 0818b2f to b844daf Compare September 17, 2026 04:28
@github-actions

github-actions Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Open the Diff Tour for this PR in Sourcegraph

@github-actions

This comment has been minimized.

Sign in to view
@marcleblanc2 marcleblanc2 changed the title build: Cache contentlayer on Vercel, switch to Turbopack build: Cache content between builds Sep 17, 2026
@marcleblanc2 marcleblanc2 mentioned this pull request Sep 17, 2026
Merged
marcleblanc2 added a commit that referenced this pull request Sep 17, 2026
@marcleblanc2 @ampagent
ci: Do not fail or comment when a PR check crashes
fc375c6 
The redirects check on #1982 posted a comment containing only its hidden
marker, then failed. The PR changed src/data/redirects.ts from
module.exports to export {}, and its copy of dev/check-redirects.mjs could
no longer read the base branch's file. The workflow could not tell that
crash (Node exits 1) from findings (also exit 1), so it treated an empty
report as a finding.

- check-redirects.mjs accepts both export styles, since CI runs the PR's
  script against the base branch too
- Both check workflows classify by output: the scripts write their result in
  one go at the end, so a crash leaves it empty. A crash is a workflow
  warning; the comment, review and fail steps only run on a real result
- The comment step never posts a report it does not have

Amp-Thread-ID: https://ampcode.com/threads/T-01a0adcb-1b03-716a-852a-cc3746148488
Co-authored-by: Amp <amp@ampcode.com>
@marcleblanc2
marcleblanc2 force-pushed the marc/build/turbopack-and-contentlayer-cache branch from 153e463 to d7ce16a Compare September 17, 2026 06:01
marcleblanc2 added a commit that referenced this pull request Sep 17, 2026
@marcleblanc2 @ampagent
ci: Do not fail or comment when a PR check crashes
8b1f6d2 
The redirects check on #1982 posted a comment containing only its hidden
marker, then failed. The PR changed src/data/redirects.ts from
module.exports to export {}, and its copy of dev/check-redirects.mjs could
no longer read the base branch's file. The workflow could not tell that
crash (Node exits 1) from findings (also exit 1), so it treated an empty
report as a finding.

- check-redirects.mjs accepts both export styles, since CI runs the PR's
  script against the base branch too
- Both check workflows classify by output: the scripts write their result in
  one go at the end, so a crash leaves it empty. A crash is a workflow
  warning; the comment, review and fail steps only run on a real result
- The comment step never posts a report it does not have

Amp-Thread-ID: https://ampcode.com/threads/T-01a0adcb-1b03-716a-852a-cc3746148488
Co-authored-by: Amp <amp@ampcode.com>
@sourcegraph sourcegraph deleted a comment from github-actions Bot Sep 17, 2026
@marcleblanc2
marcleblanc2 marked this pull request as ready for review September 17, 2026 06:24
@marcleblanc2
marcleblanc2 requested review from a team and enriquegh September 17, 2026 06:24
@marcleblanc2 marcleblanc2 changed the title build: Cache content between builds build: Fix build cache of page content Sep 17, 2026
marcleblanc2 added a commit that referenced this pull request Sep 17, 2026
@marcleblanc2 @ampagent
ci: Do not fail or comment when a PR check crashes
69fed7f 
The redirects check on #1982 posted a comment containing only its hidden
marker, then failed. The PR changed src/data/redirects.ts from
module.exports to export {}, and its copy of dev/check-redirects.mjs could
no longer read the base branch's file. The workflow could not tell that
crash (Node exits 1) from findings (also exit 1), so it treated an empty
report as a finding.

- check-redirects.mjs accepts both export styles, since CI runs the PR's
  script against the base branch too
- Both check workflows classify by output: the scripts write their result in
  one go at the end, so a crash leaves it empty. A crash is a workflow
  warning; the comment, review and fail steps only run on a real result
- The comment step never posts a report it does not have

Amp-Thread-ID: https://ampcode.com/threads/T-01a0adcb-1b03-716a-852a-cc3746148488
Co-authored-by: Amp <amp@ampcode.com>
marcleblanc2 added a commit that referenced this pull request Sep 17, 2026
@marcleblanc2 @ampagent
ci: Crashed PR check scripts now warn instead of block (#1992)
71a74eb 
When a PR check script crashes, the workflow now comments that the check
could not run instead of failing the PR

Unblocks #1982

## Testing

- Tested the `run:` steps locally for the crash, clean and findings
paths
- `actionlint` passes

---------

Co-authored-by: Amp <amp@ampcode.com>
@marcleblanc2
marcleblanc2 force-pushed the marc/build/turbopack-and-contentlayer-cache branch from 245a391 to d530440 Compare September 17, 2026 10:07
marcleblanc2 and others added 4 commits September 17, 2026 04:10
@marcleblanc2 @ampagent
build: Turbopack for next build, keep the contentlayer cache between …
a2bcb29 
…Vercel deploys

Vercel builds spend ~25-30s regenerating all 506 MDX documents on every
deploy, because contentlayer only writes its cache to .contentlayer/.cache
and Vercel only keeps .next/cache. dev/build-content.mjs symlinks the one
into the other and runs contentlayer2 build, so only changed pages are
re-rendered.

With contentlayer run as its own step, next build no longer needs the
next-contentlayer2 webpack hook, so it can use Turbopack (the Next 16
default): no --webpack, no webpack persistent cache to restore and upload,
no 'Collecting build traces' step. next dev --webpack keeps the plugin
through the PHASE_DEVELOPMENT_SERVER branch in next.config.js.

Two things webpack tolerated that Turbopack does not:
- src/proxy.ts loaded src/data/redirects.ts with require() of a file that
  mixes import and module.exports; it is now import/export on both sides,
  and dev/check-redirects.mjs strips the new export line
- contentlayer.config.ts read the code theme with a path relative to where
  contentlayer compiles the config; it is now a JSON import esbuild bundles

Amp-Thread-ID: https://ampcode.com/threads/T-01a0acbf-1cb1-7317-92d6-cd9399e6112a
Co-authored-by: Amp <amp@ampcode.com>
@marcleblanc2 @ampagent
build: Set .mdx mtimes from content so contentlayer's cache hits on V…
76574bc 
…ercel

contentlayer2 keys its document cache on the source file's mtime, and a
fresh clone sets every mtime to the clone time, so on Vercel every entry
missed. The second preview deploy of this branch restored the cache and
still spent 23s re-rendering all 506 documents.

Amp-Thread-ID: https://ampcode.com/threads/T-01a0ad75-46cf-77ed-889b-438e2521bc89
Co-authored-by: Amp <amp@ampcode.com>
@marcleblanc2 @ampagent
spelling: Allow mtimes and timlrx
199d185 
Amp-Thread-ID: https://ampcode.com/threads/T-01a0ad75-46cf-77ed-889b-438e2521bc89
Co-authored-by: Amp <amp@ampcode.com>
@marcleblanc2 @ampagent
dev/todo: Note the webpack to Turbopack dev migration
0ab4acd 
Amp-Thread-ID: https://ampcode.com/threads/T-01a0ad75-46cf-77ed-889b-438e2521bc89
Co-authored-by: Amp <amp@ampcode.com>
@marcleblanc2
marcleblanc2 force-pushed the marc/build/turbopack-and-contentlayer-cache branch from d530440 to 0ab4acd Compare September 17, 2026 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@enriquegh enriquegh Awaiting requested review from enriquegh

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@marcleblanc2