Fix interact sample variables, dead paths, and crawler access guidance

[hmishra2250]

Stack

These three PRs all edit the same navigation lines in docs.json, so they are stacked in this order:

1. docs: add Browser Sandbox page to navigation #1286, base main. Adds features/browser to the Features nav.
2. Scope the capabilities page and add Cloud availability per row #1364, base docs/add-browser-to-navigation. Depends on docs: add Browser Sandbox page to navigation #1286. Groups features/browser with features/interact so the page appears exactly once.
3. This PR (Fix interact sample variables, dead paths, and crawler access guidance #1369), base docs/capabilities-and-licensing. Depends on Scope the capabilities page and add Cloud availability per row #1364.

This PR's own docs.json change is the five common-sites redirects, which sit in a different block from the nav edits above and merge cleanly on top of them.

Review and merge bottom up. Retarget to main after the parent merges; the diff then shrinks to this PR's own changes.

---

Summary

Four small docs defects that agents and site operators hit on the live site. A copyable interact sample sends an undefined shell variable to the API, the retired common-sites guides still 404, the monitoring pages do not link the two facts a monitoring reader needs next, and the advanced scraping guide tells operators to allowlist a user agent Firecrawl does not send.

Why

Interact samples. Deep Insights report DI-2026-09-05-WEEKLY records the retrieved body Cannot GET /v2/scrape/$SCRAPE_ID/interact (source page:ee2fc1b4..., SHA-256 d9144198...), and names it "a path shaped exactly like an unresolved '$SCRAPE_ID' placeholder from the interact docs". On main, six of the eight curl samples on features/interact.mdx use $SCRAPE_ID in the request path and never set it, while the quickstart and profile samples on the same page do set it. Each sample has its own copy button, so the six ship a variable that expands to nothing.

Common sites. The same report notes the Amazon common-sites guide 404s. That tree was deleted on purpose in 2ed95d92 "rm-common-sites" on 2026-08-04 and nothing was left behind. The commit removed four pages: amazon, etsy, github and wikipedia. Checked live on 2026-09-06: all four paths under /developer-guides/common-sites/ and the bare /developer-guides/common-sites return 404.

Monitoring cross-links. Report DI-2026-09-04-WEEKLY, finding OB-04, found that features/monitoring.mdx answers six of the seven questions a monitoring reader asks. The seventh, webhook retry, lives in webhooks/overview.mdx lines 50 to 62, and the monitoring page had no link to it. Separately, features/change-tracking.mdx had zero links to /features/monitoring, so the page an agent actually retrieves dead-ends with respect to the scheduled product built on it.

Crawler access guidance. Finding OB-06 in report DI-2026-09-05-WEEKLY, Astra edition, run 34015460876, flagged advanced-scraping-guide.mdx:667:

User Agent: Allow FirecrawlAgent in your firewall or security rules.

Product truth on firecrawl main: apps/api/src/lib/robots-txt.ts:140 passes ["FireCrawlAgent", "FirecrawlAgent"] to the robots parser, so the token decides which robots.txt block applies and is never written into an outgoing header. apps/api/src/scraper/scrapeURL/engines/fetch/index.ts:170 forwards meta.options.headers only, and no core fetch path sets a default User-Agent. apps/api/src/scraper/WebScraper/crawler.ts:523 runs the robots check on crawl. Live on 2026-09-06, a scrape of https://httpbin.org/headers through api.firecrawl.dev on default settings returned:

"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36"

Repeating with proxy: basic and no cache gave the same result. An operator who follows the current instruction gets a rule that never fires. The same behavior is reported from outside in firecrawl/firecrawl#1169.

Changes

• snippets/v2/interact/{prompt,execute-python,execute-node,execute-bash}/*curl*.mdx: each standalone curl sample now scrapes first and reads data.metadata.scrapeId into SCRAPE_ID, matching the quickstart and profile samples. snippets/v2/interact/stop/curl.mdx keeps its single DELETE call and states in a comment where SCRAPE_ID comes from.
• docs.json: five redirects sending /developer-guides/common-sites and its four removed pages to /features/scrape, the closest current page. All four taught the same thing applied to one site: scrape with JSON mode, search, scrape, map, crawl, batch scrape.
• features/monitoring.mdx: the notifications section states the 10 second timeout and the three retries at 1, 5, and 15 minutes and links /webhooks/overview#timeouts--retries. Its change tracking section links /features/change-tracking.
• features/change-tracking.mdx: links /features/monitoring in the lead-in and again in the scheduling section.
• advanced-scraping-guide.mdx, one section, three bullets: a new robots.txt bullet stating that Firecrawl reads robots.txt before crawling and applies rules written for FirecrawlAgent and for *; the User Agent bullet now says Firecrawl does not send FirecrawlAgent in the User-Agent header of its page requests, so firewall and WAF rules matching that token will not match Firecrawl traffic; the IP addresses bullet is unchanged.

gt-lock.json still carries five stale translation records for each removed common-sites page. That file is generated by gtx-cli from gt.config.json, so it is not hand-edited here and will settle on the next translation run. Localized copies of the interact snippets and of the crawler-access section are generated the same way and are also left alone, per CLAUDE.md.

Verification

• mintlify broken-links (Mintlify CLI 4.2.773, Node 22.23.2): 208 broken links, byte-identical to the same run on a clean origin/main worktree. None in the changed files.
• mintlify validate: 17 warnings, all missing snippet imports, byte-identical to the same run on origin/main.
• Live 404s confirmed by curl on 2026-09-06 for all five common-sites paths above.
• Live user-agent check described above, run twice.
• Repo search for Signature-Agent, Signature-Input, web-bot-auth, RFC 9421, ed25519 and http-message-signatures across firecrawl, firecrawl-docs, firecrawl-web and firecrawl-mcp-server on main: no matches. Nothing in the change assumes request signing that does not exist.
• git diff --name-only origin/main...HEAD lists 11 files, all docs content.

Not in this PR

Shipping an identifiable user agent, publishing an outbound IP list, and applying for Cloudflare bot verification are product decisions, not docs edits. PR firecrawl/firecrawl#2766 attempted the first and was closed unmerged on 2026-03-04.

Added 2026-09-07

The Monitor operating contract is now on the page. features/monitoring.mdx listed six check statuses and answered none of the three questions an operator asks next: what happens when a check cannot run, whether there is a failure count to read, and whether Firecrawl ever stops a schedule by itself. A new "Skipped checks and paused monitors" section answers all three, and one existing line is corrected. Reference: Deep Insights DI-2026-09-05-WEEKLY, path IP-D5.

skipped_no_credits was missing from the status set. The page said checks can be filtered by queued, running, completed, failed, partial, or skipped_overlap. The status exists and is written at apps/api/src/services/monitoring/runner.ts:1033, typed at apps/api/src/services/monitoring/types.ts:277-288 and :342-349, and declared in both SDKs at apps/js-sdk/firecrawl/src/v2/types.ts:1185-1195 and apps/python-sdk/firecrawl/v2/types.py:1356. The two status enums in api-reference/v2-openapi.json omitted it too, at the checks-list query parameter and on the MonitorCheck schema, so the API reference under-reported the set as well. Both enums and the prose line now carry it, and the prose says the set of seven is complete.

There is no aggregate failure counter, and stating that is safe. apps/api/src/services/monitoring/store.ts:849-853 derives the credit-skip streak by query and says why: "Derived rather than counted: a stored counter would be a second source of truth to keep correct across retries." No failure count, error rate or consecutive-skip field appears on either the Monitor or the MonitorCheck schema in api-reference/v2-openapi.json. The page now names what a monitor does return, status, nextRunAt, lastRunAt, currentCheckId and lastCheckSummary, and shows how to derive a streak from the checks list, which is returned newest first.

Auto-pause exists, so the page describes it instead of denying its absence. apps/api/src/services/monitoring/runner.ts:1047-1058 calls pauseMonitor (store.ts:883-901) when the credit lock is denied with reason job_revoked and the leading run of credit-skipped checks has reached three. That reason comes from the partner gate rather than from the team's own balance, per the comment at apps/api/src/services/autumn/types.ts:59-67: "job_revoked is the only one that never resolves on its own, and so the only one a caller may answer by stopping a schedule." So the page states the partner-revocation rule, states explicitly that an ordinary team running out of credits is not paused and keeps recording skipped checks, and notes that the pause is a status change with no webhook event and no email behind it. Returning the monitor to active is already in the MonitorUpdateRequest status enum.

Verification on this branch, Node 22.23.2:

• mintlify broken-links: 208 broken-link entries across 95 files, identical to the same run on a detached worktree at this branch's previous head 32f8ac97. None in features/monitoring.mdx.
• mintlify validate: 17 warnings, the same count as the detached baseline.
• api-reference/v2-openapi.json parses after the edit.

Rebase note

This branch was rebased onto the new head of its base branch docs/capabilities-and-licensing (93dc8d5e), which gained one commit on #1364 the same day. The rebase was clean and no earlier commit changed.

Added 2026-09-07

Narrowed the user-agent bullet in advanced-scraping-guide.mdx after external review. The bullet previously said Firecrawl does not send FirecrawlAgent in the User-Agent header of its page requests. The code evidence supports only that the token is used for robots.txt rule matching, at apps/api/src/lib/robots-txt.ts:143 on firecrawl origin/main: userAgents: string[] = ["FireCrawlAgent", "FirecrawlAgent"],. That default does not establish the outbound identity of every managed request, so a universal absence claim is wider than the evidence. The bullet now gives the actionable distinction instead: FirecrawlAgent is the token Firecrawl matches against in robots.txt, do not assume it is the User-Agent header on the fetch, and allow-list by robots directive rather than by User-Agent string. The corrected robots.txt bullet is unchanged. mintlify broken-links on Node 22.23.2 is identical to a detached worktree at the previous head 9d0b60e2.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SL5chNkWnr4Gy6uuB8PeKS

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
firecrawl	🟢 Ready	View Preview	Sep 6, 2026, 10:27 AM

💡 Tip: Enable Automations to automatically generate PRs for you.