fix: filter sitemap-derived URLs by enqueue strategy by vdusek · Pull Request #3797 · apify/crawlee

vdusek · 2026-06-30T10:08:13Z

SitemapRequestList, Sitemap.load / parseSitemap, and RobotsTxtFile.getSitemaps() now apply an enqueue strategy to the URLs they extract. They keep only entries that match the strategy (default same-hostname) relative to their parent sitemap, and always drop non-http(s) schemes. This brings sitemap loading in line with enqueueLinks, which already scopes discovered links to the same hostname by default.

The selected strategy is also stamped onto the emitted Request objects, so it keeps being enforced after navigation (e.g. across redirects).

This changes the default behavior: cross-host sitemap entries are no longer enqueued unless you opt in with enqueueStrategy: 'all' (or 'same-domain' / 'same-origin').

B4nan · 2026-06-30T10:57:13Z

 export type SearchParams = string | URLSearchParams | Record<string, string | number | boolean | null | undefined>;

+/** Enqueue strategy values, mirroring the `EnqueueStrategy` enum from `@crawlee/core` (which `@crawlee/utils` can't import). */
+export type EnqueueStrategyValue = 'all' | 'same-hostname' | 'same-domain' | 'same-origin';


same here

Suggested change

export type EnqueueStrategyValue = 'all' | 'same-hostname' | 'same-domain' | 'same-origin';

export type EnqueueStrategyValue = `${EnqueueStrategy}`;

i would rather inline this instead of introducing a new exported type for it

This is not possible because crawlee/core already depends on crawlee/utils, so that would be a circular reference. Am I correct? 🙂

type-level cycles are fine

I won't pretend to be a guru here, but this still doesn't seem doable (or at least not doable in a right & easy way):

Type-level cycles are fine within a single tsc program (or with project references) — but that's not the situation here. @crawlee/utils and @crawlee/core are two separately-compiled packages, and this repo doesn't use project references: each package is built by its own tsc -p tsconfig.build.json, ordered topologically by Turbo from the package.json dependency graph.

To write ${EnqueueStrategy} in utils I'd have to import type { EnqueueStrategy } from '@crawlee/core', and that hits two concrete walls:

It requires @crawlee/core in utils' package.json. As soon as I add it, Turbo refuses to build: x Cyclic dependency detected: @crawlee/core#build, @crawlee/utils#build

Even without declaring it (relying on the workspace symlink), the build fails. tsconfig.build.json has no paths mapping — only the root dev/type-check tsconfig.json does — so cross-package types resolve from the emitted node_modules/@crawlee/core/dist/index.d.ts. That file only exists after core is built, but core builds after utils (core depends on utils), so on a clean build utils compiles first and gets TS2307: Cannot find module '@crawlee/core'.

So I kept the mirrored literal union in utils. I did apply the ${EnqueueStrategy} form in sitemap_request_list.ts, since that's in core where the enum is already in scope. If we ever want to genuinely dedupe it, the value union would have to move into @crawlee/types (which both packages depend on), but that's a bigger change than this nit.

Could we perhaps extract all the EnqueueStrategy logic to @crawlee/utils (and only import / reexport it in @crawlee/core and all the places we want to use it)?

It imo makes sense now that we're directly using the strategies in @crawlee/utils classes (Sitemap, Robots)

The reason is probably confirmed by #3797 (comment), so resolving this with no action.

unresolving, as a new comment from barjin landed ⛵

Could we perhaps extract all the EnqueueStrategy logic to @crawlee/utils (and only import / reexport it in @crawlee/core and all the places we want to use it)?

It imo makes sense now that we're directly using the strategies in @crawlee/utils classes (Sitemap, Robots)

moving it there sounds reasonable, thanks

vdusek · 2026-06-30T10:59:45Z

@B4nan FYI; this is still a draft - I haven't reviewed most of it yet

B4nan · 2026-06-30T11:02:08Z

no worries, i was just curious 🙃

vdusek · 2026-07-01T10:56:23Z

Requesting a review from @janbuchar, since you may have additional context from the Python solution.

janbuchar

LGTM, thanks

B4nan

few nits from my end

@barjin please take a look as well, we want to have this in the today's release

B4nan · 2026-07-16T11:01:24Z

        networkTimeouts,
        reportNetworkErrors = true,
        nestedSitemapFilter,
+        enqueueStrategy = 'same-hostname',


[medium] Defaulting to same-hostname makes parseSitemap/RobotsTxtFile.* drop cross-host URLs (notably apex↔www) that they used to return. Intended and consistent with enqueueLinks, but these are public exports — needs a changelog note.

So I guess we should update the changelog manually once this is released, after the generated section is moved to the next release.

B4nan · 2026-07-16T11:01:24Z

+ * Check whether `target` matches `origin` under the given enqueue `strategy`. The URL scheme is not
+ * considered here (use {@apilink filterUrl} for the combined scheme + strategy check).
+ */
+export function matchesEnqueueStrategy(strategy: EnqueueStrategyValue, target: URL, origin: URL): boolean {


[low] Duplicates the EnqueueStrategy matching in core/enqueue_links.ts (unavoidable — utils can't import core). Add a comment cross-referencing the core enum so they don't drift. Minor: core's same-hostname is port-sensitive, this isn't.

Let's add a cross-reference note to the docblock.

If I am not mistaken the comment also confirms this #3797 (comment)

B4nan · 2026-07-16T11:01:24Z

+                if (source.type === 'url') {
+                    const { allowed, reason } = filterUrl(item.loc, sitemapUrl!, enqueueStrategy);
+                    if (!allowed) {
+                        log.debug(`Skipping sitemap URL ${item.loc} (parent ${source.url}): ${reason}.`);


[low] Dropped <url> entries log at debug while dropped nested sitemaps log at warning (line 373). The <url> case is the more impactful one — align the levels.

Just flipping it to a warning could generate a lot of warning logs. So let's update the parseSitemap to count the dropped <url> records and emit a single aggregated warning per sitemap instead.

B4nan · 2026-07-16T11:01:24Z

+    const originUrl = toUrl(origin);
+
+    if (originUrl === null || !matchesEnqueueStrategy(strategy, targetUrl, originUrl)) {
+        return { allowed: false, reason: `does not match enqueue strategy '${strategy}'` };


[low] Reason says does not match enqueue strategy even when the real cause is an unparseable origin. Give the origin-parse-failure path its own reason.

Let's split it and return a proper reason: "invalid origin URL" instead.

barjin

Thank you @vdusek ! few ideas from me ⬇️

barjin · 2026-07-17T12:20:50Z

     */
-    getSitemaps(): string[] {
-        return this.robots.getSitemaps();
+    getSitemaps(enqueueStrategy: EnqueueStrategyValue = 'same-hostname'): string[] {


let's please make these options objects

getSitemaps(options: { enqueueStrategy: EnqueueStrategy })

in TS, this is a positional argument, so if we eventually add a new option, we'll have to always specify the enqueueStrategy

getSitemaps('same-hostname', /* <--- this has to be here in order to specify the second argument */)

Sure, thank you

Edit: applied this approach to other places as well

barjin · 2026-07-17T12:24:02Z

+    /**
+     * Keep only sitemap-derived URLs matching this strategy relative to the parent sitemap URL; non-`http(s)`
+     * schemes are always dropped. The strategy is also stamped onto emitted `Request`s, so it stays enforced
+     * after navigation (e.g. across redirects). Pass `'all'` to disable host filtering.
+     * @default EnqueueStrategy.SameHostname
+     */
+    enqueueStrategy?: EnqueueStrategy | `${EnqueueStrategy}`;


The strategy is also stamped onto emitted Requests, so it stays enforced after navigation (e.g. across redirects).

This is an implementation detail (see that Request.enqueueStrategy is private). I wish we had implemented this differently, but here we are 😄

okay, removing 🙂

barjin · 2026-07-17T12:36:04Z

 export type SearchParams = string | URLSearchParams | Record<string, string | number | boolean | null | undefined>;

+/** Enqueue strategy values, mirroring the `EnqueueStrategy` enum from `@crawlee/core` (which `@crawlee/utils` can't import). */
+export type EnqueueStrategyValue = 'all' | 'same-hostname' | 'same-domain' | 'same-origin';


Could we perhaps extract all the EnqueueStrategy logic to @crawlee/utils (and only import / reexport it in @crawlee/core and all the places we want to use it)?

It imo makes sense now that we're directly using the strategies in @crawlee/utils classes (Sitemap, Robots)

barjin

I don't see any other blockers. Thank you @vdusek !

fix: filter sitemap-derived URLs by enqueue strategy

1b1f534

vdusek added adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team. labels Jun 30, 2026

vdusek self-assigned this Jun 30, 2026

vdusek added 2 commits June 30, 2026 12:19

chore: shorten comments and docstrings

43a02d8

fix: address review findings in sitemap enqueue-strategy filtering

038eb40

B4nan reviewed Jun 30, 2026

View reviewed changes

Comment thread packages/core/src/storages/sitemap_request_list.ts Outdated

B4nan reviewed Jun 30, 2026

View reviewed changes

Comment thread packages/utils/package.json

chore: address review nits in sitemap enqueue-strategy filtering

59ff5ce

vdusek requested a review from B4nan July 1, 2026 10:50

vdusek marked this pull request as ready for review July 1, 2026 10:50

vdusek requested review from janbuchar and removed request for B4nan July 1, 2026 10:54

janbuchar approved these changes Jul 3, 2026

View reviewed changes

vdusek requested a review from B4nan July 7, 2026 07:23

B4nan approved these changes Jul 17, 2026

View reviewed changes

B4nan requested a review from barjin July 17, 2026 08:30

barjin reviewed Jul 17, 2026

View reviewed changes

Address b4nan's review findings

134855a

github-actions Bot added this to the 145th sprint - Tooling team milestone Jul 17, 2026

github-actions Bot added the tested Temporary label used only programatically for some analytics. label Jul 17, 2026

Address barjin's review findings

99e299c

vdusek requested a review from barjin July 17, 2026 13:08

barjin approved these changes Jul 20, 2026

View reviewed changes

vdusek merged commit 9dc3160 into master Jul 20, 2026
8 checks passed

vdusek deleted the fix/sitemap-enqueue-strategy branch July 20, 2026 12:30

	export type EnqueueStrategyValue = 'all' \| 'same-hostname' \| 'same-domain' \| 'same-origin';
	export type EnqueueStrategyValue = `${EnqueueStrategy}`;

Uh oh!

Conversation

vdusek commented Jun 30, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vdusek Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vdusek commented Jun 30, 2026

Uh oh!

B4nan commented Jun 30, 2026

Uh oh!

vdusek commented Jul 1, 2026

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

B4nan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

B4nan Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

B4nan Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

B4nan Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

B4nan Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

barjin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vdusek Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

barjin left a comment

Choose a reason for hiding this comment

vdusek Jul 17, 2026 •

edited

Loading

B4nan left a comment •

edited

Loading

B4nan Jul 16, 2026 •

edited

Loading

B4nan Jul 16, 2026 •

edited

Loading

B4nan Jul 16, 2026 •

edited

Loading

B4nan Jul 16, 2026 •

edited

Loading

vdusek Jul 17, 2026 •

edited

Loading