Skip to content

Add opt-in LLM priority search crawler #607

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rajeee wants to merge 9 commits into from
Open

Add opt-in LLM priority search crawler #607

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
3e72512
Add opt-in LLM priority search crawler
rajeee Nov 12, 2026
869609a
Document priority search in the developer guide
rajeee Nov 12, 2026
f3ecc68
Address review: stable URL canonicalization, per-page error handling,…
rajeee Nov 12, 2026
eaed02f
Read crawled documents with the configured backend; sample long singl…
rajeee Nov 12, 2026
f748cf9
Drop carried-over evidence before giving up on an oversized page
rajeee Nov 12, 2026
9464e53
Start priority search from saved search results when supplied
rajeee Nov 12, 2026
bfb8985
Use a realistic sample in the Docling read test
rajeee Nov 12, 2026
5b4d9f2
Describe the crawler's opt-in event log location
rajeee Nov 12, 2026
0f60752
Record full error chains in the crawler's failure events
rajeee Nov 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions compass/data/workflow.json5
View file
Open in desktop

@rajeee rajeee Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adds the priority-search branch to the workflow map.

A decision node on whether priority_search is configured, and the nodes for the crawl itself (search, rank, visit, fetch, loaded, new, assess, candidate, kept, follow, failed, skipped, stop, done). The crawler's events refer to these IDs. Data only; the tests in test_utilities_workflow.py check the map against the code.

Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,16 @@
about: "Searches for ordinance documents and downloads the top results.",
code: "compass/pipeline/collection/steps.py:SearchEngineDocumentsStep",
},
{
id: "collect.search_engine.mode",
kind: "decision",
label: "Priority search configured?",
about: "A priority_search setting replaces the standard search with an LLM-ranked crawl of the search results.",
next: [
{to: "collect.search_engine.queries", when: "no"},
{to: "collect.search_engine.priority", when: "yes"},
],
},
{
id: "collect.search_engine.queries",
kind: "step",
Expand Down Expand Up @@ -123,6 +133,143 @@
tone: "bad",
label: "No search results",
},
{
id: "collect.search_engine.priority",
kind: "group",
label: "Priority search",
each: "page",
about: "Searches every engine, has an LLM rank the results, then visits pages one at a time in priority order: each page is fetched, assessed as an ordinance document, and its best links are queued. It stops once enough candidates reach the stopping score, the queue runs out, or the page limit is hit.",
code: "compass/web/priority_crawl.py:PriorityCrawler.run",
events: ["run_started"],
},
{
id: "collect.search_engine.priority.search",
kind: "step",
label: "Search all engines",
code: "compass/scripts/download.py:_priority_search_seeds",
events: ["search_queries", "search_request", "search_response", "search_results"],
next: "collect.search_engine.priority.rank",
},
{
id: "collect.search_engine.priority.rank",
kind: "llm",
label: "Rank the search results",
about: "The best-ranked results start the queue. If ranking fails, the queue falls back to search order.",
code: "compass/web/priority_crawl.py:PriorityCrawler._rank",
events: ["llm_request", "llm_response", "ranking_resolved", "ranking_fallback", "enqueued"],
next: "collect.search_engine.priority.visit",
},
{
id: "collect.search_engine.priority.visit",
kind: "step",
label: "Visit the top queued page",
about: "Takes the pending link with the highest priority (the best score any page gave it).",
code: "compass/web/priority_crawl.py:PriorityCrawler._visit_next",
events: ["visit_started"],
next: "collect.search_engine.priority.fetch",
},
{
id: "collect.search_engine.priority.fetch",
kind: "step",
label: "Fetch it (HTTP, then a browser)",
about: "Plain HTTP first. A refused or failed request (other than 404/410) is retried in a browser, except for document links (PDF, Word), which a browser cannot render. A browser page without text still loads if it links to or frames something. Each attempt's request, response headers, redirects, and body are recorded.",
code: "compass/web/priority_crawl.py:PriorityCrawler._fetch",
events: ["fetched", "browser_retry", "page_read"],
next: "collect.search_engine.priority.loaded",
},
{
id: "collect.search_engine.priority.loaded",
kind: "decision",
label: "Did it load?",
next: [
{to: "collect.search_engine.priority.new", when: "yes"},
{to: "collect.search_engine.priority.failed", when: "no"},
],
},
{
id: "collect.search_engine.priority.new",
kind: "decision",
label: "New, with text or links?",
about: "Pages already seen (the same final URL or identical bytes) are skipped. A page with no text is still assessed when its HTML links to or embeds something, such as a viewer page framing a PDF; one with nothing to follow is skipped.",
next: [
{to: "collect.search_engine.priority.assess", when: "yes"},
{to: "collect.search_engine.priority.assess", when: "no text, has links"},
{to: "collect.search_engine.priority.skipped", when: "duplicate"},
{to: "collect.search_engine.priority.skipped", when: "unreadable"},
],
},
{
id: "collect.search_engine.priority.assess",
kind: "llm",
label: "Assess the page",
about: "Scores the page as an ordinance document (0-10), classifies and dates it, and rates its links. A page with no text is judged from its HTML, passes its parents' evidence on, and cannot become a candidate itself.",
code: "compass/web/priority_crawl.py:PriorityCrawler._assess",
events: ["llm_request", "llm_response", "assessed"],
next: [
{to: "collect.search_engine.priority.candidate", when: "assessed"},
{to: "collect.search_engine.priority.failed", when: "error"},
],
},
{
id: "collect.search_engine.priority.candidate",
kind: "decision",
label: "A candidate document?",
about: "A scored ordinance, code, draft, resolution, or moratorium is kept as a candidate.",
next: [
{to: "collect.search_engine.priority.kept", when: "yes"},
{to: "collect.search_engine.priority.follow", when: "no"},
],
},
{
id: "collect.search_engine.priority.kept",
kind: "step",
label: "Keep it as a candidate",
tone: "good",
next: "collect.search_engine.priority.follow",
},
{
id: "collect.search_engine.priority.follow",
kind: "decision",
label: "Queue its best links?",
about: "Up to max_links of the page's links are queued, while the depth limit allows; a link already queued keeps its best score.",
events: ["enqueued"],
next: [
{to: "collect.search_engine.priority.stop", when: "yes"},
{to: "collect.search_engine.priority.stop", when: "no: max depth"},
],
},
{
id: "collect.search_engine.priority.failed",
kind: "end",
tone: "bad",
label: "Failed",
about: "The page could not be loaded (or assessed); its fetch attempts show why.",
events: ["visit_failed"],
next: "collect.search_engine.priority.stop",
},
{
id: "collect.search_engine.priority.skipped",
kind: "end",
label: "Skipped",
events: ["skipped"],
next: "collect.search_engine.priority.stop",
},
{
id: "collect.search_engine.priority.stop",
kind: "decision",
label: "Stop crawling?",
about: "Stops once stop_count candidates reach stop_score, when the queue is empty, or at the page limit.",
next: [
{to: "collect.search_engine.priority.visit", when: "keep going"},
{to: "collect.search_engine.priority.done", when: "stop"},
],
},
{
id: "collect.search_engine.priority.done",
kind: "end",
label: "Candidates handed on, best first",
events: ["run_finished"],
},
{
id: "collect.search_results_crawl",
kind: "group",
Expand Down
15 changes: 15 additions & 0 deletions compass/pipeline/collection/steps.py
View file
Open in desktop

@rajeee rajeee Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The search-engine collection step branches on priority_search.

When priority_search is configured, SearchEngineDocumentsStep calls download_prioritized_ordinances instead of the standard search-and-download, and records which path was taken. Errors are caught and return no documents, as the standard path does, so later collection steps still run.

With priority_search unset (the default) this step behaves exactly as before.

Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
from elm.web.utilities import get_redirected_url

from compass.scripts.download import (
download_prioritized_ordinances,
download_jurisdiction_ordinance_from_search_results,
download_jurisdiction_ordinance_using_search_engine,
download_jurisdiction_ordinances_from_website,
Expand Down Expand Up @@ -193,6 +194,20 @@ async def collect(self, workflow): # ruff:ignore[no-self-use]
"Collecting documents using a search engine for jurisdiction: %s",
workflow.jurisdiction.full_name,
)
priority = workflow.runtime.search_params.priority_search is not None
emit_branch("collect.search_engine.mode", "yes" if priority else "no")
if priority:
logger.info("Search path: LLM priority queue")
try:
with event_step("collect.search_engine.priority"):
return await download_prioritized_ordinances(workflow)
except Exception:
logger.exception(
"Error collecting documents using priority search for %s",
workflow.jurisdiction.full_name,
)
return []
logger.info("Search path: standard retrieval")
try:
if workflow.known_se_results is not None:
docs = await _download_docs_using_se_results(workflow)
Expand Down
30 changes: 30 additions & 0 deletions compass/pipeline/data_classes.py
View file
Open in desktop

@rajeee rajeee Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New opt-in config option: priority_search.

None (the default) keeps the standard search. A dict turns on the priority crawler and may set results_per_engine, max_ranked_links, max_pages, max_depth, max_links, stop_score and stop_count.

The option is threaded through WebSearchParams, BaseRequest and CollectionRequest, which is why the same parameter and docstring appear three times; that follows how the neighbouring search options are declared in this file.

Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@ def __init__(
url_keep_substrings=None,
search_engines=None,
simple_se_result_sort=False,
priority_search=None,
pytesseract_exe_fp=None,
num_search_results_to_crawl=0,
search_results_crawl_depth=3,
Expand Down Expand Up @@ -307,6 +308,14 @@ def __init__(
first search engine that gives results (``True``) or to
apply a holistic link sorting based on all results from all
search engines (``False``). By default, ``False``.
priority_search : dict, optional
Disabled by default (None). Pass a dict to opt into
LLM-ranked search and link traversal. Options (defaults)
are results_per_engine (10), max_ranked_links (10),
max_pages (15), max_depth (3), max_links (3),
stop_score (10; ``None`` crawls until a limit is hit), and
stop_count (1; how many documents must reach stop_score
before the crawl stops).
pytesseract_exe_fp : path-like, optional
Path to the `pytesseract` executable. If specified, OCR will
be used to extract text from scanned PDFs using Google's
Expand Down Expand Up @@ -337,6 +346,7 @@ def __init__(
]
self._search_engines_input = search_engines
self.simple_se_result_sort = simple_se_result_sort
self.priority_search = priority_search
self.pytesseract_exe_fp = pytesseract_exe_fp
self.num_search_results_to_crawl = max(
0, int(num_search_results_to_crawl)
Expand Down Expand Up @@ -427,6 +437,7 @@ def __init__( # ruff:ignore[too-many-arguments]
file_loader_kwargs=None,
search_engines=None,
simple_se_result_sort=False,
priority_search=None,
pytesseract_exe_fp=None,
td_kwargs=None,
tpe_kwargs=None,
Expand Down Expand Up @@ -647,6 +658,14 @@ def __init__( # ruff:ignore[too-many-arguments]
first search engine that gives results (``True``) or to
apply a holistic link sorting based on all results from all
search engines (``False``). By default, ``False``.
priority_search : dict, optional
Disabled by default (None). Pass a dict to opt into
LLM-ranked search and link traversal. Options (defaults)
are results_per_engine (10), max_ranked_links (10),
max_pages (15), max_depth (3), max_links (3),
stop_score (10; ``None`` crawls until a limit is hit), and
stop_count (1; how many documents must reach stop_score
before the crawl stops).
pytesseract_exe_fp : path-like, optional
Path to the `pytesseract` executable. If specified, OCR will
be used to extract text from scanned PDFs using Google's
Expand Down Expand Up @@ -767,6 +786,7 @@ def __init__( # ruff:ignore[too-many-arguments]
url_keep_substrings=url_keep_substrings,
search_engines=search_engines,
simple_se_result_sort=simple_se_result_sort,
priority_search=priority_search,
pytesseract_exe_fp=pytesseract_exe_fp,
num_search_results_to_crawl=num_search_results_to_crawl,
search_results_crawl_depth=search_results_crawl_depth,
Expand Down Expand Up @@ -1007,6 +1027,7 @@ def __init__( # ruff:ignore[too-many-arguments]
file_loader_kwargs=None,
search_engines=None,
simple_se_result_sort=False,
priority_search=None,
pytesseract_exe_fp=None,
td_kwargs=None,
tpe_kwargs=None,
Expand Down Expand Up @@ -1204,6 +1225,14 @@ def __init__( # ruff:ignore[too-many-arguments]
first search engine that gives results (``True``) or to
apply a holistic link sorting based on all results from all
search engines (``False``). By default, ``False``.
priority_search : dict, optional
Disabled by default (None). Pass a dict to opt into
LLM-ranked search and link traversal. Options (defaults)
are results_per_engine (10), max_ranked_links (10),
max_pages (15), max_depth (3), max_links (3),
stop_score (10; ``None`` crawls until a limit is hit), and
stop_count (1; how many documents must reach stop_score
before the crawl stops).
pytesseract_exe_fp : path-like, optional
Path to the `pytesseract` executable. If specified, OCR will
be used to extract text from scanned PDFs using Google's
Expand Down Expand Up @@ -1343,6 +1372,7 @@ def __init__( # ruff:ignore[too-many-arguments]
file_loader_kwargs=file_loader_kwargs,
search_engines=search_engines,
simple_se_result_sort=simple_se_result_sort,
priority_search=priority_search,
pytesseract_exe_fp=pytesseract_exe_fp,
td_kwargs=td_kwargs,
tpe_kwargs=tpe_kwargs,
Expand Down
5 changes: 4 additions & 1 deletion compass/pipeline/runtime.py
View file
Open in desktop

@rajeee rajeee Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Collect-only runs start LLM services when priority search is on.

_llm_services returned nothing in collect mode, since standard collection makes no LLM calls. The priority crawler does (ranking and page assessment), so collect mode now starts the services when priority_search is configured. Unchanged otherwise.

Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,10 @@ def _base_services(self):
@cached_property
def _llm_services(self):
"""LLM services for modes that require them"""
if self.mode == self.mode.COLLECT:
if (
self.mode == self.mode.COLLECT
and self.search_params.priority_search is None
):
return []
return [model.llm_service for model in set(self.models.values())]

Expand Down
Loading
Loading