Skip to content

Add data center one-shot extraction plugin - #608

Open
rajeee wants to merge 13 commits into
add-priority-crawlerfrom
add-data-center-plugin
Open

rajeee wants to merge 13 commits into
add-priority-crawlerfrom
add-data-center-plugin

Conversation

@rajeee

@rajeee rajeee commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Adds the data center schema and plugin config, post-processing for
data center type labels, and scripts that export the human-extracted
reference ordinances.

Also loads jurisdictions by FIPS when the input file has a FIPS
column, keeps leading zeros in known-source lookups, and stops
definitions-only documents from counting as an ordinance.


Stack created with GitHub Stacks CLI • Give Feedback 💬

@codecov-commenter

codecov-commenter commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.83333% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 76.69%. Comparing base (0f60752) to head (e6bd7ed).

Files with missing lines Patch % Lines
compass/plugin/one_shot/components.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@                   Coverage Diff                    @@
##           add-priority-crawler     #608      +/-   ##
========================================================
+ Coverage                 76.44%   76.69%   +0.24%     
========================================================
  Files                        86       86              
  Lines                      9927     9949      +22     
  Branches                   1088     1094       +6     
========================================================
+ Hits                       7589     7630      +41     
+ Misses                     2129     2106      -23     
- Partials                    209      213       +4     
Flag Coverage Δ
unittests 76.69% <95.83%> (+0.24%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rajeee rajeee left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer guide: one note per changed source file, saying what changed and why the existing structure couldn't do it. Tests and docs are not annotated.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New: the extraction schema for data center ordinances.

A one-shot schema: the model returns a sparse table with one row per feature the ordinance addresses, plus extraction_notes saying what was and was not extracted and why.

Points worth reviewing:

  • 35 features, with IDs that mirror the vocabulary of the hand-coded research workbook so output can be compared against it directly. Each has a definition under $definitions stating what counts, what value to record, and what to ignore.
  • data_center_type is a fixed list: Micro, Small, Medium, Large or Hyperscale Data Center, Accessory Data Center, or null for data centers of any size. The extraction call uses strict structured output, so the model cannot emit anything else. This replaced free-text class names, which produced near-duplicates across runs ('Large-Scale Data Centers' / 'Large-Scale Data Center'). The jurisdiction's own class name goes at the start of summary.
  • One row per feature and type. Rows are typed by how the ordinance states each requirement (per tier, or generally). Other distinctions, such as whether a site is part of a unified development plan, are conditions: the most restrictive value is recorded and the others are listed in summary.
  • Numeric features carry a single number, the most restrictive one, chosen by the rules in $instructions.numeric_prioritization.
  • Drafts and other jurisdictions' text are excluded. Meeting packets often attach a draft ordinance or reproduce neighbouring counties' rules for comparison. Redline markup alone does not make a text a draft.

tests/python/unit/extraction/test_extraction_data_centers_schema.py checks the contract (feature list matches the definitions, the type list, unit and value rules).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New: plugin configuration for data centers.

Search query templates, website and heuristic keywords, and the extraction system prompt. The prompt names the jurisdiction ({jur_full_name}), which the model was previously never told, and repeats the schema's key rules in prose: the fixed type list, one row per feature and type, most restrictive value, and no rows from drafts or from other jurisdictions' regulations.

No Python is needed for the plugin; __init__.py builds it from this file with the existing create_schema_based_one_shot_extraction_plugin.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Registers the plugin.

Builds COMPASSDataCentersExtractor from plugin_config.yaml with the existing one-shot factory, under tech="data_centers". The two one-line imports in compass/extraction/__init__.py and compass/__init__.py make it load with the package, the same way the other bundled plugins do.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jurisdictions can be selected by FIPS code, and codes keep their leading zeros.

  • A jurisdiction CSV with a FIPS column is now matched on that column alone (_load_jurisdictions_by_fips). A blank or unknown code is skipped with the usual warning, without falling back to names.
  • fips_to_str returns a string unchanged instead of round-tripping it through an integer.

Why: names are ambiguous for cities and townships (the same place name exists in several counties and states), and the data center reference set is keyed by FIPS. Converting through an integer turned 0446000 into 446000, which no longer matches the catalog.

This affects every technology, not only data centers: jurisdiction codes in outputs now keep their leading zero (08001 rather than 8001). The integration test test_collection_cli_selectively_resets_and_collects_missing_shards was updated for that.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Known documents are looked up by the full code, with a fallback for existing configs.

known_local_docs and known_doc_urls are keyed by jurisdiction code. Since codes now keep leading zeros, _known_source_for_code first tries the full code and then the same code without leading zeros, so a config written with 8001 still finds its documents for 08001.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two changes to the shared one-shot components. Both apply to every one-shot plugin, not only data centers.

  • The relevance prompt gains a sentence: the content must be about the specific technology or use the schema targets, and a definition or provision for some other use that merely mentions a related activity does not count. Without it, zoning-code definitions chapters were kept as relevant because one definition (for example, a business office that "includes data processing") matched.
  • When the parser extracts nothing, it now logs at INFO and includes the schema's extraction_notes, so the log says why. Schemas without that field log "no explanation given".

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"definition" is added to EXCLUDE_FROM_ORD_DOC_CHECK.

That set lists features that do not, on their own, make a document count as an ordinance. A definitions chapter defines terms but regulates nothing, so a document yielding only definition rows is not accepted as the jurisdiction's ordinance and the pipeline moves on to the next document.

Worth knowing: together with the district features already in this set, this means a document whose only rows are definitions and district lists is discarded. In a recent 22-jurisdiction run that dropped two jurisdictions whose adopted ordinance contains little else.

This branch was previously deployed

2 inactive deployments
codecov — e6bd7ede Deployed Nov 13, 2026 by rajeee via Codecov #649
pypi — e6bd7ede Deployed Nov 13, 2026 by rajeee via Build, check, and optionally upload wheels #481
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants