Skip to content

changed rule to new rule Bls_cpi_Category#2044

Merged
balit-raibot merged 10 commits into
datacommonsorg:masterfrom
niveditasing:rule_change_validation_config
Jul 10, 2026
Merged

changed rule to new rule Bls_cpi_Category#2044
balit-raibot merged 10 commits into
datacommonsorg:masterfrom
niveditasing:rule_change_validation_config

Conversation

@niveditasing

@niveditasing niveditasing commented May 28, 2026

Copy link
Copy Markdown
Contributor

This PR includes:

  1. Updated the threshold rule ID. This import previously had a threshold of 1230 with rule id that checks deletions in numbers, but because the rule was recently changed to check the deletion threshold as a percentage rather than a raw number, the import is currently failing due to deletions.

Fix: Added the relevant rule ID and set the threshold to 0.6% (representing 1,230 rows out of 211,256).

1230/211256 x 100 ~ 0.58223% ~ 0.6%
Keeping the threshold at this level ensures we capture a meaningful baseline of the data without letting minor anomalies skew the results.

  1. Pvmap fix as per new Data additions

  2. The Problem
    The CI build (python_install step in Cloud Build) recently started failing with:

    1 pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000
    Because dependencies in requirements_all.txt are unpinned, standard pip3 attempts to fetch the latest versions. Recent upstream updates on
    PyPI (within the google-cloud-* / protobuf ecosystems) introduced a complex version conflict. Standard pip3's backtracking resolver got
    stuck in an infinite loop trying to reconcile these versions, exceeding its 200,000-round limit and crashing.

Fix:
Updated the setup_python function in run_tests.sh to:

  1. Bootstrap uv: Check if uv is installed; if not (as is the case in the raw Cloud Build container), install it into the virtual
    environment using pip3 (takes < 1 second).
  2. Install via uv: Use uv pip install instead of standard pip3 to install requirements.

Why it works: uv is powered by a high-performance Rust resolver (PubGrub) that handles deep backtracking effortlessly. It resolves and installs the entire unpinned dependency graph in under 2 seconds.

@niveditasing niveditasing changed the title changed rule to new rule changed rule to new rule Bls_cpi_Category May 28, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the validation configuration for the US BLS CPI category import by replacing the deleted records count validator with a percentage-based validator. A critical issue was identified where the threshold value was set to 0.1, which translates to an extremely restrictive 0.1% threshold due to the validator's implementation using a 0-100 scale. It is recommended to change this threshold to 10 to represent a 10% limit.

Comment thread statvar_imports/us_bls/cpi_category/validation_config.json Outdated
Comment thread statvar_imports/us_bls/cpi_category/validation_config.json
@balit-raibot
balit-raibot requested a review from hareesh-ms July 9, 2026 08:12
@balit-raibot

Copy link
Copy Markdown
Contributor

@gemini-code-assist review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the test runner script to install uv if it is missing, adds several new CPI-U and CPI-W category mappings, introduces a golden summary report, and updates the validation configuration to use a percentage-based deleted records check and a goldens check. Feedback was provided regarding an incorrect relative path for the golden files in the validation configuration, which would lead to validation failures.

Comment thread statvar_imports/us_bls/cpi_category/validation_config.json

@hareesh-ms hareesh-ms left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets revisit the logic to chose golden

@balit-raibot
balit-raibot merged commit e472800 into datacommonsorg:master Jul 10, 2026
14 checks passed
@balit-raibot
balit-raibot deleted the rule_change_validation_config branch July 10, 2026 10:25
Comment thread statvar_imports/us_bls/cpi_category/validation_config.json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants