Skip to content

Add validation config and golden files for WHO COVID19#2097

Open
abhishekjaisw wants to merge 1 commit into
datacommonsorg:masterfrom
abhishekjaisw:who_covid19_validation
Open

Add validation config and golden files for WHO COVID19#2097
abhishekjaisw wants to merge 1 commit into
datacommonsorg:masterfrom
abhishekjaisw:who_covid19_validation

Conversation

@abhishekjaisw

@abhishekjaisw abhishekjaisw commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

This PR adds the validation configuration file and golden observation/summary files for the WHO COVID19 import.

Summary
This PR introduces the automated validation configuration and companion golden verification files for the WHO COVID-19 statistical variable imports. These additions ensure that all future scheduled runs of the WHO pipeline are automatically validated against verified baseline metrics for data completeness, schema correctness, and anomalous fluctuations.
Changes Introduced
Added validation_config.json:
Configures a threshold of 0.5 (50%) for the DELETED_RECORDS_PERCENT validator rule (check_deleted_records_percent).
Establishes GOLDENS_CHECK validator rules for both the generated observation CSVs and their run summary reports across all five WHO data subsets:
COV_VAC_UPTAKE_2021_2023
COV_VAC_UPTAKE_2024
death_by_age
global_daily_data
global_hosp_icu_data
Checked-In Golden Datasets (located under golden_data/):
Added 5 golden observation files containing verified historical baseline records.
Added 5 golden summary report files outlining expected place coverage, min/max dates, scaling factors, and statistical variables.
Updated manifest.json:
Registered validation_config.json as the active validation_config_file.
Included the newly added golden CSVs in the source_files wildcard arrays to ensure they are packaged with the run.
Verification
Local validation runs have been executed and verified. The GKE runner relative paths resolve successfully to the respective golden_data/ and generated output directories.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces validation configuration and golden data files for the WHO COVID-19 statistical variable imports. Specifically, it adds several golden observation and summary report CSV files, configures validation rules in a new validation_config.json file, and updates the import manifest to reference these validation settings and golden files. There are no review comments, so I have no feedback to provide.

@abhishekjaisw

abhishekjaisw commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

LM notebook review :

Since the paths are fully verified on your end, the rest of your PR—including the manifest.json configurations and the script mapping patterns—looks solid, structurally correct, and ready to go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant