Add validation config and golden files for WHO COVID19#2097
Open
abhishekjaisw wants to merge 1 commit into
Open
Add validation config and golden files for WHO COVID19#2097abhishekjaisw wants to merge 1 commit into
abhishekjaisw wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces validation configuration and golden data files for the WHO COVID-19 statistical variable imports. Specifically, it adds several golden observation and summary report CSV files, configures validation rules in a new validation_config.json file, and updates the import manifest to reference these validation settings and golden files. There are no review comments, so I have no feedback to provide.
Contributor
Author
|
LM notebook review : Since the paths are fully verified on your end, the rest of your PR—including the manifest.json configurations and the script mapping patterns—looks solid, structurally correct, and ready to go. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the validation configuration file and golden observation/summary files for the WHO COVID19 import.
Summary
This PR introduces the automated validation configuration and companion golden verification files for the WHO COVID-19 statistical variable imports. These additions ensure that all future scheduled runs of the WHO pipeline are automatically validated against verified baseline metrics for data completeness, schema correctness, and anomalous fluctuations.
Changes Introduced
Added validation_config.json:
Configures a threshold of 0.5 (50%) for the DELETED_RECORDS_PERCENT validator rule (check_deleted_records_percent).
Establishes GOLDENS_CHECK validator rules for both the generated observation CSVs and their run summary reports across all five WHO data subsets:
COV_VAC_UPTAKE_2021_2023
COV_VAC_UPTAKE_2024
death_by_age
global_daily_data
global_hosp_icu_data
Checked-In Golden Datasets (located under golden_data/):
Added 5 golden observation files containing verified historical baseline records.
Added 5 golden summary report files outlining expected place coverage, min/max dates, scaling factors, and statistical variables.
Updated manifest.json:
Registered validation_config.json as the active validation_config_file.
Included the newly added golden CSVs in the source_files wildcard arrays to ensure they are packaged with the run.
Verification
Local validation runs have been executed and verified. The GKE runner relative paths resolve successfully to the respective golden_data/ and generated output directories.