Data Validator for Custom UN Imports

scripts/un/un_dataset_validator/README.md

@@ -0,0 +1,90 @@
+# UN Data Commons - Dataset Validation Suite
+
+This directory contains the Python-based validation suite used to ensure datasets conform to the UN Data Commons Schema mapping rules before ingestion.
+
+## Prerequisites
+
+1.  **Python 3:** Ensure Python 3.8+ is installed on your system.
+2.  **Required Libraries:** Install the dependencies required by the validation scripts:
+    ```bash
+    cd un_dataset_validator
+    pip install -r requirements.txt
+    ```
+
+## Directory Structure Expectations
+
+To run the validation successfully, your datasets and configurations must be organized into specific nested folders. The validation suite requires paths to **three** distinct directories:
+
+### 1. Processed Directory (`<processed_dir>`)
+This is the main directory containing your custom dataset's final generated mappings and outputs. It **must** follow this nested folder structure:
+
+```text
+my_processed_dir/                         <-- This is the <processed_dir> you provide
+├── processed_data/                       <-- (Required) Contains your final processed *_data.csv files
+│   ├── SDG_q1-2026_OBS_AG_FLS_INDEX_data.csv     
+│   └── SDG_q1-2026_OBS_AG_FLS_PCT_data.csv
+├── schema/                               <-- (Required) Contains your generated schemas
+│   ├── SDG_q1-2026_OBS_AG_FLS_INDEX_data_stat_vars.mcf          
+│   └── SDG_q1-2026_OBS_AG_FLS_PCT_data_stat_vars.mcf
+├── pvmap/                                <-- (Optional) Contains local Property-Value maps for this dataset
+│   └── CL_UNIT_MEASURE_pvmap.csv         
+└── dc_generated/                         <-- (Optional) Intermediate generated outputs
+```
+*Note: If your dataset relies on global mapping files (like a global `un_geography_pvmap.csv`), the suite will automatically look for a `pvmap/` folder in the **parent** directory of your `<processed_dir>` (e.g., `../my_processed_dir/../pvmap/`).*
+
+### 2. Raw Input Data Directory (`<input_data_directory>`)
+This folder contains the original, raw CSV files (the data you processed into the `processed_data` folder). It is critical that the base names correspond to the processed files.
+```text
+raw_data/
+└── DATA/                                 <-- This is the <input_data_directory> you provide
+    ├── SDG_q1-2026_OBS_AG_FLS_INDEX.csv
+    └── SDG_q1-2026_OBS_AG_FLS_PCT.csv
+```
+
+### 3. DSD Schema Directory (`<dsd_directory>`)
+This folder contains the SDMX Data Structure Definition (DSD) files used to validate your dimensions and attributes. The files typically have `_DSD_` in their names.
+```text
+raw_data/
+└── DSD/                                  <-- This is the <dsd_directory> you provide
+    ├── SDG_q1-2026_DSD_AG_FLS_INDEX.csv
+    └── SDG_q1-2026_DSD_AG_FLS_PCT.csv
+```
+
+## How to Run Validations
+
+The easiest way to run the full validation suite on your custom dataset is using the provided shell wrapper script:
+
+```bash
+cd un_dataset_validator
+./run_custom_dataset.sh <dataset_name> <processed_dir> <input_data_directory> <dsd_directory>
+```
+
+### Example Usage:
+
+Assume you have a project directory organized like this:
+```text
+my_project/
+├── raw_inputs/
+│   ├── DATA/         <-- Raw CSVs
+│   └── DSD/          <-- Raw SDMX Schemas
+├── processed/
+│   └── health_v1/    <-- Processed Dataset Folder
+│       ├── processed_data/
+│       └── schema/
+└── un_dataset_validator/       <-- This validation suite
+```
+
+To run the validation on `health_v1`, you would execute:
+
+```bash
+cd un_dataset_validator
+./run_custom_dataset.sh health_v1 ../processed/health_v1 ../raw_inputs/DATA ../raw_inputs/DSD
+```
+
+### Output & Logs
+
+The script will automatically create a dedicated log directory inside `un_dataset_validator/logs/` named `<dataset_name>_validation_logs/`.
+
+Inside this folder, you will find:
+1.  Individual `ruleX_..._validation.log` files containing detailed tracing of each rule's execution.
+2.  A final **`summary.md`** file providing a clean, comprehensive markdown matrix of which rules passed/failed alongside detailed metrics and error samples.

scripts/un/un_dataset_validator/requirements.txt

@@ -0,0 +1 @@
+pandas>=1.0.0

scripts/un/un_dataset_validator/rules/rule_10_1_coded_attributes.md

@@ -0,0 +1,23 @@
+# Rule 10.1: Coded Attributes Mapping to Property Enum
+
+## Requirement
+All coded attributes within the dataset must be correctly mapped and assigned a `property:enum` value in the Data Commons Project (DCP) schema.
+
+## Context & Rules
+- According to Rule 10, all attributes (columns marked as 'Attribute' in the DSD) should become output columns along with the observation.
+- For attributes that are specifically *coded* (meaning they pull from a defined codelist or restricted set of values, as opposed to free-text), the schema must reflect this by assigning a `property:enum` mapping.
+- This ensures that coded attributes maintain their structural integrity and defined value set in the output DCP schema.
+
+## Implementation Logic
+1. **Target Files**: Data Structure Definition (DSD) files, source data files, and schema mapping files.
+2. **Identification of Coded Attributes**:
+   - Parse the DSD file to identify columns where the ROLE is defined as 'Attribute'.
+   - Determine which of these attributes are "coded" (i.e., they reference a Code List / CL file).
+3. **Enum Mapping Validation**:
+   - Verify that for every coded attribute identified, the resulting schema generation logic assigns it a `property:enum` value mapping.
+   - Check the output files to ensure that the schema correctly reflects the enum type for these specific attributes, while leaving text attributes as raw values.
+4. **Error Flagging**:
+   - If a coded attribute is found that is NOT mapped to a `property:enum` in the schema (e.g., if it is mapped as a plain text string), flag this as a validation error.
+
+## Implementation Deviations
+- **Temporarily Excluded:** During the June 23rd meeting, it was explicitly decided that all validations enforcing base DCP schema mappings are to be temporarily ignored pending further discussion with AJ. The validation script currently skips this check to align with this decision.

scripts/un/un_dataset_validator/rules/rule_10_attributes_as_columns.md

@@ -0,0 +1,83 @@
+# Rule 10: Attributes as Output Columns
+
+## Description
+According to the UN Data Commons mapping rules, every concept defined in the dataset's Data Structure Definition (DSD) file with a `ROLE` of `Attribute` must be present as a separate column in the output `data.csv` file. 
+
+Attributes provide supplementary information (such as footnotes, observation statuses, or flags) about the observation value. Unlike dimensions, attributes must **NOT** be attached as properties to the Statistical Variables.
+
+**Note on Exclusions (Rule 10.1):** Parsing and validating the internal string structures of complex attributes (e.g., verifying comma-separated multiple footnotes inside a single `FOOTNOTE` cell) is marked as an "Ask Ajai" item and is strictly excluded from this validation check. This rule only validates the *presence* of the attribute column in the output.
+
+## Files Involved
+- **Input:** Dataset-specific DSD file (e.g., `schema/dsd.csv` or similar file defining `ROLE`).
+- **Output:** The generated data CSV file (e.g., `SDG_q1-2026_OBS_AG_FOOD_WST_data.csv`).
+
+## Validation Logic
+1. **Identify Attributes:** Parse the DSD file and filter for all rows where the `ROLE` column is equal to `Attribute` (case-insensitive).
+2. **Extract Identifiers:** Extract the concept identifier/name for each attribute (e.g., `OBS_STATUS`, `FOOTNOTE`).
+3. **Verify Output Columns:** Read the header of the generated output `data.csv` file.
+4. **Compare:** Ensure that every attribute identified in step 1 exists as a distinct column in the output CSV header.
+5. **Report Missing Columns:** If an attribute defined in the DSD is missing from the output CSV, flag it as a validation failure.
+
+## Python Implementation
+
+```python
+import pandas as pd
+import glob
+import os
+
+def validate_rule_10(dsd_file_path: str, output_csv_path: str) -> bool:
+    """
+    Validates that all concepts with ROLE='Attribute' in the DSD are present as 
+    columns in the output CSV.
+    """
+    print(f"Validating Rule 10: Attributes as Output Columns for {os.path.basename(output_csv_path)}")
+
+    try:
+        # 1. Read DSD and find Attributes
+        dsd_df = pd.read_csv(dsd_file_path)
+
+        # Ensure required columns exist
+        # Note: Actual column names for Concept/Identifier and Role might vary slightly
+        # Adjust 'concept' and 'role' based on the exact DSD schema structure.
+        role_col = next((c for c in dsd_df.columns if c.strip().lower() == 'role'), None)
+        concept_col = next((c for c in dsd_df.columns if c.strip().lower() in ['concept', 'id', 'name']), None)
+
+        if not role_col or not concept_col:
+            print(f"  [ERROR] DSD file missing 'ROLE' or Concept identifier column.")
+            return False
+
+        # Filter attributes and get their names
+        attributes = dsd_df[dsd_df[role_col].str.lower() == 'attribute'][concept_col].dropna().tolist()
+        # Clean up strings
+        expected_attribute_cols = [attr.strip() for attr in attributes]
+
+        if not expected_attribute_cols:
+            print("  [INFO] No attributes found in DSD. Validation passed.")
+            return True
+
+        # 2. Read Output CSV headers
+        output_df = pd.read_csv(output_csv_path, nrows=0) # Read only header
+        output_columns = [col.strip() for col in output_df.columns]
+
+        # 3. Verify presence
+        missing_attributes = []
+        for attr in expected_attribute_cols:
+            # Check exact match or case-insensitive match based on pipeline behavior
+            match_found = any(attr.lower() == col.lower() for col in output_columns)
+            if not match_found:
+                missing_attributes.append(attr)
+
+        if missing_attributes:
+            print(f"  [FAILURE] Missing Attribute columns in output CSV: {missing_attributes}")
+            return False
+
+        print("  [SUCCESS] All DSD Attributes are present as columns in the output CSV.")
+        return True
+
+    except Exception as e:
+        print(f"  [ERROR] Validation failed due to exception: {e}")
+        return False
+
+# Example Usage:
+# validate_rule_10('schema/dsd.csv', 'extracted_data_new/20260615/SDG_q1-2026_OBS_AG_FOOD_WST_data.csv')
+```

scripts/un/un_dataset_validator/rules/rule_11_statvar_dcid.md

@@ -0,0 +1,41 @@
+# Rule 11: StatVar DCID Format & Special Characters
+
+## Overview
+This rule validates the structural format of the Data Commons Identifier (DCID) generated for each Statistical Variable (StatVar). The DCID must strictly adhere to a specific templated format, and any illegal or special characters within the originating codes must be converted to underscores (`_`) to ensure valid identifier syntax.
+
+## Files Involved
+- **Output:** `output_stat_vars.mcf` (Specifically examining the `Node: dcid:...` lines).
+
+## Implementation Details
+
+### 1. DCID Template
+The DCID must be constructed using the following template:
+`<prefix>/<agency>/<SERIES>[.<CONCEPT>--<CODE>__…]`
+
+Where:
+- `<prefix>` is generally `undata`.
+- `<agency>` is the agency identifier (e.g., `sdg`).
+- `<SERIES>` is the identifier of the series (e.g., `AG_FLS_INDEX`).
+- The `[.<CONCEPT>--<CODE>__…]` portion represents the dimensional constraints attached to the StatVar, chained together. 
+
+### 2. Special Character Conversion
+Any special characters present in the original `<SERIES>`, `<CONCEPT>`, or `<CODE>` values (such as spaces, dashes, slashes, or parentheses) MUST be converted to underscores (`_`) before being concatenated into the DCID string.
+
+## Example
+Given an agency of `sdg`, a series of `AG_FLS_INDEX`, a concept of `PRODUCT`, and a code of `AGG_ANIMAL_PROD`:
+
+**Expected MCF Node Definition:**
+```
+Node: dcid:undata/sdg/AG_FLS_INDEX.PRODUCT--AGG_ANIMAL_PROD
+```
+
+## Python Implementation Strategy
+
+1. **Parse MCF:** Read the `output_stat_vars.mcf` file and extract all values from the `Node:` property that begin with `dcid:`.
+2. **Regex Validation:** Use a regular expression to validate the structural integrity of the extracted DCID against the expected template. A simplified regex structure would look like: `^dcid:undata/[a-z0-9_-]+/[A-Z0-9_-]+(\.[A-Z0-9_-]+--[A-Z0-9_-]+(__[A-Z0-9_-]+--[A-Z0-9_-]+)*)?$`. (Note: The exact regex will need to be refined based on the full scope of allowed characters in standard DCIDs, but it must enforce the template hierarchy).
+3. **Character Check:** Independently verify that no spaces or unescaped/unconverted special characters exist within the identifier string after the `dcid:` prefix.
+4. **Reconciliation (Advanced):** If the input data is available during this check, independently reconstruct the expected DCID using the input series and dimension codes (applying the underscore conversion logic) and verify it matches the DCID found in the MCF.
+
+## Implementation Deviations
+- **Flexible Agency/Prefix:** The implemented regular expression (`^dcid:[a-zA-Z0-9_]+/[a-zA-Z0-9_]+/...`) dynamically accepts any alphanumeric sequence for the prefix and agency, rather than hardcoding `undata`.
+- **Hyphens vs. Underscores:** The strict regex implementation exclusively expects underscores (`_`) as word separators in series, concepts, and codes, differing from earlier examples that suggested hyphens (`-`) might be allowed.

scripts/un/un_dataset_validator/rules/rule_12_names.md

@@ -0,0 +1,59 @@
+# Rule 12: Names (alternateName, Value names, nameWithLanguage)
+
+## 1. Rule Description
+* **Core Rule (12):** This rule ensures that names for properties and values generated in the schema accurately reflect their source definitions in the DSD (Data Structure Definition) and Codelists.
+* **Sub-rule (12.1):** A property's `alternateName` must match the corresponding name defined in the DSD file.
+* **Sub-rule (12.2):** A value's name must match the name defined in the specific concept's codelist (`CL` file).
+* **Sub-rule (12.3):** Any names available in languages other than the default must be appropriately added to the `nameWithLanguage` property.
+
+## 2. Files Involved
+* **Data Input:** The transcoded dataset.
+* **DSD File:** Defines the structural metadata and concepts.
+* **Codelists (CL):** Define the valid values and their corresponding names for each concept.
+* **Output MCF Files:** The generated schema and stat vars where these properties are defined.
+
+## 3. Validation Logic & Flow
+1. **Property Names (12.1):** 
+   - Parse the generated schema.
+   - For each property, locate its corresponding concept in the DSD.
+   - Assert that the `alternateName` in the generated schema exactly matches the name string in the DSD.
+2. **Value Names (12.2):**
+   - Parse the generated schema.
+   - For each value, identify its concept and find the corresponding codelist file.
+   - Assert that the value name in the generated schema matches the name in the codelist.
+3. **Multi-lingual Names (12.3):**
+   - If the DSD or codelists provide names in multiple languages (e.g., using language tags), check that the generated schema utilizes the `nameWithLanguage` property correctly to represent these translations.
+
+## 4. Python Implementation Strategy (Draft)
+
+```python
+import pandas as pd
+# Pseudocode structure for Rule 12 Validation
+
+def validate_rule_12(schema_mcf_path: str, dsd_path: str, cl_dir_path: str) -> dict:
+    """
+    Validates naming conventions against DSD and Codelists.
+    """
+    errors = []
+
+    # 1. Load DSD for property names
+    # dsd_df = pd.read_csv(dsd_path)
+
+    # 2. Parse MCF to extract properties and values
+    # ...
+
+    # 3. Check 12.1: Property alternateName vs DSD
+    # ...
+
+    # 4. Check 12.2: Value names vs Codelists
+    # ...
+
+    # 5. Check 12.3: nameWithLanguage
+    # ...
+
+    return {"status": "PASSED" if not errors else "FAILED", "errors": errors}
+```
+
+## Implementation Deviations
+- **String Normalization:** To account for pipeline inconsistencies (such as varying quotes, spacing, or punctuation), the script implements a `normalize_name()` function. This strips all punctuation, converts strings to lowercase, and normalizes spacing before performing the string comparison.
+- **Missing StatVar Names:** Due to a known pipeline issue where `name` properties are omitted from `StatisticalVariable` nodes, the validator currently logs a warning (rather than a failure) when this occurs.

scripts/un/un_dataset_validator/rules/rule_13_1_statvar_name_template.md

@@ -0,0 +1,27 @@
+# Rule 13.1: Statvar Name Template
+
+## Requirement
+The name assigned to a Statistical Variable (StatVar) must adhere to a specific template format to ensure consistency and readability across the Data Commons Project (DCP).
+
+## Context & Rules
+- According to the checklist, the required template for a StatVar name is:
+  `"<series name> [<concept name>=<code name>, ...]"`
+- This template provides a clear, human-readable summary of the underlying data series and the specific constraint properties (concepts and codes) that define the statistical variable.
+- For example, if the series is "Unemployment Rate" and the concepts are "Age" and "Gender" with codes "15-24" and "Female" respectively, the name should be constructed as:
+  `"Unemployment Rate [Age=15-24, Gender=Female]"`
+
+## Implementation Logic
+1. **Target Files**: Output MCF files where StatVars are defined (e.g., `output_stat_vars.mcf`), and relevant source data/DSD files to fetch names.
+2. **Template Validation**:
+   - Extract the generated `name` property for each StatVar node.
+   - Deconstruct the underlying series name and its associated constraint concepts and codes.
+   - Verify that the generated name strictly matches the format: `"<series name> [<concept name>=<code name>, ...]"`
+3. **Format Checks**:
+   - Ensure the square brackets `[]` are used correctly to enclose the constraints.
+   - Ensure a comma and a space `, ` separate multiple concept=code pairs.
+   - Ensure an equals sign `=` separates the concept name and the code name.
+4. **Error Flagging**:
+   - Flag a validation error if a StatVar name does not conform to this template or if the constituent parts (series name, concept name, code name) do not accurately reflect the underlying data definition.
+
+## Implementation Deviations
+- **Skipped Execution:** This validation is currently skipped entirely by the script. Because the data pipeline does not reliably generate the `name` property for `StatisticalVariable` nodes (a known missing feature tracked under "Ask Ajai" rules), the script logs a warning and marks the validation as "PASSED (Skipped)".

scripts/un/un_dataset_validator/rules/rule_15_duplicate_statvar_properties.md

@@ -0,0 +1,15 @@
+# Rule 15: Duplicate Properties in Statistical Variables
+
+## Description
+This rule ensures that no property or concept code is repeated twice within the same Statistical Variable (StatVar) definition. Duplicating dimension values unnecessarily violates the Data Commons structural requirements and can lead to inconsistent behavior downstream.
+
+## Implementation Details
+* **Script:** `scripts/test_rule_15.py`
+* **Target:** Parsed `*_stat_vars.mcf` (or `.tmcf`) files located in the `processed_data` directory.
+* **Validation Logic:** 
+    1. Scan each file for StatVar definitions, identifying blocks that start with `Node: dcid:...`.
+    2. Track all property keys defined inside that specific block.
+    3. If any key is encountered more than once within the same node, flag it as a violation.
+
+## Remediation
+If this rule fails, review the PV map generation logic or custom mapping script. Ensure that dimension-value pairs are properly aggregated and that a single concept/property is only mapped once for a given Statistical Variable.