Skip to content

Which acceptance findings should fail a store, and which should only warn? #8

Open

Description

@lhoupert

I created eopf-accept as a first step in running acceptance checks that confirm new (Geo)Zarr stores work with our existing tools. It reports each finding as FAIL or WARN. A FAIL fails the run (exit 1), so a gate built on it would block the store. A WARN shows in the report and doesn't change the exit code. A FAIL can also be accepted for a limited time as a known issue: it is then reported as KNOWN, and each known issue names the check, the text it must match and an expiry date.

Several of these FAIL/WARN choices are my first guesses rather than something a spec decides. Before we use the suite as a gate for new conversions, I'd like to agree on them. Can you tell me your thoughts on that @emmanuelmathot?

Table updated on 4 Nov with the changes in this comment: GR03 split in two, the ST01 level-group warning dropped, and an RG04 row added. Updated on 5 Nov with rows for TI03, TI04, TI10 and unreadable tiles (#14) and for undecodable thumbnails (#10). The suite's code now applies the levels in this table (the 4 Nov ones came in #10), though GR02 and GR03 run only for configs with a [generic] table, today S1 GRD.

Finding Check Level Why it's at that level
A group that the collection's config expects is missing (e.g. S1 GRD without the overviews group listed in EOPF-Explorer/data-model#249) ST01, ST03, ST04, HT02 FAIL Nothing in the product can be rendered without it
Data arrays that don't follow the generic_rechunker layout (a chunk ≠ min(spatial_chunk, size), or not stored as a single shard covering the whole array) GR02 FAIL It's the writer's own rule (EOPF-Explorer/data-model#292)
CF scale_factor/add_offset on an array that also uses the scale_offset codec; a _FillValue that the dtype can't hold GR03 FAIL The values would decode wrongly (scaled twice), or nodata can't be represented
A valid_min/valid_max/valid_range that the dtype can't hold GR03 WARN (was FAIL) It hides no data. Every Sentinel-2 L2A store checked so far (six) has valid_max 65535 on the 8-bit scene classification band, so as a FAIL it would block them all
Inner chunks that make OpenLayers ≤ 10.10 draw 64 px tiles (e.g. 1024 px chunks, the S2 products of EOPF-Explorer/data-model#292) ST07 WARN (FAIL if 10.11 has it too) The OpenLayers versions in our configs (10.9/10.10) would decode 16 times or more pixels than they draw; 10.11 already picks 512 px tiles, so the WARN names the viewers to upgrade. It stays a FAIL when 10.11, the newest release modelled, still decodes 16 times or more (e.g. 2048 px inner chunks). Predicted from OpenLayers' tile-size code, not measured in a browser
zarr fill_value ≠ CF _FillValue (e.g. 0 vs 65535 on OLCI) GR03 WARN Missing chunks read as the zarr fill value, and readers that take the zarr fill value as nodata mask a different value from the one CF declares. I don't know yet whether the mismatch is already in the CPM input
2-D coordinates (swath lat/lon) stored uncompressed and unsharded GR02 WARN It's the writer's default (chunk_and_shard_coords is off); on OLCI EFR each of them is about 80 MB, read in one piece
Arrays stored the way coordinates are (uncompressed, unsharded) although no variable declares them as coordinates (OLCI altitude, time_stamp) GR02 WARN From the store alone we can't tell whether they are coordinates missing a declaration or data arrays that should have been sharded
Level groups without their own consolidated metadata ST01 None (dropped; was WARN) EOPF-Explorer/data-model#335 now says a zoom level does not need its own consolidated metadata. It was a WARN for readers that open a level group directly
zarr_conventions declarations whose fields differ from the constants in the v0.1 schemas ST11 WARN (FAIL if the config sets strict_declarations) titiler-eopf matches conventions by UUID, so stale names don't break it
spatial/proj/multiscales attribute values that geozarr-toolkit's validators reject ST12 WARN The toolkit is still 0.1.x, so a rejection may be the toolkit's bug rather than the store's
A registered item's thumbnail is an image type other than PNG (the EOPF Explorer's thumbnails are webp) RG04 None (any image type the suite can decode is accepted; was FAIL) It is a rule in the suite, not a problem with the data
A registered item's thumbnail isn't served as an image type, or the suite can't decode it (e.g. SVG, a corrupt body) RG04 FAIL The suite checks that a thumbnail has valid pixels and can't without decoding it, so a valid SVG fails too
A store level's tile has enough valid pixels but too few distinct values (fewer than 5 on L2A at stretch 0,1, fewer than 16 elsewhere), and none of up to 4 other tiles at that zoom varies with enough valid pixels TI03 FAIL (PASS, named in the summary, if another tile varies; was always FAIL) A cloud clips under the stretch in one place, a constant render is the same everywhere. A level that passes this way leaves its centre tile unverified, and a scene bright or dark on every tile tried still FAILs. A tile with too few valid pixels FAILs with no comparison. L2A's 5 values at 0,1 span about the same reflectance as 16 did at 0,0.3
An RGB tile with valid pixels has three identical channels, and none of up to 4 other tiles at that zoom has channels that differ TI04 FAIL (PASS if another tile differs; was always FAIL) A cloud that clips in all three bands is grey in one place, one band served three times is grey everywhere. A scene grey on every tile tried still FAILs, and so does a tile with no valid pixels
At a store level, the tile under the Explorer's true colour stretch (0,0.5, L2A only, on the 0.12 API) has valid pixels but fewer than 10 distinct values, and the one other tile tried doesn't vary either TI10 WARN A fully clouded scene clipping isn't a store defect, but a viewer using this stretch shows one flat colour there. TI03 decides whether the render works, and judges a tile with no valid pixels
A tile that can't be read or decoded, including one fetched only for the comparison TI03, TI04, TI10 FAIL Nothing can be judged from it, and a server error shouldn't pass unnoticed, so it counts even when another tile varies

Questions:

  1. @emmanuelmathot @d-v-b Which of these should block a store? Is anything at the wrong level?
  2. Which layout should the checks expect for each product? @shollendEODC, this one is for you in particular: Integrate/cpm geozarr main feat generic conversion s1slc data-model#292 describes the generic processor as rechunking and sharding "without the addition of geozarr multiscales", while your S1 GRD listing in Discussion: GeoZarr structure for non-S2 products data-model#249 shows an overviews group. Will that group come from a separate S1 step later? If so, the S1 GRD config shouldn't expect it from the Integrate/cpm geozarr main feat generic conversion s1slc data-model#292 products yet, and their FAIL says more about the config than about the products. The same question applies to the OLCI L1 multiscales and the S2 mask multiscales.

The checks are described in https://github.com/EOPF-Explorer/data_model_validation_tests/tree/main/acceptance#checks.

Activity

  1. shollendEODC commented on Oct 8, 2026

    @shollendEODC

    just a short answer regarding 2.)

    the layouts defined in EOPF-Explorer/data-model#249 are not yet final, and currently under development. The goal would be to merge EOPF-Explorer/data-model#292 and some other utility PRs and then make a single PR for each product, during which we can define and standardise the final output shape. Some sort of validation and product schema would be required there. This also applies to OLCI and S2, both products are not finished yet and need some generalization/refactoring from my point of view

  2. lhoupert commented on Nov 4, 2026

    @lhoupert
    CollaboratorAuthor

    Update after testing the suite on recent realease data.

    What was tested. One Sentinel-2 L2A product that EODC made with the band statistics of EOPF-Explorer/data-model#333 and the T33WWP item.

    Result. The data passes. Two checks fail, but both are problems in the suite, not in the data: RG04, whose rule is too strict (see 3 below), and TI03, which tests zoom levels this item does not use (a suite bug, to fix separately). The warnings are the usual ones for Sentinel-2 stores (ST01, ST07 and TI02). The codes are explained at the end.

    I edit the table in the issue with the changes below

    1. Row 3 (GR03): split it in two.
      • Keep as FAIL: the data would be decoded twice, or the fill value does not fit the data type.
      • Make it a WARN: a valid_min/valid_max value does not fit the data type. Every Sentinel-2 L2A store checked so far (six) has this on the scene classification band, an 8-bit band whose valid_max is 65535. It hides no data, so it should not block a store.
    2. Row 8 (ST01): drop this warning. Enhance GeoZarr mini spec with consolidated metadata details data-model#335 now says a zoom level does not need its own consolidated metadata.
    3. RG04: accept any image type for the thumbnail. The check only accepts PNG, and the Explorer's thumbnails are webp. That is a fix in the suite, not a problem with the data.

    The other rows stay as they are. No check reads the new band statistics or raster:scale yet; checks for them could go in a separate issue.

    Codes used above

    Code What it checks
    ST01 The groups the suite expects have consolidated metadata (one file listing their arrays), so a reader does not have to list the store. It warns when a zoom level has none of its own.
    ST07 Whether the chunk size lets OpenLayers draw map tiles efficiently. It warns when only older OpenLayers versions (10.10 and earlier) would struggle.
    TI02 The map-tile description (tilejson) is valid and covers the item. It warns when its zoom range cannot be compared with the store's.
    TI03 One map tile per zoom level is not blank and not a single colour.
    GR03 Scaling and fill values: the data is not scaled twice (CF scale_factor/add_offset and a scale codec together), and the fill value and valid range fit the data type.
    RG04 The links of an item registered in the Explorer (viewer, map tiles, thumbnail) work and return images with data.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions