Skip to content

fix!: apply the accepted GeoZarr reader findings from EOPF-Explorer/data-model#349 - #10

Open
d-v-b wants to merge 21 commits into
claude/s2-plan-writerfrom
claude/issue-349-fixes
Open

d-v-b wants to merge 21 commits into
claude/s2-plan-writerfrom
claude/issue-349-fixes

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

This fixes the findings accepted from the independent reader's review in EOPF-Explorer#349: EOPF-Explorer#351, EOPF-Explorer#353, EOPF-Explorer#355, EOPF-Explorer#356 and EOPF-Explorer#357 in the S2 output, and EOPF-Explorer#352 across the S2, S1 RTC and OLCI converters. It is stacked on #11 (the plan-driven writer), which is stacked on EOPF-Explorer#330, and should merge after both. EOPF-Explorer#360 is the draft of this branch against upstream main; this PR stays open to show the fixes' own diff.

Warning

EOPF-Explorer#352 breaks CRS resolution in rioxarray as released (0.23 and 0.24, and its main branch). rioxarray recognizes the proj convention only under the name "proj:", while the proj v0.1 spec and zarr-cm name it "proj". With the spatial_ref grid-mapping array gone, ds.rio.crs is None for this output. The CRS is still declared, as proj:code on each group. A strict xfail test (test_rioxarray_resolves_the_level_crs) records the break and fails once a rioxarray release reads the spec's name.

What changed, by issue
Design notes and scope
  • The meteorology georeferencing needs coordinate values, not just metadata. The plan reads only metadata, so plan_geozarr takes an optional coordinate_values lookup, and write_geozarr passes its source reader's. Without the lookup, those groups get no georeferencing.
  • Remove GeoZarr V0 grid_mapping and spatial_ref from the output EOPF-Explorer/data-model#352 leaves the generic converter alone. create_geozarr_dataset, behind the CLI convert generic path and the CPM generic pipeline, is built around the GeoZarr V0 grid mapping and still writes it. Its contract test cases are marked that way.
  • The validators are unchanged. check_grid_mapping checks only a declared grid mapping, and the minispec validator tolerates one, so older stores still validate.
  • sanitize_array_attrs keeps valid_* on float arrays. Its callers pass is_decoded_float for native floats, whose ranges are in physical units, so only its docstring was corrected.
  • Fixtures were regenerated: the S2 snapshots, the OLCI snapshot and the S1 RTC fixture. The S1 fixture also picks up _FillValue and resampling_method attributes the ingester was already writing; the checked-in copy was stale.
Verification

The roborev post-commit hook reviewed every commit, and the agent fixed each valid finding; the last commits' reviews pass. The agent ran:

Check Result
Full test suite (not network), with the venv on PATH 878 passed, 5 skipped, 3 xfailed (one is the rioxarray break)
pyright clean
ruff 0.14.3 check and format clean
The issue-349 findings, recounted in the planned output of the V2 snapshots and the native V3 fixtures none left, except the meteorology groups without coordinate values in the snapshot test

🤖 Generated with Claude Code

d-v-b and others added 16 commits November 13, 2026 18:25
The five multiscale parent groups carried the finest level's
`spatial:transform` without a `spatial:shape`, so a reader could not tell
what grid the transform described. Both multiscale planners now write
the finest level's shape with its transform. A test checks that every
planned group with a transform also has its shape, and the snapshots are
regenerated (EOPF-Explorer#355).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A `flag_meanings` list is joined into the one blank-separated string CF
  defines. The SCL flag table is written as a string too.
- The `dtype` attribute, which repeats the array's data type, is dropped.
- A `valid_min` / `valid_max` outside the range of the integer data type
  is dropped. The scl `uint8` array's `valid_max: 65535` was one.
- A carried `short_name` is written for the level the array is in, as
  `<variable>_r<resolution>m`. Coarsened levels copied the source level's
  name, and the source styles were mixed.

The first three rules live in one helper, `normalize_cf_array_attrs`,
which the S2 plan and the shared `sanitize_array_attrs` both apply.
`sanitize_array_attrs` now also drops `valid_*` for decoded floats, as
its docstring said. Tests cover the helper's cases, the snapshots'
attributes, and a planted array on native and coarsened levels. The
snapshots are regenerated (EOPF-Explorer#351).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every band and carried data array got a shard, even when the whole array
fit in one inner chunk. Those shards held that one chunk, so a reader made
an index request for nothing. 201 such arrays appeared across the three
snapshots, for example the 9 x 9 meteorology and 23 x 23 angle arrays.
Such an array is now written in plain chunks, with one chunk covering it,
under the same bytes + blosc codecs. Arrays that span several inner
chunks keep their shards, including a shard larger than the array. A
test checks every planned array, and the snapshots are regenerated
(EOPF-Explorer#357).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on a2b8247. That commit made
`sanitize_array_attrs(..., is_decoded_float=True)` drop `valid_min` /
`valid_max`, as its docstring had claimed. But callers pass the flag for
every float array: `create_uniform_encoding` and the generic converter
do, for native floats whose range is in physical units. The packed-integer
output passes it too, and its range is in packed units. So correct
ranges were lost. The removal is reverted, and the docstring now states
what the code does: the decode path that unpacks integers drops `valid_*`
itself. A test pins that a float array keeps an in-range `valid_*`.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g type

The geometry label coordinates (`angle`, `band`, `detector`) carried
their source data types: `fixed_length_utf32` and `null_terminated_bytes`.
The second is in no registry, and a strict Zarr V3 reader must refuse an
unknown data type. A carried unicode or bytes array is now planned as
the registered `string` data type. `create_default` gives it the
`vlen-utf8` codec, the empty fill and one chunk. The writer decodes
bytes labels as ASCII rather than leave the conversion to zarr. A test
checks that no planned array uses either unregistered type, and the
snapshots are regenerated (EOPF-Explorer#356).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CAMS and ECMWF groups under `conditions/meteorology` held arrays on
latitude/longitude dimensions with no CRS. They also carried the source
`Conventions: "CF-1.7"`, which contradicts the output's no-CF-claim
decision.

- A carried group on a regular geographic grid gets `proj` and `spatial`
  attributes derived from its coordinate values. Such a group has 1-D
  `latitude` and `longitude` coordinates and no `proj:*` attributes. The
  attributes are `proj:code` EPSG:4326, the dimensions, the shape, and
  the transform and bbox. The values are grid points, so the
  registration is `node`, and a decreasing latitude gives a negative
  step. Irregular spacing is refused; spacing within a thousandth of the
  step, as float32 storage gives, counts as regular.
- The plan stays a function of what it is given. `plan_geozarr` takes an
  optional `coordinate_values` lookup, and `write_geozarr` passes its
  source reader's. Without the lookup, these groups get no
  georeferencing.
- The source `Conventions` attribute is no longer carried on any group.

Tests cover the derived attributes for increasing, decreasing and
float32 latitudes, the two refusals, and the plan with and without
values. The writer test seeds regular latitude/longitude values, and
the snapshots are regenerated (EOPF-Explorer#353).

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on c8c0fbd. The V2 fixtures hold only
unicode labels, so nothing reached the bytes path of EOPF-Explorer#356. A plan test
plants a `|S3` label array and checks that it is planned as `string`
with `vlen-utf8`. A writer test fills an `S3` label array into a string
array and reads back the decoded labels.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pyright refused `np.iinfo` on an unnarrowed data type in the valid-range
guardrail. It also refused `zarr.storage` attribute access and an
untyped array read in the bytes-label write test.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on bbd7637.

- A latitude/longitude group whose values `_lat_lon_attributes` refuses
  (irregular spacing, NaN fill, a single point) stopped the whole
  conversion. Such a group is now carried without georeferencing, as
  before EOPF-Explorer#353, and a warning is logged.
- Longitudes are unwrapped before the spacing check, so a grid crossing
  the antimeridian or 0/360 keeps a regular step and gets its transform.
- `_lat_lon_attributes` returns a `LatLonGroupAttrs` TypedDict, like its
  sibling builders.
- The write test asserts the derived `proj:code` and transform on the
  written meteorology groups, which shows the source reader feeds the
  plan. Plan tests cover an irregular grid, a NaN grid and an
  antimeridian grid.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 3ac4926. Longitudes are unwrapped for the
spacing check and the transform. The bbox took the unwrapped extremes,
so a grid crossing the antimeridian got an east longitude past 180. The
bbox longitudes are now moved back into [-180, 180], so the crossing
reads as west > east, as STAC and GeoJSON spell it.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…referencing

BREAKING CHANGE: no `spatial_ref` grid-mapping arrays (with `crs_wkt` /
`spatial_ref` attributes) and no `grid_mapping` attributes are written
by the S2 plan, the S1 RTC ingest, the OLCI converter or the shared
write path. The `proj` and `spatial` conventions are the only
georeferencing. An array opened on its own has no CRS; read `proj:code`
from its group.

This breaks CRS resolution in rioxarray as released (0.23, 0.24 and its
main branch). rioxarray recognizes the proj convention only under the
name "proj:", while the proj v0.1 spec and zarr-cm 0.5 name it "proj".
With the `spatial_ref` array gone, `ds.rio.crs` is None for this output.
A strict xfail test records the break and fails once a rioxarray release
reads the spec's name (EOPF-Explorer#352).

- S2 plan: no `spatial_ref` members, no `grid_mapping` on groups or
  arrays. `GridMappingAttrs` and the WKT plumbing are gone, so the
  product-level fallback no longer requires `proj:wkt2`. The writer and
  converter drop their `spatial_ref` cases.
- Shared path: `strip_cf_grid_mapping` removes the grid-mapping variables
  and every `grid_mapping` attribute and encoding. `stream_write_dataset`
  applies it, and drops the encoding of the variables it removes.
  `write_dataset_geo_metadata` writes only the convention attributes,
  and now takes a CRS given as a string.
- OLCI: the reprojection keeps rioxarray's CRS in memory for the
  computations downstream. The converter strips the grid mapping before
  each write.
- S1: the ingest no longer adds the grid mapping, and the model no
  longer requires `spatial_ref`. It still accepts it, so stores ingested
  before validate.
- The generic converter (`create_geozarr_dataset`) is built around the
  grid mapping and is left as it is. The output-contract test names it.
- Validators: `check_grid_mapping` checks only a declared grid mapping,
  and the minispec validator tolerates one. Both stay, so older stores
  still validate.
- Docs, the OLCI notebook and the changelog mark the grid mapping
  obsolete. The S2, OLCI and S1 fixtures are regenerated; the S1
  fixture also picks up attributes the ingester already wrote.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 85366f4. Moving a longitude by one turn
left a value more than one turn out of range past 180. That happens when
a wide 0-360 grid starts above 180 and wraps past 0. Longitudes outside
[-180, 180] are now normalized modulo 360. In-range values, the
antimeridian's 180 included, are kept as written. A test covers a wide
0-360 grid.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on 3c881ac.

- `stream_write_dataset` stripped the CF grid mapping even when it added
  no `proj` attributes. That is the case for the generic rechunker, so an
  input whose CRS lived only in the grid mapping was written with no CRS.
  `strip_cf_grid_mapping` now carries the CRS over as the `proj`
  convention attributes when the dataset declares no `proj:*` attribute.
  It drops a `spatial_ref` variable only when that variable is a grid
  mapping.
- The CRS parameters of `write_dataset_geo_metadata` and
  `stream_write_dataset` take a string or EPSG code, matching what the
  code accepts.
- The changelog and architecture docs say that the generic converter
  still writes the grid mapping, and that rioxarray does not yet read the
  proj convention under its spec name.

Tests cover the strip helper (CRS carried over, a declared `proj:code`
kept, an unrelated `spatial_ref` kept) and a stream write of a
grid-mapped input.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ping's CRS

Addresses roborev's findings on 2d8e58a.

- The carried `proj` attributes replaced the dataset's `zarr_conventions`,
  dropping a declared spatial or multiscales convention. The proj entry
  is now appended to the declared ones, unless already declared.
- The carry-over is skipped only when the dataset declares a proj
  convention key (`proj:code`, `proj:wkt2`, `proj:projjson`), the rule
  the validator uses. A legacy `proj:epsg` no longer counts.

Test cases cover a legacy `proj:epsg` and a declared spatial convention.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 997b123: the docstring still described the
old any-`proj:*` rule.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ixes

Assisted-by: ClaudeCode:claude-opus-5-5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d-v-b and others added 4 commits November 15, 2026 22:32
PR 330 merged EOPF-Explorer main and ported its S2 changes into the plan:
the atmosphere pyramid, the geometry and detector-footprint
georeferencing, and the band statistics. They are carried into this
branch's EOPF-Explorer#351, EOPF-Explorer#352 and EOPF-Explorer#353 work:

- s2_plan: this branch's version, with the port applied on top. The
  geometry grid and the footprint levels get no `spatial_ref` and no
  `grid_mapping` (EOPF-Explorer#352). The tile georeferencing is (origin, EPSG), with
  no WKT. The meteorology (EOPF-Explorer#353) and geometry grids share one
  regular-spacing helper (`_regular_axis`) and one refusal path, which
  carries a grid no transform describes without georeferencing, with a
  warning. `level_meters` is the one parser of level names, for the plan
  and the writer.
- conversion/utils: `strip_cf_grid_mapping` and main's node-transform
  helpers both kept. `write_dataset_geo_metadata` and
  `stream_write_dataset` take main's `registration` and this branch's
  wider CRS type, and write no grid mapping.
- CHANGELOG: this branch's Unreleased section above main's 0.13.x
  releases.
- Tests: both sides kept. PR 330's geometry test now expects no grid
  mapping. Snapshots regenerated.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 47481e8. The `level_meters` keyword
parameter of `_carried_array_attributes` and `_plan_carried_array`
shadowed the module-level `level_meters` once the merge made it public.
The parameter is now `meters`, as in `_with_level_short_name`.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR 330 now skips only the nodata the input declares when it coarsens,
so probability and atmosphere arrays that declare none are averaged
whole. The writer's loop takes this branch's side, without the
`spatial_ref` case (EOPF-Explorer#352). The tests keep both sides, and the bytes-label
test's source stub gains the readers' new `nodata` method.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
PR 330 is now validation-only, and the plan-driven writer lives on
`claude/s2-plan-writer`, which this branch is stacked on. Merging it brings
the conformance test, the int64 x/y coordinates on native levels, and the
writer's new refusals.

Conflict resolution:
- `s2_plan.py`: native levels carry the input's x/y arrays, with no
  `spatial_ref` member (EOPF-Explorer#352).
- `s2_write.py`: values go through the safe-cast check. Fixed-length unicode
  labels pass it into the `string` arrays they are planned as
  (EOPF-Explorer#356).
- The conformance test treats a codec with an empty `configuration` as its
  name-only form, which is how zarr-python stores `vlen-utf8`.
- Snapshots regenerated.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@d-v-b
d-v-b changed the base branch from feat/pydantic-zarr-scoped-validation to claude/s2-plan-writer November 16, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant