Repository navigation
Conversation
The five multiscale parent groups carried the finest level's `spatial:transform` without a `spatial:shape`, so a reader could not tell what grid the transform described. Both multiscale planners now write the finest level's shape with its transform. A test checks that every planned group with a transform also has its shape, and the snapshots are regenerated (EOPF-Explorer#355). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A `flag_meanings` list is joined into the one blank-separated string CF defines. The SCL flag table is written as a string too. - The `dtype` attribute, which repeats the array's data type, is dropped. - A `valid_min` / `valid_max` outside the range of the integer data type is dropped. The scl `uint8` array's `valid_max: 65535` was one. - A carried `short_name` is written for the level the array is in, as `<variable>_r<resolution>m`. Coarsened levels copied the source level's name, and the source styles were mixed. The first three rules live in one helper, `normalize_cf_array_attrs`, which the S2 plan and the shared `sanitize_array_attrs` both apply. `sanitize_array_attrs` now also drops `valid_*` for decoded floats, as its docstring said. Tests cover the helper's cases, the snapshots' attributes, and a planted array on native and coarsened levels. The snapshots are regenerated (EOPF-Explorer#351). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every band and carried data array got a shard, even when the whole array fit in one inner chunk. Those shards held that one chunk, so a reader made an index request for nothing. 201 such arrays appeared across the three snapshots, for example the 9 x 9 meteorology and 23 x 23 angle arrays. Such an array is now written in plain chunks, with one chunk covering it, under the same bytes + blosc codecs. Arrays that span several inner chunks keep their shards, including a shard larger than the array. A test checks every planned array, and the snapshots are regenerated (EOPF-Explorer#357). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on a2b8247. That commit made `sanitize_array_attrs(..., is_decoded_float=True)` drop `valid_min` / `valid_max`, as its docstring had claimed. But callers pass the flag for every float array: `create_uniform_encoding` and the generic converter do, for native floats whose range is in physical units. The packed-integer output passes it too, and its range is in packed units. So correct ranges were lost. The removal is reverted, and the docstring now states what the code does: the decode path that unpacks integers drops `valid_*` itself. A test pins that a float array keeps an in-range `valid_*`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g type The geometry label coordinates (`angle`, `band`, `detector`) carried their source data types: `fixed_length_utf32` and `null_terminated_bytes`. The second is in no registry, and a strict Zarr V3 reader must refuse an unknown data type. A carried unicode or bytes array is now planned as the registered `string` data type. `create_default` gives it the `vlen-utf8` codec, the empty fill and one chunk. The writer decodes bytes labels as ASCII rather than leave the conversion to zarr. A test checks that no planned array uses either unregistered type, and the snapshots are regenerated (EOPF-Explorer#356). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CAMS and ECMWF groups under `conditions/meteorology` held arrays on latitude/longitude dimensions with no CRS. They also carried the source `Conventions: "CF-1.7"`, which contradicts the output's no-CF-claim decision. - A carried group on a regular geographic grid gets `proj` and `spatial` attributes derived from its coordinate values. Such a group has 1-D `latitude` and `longitude` coordinates and no `proj:*` attributes. The attributes are `proj:code` EPSG:4326, the dimensions, the shape, and the transform and bbox. The values are grid points, so the registration is `node`, and a decreasing latitude gives a negative step. Irregular spacing is refused; spacing within a thousandth of the step, as float32 storage gives, counts as regular. - The plan stays a function of what it is given. `plan_geozarr` takes an optional `coordinate_values` lookup, and `write_geozarr` passes its source reader's. Without the lookup, these groups get no georeferencing. - The source `Conventions` attribute is no longer carried on any group. Tests cover the derived attributes for increasing, decreasing and float32 latitudes, the two refusals, and the plan with and without values. The writer test seeds regular latitude/longitude values, and the snapshots are regenerated (EOPF-Explorer#353). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on c8c0fbd. The V2 fixtures hold only unicode labels, so nothing reached the bytes path of EOPF-Explorer#356. A plan test plants a `|S3` label array and checks that it is planned as `string` with `vlen-utf8`. A writer test fills an `S3` label array into a string array and reads back the decoded labels. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pyright refused `np.iinfo` on an unnarrowed data type in the valid-range guardrail. It also refused `zarr.storage` attribute access and an untyped array read in the bytes-label write test. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on bbd7637. - A latitude/longitude group whose values `_lat_lon_attributes` refuses (irregular spacing, NaN fill, a single point) stopped the whole conversion. Such a group is now carried without georeferencing, as before EOPF-Explorer#353, and a warning is logged. - Longitudes are unwrapped before the spacing check, so a grid crossing the antimeridian or 0/360 keeps a regular step and gets its transform. - `_lat_lon_attributes` returns a `LatLonGroupAttrs` TypedDict, like its sibling builders. - The write test asserts the derived `proj:code` and transform on the written meteorology groups, which shows the source reader feeds the plan. Plan tests cover an irregular grid, a NaN grid and an antimeridian grid. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 3ac4926. Longitudes are unwrapped for the spacing check and the transform. The bbox took the unwrapped extremes, so a grid crossing the antimeridian got an east longitude past 180. The bbox longitudes are now moved back into [-180, 180], so the crossing reads as west > east, as STAC and GeoJSON spell it. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…referencing BREAKING CHANGE: no `spatial_ref` grid-mapping arrays (with `crs_wkt` / `spatial_ref` attributes) and no `grid_mapping` attributes are written by the S2 plan, the S1 RTC ingest, the OLCI converter or the shared write path. The `proj` and `spatial` conventions are the only georeferencing. An array opened on its own has no CRS; read `proj:code` from its group. This breaks CRS resolution in rioxarray as released (0.23, 0.24 and its main branch). rioxarray recognizes the proj convention only under the name "proj:", while the proj v0.1 spec and zarr-cm 0.5 name it "proj". With the `spatial_ref` array gone, `ds.rio.crs` is None for this output. A strict xfail test records the break and fails once a rioxarray release reads the spec's name (EOPF-Explorer#352). - S2 plan: no `spatial_ref` members, no `grid_mapping` on groups or arrays. `GridMappingAttrs` and the WKT plumbing are gone, so the product-level fallback no longer requires `proj:wkt2`. The writer and converter drop their `spatial_ref` cases. - Shared path: `strip_cf_grid_mapping` removes the grid-mapping variables and every `grid_mapping` attribute and encoding. `stream_write_dataset` applies it, and drops the encoding of the variables it removes. `write_dataset_geo_metadata` writes only the convention attributes, and now takes a CRS given as a string. - OLCI: the reprojection keeps rioxarray's CRS in memory for the computations downstream. The converter strips the grid mapping before each write. - S1: the ingest no longer adds the grid mapping, and the model no longer requires `spatial_ref`. It still accepts it, so stores ingested before validate. - The generic converter (`create_geozarr_dataset`) is built around the grid mapping and is left as it is. The output-contract test names it. - Validators: `check_grid_mapping` checks only a declared grid mapping, and the minispec validator tolerates one. Both stay, so older stores still validate. - Docs, the OLCI notebook and the changelog mark the grid mapping obsolete. The S2, OLCI and S1 fixtures are regenerated; the S1 fixture also picks up attributes the ingester already wrote. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 85366f4. Moving a longitude by one turn left a value more than one turn out of range past 180. That happens when a wide 0-360 grid starts above 180 and wraps past 0. Longitudes outside [-180, 180] are now normalized modulo 360. In-range values, the antimeridian's 180 included, are kept as written. A test covers a wide 0-360 grid. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on 3c881ac. - `stream_write_dataset` stripped the CF grid mapping even when it added no `proj` attributes. That is the case for the generic rechunker, so an input whose CRS lived only in the grid mapping was written with no CRS. `strip_cf_grid_mapping` now carries the CRS over as the `proj` convention attributes when the dataset declares no `proj:*` attribute. It drops a `spatial_ref` variable only when that variable is a grid mapping. - The CRS parameters of `write_dataset_geo_metadata` and `stream_write_dataset` take a string or EPSG code, matching what the code accepts. - The changelog and architecture docs say that the generic converter still writes the grid mapping, and that rioxarray does not yet read the proj convention under its spec name. Tests cover the strip helper (CRS carried over, a declared `proj:code` kept, an unrelated `spatial_ref` kept) and a stream write of a grid-mapped input. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ping's CRS Addresses roborev's findings on 2d8e58a. - The carried `proj` attributes replaced the dataset's `zarr_conventions`, dropping a declared spatial or multiscales convention. The proj entry is now appended to the declared ones, unless already declared. - The carry-over is skipped only when the dataset declares a proj convention key (`proj:code`, `proj:wkt2`, `proj:projjson`), the rule the validator uses. A legacy `proj:epsg` no longer counts. Test cases cover a legacy `proj:epsg` and a declared spatial convention. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 997b123: the docstring still described the old any-`proj:*` rule. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ixes Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR 330 merged EOPF-Explorer main and ported its S2 changes into the plan: the atmosphere pyramid, the geometry and detector-footprint georeferencing, and the band statistics. They are carried into this branch's EOPF-Explorer#351, EOPF-Explorer#352 and EOPF-Explorer#353 work: - s2_plan: this branch's version, with the port applied on top. The geometry grid and the footprint levels get no `spatial_ref` and no `grid_mapping` (EOPF-Explorer#352). The tile georeferencing is (origin, EPSG), with no WKT. The meteorology (EOPF-Explorer#353) and geometry grids share one regular-spacing helper (`_regular_axis`) and one refusal path, which carries a grid no transform describes without georeferencing, with a warning. `level_meters` is the one parser of level names, for the plan and the writer. - conversion/utils: `strip_cf_grid_mapping` and main's node-transform helpers both kept. `write_dataset_geo_metadata` and `stream_write_dataset` take main's `registration` and this branch's wider CRS type, and write no grid mapping. - CHANGELOG: this branch's Unreleased section above main's 0.13.x releases. - Tests: both sides kept. PR 330's geometry test now expects no grid mapping. Snapshots regenerated. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 47481e8. The `level_meters` keyword parameter of `_carried_array_attributes` and `_plan_carried_array` shadowed the module-level `level_meters` once the merge made it public. The parameter is now `meters`, as in `_with_level_short_name`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR 330 now skips only the nodata the input declares when it coarsens, so probability and atmosphere arrays that declare none are averaged whole. The writer's loop takes this branch's side, without the `spatial_ref` case (EOPF-Explorer#352). The tests keep both sides, and the bytes-label test's source stub gains the readers' new `nodata` method. Assisted-by: ClaudeCode:claude-fable-5.2 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
PR 330 is now validation-only, and the plan-driven writer lives on `claude/s2-plan-writer`, which this branch is stacked on. Merging it brings the conformance test, the int64 x/y coordinates on native levels, and the writer's new refusals. Conflict resolution: - `s2_plan.py`: native levels carry the input's x/y arrays, with no `spatial_ref` member (EOPF-Explorer#352). - `s2_write.py`: values go through the safe-cast check. Fixed-length unicode labels pass it into the `string` arrays they are planned as (EOPF-Explorer#356). - The conformance test treats a codec with an empty `configuration` as its name-only form, which is how zarr-python stores `vlen-utf8`. - Snapshots regenerated. Assisted-by: ClaudeCode:claude-fable-5.2 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
d-v-b
changed the base branch from
feat/pydantic-zarr-scoped-validation
to
claude/s2-plan-writer
November 16, 2026 08:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
This fixes the findings accepted from the independent reader's review in EOPF-Explorer#349: EOPF-Explorer#351, EOPF-Explorer#353, EOPF-Explorer#355, EOPF-Explorer#356 and EOPF-Explorer#357 in the S2 output, and EOPF-Explorer#352 across the S2, S1 RTC and OLCI converters. It is stacked on #11 (the plan-driven writer), which is stacked on EOPF-Explorer#330, and should merge after both. EOPF-Explorer#360 is the draft of this branch against upstream
main; this PR stays open to show the fixes' own diff.Warning
EOPF-Explorer#352 breaks CRS resolution in rioxarray as released (0.23 and 0.24, and its main branch). rioxarray recognizes the proj convention only under the name
"proj:", while the proj v0.1 spec and zarr-cm name it"proj". With thespatial_refgrid-mapping array gone,ds.rio.crsisNonefor this output. The CRS is still declared, asproj:codeon each group. A strict xfail test (test_rioxarray_resolves_the_level_crs) records the break and fails once a rioxarray release reads the spec's name.What changed, by issue
spatial:shapeon multiscale parent groups EOPF-Explorer/data-model#355: the multiscale parent groups state thespatial:shapetheirspatial:transformdescribes.flag_meaningslist becomes one blank-separated string, the SCL table included.dtypeattribute is dropped.valid_min/valid_maxoutside the integer data type's range is dropped.short_nameis written per level, as<variable>_r<resolution>m, coarsened levels included.normalize_cf_array_attrs, which the sharedsanitize_array_attrsalso applies.stringEOPF-Explorer/data-model#356: fixed-length unicode and bytes labels (angle,band,detector) are stored as the registeredstringdata type. The writer decodes bytes as ASCII.projandspatialattributes derived from their latitude/longitude values. They usenoderegistration, signed steps, unwrapped longitudes, and a bbox within [-180, 180].Conventions: "CF-1.7".spatial_refarrays and nogrid_mappingattributes are written by the S2 plan, the S1 RTC ingest, the OLCI converter or the shared write path.strip_cf_grid_mappingremoves them before writing. When a dataset has no proj convention key, it first carries the grid mapping's CRS over asproj:code.spatial_ref, but still accepts it, so stores ingested before validate.CHANGELOG.md(Unreleased, Breaking) mark the grid mapping obsolete.Design notes and scope
plan_geozarrtakes an optionalcoordinate_valueslookup, andwrite_geozarrpasses its source reader's. Without the lookup, those groups get no georeferencing.create_geozarr_dataset, behind the CLIconvertgeneric path and the CPMgenericpipeline, is built around the GeoZarr V0 grid mapping and still writes it. Its contract test cases are marked that way.check_grid_mappingchecks only a declared grid mapping, and the minispec validator tolerates one, so older stores still validate.sanitize_array_attrskeepsvalid_*on float arrays. Its callers passis_decoded_floatfor native floats, whose ranges are in physical units, so only its docstring was corrected._FillValueandresampling_methodattributes the ingester was already writing; the checked-in copy was stale.Verification
The roborev post-commit hook reviewed every commit, and the agent fixed each valid finding; the last commits' reviews pass. The agent ran:
PATH🤖 Generated with Claude Code