Repository navigation
Conversation
…tic-zarr main
pydantic-zarr now validates every Zarr V3 array spec as a whole metadata
document, judging each extension-point name by the definition that
claims it in a scope. The GeoZarr DataArray sets zarr_scope_strict, so
a typo'd or unknown codec or data type name is refused instead of
passing unjudged, against a GeoZarr scope that extends the default with
zarr-python's provisional fixed_length_utf32 data type (band and angle
coordinates in real stores hold it, and no zarr-metadata scope claims
it yet). The pyz wrappers drop their attributes redeclaration: the
parent field now carries a validated empty-dict default, so narrowed
attribute types that refuse {} stay required without it.
pydantic-zarr comes from git main pending its next release.
Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bers override The pyz ArraySpec type parameter was phantom (both wrappers subclassed ArraySpec[Any]), so every declared attributes model on the Sentinel array classes was decorative: Sentinel2DataArray accepted arbitrary attributes and held plain dicts. The parameter is now passed through, so attributes validate against the declared model and read back typed. The GroupSpec wrappers stop binding the members mapping type into the parent's member-item slot (the old type-var hack and both its ignores): with pydantic-zarr's covariant parameters, parameterizing the parent at its widest member type and narrowing the members field is a legal override. s3_olci spells unconstrained attributes Mapping[str, object] (object no longer satisfies the enforced bound), and the GeoZarr v3 group classes use AnyGroupSpec, now an ordinary wide type, so member reads stay checked. pydantic-zarr is bumped to main with covariant spec parameters (zarr-developers/pydantic-zarr#191). Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
load_sentinel2 reads a Zarr store into the validated Sentinel2Root input model — keeping the model as the carrier instead of validating and discarding it — and plan_geozarr is a pure function from that model to the GeoZarr hierarchy to write, as a Zarr V3 GroupSpec tree: the metadata plane of the conversion, independent of chunk copying. The reflectance pyramid (native levels with the product level's finer bands injected, coarser levels derived, sharded uint16 bands, float64 coordinates, CF grid mappings from the product's own proj:wkt2 via a CF round-trip, spatial/proj/multiscales convention attributes) is derived entirely from the input model's metadata, and reproduces the converter's stored snapshot byte for byte for the product whose tile the snapshot harness models; for the other products the snapshots carry the harness's synthetic tile georeferencing, so the tests mask those values and pin the plan's georeferencing to the product's own band attributes instead. conditions and quality stay on the DataTree converter for now. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…quality plan_geozarr now plans the whole hierarchy. The aux multiscale groups (l1c/l2a classification, cloud probabilities) get the same pyramid treatment as reflectance via AUX_MULTISCALE_LEVELS — native levels carried, coarser levels derived with attributes verbatim, nearest resampling when the arrays carry a flag table — and every other conditions/quality dataset is carried: georeferenced datasets (detector footprints, quality masks, atmosphere) gain spatial attributes and a spatial_ref grid mapping derived from their own proj attributes, geometry and meteorology are translated as-is (string arrays to fixed_length_utf32, coordinates uncompressed, data arrays sharded), and the quicklook groups and scalar arrays are dropped, as the converter drops them. The planner walks the input model's dump, since the input model types these subtrees loosely. Runtime assert statements are replaced with raises or restructured away. The whole planned tree is byte-for-byte equal to the stored converter snapshot for the product whose tile the snapshot harness models; for the others the tests mask the harness's synthetic-tile georeferencing and the generated aux levels' chunk shapes, which the converter derives from incidental dask chunking (inner chunks of 915, 512 and 1830 for the same situation across the snapshots) where the plan uses its uniform rule. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…re encoding Coarsening divides whatever dask chunks the source dataset happens to have, so the stored encoding of a generated level depended on the incidental chunking of the object it was coarsened from: the snapshots held inner chunks of 915, 512 and 1830 for like levels. generic multiscales now rechunks the coarsened dataset to the spatial chunk target (min(spatial_chunk, dim) per dimension) before deriving the encoding, matching the native levels' policy and the spec-level plan's uniform rule. The snapshots are regenerated (the only other change is the omission of null dimension_names, which current pydantic-zarr no longer serializes), and the plan tests drop their generated-level chunk masking: the whole tree is now pinned. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Picks up the unreleased extra-fields support (consolidated root documents now validate, carried in extra_fields and refused by to_zarr), the typed to_metadata_json / to_stored_metadata_json exports, the always-complete GroupSpec hierarchy, and whole-document validation for Zarr V2 arrays. Whole-repo pyright and the full non-network suite are green against it. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…data and zarr-cm Every zarr metadata field the Sentinel-2 plan writes now carries its published type: zarr-metadata's *Object TypedDicts for the extension points (regular chunk grid, default chunk key encoding, and the bytes, blosc, zstd, crc32c and sharding_indexed codecs), zarr-cm's convention types for the zarr-conventions attributes (the CMO declarations replace the hand-written registry dicts, and layout entries are LayoutObject with the spatial convention's keys as typed extra items), and dedicated TypedDicts for the fixed-shape documents the plan builds itself: GridMappingAttrs (narrowed at runtime from pyproj's CF output via zarr_metadata.typed_json.check), SpatialGroupAttrs, MultiscaleGroupAttrs, and FixedLengthUtf32Object in the GeoZarr scope. Function-scoped imports move to the top of their modules. No runtime change: the stored snapshots are byte-identical. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds data_api/s2_write.py: write_geozarr realizes the plan with zarr-python and numpy alone. The planned metadata documents are written first (pydantic-zarr to_zarr, so every stored document is exactly the plan), then the chunk data is filled in: native arrays copied from the input store, injected bands block-resampled from the finest input level that holds them, generated pyramid levels chained from the level before them with the converter's arithmetic (nodata-masked rounded block mean, window-centre subsampling for classifications, window means for the coordinates), and grid-mapping arrays set to zero. Product-level publication (root attributes, consolidation, STAC) stays the converter pipeline's separate final step. The parity test runs write_geozarr and create_multiscale_from_datatree on the same seeded input (each product's own proj:transform drives the coordinates; small arrays and the scene classifications carry patterned values) and compares the stores file by file: identical file sets, identical metadata documents, and byte-identical chunks, for all three fixture products. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Converts the pyz GroupSpec/ArraySpec wrappers from Generic[...] bases with module-level TypeVars to PEP 695 type parameters (ruff UP046), and drops an unused import this branch had left in geozarr/v3.py. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…move the DataTree writer convert_s2_optimized now validates the input store into Sentinel2Root, plans the output with plan_geozarr, and writes it with write_geozarr — metadata documents first, chunks after, root consolidation and STAC unchanged as the final step, fed by a dataset index derived from the plan. The DataTree-based write path (create_multiscale_from_datatree, generic_multiscales, stream_write_s2dataset, inject_missing_bands, the coarsening kernels and write_geo_metadata) is removed, after the plan-driven writer reproduced its stores byte for byte; s2_multiscale keeps only the declarative policy tables and the SCL vocabulary. The write policy is the plan's fixed policy (sharding on, 1024-pixel inner chunks, blosc-zstd level 3, ESA packed-integer layout): deviating option values are refused with a message naming them, the layout flags are gone from the convert-s2-optimized CLI command and ignored on the auto-detect convert path, and the CPM writer forwards only the options the S2 pipeline still interprets. The migration also widens the inputs the plan accepts, which the DataTree path tolerated and the Zarr V3 (N0512) fixtures exercise: a tree with no backing store or a non-V2 backing store is materialized to a scratch V2 store (minus its per-variable zarr storage encoding), conditions/meteorology is optional in the input model, georeferencing falls back to the product-level stac_discovery proj declaration when the band and mask arrays carry none, and numpy S dtypes map to zarr-python's null_terminated_bytes, now claimed by the GeoZarr scope. Tests: the writer tests pin plan realization and the resampling semantics against independently formulated references (the byte parity against the DataTree writer was proven before its removal); snapshot regeneration moves to the plan tests; the multiscale tests keep the encoding and end-to-end structural coverage; the geo-metadata unit tests go with the code they tested. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…and drop the scratch copy The converter no longer materializes in-memory or Zarr V3-backed trees to a scratch Zarr V2 store. New `pyz.datatree` is the DataTree analogue of `GroupSpec.from_zarr`: `group_spec_from_datatree` extracts a tree's metadata plane as the Zarr V2 documents xarray's own zarr writer would store (every variable through `encode_zarr_variable`, CF packing applied, storage-layout encoding keys dropped), so `Sentinel2Root` validates a tree's named members and typed attributes exactly as it validates a store. `load_sentinel2` accepts a DataTree, and `write_geozarr` reads the input through a `SourceReader` protocol with a zarr-store and a DataTree implementation, the latter reading values in the same encoded form. A parity test pins the equivalence: on the seeded Sentinel-2 fixtures, the tree source writes the same store as the zarr-store source, chunk for chunk, whether the tree holds the raw packed values or the decoded ones (which the reader re-encodes as xarray would have on a store write). Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Correctness and robustness: - `_window_mean` keeps an all-NaN float window NaN instead of writing 0.0, matching the removed converter's skipna mean and `_coarsen`'s docstring. - `_plan_subtree` carries a mixed group's own arrays alongside its planned subgroups instead of dropping them silently. - `plan_reflectance` raises its deliberate ValueErrors (no band arrays; missing proj:transform/proj:epsg/proj:wkt2 with no product fallback) instead of bare StopIteration/KeyError. Consistency and single sources of truth: - Root consolidation consolidates every multiscale root the plan maps to None, dropping the converter's dependency on the duplicate `_AUX_MULTISCALES` table; the dead `REFLECTANCE_MULTISCALE` is deleted. - The CPM writer forwards spatial_chunk and compression_level to the S2 policy check, so an explicit deviation is refused loudly rather than silently ignored (enable_sharding stays unforwarded: sharding is fixed on and its CPM default of False is indistinguishable from an explicit False). - The pydantic-zarr git source pins the validated commit (9f2e847) instead of floating on main. Snapshots and docs: - The plan snapshots are regenerated from the plan, replacing the snapshot harness's synthetic-tile georeferencing with each product's own; the masking machinery and `_EXACT_PRODUCT` special case are retired, so every product's snapshot comparison is now exact. - The docs drop the removed convert-s2-optimized layout flags, scope --scale-offset-codec to the generic rechunker, document the fixed S2 write policy, and replace the deleted create_multiscale_from_datatree reference with the plan/write API. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…to feat/pydantic-zarr-scoped-validation Resolves the conflicts between this branch's plan-driven S2 pipeline and upstream's root-consolidation harmonization (EOPF-Explorer#328) and spatial-chunking fix (EOPF-Explorer#331): - convert_s2_optimized now feeds the planned dataset index into the harmonized utils.updated_root_consolidation, passing the plan's None-mapped multiscale roots as additional_consolidate_paths; the converter-local simple_root_consolidation, add_multiscale_pyramids_to_stac_metadata and write_store_root_bbox are gone, and add_s2_stac_assets (from EOPF-Explorer#328) is adapted to take the index and the planned reflectance shape. - utils.add_multiscale_pyramids_to_stac_metadata accepts a dataset's variable names in place of the dataset, which is what the plan-driven converter has. - convert_s2 and the DataTree-writer code EOPF-Explorer#328/EOPF-Explorer#331 touched stay deleted (no remaining callers); max_retries leaves convert_s2_optimized as upstream removed it from the call sites. - tests follow: the consolidation tests run against updated_root_consolidation with explicit None roots, upstream's revived test_s2_multiscale_geo_metadata (now against conversion.utils) is kept, and the DataTree-writer tests stay deleted. - uv.lock re-locked: upstream's bumps plus this branch's pinned pydantic-zarr commit; the regenerated plan snapshots are byte-identical to the merged files, so both pipelines had converged on the same output. Assisted-by: ClaudeCode:claude-fable-5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Pin pydantic-zarr to main at b0efedb, which requires zarr-metadata 0.8. zarr-metadata is now declared as a direct dependency, since the package imports it. - The GeoZarr scope is zarr-metadata's ZARR_PYTHON_CONTEXT. It claims `fixed_length_utf32` and `null_terminated_bytes` with their full definitions, which the hand-built scope only approximated, and `ZarrV3Context.update` no longer exists. s2_plan types those data types with zarr-metadata's `FixedLengthUtf32` and `NullTerminatedBytes`. - The coordinate and grid-mapping builders use `create_default`. It supplies the one-chunk grid, the `/` chunk key encoding, the derived fill value and the little-endian `bytes` codec those builders spelled out by hand. - Tests copy a spec with the validated `replace`, not `model_copy(update=...)`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Require pydantic-zarr>=0.11.0, sourced from the v0.11.0 tag until the release reaches PyPI, instead of a commit on main. - s2_plan converts a V2 dtype to its Zarr V3 form with `parse_dtype_v3(..., scope=GEOZARR_SCOPE)` instead of its own mapping. 0.11 reads numpy's fixed-length unicode and bytes types as zarr-python writes them. The local mapping turned a datetime dtype into an unclaimed name such as "datetime64[ns]"; `parse_dtype_v3` gives the `numpy.datetime64` form. - 0.11 accepts zarr-metadata's typed codec objects on `create_default`, so the builders that pass them type-check. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A carried V2 `<M8[ns]` array is planned as `numpy.datetime64` with its unit, the form zarr-python writes and the GeoZarr scope claims. The local mapping that `parse_dtype_v3` replaced gave the unclaimed name "datetime64[ns]". Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The five multiscale parent groups carried the finest level's `spatial:transform` without a `spatial:shape`, so a reader could not tell what grid the transform described. Both multiscale planners now write the finest level's shape with its transform. A test checks that every planned group with a transform also has its shape, and the snapshots are regenerated (EOPF-Explorer#355). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A `flag_meanings` list is joined into the one blank-separated string CF defines. The SCL flag table is written as a string too. - The `dtype` attribute, which repeats the array's data type, is dropped. - A `valid_min` / `valid_max` outside the range of the integer data type is dropped. The scl `uint8` array's `valid_max: 65535` was one. - A carried `short_name` is written for the level the array is in, as `<variable>_r<resolution>m`. Coarsened levels copied the source level's name, and the source styles were mixed. The first three rules live in one helper, `normalize_cf_array_attrs`, which the S2 plan and the shared `sanitize_array_attrs` both apply. `sanitize_array_attrs` now also drops `valid_*` for decoded floats, as its docstring said. Tests cover the helper's cases, the snapshots' attributes, and a planted array on native and coarsened levels. The snapshots are regenerated (EOPF-Explorer#351). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every band and carried data array got a shard, even when the whole array fit in one inner chunk. Those shards held that one chunk, so a reader made an index request for nothing. 201 such arrays appeared across the three snapshots, for example the 9 x 9 meteorology and 23 x 23 angle arrays. Such an array is now written in plain chunks, with one chunk covering it, under the same bytes + blosc codecs. Arrays that span several inner chunks keep their shards, including a shard larger than the array. A test checks every planned array, and the snapshots are regenerated (EOPF-Explorer#357). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on a2b8247. That commit made `sanitize_array_attrs(..., is_decoded_float=True)` drop `valid_min` / `valid_max`, as its docstring had claimed. But callers pass the flag for every float array: `create_uniform_encoding` and the generic converter do, for native floats whose range is in physical units. The packed-integer output passes it too, and its range is in packed units. So correct ranges were lost. The removal is reverted, and the docstring now states what the code does: the decode path that unpacks integers drops `valid_*` itself. A test pins that a float array keeps an in-range `valid_*`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g type The geometry label coordinates (`angle`, `band`, `detector`) carried their source data types: `fixed_length_utf32` and `null_terminated_bytes`. The second is in no registry, and a strict Zarr V3 reader must refuse an unknown data type. A carried unicode or bytes array is now planned as the registered `string` data type. `create_default` gives it the `vlen-utf8` codec, the empty fill and one chunk. The writer decodes bytes labels as ASCII rather than leave the conversion to zarr. A test checks that no planned array uses either unregistered type, and the snapshots are regenerated (EOPF-Explorer#356). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CAMS and ECMWF groups under `conditions/meteorology` held arrays on latitude/longitude dimensions with no CRS. They also carried the source `Conventions: "CF-1.7"`, which contradicts the output's no-CF-claim decision. - A carried group on a regular geographic grid gets `proj` and `spatial` attributes derived from its coordinate values. Such a group has 1-D `latitude` and `longitude` coordinates and no `proj:*` attributes. The attributes are `proj:code` EPSG:4326, the dimensions, the shape, and the transform and bbox. The values are grid points, so the registration is `node`, and a decreasing latitude gives a negative step. Irregular spacing is refused; spacing within a thousandth of the step, as float32 storage gives, counts as regular. - The plan stays a function of what it is given. `plan_geozarr` takes an optional `coordinate_values` lookup, and `write_geozarr` passes its source reader's. Without the lookup, these groups get no georeferencing. - The source `Conventions` attribute is no longer carried on any group. Tests cover the derived attributes for increasing, decreasing and float32 latitudes, the two refusals, and the plan with and without values. The writer test seeds regular latitude/longitude values, and the snapshots are regenerated (EOPF-Explorer#353). Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on c8c0fbd. The V2 fixtures hold only unicode labels, so nothing reached the bytes path of EOPF-Explorer#356. A plan test plants a `|S3` label array and checks that it is planned as `string` with `vlen-utf8`. A writer test fills an `S3` label array into a string array and reads back the decoded labels. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pyright refused `np.iinfo` on an unnarrowed data type in the valid-range guardrail. It also refused `zarr.storage` attribute access and an untyped array read in the bytes-label write test. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on bbd7637. - A latitude/longitude group whose values `_lat_lon_attributes` refuses (irregular spacing, NaN fill, a single point) stopped the whole conversion. Such a group is now carried without georeferencing, as before EOPF-Explorer#353, and a warning is logged. - Longitudes are unwrapped before the spacing check, so a grid crossing the antimeridian or 0/360 keeps a regular step and gets its transform. - `_lat_lon_attributes` returns a `LatLonGroupAttrs` TypedDict, like its sibling builders. - The write test asserts the derived `proj:code` and transform on the written meteorology groups, which shows the source reader feeds the plan. Plan tests cover an irregular grid, a NaN grid and an antimeridian grid. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 3ac4926. Longitudes are unwrapped for the spacing check and the transform. The bbox took the unwrapped extremes, so a grid crossing the antimeridian got an east longitude past 180. The bbox longitudes are now moved back into [-180, 180], so the crossing reads as west > east, as STAC and GeoJSON spell it. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…referencing BREAKING CHANGE: no `spatial_ref` grid-mapping arrays (with `crs_wkt` / `spatial_ref` attributes) and no `grid_mapping` attributes are written by the S2 plan, the S1 RTC ingest, the OLCI converter or the shared write path. The `proj` and `spatial` conventions are the only georeferencing. An array opened on its own has no CRS; read `proj:code` from its group. This breaks CRS resolution in rioxarray as released (0.23, 0.24 and its main branch). rioxarray recognizes the proj convention only under the name "proj:", while the proj v0.1 spec and zarr-cm 0.5 name it "proj". With the `spatial_ref` array gone, `ds.rio.crs` is None for this output. A strict xfail test records the break and fails once a rioxarray release reads the spec's name (EOPF-Explorer#352). - S2 plan: no `spatial_ref` members, no `grid_mapping` on groups or arrays. `GridMappingAttrs` and the WKT plumbing are gone, so the product-level fallback no longer requires `proj:wkt2`. The writer and converter drop their `spatial_ref` cases. - Shared path: `strip_cf_grid_mapping` removes the grid-mapping variables and every `grid_mapping` attribute and encoding. `stream_write_dataset` applies it, and drops the encoding of the variables it removes. `write_dataset_geo_metadata` writes only the convention attributes, and now takes a CRS given as a string. - OLCI: the reprojection keeps rioxarray's CRS in memory for the computations downstream. The converter strips the grid mapping before each write. - S1: the ingest no longer adds the grid mapping, and the model no longer requires `spatial_ref`. It still accepts it, so stores ingested before validate. - The generic converter (`create_geozarr_dataset`) is built around the grid mapping and is left as it is. The output-contract test names it. - Validators: `check_grid_mapping` checks only a declared grid mapping, and the minispec validator tolerates one. Both stay, so older stores still validate. - Docs, the OLCI notebook and the changelog mark the grid mapping obsolete. The S2, OLCI and S1 fixtures are regenerated; the S1 fixture also picks up attributes the ingester already wrote. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 85366f4. Moving a longitude by one turn left a value more than one turn out of range past 180. That happens when a wide 0-360 grid starts above 180 and wraps past 0. Longitudes outside [-180, 180] are now normalized modulo 360. In-range values, the antimeridian's 180 included, are kept as written. A test covers a wide 0-360 grid. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's findings on 3c881ac. - `stream_write_dataset` stripped the CF grid mapping even when it added no `proj` attributes. That is the case for the generic rechunker, so an input whose CRS lived only in the grid mapping was written with no CRS. `strip_cf_grid_mapping` now carries the CRS over as the `proj` convention attributes when the dataset declares no `proj:*` attribute. It drops a `spatial_ref` variable only when that variable is a grid mapping. - The CRS parameters of `write_dataset_geo_metadata` and `stream_write_dataset` take a string or EPSG code, matching what the code accepts. - The changelog and architecture docs say that the generic converter still writes the grid mapping, and that rioxarray does not yet read the proj convention under its spec name. Tests cover the strip helper (CRS carried over, a declared `proj:code` kept, an unrelated `spatial_ref` kept) and a stream write of a grid-mapped input. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ping's CRS Addresses roborev's findings on 2d8e58a. - The carried `proj` attributes replaced the dataset's `zarr_conventions`, dropping a declared spatial or multiscales convention. The proj entry is now appended to the declared ones, unless already declared. - The carry-over is skipped only when the dataset declares a proj convention key (`proj:code`, `proj:wkt2`, `proj:projjson`), the rule the validator uses. A legacy `proj:epsg` no longer counts. Test cases cover a legacy `proj:epsg` and a declared spatial convention. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 997b123: the docstring still described the old any-`proj:*` rule. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pydantic-zarr 0.11.0 is published, so the git-tag source override is removed and the lock resolves `pydantic-zarr>=0.11.0` from the registry. The resolved version and code are unchanged. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ixes Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…S2 changes main changed the DataTree-based S2 write path, which this branch replaces with the model-based plan. Its changes are ported to the plan rather than taken as written: - Atmosphere (EOPF-Explorer#343): `quality/atmosphere` is a multiscale pyramid, 10-720 m, in the plan's `AUX_MULTISCALE_LEVELS` as in main's `_AUX_MULTISCALES`. Only L2A products hold the group, so presence gives main's L2A-only rule. `aot` / `wvp` resample as reflectance, per main's `determine_variable_type`. - Geometry (EOPF-Explorer#347, EOPF-Explorer#346): `conditions/geometry` declares the tile CRS with node registration. Its shape, transform and bbox come from the grid's x/y values: `plan_geozarr` takes an optional `coordinate_values` lookup, and `write_geozarr` passes its source reader's. The (y, x) angle arrays reference the grid mapping. A grid no transform describes is carried without georeferencing, with a warning. Resolution levels under `conditions/mask` and `quality` whose arrays carry no proj attributes, as in native Zarr V3 products, declare the tile CRS with pixel registration from the tile origin. - Band statistics (EOPF-Explorer#333, EOPF-Explorer#344): computed by the shared root consolidation, which the plan path already calls. The STAC asset function takes either datasets or the plan's variable names, and keeps main's href-based duplicate check and key-collision warning. Conflicts in s2_multiscale.py and the old-path tests take this branch's side, since that code was removed. main's new statistics assertion is kept, and its atmosphere and geometry tests are rewritten against the plan. The snapshots are regenerated from the plan: the atmosphere and detector-footprint groups match main's in structure and attribute keys. The geometry group's value-derived attributes are tested with coordinate values, in the plan and through the writer. The lock is updated for the 0.13.1 version bump. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 47265cc. The merge left two parsers of level names (`r60m`): a regex in s2_plan and string slicing in s2_write. s2_plan's `level_meters` is now the only one. The writer's `_level_meters` and `_is_level_name` call it, and `_level_meters` raises for a non-level name. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR 330 merged EOPF-Explorer main and ported its S2 changes into the plan: the atmosphere pyramid, the geometry and detector-footprint georeferencing, and the band statistics. They are carried into this branch's EOPF-Explorer#351, EOPF-Explorer#352 and EOPF-Explorer#353 work: - s2_plan: this branch's version, with the port applied on top. The geometry grid and the footprint levels get no `spatial_ref` and no `grid_mapping` (EOPF-Explorer#352). The tile georeferencing is (origin, EPSG), with no WKT. The meteorology (EOPF-Explorer#353) and geometry grids share one regular-spacing helper (`_regular_axis`) and one refusal path, which carries a grid no transform describes without georeferencing, with a warning. `level_meters` is the one parser of level names, for the plan and the writer. - conversion/utils: `strip_cf_grid_mapping` and main's node-transform helpers both kept. `write_dataset_geo_metadata` and `stream_write_dataset` take main's `registration` and this branch's wider CRS type, and write no grid mapping. - CHANGELOG: this branch's Unreleased section above main's 0.13.x releases. - Tests: both sides kept. PR 330's geometry test now expects no grid mapping. Snapshots regenerated. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 47481e8. The `level_meters` keyword parameter of `_carried_array_attributes` and `_plan_carried_array` shadowed the module-level `level_meters` once the merge made it public. The parameter is now `meters`, as in `_with_level_short_name`. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The security check's pip-audit blocked on tornado 6.6 (CVE-2026-126347, fixed in 6.6.1), a transitive dependency through dask's distributed scheduler. Only tornado moves in the lock. Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The writer masked a coarsened integer array's planned Zarr fill value, which is always 0, out of the block mean. Arrays that declare no nodata lost their zeros from the mean: the cloud and snow probabilities, where 0 is a valid value, and the atmosphere arrays in products that declare none. main averages those whole, so 14 coarsened arrays of an L2A product, and their STAC statistics, differed from main. The source readers now report the nodata an input array declares, in main's order: the CF `_FillValue` (a Zarr V2 array's `fill_value`, xarray's encoding, or the attribute), then the EOPF `fill_value` attribute. The writer carries that value down each pyramid chain and skips it alone. An array that declares none is averaged whole. Tests: both readers over each declaration, and generated probability and atmosphere levels equal to the plain block mean on seeded values that hold zeros. Assisted-by: ClaudeCode:claude-fable-5.2 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
PR 330 now skips only the nodata the input declares when it coarsens, so probability and atmosphere arrays that declare none are averaged whole. The writer's loop takes this branch's side, without the `spatial_ref` case (EOPF-Explorer#352). The tests keep both sides, and the bytes-label test's source stub gains the readers' new `nodata` method. Assisted-by: ClaudeCode:claude-fable-5.2 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e plan unwired Restore the Sentinel-2 converter, the DataTree pyramid writer, the CLI, the CPM writer, their docs, tests and output snapshots to their state on main, and remove the plan-driven writer (`data_api/s2_write.py`) with its tests. This branch now only adds the pydantic models: the pydantic-zarr 0.11 and zarr-metadata 0.8 move, the GeoZarr scope, the Sentinel-2 input model and the output plan (`data_api/s2_plan.py`). `convert_s2_optimized` writes what it writes on main. The plan-driven writer moves to a follow-up change stacked on this one. Assisted-by: ClaudeCode:claude-fable-5.2 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Replace the plan's snapshot test with a comparison against what `convert_s2_optimized` writes. Each Sentinel-2 fixture (one L1C, two L2A) is shrunk tenfold, given data and converted; every group and array document stored below the root must equal the planned one. The root's own attributes are left out, because the converter writes them in its consolidation step. The comparison found one difference, fixed here: the plan typed every x/y array as float64, while the converter keeps the input's own coordinate arrays on native levels (int64 metres in ESA products). The plan now carries the input's x/y arrays on native levels, data type included, and keeps the float64 coordinate array for generated levels. A second parametrization covers a product whose coordinates are float64. The API reference describes the plan and what it is checked against. Assisted-by: ClaudeCode:claude-fable-5.2 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The reflectance plan's docstring said every level holds float64 x/y coordinates, and `level_meters` named a writer this branch no longer has. Assisted-by: ClaudeCode:claude-fable-5.2 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…move the DataTree writer Bring back the plan-driven Sentinel-2 writer (`data_api/s2_write.py`) that the previous change set aside, on top of the plan that change checked against the DataTree writer's output. `convert_s2_optimized` validates the input into `Sentinel2Root`, plans the output hierarchy, writes the planned documents and fills the chunk data. The DataTree pyramid writer in `s2_multiscale.py` is removed. BREAKING CHANGE: `convert_s2_optimized` has a fixed write policy. A `spatial_chunk` other than 1024, a `compression_level` other than 3, `enable_sharding=False` and `scale_offset_codec=True` raise `ValueError`. The CPM writer's default spatial chunk for the `s2-optimized` pipeline changes from 256 to 1024. The output is the same as the DataTree writer's at that policy. The test that compares the plan with the converter's stored documents passes unchanged. Converting the three fixture products (shrunk, with data) with both writers gives identical chunk data and identical documents in every node; the root's STAC assets differ in one href, where two arrays compete for the key `b10_r60m`. The output snapshots are the plan's again, regenerated with the int64 x/y coordinates that native levels now carry. Assisted-by: ClaudeCode:claude-fable-5.2 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…tions before writing Apply the review findings on the plan-driven writer: - The CPM writer checks the Sentinel-2 write policy before it prepares the target, so a refused option no longer removes an existing product. The check is `check_s2_write_policy`, which `convert_s2_optimized` also calls. - The writer refuses input values whose data type does not cast safely to the planned one. zarr casts on assignment, which stored float reflectance as 0 and 1 in the planned uint16 band without an error. - An output location that already holds a different hierarchy raises `FileExistsError` saying to remove it, in place of zarr's `ContainsGroupError`. - The quickstart and examples pages no longer pass `spatial_chunk=256`, which the policy refuses. The CPM writer's docstring, CLI help and options table describe the Sentinel-2 options as fixed. Assisted-by: ClaudeCode:claude-fable-5.2 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
PR 330 is now validation-only, and the plan-driven writer lives on `claude/s2-plan-writer`, which this branch is stacked on. Merging it brings the conformance test, the int64 x/y coordinates on native levels, and the writer's new refusals. Conflict resolution: - `s2_plan.py`: native levels carry the input's x/y arrays, with no `spatial_ref` member (EOPF-Explorer#352). - `s2_write.py`: values go through the safe-cast check. Fixed-length unicode labels pass it into the `string` arrays they are planned as (EOPF-Explorer#356). - The conformance test treats a codec with an empty `configuration` as its name-only form, which is how zarr-python stores `vlen-utf8`. - Snapshots regenerated. Assisted-by: ClaudeCode:claude-fable-5.2 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This was referenced Nov 16, 2026
feat(s2-optimization)!: drive convert_s2_optimized from the plan and remove the DataTree writer
#359
Draft
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
This fixes the findings accepted from the independent reader's review in #349: #351, #353, #355, #356 and #357 in the S2 output, and #352 across the S2, S1 RTC and OLCI converters. It is stacked on #359 (the plan-driven writer), which is stacked on #330, and merges after both.
Warning
#352 breaks CRS resolution in rioxarray as released (0.23 and 0.24, and its main branch). rioxarray recognizes the proj convention only under the name
"proj:", while the proj v0.1 spec and zarr-cm name it"proj". With thespatial_refgrid-mapping array gone,ds.rio.crsisNonefor this output. The CRS is still declared, asproj:codeon each group. A strict xfail test (test_rioxarray_resolves_the_level_crs) records the break and fails once a rioxarray release reads the spec's name.What changed, by issue
spatial:shapeon multiscale parent groups #355: the multiscale parent groups state thespatial:shapetheirspatial:transformdescribes.flag_meaningslist becomes one blank-separated string, the SCL table included.dtypeattribute is dropped.valid_min/valid_maxoutside the integer data type's range is dropped.short_nameis written per level, as<variable>_r<resolution>m, coarsened levels included.normalize_cf_array_attrs, which the sharedsanitize_array_attrsalso applies.string#356: fixed-length unicode and bytes labels (angle,band,detector) are stored as the registeredstringdata type. The writer decodes bytes as ASCII.projandspatialattributes derived from their latitude/longitude values. They usenoderegistration, signed steps, unwrapped longitudes, and a bbox within [-180, 180].Conventions: "CF-1.7".spatial_refarrays and nogrid_mappingattributes are written by the S2 plan, the S1 RTC ingest, the OLCI converter or the shared write path.strip_cf_grid_mappingremoves them before writing. When a dataset has no proj convention key, it first carries the grid mapping's CRS over asproj:code.spatial_ref, but still accepts it, so stores ingested before validate.CHANGELOG.md(Unreleased, Breaking) mark the grid mapping obsolete.Design notes and scope
plan_geozarrtakes an optionalcoordinate_valueslookup, andwrite_geozarrpasses its source reader's. Without the lookup, those groups get no georeferencing.create_geozarr_dataset, behind the CLIconvertgeneric path and the CPMgenericpipeline, is built around the GeoZarr V0 grid mapping and still writes it. Its contract test cases are marked that way.check_grid_mappingchecks only a declared grid mapping, and the minispec validator tolerates one, so older stores still validate.sanitize_array_attrskeepsvalid_*on float arrays. Its callers passis_decoded_floatfor native floats, whose ranges are in physical units, so only its docstring was corrected._FillValueandresampling_methodattributes the ingester was already writing; the checked-in copy was stale.Verification
roborev reviewed every commit, and the agent fixed each valid finding. The low findings on the writer that are still open are listed in #359. On the current head, the agent ran:
PATHcpmextraBefore the merge with the split writer branch, the agent also recounted the #349 findings in the planned output of the V2 snapshots and the native V3 fixtures: none were left, except the meteorology groups without coordinate values in the snapshot test. That recount was not repeated after the merge.
Stack
main, because the branches below it live on a fork. Until they merge, the diff shown here includes their changes. This PR's own diff is fix!: apply the accepted GeoZarr reader findings from EOPF-Explorer/data-model#349 d-v-b/data-model#10.main. It stays a draft until then.🤖 Generated with Claude Code