Skip to content

feat(s2-optimization)!: drive convert_s2_optimized from the plan and remove the DataTree writer - #359

Draft
d-v-b wants to merge 27 commits into
EOPF-Explorer:mainfrom
d-v-b:claude/s2-plan-writer
Draft

d-v-b wants to merge 27 commits into
EOPF-Explorer:mainfrom
d-v-b:claude/s2-plan-writer

Conversation

@d-v-b

@d-v-b d-v-b commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖

This PR makes convert_s2_optimized write the hierarchy that plan_geozarr describes, and removes the DataTree pyramid writer. With the default layout the output is the same as before; other layouts are refused.

Warning

Breaking changes

  • convert_s2_optimized has a fixed write policy. spatial_chunk other than 1024, compression_level other than 3, enable_sharding=False and scale_offset_codec=True raise ValueError.
  • The CPM writer's default spatial_chunk for the s2-optimized pipeline changes from 256 to 1024.
  • create_multiscale_from_datatree and the other DataTree pyramid functions in s2_optimization.s2_multiscale are removed.
  • An output location that already holds a different hierarchy raises FileExistsError. The DataTree writer skipped levels that existed and carried on.
  • Input values whose data type does not cast safely to the planned one raise ValueError. Reflectance bands are planned as uint16.
Evidence that the output is unchanged
Fixture Nodes Documents differing Arrays with differing data
S2A L2A, T32TQM 295 0 0
S2B L1C, T32TLQ 204 0 0
S2C L2A, T29TPF 295 0 0
  • Root STAC assets: one href differs in the L2A products. Two arrays compete for the asset key b10_r60m, and the two writers visit them in a different order.

  • Not covered: the comparison used products shrunk tenfold. No full-size product was compared.

What changed
  • Writer: data_api/s2_write.py writes the planned documents, then fills chunk data from a SourceReader over the input (a Zarr V2 group or an xarray DataTree). Native levels are copied and finer bands are injected. Coarser levels are block means that skip only the nodata the input declares, or window-centre samples for classifications.
  • Converter: convert_s2_optimized validates the input into Sentinel2Root, calls the writer, and keeps the root consolidation and STAC statistics steps.
  • Removed: the DataTree pyramid writer in s2_multiscale.py, about 700 lines, with its tests. tests/test_s2_multiscale.py keeps the 4 tests of what remains.
  • Snapshots: tests/_test_data/optimized_geozarr_examples now holds the plan's output for each fixture. Before, it held the DataTree writer's output for made-up tile coordinates, so most of the JSON diff is coordinate values.
  • Docs: the converter, CLI and CPM driver pages describe the fixed policy.
Review notes
  • --dask-cluster: the plan-driven writer does not use dask, so the flag has no effect on Sentinel-2 writes.
  • Fixed policy or options: the plan could take the chunk size and compression level as parameters. This PR does not, to keep the comparison with the DataTree writer at one layout.
  • Open review findings, not fixed here:
    • The existing-output refusal only sees a Zarr V3 node. A directory that holds a Zarr V2 hierarchy or stray files is written into.
    • The unsafe-cast refusal fires while filling data, after the documents are written, so a refused input leaves a metadata-only store. Refusing a non-uint16 band in the plan would fail earlier.
    • No test checks the values of int64 x/y coordinates on native levels. The agent's comparison with the DataTree writer covered them once, by hand.
    • The writer stores the plan with GroupSpec.to_zarr, which pydantic-zarr 0.12 removes. The dependency has no upper bound.
Stack

🤖 Generated with Claude Code

d-v-b and others added 27 commits October 28, 2026 12:28
…tic-zarr main

pydantic-zarr now validates every Zarr V3 array spec as a whole metadata
document, judging each extension-point name by the definition that
claims it in a scope. The GeoZarr DataArray sets zarr_scope_strict, so
a typo'd or unknown codec or data type name is refused instead of
passing unjudged, against a GeoZarr scope that extends the default with
zarr-python's provisional fixed_length_utf32 data type (band and angle
coordinates in real stores hold it, and no zarr-metadata scope claims
it yet). The pyz wrappers drop their attributes redeclaration: the
parent field now carries a validated empty-dict default, so narrowed
attribute types that refuse {} stay required without it.

pydantic-zarr comes from git main pending its next release.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bers override

The pyz ArraySpec type parameter was phantom (both wrappers subclassed
ArraySpec[Any]), so every declared attributes model on the Sentinel
array classes was decorative: Sentinel2DataArray accepted arbitrary
attributes and held plain dicts. The parameter is now passed through,
so attributes validate against the declared model and read back typed.
The GroupSpec wrappers stop binding the members mapping type into the
parent's member-item slot (the old type-var hack and both its ignores):
with pydantic-zarr's covariant parameters, parameterizing the parent at
its widest member type and narrowing the members field is a legal
override. s3_olci spells unconstrained attributes Mapping[str, object]
(object no longer satisfies the enforced bound), and the GeoZarr v3
group classes use AnyGroupSpec, now an ordinary wide type, so member
reads stay checked. pydantic-zarr is bumped to main with covariant
spec parameters (zarr-developers/pydantic-zarr#191).

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
load_sentinel2 reads a Zarr store into the validated Sentinel2Root
input model — keeping the model as the carrier instead of validating
and discarding it — and plan_geozarr is a pure function from that model
to the GeoZarr hierarchy to write, as a Zarr V3 GroupSpec tree: the
metadata plane of the conversion, independent of chunk copying. The
reflectance pyramid (native levels with the product level's finer bands
injected, coarser levels derived, sharded uint16 bands, float64
coordinates, CF grid mappings from the product's own proj:wkt2 via a
CF round-trip, spatial/proj/multiscales convention attributes) is
derived entirely from the input model's metadata, and reproduces the
converter's stored snapshot byte for byte for the product whose tile
the snapshot harness models; for the other products the snapshots carry
the harness's synthetic tile georeferencing, so the tests mask those
values and pin the plan's georeferencing to the product's own band
attributes instead. conditions and quality stay on the DataTree
converter for now.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…quality

plan_geozarr now plans the whole hierarchy. The aux multiscale groups
(l1c/l2a classification, cloud probabilities) get the same pyramid
treatment as reflectance via AUX_MULTISCALE_LEVELS — native levels
carried, coarser levels derived with attributes verbatim, nearest
resampling when the arrays carry a flag table — and every other
conditions/quality dataset is carried: georeferenced datasets (detector
footprints, quality masks, atmosphere) gain spatial attributes and a
spatial_ref grid mapping derived from their own proj attributes,
geometry and meteorology are translated as-is (string arrays to
fixed_length_utf32, coordinates uncompressed, data arrays sharded), and
the quicklook groups and scalar arrays are dropped, as the converter
drops them. The planner walks the input model's dump, since the input
model types these subtrees loosely. Runtime assert statements are
replaced with raises or restructured away.

The whole planned tree is byte-for-byte equal to the stored converter
snapshot for the product whose tile the snapshot harness models; for
the others the tests mask the harness's synthetic-tile georeferencing
and the generated aux levels' chunk shapes, which the converter derives
from incidental dask chunking (inner chunks of 915, 512 and 1830 for
the same situation across the snapshots) where the plan uses its
uniform rule.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…re encoding

Coarsening divides whatever dask chunks the source dataset happens to
have, so the stored encoding of a generated level depended on the
incidental chunking of the object it was coarsened from: the snapshots
held inner chunks of 915, 512 and 1830 for like levels. generic
multiscales now rechunks the coarsened dataset to the spatial chunk
target (min(spatial_chunk, dim) per dimension) before deriving the
encoding, matching the native levels' policy and the spec-level plan's
uniform rule. The snapshots are regenerated (the only other change is
the omission of null dimension_names, which current pydantic-zarr no
longer serializes), and the plan tests drop their generated-level
chunk masking: the whole tree is now pinned.

Assisted-by: ClaudeCode:claude-fable-5
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Picks up the unreleased extra-fields support (consolidated root documents
now validate, carried in extra_fields and refused by to_zarr), the typed
to_metadata_json / to_stored_metadata_json exports, the always-complete
GroupSpec hierarchy, and whole-document validation for Zarr V2 arrays.
Whole-repo pyright and the full non-network suite are green against it.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…data and zarr-cm

Every zarr metadata field the Sentinel-2 plan writes now carries its
published type: zarr-metadata's *Object TypedDicts for the extension
points (regular chunk grid, default chunk key encoding, and the bytes,
blosc, zstd, crc32c and sharding_indexed codecs), zarr-cm's convention
types for the zarr-conventions attributes (the CMO declarations replace
the hand-written registry dicts, and layout entries are LayoutObject
with the spatial convention's keys as typed extra items), and dedicated
TypedDicts for the fixed-shape documents the plan builds itself:
GridMappingAttrs (narrowed at runtime from pyproj's CF output via
zarr_metadata.typed_json.check), SpatialGroupAttrs,
MultiscaleGroupAttrs, and FixedLengthUtf32Object in the GeoZarr scope.
Function-scoped imports move to the top of their modules. No runtime
change: the stored snapshots are byte-identical.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds data_api/s2_write.py: write_geozarr realizes the plan with
zarr-python and numpy alone. The planned metadata documents are written
first (pydantic-zarr to_zarr, so every stored document is exactly the
plan), then the chunk data is filled in: native arrays copied from the
input store, injected bands block-resampled from the finest input level
that holds them, generated pyramid levels chained from the level before
them with the converter's arithmetic (nodata-masked rounded block mean,
window-centre subsampling for classifications, window means for the
coordinates), and grid-mapping arrays set to zero. Product-level
publication (root attributes, consolidation, STAC) stays the converter
pipeline's separate final step.

The parity test runs write_geozarr and create_multiscale_from_datatree
on the same seeded input (each product's own proj:transform drives the
coordinates; small arrays and the scene classifications carry patterned
values) and compares the stores file by file: identical file sets,
identical metadata documents, and byte-identical chunks, for all three
fixture products.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Converts the pyz GroupSpec/ArraySpec wrappers from Generic[...] bases
with module-level TypeVars to PEP 695 type parameters (ruff UP046), and
drops an unused import this branch had left in geozarr/v3.py.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…move the DataTree writer

convert_s2_optimized now validates the input store into Sentinel2Root,
plans the output with plan_geozarr, and writes it with write_geozarr —
metadata documents first, chunks after, root consolidation and STAC
unchanged as the final step, fed by a dataset index derived from the
plan. The DataTree-based write path (create_multiscale_from_datatree,
generic_multiscales, stream_write_s2dataset, inject_missing_bands, the
coarsening kernels and write_geo_metadata) is removed, after the
plan-driven writer reproduced its stores byte for byte; s2_multiscale
keeps only the declarative policy tables and the SCL vocabulary.

The write policy is the plan's fixed policy (sharding on, 1024-pixel
inner chunks, blosc-zstd level 3, ESA packed-integer layout): deviating
option values are refused with a message naming them, the layout flags
are gone from the convert-s2-optimized CLI command and ignored on the
auto-detect convert path, and the CPM writer forwards only the options
the S2 pipeline still interprets.

The migration also widens the inputs the plan accepts, which the
DataTree path tolerated and the Zarr V3 (N0512) fixtures exercise: a
tree with no backing store or a non-V2 backing store is materialized to
a scratch V2 store (minus its per-variable zarr storage encoding),
conditions/meteorology is optional in the input model, georeferencing
falls back to the product-level stac_discovery proj declaration when
the band and mask arrays carry none, and numpy S dtypes map to
zarr-python's null_terminated_bytes, now claimed by the GeoZarr scope.

Tests: the writer tests pin plan realization and the resampling
semantics against independently formulated references (the byte parity
against the DataTree writer was proven before its removal); snapshot
regeneration moves to the plan tests; the multiscale tests keep the
encoding and end-to-end structural coverage; the geo-metadata unit
tests go with the code they tested.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…and drop the scratch copy

The converter no longer materializes in-memory or Zarr V3-backed trees to
a scratch Zarr V2 store. New `pyz.datatree` is the DataTree analogue of
`GroupSpec.from_zarr`: `group_spec_from_datatree` extracts a tree's
metadata plane as the Zarr V2 documents xarray's own zarr writer would
store (every variable through `encode_zarr_variable`, CF packing applied,
storage-layout encoding keys dropped), so `Sentinel2Root` validates a
tree's named members and typed attributes exactly as it validates a store.
`load_sentinel2` accepts a DataTree, and `write_geozarr` reads the input
through a `SourceReader` protocol with a zarr-store and a DataTree
implementation, the latter reading values in the same encoded form.

A parity test pins the equivalence: on the seeded Sentinel-2 fixtures,
the tree source writes the same store as the zarr-store source, chunk for
chunk, whether the tree holds the raw packed values or the decoded ones
(which the reader re-encodes as xarray would have on a store write).

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Correctness and robustness:
- `_window_mean` keeps an all-NaN float window NaN instead of writing 0.0,
  matching the removed converter's skipna mean and `_coarsen`'s docstring.
- `_plan_subtree` carries a mixed group's own arrays alongside its planned
  subgroups instead of dropping them silently.
- `plan_reflectance` raises its deliberate ValueErrors (no band arrays;
  missing proj:transform/proj:epsg/proj:wkt2 with no product fallback)
  instead of bare StopIteration/KeyError.

Consistency and single sources of truth:
- Root consolidation consolidates every multiscale root the plan maps to
  None, dropping the converter's dependency on the duplicate
  `_AUX_MULTISCALES` table; the dead `REFLECTANCE_MULTISCALE` is deleted.
- The CPM writer forwards spatial_chunk and compression_level to the S2
  policy check, so an explicit deviation is refused loudly rather than
  silently ignored (enable_sharding stays unforwarded: sharding is fixed on
  and its CPM default of False is indistinguishable from an explicit False).
- The pydantic-zarr git source pins the validated commit (9f2e847) instead
  of floating on main.

Snapshots and docs:
- The plan snapshots are regenerated from the plan, replacing the snapshot
  harness's synthetic-tile georeferencing with each product's own; the
  masking machinery and `_EXACT_PRODUCT` special case are retired, so every
  product's snapshot comparison is now exact.
- The docs drop the removed convert-s2-optimized layout flags, scope
  --scale-offset-codec to the generic rechunker, document the fixed S2
  write policy, and replace the deleted create_multiscale_from_datatree
  reference with the plan/write API.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…to feat/pydantic-zarr-scoped-validation

Resolves the conflicts between this branch's plan-driven S2 pipeline and
upstream's root-consolidation harmonization (EOPF-Explorer#328) and spatial-chunking fix
(EOPF-Explorer#331):

- convert_s2_optimized now feeds the planned dataset index into the
  harmonized utils.updated_root_consolidation, passing the plan's None-mapped
  multiscale roots as additional_consolidate_paths; the converter-local
  simple_root_consolidation, add_multiscale_pyramids_to_stac_metadata and
  write_store_root_bbox are gone, and add_s2_stac_assets (from EOPF-Explorer#328) is
  adapted to take the index and the planned reflectance shape.
- utils.add_multiscale_pyramids_to_stac_metadata accepts a dataset's variable
  names in place of the dataset, which is what the plan-driven converter has.
- convert_s2 and the DataTree-writer code EOPF-Explorer#328/EOPF-Explorer#331 touched stay deleted
  (no remaining callers); max_retries leaves convert_s2_optimized as upstream
  removed it from the call sites.
- tests follow: the consolidation tests run against updated_root_consolidation
  with explicit None roots, upstream's revived test_s2_multiscale_geo_metadata
  (now against conversion.utils) is kept, and the DataTree-writer tests stay
  deleted.
- uv.lock re-locked: upstream's bumps plus this branch's pinned pydantic-zarr
  commit; the regenerated plan snapshots are byte-identical to the merged
  files, so both pipelines had converged on the same output.

Assisted-by: ClaudeCode:claude-fable-5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Pin pydantic-zarr to main at b0efedb, which requires zarr-metadata
  0.8. zarr-metadata is now declared as a direct dependency, since the
  package imports it.
- The GeoZarr scope is zarr-metadata's ZARR_PYTHON_CONTEXT. It claims
  `fixed_length_utf32` and `null_terminated_bytes` with their full
  definitions, which the hand-built scope only approximated, and
  `ZarrV3Context.update` no longer exists. s2_plan types those data
  types with zarr-metadata's `FixedLengthUtf32` and
  `NullTerminatedBytes`.
- The coordinate and grid-mapping builders use `create_default`. It
  supplies the one-chunk grid, the `/` chunk key encoding, the derived
  fill value and the little-endian `bytes` codec those builders spelled
  out by hand.
- Tests copy a spec with the validated `replace`, not
  `model_copy(update=...)`.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Require pydantic-zarr>=0.11.0, sourced from the v0.11.0 tag until the
  release reaches PyPI, instead of a commit on main.
- s2_plan converts a V2 dtype to its Zarr V3 form with
  `parse_dtype_v3(..., scope=GEOZARR_SCOPE)` instead of its own mapping.
  0.11 reads numpy's fixed-length unicode and bytes types as zarr-python
  writes them. The local mapping turned a datetime dtype into an
  unclaimed name such as "datetime64[ns]"; `parse_dtype_v3` gives the
  `numpy.datetime64` form.
- 0.11 accepts zarr-metadata's typed codec objects on `create_default`,
  so the builders that pass them type-check.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A carried V2 `<M8[ns]` array is planned as `numpy.datetime64` with its
unit, the form zarr-python writes and the GeoZarr scope claims. The local
mapping that `parse_dtype_v3` replaced gave the unclaimed name
"datetime64[ns]".

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pydantic-zarr 0.11.0 is published, so the git-tag source override is
removed and the lock resolves `pydantic-zarr>=0.11.0` from the registry.
The resolved version and code are unchanged.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…S2 changes

main changed the DataTree-based S2 write path, which this branch
replaces with the model-based plan. Its changes are ported to the plan
rather than taken as written:

- Atmosphere (EOPF-Explorer#343): `quality/atmosphere` is a multiscale pyramid,
  10-720 m, in the plan's `AUX_MULTISCALE_LEVELS` as in main's
  `_AUX_MULTISCALES`. Only L2A products hold the group, so presence
  gives main's L2A-only rule. `aot` / `wvp` resample as reflectance, per
  main's `determine_variable_type`.
- Geometry (EOPF-Explorer#347, EOPF-Explorer#346): `conditions/geometry` declares the tile CRS with
  node registration. Its shape, transform and bbox come from the grid's
  x/y values: `plan_geozarr` takes an optional `coordinate_values`
  lookup, and `write_geozarr` passes its source reader's. The
  (y, x) angle arrays reference the grid mapping. A grid no transform
  describes is carried without georeferencing, with a warning.
  Resolution levels under `conditions/mask` and `quality` whose arrays
  carry no proj attributes, as in native Zarr V3 products, declare the
  tile CRS with pixel registration from the tile origin.
- Band statistics (EOPF-Explorer#333, EOPF-Explorer#344): computed by the shared root
  consolidation, which the plan path already calls. The STAC asset
  function takes either datasets or the plan's variable names, and keeps
  main's href-based duplicate check and key-collision warning.

Conflicts in s2_multiscale.py and the old-path tests take this branch's
side, since that code was removed. main's new statistics assertion is
kept, and its atmosphere and geometry tests are rewritten against the
plan. The snapshots are regenerated from the plan: the atmosphere and
detector-footprint groups match main's in structure and attribute keys.
The geometry group's value-derived attributes are tested with
coordinate values, in the plan and through the writer. The lock is
updated for the 0.13.1 version bump.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Addresses roborev's finding on 47265cc. The merge left two parsers of
level names (`r60m`): a regex in s2_plan and string slicing in s2_write.
s2_plan's `level_meters` is now the only one. The writer's
`_level_meters` and `_is_level_name` call it, and `_level_meters`
raises for a non-level name.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The security check's pip-audit blocked on tornado 6.6
(CVE-2026-126347, fixed in 6.6.1), a transitive dependency through
dask's distributed scheduler. Only tornado moves in the lock.

Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The writer masked a coarsened integer array's planned Zarr fill value,
which is always 0, out of the block mean. Arrays that declare no nodata
lost their zeros from the mean: the cloud and snow probabilities, where
0 is a valid value, and the atmosphere arrays in products that declare
none. main averages those whole, so 14 coarsened arrays of an L2A
product, and their STAC statistics, differed from main.

The source readers now report the nodata an input array declares, in
main's order: the CF `_FillValue` (a Zarr V2 array's `fill_value`,
xarray's encoding, or the attribute), then the EOPF `fill_value`
attribute. The writer carries that value down each pyramid chain and
skips it alone. An array that declares none is averaged whole.

Tests: both readers over each declaration, and generated probability
and atmosphere levels equal to the plain block mean on seeded values
that hold zeros.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e plan unwired

Restore the Sentinel-2 converter, the DataTree pyramid writer, the CLI, the
CPM writer, their docs, tests and output snapshots to their state on main,
and remove the plan-driven writer (`data_api/s2_write.py`) with its tests.

This branch now only adds the pydantic models: the pydantic-zarr 0.11 and
zarr-metadata 0.8 move, the GeoZarr scope, the Sentinel-2 input model and
the output plan (`data_api/s2_plan.py`). `convert_s2_optimized` writes what
it writes on main. The plan-driven writer moves to a follow-up change
stacked on this one.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Replace the plan's snapshot test with a comparison against what
`convert_s2_optimized` writes. Each Sentinel-2 fixture (one L1C, two L2A)
is shrunk tenfold, given data and converted; every group and array document
stored below the root must equal the planned one. The root's own attributes
are left out, because the converter writes them in its consolidation step.

The comparison found one difference, fixed here: the plan typed every x/y
array as float64, while the converter keeps the input's own coordinate
arrays on native levels (int64 metres in ESA products). The plan now carries
the input's x/y arrays on native levels, data type included, and keeps the
float64 coordinate array for generated levels. A second parametrization
covers a product whose coordinates are float64.

The API reference describes the plan and what it is checked against.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The reflectance plan's docstring said every level holds float64 x/y
coordinates, and `level_meters` named a writer this branch no longer has.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…move the DataTree writer

Bring back the plan-driven Sentinel-2 writer (`data_api/s2_write.py`) that
the previous change set aside, on top of the plan that change checked
against the DataTree writer's output. `convert_s2_optimized` validates the
input into `Sentinel2Root`, plans the output hierarchy, writes the planned
documents and fills the chunk data. The DataTree pyramid writer in
`s2_multiscale.py` is removed.

BREAKING CHANGE: `convert_s2_optimized` has a fixed write policy. A
`spatial_chunk` other than 1024, a `compression_level` other than 3,
`enable_sharding=False` and `scale_offset_codec=True` raise `ValueError`.
The CPM writer's default spatial chunk for the `s2-optimized` pipeline
changes from 256 to 1024.

The output is the same as the DataTree writer's at that policy. The test
that compares the plan with the converter's stored documents passes
unchanged. Converting the three fixture products (shrunk, with data) with
both writers gives identical chunk data and identical documents in every
node; the root's STAC assets differ in one href, where two arrays compete
for the key `b10_r60m`.

The output snapshots are the plan's again, regenerated with the int64 x/y
coordinates that native levels now carry.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…tions before writing

Apply the review findings on the plan-driven writer:

- The CPM writer checks the Sentinel-2 write policy before it prepares the
  target, so a refused option no longer removes an existing product. The
  check is `check_s2_write_policy`, which `convert_s2_optimized` also calls.
- The writer refuses input values whose data type does not cast safely to
  the planned one. zarr casts on assignment, which stored float reflectance
  as 0 and 1 in the planned uint16 band without an error.
- An output location that already holds a different hierarchy raises
  `FileExistsError` saying to remove it, in place of zarr's
  `ContainsGroupError`.
- The quickstart and examples pages no longer pass `spatial_chunk=256`,
  which the policy refuses. The CPM writer's docstring, CLI help and options
  table describe the Sentinel-2 options as fixed.

Assisted-by: ClaudeCode:claude-fable-5.2
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant