Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions acceptance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Slice 1 of the plan. Benchmarks and the OpenLayers browser runner come in later
```bash
cd acceptance
uv sync # core
uv sync --extra reader # + titiler-eopf 0.12.2's own reader, for TR01 and the integration tests
uv sync --extra reader # + titiler-eopf 0.12.4's own reader, for TR01 and the integration tests
```

## Use
Expand All @@ -31,7 +31,7 @@ uv run eopf-accept run --collection sentinel-3-olci-l1-efr-staging --stage scrat
uv run eopf-accept run …same as plan…

# A local or ad-hoc endpoint: NAME=URL@REPORTED_VERSION:API (API 0.11 or 0.12)
uv run eopf-accept run … --endpoint local=http://127.0.0.1:8000@0.12.2:0.12
uv run eopf-accept run … --endpoint local=http://127.0.0.1:8000@0.12.4:0.12
```

Reports go to `~/DevDS/EOPF/acceptance_runs/<UTC>_<collection>_<stage>_<label>/` (`report.md`,
Expand Down Expand Up @@ -101,7 +101,7 @@ Numbers are IDs, not an order; gaps are unused IDs.
| ST09 | the finest level has data where the coarsest level does | empty conversions |
| HT02 | each opened group, rooted where titiler roots it, opens with listing forbidden | PROPFIND 405 over HTTP |
| TR01 | titiler-eopf 0.12's own `GeoZarrReader`: open, zooms, tiles (scratch only) | anything the server would 500 on |
| TI00–TI09 | per endpoint: version + route fingerprint, /info, tilejson without zoom params, decoded tiles scaled by footprint coverage, RGB channels, cold cache, viewer, URL-form contract, fit-bounds vs minzoom | C1, C8, C13, C14 |
| TI00–TI10 | per endpoint: version + route fingerprint, /info, tilejson without zoom params, decoded tiles per store level scaled by footprint coverage (a flat tile compared with others in the footprint), RGB channels, cold cache, viewer, URL-form contract, fit-bounds vs minzoom, and where a config names a second stretch, the same levels under it (WARN if flat) | C1, C8, C13, C14 |
| RG04/05/07/08 | the item's links render (xyz as PNG, the thumbnail as any image type the suite can decode, issue #8); do they also work on 0.12 (flip readiness); item newer than its store; asset hrefs point exactly at this store's groups | broken links, the /raster flip, stale registrations |
| GR02 | `generic_rechunker` layout: each dimension chunked to `min(spatial_chunk, size)`, one shard per array (FAIL); 2-D coordinates left uncompressed, and arrays written like coordinates that nothing declares as one (WARN) | rechunking or sharding that drifts from data-model#292's rule |
| GR03 | CF packing: CF `scale_factor`/`add_offset` and the `scale_offset` codec never both; `_FillValue` fits the dtype (FAIL); a valid range the dtype can't hold (WARN, named first in the summary, issue #8); zarr `fill_value` ≠ CF `_FillValue` (WARN) | double decoding; nodata that readers mask differently |
Expand Down
4 changes: 2 additions & 2 deletions acceptance/configs/sentinel-1-grd-rtc.toml
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ rescale = ["0,0.4", "0,0.1"]

[endpoints]
# `api` drives the URL syntax; `expect_version` is what /api reports. /rstaging runs
# titiler-eopf v0.12.2 (73b01a5) since 8 Oct.
rstaging = { base = "https://api.explorer.eopf.copernicus.eu/rstaging", api = "0.12", expect_version = "0.12.2" }
# titiler-eopf v0.12.4 (d7f86e8), read on 3 Nov.
rstaging = { base = "https://api.explorer.eopf.copernicus.eu/rstaging", api = "0.12", expect_version = "0.12.4" }
raster = { base = "https://api.explorer.eopf.copernicus.eu/raster", api = "0.11", expect_version = "0.11.1" }

# Baseline before data-model#292 (chosen 6 Oct): the Venice lagoon tile, 11 slices 23 Jun – 13 Jul
Expand Down
4 changes: 2 additions & 2 deletions acceptance/configs/sentinel-2-l1c.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ rescale = ["0,1", "0,1", "0,1"] # TOA reflectance over cloud or snow exceeds 0

[endpoints]
# `api` drives the URL syntax; `expect_version` is what /api reports. /rstaging runs
# titiler-eopf v0.12.2 (73b01a5) since 8 Oct.
rstaging = { base = "https://api.explorer.eopf.copernicus.eu/rstaging", api = "0.12", expect_version = "0.12.2" }
# titiler-eopf v0.12.4 (d7f86e8), read on 3 Nov.
rstaging = { base = "https://api.explorer.eopf.copernicus.eu/rstaging", api = "0.12", expect_version = "0.12.4" }
raster = { base = "https://api.explorer.eopf.copernicus.eu/raster", api = "0.11", expect_version = "0.11.1" }

# Verified on /rstaging for an L2A item, S2A_MSIL2A_20260928T140901_N0513_R053_T26WNE_20260928T203014 (29 Sep); not yet on L1C.
Expand Down
10 changes: 6 additions & 4 deletions acceptance/configs/sentinel-2-l2a.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ sample_variables = ["b04", "b03", "b02"]
dtype_allow = ["uint16", "float32"]
dtype_check_pattern = "^b(0[1-9]|1[0-2]|8a)$" # b01–b12 and b8a
min_valid = 0.25
min_distinct = 16
min_distinct = 5 # the bar 16 had at 0,0.3: 5 PNG levels at 0,1 span 0.0196 of reflectance, 16 at 0,0.3 span 0.0188. At 16 a healthy ocean scene fails (T27SVB, 2 Nov)

stac = "https://api.explorer.eopf.copernicus.eu/stac"
flip = { from = "/raster/", to = "/rstaging/" } # RG05: does each item link also work on 0.12?
Expand All @@ -25,14 +25,16 @@ upstream = "10.11.0" # first release with the largest-divisor tile si

[render]
variables = ["b04", "b03", "b02"]
rescale = ["0,0.3", "0,0.3", "0,0.3"]
rescale = ["0,1", "0,1", "0,1"] # what the items link (data-pipeline scripts/register_v1.py); under 0,0.3 a cloud or snow scene renders as one colour (T25WFP, 2 Nov)
# TI10: the Explorer's true colour for this collection (landing-page sentinel-explorer-config.js). 10 PNG levels at 0,0.5 span the same 0.0196.
second_stretch = { name = "the Explorer's true colour", rescale = "0,0.5", min_distinct = 10, api = "0.12" }
"0.12" = { route = "item", asset = "reflectance" }
"0.11" = { group = "/measurements/reflectance" }

[endpoints]
# `api` drives the URL syntax; `expect_version` is what /api reports. /rstaging runs
# titiler-eopf v0.12.2 (73b01a5) since 8 Oct.
rstaging = { base = "https://api.explorer.eopf.copernicus.eu/rstaging", api = "0.12", expect_version = "0.12.2" }
# titiler-eopf v0.12.4 (d7f86e8), read on 3 Nov.
rstaging = { base = "https://api.explorer.eopf.copernicus.eu/rstaging", api = "0.12", expect_version = "0.12.4" }
raster = { base = "https://api.explorer.eopf.copernicus.eu/raster", api = "0.11", expect_version = "0.11.1" }

# Verified on /rstaging for S2A_MSIL2A_20260928T140901_N0513_R053_T26WNE_20260928T203014 (29 Sep).
Expand Down
8 changes: 4 additions & 4 deletions acceptance/configs/sentinel-3-olci-l1-efr.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ rescale = ["10,300", "22,345", "36,390"]

[endpoints]
# `api` drives the URL syntax; `expect_version` is what /api reports. /rstaging runs
# titiler-eopf v0.12.2 (73b01a5) since 8 Oct.
rstaging = { base = "https://api.explorer.eopf.copernicus.eu/rstaging", api = "0.12", expect_version = "0.12.2" }
# titiler-eopf v0.12.4 (d7f86e8), read on 3 Nov.
rstaging = { base = "https://api.explorer.eopf.copernicus.eu/rstaging", api = "0.12", expect_version = "0.12.4" }
raster = { base = "https://api.explorer.eopf.copernicus.eu/raster", api = "0.11", expect_version = "0.11.1" }

# `zooms` is an optional anchor: TI02 compares the tilejson with the range the store's own
Expand Down Expand Up @@ -99,7 +99,7 @@ until = "2026-12-31"
# /raster TI02: the minzoom = maxzoom collapse is the link form, not titiler 0.11.1.
# register_v1 (v1.15.0-s3olci-rc8, _S3_OLCI_BASE_LEVEL = "r0") links the level group
# /measurements/r0, which has no multiscales, so titiler returns that level's zoom for both
# (reader.py get_minzoom, same in 0.11.1, 5fbea81 and v0.12.2); /rstaging passes. The first known issue
# (reader.py get_minzoom, same in 0.11.1, 5fbea81 and v0.12.4); /rstaging passes. The first known issue
# (6 Oct) blamed titiler and was retired on 7 Oct. This one names the real cause and won't be
# fixed on 0.11: the titiler-eopf >0.12 flip replaces these links (Loïc, 7 Oct, option b; RG05
# lists the 0.11 links that 422 on /rstaging). It matches only that mechanism, on /raster only.
Expand All @@ -110,4 +110,4 @@ match = "the render reads level group measurements/r0, not the multiscales group
ref = "register_v1 rc8 links /measurements/r0 on the 0.11 API; superseded by the titiler-eopf >0.12 flip"
# A review date, not the flip date (unset on 8 Oct: confirmed when everything is ready to
# flip). After it this is a FAIL again: extend it if the flip isn't ready, drop it once flipped.
until = "2026-10-31"
until = "2026-11-30"
72 changes: 39 additions & 33 deletions acceptance/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,19 @@ requires-python = ">=3.12,<3.14" # titiler-eopf (the reader extra) caps at <3.1
dependencies = [
"geozarr-toolkit==0.1.2",
"httpx[http2]>=0.27",
"morecantile==7.1.0", # with rasterio: the TI02 zoom oracle uses titiler's own zoom arithmetic
"morecantile==7.0.3", # with rasterio: the TI02 zoom oracle uses titiler's own zoom arithmetic
"numpy>=1.26",
"obstore==0.12.1",
"obstore==0.12.0",
"pillow>=10",
"pydantic>=2.12", # ST12 catches its ValidationError (geozarr-toolkit's models)
"rasterio==1.5.2",
"rasterio==1.5.0",
"zarr[cast-value-rs]==3.2.0",
]

[project.optional-dependencies]
# The local-reader check (TR01) runs titiler-eopf's own GeoZarrReader at v0.12.2 (73b01a5), the
# The local-reader check (TR01) runs titiler-eopf's own GeoZarrReader at v0.12.4 (d7f86e8), the
# release /rstaging runs. It imports redis at load even with the cache off.
reader = ["titiler-eopf @ git+https://github.com/EOPF-Explorer/titiler-eopf@73b01a5a53896958da7bd70a9565cb0358a2dc2a", "redis", "xarray==2026.9.0"]
reader = ["titiler-eopf @ git+https://github.com/EOPF-Explorer/titiler-eopf@d7f86e88222d2fe3c1f09292ae105616dd057819", "redis", "xarray==2026.4.0"]

[project.scripts]
eopf-accept = "eopf_accept.cli:main"
Expand All @@ -44,8 +44,8 @@ testpaths = ["tests"]
[tool.uv]
# Every package the suite shares with the /rstaging image resolves to the image's version, so the
# store checks, the zoom oracle, TR01 and the real-app tests run the code /rstaging runs. This is the
# uv.lock of titiler-eopf 73b01a5a53896958da7bd70a9565cb0358a2dc2a (the reader pin), which the image
# installs with `uv sync --frozen` on Python 3.12 (.python-version). When the image moves, regenerate it
# uv.lock of titiler-eopf d7f86e88222d2fe3c1f09292ae105616dd057819 (the reader pin), which the image
# installs with `uv sync --locked` on Python 3.12 (.python-version). When the image moves, regenerate it
# in a titiler-eopf checkout at the new commit:
# uv export --frozen --no-hashes --no-emit-project --no-dev --extra server --extra cache --extra openeo --no-header --no-annotate
constraint-dependencies = [
Expand All @@ -56,7 +56,7 @@ constraint-dependencies = [
"attrs==26.1.0",
"cachetools==7.0.6",
"cast-value-rs==0.4.0",
"certifi==2026.10.23", # titiler-eopf v0.12.4's version (security fix), ahead of the rest of this block
"certifi==2026.10.23",
"cffi==2.0.0 ; platform_python_implementation != 'PyPy'",
"charset-normalizer==3.4.7",
"click==8.3.3",
Expand All @@ -65,88 +65,94 @@ constraint-dependencies = [
"color-operations==0.2.0",
"colorama==0.4.6 ; sys_platform == 'win32'",
"cramjam==2.11.0",
"cryptography==50.0.0",
"cryptography==50.0.2",
"dask==2026.3.0",
"defusedxml==0.7.1",
"deprecated==1.3.1",
"donfig==0.8.1.post1",
"duckdb==1.5.2",
"duckdb==1.5.6",
"fastapi==0.141.1",
"fsspec==2026.4.0",
"fsspec==2026.6.0",
"geojson-pydantic==1.2.0",
"geopandas==1.2.0",
"google-crc32c==1.8.0",
"greenlet==3.5.0 ; platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64'",
"gunicorn==26.1.0",
"gunicorn==26.2.0",
"h11==0.16.0",
"httpcore==1.0.9",
"httpcore2==2.3.0",
"httpcore2==2.12.0 ; sys_platform != 'emscripten'",
"httptools==0.7.1",
"httpx==0.28.1",
"httpx2==2.3.0",
"idna==3.15",
"httpx2==2.12.0",
"httpx2-jsfetch==1.0 ; sys_platform == 'emscripten'",
"idna==3.20",
"isochron==0.1.0",
"jinja2==3.1.6",
"jsonpath-ng==1.7.0",
"locket==1.0.0",
"markupsafe==3.0.3",
"morecantile==7.1.0",
"morecantile==7.0.3",
"networkx==3.6.1",
"numcodecs==0.16.5",
"numexpr==2.14.1",
"numpy==2.4.4",
"obstore==0.12.1",
"openeo==0.39.1",
"obstore==0.12.0",
"openeo==0.53.0",
"openeo-pg-parser-networkx==2026.3.3",
"orjson==3.11.8",
"orjson==3.13.0",
"oschmod==0.3.12 ; sys_platform == 'win32'",
"packaging==26.2",
"pandas==3.0.2",
"pandas==2.3.3",
"partd==1.4.2",
"pendulum==3.2.0",
"ply==3.11",
"psycopg2-binary==2.9.12",
"psycopg==3.3.6",
"psycopg-binary==3.3.6 ; implementation_name != 'pypy'",
"psycopg2-binary==2.9.13",
"pycparser==3.0 ; implementation_name != 'PyPy' and platform_python_implementation != 'PyPy'",
"pydantic==2.13.3",
"pydantic-core==2.46.3",
"pydantic-settings==2.14.2",
"pydantic-settings==2.16.0",
"pyogrio==0.13.0",
"pyparsing==3.3.2",
"pyproj==3.7.2",
"pystac==1.14.3",
"pystac-client==0.9.0",
"python-dateutil==2.9.0.post0",
"python-dotenv==1.2.2",
"pytz==2026.5",
"pywin32==312 ; sys_platform == 'win32'",
"pyyaml==6.0.3",
"rasterio==1.5.2",
"redis==7.4.0",
"rasterio==1.5.0",
"redis==8.1.0",
"requests==2.33.1",
"rio-tiler==9.1.0",
"rioxarray==0.22.0",
"rio-tiler==9.4.12",
"rioxarray==0.23.0",
"shapely==2.1.2",
"simplejson==4.1.1",
"six==1.17.0",
"sqlalchemy==2.0.49",
"sqlalchemy==2.1.5",
"starlette==1.6.0",
"starlette-cramjam==0.5.1",
"starlette-cramjam==0.7.0",
"titiler-core==2.2.1",
"titiler-extensions==2.2.1",
"titiler-mosaic==2.2.1",
"titiler-openeo==0.18.0",
"titiler-openeo==0.19.0",
"titiler-stacapi==2.1.1",
"titiler-xarray==2.2.1",
"toolz==1.1.0",
"traitlets==5.14.3",
"truststore==0.10.4",
"truststore==0.10.4 ; sys_platform != 'emscripten'",
"typing-extensions==4.16.0",
"typing-inspection==0.4.2",
"tzdata==2026.2",
"urllib3==2.7.0",
"urllib3==2.8.0",
"uvicorn==0.46.0",
"uvicorn-worker==0.4.0",
"uvloop==0.22.1 ; platform_python_implementation != 'PyPy' and sys_platform != 'cygwin' and sys_platform != 'win32'",
"watchfiles==1.1.1",
"websockets==16.0",
"wrapt==2.4.0",
"xarray==2026.9.0",
"xarray==2026.4.0",
"yprov4wfs==0.0.8",
"zarr==3.2.0",
]
Expand Down
2 changes: 1 addition & 1 deletion acceptance/src/eopf_accept/conventions.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def declaration_problems(attrs: dict) -> list[str]:


def titiler_visible(group_attrs: dict, array_attrs: list[dict]) -> bool:
"""titiler-eopf 0.12 `GeoZarrReader._get_groups` for one group (reader.py, v0.12.0 to v0.12.2).
"""titiler-eopf 0.12 `GeoZarrReader._get_groups` for one group (reader.py, v0.12.0 to v0.12.4).

A group with `zarr_conventions` is visible only if it declares both spatial and proj.
A group without them is visible only if one of its arrays declares both.
Expand Down
16 changes: 15 additions & 1 deletion acceptance/src/eopf_accept/geometry.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,10 @@ def _rings(geometry: dict) -> list[list[list[float]]]:
raise ValueError(f"unsupported geometry type {geometry['type']}")


def _largest_ring(geometry: dict) -> list[list[float]]:
return max(_rings(geometry), key=lambda r: abs(_area(r)))


def bbox_polygon(bounds) -> dict:
w, s, e, n = bounds
return {"type": "Polygon", "coordinates": [[[w, s], [e, s], [e, n], [w, n], [w, s]]]}
Expand All @@ -34,7 +38,7 @@ def contains(geometry: dict, lon: float, lat: float) -> bool:

def interior_point(geometry: dict) -> tuple[float, float]:
"""A point inside the largest ring: its centroid if inside, else the ring point nearest it."""
ring = max(_rings(geometry), key=lambda r: abs(_area(r)))
ring = _largest_ring(geometry)
cx, cy = _centroid(ring)
if contains(geometry, cx, cy):
return cx, cy
Expand All @@ -54,6 +58,16 @@ def interior_point(geometry: dict) -> tuple[float, float]:
return best[1], best[2]


SPREAD = ((1, 1), (3, 3), (1, 3), (3, 1)) # quarters of a bbox, the two farthest apart first


def spread_points(geometry: dict) -> list[tuple[float, float]]:
"""The SPREAD points of the largest ring's bbox that fall inside the geometry."""
xs, ys = zip(*_largest_ring(geometry))
points = [(min(xs) + (max(xs) - min(xs)) * i / 4, min(ys) + (max(ys) - min(ys)) * j / 4) for i, j in SPREAD]
return [p for p in points if contains(geometry, *p)]


def _area(ring) -> float:
return sum(x1 * y2 - x2 * y1 for (x1, y1), (x2, y2) in zip(ring, ring[1:])) / 2

Expand Down
11 changes: 6 additions & 5 deletions acceptance/src/eopf_accept/registration.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from .budget import Http
from .model import FAIL, PASS, SKIP, WARN, Result
from .storeio import StoreReader
from .titiler_checks import lonlat_to_tile, png_stats, resolve_group, urls
from .titiler_checks import decoded_stats, lonlat_to_tile, resolve_group, urls

LINK_RELS = ("viewer", "tilejson", "xyz")
REQUEST_BOUND = 13 # item + 3 links + thumbnail, each once more for RG05, RG07's source item, plus slack
Expand Down Expand Up @@ -126,6 +126,8 @@ def rg07_fresh(item: dict, reader: StoreReader, http: Http) -> Result:
item_t = _parse_time(props.get("updated") or props.get("created"))
meta = reader.head("zarr.json")
store_t = meta.get("last_modified") if meta else None
if store_t and store_t.timestamp() == 0: # no Last-Modified header: obstore reports the epoch
store_t = None
if not item_t or not store_t:
return Result("RG07", "registration", SKIP, f"no item updated/created ({item_t}) or store Last-Modified ({store_t})")
lag = (item_t - store_t).total_seconds()
Expand Down Expand Up @@ -187,10 +189,9 @@ def _check_link(http: Http, rel: str, href: str, ztile, expected=None, footprint
return 0, f"200 but {ctype!r}, not a PNG"
if rel == "thumbnail" and not ctype.startswith("image/"):
return 0, f"200 but {ctype!r}, not an image"
try:
st = png_stats(r.content)
except (OSError, SyntaxError) as exc: # uncaught, this would stop the run and lose the titiler battery
return 0, f"200 but {ctype!r} that can't be decoded ({type(exc).__name__})"
st = decoded_stats(r.content)
if st is None:
return 0, f"200 but {ctype!r} that can't be decoded"
if st["valid"] == 0:
return 0, "200 but no valid pixels"
return 200, f"200, {st['valid']:.0%} valid"
Expand Down
2 changes: 1 addition & 1 deletion acceptance/src/eopf_accept/store_checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,7 @@ def st03_multiscales(ctx: StoreContext) -> Result:
if i in (0, len(levels) - 1) and not (in_layout or in_group):
which = "maxzoom" if i == 0 else "minzoom"
fails.append(f"{path}: no spatial:shape+spatial:transform in the layout entry or the group; {which} can't be derived, so tilejson without zoom params returns 500")
# titiler-eopf 0.12 (5fbea81 to v0.12.2) reads layout["spatial:transform"] with no fallback:
# titiler-eopf 0.12 (5fbea81 to v0.12.4) reads layout["spatial:transform"] with no fallback:
# get_multiscale_level for every level, and _get_variable's
# `attrs.get("spatial:transform", layout["spatial:transform"])` evaluates its default
# first. A transform on the level group alone doesn't help: every tile returns 500.
Expand Down
2 changes: 1 addition & 1 deletion acceptance/src/eopf_accept/storeio.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ def __init__(self, url: str, budget: Budget, *, forbidden_is_missing: bool = Fal
self.budget = budget
self._stores: dict[str, object] = {}
scheme = urlparse(self.url).scheme
# object_store's own truthy values (y and Y included; verified against obstore 0.11)
# object_store's own truthy values (y and Y included; verified against obstore 0.12)
self.unsigned = scheme in ("http", "https") or (
scheme == "s3" and os.environ.get("AWS_SKIP_SIGNATURE", "").lower() in ("1", "true", "yes", "y", "on"))
self.absent = (FileNotFoundError, PermissionDeniedError) if forbidden_is_missing and self.unsigned else (FileNotFoundError,)
Expand Down
Loading
Loading