[Cosmos] feat: Hedging Detection API (accessors on response/error types)

[NaluTripician]

Cosmos: Hedging Detection API (Python, Option B)

Closes #46899

Summary

Adds the cross-SDK Hedging Detection API to azure-cosmos (Python),
shipping Option B from the public spec: three accessor methods —
is_hedging_started(), get_requested_regions(), get_responded_regions()
— added directly to the four wrapper types and the three exception types
that customers receive from data-plane operations, backed by a shared
private _HedgingDetectionState.

response = container.read_item(item="abc", partition_key="pk1",
                               availability_strategy={"threshold_ms": 100})
if response.is_hedging_started():
    print("dispatched to:", [r.region_name for r in response.get_requested_regions()])
    print("responded from:", response.get_responded_regions())

Public surface (new)

Symbol	Module
RequestedRegion (frozen dataclass: region_name, reason)	azure.cosmos
RequestedRegionReason (enum, with _missing_ → UNKNOWN)	azure.cosmos
is_hedging_started() -> bool	added to 7 types
get_requested_regions() -> tuple[RequestedRegion, ...]	added to 7 types
get_responded_regions() -> tuple[str, ...]	added to 7 types

7 carrier types: CosmosDict, CosmosList, CosmosItemPaged,
CosmosAsyncItemPaged, CosmosHttpResponseError, CosmosBatchOperationError,
CosmosClientTimeoutError.

Design highlights

• Closure-argument state propagation (SE-002 / AC8). The hedging handler
  deep-copies request_params at the worker boundary (line 96 of
  _availability_strategy_handler.py). Storing state on request_params
  would silently swallow child appends. State is therefore threaded through
  the handler as an explicit hedging_state= closure argument, into the
  retry utility via kwargs["_hedging_state"], and back to the wrapper
  construction site by stashing on the response-headers dict under a
  private sentinel key that each wrapper pops in its __init__. Internal
  flag is also popped in _Request before reaching the HTTP pipeline.
• HEDGING recorded "dispatched, not necessarily wire-issued" (AC10). The
  hedging-handler append happens AFTER the threshold sleep AND AFTER the
  cancellation check, so a hedge arm that loses the race before its delay
  elapses produces NO phantom entry.
• Forward compat (SE-016).
  RequestedRegionReason._missing_(cls, raw) returns cls.UNKNOWN for any
  unrecognized payload — calling RequestedRegionReason("future_v42")
  never raises.
• Thread / coroutine safety (SE-017). _HedgingDetectionState uses a
  threading.Lock to guard the compound (append + flag set) update;
  snapshots are returned as immutable tuples.
• TRANSPORT_RETRY / CIRCUIT_BREAKER_PROBE are RESERVED in v1 (SE-005).
  Enum values are exported (forward compat) but a repo-wide grep test
  asserts they are never emitted by production code.

Acceptance-criteria matrix

AC	What	Test
AC1	Single-region success → is_hedging_started()==False	test_record_initial_does_not_set_hedging_flag
AC2	Primary wins under threshold → no phantom HEDGING entry	test_cancelled_before_delay_records_nothing (sync + async)
AC3	Hedge arm wins → is_hedging_started()==True, HEDGING entry present	test_record_hedging_sets_flag (sync + async)
AC4	Same-region retry → OPERATION_RETRY, not HEDGING	test_record_operation_retry_does_not_set_flag
AC5	Error path exposes accessors	test_cosmos_http_response_error_forwards
AC6	RequestedRegion frozen / hashable / picklable	test_diagnostics_types.py (9 tests)
AC7	All 7 types carry the 3 accessors	wrapper-accessors suite (sync + async)
AC8	Closure-arg pattern survives copy.deepcopy(request_params)	test_handler_dispatches_state_through_closure_not_params (sync + async)
AC9	Sync↔async parity guaranteed at test time	TestSyncAsyncParity.{test_parity_class_coverage,test_parity_method_coverage}
AC10	Cancelled-before-delay hedge arm records nothing	test_cancelled_before_delay_records_nothing (sync + async)
AC11	Duplicate responses preserved	test_responded_regions_preserves_duplicates
AC12	TRANSPORT_RETRY / CIRCUIT_BREAKER_PROBE never emitted in v1	test_reserved_reasons_absent_from_production_code (repo-wide grep)
AC13	Live multi-region smoke	tests/livecanary/test_hedging_detection_live[_async].py (opt-in via env vars)

Test execution

==== TOTAL: 53 passed, 0 failed ====
test_diagnostics_types.py        9 passed
test_hedging_detection.py        21 passed
test_hedging_detection_async.py  23 passed

Live canary suite is opt-in (skipped unless COSMOS_MULTI_REGION_ENDPOINT
and COSMOS_MULTI_REGION_KEY env vars are set).

Files changed

Production (12)

• azure/cosmos/_diagnostics_types.py (NEW)
• azure/cosmos/_diagnostics.py (NEW)
• azure/cosmos/__init__.py
• azure/cosmos/_cosmos_responses.py
• azure/cosmos/exceptions.py
• azure/cosmos/_availability_strategy_handler.py
• azure/cosmos/aio/_asynchronous_availability_strategy_handler.py
• azure/cosmos/_synchronized_request.py
• azure/cosmos/aio/_asynchronous_request.py
• azure/cosmos/_retry_utility.py
• azure/cosmos/aio/_retry_utility_async.py
• CHANGELOG.md

Tests / samples (7)

• tests/test_diagnostics_types.py (NEW)
• tests/test_hedging_detection.py (NEW)
• tests/test_hedging_detection_async.py (NEW)
• tests/livecanary/__init__.py (NEW)
• tests/livecanary/test_hedging_detection_live.py (NEW)
• tests/livecanary/test_hedging_detection_live_async.py (NEW)
• samples/hedging_detection.py + samples/hedging_detection_async.py (NEW)

Why no .pyi?

azure-cosmos doesn't ship a __init__.pyi; only py.typed is present.
Inline annotations on the new public types provide equivalent type-checker
coverage. Spec S2 was a SHOULD, not a MUST.

Notes for reviewers

• This is the Python half of a cross-SDK API (see also pending .NET/Java
  PRs).
• All new public symbols are exported through azure/cosmos/__init__.py.
• The implementation is deliberately decoupled from logging — the public
  spec §7 forbids log-string scraping as a customer contract.

[Copilot]

Pull request overview

Adds the cross-SDK Hedging Detection API (Option B) to azure-cosmos: three accessor methods (is_hedging_started, get_requested_regions, get_responded_regions) backed by a private _HedgingDetectionState, plus two new public value types (RequestedRegion, RequestedRegionReason). State is propagated as an explicit closure argument through the sync/async hedging handlers and retry utility, and stashed on response headers under a private sentinel key that wrappers pop in __init__.

Changes:

• New _diagnostics.py / _diagnostics_types.py modules with the state object, accessor mixin, and value types; exported from azure.cosmos.
• Threaded hedging_state= through sync + async availability-strategy handlers, _Request, retry utilities, and the seven public wrapper/exception types.
• Added sync/async unit tests, a parity checker, opt-in live multi-region canary tests, and two samples.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
azure/cosmos/_diagnostics_types.py	New public RequestedRegion dataclass + non-exhaustive RequestedRegionReason enum with _missing_ → UNKNOWN.
azure/cosmos/_diagnostics.py	_HedgingDetectionState, accessor mixin, and helpers to attach/pop state on headers.
azure/cosmos/init.py	Exports the two new public types.
azure/cosmos/_cosmos_responses.py	Adds mixin to CosmosDict/CosmosList/CosmosItemPaged/CosmosAsyncItemPaged, pops sentinel from headers.
azure/cosmos/exceptions.py	Adds mixin to CosmosHttpResponseError, CosmosBatchOperationError, CosmosClientTimeoutError.
azure/cosmos/_availability_strategy_handler.py	Records INITIAL/HEDGING dispatch + responding region in sync handler.
azure/cosmos/aio/_asynchronous_availability_strategy_handler.py	Same for async handler.
azure/cosmos/_synchronized_request.py	Creates per-op state, threads it through retry, attaches to response headers; adds _resolve_region_name.
azure/cosmos/aio/_asynchronous_request.py	Async parity for the above.
azure/cosmos/_retry_utility.py	New _record_retry_diagnostics_and_attach helper to record REGION_FAILOVER/OPERATION_RETRY and attach state to raised errors.
azure/cosmos/aio/_retry_utility_async.py	Async parity for retry recording.
CHANGELOG.md	Feature entry under 4.16.0b3 (Unreleased).
tests/test_diagnostics_types.py	Enum + frozen-dataclass tests.
tests/test_hedging_detection.py	Sync state + wrapper + handler tests, repo-wide grep for reserved reasons.
tests/test_hedging_detection_async.py	Async twin of the above plus sync↔async parity checker.
tests/livecanary/*	Opt-in live multi-region smoke tests (sync + async).
samples/hedging_detection*.py	New customer-facing samples.

Comments suppressed due to low confidence (2)

sdk/cosmos/azure-cosmos/azure/cosmos/_cosmos_responses.py:85

• HEDGING_STATE_HEADER_KEY = "__hedging_state__" is set as a normal string key on the headers dict. _response_headers_list for paged operations holds the actual CaseInsensitiveDict instances mutated by the orchestrator; once _pop_state_from_headers runs on a wrapper-constructed page (e.g., for queries that produce CosmosDict items in addition to the page-level headers), the sentinel disappears from the same CaseInsensitiveDict reference. After that, _refresh_hedging_state_from_pages (which scans _response_headers_list looking for the sentinel) will miss the state on subsequent calls. Verify whether the paged-item construction path can ever pop the sentinel before CosmosItemPaged itself caches it, otherwise the accessors may regress to defaults after the first call.

    def _refresh_hedging_state_from_pages(self) -> None:
        """Internal: scan pages from newest to oldest and update
        ``self._hedging_state`` with the most recent attached state, if any.
        Does not mutate the underlying headers dicts."""
        if not self._response_headers_list:
            return
        for h in reversed(self._response_headers_list):
            from ._diagnostics import HEDGING_STATE_HEADER_KEY
            state = h.get(HEDGING_STATE_HEADER_KEY) if h is not None else None
            if state is not None:
                self._hedging_state = state
                return

sdk/cosmos/azure-cosmos/tests/test_hedging_detection_async.py:334

• The reserved-reasons grep test is duplicated verbatim between test_hedging_detection.py and test_hedging_detection_async.py. Both walk every .py under azure/cosmos/ on every test run, and the comment-stripping regex only strips trailing #… plus docstring boundary lines, missing the case where the reserved names appear inside multi-line strings or pylint-disable comments. Consolidate into a single helper to avoid maintenance drift and reduce duplicate I/O in CI.

    def test_reserved_reasons_absent_from_production_code_async(self):
        """Same repo-wide guarantee as the sync test, executed via the async
        suite so it runs in CI even when the sync test is skipped."""
        import os
        import re

        here = os.path.dirname(os.path.abspath(__file__))
        prod_root = os.path.abspath(os.path.join(here, "..", "azure", "cosmos"))
        assert os.path.isdir(prod_root)

        offenders = []
        for dirpath, _dirs, files in os.walk(prod_root):
            for fn in files:
                if not fn.endswith(".py") or fn == "_diagnostics_types.py":
                    continue
                full = os.path.join(dirpath, fn)
                with open(full, "r", encoding="utf-8") as fh:
                    for ln, line in enumerate(fh, start=1):
                        stripped = line.split("#", 1)[0]
                        if re.match(r"\s*('''|\"\"\")", stripped):
                            continue
                        for offender in (
                            "RequestedRegionReason.TRANSPORT_RETRY",
                            "RequestedRegionReason.CIRCUIT_BREAKER_PROBE",
                        ):
                            if offender in stripped:
                                offenders.append((full, ln, offender))
        assert offenders == [], repr(offenders)