Skip to content

feat(firestore): add BSON cross-type query ordering support - #18405

Merged
ohmayr merged 13 commits into
mainfrom
bson-pr3-ordering
Sep 26, 2026
Merged

ohmayr merged 13 commits into
mainfrom
bson-pr3-ordering

Conversation

@ohmayr

@ohmayr ohmayr commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Context & Problem

Firestore queries require client-side and cross-type value ordering support for BSON types, including BSONMinKey, BSONMaxKey, BSONObjectId, BSONInt32, BSONDecimal128, BSONBinary, BSONRegex, and BSONTimestamp, aligning cross-type ordering with backend specifications.

Summary of Changes

  • Add _BSON_KEY_TO_TYPE_ORDER mapping dictionary in order.py for $O(1)$ wire-key resolution to TypeOrder.
  • Consolidate cross-type numeric comparisons in Order.compare_numbers with inline unboxing, unified NaN handling, and safe float-to-Decimal conversion to prevent float overflow and precision loss.
  • Restore Order.compare_doubles to standard float comparisons.
  • Implement cross-type timestamp comparison in Order.compare_timestamps supporting native Firestore timestamps and BSONTimestamp.
  • Add unit tests in test_order.py verifying BSON type ordering, 1e1000 large Decimal comparisons, Decimal NaN handling, and wire-key mapping lookups.
  • Update system test to verify query ordering across BSON types.

Verification

  • Ran local unit test suite: pytest tests/unit/v1/test_order.py tests/unit/v1/test_bson.py (112 passed in 1.18s).
  • Ran full test suite across Python 3.11 with python and upb protobuf implementations (3,708 passed).
  • Ran linter suite: nox -e lint (all checks passed, 276 files formatted).
  • Ran type checker: nox -s mypy-3.11 (Success: no issues found in 111 source files).

@ohmayr
ohmayr added this pull request to stack #18386 September 16, 2026 22:49

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for ordering and comparing BSON types (such as BSON min/max keys, object IDs, binaries, regexes, timestamps, and numbers) in Firestore. Feedback on these changes suggests adding defensive checks for keys in BSON regex and timestamp maps to prevent potential KeyError exceptions, as well as simplifying the number comparison logic by removing redundant float attribute checks.

Comment thread packages/google-cloud-firestore/google/cloud/firestore_v1/order.py Outdated
Comment thread packages/google-cloud-firestore/google/cloud/firestore_v1/order.py Outdated
Comment thread packages/google-cloud-firestore/google/cloud/firestore_v1/order.py Outdated
@ohmayr
ohmayr marked this pull request as ready for review September 16, 2026 22:51
@ohmayr
ohmayr requested a review from a team as a code owner September 16, 2026 22:51
@ohmayr
ohmayr force-pushed the bson-pr3-ordering branch 3 times, most recently from b92472b to 9e8a85b Compare September 16, 2026 23:18
Comment thread packages/google-cloud-firestore/google/cloud/firestore_v1/order.py
Comment thread packages/google-cloud-firestore/google/cloud/firestore_v1/order.py
left_val = left_val.value
if hasattr(right_val, "value"):
right_val = right_val.value
return Order.compare_doubles(float(left_val), float(right_val))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like BSONDecimal can hold larger values than float. maybe we should use decimal.Decimal here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly right. Casting to float caused overflow/precision loss on large values (e.g. 1e1000).

We updated compare_numbers to extract decimal.Decimal via .to_decimal(), unified NaN handling across both float and Decimal, and convert float to Decimal(str(float)) when comparing across types to prevent Python's TypeError and avoid float overflow. We've also added unit tests for 1e1000 and Decimal NaN comparisons.

if hasattr(left_val, "value"):
left_val = left_val.value
if hasattr(right_val, "value"):
right_val = right_val.value

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wouldn't be needed if we implement __int__ and __float__ in the BSON types, so they are automatically treated as numbers (We would still need to compare decimals for BSONDecimal though)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementing int or float on BSONDecimal128 would still lose precision or overflow for values exceeding standard IEEE 754 float limits (1e1000).

By using a small inline extractor in compare_numbers:

def _to_number(val):
    num = decode_value(val, None)
    to_decimal = getattr(num, "to_decimal", None)
    return to_decimal() if callable(to_decimal) else getattr(num, "value", num)

we cleanly unbox native ints/floats, BSONInt32 (.value), and BSONDecimal128 (.to_decimal()) without modifying the public interfaces or type contracts of the BSON classes.

@ohmayr
ohmayr requested a review from a team as a code owner September 21, 2026 09:06
Comment thread packages/google-cloud-firestore/google/cloud/firestore_v1/order.py
elif isinstance(right_val, decimal.Decimal) and isinstance(left_val, float):
left_val = decimal.Decimal(str(left_val))

return Order._compare_to(left_val, right_val)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • math.isnan can raise OverflowError if the value is too large, so we might have to guard against that
  • Can we pull _to_number and _is_nan out into helper methods, so we don't have to re-define them on each invocation?
  • [optional] if we can avoid calling decode_value() in _to_number and inspect the protobuf fields directly (like the other compare_* methods), that should keep things fast

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

  • Extracted module-level _to_number and _is_nan helpers.
  • _to_number now directly inspects the protobuf fields (integer_value, double_value, and map_value for __int__ / __decimal128__) to avoid the overhead of decode_value().
  • _is_nan safely handles non-floats, returns False for int and Decimal, and catches OverflowError.

@ohmayr
ohmayr force-pushed the bson-pr3-ordering branch 2 times, most recently from 02d750d to 9a60436 Compare September 24, 2026 01:44
…ation

- Perform automatic BSON deserialization in decode_dict and DocumentSnapshot.to_dict using _BSONType._from_dict.
- Remove decode_bson configuration parameter across Client, AsyncClient, BaseClient, and DocumentSnapshot.
- Preserve precise return type annotations in decode_dict and restore docstring Raises section.

Towards #18402
…ecode_value

- Restore full Union return type with _BSONType on decode_value.
- Restore Returns and Raises docstring sections in decode_value matching base branch.
- Remove unused _BSON_DECODERS import from _helpers.py.
- Revert extraneous changes to pipeline_result.py.

Towards #18402
… types with _BSONType

- Annotate decode_dict with Union[dict, Vector, _BSONType].
- Update PipelineResult.data to return dict | Vector | _BSONType | None.
- Import _BSONType under TYPE_CHECKING in pipeline_result.py.

Towards #18402
…nsions for librarian

- Make client a required positional parameter in decode_value and decode_dict.
- Format comprehensions in _helpers.py as single lines to satisfy librarian generation check.

Towards #18402
Rename abstract base class _BSONType to BSONType and export it in
google.cloud.firestore_v1 and __all__. Update return type annotations
and docstrings on decode_value, decode_dict, and PipelineResult.data.
…back

Update BSONType._from_dict to return Optional[Union[BSONType, bytes]]
and safely catch decoder exceptions. Remove Any annotations from
_BSON_DECODERS. Update decode_dict and PipelineResult.data return types
to include bytes.
Format self.data() with !r in f-string to satisfy mypy str-bytes-safe
check after adding bytes to PipelineResult.data return type.
@ohmayr
ohmayr force-pushed the bson-pr3-ordering branch 2 times, most recently from ee72b87 to 6351257 Compare September 24, 2026 20:23

@daniel-sanche daniel-sanche left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Update test_bson_decimal128_special_values and its async variant
to expect deserialized BSONDecimal128 instances instead of raw wire
dictionary representations, aligning with BSON read deserialization.
@ohmayr
ohmayr requested review from a team as code owners September 25, 2026 22:15
…risons

Use _BSON_KEY_TO_TYPE_ORDER dictionary lookup in order.py for O(1) wire
key resolution, decoupling bson.py from query ordering. Consolidate
cross-type numeric comparisons in compare_numbers with safe Decimal
handling and restore compare_doubles to standard float comparisons.
Rename ambiguous single-letter variable l to left_val to satisfy flake8
E741 and align expression formatting with ruff.
Base automatically changed from bson-pr2-reads to main September 25, 2026 22:48
@ohmayr
ohmayr force-pushed the bson-pr3-ordering branch 2 times, most recently from 8ed76bd to 7910114 Compare September 25, 2026 23:11
…pare_numbers

Separate BSON_TIMESTAMP from native TIMESTAMP to conform to the
cross-SDK 17-rank TypeOrder specification. Restore compare_timestamps
to native Firestore timestamps and introduce compare_bson_timestamps.
Extract module-level _to_number and _is_nan helpers, directly inspecting
protobuf fields to optimize numeric comparisons.
@ohmayr
ohmayr merged commit 13be295 into main Sep 26, 2026
49 checks passed
@ohmayr
ohmayr deleted the bson-pr3-ordering branch September 26, 2026 00:26
@release-please release-please Bot mentioned this pull request Sep 26, 2026
JoeWang1127 pushed a commit that referenced this pull request Sep 29, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>django-google-spanner: 5.2.0</summary>

##
[5.2.0](django-google-spanner-v5.1.0...django-google-spanner-v5.2.0)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>gapic-generator: 1.41.0</summary>

##
[1.41.0](gapic-generator-v1.40.0...gapic-generator-v1.41.0)
(2026-09-29)


### Features

* **gapic-generator:** add schema support for resumable uploads
([#18480](#18480))
([78ff458](78ff458))


### Bug Fixes

* **gapic-generator:** init mock response in version header test
([#18488](#18488))
([7871fa9](7871fa9))
</details>

<details><summary>google-api-core: 2.40.0</summary>

##
[2.40.0](google-api-core-v2.39.0...google-api-core-v2.40.0)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>google-auth-httplib2: 0.4.3</summary>

##
[0.4.3](google-auth-httplib2-v0.4.2...google-auth-httplib2-v0.4.3)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>google-auth-oauthlib: 1.5.0</summary>

##
[1.5.0](google-auth-oauthlib-v1.4.1...google-auth-oauthlib-v1.5.0)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))


### Bug Fixes

* **google-auth-oauthlib:** prevent ipv6 address reuse
([#18463](#18463))
([30b4f44](30b4f44))
</details>

<details><summary>google-cloud-bigtable: 2.48.0</summary>

##
[2.48.0](google-cloud-bigtable-v2.47.0...google-cloud-bigtable-v2.48.0)
(2026-09-29)


### Features

* **bigtable:** Reroute Mutations Batcher to use data client
([#18200](#18200))
([0b488a3](0b488a3))
* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>google-cloud-core: 2.8.0</summary>

##
[2.8.0](google-cloud-core-v2.7.0...google-cloud-core-v2.8.0)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>google-cloud-dns: 0.37.2</summary>

##
[0.37.2](google-cloud-dns-v0.37.1...google-cloud-dns-v0.37.2)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>google-cloud-documentai-toolbox: 0.17.4</summary>

##
[0.17.4](google-cloud-documentai-toolbox-v0.17.3...google-cloud-documentai-toolbox-v0.17.4)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>google-cloud-firestore: 2.33.0</summary>

##
[2.33.0](google-cloud-firestore-v2.32.0...google-cloud-firestore-v2.33.0)
(2026-09-29)


### Features

* **firestore:** add BSON cross-type query ordering support
([#18405](#18405))
([13be295](13be295))
* **firestore:** add BSON read deserialization support
([#18402](#18402))
([ce2544f](ce2544f))
* **firestore:** add PyMongo duck-typing serialization support
([#18406](#18406))
([a1e0e00](a1e0e00))
</details>

<details><summary>google-cloud-ndb: 2.7.0</summary>

##
[2.7.0](google-cloud-ndb-v2.6.1...google-cloud-ndb-v2.7.0)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>google-cloud-runtimeconfig: 0.37.2</summary>

##
[0.37.2](google-cloud-runtimeconfig-v0.37.1...google-cloud-runtimeconfig-v0.37.2)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>google-cloud-storage: 3.15.1</summary>

##
[3.15.1](google-cloud-storage-v3.15.0...google-cloud-storage-v3.15.1)
(2026-09-29)


### Bug Fixes

* **storage:** add correct app hub uri prefix to aco traces
([#18483](#18483))
([7cead02](7cead02))
</details>

<details><summary>google-cloud-testutils: 1.10.0</summary>

##
[1.10.0](google-cloud-testutils-v1.9.3...google-cloud-testutils-v1.10.0)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>google-resumable-media: 2.11.0</summary>

##
[2.11.0](google-resumable-media-v2.10.2...google-resumable-media-v2.11.0)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

<details><summary>googleapis-common-protos: 1.75.5</summary>

##
[1.75.5](googleapis-common-protos-v1.75.4...googleapis-common-protos-v1.75.5)
(2026-09-29)


### Bug Fixes

* resolve error where google/longrunning/operations.proto is missing
([#18477](#18477))
([e1d306a](e1d306a)),
refs
[#18478](#18478)
</details>

<details><summary>proto-plus: 1.29.0</summary>

##
[1.29.0](proto-plus-v1.28.4...proto-plus-v1.29.0)
(2026-09-29)


### Features

* declare Python3.15 support
([05b0c34](05b0c34))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants