feat(firestore): add BSON cross-type query ordering support - #18405
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces support for ordering and comparing BSON types (such as BSON min/max keys, object IDs, binaries, regexes, timestamps, and numbers) in Firestore. Feedback on these changes suggests adding defensive checks for keys in BSON regex and timestamp maps to prevent potential KeyError exceptions, as well as simplifying the number comparison logic by removing redundant float attribute checks.
b92472b to
9e8a85b
Compare
9e8a85b to
2c6fc33
Compare
| left_val = left_val.value | ||
| if hasattr(right_val, "value"): | ||
| right_val = right_val.value | ||
| return Order.compare_doubles(float(left_val), float(right_val)) |
There was a problem hiding this comment.
It looks like BSONDecimal can hold larger values than float. maybe we should use decimal.Decimal here?
There was a problem hiding this comment.
Exactly right. Casting to float caused overflow/precision loss on large values (e.g. 1e1000).
We updated compare_numbers to extract decimal.Decimal via .to_decimal(), unified NaN handling across both float and Decimal, and convert float to Decimal(str(float)) when comparing across types to prevent Python's TypeError and avoid float overflow. We've also added unit tests for 1e1000 and Decimal NaN comparisons.
| if hasattr(left_val, "value"): | ||
| left_val = left_val.value | ||
| if hasattr(right_val, "value"): | ||
| right_val = right_val.value |
There was a problem hiding this comment.
This wouldn't be needed if we implement __int__ and __float__ in the BSON types, so they are automatically treated as numbers (We would still need to compare decimals for BSONDecimal though)
There was a problem hiding this comment.
Implementing int or float on BSONDecimal128 would still lose precision or overflow for values exceeding standard IEEE 754 float limits (1e1000).
By using a small inline extractor in compare_numbers:
def _to_number(val):
num = decode_value(val, None)
to_decimal = getattr(num, "to_decimal", None)
return to_decimal() if callable(to_decimal) else getattr(num, "value", num)we cleanly unbox native ints/floats, BSONInt32 (.value), and BSONDecimal128 (.to_decimal()) without modifying the public interfaces or type contracts of the BSON classes.
2c6fc33 to
25549bf
Compare
25549bf to
dc953a7
Compare
dc953a7 to
3250d05
Compare
3250d05 to
aeb9997
Compare
| elif isinstance(right_val, decimal.Decimal) and isinstance(left_val, float): | ||
| left_val = decimal.Decimal(str(left_val)) | ||
|
|
||
| return Order._compare_to(left_val, right_val) |
There was a problem hiding this comment.
- math.isnan can raise OverflowError if the value is too large, so we might have to guard against that
- Can we pull
_to_numberand_is_nanout into helper methods, so we don't have to re-define them on each invocation? - [optional] if we can avoid calling
decode_value()in_to_numberand inspect the protobuf fields directly (like the othercompare_*methods), that should keep things fast
There was a problem hiding this comment.
Done!
- Extracted module-level
_to_numberand_is_nanhelpers. _to_numbernow directly inspects the protobuf fields (integer_value,double_value, andmap_valuefor__int__/__decimal128__) to avoid the overhead ofdecode_value()._is_nansafely handles non-floats, returnsFalseforintandDecimal, and catchesOverflowError.
17ac4e9 to
96b41bc
Compare
96b41bc to
4c62e1e
Compare
4c62e1e to
309ce0b
Compare
02d750d to
9a60436
Compare
…ation - Perform automatic BSON deserialization in decode_dict and DocumentSnapshot.to_dict using _BSONType._from_dict. - Remove decode_bson configuration parameter across Client, AsyncClient, BaseClient, and DocumentSnapshot. - Preserve precise return type annotations in decode_dict and restore docstring Raises section. Towards #18402
…ecode_value - Restore full Union return type with _BSONType on decode_value. - Restore Returns and Raises docstring sections in decode_value matching base branch. - Remove unused _BSON_DECODERS import from _helpers.py. - Revert extraneous changes to pipeline_result.py. Towards #18402
… types with _BSONType - Annotate decode_dict with Union[dict, Vector, _BSONType]. - Update PipelineResult.data to return dict | Vector | _BSONType | None. - Import _BSONType under TYPE_CHECKING in pipeline_result.py. Towards #18402
…nsions for librarian - Make client a required positional parameter in decode_value and decode_dict. - Format comprehensions in _helpers.py as single lines to satisfy librarian generation check. Towards #18402
Rename abstract base class _BSONType to BSONType and export it in google.cloud.firestore_v1 and __all__. Update return type annotations and docstrings on decode_value, decode_dict, and PipelineResult.data.
…back Update BSONType._from_dict to return Optional[Union[BSONType, bytes]] and safely catch decoder exceptions. Remove Any annotations from _BSON_DECODERS. Update decode_dict and PipelineResult.data return types to include bytes.
Format self.data() with !r in f-string to satisfy mypy str-bytes-safe check after adding bytes to PipelineResult.data return type.
ee72b87 to
6351257
Compare
Update test_bson_decimal128_special_values and its async variant to expect deserialized BSONDecimal128 instances instead of raw wire dictionary representations, aligning with BSON read deserialization.
…risons Use _BSON_KEY_TO_TYPE_ORDER dictionary lookup in order.py for O(1) wire key resolution, decoupling bson.py from query ordering. Consolidate cross-type numeric comparisons in compare_numbers with safe Decimal handling and restore compare_doubles to standard float comparisons.
Rename ambiguous single-letter variable l to left_val to satisfy flake8 E741 and align expression formatting with ruff.
6351257 to
eca1b4a
Compare
8ed76bd to
7910114
Compare
…pare_numbers Separate BSON_TIMESTAMP from native TIMESTAMP to conform to the cross-SDK 17-rank TypeOrder specification. Restore compare_timestamps to native Firestore timestamps and introduce compare_bson_timestamps. Extract module-level _to_number and _is_nan helpers, directly inspecting protobuf fields to optimize numeric comparisons.
7910114 to
d5163ff
Compare
🤖 I have created a release *beep* *boop* --- <details><summary>django-google-spanner: 5.2.0</summary> ## [5.2.0](django-google-spanner-v5.1.0...django-google-spanner-v5.2.0) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>gapic-generator: 1.41.0</summary> ## [1.41.0](gapic-generator-v1.40.0...gapic-generator-v1.41.0) (2026-09-29) ### Features * **gapic-generator:** add schema support for resumable uploads ([#18480](#18480)) ([78ff458](78ff458)) ### Bug Fixes * **gapic-generator:** init mock response in version header test ([#18488](#18488)) ([7871fa9](7871fa9)) </details> <details><summary>google-api-core: 2.40.0</summary> ## [2.40.0](google-api-core-v2.39.0...google-api-core-v2.40.0) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>google-auth-httplib2: 0.4.3</summary> ## [0.4.3](google-auth-httplib2-v0.4.2...google-auth-httplib2-v0.4.3) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>google-auth-oauthlib: 1.5.0</summary> ## [1.5.0](google-auth-oauthlib-v1.4.1...google-auth-oauthlib-v1.5.0) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) ### Bug Fixes * **google-auth-oauthlib:** prevent ipv6 address reuse ([#18463](#18463)) ([30b4f44](30b4f44)) </details> <details><summary>google-cloud-bigtable: 2.48.0</summary> ## [2.48.0](google-cloud-bigtable-v2.47.0...google-cloud-bigtable-v2.48.0) (2026-09-29) ### Features * **bigtable:** Reroute Mutations Batcher to use data client ([#18200](#18200)) ([0b488a3](0b488a3)) * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>google-cloud-core: 2.8.0</summary> ## [2.8.0](google-cloud-core-v2.7.0...google-cloud-core-v2.8.0) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>google-cloud-dns: 0.37.2</summary> ## [0.37.2](google-cloud-dns-v0.37.1...google-cloud-dns-v0.37.2) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>google-cloud-documentai-toolbox: 0.17.4</summary> ## [0.17.4](google-cloud-documentai-toolbox-v0.17.3...google-cloud-documentai-toolbox-v0.17.4) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>google-cloud-firestore: 2.33.0</summary> ## [2.33.0](google-cloud-firestore-v2.32.0...google-cloud-firestore-v2.33.0) (2026-09-29) ### Features * **firestore:** add BSON cross-type query ordering support ([#18405](#18405)) ([13be295](13be295)) * **firestore:** add BSON read deserialization support ([#18402](#18402)) ([ce2544f](ce2544f)) * **firestore:** add PyMongo duck-typing serialization support ([#18406](#18406)) ([a1e0e00](a1e0e00)) </details> <details><summary>google-cloud-ndb: 2.7.0</summary> ## [2.7.0](google-cloud-ndb-v2.6.1...google-cloud-ndb-v2.7.0) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>google-cloud-runtimeconfig: 0.37.2</summary> ## [0.37.2](google-cloud-runtimeconfig-v0.37.1...google-cloud-runtimeconfig-v0.37.2) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>google-cloud-storage: 3.15.1</summary> ## [3.15.1](google-cloud-storage-v3.15.0...google-cloud-storage-v3.15.1) (2026-09-29) ### Bug Fixes * **storage:** add correct app hub uri prefix to aco traces ([#18483](#18483)) ([7cead02](7cead02)) </details> <details><summary>google-cloud-testutils: 1.10.0</summary> ## [1.10.0](google-cloud-testutils-v1.9.3...google-cloud-testutils-v1.10.0) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>google-resumable-media: 2.11.0</summary> ## [2.11.0](google-resumable-media-v2.10.2...google-resumable-media-v2.11.0) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> <details><summary>googleapis-common-protos: 1.75.5</summary> ## [1.75.5](googleapis-common-protos-v1.75.4...googleapis-common-protos-v1.75.5) (2026-09-29) ### Bug Fixes * resolve error where google/longrunning/operations.proto is missing ([#18477](#18477)) ([e1d306a](e1d306a)), refs [#18478](#18478) </details> <details><summary>proto-plus: 1.29.0</summary> ## [1.29.0](proto-plus-v1.28.4...proto-plus-v1.29.0) (2026-09-29) ### Features * declare Python3.15 support ([05b0c34](05b0c34)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Context & Problem
Firestore queries require client-side and cross-type value ordering support for BSON types, including BSONMinKey, BSONMaxKey, BSONObjectId, BSONInt32, BSONDecimal128, BSONBinary, BSONRegex, and BSONTimestamp, aligning cross-type ordering with backend specifications.
Summary of Changes
_BSON_KEY_TO_TYPE_ORDERmapping dictionary inorder.pyforTypeOrder.Order.compare_numberswith inline unboxing, unified NaN handling, and safe float-to-Decimalconversion to prevent float overflow and precision loss.Order.compare_doublesto standard float comparisons.Order.compare_timestampssupporting native Firestore timestamps andBSONTimestamp.test_order.pyverifying BSON type ordering,1e1000large Decimal comparisons, DecimalNaNhandling, and wire-key mapping lookups.Verification
pytest tests/unit/v1/test_order.py tests/unit/v1/test_bson.py(112 passed in 1.18s).nox -e lint(all checks passed, 276 files formatted).nox -s mypy-3.11(Success: no issues found in 111 source files).