You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Coordinate Arrow reader close with the complete batch operation, preventing idle SQLCancel from closing a result before metadata/fetch work finishes.
Cancel only an active fetch and fence fetch completion against outstanding SQLCancel. Idle close returns before weak-handle acquisition, GIL release, or mutex acquisition.
Serialize complete iterator advancement and cleanup, including delegated reads and exported Arrow C streams. Reentrant close on the reading thread defers cleanup until advancement unwinds. Closing a never-started generator also performs cleanup.
Check both the existing result-description identity and weak statement identity. Stale readers cannot consume, close, or reset bookkeeping for a reexecuted cursor, a subsequent result set, or a replacement statement after failed execute. Explicit parent-cursor close still raises ProgrammingError. No new execute-path generation counter or strong statement ownership is introduced.
Stop on HY008 even with SQL_SUCCESS_WITH_INFO. Recognize HY007/HY010/24000 as shutdown results only after this reader requested close and issued cancellation. Preserve unrelated diagnostics regardless of record ordering.
Run native-boundary probes in Windows PR CI using the default-OFF MSSQL_ARROW_TEST_INTERPOSITION option. Requested but unavailable controls fail instead of skipping; ordinary Release builds expose no test pointer API.
Bound concurrency probes in subprocesses, guard from_stream cases on supported PyArrow versions lacking that API, and add a real-driver cancellation test. It observes the same outstanding SQLFetch across a DMV round trip while the reader is blocked by a known lock owner, invokes the real SQLCancel, and verifies termination and cursor reuse before releasing the blocking transaction. LocalDB CI receives the required DMV-read permissions.
The native completion fence and Python batch lock protect different boundaries and remain necessary. No public API/type-stub changes or throughput improvement claims.
Latest review fixes and validation
Revision: 4f5a5b11241770d4b2ad33e2cd0932e03537988f (after merging current main).
Removed the Unix exported-symbol fallback. Only the default-OFF compiled _arrow_test_pointer_addresses API enables interposition. Ordinary builds skip; explicitly requested but missing controls fail. Focused regressions guard both policies without loading exported globals.
Accept both SQL_SUCCESS and SQL_SUCCESS_WITH_INFO for exactly one SQLCancel. The live blocked-fetch probe now runs with the driver's actual result and with an injected success-with-info return after a genuinely successful real SQLCancel; both still require termination and cursor reuse before releasing the blocking transaction.
.\build.bat x64 from mssql_python\pybind with MSSQL_ARROW_TEST_INTERPOSITION=1: passed.
python -m pytest tests\test_004_cursor_arrow.py -m "not slow" -q --tb=short with compiled controls: 82 passed, 1 skipped per provider (msodbcsql18 and mssql-odbc, separate processes).
.\build.bat x64 with the flag unset: passed.
The same pytest command on the ordinary Release build: 71 passed, 12 skipped per provider. Eleven cases require the compiled controls; ordinary runs do not acquire a new DMV-permission requirement.
python -m pre_commit run black-check --all-files --hook-stage pre-push: passed, including the push hook.
git diff --check: passed.
All three newly added regression cases failed before these fixes. A separate requested-controls/ordinary-build negative run failed with the expected missing-controls message rather than skipping. The full repository suite and Linux/macOS runtime validation were not run locally; fresh CI is pending for this revision. Earlier-revision evidence follows.
Validation (previous revision)
Revision: d84b6ebc231d11456ac3ccb482507fb389fc50c6. Windows x64, Python 3.13.17, PyArrow 25.0.1, live SQL Server. Both msodbcsql18 and mssql-odbc ran in separate processes.
.\build.bat x64 from mssql_python\pybind, with MSSQL_ARROW_TEST_INTERPOSITION=1: passed, clean Release build using the Windows PR CI opt-in.
python -m pytest tests\test_004_cursor_arrow.py -m "not slow" -q --tb=short, with that build: 79 passed, 1 skipped per provider, including all native-boundary cases, the real blocked-driver-fetch probe, bounded concurrency probes, and stress coverage.
.\build.bat x64 with MSSQL_ARROW_TEST_INTERPOSITION unset: passed, clean ordinary Release build.
python -c "from mssql_python import ddbc_bindings; assert not hasattr(ddbc_bindings.module, '_arrow_test_pointer_addresses')": passed with the ordinary build.
python -m pytest tests\test_004_cursor_arrow.py -m "not slow" -q --tb=short, with the ordinary build: 69 passed, 11 skipped per provider. Ten skips require interposition, explicitly enabled in Windows PR CI.
python -m pytest tests\test_039_rs_runtime_pipelines.py::test_modified_pipeline_yaml_parses -q --tb=short: 1 passed. Parsed configuration checks confirmed Windows test controls and LocalDB DMV grants.
python -m pre_commit run black-check --all-files --hook-stage pre-push: passed, including the push hook.
python -m pre_commit validate-config: passed.
git diff --check: passed.
Additional controls: both replacement-statement regressions failed before the weak-identity fix; both C-stream parameters skipped when from_stream was deliberately absent; deliberately frozen cleanup was terminated by a subprocess timeout. PyArrow 14 itself was not installed on this Python 3.13 environment. Earlier controls reproduced stale-result cleanup, reentrant deadlock, and cancellation-first mixed-diagnostic failures before their fixes.
An intermediate full Arrow run caught a parent-closed exception regression in the new identity check; it was corrected before the final clean runs above. The full repository suite and Linux/macOS runtime validation were not run locally. The previous revision passed its remote PR validation matrix; the new revision requires a fresh CI run.
Existing scanner warning
DevSkim DS121708 flags an existing Arrow metadata memcpy. The flagged copies and allocation sizes are unchanged from the PR base; this fix introduces no memcpy calls. No scanner suppression or unrelated buffer-copy rewrite is included.
Unix / SQL Server 2022: Python 3.12.3, x86_64, SQL 16.0.4325.4; 5 paired comparisons and 1 warmup.
Unix / SQL Server 2025: Python 3.12.3, x86_64, SQL 17.0.5065.9; 5 paired comparisons and 1 warmup.
A consistent change requires more than 20% median paired movement, at least 1 ms between the median runtimes, and at least 80% of pairs exceeding the relative threshold in the same direction. A slowdown without enough pair agreement is reported as inconsistent.
The displayed change is the median of paired before-and-after ratios. It is not recalculated from the two displayed median runtimes.
Both revisions use profiling-enabled builds on the same agent and database, with alternating order and discarded warmups. Results are diagnostic and do not represent production-wheel latency.
Raw samples and logs are attached to the ADO run as profiler-* artifacts.
123defclose(self):
124withself._lock:
125ifself._generator.gi_running:
126# A signal handler on the reading thread must let __next__ unwind.
! 127returnFalse128self._generator.close()
129# Closing a generator that has never been advanced skips its finally.130self._cleanup()
131returnTrue
Lines 345-353
345# the cleanup steps below fail and we end up retried later.346self._closed=True347try:
348self._close_requested.request_close()
! 349exceptExceptionase: # pylint: disable=broad-exception-caught350logger.warning("arrow_reader.close: SQLCancel raised: %s", e)
351352# Wait for any iterator advancement before closing the generator. The353# native close request interrupts a blocked fetch without taking this lock.
Preserve result ownership and unrelated diagnostics, defer reentrant cleanup, and exercise native cancellation boundaries in Windows CI.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A stale reader can access or close a replacement statement, and the tests have compatibility, deadlock-timeout, and live-driver cancellation coverage gaps.
Test cancellation of a synchronized in-flight SQLFetch
mssql_python/pybind/ddbc_bindings.cpp:5439
The deterministic cancellation tests do not exercise cancellation of an actual in-flight driver fetch: their interposed fetch returns from real_fetch before waiting, and their cancel replaces SQLCancel entirely. The remaining real-driver test still relies on a sleep, so it cannot establish that this call was blocked when close occurred. Please add a synchronized live-driver case that cancels a genuinely outstanding SQLFetch and verifies both termination and cursor reuse.
Guard the PyArrow 14-incompatible from_stream cases
tests/test_004_cursor_arrow.py:570
RecordBatchReader.from_stream was added in PyArrow 15, but the project's PyArrow extra allows version 14. With that supported version, both c_stream cases fail with AttributeError before exercising shutdown. Skip this parameter when from_stream is unavailable, as test_arrow_reader_pycapsule_protocol already does.
Bound deadlock-prone concurrency probes with a subprocess timeout
tests/test_004_cursor_arrow.py:626
If the shutdown code deadlocks, the timed joins expire but the unconditional reader.close() then waits on the same iterator lock indefinitely, so this test never reaches its thread-liveness assertion and can hang the entire test job. Run this concurrency probe in a subprocess with a timeout, as for the native cancellation probe, so a deadlock is reported as a bounded failure.
Check weak statement identity, bound concurrency probes, support PyArrow 14, and exercise real blocked SQLFetch cancellation before releasing the server lock.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The interposition tests can run without explicit opt-in and fail for logins lacking DMV permissions, while the cancellation probe rejects a valid successful return code.
Remove exported-symbol fallback and test both missing-control policies. Exercise SQL_SUCCESS_WITH_INFO after real driver cancellation while preserving the exactly-once assertion.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
The shutdown implementation addresses the reported interleavings and is consistent with the applicable Arrow reader lifecycle and ODBC cancellation requirements. One Medium regression-coverage gap remains: the new statement-identity guard makes test_arrow_reader_cleanup_survives_close_cursor_failure pass without reaching its injected SQL_CLOSE failure or checking recovery. The existing inline thread has the requested correction: inject failure on the original handle type and assert the close attempt, warning, and bookkeeping reset. No production functional defect was established.
Inject failure on the original statement type and assert one close attempt, the warning, bookkeeping reset, idempotent close, and parent cursor recovery. Keep production ownership guards unchanged.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Cross-thread fetch/cancel ordering and diagnostics require live-driver validation, particularly around the completion-to-cancellation window, with Linux/macOS coverage and fresh CI still pending.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The cross-thread cancellation and statement-lifetime changes require live-driver, cross-platform validation, but CI is pending and the necessary tests cannot run in this environment.
No open issues 🎉
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work Item / Issue Reference
Summary
Coordinate Arrow reader close with the complete batch operation, preventing idle SQLCancel from closing a result before metadata/fetch work finishes.
The native completion fence and Python batch lock protect different boundaries and remain necessary. No public API/type-stub changes or throughput improvement claims.
Latest review fixes and validation
Revision:
4f5a5b11241770d4b2ad33e2cd0932e03537988f(after merging current main)._arrow_test_pointer_addressesAPI enables interposition. Ordinary builds skip; explicitly requested but missing controls fail. Focused regressions guard both policies without loading exported globals..\build.bat x64frommssql_python\pybindwithMSSQL_ARROW_TEST_INTERPOSITION=1: passed.python -m pytest tests\test_004_cursor_arrow.py -m "not slow" -q --tb=shortwith compiled controls: 82 passed, 1 skipped per provider (msodbcsql18 and mssql-odbc, separate processes)..\build.bat x64with the flag unset: passed.python -m pre_commit run black-check --all-files --hook-stage pre-push: passed, including the push hook.git diff --check: passed.All three newly added regression cases failed before these fixes. A separate requested-controls/ordinary-build negative run failed with the expected missing-controls message rather than skipping. The full repository suite and Linux/macOS runtime validation were not run locally; fresh CI is pending for this revision. Earlier-revision evidence follows.
Validation (previous revision)
Revision:
d84b6ebc231d11456ac3ccb482507fb389fc50c6. Windows x64, Python 3.13.17, PyArrow 25.0.1, live SQL Server. Both msodbcsql18 and mssql-odbc ran in separate processes..\build.bat x64frommssql_python\pybind, withMSSQL_ARROW_TEST_INTERPOSITION=1: passed, clean Release build using the Windows PR CI opt-in.python -m pytest tests\test_004_cursor_arrow.py -m "not slow" -q --tb=short, with that build: 79 passed, 1 skipped per provider, including all native-boundary cases, the real blocked-driver-fetch probe, bounded concurrency probes, and stress coverage..\build.bat x64withMSSQL_ARROW_TEST_INTERPOSITIONunset: passed, clean ordinary Release build.python -c "from mssql_python import ddbc_bindings; assert not hasattr(ddbc_bindings.module, '_arrow_test_pointer_addresses')": passed with the ordinary build.python -m pytest tests\test_004_cursor_arrow.py -m "not slow" -q --tb=short, with the ordinary build: 69 passed, 11 skipped per provider. Ten skips require interposition, explicitly enabled in Windows PR CI.python -m pytest tests\test_039_rs_runtime_pipelines.py::test_modified_pipeline_yaml_parses -q --tb=short: 1 passed. Parsed configuration checks confirmed Windows test controls and LocalDB DMV grants.python -m pre_commit run black-check --all-files --hook-stage pre-push: passed, including the push hook.python -m pre_commit validate-config: passed.git diff --check: passed.Additional controls: both replacement-statement regressions failed before the weak-identity fix; both C-stream parameters skipped when from_stream was deliberately absent; deliberately frozen cleanup was terminated by a subprocess timeout. PyArrow 14 itself was not installed on this Python 3.13 environment. Earlier controls reproduced stale-result cleanup, reentrant deadlock, and cancellation-first mixed-diagnostic failures before their fixes.
An intermediate full Arrow run caught a parent-closed exception regression in the new identity check; it was corrected before the final clean runs above. The full repository suite and Linux/macOS runtime validation were not run locally. The previous revision passed its remote PR validation matrix; the new revision requires a fresh CI run.
Existing scanner warning
DevSkim DS121708 flags an existing Arrow metadata memcpy. The flagged copies and allocation sizes are unchanged from the PR base; this fix introduces no memcpy calls. No scanner suppression or unrelated buffer-copy rewrite is included.