Repository navigation
feat(csharp): collect garbage when an exported driver goes idle - #4917
Merged
Merged
Conversation
A driver published with NativeAOT has its own GC heap, and the runtime only collects it in response to allocations. Once a caller releases its databases, the memory the driver last used stays committed for the life of the process. CAdbcDriverExporter now counts open databases and, once the last one is released and none is opened for IdleCollectionDelay, forces a full, compacting collection on a thread-pool thread. IdleCollectionDelay defaults to one second under NativeAOT and to null (disabled) under JIT, where the heap is shared with the host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Collection scheduling has unresolved lifecycle and concurrency issues, and unsupported delays can cause release operations to report failure.
4 open issues
What changed in this PR
Adds delayed garbage collection to reclaim memory retained by idle NativeAOT C# drivers.
Changes:
- Adds configurable idle collection, enabled by default under NativeAOT.
- Tracks database lifetimes and driver releases to schedule collection.
- Adds tests for scheduling, reopening, disabling, and defaults.
| File | Description |
|---|---|
| csharp/test/Apache.Arrow.Adbc.Tests/IdleCollectorTests.cs | Tests idle collection behavior and configuration. |
| csharp/src/Apache.Arrow.Adbc/C/IdleCollector.cs | Tracks activity and schedules delayed collection. |
| csharp/src/Apache.Arrow.Adbc/C/CAdbcDriverExporter.cs | Integrates lifecycle tracking and garbage collection. |
Review effort: Balanced
- Reject delays longer than Task.Delay supports, so that scheduling can't make a completed release report a failure. - Recheck the delay before collecting, so that disabling the feature stops a pending collection, and store it so it's read atomically. - Start a new idle period when the driver is released, so that its collection isn't skipped after the last database's has already run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Releasing an exported driver now starts a timer, so a NativeAOT driver starts a thread of its own. On Linux, that thread's first UTF-8 marshalling (to set its name) can run UTF8Encoding's static constructor while contending on another type's constructor lock. Lock's slow path then reads an environment variable, re-entering UTF8Encoding while its static instance is still null, and the runtime fails fast. This crashed the AOT fixture test host on every run in WSL. Running the constructor in AdbcDriverInit, on the caller's thread, avoids it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…workaround Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A collection still pending when the driver is released runs after the release returns, so a host that unloads the library then would crash. NativeAOT libraries can't be safely unloaded anyway; say so on the importer's canUnload parameter as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's changed
A driver published with NativeAOT gets its own GC heap, and the runtime only collects that heap in response to allocations. Nothing collects it on a timer, and the low-memory notification doesn't collect an idle heap either. So once a caller releases its databases, the memory the driver last used stays committed for the rest of the process's life.
CAdbcDriverExporternow counts the databases it has open. A collection is scheduled when the last database is released, or when the driver is released while no databases are open. It runs on a thread-pool thread afterIdleCollectionDelay, and only if no database was opened in the meantime. The collection isCollect+WaitForPendingFinalizers+ an aggressive, compacting gen2 collection.IdleCollectionDelayis a new public staticTimeSpan?, andnullturns the feature off.!RuntimeFeature.IsDynamicCodeSupported). It defaults tonullunder JIT, where the driver shares its heap with the host, and on thenetstandard2.0build.GC.Collect()empties the heap but leaves the memory committed.GCCollectionMode.Aggressivedecommits it.Measurements
A NativeAOT driver built on the exporter was loaded by a C host, which ran a workload and then released everything. The host never asks for a collection. Numbers are process private bytes; the baseline is about 3 MB.
Without this change, the 64K-row run was sampled every 10 s for 60 s after release, and its memory never went down.
The forced collection took 2–32 ms in these runs.
NativeAOT workaround
AdbcDriverInitnow evaluatesEncoding.UTF8on the caller's thread. Starting the idle timer starts a thread in the driver's runtime, and on Linux that can hit a NativeAOT bug that fails the process fast (dotnet/runtime#136031). The thread runsUTF8Encoding's static constructor while it sets its name, and if it contends on another type's constructor lock at that moment,Lock's statics initialization re-enters the encoding while its static instance is still null. Without the workaround, the AOT fixture tests crashed on every run on Linux.Tests
IdleCollectorTestscovers:nulldelay, and rejecting a negative delay;🤖 Generated with Claude Code