You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(csharp): support extension types in the ADO.NET client - #4919
The ADO.NET client now reads and binds the canonical extension types that Apache.Arrow 23 implements.
Extension
.NET value
Binding
arrow.uuid
Guid
DbType.Guid (from a Guid, a string, or 16 bytes)
arrow.bool8
bool
—
arrow.timestamp_with_offset
DateTimeOffset, keeping the original offset
—
arrow.parquet.variant
JSON string or Dictionary/List/scalar, depending on StructBehavior
—
any other extension type
the storage value
—
Previously, any extension column made the reader throw NotSupportedException.
Drivers loaded through the C API
CArrowArrayStreamImporter doesn't take an extension registry, so a native driver's UUID column arrives as fixed_size_binary(16) with only ARROW:extension:name in its field metadata. The reader resolves these fields using the built-in definitions plus anything registered in ExtensionTypeRegistry.Default, then wraps the storage column in the extension array the first time it's read in each batch. The new AdbcDataReader.GetArrowArray(ordinal) returns that wrapped array. Once apache/arrow-dotnet#466 adds a registry overload to the stream importer, this becomes a fallback.
Other changes
IArrowArrayExtensions.ValueAt and GetValueConverter handle GuidArray, Bool8Array, TimestampWithOffsetArray and VariantArray, and fall back to storage for other extension arrays. They also now support fixed_size_binary. Extension types nested in structs and lists are resolved and converted the same way as top-level ones.
Prepare() maps uuid, bool8 and timestamp_with_offset parameters to DbType.Guid, DbType.Boolean and DbType.DateTimeOffset.
DbType.DateTimeOffset parameters are bound as UTC timestamps, which every driver understands; the original offset is not sent.
Shredded variants are opt-in. Unshredded variants need only Apache.Arrow, but reading a shredded one needs Apache.Arrow.Operations, which neither package references. An application that expects shredded results references it and sets ShreddedVariants.Reader = (array, index) => array.GetLogicalVariantValue(index) once; otherwise reading a shredded variant throws NotSupportedException. Variant JSON uses the same writer as structs.
Mocked-driver tests cover each type, both as real extension arrays and as storage with extension metadata, plus user-registered definitions, unknown extension types, parameter binding and Prepare().
A DuckDB test with arrow_lossless_conversion reads UUID and BOOLEAN as Guid and bool and round-trips Guid parameters. DuckDB 1.5.6 can't export VARIANT over Arrow, so variants are only covered by the mocked tests.
Read arrow.uuid, arrow.bool8, arrow.timestamp_with_offset and
arrow.parquet.variant columns as Guid, bool, DateTimeOffset and
JSON/.NET values, resolving them from field metadata when a driver
loaded through the C API returns only the storage type. Other extension
types are read as their storage values. Bind DbType.Guid and
DbType.DateTimeOffset parameters.
Closesapache#4918.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Don't return cached columns from a closed reader.
- Bind DBNull.Value parameters as null for every DbType.
- Resolve extension metadata on fields nested in struct and list types.
- Return variant TimestampNtzNanos values with an unspecified DateTimeKind.
- Work around ListArray.GetSlicedValues returning null for extension values.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Convert variants in the core value conversions so that variants nested
in structs get the same JSON or .NET values as top-level ones. The
Apache.Arrow.Operations dependency moves from the client to the core
library.
- Map extension and fixed-size binary list element types in the client
schema APIs.
- Read 16-byte Guid parameter values in RFC 4122 order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read unshredded variants with Apache.Arrow alone and format them with the
existing struct JSON writer, so neither Apache.Arrow.Adbc nor the client
references Apache.Arrow.Operations. Applications that expect shredded
variants can reference that package and set ShreddedVariants.Reader;
otherwise reading a shredded variant throws NotSupportedException.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LargeList and FixedSizeList extension types lack conversion support
csharp/src/Client/ExtensionTypes.cs:116
These cases resolve extension children inside LargeListType and FixedSizeListType, but the conversion paths only handle ArrowTypeId.List: IArrowArrayExtensions.ValueAt/GetValueConverter do not read either container, and SchemaConverter.ConvertArrowType does not expose their element array type. A C-Data field such as large_list<arrow.uuid> is therefore reported with an unusable schema and yields null/DBNull instead of the nested values. Add matching conversion/schema support (and tests), or do not resolve these container types until they are supported.
The client resolves extension types inside large_list and fixed_size_list
fields, but the value conversions and schema APIs only handled list. Read
both, including when their values are extension arrays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#4920)
## What's changed
`AdbcDataReader.IsDBNull` checked `GetValue(ordinal) == null`, but
`GetValue` returns `DBNull.Value` for null values, so `IsDBNull` always
returned `false`. It now checks for `DBNull`.
It still goes through `GetValue`, so a value supplied by an `OnGetValue`
handler continues to count as not null.
Found while working on #4919.
### Tests
`TestIsDBNull` reads a column whose fourth row is null and checks
`IsDBNull` for every row.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
With the default JsonString behavior, a variant containing NaN or either infinity throws instead of returning a value. ToObject preserves the float/double, but the shared WriteJsonValue helper calls Utf8JsonWriter.WriteNumberValue, which rejects non-finite numbers. This also affects values nested in variant objects or arrays. Handle non-finite values explicitly in the shared writer using a documented JSON representation, preserve the numeric values under Strict, and add regression tests for both behaviors.
Re the review note about NaN and Infinity in variant JSON: variants use the same JSON writer as structs, and on main that writer already throws for non-finite float and double struct fields. Arrow's own VariantJsonWriter also rejects them. Since this predates this PR, it's fixed separately in #4921, which writes them as "NaN", "Infinity" and "-Infinity". Variants will get the same behavior through the shared writer; once #4921 merges, I'll add a variant regression test here.
## What's changed
When a struct is serialized to JSON (the default
`StructBehavior.JsonString`), float and double fields are written with
`Utf8JsonWriter.WriteNumberValue`. That throws `ArgumentException` for
NaN and ±Infinity, so a single non-finite value made its whole row
unreadable.
These values are now written as the strings `"NaN"`, `"Infinity"` and
`"-Infinity"`. That's how System.Text.Json writes them when
`JsonNumberHandling.AllowNamedFloatingPointLiterals` is set.
`StructBehavior.Strict` is unchanged and still returns the actual
`float`/`double` values.
This came up in review of #4919, where variants serialized to JSON use
the same writer.
### Tests
`NonFiniteFloat` and `NonFiniteDouble` check the JSON for NaN, Infinity
and -Infinity. `NonFiniteDouble` also checks that the struct result
keeps the actual NaN value. Without the fix, both throw the
`ArgumentException`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## What's changed
Adds the variant test promised in #4919. Variants converted to JSON use
the same writer as structs, so they already get the behavior from #4921:
NaN and ±Infinity are written as the strings `"NaN"`, `"Infinity"` and
`"-Infinity"`. This test pins that down.
`VariantNonFiniteValues` reads a variant column with a non-finite double
and float inside an object and an array, and a top-level NaN:
- With `StructBehavior.JsonString`, it checks the exact JSON.
- With `StructBehavior.Strict`, it checks that the .NET values keep the
actual NaN and infinity values.
With the #4921 change reverted, the JSON case fails with the original
`ArgumentException`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's changed
The ADO.NET client now reads and binds the canonical extension types that Apache.Arrow 23 implements.
arrow.uuidGuidDbType.Guid(from aGuid, a string, or 16 bytes)arrow.bool8boolarrow.timestamp_with_offsetDateTimeOffset, keeping the original offsetarrow.parquet.variantDictionary/List/scalar, depending onStructBehaviorPreviously, any extension column made the reader throw
NotSupportedException.Drivers loaded through the C API
CArrowArrayStreamImporterdoesn't take an extension registry, so a native driver's UUID column arrives asfixed_size_binary(16)with onlyARROW:extension:namein its field metadata. The reader resolves these fields using the built-in definitions plus anything registered inExtensionTypeRegistry.Default, then wraps the storage column in the extension array the first time it's read in each batch. The newAdbcDataReader.GetArrowArray(ordinal)returns that wrapped array. Once apache/arrow-dotnet#466 adds a registry overload to the stream importer, this becomes a fallback.Other changes
IArrowArrayExtensions.ValueAtandGetValueConverterhandleGuidArray,Bool8Array,TimestampWithOffsetArrayandVariantArray, and fall back to storage for other extension arrays. They also now supportfixed_size_binary. Extension types nested in structs and lists are resolved and converted the same way as top-level ones.Prepare()maps uuid, bool8 and timestamp_with_offset parameters toDbType.Guid,DbType.BooleanandDbType.DateTimeOffset.DbType.DateTimeOffsetparameters are bound as UTC timestamps, which every driver understands; the original offset is not sent.Apache.Arrow, but reading a shredded one needsApache.Arrow.Operations, which neither package references. An application that expects shredded results references it and setsShreddedVariants.Reader = (array, index) => array.GetLogicalVariantValue(index)once; otherwise reading a shredded variant throwsNotSupportedException. Variant JSON uses the same writer as structs.ListArray.GetSlicedValuesreturns null for lists of extension values (ListArray.GetSlicedValues returns null when the values are an ExtensionArray arrow-dotnet#468), so the list conversion slices the values directly in that case.Tests
Prepare().arrow_lossless_conversionreads UUID and BOOLEAN asGuidandbooland round-tripsGuidparameters. DuckDB 1.5.6 can't export VARIANT over Arrow, so variants are only covered by the mocked tests.Closes #4918.
🤖 Generated with Claude Code