fix(arrow/array): Parity with Arrow C++ when reading/writing null to/from JSON

[serramatutu]

Rationale for this change

I raised an issue at the Arrow Community meeting regarding how Arrow Go allows the user to construct invalid record batches by reading JSON with nulls in it even if the schema says it's not null. It is also possible to construct invalid batches by calling AppendNull() in the builders, and currently there is no way to validate that. I was instructed to look at how Arrow C++/PyArrow do it, and replicate it here.

TL;DR:

1. Arrow C++ raises errors when reading JSON data that does not conform to the schema
2. Arrow C++ never JSON-encodes values as null if the parent field is non-nullable

Here's a link to my full investigation, including the code so you can run it yourself: https://github.com/serramatutu/arrow-internal-nulls

What changes are included in this PR?

Rebased on main, and reworked to address @zeroshade's review.

Reading (review point 4, plus the original issue 1). Nullability is now validated recursively, in one place, before anything is appended:

• arrow/array/nullability.go adds validateJSONNullability, which walks a buffered row against the arrow.Field tree. It covers struct fields, the element field of lists / large lists / list views / fixed-size lists, map key and item fields, union children, dictionary values, run-end-encoded values and extension storage types. So ListOfNonNullable(int32) now rejects [1, null], and a missing non-nullable field is rejected at any depth.
• RecordBuilder and StructBuilder keep upstream's streaming append path untouched. They only buffer the row and validate it when the schema declares a non-nullable field somewhere in its type tree (typeHasNonNullableField), so fully nullable schemas take exactly the same path as before this PR.
• Buffering the row also means a rejected row leaves the decoder positioned at the next row, so callers can keep reading. That is not achievable with streaming decode: an error raised inside a nested builder leaves the decoder at an unknown depth, which cannot be drained reliably.
• Roots that are not records or structs are validated in FromJSON, so array.FromJSON(mem, arrow.ListOfNonNullable(...), ...) is checked too.

Rollback (review points 1 and 2). #1113 landed while this PR was open and provides exactly the pre-row checkpoint the review asked for, so the Resize(-1) rollbacks are gone:

• RecordBuilder uses upstream's builderCheckpoint capture/restore as-is.
• StructBuilder now builds its own reusable builderCheckpoint, because it is the root builder for array.FromJSON on a struct type and had no rollback of its own. Resize(-1) is no longer used, and the panic the review reported (columnLenRange returning -1 into child Resize) is gone with it.

Writing (review point 3, plus the original issue 2). Writers now fail instead of emitting a null that this PR's reader would reject:

• RecordToJSON and Struct.MarshalJSON (and therefore RecordBatch.MarshalJSON, which goes through RecordToStructArray) return arrow.ErrInvalid when a non-nullable field holds a null.
• The check honors parent-validity priority, so a null child under a null struct parent is not an error — that matches what the encoders actually emit, since they stop descending at a null parent.
• The previous dtype.Field(i).Nullable && IsNull(i) branches were no-ops (GetOneForMarshal returns nil for a null slot either way), so they were reverted.

I had to change a bunch of tests that implicitly depended on fields being nullable, even if they didn't declare the fields as such.

Scope

MarshalJSON on non-struct array types (a list array marshalled on its own, say) still emits nulls for non-nullable element fields. Reading is validated for those roots; writing is not. Happy to extend it if you'd like it in this PR.

Are these changes tested?

Yes — arrow/array/nullability_test.go covers nested list/map/union/struct element nullability, root-level FromJSON, writer errors, parent-validity priority, decoding the row after a rejected row, and a JSON round trip over a schema mixing nullable and non-nullable fields. There is also a regression test for the reported StructBuilder panic (a malformed nested list value followed by a valid row).

go test ./arrow/... ./parquet/... passes, with the arrow_json_stdlib backend as well.

Performance

Buffering the row turned out to be a large speed-up, not a cost. go test ./arrow/array -run '^$' -bench 'BenchmarkRecordFromJSON/Size_1000$' -benchtime=3x -count=2 -benchmem (that benchmark's schema declares all five fields non-nullable, so it takes the new path):

	ns/op	B/op	allocs/op
main (b3dacd2a)	7,769,030,778 / 7,836,212,958	6,141,082	16,130
this branch	26,246,764 / 25,082,431	29,142,202	53,071

That is ~305x faster for ~4.7x the allocated bytes. A CPU profile of the old path attributes 98% of samples to runtime.memmove under goccy/go-json's decodeEscapeString ← (*Stream).Token: decoding string tokens one at a time straight off the document decoder is quadratic in the remaining buffer, so it gets worse the larger the document. Decoding each row from its own small json.RawMessage avoids that entirely.

The flip side: a schema with no non-nullable field anywhere still takes the streaming path and so keeps the quadratic behaviour (~7.8 s/op on the same data). I kept the fast path opt-in by nullability to keep this PR's behaviour change scoped — an unconditional row buffer would fix it for every schema, but it would also force UseNumber on caller-supplied decoders in UnmarshalOne. Happy to do that here, or file it separately, whichever you prefer.

Are there any user-facing changes?

Yes, two breaking changes:

1. The JSON decoder is stricter: null or a missing value for a non-nullable field is now an error, at any nesting depth. This breaks callers relying on the decoder's leniency.
2. The JSON encoders now return an error for arrays that hold a null in a non-nullable field, instead of silently writing null.

[zeroshade]

Is it possible for us to do this without a breaking change? I'd rather avoid the breaking change of changing a public API method if at all possible.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

arrow/array/record.go:424

• The UnmarshalOne docstring says it “receives an already-configured json.Decoder, so options such as UseNumber set by the caller are honored”, but the implementation now buffers values as RawMessage and re-decodes them with a fresh decoder that always calls UseNumber. This no longer honors caller decoder configuration (and in particular forces UseNumber even if the caller didn’t enable it).

// UnmarshalOne reads one row (a JSON object) from the supplied decoder and
// appends a value to each field in the RecordBuilder.
//
// Unlike UnmarshalJSON, this method receives an already-configured
// json.Decoder, so options such as UseNumber set by the caller are honored
// for nested field decoding. This is critical for preserving large integer
// values (>2^53) that cannot be represented exactly as float64.

[zeroshade]

@serramatutu are you still working on this?

[serramatutu]

@zeroshade I haven't had the time to push this forward since I last worked on it. Currently on PTO, will be back in July.

[serramatutu]

@zeroshade I addressed all the issues in new commits and rebased onto main. I don't have permissions to request another Copilot review. I did run a Claude Code review locally and it says it's good.

There is one issue that I found where b.Resize(-1) can leak memory which is causing a test to break. This was a pre-existing bug. I'm making a PR to fix that here. Will need to rebase onto that to make that test pass :)

[serramatutu]

@zeroshade I've rebased this onto the latest main and the tests are now passing :)

[zeroshade]

The stricter top-level nullability checks are directionally correct, but four correctness issues remain:

1. RecordBuilder rollback can retain rejected nested rows when top-level lengths remain equal.
2. StructBuilder rollback can pass -1 into child builders and panic.
3. Record and struct writers still emit non-nullable nulls that the new readers reject, breaking round trips.
4. Nested field metadata remains unenforced; for example, non-nullable list elements still accept null.

See the inline comments on record.go, struct.go, and util.go.

The existing array tests pass, as do focused race tests and all 23 CI checks. The PR is currently conflicting with main and will also require a rebase.

---

This review was drafted by an AI-assisted tool and confirmed by an Apache Arrow Go maintainer. After you've addressed the points above and pushed an update, an Apache Arrow Go maintainer — a real person — will take the next look at the PR. The findings cite the project's review criteria; if you think one of them is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how Apache Arrow Go handles maintainer review: CONTRIBUTING.md.

[zeroshade]

This still serializes a null in a non-nullable column: when the condition is false, the column’s GetOneForMarshal returns nil. The resulting {"x":null} is then rejected by this PR’s reader, breaking writer/reader round trips.

Please define and implement the intended behavior for invalid non-nullable data—likely return an encoding error—and add a round-trip regression. The equivalent struct path has the same issue.

[serramatutu]

Per this: https://github.com/serramatutu/arrow-internal-nulls

In C++, it looks like the general approach is "validate all data in", then assume it's OK/consistent when "writing out". I.e it is an invariant that the data is correct when writing out.

If the user has messed around with the data:

• by tweaking memory directly: they need to ensure their code respects invariants OR call ValidateFull
• by using a public API: the API will always leave the array in a consistent internal state

This is my rationale for calling GetOneForMarshal() here: we assume the data is correct, if it's not then it's UB.

If we're very pedantic about this, we should just revert this change and use c.GetOneForMarshal(i) as we should assume IsNull(i) will always return false if the data is consistent with the schema. This is what C++ does actually, so here we're doing even more validation by checking rec.Schema().Field(j).Nullable. If this is writing wrong data that is failing validation upstream, I think we should fix the thing that allows the bad data to be constructed in the first place, not validate here while writing.

Sources:

• JsonWriter::WriteArray uses an ArrayWriter: https://github.com/apache/arrow/blob/b12755a1cc41ee5e9b1874ece3bcc92d6d3e3e78/cpp/src/arrow/integration/json_internal.cc#L2079
• ArrayWriter::WriteDataValues does not check for the schema at all, it just calls IsValid() directly: https://github.com/apache/arrow/blob/b12755a1cc41ee5e9b1874ece3bcc92d6d3e3e78/cpp/src/arrow/integration/json_internal.cc#L505

[zeroshade]

The checkpoint-based rollback changes resolve the prior record/struct contamination and panic findings. Two previously reported correctness gaps remain: nested nullability metadata is still not enforced, and record/struct serialization still emits nulls for non-nullable fields that the updated readers reject. Targeted, race, repeated rollback, impacted-package, formatting, vet, and diff checks pass. The PR is also still conflicting with main and currently has no reported CI checks.

[zeroshade]

Blocking: This still delegates the nested value directly to the child builder, which has no access to the containing nested arrow.Field.Nullable metadata. I reproduced RecordFromJSON accepting {"x":[1,null]} for a schema whose field type is ListOfNonNullable(int32). Please enforce nested nullability recursively (including corresponding nested container types) and add a regression that expects this input to fail.

[serramatutu]

Sorry, I was still fixing this yesterday but it got late...

I just fixed it for list, map, union and REE, and added tests for all of those. I also made all of them have individual checkpointing so if users use e.g a standalone ListBuilder with a non-nullable field, it'll still keep a consistent state.

Note that I had to add a new NewListBuilderWithField (also for list view and large lists) so that the builder can know the schema it's trying to build.

[serramatutu]

@zeroshade regarding the previous comments:

What I fixed

Fully nullable records still take the buffering/re-decoding path, with roughly 1.5× slower decoding and 2× allocated bytes than the base implementation in a 1,000-row probe.

I have fixed the perf issue, now the row buffering only happens at the top-level struct, and we avoid re-decoding nested rows. Had to add some plumbing to Builder for that. Look at commit Don't buffer rows on child struct builders

What I didn't fix (on purpose)

arrow/array/builder.go:451 unconditionally enables UseNumber() while replaying buffered rows, overriding the caller's decoder semantics.

I kept it like this as I assume this is intender behavior for 2 reasons:

• WithUseNumber() is deprecated and the docs state it's a no-op and always enabled by default. So I'm always enabling it.
• Not enabling it makes the JSON decoder lenient as it can decode floats like 1.5 into integers by truncating, silently violating the schema. There was a test that asserted that lenient behavior (TestUnionBuilderUnmarshalOnePreservesDecoderConfiguration), which I have fixed (see my other comment above on that test)

arrow/array/builder.go:515-521 skips unknown fields before recording seen keys, so duplicate unknown keys are silently accepted.

Also intended behavior. I added a comment that the decoder behaves like the default C++ decoder, with ParseOptions::Ignore, which ignores unknown keys. I added it as a TODO comment to possibly support ParseOptions with Error and InferType as well for strict/lenient parsing.

[zeroshade]

I have fixed the perf issue, now the row buffering only happens at the top-level struct, and we avoid re-decoding nested rows. Had to add some plumbing to Builder for that. Look at commit Don't buffer rows on child struct builders

Can we add a benchmark for this (and post the main vs this PR result of the benchmark just to confirm)?

What I didn't fix (on purpose)

That's fair, this logic is sound and makes sense to me. I'm good with it.

Also intended behavior. I added a comment that the decoder behaves like the default C++ decoder, with ParseOptions::Ignore, which ignores unknown keys.

Gotcha, this is fine then. Following the default behavior of the c++ decoder is more desirable anyways. Thanks!

So you just gotta fix the linting issue and then we're all good here :)

Thanks for putting up with the reviews!

[serramatutu]

Benchmark

Results for BenchmarkJSONReaderRowShape and BenchmarkJSONReaderEscapedString in arrow/array/json_reader_test.go, comparing this branch against main (227f6e3e).

Method

I ran it with Go 1.27.0, on an Apple M3 Pro. Every figure below is the median of six
runs of 20 iterations for RowShape and 100 iterations for EscapedString.

go test ./arrow/array/ -run XXX -bench BenchmarkJSONReaderRowShape -benchtime 20x -count 6
go test ./arrow/array/ -run XXX -bench BenchmarkJSONReaderEscapedString -benchtime 100x -count 6

The test cases

BenchmarkJSONReaderRowShape

shape	rows	fields per row	document size
small	10,000	3	≈404 KB
medium	1,000	10	≈119 KB
large	1,000	100	≈1.3 MB

All three shapes have 10,000 rows, so fields per row is the only variable. Every shape runs twice, once with all fields non-nullable and once with all
fields nullable.

BenchmarkJSONReaderEscapedString

10,000 rows of one string field each. The field contains one \n escaped character. Runs twice, once with nullable and the other non-nullable.

Results

BenchmarkJSONReaderRowShape

TL;DR: the new encoder is slower on small field counts, but it's faster for lots of fields. Nullability has no impact on performance.

Non-nullable fields:

shape		ns/op	B/op	allocs/op
small	base	4,707,431	3,210,491	100,111
	branch	7,080,244	10,011,996	170,054
		1.50× slower	3.12×	1.70×
medium	base	18,533,913	14,962,940	340,306
	branch	19,983,839	16,638,869	379,430
		1.08× slower	1.11×	1.11×
large	base	205,269,110	160,635,683	3,092,727
	branch	185,001,977	141,320,030	3,097,500
		1.11× faster	0.88×	1.00×

Nullable fields:

shape		ns/op	B/op	allocs/op
small	base	4,697,328	3,210,491	100,111
	branch	6,720,561	10,011,997	170,054
		1.43× slower	3.12×	1.70×
medium	base	18,532,919	14,962,933	340,306
	branch	18,884,480	16,638,866	379,430
		1.02× slower	1.11×	1.11×
large	base	206,265,585	160,635,687	3,092,727
	branch	176,193,939	141,320,040	3,097,500
		1.17× faster	0.88×	1.00×

BenchmarkJSONReaderEscapedString

TL;DR: the row buffering approach makes it faster because we're avoiding the big shift-left of the entire JSON document that goccy does when it finds an escaped character.

	ns/op	MB/s	B/op	allocs/op
main, non-nullable	9,751,840	19.37	1,532,712	50,064
branch, non-nullable	3,830,202	49.34	8,087,729	110,000
	2.55× faster		5.28×	2.20×
main, nullable	9,752,524	19.38	1,532,712	50,064
branch, nullable	3,758,762	50.28	8,087,729	110,000
	2.59× faster		5.28×	2.20×