[Python] Casting a struct with a null-typed field to its own type gives an invalid array

[lmmx]

Describe the bug, including details regarding any error messages, version, and platform.

I was developing a routine to 'compact' my parquet files down to some different row group sizes (etc) with PyArrow, and along the way used a RecordBatch.cast which failed. I got a coding agent (Claude Opus 5.5) to dig into this and bisect it down to a minimal repro and it identified the cause as follows.

---

Repro files:

matrix.py

Details

"""Which casts fail, and the length of the null-typed child in each result.

Run: python matrix.py (needs only pyarrow). Prints one line per case, ok or FAIL.
"""

import pyarrow as pa
import pyarrow.compute as pc

print("pyarrow", pa.__version__)

s = pa.struct([("a", pa.int64()), ("n", pa.null())])
s_int32 = pa.struct([("a", pa.int32()), ("n", pa.null())])
s_no_null = pa.struct([("a", pa.int64()), ("n", pa.int64())])
L = pa.list_

cases = [
    # Lists, not sliced: fails when the list has fewer rows than structs
    ("list [[x]]: 1 row, 1 struct", pa.array([[{"a": 1}]], L(s)), L(s)),
    ("list [[x, x]]: 1 row, 2 structs", pa.array([[{"a": 1}, {"a": 2}]], L(s)), L(s)),
    ("list [[x], [x]]: 2 rows, 2 structs", pa.array([[{"a": 1}], [{"a": 2}]], L(s)), L(s)),
    ("list [None, [x, x]]: 2 rows, 2 structs", pa.array([None, [{"a": 1}, {"a": 2}]], L(s)), L(s)),
    ("list [[x, x], [x]]: 2 rows, 3 structs", pa.array([[{"a": 1}, {"a": 2}], [{"a": 3}]], L(s)), L(s)),
    ("large_list [[x, x]]", pa.array([[{"a": 1}, {"a": 2}]], pa.large_list(s)), pa.large_list(s)),
    ("list_view [[x, x]]", pa.array([[{"a": 1}, {"a": 2}]], pa.list_view(s)), pa.list_view(s)),
    ("list [[x, x]] with n: int64 (control)", pa.array([[{"a": 1}, {"a": 2}]], L(s_no_null)), L(s_no_null)),
    # Casts to another type
    ("list [[x, x]] -> large_list<same struct>", pa.array([[{"a": 1}, {"a": 2}]], L(s)), pa.large_list(s)),
    ("list [[x, x]] -> list<struct<a: int32, n: null>>", pa.array([[{"a": 1}, {"a": 2}]], L(s)), L(s_int32)),
    # Structs, not in a list
    ("struct, 2 rows, not sliced", pa.array([{"a": 0}, {"a": 1}], s), s),
    ("struct, 2 rows, slice(1)", pa.array([{"a": 0}, {"a": 1}], s).slice(1), s),
    ("struct, 2 rows, slice(1) -> struct<a: int32, n: null>", pa.array([{"a": 0}, {"a": 1}], s).slice(1), s_int32),
]


for label, arr, to in cases:
    arr.validate(full=True)
    out = pc.cast(arr, to)
    try:
        out.validate(full=True)
        print(f"ok    {label}")
    except pa.ArrowInvalid as e:
        print(f"FAIL  {label}: input length {len(arr)}, {e}")

# Array.cast and ChunkedArray.cast give the same invalid result, also without raising
arr = pa.array([[{"a": 1}, {"a": 2}]], L(s))
for label, out in [
    ("Array.cast", arr.cast(arr.type)),
    ("ChunkedArray.cast", pa.chunked_array([arr]).cast(arr.type)),
]:
    try:
        out.validate(full=True)
        print(f"ok    {label} of list [[x, x]]")
    except pa.ArrowInvalid as e:
        print(f"FAIL  {label} of list [[x, x]]: {e}")

repro.py

Details

"""Casting a struct with a null-typed field, nested in a list or sliced, to its own type
returns an invalid array, without raising.

Run: python repro.py (needs only pyarrow). Exits non-zero while the bug is present.
"""

import sys

import pyarrow as pa
import pyarrow.compute as pc

print("pyarrow", pa.__version__)

s = pa.struct([("a", pa.int64()), ("n", pa.null())])
failed = False

# 1. A list of 1 row holding 2 structs
arr = pa.array([[{"a": 1}, {"a": 2}]], type=pa.list_(s))
arr.validate(full=True)  # the input is valid
out = pc.cast(arr, arr.type)  # does not raise
print(
    f"list: {len(arr)} row, struct length {len(out.values)},"
    f" child a length {len(out.values.field(0))}, child n length {len(out.values.field(1))}"
)
try:
    out.validate(full=True)
except pa.ArrowInvalid as e:
    print("  BUG: the cast result is invalid:", e)
    failed = True

# 2. A struct array sliced to its second row
arr = pa.array([{"a": 0}, {"a": 1}], type=s).slice(1)
arr.validate(full=True)
out = pc.cast(arr, s)
print(f"sliced struct: offset {out.offset}, length {len(out)}")
try:
    out.validate(full=True)
except pa.ArrowInvalid as e:
    print("  BUG: the cast result is invalid:", e)
    failed = True

sys.exit(1 if failed else 0)

repro_parquet.py

Details

"""The same bug reached through Parquet: batches from ParquetFile.iter_batches are slices,
so RecordBatch.cast to the file's own schema fails once a batch starts past row 0.

Run: python repro_parquet.py (needs only pyarrow). Writes repro.parquet next to itself.
"""

from pathlib import Path

import pyarrow as pa
import pyarrow.parquet as pq

print("pyarrow", pa.__version__)

path = Path(__file__).with_name("repro.parquet")
t = pa.list_(pa.struct([("a", pa.int64()), ("n", pa.null())]))
pq.write_table(pa.table({"c": pa.array([None, [{"a": 1}, {"a": 2}]], type=t)}), path)

f = pq.ParquetFile(path)
for i, batch in enumerate(f.iter_batches(batch_size=1)):
    batch.validate(full=True)  # every batch is valid
    try:
        batch.cast(f.schema_arrow)
        print(f"batch {i}: RecordBatch.cast to the file's schema ok")
    except pa.ArrowInvalid as e:
        print(f"batch {i}: RecordBatch.cast to the file's schema FAILS: {e}")

# Reading the whole file in one batch, nothing is sliced and the cast succeeds
pq.read_table(path).cast(f.schema_arrow)
print("read_table(...).cast(schema) ok")

Description:

Take a struct with a field of type null, nested in a list holding more structs than the list has
rows, or sliced. Cast it to its own type, and the result is invalid. The cast does not raise: the
result's null-typed child comes back with the length of the array passed to cast, not the
struct's own length. It fails validate(), and to_pylist() on it can raise.

• Silent: Array.cast, ChunkedArray.cast and pyarrow.compute.cast return the invalid
  array without raising.
• Raises: RecordBatch.cast raises ArrowInvalid, because it validates its result.

Reproduced on pyarrow 25.0.1, the latest release on PyPI on 2026-09-29, on Linux x86_64 with
Python 3.11. A search of apache/arrow issues on 2026-09-29 found no report of it. The nearest,
GH-50515 ("Respect parent validity bitmap when
casting nested structs with non-nullable fields"), is a different struct-cast bug.

Minimal repro

repro.py, which needs only pyarrow:

import pyarrow as pa

s = pa.struct([("a", pa.int64()), ("n", pa.null())])
arr = pa.array([[{"a": 1}, {"a": 2}]], type=pa.list_(s))  # 1 row, 2 structs
arr.validate(full=True)  # the input is valid

out = arr.cast(arr.type)  # does not raise
len(out.values)           # 2
len(out.values.field(0))  # 2, the int64 child
len(out.values.field(1))  # 1, the null child: the list's length, not the struct's
out.validate(full=True)
# pyarrow.lib.ArrowInvalid: List child array invalid: Invalid: Struct child array #1 has length
# smaller than expected for struct array (1 < 2)

A sliced struct array, not in a list, fails the same way:

arr = pa.array([{"a": 0}, {"a": 1}], type=s).slice(1)
pa.compute.cast(arr, s).validate(full=True)
# pyarrow.lib.ArrowInvalid: Struct child array #1 has length smaller than expected for struct array (1 < 2)

Output of python repro.py (exits 1 while the bug is present):

pyarrow 25.0.1
list: 1 row, struct length 2, child a length 2, child n length 1
  BUG: the cast result is invalid: List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2)
sliced struct: offset 1, length 1
  BUG: the cast result is invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2)

Which casts fail

matrix.py validates each cast's result. Output on 25.0.1:

ok    list [[x]]: 1 row, 1 struct
FAIL  list [[x, x]]: 1 row, 2 structs: input length 1, List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2)
ok    list [[x], [x]]: 2 rows, 2 structs
ok    list [None, [x, x]]: 2 rows, 2 structs
FAIL  list [[x, x], [x]]: 2 rows, 3 structs: input length 2, List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (2 < 3)
FAIL  large_list [[x, x]]: input length 1, List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2)
FAIL  list_view [[x, x]]: input length 1, List-view child array is invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2)
ok    list [[x, x]] with n: int64 (control)
ok    list [[x, x]] -> large_list<same struct>
ok    list [[x, x]] -> list<struct<a: int32, n: null>>
ok    struct, 2 rows, not sliced
FAIL  struct, 2 rows, slice(1): input length 1, Struct child array #1 has length smaller than expected for struct array (1 < 2)
ok    struct, 2 rows, slice(1) -> struct<a: int32, n: null>
FAIL  Array.cast of list [[x, x]]: List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2)
FAIL  ChunkedArray.cast of list [[x, x]]: In chunk 0: Invalid: List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2)

• Every failure is a struct whose length differs from the length of the array passed to cast:
  a list with more structs than rows (list, large_list and list_view alike), or a sliced
  struct. The error's first number is the input's length, the second the length the struct needs.
• Lists whose row count equals their struct count cast correctly, including one with a null row.
• The same struct with the field as int64 casts correctly.
• Casting to a different type, one that changes another field (a to int32) or the list type
  (list to large_list), gives a valid result. Only a cast to the array's own type fails.

Through Parquet

repro_parquet.py shows how the bug arises in ordinary use. It writes
[None, [x, x]] to Parquet and reads it back in 1-row batches. The second batch is [[x, x]], a
list with 1 row and 2 structs. So RecordBatch.cast to the file's own schema fails on it, although
the batch passes validate(full=True):

pyarrow 25.0.1
batch 0: RecordBatch.cast to the file's schema ok
batch 1: RecordBatch.cast to the file's schema FAILS: In column 0: Invalid: List child array invalid: Invalid: Struct child array #1 has length smaller than expected for struct array (1 < 2)
read_table(...).cast(schema) ok

Read whole, the file has 2 rows and 2 structs, and the cast succeeds.

Component(s)

Python

[kartik00052]

I want to work on this issue