Skip to content

perf(parquet): use typed memo insertion for fixed-width dictionaries - #1312

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-typed-memo-flba-int96
Open

perf(parquet): use typed memo insertion for fixed-width dictionaries#1312
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-typed-memo-flba-int96

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What changed

  • Use BinaryMemoTable.InsertOrGet for FLBA dictionary Put and PutDictionary.
  • Use the same typed insertion path for DictInt96Encoder.Put.
  • Keep empty Put calls as no-ops.
  • Add Int96 dictionary round-trip coverage and FLBA/Int96 benchmarks.

Why

These encoders already use the binary memo table, but still called GetOrInsert through the generic interface for every value. The typed entry point avoids the extra interface conversion. Dictionary order and encoded output stay unchanged.

Benchmark

Apple M1 Pro, macOS arm64, Go 1.26.3. 65,535 values and 100 unique values, median of five runs.

Case Before After Change
FLBA, width 16 4.59 ms 3.09 ms 33% faster
Int96 5.41 ms 3.87 ms 29% faster
FLBA allocations 131,117 65,581 50% fewer
Int96 allocations 196,655 131,118 33% fewer

Allocated bytes also fell by about 42% for FLBA and 33% for Int96.

Tests

  • go test ./... with Arrow and Parquet test data
  • go test ./parquet/internal/encoding/...
  • go test -race ./parquet/internal/encoding/...
  • go vet ./parquet/internal/encoding/...
  • go test -tags noasm ./parquet/internal/encoding/...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant