chore(): Added benchmark tests for parquet and hdf5

[Ichbinkiana]

No description provided.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.56%. Comparing base (34a19f0) to head (359c381).
⚠️ Report is 72 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2509      +/-   ##
============================================
+ Coverage     71.38%   71.56%   +0.17%     
- Complexity    48756    49047     +291     
============================================
  Files          1571     1574       +3     
  Lines        188912   189565     +653     
  Branches      37067    37188     +121     
============================================
+ Hits         134858   135663     +805     
+ Misses        43603    43430     -173     
- Partials      10451    10472      +21     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[janniklinde]

Thanks for the first PR @Ichbinkiana. I am missing concrete results and interpretations. Did you get some insights from the benchmarks that you could share?

Please move those benchmarks to the performance test package and do not use JUnit to run those, because it adds overhead which may distort the results.

[Ichbinkiana]

Summary of changes

Latest commits moved the HDF5 benchmark away from JUNIT and also introduced two HDF5 I/O optimizations as below:

• Added HDF5 benchmarks which by default disabled
• My Benchmark records write/read time, file size, number of files, heap usage, GC stats, parallelism, and correctness status in both Json and CSV format
• Added batched HDF5 writing for both dense and sparse inputs
• Cleaned up HDF5 write to avoid unnecessary boxed Double and used Java primitive types instead.
• Now the write and read are sparse-aware and this can be enabled for sparse matrices with below args:

• Dsysds.hdf5.write.sparse.layout=coo
• Dsysds.hdf5.read.sparse.layout=coo

• Added a fallback so ReaderHDF5Parallel uses the sparse-aware sequential reader when Coordinate layout is enabled for sparse test cases.

---

Results

For dense sequential writing:

• Baseline: ~7.1 s
• After batching: ~4.4 s
• Improvement: ~37% faster

For sparse matrix writing with 125000 x 1000 and 125000 nonzeros:

• Baseline dense-layout file size: ~1.0 GB

• optimized file size: ~3.0 MB

• Baseline sparse write: several seconds

• Optimized sparse write: ~23 ms

• Optimized sparse read: ~18 ms

---

Interpretations/ Reasoning

• Batching improved dense write performance by calling write less and in batches instead of row-by-row.
• The large sparse improvement came from writing only nonzero entries instead of all entries.

Why did i use synthetic Data

• SystemDS HDF5 implementation works with matrices, which only support doubles. Real HDF5 files can contain non-double entries that are outside the current matrix reader/writer scope.
  I also used synthetic dense and sparse matrices to isolate and validate my optimized code behavior under controlled conditions.

[Ichbinkiana]

I'll also push the optimization with results and interpretation for Parquet reader and writer by tomorrow EoD.
And i'll finalise the comparison to external benchmarks before deadline.
Many thanks for your patience and review.

[Ichbinkiana]

Next steps:

• Sharing a summary of results for parquet optimization + interpretation by Thursday EoD.
• Finalise the comparison to external benchmarks before the deadline.

[Ichbinkiana]

HDFS Comparison to external Benchmark

• I extended the HDF5 benchmark with a local external Java baseline using io.jhdf:jhdf:0.11.0.
• I picked jHDF because it is a pure-Java HDF5 library, resolves from Maven Central, and was easy to run locally on my system; it is only for local comparison and is not intended as a required SystemDS dependency.
• The benchmark uses 1,000,000 x 200 FP64 matrices, giving a dense logical size of about 1.6 GB, with -Xmx12g, read/write parallelism 16, and medians over 3 non-warmup repetitions.
• I ran two kinds of tests: dense-layout HDF5 comparisons and a separate sparse version with the optimized code of mine that was sparse-aware.
• The dense-layout comparison is fair because SystemDS and jHDF write/read the same logical dense FP64 dataset with the same shape and dataset name.
• For sparse-like input without spaarse-aware optimization flag, the comparison is also fair as a dense-layout comparison: both SystemDS and jHDF store the logical matrix as a dense HDF5 dataset, including zeros.
• The test aginst sparse-aware optimization version is intentionally separate, because it measures the benefit of sparse-aware SystemDS storage.
• For dense FP64, SystemDS sequential write was ~7.13 s, SystemDS parallel write was ~3.30 s, and jHDF write was ~10.03 s.
• This means SystemDS sequential write was about 1.4× faster than jHDF, and SystemDS parallel write was about 3.0× faster than jHDF.
• For dense reads from the SystemDS sequential file, SystemDS sequential read was ~883 ms and SystemDS parallel read was ~448 ms, giving about 2.0× read speedup.
• An important issue appeared for dense parallel output: the writer produced 11 part files and wrote fast, but the current read path failed to materialize that directory output, so this should not be reported as a successful write/read roundtrip yet.
• For sparse-like dense-layout HDF5, the input had only 1,000,000 nonzeros, having 0.005 sparsity, but the output was still ~1.6 GB, showing that sparse input alone does not necessarily cause sparse HDF5 storage.
• In that dense-layout sparse mode, SystemDS write was ~7.35–7.42 s, while jHDF write was ~10.40 s; parallel write did not help because the output stayed as one file.
• With SystemDS sparse-aware optimization enabled, the same sparse-like matrix dropped from ~1.6 GB to ~24 MB, a ~66.7× file-size reduction.
• In this case write time improved from ~7.35 s dense-layout sparse write to ~112 ms sequential sprse-aware write, and from ~7.42 s dense-layout parallel write to ~106 ms sparse-aware parallel write.
• Sparse-aware read also improved from ~695 ms dense-layout sparse sequential read to ~63 ms COO sequential read.
  jHDF read/materialization used much more heap, around 3.2–3.3 GB, because the benchmark reads into a dense Java array before materializing a MatrixBlock.
• Important observation: write-only; read-back failed. Happened also with baseline.

[Ichbinkiana]

Parquet Results and observations Summary

Summary of Changes

• The Parquet frame reader path was optimized, so instead of converting each vaalue to string first, the reader uses primitive type and reads values through integer, long, float, double, boolean, and etc. This impacted the read path for both sequential and parallel readers.
• Improved write throughput by allowing multiple writer tasks to run concurrently instead of writing one large single file sequentially. (Parallel parquet writer)
• The parallel writer now uses a target part-size configuration rather than writing baased on row/cell count. Runtime settings used in the benchmark were:

sysds.io.parquet.writer.target.part.size.mb = 128
sysds.io.parquet.writer.threads = 4

• Limitation: Issue with sparse-like FP64, the writer can still create relatively small part files.

---

Benchmark Setup

Synthetic Benchmark

My benchmark used three data profiles:

• dense_fp64_only
• mixed_schema
• sparse_like_fp64

Main settings:

• rows = 1,000,000
• cols = 100
• warmup = 2
• measured reps = 3
• writer target part size = 128 MB
• writer thread limit = 4
• JVM heap = -Xmx12g

External Real-Data Benchmark

The external benchmark used, Data properties:

• rows = 2,964,624
• cols = 19
• original external file size = 50 Mb
• original input files = 1

---

Results and Baseline Comparison

1. Dense FP64 Synthetic Data

Read

Case	Files	Average Time
SystemDS sequential read	1	7.74 s
SystemDS parallel read, single-file input	1	7.75 s
Direct parquet-java read	1	8.23 s
SystemDS parallel multipart read-back	6	3.17 s
Direct parquet-java multipart scan	6	8.33 s

Write

Case	Files	Average Time
SystemDS sequential write	1	12.74 s
SystemDS parallel write	6	6.20 s
Direct parquet-java write	1	12.74 s

Interpretation

For dense FP64 data, the single-file SystemDS path and direct parquet-java baseline are close.
The main improvement appears in the optimized parallel path:

• SystemDS parallel write is about 2.1× faster than sequential/direct single-file write.
• SystemDS multipart read-back is about 2.4–2.6× faster than the single-file read paths.
  This shows that the benefit comes from multipart parallel writing and parallel read-back.

---

2. Mixed-Schema Synthetic Data

Read

Case	Files	Average Time
SystemDS sequential read	1	10.21 s
SystemDS parallel read, single-file input	1	10.21 s
Direct parquet-java read	1	10.41 s
SystemDS parallel multipart read-back	6	5.20 s
Direct parquet-java multipart scan	6	10.08 s

Write

Case	Files	Average Time
SystemDS sequential write	1	15.76 s
SystemDS parallel write	6	7.03 s
Direct parquet-java write	1	15.89 s

Interpretation

The mixed-schema profile is slower than dense FP64 because it includes multiple value types, including strings and booleans imo.

Similar to dense profile test:

• SystemDS parallel write is about 2.2× faster than sequential/direct single-file write.
• SystemDS multipart read-back is about 2× faster than single-file read.

---

3. Sparse-Like FP64 Synthetic Data

Read

Case	Files	Average Time
SystemDS sequential read	1	5.90 s
SystemDS parallel read, single-file input	1	6.20 s
Direct parquet-java read	1	distorted by one large outlier
Direct parquet-java read median	1	9.72 s
SystemDS parallel multipart read-back	6	2.56 s
Direct parquet-java multipart scan	6	5.94 s

Write

Case	Files	Average Time
SystemDS sequential write	1	9.09 s
SystemDS parallel write	6	3.65 s
Direct parquet-java write	1	9.21 s

Interpretation

The optimized parallel path still shows a clear improvement:

• SystemDS parallel write is about 2.5× faster than sequential/direct single-file write.
• SystemDS multipart read-back is about 2.3× faster than SystemDS sequential read.

---

4. External Yellow Taxi Real Data

External Single-File Read

Case	Files	Average Time
SystemDS sequential read	1	4.07 s
SystemDS parallel read, single-file input	1	3.93 s
Direct parquet-java read	1	4.18 s

Multipart Read-Back

Case	Files	Average Time
SystemDS parallel multipart read-back	4	1.69 s
Direct parquet-java multipart scan	4	4.67 s

Write from External FrameBlock

Case	Files	Output Size	Average Time
SystemDS sequential write	1	80 Mb	5.71 s
SystemDS parallel write	4	77 Mb	1.79 s
Direct parquet-java write	1	80 Mb	5.41 s

Interpretation

The external Yellow Taxi benchmark shows that the same behavior appears on real Parquet data.

• SystemDS sequential read: ~4.07 s
• SystemDS parallel read: ~3.93 s
• Direct parquet-java read: ~4.18 s
• SystemDS parallel write is about 3× faster than the single-file writers.
• SystemDS parallel multipart read-back is about 2.4× faster than reading the original single file.

---

Overall Interpretation

• NOT that SystemDS is generally faster than Apache parquet-java.
• SystemDS sequential and direct parquet-java single-file paths are usually close.
• The major improvement comes from SystemDS parallel write and SystemDS parallel read-back of multipart output.