[Feature] Spark WAP (write, audit, publish) branch support

[tub]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Problem

When combining batch generated & streaming data, a single large snapshot in a Paimon table can stall Flink checkpointing for a long time.

flowchart TD
Spark --> BatchData[(Batch Data)]
BatchData --> FlinkJoin
EventData[(Event Data)] --> FlinkJoin
FlinkJoin --> JoinedData[(Joined Data)]

Loading

For example, if we rewrite every row in Batch Data in a single snapshot from Spark, the realtime Event Data stalls until the full table is scanned.

Solution

Using Paimon's branching support, we can split the Spark changes up into smaller commits on a staging branch, then fast-forward the main branch at the end of the batch.
This allows us to have smaller, incremental snapshots while keeping the batch update atomic & avoiding partial data if the batch fails half way through.

Solution

This is achievable today, but it would be great to have a similar feature to Iceberg, where you can specify write.wap.enabled and the branch is created & merged automatically.

Prior art from Iceberg: https://iceberg.apache.org/docs/latest/branching/#audit-branch

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!