[fix](iceberg) Harden write lifecycle and cleanup - #66348
Conversation
Problem Summary: Iceberg writes could bypass cleanup, miss overwrite conflicts, undercount partition sorter memory, hide close errors, exceed report limits, or leave incomplete remote files. Partition dispatch also allocated full-block filters for every partition. Solution: Unify failure handling, preserve the durable commit point, validate overwrite conflicts, account for all active sort writers, propagate close failures, bound commit metadata, abort incomplete uploads, add orphan cleanup, and use compact row permutations.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Requesting changes: the static review found nine actionable issues (six P1, three P2), including three paths that can delete or reference missing Iceberg data, a concurrent use-after-free, an aggregate Thrift limit bypass, pathological spill reservation, and two incomplete object-storage cleanup paths.
Critical checkpoint conclusions:
- Goal and proof: The PR partially hardens write/cleanup/OCC behavior, and the overwrite and external-insert lifecycle changes are credible, but the nine inline findings prevent the stated lifecycle and cleanup goals from being met. Added tests prove only narrow single-state, AWS-abort, permutation, and executor cases.
- Scope and clarity: The edits are generally focused, but the new destructive action reimplements Iceberg safety policy incompletely, and the provider-neutral abort surface is implemented with AWS-specific state.
- Concurrency: The async writer publishes active-writer snapshots while a separately scheduled revocation task reads them. Per-writer
_sorter_mutexlocking is consistent and no deadlock was found, but it cannot protect the snapshot container because the range-for drops its temporary owner before iteration (MAIN-BE-003). - Lifecycle: Sort-writer close/cancel/deferred cleanup and external commit/listener ordering were traced end to end and otherwise preserve error ownership. S3/Azure abort lifecycle is incomplete: a rejected abort is never retried, and Azure reports successful abandonment without provider cleanup (MAIN-IO-001/002).
- Configuration: No new configuration was added. Existing
thrift_max_message_sizeand S3 limiter settings are read dynamically, but their consumers do not enforce the correct aggregate/cleanup contracts. - Compatibility: No storage-format or wire-field change was introduced. Default virtual methods preserve source-level parallel implementations, but that default is exactly why Azure silently lacks the new cleanup behavior.
- Parallel paths: Static/dynamic/branch/empty overwrites, data/delete writers, sync/async/destructor close, S3/Azure providers, and all insert executors were checked. The accepted findings identify the paths where parity is missing.
- Conditional checks: Existing overwrite conflict conditions are sound. The orphan action is missing the required
gc.enabledfence, safe retention interval, canonical URI identity handling, and version-hint reachability. - Test coverage: Coverage is not comprehensive. Missing cases include multi-task aggregate reports, concurrent snapshot publication/traversal, many-partition reserve accounting, hard-limit abort rejection, Azure staged-block abort, and execution-level orphan deletion tests for GC, retention, URI aliases, and the version hint.
- Test results: The new assertions are logically consistent within their narrow scopes; no result files changed. Builds and tests were intentionally not run because the authoritative review prompt forbids them, so this conclusion is static-only.
- Observability: Existing timers/logs cover writer/spill/commit paths, and abort failures are warned. No additional standalone observability defect was found, but logging does not substitute for retained cleanup ownership or retry.
- Transaction and persistence: No Doris EditLog state is added. Iceberg overwrite OCC anchors for main/branch/empty/static/dynamic cases are sound, and the durable external commit marker precedes refresh/listener suppression. The orphan action can nevertheless race a concurrent external commit (MAIN-FE-004).
- Data writes and atomicity: The aggregate-report failure can strand uncommitted objects; unsafe orphan deletion can corrupt shared, aliased, or concurrent writes; and multipart cleanup can leak provider state. These are blocking write-lifecycle defects.
- FE/BE propagation: No new FE-BE variable or Thrift field is introduced. Existing commit metadata is propagated on every producer path, but its budget is enforced per task rather than on the final aggregated RPC (MAIN-BE-001).
- Performance: Compact permutation dispatch is semantically sound, and bounded snapshot-copy cost is not independently material. Summing one full-batch reserve estimate per active partition can request roughly 128 blocks and repeatedly spill tiny partitions (MAIN-BE-002).
- Other issues: Error precedence, close idempotence, spill cleanup, overwrite validation, permission checks, location containment, and post-commit listener behavior produced no additional substantiated defects.
User focus: review_focus.txt contained no additional focus guidance, so the whole PR was reviewed without a narrower focus.
Review status: complete and converged in round 3. All final full-scope and risk-focused reviewers returned NO_NEW_VALUABLE_FINDINGS for this exact comment set. Static review only; no builds or tests were run.
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 29107 ms |
TPC-DS: Total hot run time: 169793 ms |
ClickBench: Total hot run time: 23.92 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: None Related PR: apache#66348 Problem Summary: Parallel Iceberg sink tasks could exceed the shared report limit, partition writers could over-reserve memory or outlive a temporary snapshot, failed multipart uploads could skip mandatory cleanup, and orphan-file deletion lacked Iceberg safety fences. Share the report budget, retain writer snapshots, bound reservations, make abort cleanup provider-aware, and enforce safe canonical orphan detection. ### Release note Improve Iceberg write failure cleanup and orphan-file deletion safety. ### Check List (For Author) - Test: Unit Test - BE ASAN focused tests with Azure support enabled - Full fe-connector-iceberg unit test suite - Behavior changed: Yes, unsafe orphan cutoffs and GC-disabled deletion are rejected, and failed multipart uploads are cleaned up reliably. - Does this need documentation: No
|
run buildall |
|
/review |
There was a problem hiding this comment.
Request changes: two distinct issues remain after the current fixes: Azure abort can delete a valid pre-existing Put Blob object, and orphan-file reachability repeatedly scans inherited manifests with quadratic cost. I did not duplicate the existing review threads.
Critical checkpoint conclusions:
- Goal and proof: The PR broadly hardens Iceberg write/cleanup lifecycle, OCC, reporting limits, and orphan cleanup, but the Azure data-preservation goal is not yet met. The added tests cover nominal cleanup and OCC paths, not the two cases below.
- Scope and clarity: The changes are cohesive for the stated lifecycle work. The orphan action's hand-built reachability scan is the one area that should reuse Iceberg's unique-manifest approach.
- Concurrency: Async writer publication is protected by retained shared snapshots and per-sorter locking; the shared report budget has a consistent lock order. Azure's committed-block test is not a valid ownership fence for an existing object.
- Lifecycle: FE before-execution cleanup and post-commit error boundaries are sound, as are the reviewed nested writer close paths. Azure abort still destroys state it does not own.
- Configuration: No new configuration behavior requires dynamic-reload handling.
- Compatibility: No FE-BE or storage-format incompatibility was found. The changed Azure block-ID format has a bounded retry impact and did not survive as a separate finding.
- Parallel paths: S3, Azure, single-shot, multipart, committer, overwrite, branch, static/dynamic overwrite, and delete/merge paths were traced; the surviving provider-specific issue is inline.
- Conditional checks: GC, retention, URI containment, identity ambiguity, and overwrite validation checks are fail-closed.
CommittedBlocks.empty()is the invalid special case. - Test coverage: Unit coverage is substantial but misses preservation of an existing Put Blob target and an append-heavy multi-snapshot reachability scan. No end-to-end result covers either.
- Test results: No tests or builds were run because the authoritative review task is static-only; no generated result files were changed.
- Observability: Existing status/log paths are adequate for the reviewed failures; no additional metric was required for a correctness conclusion.
- Transactions and persistence: Iceberg overwrite OCC, including empty-table and branch anchors, is sound. No EditLog or master-failover persistence change is involved.
- Data writes: Commit/report cleanup ownership is sound outside the Azure abort branch; that branch can lose previously committed data and blocks approval.
- FE-BE variables: No new cross-version transmitted variable or missing propagation path was found.
- Performance: Per-snapshot
planFiles()causes the P2 quadratic manifest-read issue below; no other distinct hot-path issue survived duplicate suppression. - Other issues: The sorter aggregate-reservation concern remains covered by the existing thread, and cleanup admission/terminal-state concerns remain covered by the existing Azure/rate-limit threads, so they are not repeated here.
User focus: no additional user-provided focus was present; the full PR was reviewed.
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 29004 ms |
TPC-DS: Total hot run time: 169560 ms |
ClickBench: Total hot run time: 23.98 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
|
/review |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
There was a problem hiding this comment.
Request changes: one rolling-upgrade compatibility bug remains in Azure multipart uploads. A failed pre-upgrade upload can leave 4-byte uncommitted block IDs, while the new FE/BE path stages 47-byte IDs; Azure rejects the first new Put Block until those legacy blocks are garbage-collected. The inline comment contains the concrete scenario, fix direction, and missing regression.
Review scope: I reviewed the full 65-file authoritative patch and its FE/BE, Iceberg, filesystem, pipeline, reporting, cleanup, and test call paths. review_focus.txt supplied no additional focus, so the whole PR was reviewed. Existing live threads were treated as the duplicate baseline; no already-reported issue is repeated here.
Part 1.3 checkpoint conclusions:
- Goal and proof: The patch substantially hardens Iceberg write OCC, reporting, memory reservation, publication, and cleanup, but it does not yet accomplish rolling-safe Azure retry because of the blocker above. The new Azure tests cover new-format IDs, not a retained legacy block followed by a new upload.
- Scope and clarity: The patch is broad because it closes interacting FE/BE lifecycle gaps across 65 files. The individual changes generally follow existing abstractions; I found no smaller standalone defect beyond the inline compatibility issue and existing live threads.
- Concurrency: I traced pipeline-task report aggregation, async-writer queue handoff, sorter snapshots/reservations, and same-key Azure writers. Shared report-budget and reservation ownership are consistent on current head; no new deadlock, lock-order, or token-release bug remains beyond existing comments.
- Lifecycle: Reservation tokens, async close/abort, commit-data ownership, connector/TCCL scope, post-commit hooks, and orphan-file cleanup were followed through success, failure, cancellation, and crash-adjacent paths. The accepted Azure legacy-block namespace transition is the only new lifecycle failure.
- Configuration: No new user-facing configuration knob is introduced. The existing coordinator Thrift limit is propagated to BE for the report budget; no separate dynamic-configuration defect was established.
- Compatibility: Optional Thrift fields, FE/BE completion fallbacks, exact block-ID transport, and Iceberg snapshot pins were reviewed. The 4-byte-to-47-byte Azure block-ID transition is not compatible with legacy uncommitted state and blocks approval.
- Parallel paths: Both FE and BE Azure multipart implementations, the shared object-storage writer/abort layer, S3 behavior, and deferred FE completion were checked. No missing parallel-path update remains outside the accepted finding and existing threads.
- Special conditions: Explicit-empty snapshots, exact-versus-legacy block-ID selection, ownership fences, retention cutoffs, rate limits, and terminal-report gating were checked against upstream guarantees. No additional unsafe condition was found.
- Test coverage: The patch adds targeted BE and FE unit coverage for its principal changes, but the Azure suite lacks the negative rolling-upgrade case that seeds a legacy uncommitted block and retries the same key with the new format. End-to-end behavior for that case is therefore unproved.
- Test results: This review was static-only as required; I did not run builds or tests. I inspected the added test code and expected assertions, but do not independently claim execution results.
- Observability: Error propagation, cleanup logs, report rejection, and provider exception context were reviewed. Existing diagnostics are adequate for the changed paths; no distinct logging or metrics blocker was found.
- Transaction and persistence: Iceberg statement pins, branch ancestry/OCC checks, commit/rollback boundaries, cache refresh, and edit-log-adjacent post-commit behavior were traced. Durable publication precedes best-effort hooks and no new failover/persistence hole was established.
- Data writes: Exact snapshot fencing and atomic Iceberg/Azure publication paths otherwise preserve committed/uncommitted visibility. The accepted issue causes valid post-upgrade Azure retries to fail before publication; it does not rely on speculative corruption.
- FE/BE variables: The new report limit, snapshot/write context, upload ID, and exact staged block ID are carried through the relevant send/receive paths. Rolling handling is otherwise fail-closed, but the Azure ID-length transition remains unsupported.
- Performance: Shared report budgeting, bounded orphan enumeration, streamed deletion candidates, and memory reservations were reviewed. The remaining full-batch-per-sorter over-reservation is already covered by live comment 3695317514; no new nonduplicate performance issue remains.
- Other issues: Security review was not requested. After duplicate suppression and two converged review rounds, no other valuable current-head finding remains.
| BlockBlobClient blockBlobClient = getClient().getBlobContainerClient(uri.container()) | ||
| .getBlobClient(uri.key()).getBlockBlobClient(); | ||
| String blockId = toBlockId(partNum); | ||
| String blockId = multipartBlockId(uploadId, partNum); |
There was a problem hiding this comment.
[P1] Preserve compatibility with legacy uncommitted block IDs. A failed or crashed pre-upgrade upload can leave 4-byte decoded block IDs on this blob (the old toBlockId format), while this retry now stages a 47-byte decoded UUID:part ID. Azure rejects Put Block with HTTP 400 when its ID length differs from existing uncommitted blocks, so a valid same-key retry remains unusable until service GC. This happens before the exact-ID completion safeguards and is not covered by the new-ID-only tests. Please keep a compatible decoded length during the rolling window or isolate the new upload from the legacy block namespace, and seed a legacy residual block in the regression test. Azure contract: https://learn.microsoft.com/en-us/rest/api/storageservices/put-block
|
run buildall |
|
/review |
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
Codex automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
924060929
left a comment
924060929
left a comment
There was a problem hiding this comment.
Requesting changes for one remaining Azure multipart correctness blocker. Matching the legacy four-byte block-ID length fixes the rolling-upgrade 400 response, but hashing each upload UUID into 32 bits no longer guarantees same-target writer isolation and can publish another writer’s bytes.
| for (byte value : uploadId.getBytes(StandardCharsets.UTF_8)) { | ||
| uploadNamespace = (uploadNamespace ^ (value & 0xFF)) * 0x01000193; | ||
| } | ||
| int namespacedPart = uploadNamespace + partNum; |
There was a problem hiding this comment.
[P1] Do not compress writer identity into a colliding 32-bit block namespace
The four-byte length fixes compatibility with legacy residual blocks, but FNV-1a-32(uploadId) + partNum cannot isolate concurrent writers. For example, these two valid UUIDs both hash to 541563024 under this exact implementation:
c8e3f981-50d5-46d3-8c3a-2d2723f686eedd3cf642-9f55-4b6b-957d-0616633116ea
For part 1 both therefore generate kZhHIA==. If writer A stages that ID, writer B stages the same ID, and A commits its block list, Azure commits B’s last-staged bytes while A reports success. Partial overlap is also possible because each upload occupies the additive range [hash + 1, hash + partCount]. This can publish a mixed/corrupt data file whose Iceberg metadata belongs to another writer.
The current test only proves upload-a and upload-b differ; it cannot establish uniqueness in a 32-bit space. Azure documents that the last upload of a repeated uncommitted block ID is what a subsequent block-list commit selects: https://learn.microsoft.com/en-us/rest/api/storageservices/put-block
Please use target-level coordination such as a lease, or another publication design that preserves both legacy-length compatibility and deterministic writer isolation. Changing the 32-bit hash function cannot fix the namespace-size limitation.
|
run buildall |
|
/review |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
There was a problem hiding this comment.
Static review of the full 66-file diff found two new blocking correctness issues, annotated inline.
Checkpoint conclusions:
- Goal and tests: the intended hardening is not yet proved end to end. Queued sorter reservations are based on stale state, and old-BE/new-FE Azure compatibility can publish Hive metadata before object publication. Added tests do not exercise either actual ordering or payload.
- Scope and architecture: the changes are generally focused and reuse existing writer and transaction abstractions; the remaining failures occur at async-queue and rolling-protocol boundaries.
- Concurrency and lifecycle: active-writer snapshots, per-sorter locking, transfer of the dequeued reservation, and the new-version Azure lease flow are coherent. Existing threads already cover terminal token cleanup, retained sorted memory, cross-task cleanup, the FE output-stream lease bypass, and provider abort retry, so those were not duplicated.
- Compatibility and configuration: the optional Thrift field is wire-compatible, while the framed receiver-limit gap remains tracked in an existing thread. New-BE/old-FE Azure completion fails closed; old-BE/new-FE does not.
- Data integrity and persistence: Iceberg OCC and explicit-empty improvements are coherent, while the selector-canonicalization and shared-root orphan-cleanup concerns remain tracked in existing threads. The two new P1 issues still block safely completing this patch.
- Tests, observability, and performance: this was a static-only review, so no build or tests were run. Existing tests miss multi-queued sorter-state advancement and the real base-BE payload with no pending upload.
No additional user review focus was provided; the whole PR was reviewed.
| per_partition_reservations.reserve(active_writers->size()); | ||
| for (const auto& writer : *active_writers) { | ||
| if (auto* sort_writer = dynamic_cast<VIcebergSortWriter*>(writer.get())) { | ||
| auto reservation = sort_writer->get_reserve_mem_size_components(state, eos); |
There was a problem hiding this comment.
[P1] Account for sorter state already queued ahead of this block
This estimate samples the async writer's currently published sorter state, but its queue accepts three blocks. If the consumer is stalled, all three producer-side calls can see the same pre-queue rows/capacity; after earlier queued blocks append, a later token can cross the 85% growth or sort threshold without reserving that allocation. The earlier tokens have already been shrunk by then, so transferring each token across threads still lets this sink exceed the hard memory limit instead of spilling. Please account for projected sorter state from blocks already queued ahead (or allow only one outstanding block), and add a deterministic multi-queued boundary test.
| .allMatch(part -> part.etag() != null && !part.etag().isEmpty()); | ||
| for (UploadPartResult part : sorted) { | ||
| blockIds.add(toBlockId(part.partNumber())); | ||
| // Missing IDs identify an older BE upload, whose blocks use the legacy namespace. |
There was a problem hiding this comment.
[P1] Reject old-BE Azure writes before publishing Hive metadata
The target-branch Azure BE returns success from multipart initiation without an upload ID. Because Hive defers completion, its blocks stay uncommitted, and VHivePartitionWriter omits the entire pending-upload record when that ID is empty, so this fallback is never invoked. The FE can then commit the reported file name while the blob is still invisible; if an old-BE update is merged first, HiveWriteUtils also drops later new-BE pending tokens. Please require one Azure completion record per reported file or capability-gate deferred Azure writes during rolling upgrade, and test an actual base-BE update with no pending list plus the mixed merge order, rather than a fabricated ID with empty ETags. Azure publishes staged blocks only through Put Block List.
|
run buildall |
|
/review |
|
run buildall |
|
/review |
FE UT Coverage ReportIncrement line coverage |
There was a problem hiding this comment.
Automated review: changes requested.
I found six correctness/resource-safety issues: five P1 and one P2. Three are attached inline; three narrower follow-ups are posted on the existing Azure rolling-compatibility, orphan-cleanup ownership, and report-size discussions.
Critical checkpoint conclusions:
- Goal/fitness: The PR aims to harden external-table write lifecycle, commit validation, cleanup, memory admission, report sizing, and Azure compatibility. It makes substantial progress, but MAIN-F1 through MAIN-F6 show the goal is not yet met on important failure/race paths.
- Scope/clarity: All 71 changed files form a coherent hardening change; I found no unrelated edits. The main paths are reasonably factored and documented, but the six edge contracts below remain incomplete.
- Concurrency/locking: I traced pipeline producer vs async writer, spill/revoke/final merge, concurrent Iceberg commits, and Azure same-key writers. Reservation ownership and lock ordering are mostly coherent and I found no new deadlock, but the EOS fan-in is under-reserved (F1) and async-open failure retains a queued token (F2).
- Lifecycle/static initialization: Multipart upload, queued-token, writer cleanup, transaction, and action lifecycles were traced through success, cancellation, and failures. F2 and F6 leave resource ownership at the wrong boundary. No cross-TU static/global initialization problem was introduced.
- Configuration: The new session/query-option budget path has defaults and FE-to-BE propagation. No dynamic-config-specific defect was found; the report budget itself is incomplete because it excludes the variable envelope (F6).
- Compatibility: Old-BE/new-FE, new-BE/old-FE, optional Thrift fields, and Azure four-byte block-ID behavior were checked. The finite lease stops fencing an old FE after expiry (F4), so rolling compatibility is not safe.
- Parallel/special paths: Sorted and unsorted table writes, delete writes, full/static/dynamic overwrite, RowDelta/branch paths, S3/Azure completion, and custom/default orphan scan scopes were checked. F3 and F5 remain uncovered special cases; no additional parallel-path defect survived verification.
- Tests/results: The PR adds broad BE and FE unit coverage, but it misses the empty-sorter multi-run EOS merge, enqueue-before-open-failure, empty-head check-to-refresh race, expired-lease rolling completion, same-suffix shared-root cleanup, and full near-limit report-envelope cases. Tests were not run because this review environment explicitly permits static review only.
- Observability: Existing status propagation, logs, and profile accounting cover most touched paths; I found no separate metrics-only blocker. F6 is more severe than observability because the final report may fail after cleanup ownership is released.
- Transactions/persistence/failover: No Doris EditLog schema/replay change was introduced. Connector OCC, metadata publication, cleanup, and failure boundaries were traced; F3, F4, and F5 violate commit/ownership fencing.
- Data-write atomicity/crash safety: Multipart publication, Iceberg commit, abort, and orphan cleanup were checked. The accepted findings can overwrite concurrent data, republish stale Azure bytes, delete another table's live files, or strand successfully written files.
- FE-BE variables/protocol: The new query option and exact block IDs are carried through the touched send/receive paths, but rolling behavior (F4) and whole-report sizing (F6) remain unsafe. No other stable mismatch was found.
- Performance/other: Reachability caps, hash scans, admission bookkeeping, and merge costs were reviewed. F1 permits final merge allocation beyond the admitted estimate; no additional stable performance or Nereids semantic issue emerged.
User focus: no additional focus was supplied, so I reviewed the entire PR.
Please address all six findings before approval.
|
|
||
| return sort_writer->get_reserve_mem_size(state, eos); | ||
| // Column growth remains in every touched sorter, while sorting workspace is reused by serial dispatch. | ||
| // The final queued item may contain rows and also owns the reservation used by async finish(). |
There was a problem hiding this comment.
[P1] Reserve the final spill-merge fan-in on EOS. When a sorter has already spilled its in-memory block, its reserve components are zero, so this path retains only the minimum allowance. finish() nevertheless enters _combine_files_output(), whose merger constructs every input cursor and immediately reads one spill block per run (up to the configured merge fan-in, 64 MiB by default) before producing output. That work runs under this transferred reservation but is materially under-reserved and can cross a hard query/process limit. Please include EOS merge inputs/output workspace in the estimate and cover an empty sorter with several spill runs under a tight limit.
| if (rows) { | ||
| _memory_used_counter->update(add_block->allocated_bytes()); | ||
| } | ||
| _data_queue.emplace_back(QueuedBlock { |
There was a problem hiding this comment.
[P2] Release queued reservation tokens when asynchronous open() fails. The producer can enqueue here while the writer thread is still opening; force_close() then makes while (_writer_status.ok()) skip the only _data_queue.clear() path, and the token stays charged until writer destruction even though the finish dependency is ready. This can retain a large query/process/workload-group reservation throughout cancellation. Please clear or swap the queue on every terminal exit and add an enqueue-before-open-failure test that verifies the accounting returns to baseline.
| if (readSnapshotId < 0) { | ||
| // An explicit empty read must conflict with any snapshot created before beginWrite. | ||
| if (targetHead != null) { | ||
| throw new DorisConnectorException("Iceberg table " + tableName |
There was a problem hiding this comment.
[P1] Preserve the explicit-empty generation fence across the transaction refresh. applyBeginGuards() observes an empty head and converts the pinned -1 to null here, but openTransaction() immediately follows and newTransaction() refreshes the mutable table. If another writer creates the first snapshot in that interval, it becomes this transaction's base; the overwrite paths then skip validateFromSnapshot because baseSnapshotId is null and can replace that concurrent data. Please retain the -1 OCC sentinel (as the RowDelta path does) or perform the empty-head check against the refreshed transaction base, with an injected check-to-refresh race test.
|
Codex automated review failed and did not complete. Error: Review step was failure (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
3 participants
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Iceberg writes had several correctness and resource-safety gaps across the insert lifecycle:
This change unifies lifecycle cleanup, makes post-commit refresh/listener work best-effort, adds overwrite OCC validation, publishes all active partition writers for memory accounting, propagates close failures, bounds commit metadata before reporting, adds explicit writer abort and orphan cleanup support, and dispatches rows with compact permutations.
Release note
Harden Iceberg write correctness, memory accounting, and orphan-file cleanup.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)