borg2 compact: ctrl-c causes slow index-rebuild

[ThomasWaldmann]

borg2 compact can be safely interrupted / killed without causing backup data loss, but it achieves that using a quite radical way:

it kills the complete index from the repository and only writes a new one if it is not interrupted.

if it is interrupted, borg will not write an index, but rebuild a new index from packs the slow way at the start of the next borg operation.

depending on the store backend (range request support), caching (might always fetch the full object), etc., this may cause a full repository transfer over the network.

[ThomasWaldmann]

From Claude Opus 4.8:

The key realization is that compact already maintains a fully correct in-memory index the whole time — it just throws that correctness away on interrupt. So the rebuild isn't fundamentally necessary; it's an artifact of how persistence is structured.

The invariant any solution must respect

The reason delete_chunkindex_from_repo nukes everything up front (#9748) is to avoid a worse failure than a slow rebuild: a stale index that leads the store — pointing at a pack that compact already deleted or rewrote. That's genuinely dangerous, because archives store chunk ids, not locations; the index is the only id→(pack, offset) map. A stale "X is in pack P" entry means:

• create dedups against X and references it → but P is gone → broken archive, and
• restore of any archive referencing X fails.

So the safety rule is asymmetric:

The persisted index may lag (miss chunks that exist in packs → harmless orphans) but must never lead (reference chunks whose pack was deleted/moved → corruption).

Nuking up front is just the bluntest way to guarantee "never lead". The improvement is to keep the index correct and current instead of absent.

Best low-effort fix: flush the in-memory index on a clean Ctrl-C

compact_packs updates self.chunks in lockstep with every store change — compact_pack repoints kept ids and the drop loop deletes entries — so at every pack boundary the in-memory index is a correct, safe view of the store. But the compaction loops never check for interrupts.

Wire compact into the same soft-sig_int mechanism create already uses:

1. On first Ctrl-C, finish the current pack op, then break out of the loop at a boundary.
2. Call save_chunk_index() with the already-correct in-memory index.
3. Exit cleanly.

That eliminates the rebuild for the entire "user pressed Ctrl-C" case — no format change, ~no new risk, because you're persisting state that's already consistent. Only a hard kill -9 would still fall back to a rebuild. Highest value-to-effort change and it directly addresses this issue.

The deeper fix: incremental, crash-safe index persistence

To also cover hard kills / power loss without a full rebuild, persist the index as you go, in safe order. The existing index is a set of additive index/<hash> fragments, but the merge is last-writer-wins with no tombstones, so today it can only express "chunk added", not "chunk moved/removed" — which is exactly what compact does.

So: add tombstone + relocation fragments, and write them in the safe order index-delta-before-destructive-store-op:

• Rewrite P→P′: write new pack P′ → write a fragment relocating its kept ids to P′ → then delete P. Crash anywhere: index points at P′ (correct), P is at worst an orphan. Safe lag.
• Drop P: write a tombstone fragment removing P's ids → then delete P. Crash before delete: ids gone from index, pack is an orphan. Safe lag.

Flush these deltas periodically (every N packs / T seconds), not per-object, to bound fragment count. After a crash you just use the index as-is — it's already correct, modulo a handful of orphan packs from in-flight ops, which the next compact reclaims. No store_list("packs") scan ever.

The cost is the new tombstone semantics in the fragment format + merge, plus the deferred-delete ordering. But it makes compact crash-safe in the same incremental way create already is.

What I'd not do

Periodic full-index checkpoints (rewrite the whole index every N packs). It needs no format change and is correct if you defer deletes until after each checkpoint, but on a large repo it re-uploads the multi-GB index repeatedly — the exact churn we already avoid elsewhere. Fine as a stopgap, wrong as the design.

---

Suggested path: ship the soft-sig_int pack-boundary flush now (kills the annoyance for real Ctrl-C, trivially safe), and treat tombstone/relocation fragments as the proper follow-up that extends crash-safety to hard kills as well.