Skip to content

[AMORO-4264][AMS] Use a dedicated thread pool for snapshot expiration planning - #4311

Open
johntomcat7408-cmyk wants to merge 1 commit into
apache:masterfrom
johntomcat7408-cmyk:fix-4264-expire-snapshots-thread-pool
Open

[AMORO-4264][AMS] Use a dedicated thread pool for snapshot expiration planning#4311
johntomcat7408-cmyk wants to merge 1 commit into
apache:masterfrom
johntomcat7408-cmyk:fix-4264-expire-snapshots-thread-pool

Conversation

@johntomcat7408-cmyk

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Snapshot expiration planning uses Iceberg's global worker pool by default. Under load, expiration planning can occupy that shared pool while self-optimizing commits wait for worker futures, leaving tables in the committing state for a long time.

Close #4264.

Brief change log

  • Add a process-wide planning pool dedicated to snapshot expiration.
  • Configure its size with expire-snapshots.plan-thread-count (default: 10) from the Iceberg process factory.
  • Pass the dedicated executor to ExpireSnapshots.planWith(...).
  • Add regression coverage for pool isolation, configuration validation, and the expiration call path.

How was this patch tested?

  • Add test cases that check positive and negative cases.
  • Screenshots are not applicable.
  • Run tests locally and on a Linux VM.

Validated on Linux with both JDK 11 and JDK 17:

./mvnw test \
  -pl amoro-ams \
  -am \
  -Pskip-dashboard-build \
  -Dtest=TestIcebergThreadPools,TestIcebergTableMaintainer,TestIcebergProcessFactory,TestSnapshotExpireIceberg \
  -Dsurefire.failIfNoSpecifiedTests=false

Both runs completed with BUILD SUCCESS. The focused unit tests passed without failures or errors, and TestSnapshotExpireIceberg reported 0 failures and 0 errors.

Documentation

  • Does this pull request introduce a new feature? No; it isolates an existing maintenance operation and adds a tuning option.
  • The default option is included in process-factories.yaml, and the pool scope and fallback behavior are documented in JavaDocs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement]: use seperate threadpool with table.expireSnapshots()

1 participant