Skip to content

[docs] Document Paimon dynamic read options - #4021

Merged
Gabriel39 merged 2 commits into
apache:masterfrom
Gabriel39:agent/document-paimon-reader-options
Aug 3, 2026
Merged

[docs] Document Paimon dynamic read options#4021
Gabriel39 merged 2 commits into
apache:masterfrom
Gabriel39:agent/document-paimon-reader-options

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Document the complete user-visible Paimon read behavior implemented by apache/doris#66247 and its branch-4.1 backport apache/doris#66297.

  • Lists all seven supported bounded batch-read options, their defaults, accepted values, and runtime effects.
  • Documents relation, Catalog, physical-table, and Paimon-default precedence, including validation of the final effective value.
  • Explains relation-local aliases, metadata-neutral projection reuse, and the local runtime cap for manifest planning.
  • Documents atomic Catalog ALTER validation and compatibility behavior for persisted legacy properties.
  • Covers statement-consistent snapshot/schema binding across partitions, statistics, system tables, and data scans.
  • Adds the supported Paimon @options time-travel selectors, combination constraints, view behavior, and explicit failure semantics.
  • Lists the system tables that accept @options and the unsupported file-creation-time filter.
  • Documents exact quoted top-level and nested identifier preservation on both Paimon JNI scanner paths.
  • Records safe logging, the one-second async-reader profile sampling interval, and retry-safe cleanup behavior.

The same behavior is documented in current and 4.x English and Chinese pages.

Related code changes:

Version scope

The implementation targets Doris master and branch-4.1, so this change updates current and 4.x. The behavior is not present in 3.x or 2.1.

Validation

  • yarn docs:links:changed
  • yarn docs:features:changed
  • yarn docs:i18n-sync:changed
  • yarn docs:lint:changed
  • yarn build (all configured English, Chinese, Japanese, and historical-version pages; exit code 0)
  • git diff --check

All validation commands pass. The reports retain existing repository-wide Markdown, sidebar, SEO, external-link, and Japanese translation-candidate notices; no new error is introduced by these changes.

Versions

  • dev
  • 4.x
  • 3.x (feature not supported)
  • 2.1 or older (feature not supported)

Languages

  • Chinese
  • English
  • Japanese candidate translation needed

Docs Checklist

  • Checked by AI
  • Test Cases Built
  • Updated required version and language counterparts, or explained why not
  • If only one language changed, confirmed whether source/translation counterparts need sync

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 30, 2026
@Gabriel39
[fix](paimon) Expand dynamic reader option support
1ffd61f 
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66247

Problem Summary: Doris allowed only `read.batch-size` and
`file-reader-async-threshold` through Paimon Catalog properties and relation
`@options`, so safe batch-read planning controls such as split sizing, file-index
reads, and manifest planning were rejected. The missing support also made it
unclear which official Paimon option names Doris consumes; these names must remain
the Paimon names and do not all use a `scan.` prefix.

Expand the allowlist to the seven batch-read options consumed by Doris, validate
their types and safety bounds, and keep manifest/partition-order settings on the
effective metadata projection path. Catalog defaults and relation overrides now
accept the same reader tuning while context selectors, Flink enumerator settings,
streaming, layout, and write options remain excluded.

The previous BE unit-test commit also omitted the new `JniColumn.output_type`
member from two designated initializers. Clang treats that omission as an error
under the BE UT warning policy, so initialize the field explicitly in both tests.

### Release note

Paimon Catalog properties and relation `@options` now support
`read.batch-size`, `file-reader-async-threshold`, `file-index.read.enabled`,
`source.split.target-size`, `source.split.open-file-cost`,
`scan.manifest.parallelism`, and `scan.plan-sort-partition`.

### Check List (For Author)

- Test:
    - Unit and P0 regression coverage added.
    - Isolated Paimon 1.3.1 Java compile/runtime validation passed.
    - Regression framework compilation passed (466 Groovy files).
    - Doris clang-format 16 check passed.
    - Full FE/BE unit execution was not run locally because the installed
      third-party toolchain, including `protoc`, is absent; CI is requested.
- Behavior changed: Yes. Doris accepts five additional safe Paimon batch-read
  options from Catalog defaults and relation-scoped overrides.
- Does this need documentation: Yes.
  apache/doris-website#4021
@Gabriel39 Gabriel39 mentioned this pull request Jul 30, 2026
Merged
@Gabriel39
Gabriel39 marked this pull request as ready for review July 30, 2026 07:43
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 30, 2026
@Gabriel39
[fix](paimon) Expand dynamic reader option support
fb2ea10 
Issue Number: None

Related PR: apache#66247

Problem Summary: Doris allowed only `read.batch-size` and
`file-reader-async-threshold` through Paimon Catalog properties and relation
`@options`, so safe batch-read planning controls such as split sizing, file-index
reads, and manifest planning were rejected. The missing support also made it
unclear which official Paimon option names Doris consumes; these names must remain
the Paimon names and do not all use a `scan.` prefix.

Expand the allowlist to the seven batch-read options consumed by Doris, validate
their types and safety bounds, and keep manifest/partition-order settings on the
effective metadata projection path. Catalog defaults and relation overrides now
accept the same reader tuning while context selectors, Flink enumerator settings,
streaming, layout, and write options remain excluded.

The previous BE unit-test commit also omitted the new `JniColumn.output_type`
member from two designated initializers. Clang treats that omission as an error
under the BE UT warning policy, so initialize the field explicitly in both tests.

Paimon Catalog properties and relation `@options` now support
`read.batch-size`, `file-reader-async-threshold`, `file-index.read.enabled`,
`source.split.target-size`, `source.split.open-file-cost`,
`scan.manifest.parallelism`, and `scan.plan-sort-partition`.

- Test:
    - Unit and P0 regression coverage added.
    - Isolated Paimon 1.3.1 Java compile/runtime validation passed.
    - Regression framework compilation passed (466 Groovy files).
    - Doris clang-format 16 check passed.
    - Full FE/BE unit execution was not run locally because the installed
      third-party toolchain, including `protoc`, is absent; CI is requested.
- Behavior changed: Yes. Doris accepts five additional safe Paimon batch-read
  options from Catalog defaults and relation-scoped overrides.
- Does this need documentation: Yes.
  apache/doris-website#4021
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 31, 2026
@Gabriel39
[fix](paimon) Expand dynamic reader option support
b473593 
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66247

Problem Summary: Doris allowed only `read.batch-size` and
`file-reader-async-threshold` through Paimon Catalog properties and relation
`@options`, so safe batch-read planning controls such as split sizing, file-index
reads, and manifest planning were rejected. The missing support also made it
unclear which official Paimon option names Doris consumes; these names must remain
the Paimon names and do not all use a `scan.` prefix.

Expand the allowlist to the seven batch-read options consumed by Doris, validate
their types and safety bounds, and keep manifest/partition-order settings on the
effective metadata projection path. Catalog defaults and relation overrides now
accept the same reader tuning while context selectors, Flink enumerator settings,
streaming, layout, and write options remain excluded.

The previous BE unit-test commit also omitted the new `JniColumn.output_type`
member from two designated initializers. Clang treats that omission as an error
under the BE UT warning policy, so initialize the field explicitly in both tests.

### Release note

Paimon Catalog properties and relation `@options` now support
`read.batch-size`, `file-reader-async-threshold`, `file-index.read.enabled`,
`source.split.target-size`, `source.split.open-file-cost`,
`scan.manifest.parallelism`, and `scan.plan-sort-partition`.

### Check List (For Author)

- Test:
    - Unit and P0 regression coverage added.
    - Isolated Paimon 1.3.1 Java compile/runtime validation passed.
    - Regression framework compilation passed (466 Groovy files).
    - Doris clang-format 16 check passed.
    - Full FE/BE unit execution was not run locally because the installed
      third-party toolchain, including `protoc`, is absent; CI is requested.
- Behavior changed: Yes. Doris accepts five additional safe Paimon batch-read
  options from Catalog defaults and relation-scoped overrides.
- Does this need documentation: Yes.
  apache/doris-website#4021
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 31, 2026
@Gabriel39
[fix](paimon) Expand dynamic reader option support
3a8d0fa 
Issue Number: None

Related PR: apache#66247

Problem Summary: Doris allowed only `read.batch-size` and
`file-reader-async-threshold` through Paimon Catalog properties and relation
`@options`, so safe batch-read planning controls such as split sizing, file-index
reads, and manifest planning were rejected. The missing support also made it
unclear which official Paimon option names Doris consumes; these names must remain
the Paimon names and do not all use a `scan.` prefix.

Expand the allowlist to the seven batch-read options consumed by Doris, validate
their types and safety bounds, and keep manifest/partition-order settings on the
effective metadata projection path. Catalog defaults and relation overrides now
accept the same reader tuning while context selectors, Flink enumerator settings,
streaming, layout, and write options remain excluded.

The previous BE unit-test commit also omitted the new `JniColumn.output_type`
member from two designated initializers. Clang treats that omission as an error
under the BE UT warning policy, so initialize the field explicitly in both tests.

Paimon Catalog properties and relation `@options` now support
`read.batch-size`, `file-reader-async-threshold`, `file-index.read.enabled`,
`source.split.target-size`, `source.split.open-file-cost`,
`scan.manifest.parallelism`, and `scan.plan-sort-partition`.

- Test:
    - Unit and P0 regression coverage added.
    - Isolated Paimon 1.3.1 Java compile/runtime validation passed.
    - Regression framework compilation passed (466 Groovy files).
    - Doris clang-format 16 check passed.
    - Full FE/BE unit execution was not run locally because the installed
      third-party toolchain, including `protoc`, is absent; CI is requested.
- Behavior changed: Yes. Doris accepts five additional safe Paimon batch-read
  options from Catalog defaults and relation-scoped overrides.
- Does this need documentation: Yes.
  apache/doris-website#4021
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Aug 2, 2026
@Gabriel39
[fix](paimon) Expand dynamic reader option support
d6fe796 
Issue Number: None

Related PR: apache#66247

Problem Summary: Doris allowed only `read.batch-size` and
`file-reader-async-threshold` through Paimon Catalog properties and relation
`@options`, so safe batch-read planning controls such as split sizing, file-index
reads, and manifest planning were rejected. The missing support also made it
unclear which official Paimon option names Doris consumes; these names must remain
the Paimon names and do not all use a `scan.` prefix.

Expand the allowlist to the seven batch-read options consumed by Doris, validate
their types and safety bounds, and keep manifest/partition-order settings on the
effective metadata projection path. Catalog defaults and relation overrides now
accept the same reader tuning while context selectors, Flink enumerator settings,
streaming, layout, and write options remain excluded.

The previous BE unit-test commit also omitted the new `JniColumn.output_type`
member from two designated initializers. Clang treats that omission as an error
under the BE UT warning policy, so initialize the field explicitly in both tests.

Paimon Catalog properties and relation `@options` now support
`read.batch-size`, `file-reader-async-threshold`, `file-index.read.enabled`,
`source.split.target-size`, `source.split.open-file-cost`,
`scan.manifest.parallelism`, and `scan.plan-sort-partition`.

- Test:
    - Unit and P0 regression coverage added.
    - Isolated Paimon 1.3.1 Java compile/runtime validation passed.
    - Regression framework compilation passed (466 Groovy files).
    - Doris clang-format 16 check passed.
    - Full FE/BE unit execution was not run locally because the installed
      third-party toolchain, including `protoc`, is absent; CI is requested.
- Behavior changed: Yes. Doris accepts five additional safe Paimon batch-read
  options from Catalog defaults and relation-scoped overrides.
- Does this need documentation: Yes.
  apache/doris-website#4021
@Gabriel39
Gabriel39 merged commit eb0eccc into apache:master Aug 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Gabriel39