[refactor](storage) Unify BE and Recycler object clients by sollhui · Pull Request #66350 · apache/doris

sollhui · 2026-08-01T09:09:38Z

1. What does this PR do?

BE and Cloud Recycler previously maintained separate object-storage abstractions and separate S3/Azure implementations. Although both sides called the same cloud-provider SDKs, credential construction, error conversion, metrics, pagination, batch deletion, and compatibility behavior were duplicated and could evolve differently.

This PR consolidates the implementation under common/cpp/client and exposes one ObjStorageClient facade to upper layers:

ObjStorageClient owns backend-independent orchestration and is the only complete client used by BE and Recycler call sites.
ObjStorageRateLimitPolicy keeps BE- and Recycler-specific admission behavior injectable without coupling common code to either environment.
ObjStorageBackend is the storage implementation boundary, implemented by S3ObjStorageBackend and AzureObjStorageBackend.
Shared request/response types, page-based listing, upper-layer lazy iteration, recursive deletion, backend batch capabilities, credentials, metrics, and error conversion live in the common layer.
ObjectStoreInfoPB carries the optional AWS session token through the same credential path; token values are redacted from logs and debug output.

Before the refactor, BE and Recycler reached the cloud SDKs through parallel stacks:

                                     BEFORE

  +---------------------- BE ----------------------+     +------------------- Recycler -------------------+
  |  BE call sites -> ObjClientHolder              |     |  Recycler call sites -> S3Accessor            |
  |                       |                        |     |                       |                        |
  |                       v                        |     |                       v                        |
  |  BE limiter + separate S3 / Azure clients      |     |  Recycler limiter + separate S3 / Azure clients|
  |  credentials / listing / recursive deletion    |     |  credentials / listing / recursive deletion    |
  |  error conversion / metrics                    |     |  error conversion / metrics                    |
  +------------------------+-----------------------+     +------------------------+-----------------------+
                           |                                                        |
                           v                                                        v
                    AWS SDK / Azure SDK                                      AWS SDK / Azure SDK

After the refactor, BE and Recycler stay on the left and right while the shared facade and backend components are centered below them:

                                      AFTER

  +---------------------- BE ----------------------+     +------------------- Recycler -------------------+
  |  BE call sites -> ObjClientHolder / factory    |     |  Recycler call sites -> S3Accessor / adapter  |
  +-----------------------------+------------------+     +------------------+-----------------------------+
                                \                                           /
                                 \                                         /
                                  v                                       v
                          +--------------------------------------------------+
                          |            ObjStorageClient facade               |
                          |                                                  |
                          |  +----------------------+  +-------------------+  |
                          |  | RateLimitPolicy      |  | ObjStorageBackend |  |
                          |  | - BE policy          |  |        |          |  |
                          |  | - Recycler policy    |  |   +----+----+     |  |
                          |  +----------------------+  |   |         |     |  |
                          |                            |   v         v     |  |
                          |                            | S3 Backend Azure   |  |
                          |                            | Backend            |  |
                          |                            +---+---------+------+  |
                          +--------------------------------|---------|---------+
                                                           v         v
                                                        AWS SDK   Azure SDK

This boundary prevents a raw backend from being used as the complete client and accidentally bypassing runtime policy. Backend code only implements cloud mechanics; common orchestration and policy dispatch remain in the facade.

2. How are the different behaviors unified?

Behavior	BE before	Recycler before	Unified behavior
Client API	BE-specific `doris::io::ObjStorageClient` and eager list results	Recycler-specific client and iterator APIs	One `doris::ObjStorageClient` facade and one set of request/response types. `doris::io` aliases keep BE call sites source-compatible; Recycler adapters preserve its integer-facing API.
Backend implementation	Separate BE S3/Azure clients	Separate Recycler S3/Azure clients	`S3ObjStorageBackend` and `AzureObjStorageBackend` are shared by both callers.
Rate limiting	BE owned QPS/bytes limiters and bucket-selection rules	Recycler owned its limiter and fault injection	Each environment injects an `ObjStorageRateLimitPolicy`; the facade performs admission before dispatching backend work. Backend implementations do not depend on BE or Recycler configuration.
Error model	BE status codes plus HTTP metadata on selected paths	Recycler-specific return codes and messages	`ObjectStorageResponse` consistently carries a Doris status code, HTTP code, and request ID, while adapters preserve caller-facing behavior.
Listing	BE eagerly collected all pages	Recycler exposed a lazy iterator	`ObjStorageClient::list_objects` returns one fixed-size `ObjectStorageListPage`. One Client call performs one admission, one Backend call, and one SDK request. The upper `ObjectListIterator` owns the continuation token and requests the next page only after its cached page is consumed.
End of listing	Eager completion was represented by a finished vector	Iterator completion used empty/false results	`END_OF_FILE` is an internal upper-iterator sentinel and `next()` converts it to a successful empty result. Backend `NOT_FOUND` remains a real error.
Missing S3 prefix	S3-compatible `NoSuchKey` handling existed in the BE path	The compatibility behavior was maintained separately	The shared S3 backend preserves `NoSuchKey`-as-empty behavior once for both callers.
Direct batch deletion	Backend limits were embedded in separate implementations	Recycler maintained its own batching	The facade splits by backend capability (`1000` for S3, `256` for Azure), performs one admission per backend-sized batch, and backends keep defensive bounds.
Recursive deletion	Separate implementations and grouping behavior	Recycler supported expiration filtering and parallel execution	The facade owns one shared listing/filtering/grouping/error-propagation flow. Recycler injects its executor; BE uses synchronous defaults. For compatibility, the whole recursive operation is still admitted once; per-SDK-request accounting is left as a follow-up.
AWS credentials	BE built static/default/role providers in its factory	Recycler maintained another construction path	`AwsCredentialFactory` implements static AK/SK/session-token credentials, provider chains, role ARN, and external ID once. Callers explicitly retain their prior empty-credential behavior.
Azure credentials	BE and Recycler built shared-key clients separately	Separate construction and credential retention	`AzureAuthFactory` creates the container client and shared-key credential for both; BE TLS diagnostic context remains attached to Azure errors.
Session token	`S3ClientConf` had a token, but the storage-vault protobuf path did not	Recycler could not receive an AK/SK session token from `ObjectStoreInfoPB`	`ObjectStoreInfoPB -> S3Conf -> AwsCredentialFactory` carries AK/SK/token consistently, and token values are cleared or masked before logging.
Metrics and latency	Duplicated stopwatch and failure-recording paths	Separate helpers recorded equivalent data	Backends use the shared `client_bvar::ScopedLatency` timer and common failure metrics. Public stopwatch utilities are unchanged.
Backend-specific APIs	Shared interfaces forced unrelated test stubs	Lifecycle, versioning, and multipart-abort were Recycler-oriented	The backend boundary supplies default not-supported responses and implements supported APIs without leaking backend details to callers.

3. Design boundaries and follow-ups

Upper layers keep std::shared_ptr<ObjStorageClient>; they do not store ObjStorageBackend directly.
BE and Recycler own their policy configuration, but both use the same facade dispatch and the same S3/Azure backends.
The upper ObjectListIterator performs lazy iteration by repeatedly calling the one-page Client API. Therefore each requested page has exactly one facade admission and one SDK request; reading objects already cached in that page performs no network request.
A public delete_objects call may still split into multiple backend-sized SDK requests. A follow-up will move batching above the facade and introduce a one-batch Client API, so each Client call maps to one admission and one SDK request.
Recursive deletion intentionally keeps the previous BE behavior of one logical PUT admission even though it may issue multiple list/delete SDK calls. A follow-up will move recursive orchestration above the facade and compose the one-page list API with the planned one-batch delete API.
Local compilation and tests were not run as requested. Changed C++ files pass clang-format 16 dry-run, stale-name checks, and git diff --check.

hello-stephen · 2026-08-01T09:09:46Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

sollhui · 2026-08-01T09:09:48Z

run buildall

sollhui · 2026-08-01T09:09:55Z

/review

github-actions · 2026-08-01T09:57:59Z

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/30693205073

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

sollhui · 2026-08-03T07:40:42Z

run buildall

sollhui · 2026-08-03T07:40:47Z

/review

sollhui · 2026-08-03T08:34:34Z

run buildall

sollhui · 2026-08-03T08:34:39Z

/review

github-actions · 2026-08-03T09:06:17Z

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/30794530392

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

github-actions · 2026-08-03T09:49:24Z

Codex automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/30797931848

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

hello-stephen · 2026-08-03T10:09:19Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

sollhui · 2026-08-03T15:28:29Z

/review

sollhui · 2026-08-03T15:28:35Z

run buildall

github-actions

Static-only review of the full authoritative diff found six issues that should be addressed before merge (five P1, one P2).

Review-cycle status: incomplete after the three-round cap. Both normal agents returned NO_NEW_VALUABLE_FINDINGS in Round 3, but the risk-focused agent found a final FE-side scope correction that was independently verified and merged into the token round-trip comment; the review contract does not permit a fourth round. All currently known candidates are nevertheless adjudicated and included below.

Critical checkpoint conclusions:

Data correctness: failed. Session-token credentials are dropped/staled across FE DDL and meta-service paths, and Recycler exists status mapping can turn real provider failures into false not-found results.
Concurrency and lifecycle: delete-task ownership, executor waiting, batch clamping, and error propagation are sound; request admission during recursive deletion is not.
Configuration and dynamic behavior: Recycler rate limiting and PUT fault injection are bypassed for the actual recursive-delete SDK requests; AWS provider precedence, refresh-capable providers, and client cache identity otherwise remain compatible.
Compatibility and rolling behavior: the optional protobuf field is wire-compatible, but the Recycler 0/1/negative adapter contract and GCS iterator migration are broken.
Parallel paths: BE/Recycler and S3/Azure/GCS paths were traced; the GCS path has an unconditional compile failure and Recycler differs from the preserved BE admission behavior.
Tests and validation: no builds or tests were run, as required by the review prompt. Existing S3 accessor tests still require 1 for not-found, and there is no end-to-end token persistence/redaction/rotation coverage; the GCS compile error is statically evident.
Observability and security: the session token lacks SK-equivalent encryption/log/display handling, and successful S3 writes now log at INFO on the hot path. This is credential-secret handling within authenticated control paths; no unsupported cross-tenant vulnerability claim is made.
Persistence and recovery: token-bearing vault/stage records can either lose the token or retain it plaintext, so persistence round trips are not safe.
Performance: recursive deletion can evade Recycler request controls, while per-write INFO logging adds log I/O proportional to storage QPS; page and provider batch limits themselves are sound.

User focus: review_focus.txt supplied no additional focus, so the entire PR was reviewed without narrowing scope.

github-actions · 2026-08-03T18:26:37Z

    optional CredProviderTypePB cred_provider_type = 17;
    optional string role_arn = 18;  // aws assumed role's arn
    optional string external_id = 19;  // aws assumed role's external_id if configure
+    optional string token = 20; // optional session token paired with ak/sk


[P1] Give the token the same secret lifecycle as SK

This field is currently left plaintext by create_object_info_with_encrypt and external-stage creation while SK is encrypted. It can then be emitted by full-instance INFO logs that do not call hide_token, returned by the list-all external-stage display path, and serialized by SHOW STORAGE VAULT for any USAGE grantee because that converter masks only SK. Please encrypt/decrypt the token like other credential secrets and explicitly strip or mask it from every log and display response.

github-actions · 2026-08-03T18:26:37Z

    *res = std::make_unique<S3ListIterator>(
-            obj_client_->list_objects({.bucket = conf_.bucket, .key = get_key(path_prefix)}),
-            prefix_length);
+            obj_client_,


[P1] Preserve Recycler's exists return contract

The common response uses positive Thrift codes (NOT_FOUND is 31, while internal/auth errors are also positive), but StorageVaultAccessor::exists promises 0 for present, 1 for absent, and a negative value for errors. checker.cpp consequently treats an S3 permission or server failure as a missing file, and the existing accessor tests still require 1 for a 404. Map OK to 0, NOT_FOUND to 1, and every other status to a negative error at this adapter boundary.

Anchor correction: the affected S3Accessor::exists return is the new .resp.status.code at line 600.

github-actions · 2026-08-03T18:26:37Z

+    // BE behavior of charging it as one logical PUT call in this refactor. Move recursive
+    // orchestration above ObjStorageClient in a follow-up, using the one-page list API and a
+    // one-batch delete API so every client call maps to one admission and one SDK request.
+    auto rate_limit = acquire(ObjStorageRequestType::PUT);


[P1] Keep Recycler recursive deletes under request-level admission

After this single PUT admission, every list page and delete batch calls the provider directly, so RecyclerObjStorageRateLimitPolicy is never invoked for the actual SDK requests. The deleted Recycler implementations admitted every GET page and PUT batch; a large prefix can now issue unbounded pages and concurrent deletes after one token and one fault-injection decision. Route each page/batch through the policy-bearing facade, or make BE's one-logical-operation behavior an explicit BE-only mode.

github-actions · 2026-08-03T18:26:37Z

+                .request_id = outcome.GetError().GetRequestId()};
+    }
+
+    LOG(INFO) << "request_id = " << request_id << ", bucket = " << opts.bucket


[P2] Avoid INFO logging every successful object write

This unconditional success log turns normal PutObject traffic into one INFO record per object; create and complete multipart have the same pattern. The deleted BE implementation logged these only when they exceeded the 5-second threshold, and Recycler logged PutObject failures only, so ingest/file-cache traffic can now generate log I/O proportional to storage QPS. Restore slow-request-only logging or lower these success messages to VLOG while keeping latency bvars for aggregate observability.

github-actions · 2026-08-03T18:26:37Z

 }

 int S3Accessor::get_life_cycle(int64_t* expiration_days) {
-    return obj_client_->get_life_cycle(conf_.bucket, expiration_days).ret;


[P1] Finish converting the GCS iterator to value semantics

This declaration changes iter from a pointer to a stack ObjectListIterator, and the new loop correctly calls iter.next(), but the post-loop check at line 678 still calls iter->is_valid(). The common iterator defines no operator->, so the Cloud target cannot compile regardless of which provider is selected at runtime. Change that remaining check to iter.is_valid().

Anchor correction: the value-semantic iterator is declared at line 636, and the stale pointer-style use is at line 678.

hello-stephen · 2026-08-03T19:08:25Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

Move the shared S3/Azure clients, credential factories, pagination, and recursive deletion orchestration into common/client. Keep BE and Recycler policies in runtime hooks and adapters. Reuse the CPU-aware limiter manager and token-bucket fixes needed from apache#65420, but apply admission before each real provider SDK request instead of wrapping logical operations. Test: Not run (per request).

Issue Number: None Related PR: None Problem Summary: Move rate limiting out of the S3 and Azure provider clients into one common logical-call decorator used by both BE and Recycler. Lazy list iteration now fetches at most 1000 objects per page and charges once per fetched page. Batch deletion is split by each provider's request limit before admission, so one charged batch maps to one SDK request. Recursive deletion intentionally keeps the previous BE one-call accounting and records a TODO for a follow-up change. None - Test: Not run at the user's request; clang-format and git diff checks passed - Behavior changed: Yes, BE and Recycler now share logical-call rate-limiting semantics - Does this need documentation: No

### What problem does this PR solve? Issue Number: N/A Related PR: None Problem Summary: The standalone rate-limiting decorator and provider implementations shared the same client interface, so a raw provider could be used where the complete object-storage client was expected and bypass runtime policy. Refactor the shared object-storage layer into an ObjStorageClient facade composed from an environment-specific ObjStorageRateLimitPolicy and an S3 or Azure ObjStorageProvider. Keep pagination, provider-sized delete batching, credential handling, and the existing recursive-delete admission behavior unchanged. ### Release note None ### Check List (For Author) - Test: Not run per request; clang-format 16, stale-reference checks, and git diff --check passed - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: N/A Related PR: None Problem Summary: The shared list API returned a lazy provider iterator, so one ObjStorageClient list call could issue multiple SDK requests. Change the Client and Provider contract to return one fixed-size ObjectStorageListPage per call, and move continuation-token iteration above ObjStorageClient. Each Client list call now corresponds to one rate-limit admission and one SDK request while BE and Recycler retain lazy consumption. Recursive delete keeps its existing single logical PUT admission and remains a follow-up item for request-level accounting. ### Release note None ### Check List (For Author) - Test: Not run per request; clang-format 16, stale-reference checks, and git diff --check passed - Behavior changed: No (internal list pagination and admission boundaries only) - Does this need documentation: No

### What problem does this PR solve? Issue Number: N/A Related PR: None Problem Summary: Direct batch deletion and recursive deletion can each issue multiple SDK requests from one public ObjStorageClient call. Record follow-up work to move batching and recursive orchestration above the facade, so a future one-batch delete API can map each Client call to one rate-limit admission and one SDK request. ### Release note None ### Check List (For Author) - Test: Not run per request; clang-format 16 and git diff --check passed - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: N/A Related PR: None Problem Summary: ObjStorageClient is the complete facade used by upper layers, while the S3 and Azure implementations only supply storage mechanics behind that facade. Rename ObjStorageProvider and its implementations to ObjStorageBackend, S3ObjStorageBackend, and AzureObjStorageBackend so the type names express those architectural roles and do not imply that a raw backend is a complete client. Update file names, factories, build references, synchronization points, and tests consistently. ### Release note None ### Check List (For Author) - Test: Not run per request; clang-format 16, stale-name checks, and git diff --check passed - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: N/A Related PR: None Problem Summary: S3ClientFactory and Recycler credential helpers reduced AwsCredentialFactory failures to a null shared pointer after logging internally, so callers could not inspect the failure reason and Kinesis could attempt construction with an empty provider. Return AwsCredentialResult from a single version-aware helper, handle its error at each client-construction boundary, and pass the provider to the SDK only after successful creation. This also removes duplicated v1/v2 helper implementations while preserving their credential compatibility behavior. ### Release note None ### Check List (For Author) - Test: Not run per request; added credential error propagation coverage, and clang-format 16 plus git diff --check passed - Behavior changed: Yes (invalid credential-provider construction now returns its explicit error at the caller boundary) - Does this need documentation: No

sollhui · 2026-08-04T08:02:05Z

run buildall

sollhui requested review from Gabriel39, deardeng, gavinchou, liaoxin01, luwei16, morningman and yiguolei as code owners August 1, 2026 09:09

sollhui force-pushed the agent/unify-be-recycler-obj-client branch from f30abce to dc25d3e Compare August 1, 2026 09:44

gavinchou reviewed Aug 1, 2026

View reviewed changes

Comment thread common/cpp/client/obj_storage_client.cpp

sollhui force-pushed the agent/unify-be-recycler-obj-client branch from 1fab2ee to 4ea09ca Compare August 3, 2026 13:48

github-actions Bot requested changes Aug 3, 2026

View reviewed changes

sollhui added 5 commits August 4, 2026 11:17

[refactor](storage) Move object storage client under common/cpp

4918cce

[fix](cloud) Redact object storage session tokens

4095078

[chore](storage) Fix clang-format

0a57017

[fix](storage) Remove conflicting client declaration

879fd06

sollhui added 8 commits August 4, 2026 11:17

[fix](storage) Include web identity credentials provider

689d8d8

[fix](storage) Preserve credential compatibility

1ca6c85

update

bad4f89

sollhui force-pushed the agent/unify-be-recycler-obj-client branch from 92a3812 to 3f0e01e Compare August 4, 2026 03:18

sollhui added 8 commits August 4, 2026 11:31

[fix](cloud) defer recycler session token support

e04c8af

[fix](cloud) remove deferred token cleanup changes

29396c0

[refactor](be) return result from object client factory

0841929

[fix](cloud) clean object storage lifecycle interfaces

3ec9595

update

7734a51

[fix](cloud) preserve recycler exists contract

dd54d22

update

fb71485

Uh oh!

Conversation

sollhui commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. What does this PR do?

2. How are the different behaviors unified?

3. Design boundaries and follow-ups

Uh oh!

hello-stephen commented Aug 1, 2026

Uh oh!

sollhui commented Aug 1, 2026

Uh oh!

sollhui commented Aug 1, 2026

Uh oh!

github-actions Bot commented Aug 1, 2026

Uh oh!

Uh oh!

sollhui commented Aug 3, 2026

Uh oh!

sollhui commented Aug 3, 2026

Uh oh!

sollhui commented Aug 3, 2026

Uh oh!

sollhui commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

FE UT Coverage Report

Uh oh!

sollhui commented Aug 3, 2026

Uh oh!

sollhui commented Aug 3, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Aug 3, 2026

FE UT Coverage Report

Uh oh!

sollhui commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sollhui commented Aug 1, 2026 •

edited

Loading