feat(storage): separate read and hedging thread pools

[ajayky-os]

This PR separates the thread pools used for primary reads and speculative hedged reads in Cloud Storage read hedging.

Previously, both primary reads and speculative hedges shared a single thread pool. This introduced two architectural issues:

1. Resource Contention / Head-of-Line Blocking: When multiple primary reads stalled or ran concurrently, the shared pool could become saturated. Consequently, speculative hedge
   attempts were blocked from executing, defeating the primary purpose of hedging (tail latency mitigation).
2. Asymmetric Sizing Needs: Primary reads block synchronously waiting on network I/O and require a higher concurrency ceiling (typically ≥64 threads or 4 × hardware concurrency),
   whereas speculative hedges are gated by rate limits and concurrency controls (typically bounded by MaxConcurrentHedgesOption or 2 × hardware concurrency).

This change introduces a general-purpose, lazily-scaling hedging_thread_pool.h:49 and composes it within hedging_thread_pool.h:145, isolating primary read execution from speculative hedges.

Key Changes

1. Dedicated ThreadPool and Embedded HedgingThreadPool

hedging_thread_pool.h:49:
• Dynamically scales workers on demand up to max_threads.
• Workers wait on a condition variable when idle and exit gracefully on shutdown.
• Automatically clamps max_threads to ≥1 to prevent deadlock/infinite hang if configured with 0.
• Supports self-destruction from within a worker thread (safely detaches rather than joining itself).

hedging_thread_pool.h:145:
• Embeds hedging_thread_pool.h:235 by value as its execution backend (declared last to guarantee worker joining before state teardown).
• Enforces the token-bucket rate limiter (ReadHedgeRateLimitOption) and maximum concurrent hedge ceiling (MaxConcurrentHedgesOption).

2. Dual Pool Configuration & Sizing Options

• Added options.h:122: Defaults to DefaultReadThreadPoolSize() (max (64,4 × cores)).
• Added options.h:135: Defaults to DefaultHedgingThreadPoolSize() (MaxConcurrentHedgesOption if set, else max (16,2 × cores)).
• Centralized sizing defaults in DefaultReadThreadPoolSize() and DefaultHedgingThreadPoolSize() so client.cc:604 and connection_impl.cc:165 remain consistent.

3. Isolation in HedgedObjectReadSource

• Updated hedged_object_read_source.cc:90 to accept separate read_pool_ and hedge_pool_.
• Primary attempt opens are scheduled onto read_pool_.
• Speculative hedged attempts are scheduled onto hedge_pool_.

[gemini-code-assist]

Code Review

This pull request refactors the storage client's thread pool management by introducing a dedicated ThreadPool for primary read attempts, distinct from the HedgingThreadPool used for speculative hedges. It adds configuration options for thread pool sizes and updates the connection implementation, read source, and tests accordingly. The review feedback identifies a namespace compilation error in client.cc, requests the use of explicit types instead of auto for primitives in connection_impl.cc to comply with the style guide, and suggests caching thread pool size calculations in hedging_thread_pool.h for better performance.

[codecov]

Codecov Report

❌ Patch coverage is 93.57798% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.30%. Comparing base (82fcb08) to head (40cf649).

Files with missing lines	Patch %	Lines
google/cloud/storage/internal/connection_impl.cc	8.33%	11 Missing ⚠️
...storage/internal/hedged_object_read_source_test.cc	97.72%	2 Missing ⚠️
...loud/storage/internal/hedged_object_read_source.cc	83.33%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #16389    +/-   ##
========================================
  Coverage   92.30%   92.30%            
========================================
  Files        2245     2245            
  Lines      211870   211995   +125     
========================================
+ Hits       195568   195689   +121     
- Misses      16302    16306     +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.