RUM-18361: Split large client side stat batches up - #3827
Conversation
Replicate the splitting approach used by datadog-agent's own stats writer: an oversized periodic flush is split into multiple size-bounded chunks (grouped round-robin by a flat group-count cap, matching the agent's own heuristic) instead of being rejected outright by the storage layer's item size limit. Each chunk is gzip-compressed before being persisted to disk, and the upload request sets Content-Encoding: gzip so the SDK's shared upload interceptor doesn't compress it a second time — this keeps stored/uploaded payloads smaller without duplicating compression work. The per-flush metric now reports split (whether the flush was split into multiple chunks) plus succeeded/dropped bucket, group, span, and error counts, with bucket counts deduped by (start, duration) so a single original bucket whose groups landed across several successful (or several failed) chunks isn't counted more than once.
|
@codex review |
abrooksv
changed the title
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
✅ All CI checks and tests passed. Datadog automation helped this PR pass. 🎉 All green!🧪 All tests passed 🔄 Datadog retried 1 test - 1 passed on retry 🎯 Code Coverage (details) 🔗 Commit SHA: 0e38366 | Docs | View more details | Give us feedback! |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0e38366554
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // BatchStatsWriter persists one fully-wrapped, already gzip-compressed StatsPayload | ||
| // (envelope + splitPayload flag already applied) per batch, and maxItemsPerBatch = 1 | ||
| // guarantees exactly one item here. | ||
| val payload = batchData.firstOrNull()?.data |
There was a problem hiding this comment.
When two split chunks are persisted within the same millisecond, BatchFileOrchestrator.createNewFile() derives the same filename from System.currentTimeMillis(); because the writer appends, that file contains multiple RawBatchEvents despite maxItemsPerBatch = 1. Selecting only the first event then uploads one chunk and silently discards the remaining chunks when the successfully uploaded file is deleted, so the request factory must handle every returned event or storage must guarantee unique files.
Useful? React with 👍 / 👎.
1 participant
What does this PR do?
Replicate the splitting approach used by datadog-agent's own stats writer: an oversized periodic flush is split into multiple size-bounded chunks (grouped round-robin by a flat group-count cap, matching the agent's own heuristic) instead of being rejected outright by the storage layer's item size limit.
Each chunk is gzip-compressed before being persisted to disk, and the upload request sets Content-Encoding: gzip so the SDK's shared upload interceptor doesn't compress it a second time — this keeps stored/uploaded payloads smaller without duplicating compression work.
The per-flush metric now reports split (whether the flush was split into multiple chunks) plus succeeded/dropped bucket, group, span, and error counts, with bucket counts deduped by (start, duration) so a single original bucket whose groups landed across several successful (or several failed) chunks isn't counted more than once.
Motivation
Batches must contain a split flag, but we can't rely on the batching infrastructure to handle it due to we need to set the boolean on all batches that contain the bucket. Instead, we will reimplement the datadog agent algorithm and handle the batching in the stats code where we have full control.
Additional Notes
Agent's implementation: https://github.com/DataDog/datadog-agent/blob/main/pkg/trace/writer/stats.go
Review checklist (to be filled by reviewers)