Skip to content

[SLES-2971] feat(traces): gate X-Ray-driven sampling behind DD_MERGE_XRAY_TRACES - #1337

Draft
lym953 wants to merge 1 commit into
yiming.luo/sles-2971-xray-sampled-priorityfrom
yiming.luo/sles-2971-merge-xray-gate
Draft

[SLES-2971] feat(traces): gate X-Ray-driven sampling behind DD_MERGE_XRAY_TRACES#1337
lym953 wants to merge 1 commit into
yiming.luo/sles-2971-xray-sampled-priorityfrom
yiming.luo/sles-2971-merge-xray-gate

Conversation

@lym953

@lym953 lym953 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Stacked on #1325 — review that first; this diff is only the gate.

Why

#1325 stops an AWS-generated Sampled=0 from dropping Datadog traces, but an AWS-generated Sampled=1 still becomes an explicit priority 1 — X-Ray still drives Datadog sampling, in the direction that over-retains rather than loses data.

The tracer libraries make this opt-in through DD_MERGE_XRAY_TRACES (DD_MERGE_DATADOG_XRAY_TRACES in Ruby), default false. bottlecap never read it, so there was no way to decline.

What

Adds merge_xray_traces to LambdaConfig (default false) and gates the flag on it:

let honor_sampled_flag = datadog_planted || merge_xray_traces;
Root id Sampled merge off (default) merge on
…-00000000<16 hex> (Datadog-planted) 0 0 0
…-00000000<16 hex> 1 1 1
AWS-generated 0 unset 0
AWS-generated 1 1 1

Three scope decisions:

Datadog-planted headers are never gated. dd-trace-java plants context in this same AWSTraceHeader for SQS, and it is the only carrier for JMS→SQS. Gating it on an X-Ray flag would break Datadog-to-Datadog propagation for users with no X-Ray involvement.

Correlation is not gated. Trace and parent ids are taken from an AWS-generated header in both states — pre-existing v88+ behavior that loses no data, and gating it would cost the reporting customer their producer→consumer link. Gating it is separately arguable.

merge on honors an AWS-generated Sampled=0 as a drop, reinstating the reported behavior for anyone who asks for it: the flag means X-Ray's decisions count. If reviewers prefer the flag to govern correlation only and never sampling, that's a three-line change.

The flag is read off DatadogCompositePropagator, which already carries Arc<Config> to every extraction site. Threading Arc<Config> explicitly would touch five more files — the listener, LWA and interceptor state tuples, processor_service, and main.rs. Happy to switch; it's mechanical.

Open question

The docs scope trace merging to Node.js and Python, and describe it as merging X-Ray's own spans via the invocation's _X_AMZN_TRACE_ID — not as governing an SQS message system attribute. The reported case is Go. Reuse this variable and broaden its meaning, or add a distinct one? The docs need a line either way.

Testing

549 tests pass, clippy -D warnings and fmt clean — opt-in honoring a drop, Datadog-planted headers identical in both states, ids preserved in both.

Verified live on the SLES-2971 repro (Go, arm64, provided.al2023, SQS trigger), all four shapes in both flag states:

Root id Sampled merge=false merge=true
AWS-generated 0 1 0
AWS-generated 1 1 1
Datadog-planted 0 0 0
Datadog-planted 1 1 1

Only the AWS-generated Sampled=0 row moves with the flag. A trace id was present in all eight runs, confirming correlation is unaffected either way.

🤖 Generated with Claude Code

…XRAY_TRACES

Adopting an AWS-generated X-Ray `Sampled` flag as a Datadog sampling decision is
opt-in behavior in the tracer libraries, where `DD_MERGE_XRAY_TRACES` defaults to
false. bottlecap never read it, so there was no way to decline.

Add the config and gate the flag on it. A header a Datadog library planted is
never gated -- its `Sampled` is a Datadog decision regardless of the setting, and
gating it would break dd-trace-java's SQS propagation, which uses this same
header and has nothing to do with X-Ray.

Scope is the sampling verdict only. Trace and parent IDs are still taken from an
AWS-generated header either way, so correlation is unchanged by this flag; that
is pre-existing v88+ behavior and gating it is a separate argument.

The flag is read off `DatadogCompositePropagator`, which already carries the
config to every extraction site -- threading `Arc<Config>` through the listener,
LWA proxy, and interceptor state instead would have touched five more files.
@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Aug 21, 2026

Copy link
Copy Markdown

Pipelines  Tests

Unblock PR with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

Bottlecap (Rust) | Audit — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitHub Actions

Critical vulnerabilities found in job 'Audit'.

DataDog/datadog-lambda-extension | e2e-test-status (amd64, fips) — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitLab

📋 Copy prompt for your agent
CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: yiming.luo/sles-2971-merge-xray-gate

Bottlecap (Rust) | Audit
Commit: 9e2f4db93b1a855dea172e3b1c2afa25d36dbaec
Error (code / quality):
Critical vulnerabilities found in job 'Audit'.
CI job: https://github.com/DataDog/datadog-lambda-extension/actions/runs/32535191615/job/96934652750

ℹ️ Info

🔄 Datadog auto-retried 1 job - 1 passed on retry View in Datadog

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 9e2f4db | Docs | View more details | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant