[SLES-2971] fix(traces): only trust the X-Ray Sampled flag on Datadog-planted headers

[lym953]

https://datadoghq.atlassian.net/browse/SLES-2971

Overview

A reported Go Lambda dropped from ~14 spans per invocation to 2 after moving from extension v87 + DD_EXTENSION_VERSION=compatibility to v98. The spans weren't lost in transit — the tracer built the full tree, but the whole trace was marked sampled-out.

With no readable _datadog attribute on the SQS record, extract_span_context falls back to the X-Ray AWSTraceHeader and mapped its Sampled flag straight onto a Datadog priority:

let sampling_priority = i8::from(sampled == "1");

Sampled was not 1, so priority 0 went into the start-invocation response (listener.rs:216-224), dd-trace-go adopted it for its whole trace, and the backend dropped that chunk. The extension's own inferred + invocation spans survive because a priority is only read for them when trace_id == 0 (processor.rs:1401-1419) — hence exactly 2 spans left.

v87 avoided this because the Go extension guards the same fallback with rootRegex = "Root=1-[0-9a-fA-F]{8}-00000000[0-9a-fA-F]{16}" (datadog-agent, carriers.go:37), rejecting AWS-generated ids and returning no headers, so the tracer sampled for itself.

Why that flag isn't ours to honor

Two findings that shaped this version of the fix:

The header gets planted with no X-Ray involvement at all. The AWS SDKs' recursion-detection middleware copies _X_AMZN_TRACE_ID onto every outbound request inside Lambda (aws-sdk-go-v2/aws/middleware/recursion_detection.go:39-50), and SQS stores it as the AWSTraceHeader system attribute. So any Lambda→SQS→Lambda chain carries one, no X-Ray SDK and no tracing config required — and its Sampled is 0 precisely when the producer has X-Ray tracing off, which is the default. Measured on a live pair: producer PassThrough → Sampled=0 → consumer's whole trace p0; producer Active → Sampled=1 → kept. The consuming function's own tracing mode is irrelevant, since the value is written by the producer at SendMessage time.

dd-trace-java gates the same path on the id shape. It injects Root=1-<epoch>-00000000<16 hex> (XRayHttpCodec.java:41, TRACE_ID_PADDING) and on extraction ignores any header lacking that padding (:200-205, "header doesn't match our padded version"). Only after that check does it map Sampled != 1 to a drop. The Go extension's regex is the same assertion.

So a padded root id carries a Datadog decision, and Sampled=0 there should still drop. An unpadded one carries X-Ray's decision — or, with tracing off, nobody's.

Fix

Gate the priority on the id shape instead of discarding it wholesale:

Root id	Sampled	priority
…-00000000<16 hex> (Datadog-planted)	0	0 — our own decision, honored
…-00000000<16 hex>	1	1
AWS-generated	1	1
AWS-generated	0	unset — the tracer decides

Trace ids still come from the header in every row, so correlation is unchanged. This keeps v87 / dd-trace-java behavior for Datadog-planted headers while fixing the reported drop, and unlike porting the regex it leaves #452's AWS-native correlation intact.

Also switches the root-id slice to a checked get: a truncated Root= part would have panicked on the old fixed index.

Open questions

• Should an AWS-generated Sampled=1 be honored as a keep at all (row 3)? It only over-retains, but it is still X-Ray driving Datadog sampling. A DD_MERGE_XRAY_TRACES-style gate would cover that, and also cover adopting the AWS trace id for correlation in the first place. Neither is gated today: xray isn't a TracePropagationStyle in dd-trace-rs, and this fallback sits outside the propagator, so no DD_TRACE_PROPAGATION_STYLE setting reaches it — extract_generated_span_context takes no config argument.
• Behavior change: traces X-Ray sampled out are now subject to the tracer's own sampling, so ingestion rises for anyone who relied on that suppression.
• Not directly observed: that the reported case's header is AWS-generated. The flare never logs the raw value. The middleware above makes it near-certain for any Lambda→SQS chain, but a raw SQS record would close it.

Correcting my earlier note on this PR: I said porting Go's regex would regress #452's deliberate acceptance of AWS-native ids. Having read XRayHttpCodec, that fixture's unpadded root id cannot have come from dd-trace-java's injector, so the missing guard looks like an oversight rather than intent.

Testing

cargo test --lib 546 pass, clippy -D warnings clean, fmt clean. One case per row of the table above, plus a truncated-header case for the slice.

Verified on a live Lambda (Go, arm64, provided.al2023, SQS trigger, us-east-2, datadog-lambda-go v1.32.0 + dd-trace-go/v2 v2.9.2 to match the reported setup), sending each header shape with no _datadog attribute. Counts are distinct spans — the repro trace has 7, and DD_TRACE_DEBUG logs each twice.

Root id	Sampled	stock Datadog-Extension-ARM:98	this branch
…-00000000<16 hex>	0	all 7 spans prio=0	all 7 spans prio=0 (unchanged)
…-00000000<16 hex>	1	all 7 spans prio=1	all 7 spans prio=1 (unchanged)
AWS-generated	0	all 7 spans prio=0	all 7 spans prio=1
AWS-generated	1	all 7 spans prio=1	all 7 spans prio=1 (unchanged)

All four log Returning generated span context, and the tracer's trace id equals the header's low 64 bits in every row, so the fallback still runs and correlation is intact:

padded   Sampled=0  Root=1-7cab16bf-00000000cb856c5bd45f57c1 -> 14665246903262730177 == dd.trace_id
padded   Sampled=1  Root=1-794ebd8d-0000000050c0502f976f23fb ->  5818738883897009147 == dd.trace_id
unpadded Sampled=0  Root=1-9d5e9020-ee42ae1899e8e2a1c33d4539 -> 11090363266793030969 == dd.trace_id
unpadded Sampled=1  Root=1-e14003d9-cbfe0e26380cd3a67387b10d ->  4038835677700796685 == dd.trace_id

Row 1 is the case the first version of this PR got wrong: it sent priority 1 there, discarding a legitimate Datadog drop. Row 3 is the reported bug. Every row still logs Returning generated span context, so the fallback runs throughout, and the trace id still equals the header's low 64 bits.

Not re-verified: the resulting backend span count (the reported "14 → 2"). Traces flushed fine, but the sandbox API key's org isn't one I can query, so I could not count spans in the UI. That step follows from p0 traces being dropped rather than from a measurement here.

🤖 Generated with Claude Code

[datadog-prod-us1-5]

 

⚠️ Warnings

🚦 3 Pipeline jobs failed

Bottlecap (Rust) | Audit

View in Datadog · View in GitHub Actions

Critical vulnerabilities were found, marking check as failed.

DataDog/datadog-lambda-extension | e2e-test-status (amd64)

View in Datadog · View in GitLab

DataDog/datadog-lambda-extension | e2e-test-status (amd64, fips)

View in Datadog · View in GitLab

ℹ️ Info

🔄 Datadog auto-retried 1 job - 1 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e93c41c | Docs | View more details | Give us feedback!}