Skip to content

openmetrics: fix native transformer locking to first observed metric type#24299

Draft
lucia-sb wants to merge 4 commits into
masterfrom
lucia-sb/fix-native-dynamic
Draft

openmetrics: fix native transformer locking to first observed metric type#24299
lucia-sb wants to merge 4 commits into
masterfrom
lucia-sb/fix-native-dynamic

Conversation

@lucia-sb

@lucia-sb lucia-sb commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes the default native OpenMetrics V2 transformer so it no longer locks onto whichever metric type it sees first for a given metric name. It now reuses native_dynamic's per-type caching, so a metric name that legitimately reports more than one type (e.g. a gauge/counter pair colliding after OpenMetrics strips the _total suffix) gets each sample routed to the correct transformer regardless of arrival order.

Adds a regression test (test_v2/test_transformers/test_native.py) covering both the plain single-type case and the gauge/_total-counter collision.

Motivation

AI-6931: several integrations (krakend, kuma, coredns v2, istio, gitlab; argocd has a separate broken custom transformer) mis-submit go_memstats_alloc_bytes because Go exposes it as both a gauge and a _total counter. OpenMetrics strips the _total suffix from counter family names, so both samples collapse into the same metric name. The default native transformer cached the first sample's type (transformer = None, set once) and reused it forever, so the gauge's value gets silently overwritten by the counter's large cumulative value, and the .count metric is never emitted.

native_dynamic already handles this correctly by caching a transformer per observed type instead of a single slot. This PR makes native — the default type for every metric not given an explicit type — share that same per-type caching, fixing the collision at the base-check level instead of requiring a native_dynamic annotation in every affected integration.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

🤖 Generated with Claude Code

…ric type

OpenMetrics strips the `_total` suffix from counter family names, so a gauge
and its `_total` counter counterpart (e.g. go_memstats_alloc_bytes) collapse
into the same metric name. The default `native` transformer cached whichever
type it saw first and reused it forever, so one of the two samples always got
submitted under the wrong type. `native` now shares native_dynamic's per-type
caching, so both samples are always submitted correctly regardless of order.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@lucia-sb lucia-sb added the qa/required QA is required for this PR and will generate a QA card label Jul 2, 2026
@datadog-official

datadog-official Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests  Code Coverage

Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

PR All | test / j06ca546 / SNMP   View in Datadog   GitHub Actions

🧪 1 Test failed in 1 job

PR All | run   GitHub Actions

All test failures are known flaky — job may pass on retry.

❄️ Known flaky: test_e2e_snmp_listener from test_e2e_snmp_listener.py   View in Datadog
Needed at least 1 candidates for &#39;datadog.snmp.check_duration&#39;, got 0
Expected:
        MetricStub(name=&#39;datadog.snmp.check_duration&#39;, type=0, value=None, tags=[&#39;autodiscovery_subnet:172.18.0.0/28&#39;, &#39;device_vendor:apc&#39;, &#39;firmware_version:2.0.3-test&#39;, &#39;loader:python&#39;, &#39;model:APC Smart-UPS 600&#39;, &#39;serial_num:test_serial&#39;, &#39;snmp_device:172.18.0.1&#39;, &#39;snmp_profile:apc_ups&#39;, &#39;ups_name:testIdentName&#39;], hostname=None, device=None, flush_first_value=None)
Difference to closest:
        Expected tag snmp_device:172.18.0.1
        Found snmp_device:172.18.0.2

Similar submitted:
Score   Most similar
1.00    MetricStub(name=&#39;datadog.snmp.check_duration&#39;, type=0, value=0.2207350730895996, tags=[&#39;autodiscovery_subnet:172.18.0.0/28&#39;, &#39;device_vendor:apc&#39;, &#39;firmware_version:2.0.3-test&#39;, &#39;loader:python&#39;, &#39;model:APC Smart-UPS 600&#39;, &#39;serial_num:test_serial&#39;, &#39;snmp_device:172.18.0.2&#39;, &#39;snmp_profile:apc_ups&#39;, &#39;ups_name:testIdentName&#39;], hostname=&#39;runnervmfmtub&#39;, device=None, flush_first_value=False)
...

Not introduced in this PR.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 2 jobs - 2 passed on retry View in Datadog

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 88.12% (+0.11%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2865834 | Docs | Datadog PR Page | Give us feedback!

…ge/_total-counter collisions

The base-check fix in this PR (native transformer no longer locking to the
first observed metric type) now correctly emits the counter side of a
gauge/_total-counter collision that was previously silently dropped. Update
metadata.csv and test metric lists to reflect the newly-correct metrics:
go.memstats.alloc_bytes.count (coredns, istio, krakend, kuma), the mesh-agent
variant (istio), and gitaly.go.memstats_alloc_bytes.count plus a second,
previously unreported collision, gitaly.catfile_processes.count (gitlab).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
It only forwarded to get_native_dynamic_transformer. Point the 'native'
TRANSFORMERS entry at get_native_dynamic_transformer directly instead of
calling one factory from the other.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 21 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
qa-label Validate the pull request declares whether it needs QA for the next Agent release
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant