[SDTEST-3873] Add internal telemetry metrics

[anmarchenko]

What

• Added a narrow, metrics-only internal/telemetry package based on the CI Visibility telemetry implementation in dd-trace-go.
  • Supports count and distribution metrics with deterministic tag handling.
  • Aggregates concurrent submissions and restores buffered values after failed or cancelled flushes.
  • Sends count and distribution payloads together using the telemetry v2 message-batch request type when both are present.
  • Sends no app-started, lifecycle, configuration, dependency, integration, log, or heartbeat events.
• Added the telemetry v2 request envelope and headers with a per-process runtime ID, sequence IDs, host/runtime metadata, the configured ddtest service name, DD_ENV, the ddtest build version, and the constant language name ddtest.
• Added Agent and agentless telemetry delivery.
  • Agent mode uses /telemetry/proxy/api/v2/apmtelemetry and supports HTTP(S) and Unix-socket Agent URLs.
  • Agentless mode uses the instrumentation telemetry intake for DD_SITE, or DD_CIVISIBILITY_AGENTLESS_URL when a test intake override is configured.
  • API keys are attached only for agentless requests.
• Extracted shared Datadog connection discovery and HTTP transport construction into internal/httptransport, then reused it from both Test Optimization and telemetry to keep Agent URL, Unix socket, site, API key, and agentless test URL handling consistent.
• Created one telemetry client per plan or test command, passed the same client through the runner, planner, Test Optimization API transport, and Git command runner, and flushed it after the command completes. Client creation and flush failures remain best-effort and do not replace the command result.
• Added CI Visibility metrics matching the current dd-trace-go names and tags where they apply to ddtest:
  • Search-commits, object-pack, settings, skippable-tests, known-tests, and test-management request counts, errors, durations, response sizes, and response item counts.
  • Settings response feature flags.
  • Aggregate itr_skipped counts tagged as event_type:test or event_type:suite. ddtest does not emit itr_unskippable or itr_forced_run.
  • Low-level Git command counts, errors, and durations for the Git operations used by repository upload, with canonical command and exit-code tags. Existing package-level Git functions retain their non-telemetry behavior.
• Added unit and integration-style coverage for metric aggregation, batching, request metadata, Agent/agentless routing, retries, concurrency, API metrics, planner counts, Git metrics, and end-to-end dependency wiring.

Why

ddtest performs Test Optimization API requests, repository uploads, TIA planning, and supporting Git operations in its own process. Those operations need internal CI Visibility metrics so their volume, latency, response sizes, and failures can be monitored independently of the language tracer running the tests.

The implementation intentionally copies only the metrics functionality needed by ddtest. It uses the optional telemetry message-batch envelope to avoid duplicating the common request header when flushing both count and distribution payloads, while excluding unrelated application lifecycle events.

Sharing connection discovery with the existing Test Optimization transport also prevents the telemetry and API clients from resolving Agent, Unix-socket, site, API-key, and test-intake configuration differently.

E2E testing

• Ran make test successfully.
• Ran make lint successfully with zero issues.
• Manual scenario:
  1. Start a local mock intake that implements the Test Optimization settings/skippables responses and records telemetry POST requests.
  2. Set DD_CIVISIBILITY_AGENTLESS_ENABLED=true, DD_API_KEY=test, DD_CIVISIBILITY_AGENTLESS_URL=http://127.0.0.1:<port>, DD_SERVICE=<service>, and DD_ENV=e2e.
  3. Run go run . plan in a supported test project.
  4. Verify a request reaches /api/v2/apmtelemetry, uses language_name: ddtest and the configured service, contains only generate-metrics/distributions payloads (wrapped in message-batch when both exist), and includes the expected API, ITR, and Git metrics.

[datadog-datadog-prod-us1-2]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 91.23%
• Overall Coverage: 87.87% (+0.42%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bfa53c2 | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a58d07e15

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-datadog-prod-us1-2]

More details

The telemetry transport, aggregation and restoration paths behaved correctly across Agent and agentless routing, retries, cancellation, malformed configuration, concurrent submissions, and realistic metric payload shapes. No production-impacting regression was reproducible; lint remains unverified because the installed golangci-lint binary targets Go 1.24 while the repository targets Go 1.26.3.

Was this helpful? React 👍 or 👎

_{📊 Validated against 10 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 6a58d07 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[anmarchenko]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 16e13e7291

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[anmarchenko]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e2362f36b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[anmarchenko]

E2E Test Report: SUCCESS ✅

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

• Final revision tested: bfa53c2256e72c6429442b6c040b1346bc67b2dc
• Dependency: ddtest=anmarchenko/add-internal-telemetry-metrics
• Local: RuboCop, Vagrant, Forem, and focused Middleman playground scenarios with mockdog; debug logging enabled
• GitHub Actions: Forem feature branch, building the exact PR revision in every job
• Final GHA run: Forem CI run 31176376799

Results

Check	Status	Evidence
Build and inject current PR head	✅	Exact bfa53c22 revision built locally and in the final Forem GHA jobs
Full ddtest run completes	✅	Local focused RSpec run passed; Forem build and selected RSpec shard passed
Agentless and agent telemetry routing	✅	Correct direct and proxy endpoints; API key present only for agentless routing
Metrics-only telemetry batching	✅	message-batch contained generate-metrics and distributions
Application and host metadata	✅	language_name: ddtest, version, runtime/session ID, service, environment, and host metadata populated
Telemetry opt-out	✅	DD_INSTRUMENTATION_TELEMETRY_ENABLED=false produced no ddtest telemetry
Distribution payload schema	✅	Top-level namespace: civisibility is present
Telemetry session correlation	✅	DD-Session-ID matches the payload runtime_id
Object-pack telemetry	✅	Mockdog captured git_requests.objects_pack=1 plus _ms, _bytes, and _files distributions when a packfile was uploaded
Forem suite-level TIA behavior	✅	Final run fetched 0 tests and 830 suites, skipped 830 of 836 files, and executed only 6 files
No unskippable telemetry	✅	No itr_unskippable or itr_forced_run series were emitted
Suite skippable-response metric	✅	Suite-only mockdog response emitted itr_skippable_tests.response_suites=1; response_tests was absent
Suite skip telemetry	✅	itr_skipped=1 was emitted with event_type:suite

Final GitHub Actions Verification

• Revision: bfa53c2256e72c6429442b6c040b1346bc67b2dc
• Workflow commit: 79793b7bb19478a26eb33768b43c63a13bf6ce15
• ddtest fetched 0 test entities and 830 suite entities.
• TIA applied 830 suite skips, fully skipping 830 of 836 files.
• One runner was selected and executed 6 files.
• RSpec completed with 18 examples, 0 failures, 2 pending.
• Suites marked unskippable: 0.

Issue Found and Resolution

During testing of revision a630e81672805f83f3e393c2b61aca769944bd56, a suite-only backend response was incorrectly reported as itr_skippable_tests.response_tests=1; itr_skippable_tests.response_suites was absent. The same classification error was visible in the Datadog dashboard for the Forem live run.

The final revision bfa53c2256e72c6429442b6c040b1346bc67b2dc resolves the issue by defining the suite response metric and selecting response_suites for suite-level TIA responses. A focused local regression run against mockdog confirmed:

• backend response: 0 tests, 1 suite;
• planning: 1 suite skip applied, 1 of 2 files fully skipped, 50% estimated time saved;
• raw telemetry: itr_skippable_tests.response_suites=1;
• raw telemetry: no itr_skippable_tests.response_tests series;
• skip telemetry: itr_skipped=1 with event_type:suite;
• no unskippable or forced-run telemetry;
• all 10 focused plan assertions passed.

Verification

• Mockdog captured the exact telemetry request bodies, headers, namespaces, metric values, distributions, and tags.
• The final focused mockdog payload proves the response entity is now classified as a suite, closing the reported regression.
• A real packfile upload produced the expected object-pack count and distribution metrics.
• The Forem GHA run verified the exact final PR binary against live Test Optimization settings and suite-level TIA responses; all jobs passed.
• The original Datadog UI symptom was reproduced before the fix. Final metric classification was verified deterministically from the raw post-fix mockdog payload; Shepherd did not independently reopen the dashboard after the final run.

Test Methodology

1. Exercised ddtest telemetry locally through direct and proxy mockdog endpoints, including opt-out and metadata checks.
2. Verified object-pack telemetry using a real git search/packfile request sequence.
3. Ran cumulative Forem source changes in GitHub Actions to exercise live suite-level TIA planning and execution.
4. Reproduced the suite-response classification bug locally with a suite-only mockdog response and correlated it with the Datadog dashboard symptom.
5. Retested the final fix locally with a focused suite-mode fixture and inspected mockdog's parsed metrics and raw wire payload.
6. Retested the final PR revision once on the Forem feature branch and verified 830 suite skips with a passing RSpec shard.

---

This E2E test was performed by Shepherd — autonomous QA for Datadog Test Optimization.