kafka_consumer: add connector failure trace and tracked topics to Connect config events by piochelepiotr · Pull Request #24309 · DataDog/integrations-core

piochelepiotr · 2026-07-02T21:21:28Z

What does this PR do?

Adds two fields to the existing data-streams-message connector config event emitted by the kafka_consumer integration's Kafka Connect collector:

connector_trace / task_traces: the Kafka Connect status trace (stack trace text) for a connector or task in the FAILED state.
topics: the list of topics tracked for the connector via the Kafka Connect topics-tracking REST API (GET /connectors/{name}/topics), fetched in parallel across connectors and refreshed on the same cache/TTL cadence as connector plugin metadata to avoid extra load on the Connect worker.

It also changes how connector config is redacted before being included in the event: instead of hiding keys that match known-sensitive patterns, only keys on an allowlist of known-safe Connect framework and popular connector-plugin settings (JDBC, Debezium, S3/GCS/BigQuery/Snowflake/Elasticsearch sinks, HTTP Sink, MirrorMaker2) are shown; everything else is hidden by default. A substring safety net additionally hides any key that looks credential-shaped even if it's allowlisted.

Motivation

A dashboard built on top of kafka_consumer data currently has to cross-reference the Connect worker directly to find out why a connector failed or which topics it touches. Surfacing the failure trace and topic list in the config event closes that gap.

Switching config redaction from a denylist to an allowlist reduces the risk of leaking a credential through a connector-plugin-specific key we didn't know to redact.

Test plan

Unit tests cover trace/topic fields on connector events, parallel topic fetching, and the allowlist (including known connector-plugin keys and the substring safety net).

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

This PR has been created and validated using the paired-review skill from agent-integrations. Ready for human review.

datadog-prod-us1-3 · 2026-07-02T21:32:11Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 92.19%

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 7477f20 | Docs | Datadog PR Page | Give us feedback!}

…nnect config events Surfaces the Kafka Connect status trace for FAILED connectors/tasks and the connector's tracked topics (via the Connect topics-tracking API) in the existing data-streams-message connector config event, so failures and topic usage are visible without cross-referencing the Connect worker directly.

… state - Add CacheHelper.get_cached_json/set_cached_json to remove duplicated try/except read/write boilerplate; existing cache methods now delegate. - Bound the topics fetch/data caches with max_cache_size and prune stale connector entries on write, matching the connector-config cache. - Only mark a connector's topics as fetched when the request succeeds, so a transient failure retries on the next run instead of waiting a full TTL. - Hoist the topics fetch out of _emit_connector_config_events into _collect_rest, alongside the other outbound Connect HTTP calls. - Only report connector_trace when connector_state is FAILED. - Add tests for topics TTL caching and fetch-failure degradation.

Backs off persistently-failing connector topic fetches on the same TTL as successes to avoid a per-run retry storm and log spam. Extracts a _build_connector_event_content helper and a _url_cache_key helper to reduce duplication, sorts task_traces for stable event hashing, and tightens CacheHelper's new JSON cache signatures. Also fixes a line-length formatting violation introduced by the round-1 fix commit.

…nstant, unconditional topic-cache pruning

…ck tests

…tor and Any typing

…substring safety net Cover well-known non-sensitive keys from JDBC, Debezium, S3/GCS/BigQuery/ Snowflake/Elasticsearch/HTTP sinks, and MirrorMaker2, so legitimate connector config isn't hidden unnecessarily. Also hide any key matching a credential-shaped substring (password, secret, token, etc.) even if allowlisted, as a backstop against curation mistakes.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: de5a375ec9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…n allowlist helper Cap connector topic refreshes at 100 per run so a Connect worker with many expired connectors, or a slow/unsupported topics endpoint, can't stall config/plugin event emission for the rest of the run. Leftover connectors are picked up on a later run since get_items_to_fetch returns oldest-expiry-first. Rename _is_allowed_config_key to is_allowed_config_key: it's a plain module-level predicate, not a reusable-library-module helper, so AGENTS.md's naming convention says no leading underscore.

- task_traces: use a None-safe sort key so a missing task id can't raise TypeError and drop all config/plugin events for the Connect URL. - topics fetch: back off only successful fetches and permanently-unsupported (404/405) endpoints; retry transient failures on the next run, matching _collect_plugins. Only rewrite the topics data cache when it actually changed. - cache: delegate get_items_to_fetch/mark_items_fetched/get_events_to_send to the new get_cached_json/set_cached_json helpers instead of duplicating the read/write-with-try/except blocks. - Move ALLOWED_CONFIG_KEYS/SENSITIVE_KEY_SUBSTRINGS/is_allowed_config_key into connector_config_keys.py so connectors.py stays focused on collection. - Add a .changed changelog fragment for the denylist->allowlist redaction flip. - Add tests for transient-vs-unsupported topics backoff and None task ids. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-17T15:39:48Z

⚠️ Major version bump
The changelog type changed or removed was used in this Pull Request, so the next release will bump major version. Please make sure this is a breaking change, or use the fixed or added type instead.

dd-octo-sts · 2026-07-17T15:40:55Z

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
`agent-reqs`	Verify check versions match the Agent requirements file	✅
`ci`	Validate CI configuration and code coverage settings	✅
`codeowners`	Validate every integration has a CODEOWNERS entry	✅
`config`	Validate default configuration files against spec.yaml	✅
`dep`	Verify dependency pins are consistent and Agent-compatible	✅
`http`	Validate integrations use the HTTP wrapper correctly	✅
`imports`	Validate check imports do not use deprecated modules	✅
`integration-style`	Validate check code style conventions	✅
`jmx-metrics`	Validate JMX metrics definition files and config	✅
`labeler`	Validate PR labeler config matches integration directories	✅
`legacy-signature`	Validate no integration uses the legacy Agent check signature	✅
`license-headers`	Validate Python files have proper license headers	✅
`licenses`	Validate third-party license attribution list	✅
`metadata`	Validate metadata.csv metric definitions	✅
`models`	Validate configuration data models match spec.yaml	✅
`openmetrics`	Validate OpenMetrics integrations disable the metric limit	✅
`package`	Validate Python package metadata and naming	✅
`qa-label`	Validate the pull request declares whether it needs QA for the next Agent release	✅
`readmes`	Validate README files have required sections	✅
`saved-views`	Validate saved view JSON file structure and fields	✅
`version`	Validate version consistency between package and changelog	✅

View full run

sarah-witt · 2026-07-18T00:01:05Z

+
+    def _fetch_connector_topics(self, url: str, connector_names: list[str]) -> dict[str, list[str]]:
+        """Return each connector's tracked topics, refreshing entries whose TTL has expired.
+


What's an expected number of topics per connector?

sarah-witt · 2026-07-18T00:13:27Z

+                    except Exception as e:
+                        if self._is_unsupported_topics_endpoint(e):
+                            names_to_back_off.append(name)
+                        self.log.warning("Error fetching topics for connector %s from %s: %s", name, url, e)


Should this always be a warning if sometimes we expect a 404 or 405 error?

temporal-github-worker-1 Bot added agent/review-requested ecosystems/review-requested product/review-requested labels Jul 2, 2026

dd-octo-sts Bot added the integration/kafka_consumer label Jul 2, 2026

piochelepiotr added the qa/required QA is required for this PR and will generate a QA card label Jul 2, 2026

piochelepiotr added a commit that referenced this pull request Jul 2, 2026

[kafka_consumer] Add changelog entry for PR #24309

26982d0

piochelepiotr added 6 commits July 6, 2026 10:37

[kafka_consumer] Add changelog entry for PR #24309

04601a7

Address round-3 review feedback: static helper, generic cache-size co…

b581b04

…nstant, unconditional topic-cache pruning

Exclude topics fetches from connectors_fetches count in expand-fallba…

0a0ed77

…ck tests

piochelepiotr force-pushed the piotr.wolski/kafka-connect-topics-and-error-trace branch from d3578db to 0a0ed77 Compare July 6, 2026 16:40

piochelepiotr added 4 commits July 6, 2026 10:46

Parallelize per-connector topic fetches, drop unneeded cache.py refac…

33bdf3d

…tor and Any typing

Redact connector config with an allowlist instead of a denylist

8c554f1

[kafka_actions] Remove restated-by-code comments in connectors.py

de5a375

piochelepiotr marked this pull request as ready for review July 6, 2026 17:29

piochelepiotr requested a review from a team as a code owner July 6, 2026 17:29

dd-octo-sts Bot added the team/agent-integrations label Jul 6, 2026

chatgpt-codex-connector Bot reviewed Jul 6, 2026

View reviewed changes

Comment thread kafka_consumer/datadog_checks/kafka_consumer/connectors.py Outdated

Comment thread kafka_consumer/datadog_checks/kafka_consumer/connectors.py Outdated

rkaneriya approved these changes Jul 6, 2026

View reviewed changes

piochelepiotr added this to the 7.82.0 milestone Jul 6, 2026

sarah-witt reviewed Jul 18, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

kafka_consumer: add connector failure trace and tracked topics to Connect config events#24309

kafka_consumer: add connector failure trace and tracked topics to Connect config events#24309
piochelepiotr wants to merge 12 commits into
masterfrom
piotr.wolski/kafka-connect-topics-and-error-trace

piochelepiotr commented Jul 2, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-3 Bot commented Jul 2, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

dd-octo-sts Bot commented Jul 17, 2026

Uh oh!

sarah-witt Jul 18, 2026

Uh oh!

sarah-witt Jul 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		def _fetch_connector_topics(self, url: str, connector_names: list[str]) -> dict[str, list[str]]:
		"""Return each connector's tracked topics, refreshing entries whose TTL has expired.

Uh oh!

Conversation

piochelepiotr commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Test plan

Review checklist (to be filled by reviewers)

Uh oh!

datadog-prod-us1-3 Bot commented Jul 2, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 17, 2026

Uh oh!

dd-octo-sts Bot commented Jul 17, 2026

Validation Report

Uh oh!

sarah-witt Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

sarah-witt Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

piochelepiotr commented Jul 2, 2026 •

edited

Loading

datadog-prod-us1-3 Bot commented Jul 2, 2026 •

edited by datadog-official Bot

Loading