Add Kueue workload events#24311
Conversation
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 7768aaf | Docs | Datadog PR Page | Give us feedback! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cafad0b4aa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
mbertrone
left a comment
There was a problem hiding this comment.
LGTM. A few notes inline; the only one worth fixing before merge is the two suppression tests, which currently can't fail.
Review from mbertrone is dismissed. Related teams and files:
- gpu-monitoring-agent
- kueue/datadog_checks/kueue/check.py
- kueue/datadog_checks/kueue/kube_client.py
- kueue/tests/test_unit.py
|
|
||
| run_command(['kubectl', 'delete', 'job/event-workload', '-n', 'default', '--ignore-not-found'], env=kubectl_env) | ||
| check = KueueCheck('kueue', {}, [load_check_instance(kubeconfig_dict)]) | ||
| check._parse_workload_events_config() |
There was a problem hiding this comment.
issue (blocking): this call blows up with 'NoneType' object has no attribute 'collect_workload_events' (the failure CI is already reporting for test_e2e_workload_events). The config-model fix from this review round changed _parse_workload_events_config to read self.config.collect_workload_events (check.py:148), but self.config returns self._config_model_instance, which AgentCheck.__init__ initializes to None and only load_configuration_models populates. That initializer is registered in check_initializations and runs inside run(), so on the normal path it fires before _parse_workload_events_config. Here you call _parse_workload_events_config() by hand right after construction, before any initialization has run, so self.config is still None. Production is unaffected; only this manual invocation trips it.
Load the model first:
check = KueueCheck('kueue', {}, [load_check_instance(kubeconfig_dict)])
check.load_configuration_models()
check._parse_workload_events_config()
check.check(check.instance)
Validation ReportAll 21 validations passed. Show details
|
* Add workload events * Enable Kueue workload events by default * List Kueue workloads by namespace * Harden Kueue workload event E2E tests * Synchronize Kueue config models * Add Kueue workload events changelog * Fix instace * Address Kueue workload event review feedback. * Fix Kueue workload event config parsing. * Fix Kueue workload event kubeconfig handling. * Add Kueue topology domain metrics. * Update Kueue changelog entry. * Fix Kueue topology validation warning. * Revert "Fix Kueue topology validation warning." This reverts commit db191bd. * Revert "Update Kueue changelog entry." This reverts commit af1715c. * Revert "Add Kueue topology domain metrics." This reverts commit 449e60a. 9274b10
What does this PR do?
Adds optional Kueue Workload lifecycle event collection for created, quota reserved, admitted, running, evicted, and finished transitions.
Motivation
Surface workload scheduling and eviction changes in Datadog events for operators monitoring Kueue queues.
Review checklist (to be filled by reviewers)
qa/requiredif this PR needs QA validation, orqa/skip-qaif it does not. Exactly one of the two is required.backport/<branch-name>label to the PR and it will automatically open a backport PR once this one is merged