diff --git a/docs/custom-metric-filter.md b/docs/custom-metric-filter.md index 115d249c87..2a713a04b3 100644 --- a/docs/custom-metric-filter.md +++ b/docs/custom-metric-filter.md @@ -307,9 +307,77 @@ curl http://localhost:8675/metrics - **Config Sync v1.11.x and earlier**: Uses `stackdriver` exporter (deprecated) - **Config Sync v1.8+**: Custom monitoring support available +## Disabling Monitoring Telemetry + +Starting in Config Sync v1.25, users can opt out of metric collection to save resource consumption (CPU/memory) and reduce telemetry overheads. + +### Option 1: Per-Sync Declarative Disabling (RootSync / RepoSync) + +You can turn off monitoring for individual `RootSync` or `RepoSync` objects by setting `spec.monitoring.enabled: false` in their specification: + +> **Note**: The default `RootSync` object named `root-sync` created by Google Cloud management tools (like the Google Cloud console, Google Cloud CLI, or Terraform) does not support the `spec.monitoring.enabled` field. Any changes to the default `root-sync` object will be automatically reverted by the management controller. You can set `spec.monitoring.enabled: false` on custom `RootSync` objects (with names other than `root-sync`) or any `RepoSync` object. + +```yaml +apiVersion: configsync.gke.io/v1beta1 +kind: RootSync # or RepoSync +metadata: + name: SYNC_NAME + namespace: SYNC_NAMESPACE +spec: + monitoring: + enabled: false +``` + +When `spec.monitoring.enabled: false` is set: +- The `otel-agent` sidecar container is omitted from the reconciler Pod, saving container memory and CPU requests/limits. +- Metric collection and exporting for that reconciler are turned off. + +### Option 2: Complete Cluster-Wide Removal (OSS Administrators) + +If an open-source cluster administrator wants to safely delete the `config-management-monitoring` namespace entirely and stop all OpenTelemetry controller activity: + +1. Set the `DISABLE_MONITORING=true` environment variable on both `reconciler-manager` and `resource-group-controller-manager` Deployments using `kubectl set env` (or by updating their Deployment manifests): + +```shell +kubectl set env deployment/reconciler-manager -n config-management-system DISABLE_MONITORING=true +kubectl set env deployment/resource-group-controller-manager -n resource-group-system DISABLE_MONITORING=true +``` + +Alternatively, inject the environment variable directly into the Deployment manifests: + +```yaml +env: +- name: DISABLE_MONITORING + value: "true" +``` + +2. Delete the monitoring namespace: + +> **Note**: Before deleting the `config-management-monitoring` namespace, verify that the `DISABLE_MONITORING=true` rollout has completed on the manager controllers (so that active reconciler Pods no longer contain the `otel-agent` container) to ensure no reconcilers attempt to export metrics to the deleted monitoring namespace. + +```shell +kubectl delete ns config-management-monitoring +``` + +### Benchmark & Resource Savings (500 RepoSync Scale) + +The following benchmark measures the memory savings from disabling the OpenTelemetry (`otel-agent`) sidecar on a cluster with 500 `RepoSync` objects across 500 namespaces on an 8-node GKE cluster: + +| Metric / Scope | Monitoring Enabled (Baseline) | Monitoring Disabled (Optimized) | Total Memory Saved | Average Savings per RepoSync | +|----------------|-------------------------------|----------------------------------|--------------------|------------------------------| +| **Total Memory (`config-management-system`)** | 42,187 Mi (~41.20 GiB) | 25,542 Mi (~24.94 GiB) | **16,645 Mi (~16.25 GiB)** | **~33.29 MiB (-39.5%)** | +| **Containers per Reconciler Pod** | 3/3 (`reconciler`, `git-sync`, `otel-agent`) | 2/2 (`reconciler`, `git-sync`) | **-1 Sidecar Container** | **-100% Sidecar Overhead** | +| **Telemetry Network Overhead** | Active gRPC to `:4317` | None (Noop) | **Zero Exporter Overhead** | **100% Telemetry Offloaded** | + +#### Benchmark Methodology & Highlights +- **Environment**: 8-node GKE cluster running 500 active `RepoSync` instances across 500 namespaces (`test-ns-1` through `test-ns-500`). +- **Measurement**: Raw aggregated Pod memory footprint measured via `kubectl top pods -n config-management-system`. +- **Key Finding**: Disabling the OpenTelemetry sidecar across 500 `RepoSync` reconcilers reclaims **16,645 Mi (~16.25 GiB)** of memory back to the cluster (saving **~33.29 MiB** per reconciler Pod). + ## Additional Resources - [OpenTelemetry Collector Documentation](https://opentelemetry.io/docs/collector/) - [Google Cloud Monitoring Documentation](https://cloud.google.com/monitoring) - [Config Sync Monitoring Guide](http://cloud/anthos-config-management/docs/how-to/monitoring-config-sync) - [Available Config Sync Metrics](http://cloud/anthos-config-management/docs/how-to/monitoring-config-sync#metrics) +