Add configurable log levels for Test Optimization tooling

[tonyredondo]

Why

The uploader currently prints one INFO message for every cached or non-current test output it skips. On a large workspace, those paths bury the useful results. There is also no single way to reduce diagnostics across metadata sync, doctor, and the uploader.

This PR adds DD_TEST_OPTIMIZATION_LOG_LEVEL and moves individual cached-output paths to DEBUG. INFO keeps a count of skipped outputs and the upload summary.

Level	What is printed
ERROR	Failures and configuration errors
WARN	Errors and recoverable warnings
INFO	Progress and summaries, plus warnings and errors; the default
DEBUG	Detailed diagnostics, including individual cached-output paths

Behavior

The setting covers metadata-sync diagnostics, doctor, schema validation, and the Python, Bash, and PowerShell uploaders. Values are case-insensitive. An explicit level overrides the existing debug attributes, environment switches, and uploader --debug; those options still work when the new variable is unset. Invalid values fail with a configuration error.

For example:

export DD_TEST_OPTIMIZATION_LOG_LEVEL=WARN
bazel test --config=test-optimization \
  --repo_env=DD_TEST_OPTIMIZATION_LOG_LEVEL //pkg:test.topt
bazel run --config=test-optimization \
  --repo_env=DD_TEST_OPTIMIZATION_LOG_LEVEL //tools/test_optimization:dd_test_optimization_doctor
bazel run --config=test-optimization \
  --repo_env=DD_TEST_OPTIMIZATION_LOG_LEVEL //tools/test_optimization:dd_upload_payloads

The variable controls human-readable diagnostics on their existing streams. It does not filter JSON reports or helper return values, change payload selection or uploads, or change exit codes. Errors remain visible at every level, and existing secret redaction stays in place.

It is not forwarded to tests or build actions. Changing a tracked repository environment value can still cause Bazel to reevaluate and refetch metadata; the regression test checks that identical backend responses produce identical test inputs at all four levels. Post-test telemetry request timings are excluded from that comparison. Uploader analysis-time messages are removed so runtime logging does not require an environment-dependent analysis setting.

Bazel output, compiler/Orchestrion subprocess diagnostics, explicit bootstrap probe records, and test/tracer logs retain their own controls. There are no GitLab folding or consumer-repository changes in this PR.

Validation

• Root Bazel suite: 511 tests passed.
• Final Python Bazel target: 529 cases, with one existing skip; isolated runpy startup is covered too.
• CI's coverage entrypoint passed locally: 59.17% against its 40% threshold.
• Regression coverage: severity filtering, legacy-option precedence, invalid values, error visibility, secret redaction, machine-readable stdout, and identical payload selection/JSON reports across all four levels.
• A 1,000-output regression checks that INFO emits only two freshness lines, while DEBUG retains all individual paths.
• Real metadata-sync integration checks all four levels and compares generated test-input hashes.
• PowerShell mock-server integration passed, including the generated legacy runtime.
• Local-archive Bzlmod Go E2E in rules_test_optimization_tests passed: disabled mode, ordinary and optimized targets, doctor, static services, and uploads to the mock backend.
• Buildifier, uploader-template lint, ShellCheck error checks, and git diff --check passed.

The full local Bash harness reached its guided Go bootstrap fixture but stopped because dd-trace-go v2.9.1 did not expose internal/civisibility/integrations/gotesting. The same failure reproduces in a separate, unmodified worktree at base commit 42caa5c, so it is not introduced by this change. The preceding logging and uploader scenarios passed. The full local harness is not claimed as green; the complete Ubuntu harness passed in PR CI.

Release readiness

• Updated configuration, uploader reference, and README.
• No new dependencies or vendored changes; no license inventory update needed.
• No credentials or private CI links added.
• Logging redaction and filtering of diagnostic output have regression coverage.

[datadog-prod-us1-3]

⚠️ Autotest was unable to complete this review. View session

Please try again by commenting @autotest review.

[tonyredondo]

@autotest review

[datadog-prod-us1-3]

⚠️ Autotest was unable to complete this review. View session

Please try again by commenting @autotest review.