uploader: add opt-in parallel Python runtime

.github/workflows/ci.yml

@@ -4,6 +4,9 @@
 # This product includes software developed at Datadog
 # (https://www.datadoghq.com/) Copyright 2025-Present Datadog, Inc.
 
+# Validate supported hosts and generated consumer artifacts before merge.
+# The uploader smoke catches launcher/runfiles failures that imports cannot expose.
+
 name: CI
 
 on:
@@ -126,6 +129,18 @@ jobs:
         shell: bash
         run: ./bazelw test //tools/... --test_output=errors
 
+      - name: Smoke default Python uploader launcher
+        timeout-minutes: 5
+        shell: bash
+        run: |
+          ./bazelw run \
+            //tools/tests/python/fixtures/generated_uploader:generated_uploader \
+            -- \
+            --dry-run \
+            --allow-cached-payload-uploads \
+            --workers=3 \
+            --debug
+
       - name: Build examples
         timeout-minutes: 20
         shell: bash
@@ -238,6 +253,18 @@ jobs:
         shell: bash
         run: ./bazelw test //tools/... --test_output=errors
 
+      - name: Smoke default Python uploader launcher
+        timeout-minutes: 5
+        shell: bash
+        run: |
+          ./bazelw run \
+            //tools/tests/python/fixtures/generated_uploader:generated_uploader \
+            -- \
+            --dry-run \
+            --allow-cached-payload-uploads \
+            --workers=3 \
+            --debug
+
       - name: Run Bazel tests (go companion module)
         shell: bash
         run: |

README.md

@@ -1349,17 +1349,26 @@ For a generic wrapper pattern, see [Other languages (without companion macro)](#
 - **Tracer/runtime with DD Test Optimization file-mode support** - Must honor `DD_TEST_OPTIMIZATION_MANIFEST_FILE` and `DD_TEST_OPTIMIZATION_PAYLOADS_IN_FILES`
 - **rules_go v0.51.0+** (for Go importpath inference) - This repository reads `GoInfo`/`GoArchive` providers when selecting per-module payloads
 - **DD_SITE format** - Accepts bare host, app/api-prefixed host, or full URL; leading/trailing ASCII whitespace is trimmed, then normalized to `https://api.<site>`
-- **Uploader tooling (per platform)** - Required for `bazel run //:dd_upload_payloads`
-  - **Linux**: `bash`, `curl`, `find`, `stat` (GNU), `awk`, and one of `md5sum` or `shasum`
-  - **macOS**: `bash` (3.2+), `curl`, `find`, `stat` (BSD), `awk`, and one of `md5` or `shasum`
-  - **Windows**: `powershell.exe` (Windows PowerShell 5.1+ or PowerShell 7+); the uploader uses .NET `HttpClient` and is intentionally PowerShell-only (no Git Bash dependency)
+- **Default uploader tooling** - `Python 3.10+` on every platform, plus `bash`
+  on Linux/macOS or `powershell.exe` (Windows PowerShell 5.1+ or PowerShell 7+)
+  on Windows. The platform script only locates Python and Bazel runfiles; upload
+  behavior is shared.
+- **Temporary legacy rollback tooling** - Required only when
+  `use_python_uploader = False`: Linux needs `bash`, `curl`, `find`, GNU `stat`,
+  `awk`, and one of `md5sum` or `shasum`; macOS needs `bash` 3.2+, `curl`,
+  `find`, BSD `stat`, `awk`, and one of `md5` or `shasum`; Windows needs
+  `powershell.exe` and uses .NET `HttpClient`.
 
 Optional tooling:
-- **jq** (Linux/macOS) - Used to enrich test payloads with `context.json` and
-  split oversized test payloads. If missing, uploads proceed without enrichment;
-  payloads up to the 5,000,000-byte intake limit can still be sent intact, but
-  larger payloads fail because their `events` array cannot be partitioned.
-- **python3** - Used for uploader payload schema validation and Unix telemetry metadata extraction. If missing, schema validation is skipped and telemetry files fail individually with a warning.
+- **jq** (Linux/macOS, legacy rollback only) - Used by the legacy uploader to
+  enrich test payloads with `context.json` and split oversized test payloads. If
+  missing, legacy uploads proceed without enrichment; payloads up to the
+  5,000,000-byte intake limit can still be sent intact, but larger payloads fail
+  because their `events` array cannot be partitioned.
+
+See the [parallel Python uploader](docs/Uploader_Reference.md#parallel-python-uploader-default)
+for its shared cross-platform behavior. Set `use_python_uploader = False` only
+for temporary rollback to the legacy Bash or PowerShell implementation.
 
 ### Contract gate checklist
 

docs/Configuration_Reference.md

@@ -332,6 +332,8 @@ Rule: `dd_payload_uploader(...)`
 | `keep_payloads` | bool | `False` | Keep payload files after successful upload |
 | `filter_prefix` | bool | `False` | Only upload files matching `span_events_*.json` or `coverage_*.json` |
 | `gzip_payloads` | bool | `False` | Gzip test payloads before upload |
+| `workers` | int | `8` | Maximum independent payload-file workers in Python mode; `DD_TEST_OPTIMIZATION_WORKERS` and `--workers` override it at runtime |
+| `use_python_uploader` | bool | `True` | Use the default cross-platform Python uploader; set to `False` only for temporary rollback to the legacy platform runtime |
 | `data` | label_list | `[]` | Data files to include (for example, `context.json` for enrichment) |
 | `expected_targets` | string_list | `[]` | Optional exact local labels expected in the matching BEP. Fresh and cached results jointly satisfy coverage; missing results are reported while other fresh outputs continue to upload |
 | `expected_targets_file` | label | unset | Optional schema-v1 exact-target file. Static and file inputs must match when both are non-empty; missing results do not block other fresh uploads |
@@ -659,6 +661,7 @@ The doctor and/or uploader runtimes read these variables at `bazel run` time:
 | `DD_TEST_OPTIMIZATION_FILTER_PREFIX` | `0` uploads all payloads; `1` restricts to `span_events_*.json` / `coverage_*.json` |
 | `DD_TEST_OPTIMIZATION_DEBUG` | Enable verbose uploader logs |
 | `DD_TEST_OPTIMIZATION_GZIP` | Gzip test payloads before upload |
+| `DD_TEST_OPTIMIZATION_WORKERS` | Override the maximum independent payload-file workers in Python mode; must be a positive integer |
 | `DD_TEST_OPTIMIZATION_MAX_WAIT_SEC` | Override uploader max wait |
 | `DD_TEST_OPTIMIZATION_QUIESCENT_SEC` | Override uploader quiescence wait |
 | `DD_TEST_OPTIMIZATION_MAX_DEPTH` | Limit payload discovery depth in large trees |
@@ -671,7 +674,7 @@ The doctor and/or uploader runtimes read these variables at `bazel run` time:
 | `DD_TEST_OPTIMIZATION_REPORT_DIR` | Optional wrapper/report-script directory. CI wrappers write `doctor-report.json` plus `uploader-dry-run-report.json` without upload or `uploader-upload-report.json` with upload unless explicit report paths override them |
 | `DD_TEST_OPTIMIZATION_SUPPORT_BUNDLE` | Optional doctor or wrapper path for the redacted support diagnostics zip |
 | `DD_TEST_OPTIMIZATION_SUPPORT_BUNDLE_COLLECTOR` | Optional override for the support bundle collector script. Doctor targets provide this through runfiles; wrappers default to `create_support_bundle.py` beside the wrapper |
-| `DD_TEST_OPTIMIZATION_PYTHON` | Optional Python interpreter used by wrapper support-bundle generation and helper scripts before falling back to `PYTHON`, `python3`, and `python` |
+| `DD_TEST_OPTIMIZATION_PYTHON` | Python interpreter override before falling back to `PYTHON`, `python3`, and `python`; Python 3.10+ is required by the default uploader and optional only when `use_python_uploader = False` selects the legacy runtime |
 | `DD_TEST_OPTIMIZATION_ARTIFACT_SOURCE` | Artifact discovery source: `local`, `bep`, or `auto`. Recommended CI with zipped undeclared outputs should set `bep` |
 | `DD_TEST_OPTIMIZATION_REMOTE_ARTIFACTS` | Remote BEP artifact handling: `disabled`, `download`, or `required`. HTTP/HTTPS `outputs.zip` carriers can be staged natively when enabled |
 | `DD_TEST_OPTIMIZATION_ARTIFACT_STAGING_DIR` | Directory used for per-run staged BEP artifacts |
@@ -687,6 +690,7 @@ Uploader CLI flags:
 |------|---------|
 | `--dry-run` | Enrich and validate discovered payloads without uploading or deleting files |
 | `--validate-enrichment` | Require key Git and Bazel tags to exist after enrichment, before either validation-only completion or upload |
+| `--workers=<positive-integer>` | Override the maximum independent payload-file workers in Python mode; takes precedence over `DD_TEST_OPTIMIZATION_WORKERS` and the rule attribute |
 | `--expected-enriched-tag=<tag>` | Add a required enriched tag; repeatable. Defaults cover `git.repository_url`, `git.commit.sha`, `bazel.target`, and `bazel.package`. Add `bazel.go.payload_selection` explicitly when a Go rollout must prove per-module selection |
 | `--bep-json=<path>` | BEP JSON file from the matching Bazel test invocation; repeat for multiple invocations |
 | `--freshness-source=<source>` | Freshness source: `auto`, `bep`, or `execution_log` |

docs/Troubleshooting.md

@@ -46,6 +46,8 @@ in this release.
 | Doctor reports msgpack payloads | tracer is not in Bazel JSON file mode | Doctor failures |
 | Doctor reports missing Git or Bazel metadata | sync metadata context or sidecar metadata is absent | Doctor failures |
 | Uploaded tests miss Git or Bazel tags | run uploader dry-run enrichment validation | Uploader enrichment dry-run |
+| Python uploader cannot start | Python 3.10+ discovery and override | Python uploader cannot resolve Python |
+| Test payload split or `413` failure | enriched size, individual event size, fixed threshold | Python uploader split and `413` failures |
 | Upload network errors | credential mode (agentless vs EVP), intake reachability | Tests not uploading (network errors) |
 | CI failure requires log archaeology | archive the support bundle from the failing run | Collect diagnostic reports |
 | Module selection misses | `bazel query` for `module_*` targets and importpath/module label expectations | Per-module files not found |
@@ -743,9 +745,70 @@ prove per-module selection. If this fails:
 
 1. Ensure the uploader target has the right `data = ["@...//:test_optimization_context"]`.
 2. Ensure `bazel_target_metadata.json` exists beside the payloads.
-3. Ensure `jq` is available on Linux/macOS when using `--validate-enrichment`.
+3. When using the temporary legacy Bash uploader, ensure `jq` is available on
+   Linux/macOS. Python uploader mode does not require `jq`.
 4. Use `--expected-enriched-tag=<tag>` for repository-specific required tags.
 
+## Python uploader cannot resolve Python
+
+**Symptom**: The generated uploader prints `Python 3.10 or newer was not found`
+or reports that the discovered interpreter is older than 3.10.
+
+**Solution**:
+
+1. Confirm the uploader target does not explicitly set
+   `use_python_uploader = False`; that value selects the temporary legacy
+   rollback path.
+2. Set `DD_TEST_OPTIMIZATION_PYTHON` to an explicit Python 3.10+ executable.
+   `PYTHON`, `python3`, and `python` are tried afterward, in that order.
+3. On Windows, set the environment variable before invoking `bazel run`:
+   ```powershell
+   $env:DD_TEST_OPTIMIZATION_PYTHON = "C:\Python312\python.exe"
+   bazel run //:dd_upload_payloads -- --dry-run
+   ```
+4. On Linux/macOS:
+   ```bash
+   DD_TEST_OPTIMIZATION_PYTHON=/usr/local/bin/python3.12 \
+     bazel run //:dd_upload_payloads -- --dry-run
+   ```
+
+The launcher only resolves runfiles, the interpreter, and the generated config;
+all enrichment and upload behavior remains in the shared Python runtime.
+
+## Python uploader split and `413` failures
+
+Python mode calculates the compact UTF-8 size after enrichment and before any
+HTTP request. Test bodies larger than `4_718_592` bytes (4.5 MiB) are split
+deterministically by event while preserving event order and all non-event
+top-level fields.
+
+- `single_event_exceeds_payload_limit`: one enriched event cannot fit by
+  itself. Reduce the producing tracer/event metadata; increasing or overriding
+  the uploader threshold is intentionally unsupported.
+- `test_payload_not_json_serializable` or `invalid_test_json`: the source or an
+  enriched value is not standards-compliant JSON. In particular, `NaN` and
+  positive/negative `Infinity` are rejected.
+- `payload_limit_contract_mismatch` after HTTP `413`: every prepared chunk was
+  already at or below 4.5 MiB, so the backend limit or request contract differs
+  from the uploader contract. The uploader retains the source, stops later
+  chunks, does not retry `413`, and does not split adaptively.
+
+Reproduce preparation without backend traffic or deletion:
+
+```bash
+bazel run //:dd_upload_payloads -- \
+  --dry-run --allow-cached-payload-uploads --debug
+```
+
+```powershell
+bazel run //:dd_upload_payloads -- `
+  --dry-run --allow-cached-payload-uploads --debug
+```
+
+Inspect the final `split`, `requests`, and `cleanup` statistics and, when
+configured, the schema-v1 JSON report. Do not retry a retained `413` payload
+unchanged; first reconcile the configured 4.5 MiB contract with the backend.
+
 ## Non-standard bazel-testlogs location
 
 **Symptom**: Uploader cannot find the `bazel-testlogs` directory.

docs/Uploader_Reference.md

@@ -186,6 +186,36 @@ and runtime identity must agree. Missing, duplicate, mismatched, or incomplete
 runtime selections fail before payload discovery, cleanup, enrichment, or
 network access.
 
+### Parallel Python uploader (default)
+
+The cross-platform Python uploader is the default. Use `uploader_kwargs` only
+when choosing a non-default number of independent payload-file workers:
+
+```bzl
+dd_test_optimization_targets(
+    name = "test_optimization",
+    sync_repo_name = "test_optimization_data",
+    uploader_kwargs = {
+        "workers": 8,
+    },
+)
+```
+
+This mode requires a host Python 3.10 or newer on Linux, macOS, and Windows at
+`bazel run` time. The launcher resolves `DD_TEST_OPTIMIZATION_PYTHON`, then
+`PYTHON`, then `python3`, then `python`. Each worker owns one payload file and
+performs its enrichment, validation, preflight split at the conservative
+`4_718_592`-byte (4.5 MiB) threshold, and upload/retries independently. Every
+worker can process test, coverage, or telemetry payloads; chunks and other
+derived requests belonging to one source remain sequential. Test and telemetry
+JSON reject non-standard numbers (`NaN` and positive/negative `Infinity`) before
+HTTP; coverage bodies remain opaque JSON/msgpack multipart parts. The
+rule-level `workers` value defaults to `8`; `DD_TEST_OPTIMIZATION_WORKERS`
+overrides it at runtime and `--workers=<positive-integer>` has highest
+precedence. Leave
+`use_python_uploader` unset for the Python implementation. Set it to `False`
+only for temporary rollback to the legacy Bash or PowerShell implementation.
+
 If your repository is small, the same helper can live in the root package. In
 large monorepos, prefer `//tools/test_optimization` or another lightweight
 package to avoid loading unrelated root package wiring when running doctor or
@@ -295,6 +325,7 @@ bazel run --config=test-optimization //:dd_upload_payloads
 | `DD_TEST_OPTIMIZATION_FILTER_PREFIX` | `0` | `0` uploads all payload files; set to `1` to only upload `span_events_*.json` or `coverage_*.json` |
 | `DD_TEST_OPTIMIZATION_DEBUG` | `0` | Set to `1` to enable verbose attempt, success, startTime, and runfile/CODEOWNERS resolution logging. Terminal test-upload failures always report the HTTP status, a bounded response body, and payload sizes. |
 | `DD_TEST_OPTIMIZATION_GZIP` | `0` | Set to `1` to gzip test payloads before upload (adds `Content-Encoding: gzip`) |
+| `DD_TEST_OPTIMIZATION_WORKERS` | rule `workers` (`8`) | Override the maximum independent payload-file workers in Python mode; `--workers` has higher precedence |
 | `DD_TEST_OPTIMIZATION_MAX_WAIT_SEC` | `300` | Override max wait time for slow filesystems (NFS, network drives); set to `0` to skip waiting when no payloads are present |
 | `DD_TEST_OPTIMIZATION_QUIESCENT_SEC` | `10` | Override quiescence wait time |
 | `DD_TEST_OPTIMIZATION_MAX_DEPTH` | `0` (unlimited) | Limit `find` depth for large `bazel-testlogs` trees |
@@ -679,11 +710,44 @@ payload discovery/quiescence before proceeding.
 
 ## Reliability
 
-- HTTP requests use a 60-second timeout
+### Python uploader
+
+- Python mode uses a 10-second connection timeout and a 60-second socket-I/O
+  timeout.
+- Each logical request makes at most four total attempts: the initial attempt
+  plus up to three retries, normally separated by 2 seconds.
+- Connection failures, timeouts, HTTP `408`, HTTP `429`, and HTTP `5xx` are
+  retryable. `Retry-After` is honored when the backend supplies it, with a
+  60-second safety cap per retry so a response cannot stall a worker
+  indefinitely.
+- Other HTTP `4xx` responses are terminal after the first attempt. For test
+  payloads, `413` indicates that the preventive split contract was violated.
+  Coverage and telemetry are not split, so their `413` failures use the
+  `upload_http_413` reason instead. No `413` is retried or triggers adaptive
+  splitting.
+- JSON, gzip, telemetry, and multipart bodies are prepared once per logical
+  request and replayed byte-for-byte for every retry.
+- Split test chunks are uploaded in event order and fail independently. After
+  partial success, the source is atomically replaced with only the failed
+  chunks so accepted events are not replayed; if every chunk fails or the
+  replacement cannot be written, the original source is retained.
+- A telemetry source and its synthetic rule-facts request are also attempted
+  independently. After partial success, only the rejected prepared request is
+  retained for the next invocation, without regenerating augmentation or
+  replaying the accepted request. If retention fails, the original source is
+  kept and the final result reports a warning.
+- The temporary legacy Bash/curl and PowerShell implementations remain
+  available only as an explicit opt-out rollback during the rollout window.
+  The normalized policy above is the default uploader contract on Linux,
+  macOS, and Windows.
+
+### Legacy Bash and PowerShell uploaders
+
+- HTTP requests use a 60-second timeout.
 - Failed requests are retried up to 3 times with a 2-second delay between
-  attempts
+  attempts.
 - Both transient errors (connection issues) and HTTP errors (4xx/5xx) trigger
-  retries
+  retries.
 - After enrichment, test payloads larger than 4,500,000 bytes are split by
   their top-level `events` array before compression and transport. Each part
   preserves the original top-level envelope and remains at or below the split
@@ -705,7 +769,7 @@ payload discovery/quiescence before proceeding.
   characters of the response body, and the uncompressed, compressed, and
   transmitted byte counts. Response logging does not require debug mode.
 - Behavior is consistent across Linux/macOS (bash/curl) and Windows
-  (PowerShell-only runtime path; no Git Bash requirement)
+  (PowerShell-only runtime path; no Git Bash requirement).
 
 ## Metadata enrichment (`context.json`)
 
@@ -716,9 +780,12 @@ payload discovery/quiescence before proceeding.
   4. if multiple bundled contexts exist and no match is found, skip only the `context.json` merge for that payload and continue uploading
   5. if no bundled context resolves, upload without context enrichment
 - When a `context.json` file is available, the uploader enriches each test
-  payload by merging all non-null keys from `context.json` into each event's
-  `content.meta` or `content.metrics`, and it also normalizes top-level
-  `metadata.*` runtime tags.
+  payload by merging its keys into each event's `content.meta` or
+  `content.metrics`, and it also normalizes top-level `metadata.*` runtime
+  tags. Numbers become metrics; strings and booleans become meta values; other
+  JSON values use their compact JSON representation. For legacy compatibility,
+  JSON `null` therefore becomes the meta string `"null"` rather than being
+  omitted.
 - Bazel sidecar metadata from `bazel_target_metadata.json` is merged separately.
   If a multi-context payload has no repo match, those Bazel sidecar tags remain
   and only the `context.json` merge is skipped.