Problem Statement

Both codex-engine workflows that ran in the last 6 hours failed with identical 401 Unauthorized errors from OpenAI, exhausting all reconnect retries before terminating. This is a P0 credential issue blocking all codex-engine workflows.

Root Cause

Codex (v0.121.0) is configured with requires_openai_auth: true and auth: None — meaning it relies entirely on an environment credential (OPENAI_API_KEY or similar). The engine attempts `(api.openai.com/redacted) and receives:

unexpected status 401 Unauthorized: Missing bearer or basic authentication in header
url: (api.openai.com/redacted)

This surfaces after 5 reconnect retries, causing the agent job to fail with no output emitted.

Affected Workflows & Run IDs

Both workflows: pre_activation ✅ → activation ✅ → agent ❌ (1.2m and 2.6m respectively)

Additional Signals

• codex session config shows experimental_bearer_token: None — no explicit token override
• Model: gpt-5.3-codex — possibly a new model ID requiring a specific API tier
• chatgpt.com:443 is being blocked by firewall (codex tries to cache plugin IDs there) — separate but worth noting
• @pelikhan's comment on [aw] Daily Observability Report for AWF Firewall and MCP Gateway failed #27115: "codex woes. New endpoint?" — suggests possible API endpoint migration may also be relevant

Proposed Remediation

1. Verify OPENAI_API_KEY repository secret — confirm it is set, not expired, and has access to the gpt-5.3-codex model
2. Check OpenAI project/org headers — codex reads OPENAI_PROJECT and OPENAI_ORGANIZATION env vars; verify these are still valid
3. Confirm model availability — gpt-5.3-codex may require explicit API access or a different endpoint; check if /v1/responses is the correct endpoint for this model
4. Test with explicit token — if using experimental_bearer_token, verify the token hasn't rotated

Success Criteria

Codex engine completes without 401 errors on the next scheduled run of Daily Observability Report or a PR-triggered AI Moderator run.

References:

• Parent tracking: [aw] AI Moderator failed #27122 (AI Moderator failure) · [aw] Daily Observability Report for AWF Firewall and MCP Gateway failed #27115 (Daily Observability Report failure)

Generated by [aw] Failure Investigator (6h) · ● 179.2K · ◷

• expires on Apr 26, 2026, 1:18 AM UTC