[python] PoC: AI Guard test under INTEGRATION_FRAMEWORKS#7257
Open
avara1986 wants to merge 8 commits into
Open
[python] PoC: AI Guard test under INTEGRATION_FRAMEWORKS#7257avara1986 wants to merge 8 commits into
avara1986 wants to merge 8 commits into
Conversation
Contributor
|
|
🎉 All green!🧪 All tests passed 🔗 Commit SHA: d95adff | Docs | Datadog PR Page | Give us feedback! |
avara1986
force-pushed
the
avara/appsec-68977-integration-frameworks-ai-guard-poc
branch
from
78379f1 to
ca1a97f
Compare
avara1986
changed the title
…APPSEC-68977) Exercise the auto-instrumented AI Guard/OpenAI integration (ai_guard_listen) the way the LLM Obs suite does: call the OpenAI SDK directly through the existing /chat/completions weblog endpoint with DD_AI_GUARD_ENABLED=true, and assert an ai_guard span is emitted for before-model, after-model (streamed) and tool-call evaluation. The evaluation outcome (ALLOW/DENY/ABORT) stays covered by the dedicated AI_GUARD scenario and is not re-asserted here. The AI Guard endpoint is not yet wired into the INTEGRATION_FRAMEWORKS scenario (no DD_AI_GUARD_ENDPOINT / no aiguard VCR cassettes), so the integration path 500s for every language. Marked missing_feature for python, nodejs and java until the scenario wiring + cassettes land. Investigation task: APPSEC-68977 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
avara1986
force-pushed
the
avara/appsec-68977-integration-frameworks-ai-guard-poc
branch
from
ca1a97f to
4de1a84
Compare
library_env now injects real DD_API_KEY/DD_APP_KEY in --generate-cassettes mode (the AI Guard client calls the real API to record the aiguard cassette) and keeps mock keys on replay, where the VCR proxy matches on the request rather than auth. Investigation task: APPSEC-68977 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
avara1986
commented
Jul 13, 2026
avara1986
requested review from
brettlangdon,
claponcet,
manuel-alvarez-alvarez and
wconti27
and removed request for
a team
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d95adff1f0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Proof-of-concept exploring whether AI Guard coverage can be consolidated under the
INTEGRATION_FRAMEWORKSscenario instead of the dedicatedAI_GUARDscenario.Adds
tests/integration_frameworks/llm/openai/test_openai_ai_guard.py, which drives AI Guard evaluation through the OpenAI framework weblog using thetest_client/test_agentfixtures, and asserts on theai_guardspan (meta.ai_guard.*,meta_struct.ai_guard) — reusing the payloads and span shape fromtests/ai_guard/test_ai_guard_sdk.py.Status: DRAFT / investigation
This is a scaffold for the investigation tracked in APPSEC-68977. It is not runnable yet — it depends on prerequisites still to be validated:
/ai_guard/evaluateendpoint inutils/build/docker/python/openai_app/openai.pythat calls the tracer's AI Guard client;/vcr/aiguardupstream (or reusing the standaloneVCRCassettesContainer);DD_AI_GUARD_ENABLED/DD_AI_GUARD_ENDPOINTwired intoIntegrationFrameworksScenario.Jira
Investigation task: https://datadoghq.atlassian.net/browse/APPSEC-68977 (under epic APPSEC-68823)
🤖 Generated with Claude Code