The goal of NeMo Compass is to efficiently find signals in agent traces that lead to meaningful improvements in quality, cost, safety, privacy, and/or user satisfaction.
A critical input to improving any agent is understanding what it’s doing, especially once you turn it on in production. Evals often capture only a small subset of the tasks an agent actually needs to be good at. Agent developers have discovered that production trace analysis is an essential part of improving an agent, but today there's no standard for how this type of analysis gets validated and measured.
NeMo Compass is an applied research project with a goal of improving the tools and techniques available to developers for trace analysis at scale. It uses a range of techniques to identify different types of issues in a large set of traces, then summarizes findings as actionable insights. We call each of those discovery techniques "Compass Signals."
| Compass Signal | Question It Answers | How It Works |
|---|---|---|
| Anomaly Detection | What outlier behaviors exist in a set of agent traces? | Scores outliers with an Isolation Forest and reports feature-level deviations |
| Tool Issues | What recurring problems occur when the agent uses tools? | Checks tool calls and results for invalid arguments, failures, missing results, and unproductive retries |
| User Sentiment | What recurring complaints do users have about the agent? | Screens recorded human messages with a classifier, then reviews recurring complaints in context |
| ETHOS.md Divergence | Where does the agent’s behavior conflict with its stated rules and goals? | Compares traces with the agent’s ETHOS.md document and investigates possible gaps |
| Grader failures | What behaviors recur in traces with poor evaluation results? | Uses recorded scores and feedback to select traces, then investigates shared failure patterns |
Learn more about each current Compass Signal: Architecture
Important
Research preview
NeMo Compass is an early-stage applied research project. Its APIs, configuration, and output formats may change without backward compatibility. All findings should be reviewed against their supporting traces before you act on them. This project is not intended for production use.
We are sharing the implementation at this stage to encourage feedback and testing by others as we continue to improve on it.
Install uv, then choose a guide:
| Your starting point | Guide |
|---|---|
| Try it before connecting your data | Run the example |
| Braintrust project logs and experiments | Braintrust |
| LangSmith traces and feedback | LangSmith |
| Langfuse traces and scores | Langfuse |
| MLflow traces and assessments | MLflow |
| NeMo Platform Intake | Intake |
| NeMo Gym rollout trajectories | NeMo Gym |
| JSONL or ATIF files | Trace files |
| Another trace store | Convert your traces |
Install from PyPI as nemo-compass; the command is compass.
If compass is not found after installation, run uv tool update-shell
and restart your terminal.
- Read your results — understand insights, skipped Compass Signals, and saved output.
- Compass Signals — supply business rules, configure sentiment, or check findings against code.
- Data and model access — configure your inference endpoint and understand where data goes.
- Configuration reference — overrides, limits, and repeatable runs.
Purpose and expectations · Development · Releasing · Architecture · Dev Notes
- How can developers efficiently find recurring, fixable agent problems in trace datasets containing hundreds of thousands or millions of executions?
- How can developers estimate how often a problem occurs and how much it matters without asking an LLM to inspect every trace?
- How can trace evidence, evaluation results, and read-only checks against agent code help distinguish real problems from misleading patterns?
- How should we measure whether an insight is correct, useful, and worth a developer's time to investigate?
This repository explores these questions. The architecture guide explains what each part does and where judgment or coverage limits remain.
We are especially interested in feedback from teams building agent observability or evaluation tools, and from developers investigating their own agents:
- Which types of recurring issues are hardest to identify today?
- How do you decide whether an identified issue matters to your agent?
- What would make a finding more "actionable": examples, frequency, estimated impact, something else?
- How would you ideally use these tools in your own workflows? What would you need to integrate it?
- What types of additional insights or findings would be most valuable to you?
If an NVIDIA contact shared this preview with you, please send your feedback through that contact and ask them to route it to the NeMo Compass research team. Otherwise, use the NVIDIA Developer contact form with NeMo Compass research preview feedback in the subject. Please avoid sending raw traces, prompts, credentials, or user data.
This project is currently not accepting code contributions, pull requests, or GitHub issues from external contributors. Please do not submit external pull requests or open issues, including bug reports and feature requests.
For research feedback, use the channels described below. To report a security vulnerability, follow SECURITY.md.
Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
This project is licensed under the Apache License, Version 2.0. See NOTICE for project attributions, THIRD_PARTY_LICENSES.md for human-readable dependency license disclosures, and third_party/licenses.jsonl for the machine-readable inventory.