feat(mxc): add Windows ETW-to-OCSF audit trail - #3015
Conversation
|
I have read the DCO document and I hereby sign the DCO. |
|
/ok to test 3e46489 |
Follow-Up NeededI cannot validate this submission yet because it is a large, cross-cutting Windows audit feature (17 files across the MXC driver, OCSF, and server configuration) without a linked validated issue, roadmap item, reviewed RFC, or maintainer confirmation of the scope. I checked PR #2959 as well; it was closed because it targeted the wrong branch, so this PR appears to be the intended continuation rather than a competing duplicate. @araza008, please link the issue, roadmap item, or RFC that authorizes this work, or ask a maintainer to confirm in this thread that this scope should proceed. If the original submitter or a maintainer does not respond within 48 business hours, this may be closed as not planned. Weekend hours do not count toward the TTL. Gator metadata
|
drew
added
the
gator:follow-up-needed
|
Maintainer confirmation: this scope should proceed. Please continue the Gator code review on the current head SHA. This is an explicit same-SHA review request; post the resulting review disposition if warranted. |
drew
left a comment
drew
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @drew, I verified your maintainer confirmation that this cross-cutting Windows audit scope should proceed and completed the requested initial review of the current head. The review found five blocking correctness, security, and audit-integrity issues.
Action required: @araza008, address the five inline findings and push an updated head for a focused follow-up review.
Blocking findings:
GATOR-3e464893-01: the shipped example uses unsupported MXC configuration and cannot complete its advertised workflow.GATOR-3e464893-02: raw command arguments are copied into durable and routed audit logs.GATOR-3e464893-03: a second gateway stops the first gateway's shared ETW session.GATOR-3e464893-04: system-wide ETW input feeds an unbounded in-memory queue.GATOR-3e464893-05: weak and stale correlation keys can misattribute host ETW activity to a sandbox.
Carried findings:
- None.
Non-blocking suggestions:
GATOR-3e464893-06: documentetw_audit,OPENSHELL_OCSF_JSON, andOPENSHELL_OCSF_LOG_DIRin the canonical gateway and observability docs.
Gator metadata
- Validation: Project-valid because verified maintainer @drew explicitly confirmed this cross-cutting scope should proceed.
- Docs: Missing canonical Fern coverage for the new gateway configuration and audit-output controls.
- Checks: Current-head
OpenShell / Branch Checksis failing in Rust lint jobs; Helm Lint and existing E2E gate statuses are green. - E2E:
test:e2ewill be required for MXC lifecycle and gateway audit behavior after review feedback is resolved; it has not been dispatched by Gator yet. - Head SHA:
3e4648932dbe1e823ac190560a3e507c7dd97ad2 - Base SHA:
74654ac30dfe7115b6b9dfdf9480342b34290607 - Merge base SHA:
65745a06ef7b4aafc00600fd7f2d16ef7da49c26 - Patch ID:
776036d10cddf58d3776c3d2c356ba322336966b - Gator payload:
7 - Review mode:
initial - Previous reviewed SHA: none
- Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:in-review
drew
added
gator:in-review
03a78b2 to
b13cf5a
Compare
Add a Windows MXC ETW->OCSF audit trail in openshell-driver-mxc: a real-time Sandboxing-provider ETW consumer that decodes events (TDH), attributes each to an OpenShell sandbox_id, and maps them to OCSF (lifecycle 6002, config 5019, process 1007, finding 2004). cp6 Phase 1 - durable OCSF JSONL audit-file parity with Linux: - openshell-ocsf: add emit_ocsf_event_routed (populates the event-bridge thread-local AND stamps sandbox_id+message in one dispatch) plus public set/clear_current_event; OS-aware device (Device::windows/for_current_os) so device.os.name reflects the host instead of a hardcoded Linux stub. - etw_consumer: emit via the routed emit (previously fired a bare info! that never populated the bridge, so the structured event was dropped). - openshell-server: install OcsfJsonlLayer over a synchronous daily-rotated appender (durable under force-kill), gated by OPENSHELL_OCSF_JSON, path via %PROGRAMDATA%\OpenShell\logs (override OPENSHELL_OCSF_LOG_DIR). - device.hostname now resolves to the real gateway machine name. Box-proven on 7F203-MXC-001: JSONL lines == shorthand OCSF rows, all valid OCSF JSON, per-sandbox attribution intact, disabled state writes nothing. Signed-off-by: Akber Raza <akberr@nvidia.com>
Close the last three ETW->OCSF gaps so the audit trail covers the full set of events the Sandboxing provider emits (12/12): - ProcessLaunched -> Process Activity [1007] "Launch" (confirmed start; carries the real processId/threadId, the twin of CreateProcessInSandbox which only has the request + command line). - SandboxProxyConfigured -> Device Config State Change [5019] (the one network-plane setup event; surfaces proxyPort, "no proxy" when 0). - SandboxConsoleReferencePlumbed -> Device Config State Change [5019] (console-handle plumbing). map_config_state now handles the full config/hardening/setup family and carries proxyPort/hasConsoleReference/creationFlags as unmapped fields. Verified on 7F203-MXC-001: 11/12 event types emit OCSF without a proxy (SandboxProxyConfigured requires proxy config to fire). Signed-off-by: Akber Raza <akberr@nvidia.com>
Address CodeRabbit review on !31: - Prevent stale ETW attribution on a delete/launch race: register the wxc-exec pid while holding the registry lock, and bail if the sandbox entry is already gone. Previously the attribution key could be seeded after `delete` had removed the sandbox, leaving a stale key that could misroute later Sandboxing ETW events to a dead sandbox_id. Lock order (registry -> attribution) matches the delete path, so no deadlock. - Add unit tests for the new Device::windows and Device::for_current_os constructors to harden Windows/Linux OCSF device parity. Signed-off-by: Akber Raza <akberr@nvidia.com>
Addresses two ETW->OCSF attribution review items (Shailendra NVIDIA#1, NVIDIA#2). NVIDIA#2 early-event loss: ETW delivers the sandbox create/config burst the instant wxc-exec starts, which can beat the driver's register_launch (now under the registry lock post-Ready). process_event previously dropped anything unresolved, losing the racing burst. Add a bounded, time-bounded pending buffer (PENDING_MAX=4096, PENDING_TTL=5s): unresolved events are held and replayed once attribution lands, aged-out ones dropped. Consumer switched to a timed recv_timeout(200ms) so the buffer is re-driven after each event and on a tick. Emit path factored into shared emit_resolved(). NVIDIA#1 attribution collisions: a Windows PID is recycled after exit and a command line is commonly identical across sandboxes. register_launch now rebinds by_pid on reuse and clears the stale last_pid_sid hint (warns if the PID still pointed at a different, leaked sandbox); command line is held in by_cmd only while unique and demoted to a new ambiguous_cmds set on a second owner, so a duplicate command refuses to resolve rather than misroute. Unit tests: buffer replay (direct + cross-link), buffer bound, PID-reuse rebind, duplicate-cmd non-resolution. Box-verified on 7F203-MXC-001 (5 sandboxes, identical cmd -> 5 isolated sandbox_ids, 50/50 OCSF/JSONL, BuffersLost=0). Signed-off-by: Akber Raza <akberr@nvidia.com>
Review item NVIDIA#3 (Shailendra): add a PRIVACY NOTE on map_process_launch stating cmd_line is copied verbatim into OCSF process.cmd_line with no redaction, so secrets/PII on a command line land unredacted in the durable audit trail (deliberate audit-fidelity trade-off; treat the log as sensitive). Redaction is owned by an upstream privacy layer, not this path; no general audit-output PII scrubber exists today (openshell_core::secrets [CREDENTIAL] redaction is scoped to the proxy HTTP-target logging, a separate egress path). Signed-off-by: Akber Raza <akberr@nvidia.com>
…s real status Review item NVIDIA#4 (Shailendra): start_session previously returned Ok(EtwSession) as soon as the pump thread was spawned, but OpenTraceW ran later inside that thread; if it failed we still handed back a live-looking session and logged 'consumer started' (silent failure = false audit coverage). Split the two Win32 calls instead of adding a channel handshake (avoids any lost-wakeup/hang risk): the quick, synchronous OpenTraceW now runs on the caller thread (open_trace), and only the blocking ProcessTrace runs on the pump thread (run_trace). start_session returns Err if OpenTraceW fails (reclaiming the boxed Sender so the consumer disconnects, stopping the session, joining the consumer) and returns Ok/logs 'started' only once capture is genuinely open. Opened handle + LoggerName buffer + boxed Sender are carried to the pump via a Send OpenedTrace so they outlive ProcessTrace. Box-verified on 7F203-MXC-001: consumer started=True, failed-to-start=False, 50 OCSF rows / 50 JSONL, BuffersLost=0 (no regression to capture/emit). Signed-off-by: Akber Raza <akberr@nvidia.com>
CodeRabbit flagged that drain_resolved() re-resolved buffered events against the live by_pid map, so if Windows recycled a wxc-exec PID within PENDING_TTL a stale event from the dead sandbox could be emitted under the new owner. Stamp each by_pid registration with its Instant and add resolve_replay(), used only on the buffered/replay path. It (a) never falls back to the recycle-/ambiguity-prone by_cmd or last_pid_sid keys, and (b) trusts a PID match only when the registration is not newer than the buffered event by more than REPLAY_PID_GRACE (2s) - a recycled PID's registration lands well outside that window, so the stale event ages out instead of misattributing. The legitimate NVIDIA#2 seed race (registration lands ~immediately) still replays. Adds unit tests for the recycle-refusal, in-grace acceptance, and weak-fallback exclusion. Signed-off-by: Akber Raza <akberr@nvidia.com>
…DIA#4) start_session already returns Err on OpenTraceW failure (runs on the caller thread since e41a770), closing the first half of Shailendra's NVIDIA#4. This closes the second half: ProcessTrace's result was discarded, so if capture died mid-run the backend had no way to know. Add a shared CaptureHealth (stopped/stopping/exit_code) between the pump thread and EtwSession. run_trace now records ProcessTrace's WIN32_ERROR and, when the pump returns without a deliberate stop, logs at ERROR that MXC OCSF capture is no longer running. EtwSession::stop() sets `stopping` before teardown so a normal shutdown isn't misreported, and EtwSession::is_capture_alive() exposes the state for status/diagnostics. Box-verified on 7F203-MXC-001: 5 sandboxes, 50 attributed OCSF rows, JSONL parity 50/50, BuffersLost=0, clean start/stop (no false failure). Signed-off-by: Akber Raza <akberr@nvidia.com>
…figured message Add a runnable OCSF audit-trail example under examples/ (run-ocsf-audit.ps1, mxc-ocsf-audit.toml, ocsf-audit.yaml, README) that spins up sandboxes with the in-process ETW consumer and egress proxy on, emitting a full OCSF JSONL audit trail across all four classes (6002/5019/1007/2004). Fix SandboxProxyConfigured mapping to log "MXC sandbox proxy configured" instead of a misleading "(no proxy)" when the provider reports proxyPort=0; the event's presence already indicates proxy configuration. Verified on-box: 26 events, all mapped ETW event types present. Signed-off-by: Akber Raza <akberr@nvidia.com>
Improve the ETW to OCSF audit-trail example output and make it safe to ship.
Report:
- Add an event-type coverage count ("N of M expected event types fired");
the denominator auto-adjusts (8 with proxy on, 7 with -NoProxy).
- Split the checklist into expected event types vs anomaly findings
(ActivityError/FallbackError), which are reported separately and not
counted toward coverage (a clean run may emit none).
- Verdict is now coverage-based (all expected types must fire) instead of
the looser "at least 3 OCSF classes".
- Call out the absolute path to the durable OCSF JSONL log prominently.
Client-safety:
- Default -ShareOut to empty (no auto-copy); pass -ShareOut a UNC path to
opt in. Removes a hardcoded internal share path from a published example.
- Drop internal-team wording ("Hand that zip back for evaluation", "BUNDLE:")
in favor of neutral "Results bundle:".
- Update README-ocsf-audit.txt to match the opt-in -ShareOut behavior.
Verified on both MXC boxes: 7F203-MXC-001 (base-container) -> PASS, 8 of 8
event types, 26 OCSF events across 4 classes; 7F203-MXC-003 (AppContainer
fallback) -> reduced set as expected, clean output.
Signed-off-by: Akber Raza <akberr@nvidia.com>
- remove unsupported gateway-scoped workload fields from the shipped MXC audit example. - build the command, working directory, and filesystem grant from each run's ShareDir - pass the workload through --driver-config-json. - preserve the host CONNECT proxy configuration and conditional audit coverage for the future host_connect_proxy merge - require the workload output when determining the audit verdict. Signed-off-by: Akber Raza <akberr@nvidia.com>
- record only the executable basename for MXC CreateProcessInSandbox audit events - leave process.cmd_line unset so workload arguments cannot reach shorthand or JSONL logs - cover tokens, passwords, signed URLs, and PII with a secret-leak regression test - update the audit example, architecture guidance, and published logging documentation - preserve ETW attribution and future host_connect_proxy enforcement behavior Signed-off-by: Akber Raza <akberr@nvidia.com>
…ncurrent gateways
johntmyers
left a comment
johntmyers
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @drew, I checked your follow-up notes and the latest remediation. The example configuration, command-line redaction, per-gateway ETW session naming, and bounded overload handling are resolved. One carried audit-attribution obligation remains.
Action required: @araza008, make ambiguous PID reuse fail closed and add the two-direction regression described below, then push an updated head for focused follow-up review.
Blocking findings:
- No new findings.
Carried findings:
GATOR-3e464893-05: the PID-generation boundary can still misattribute a new sandbox's ETW activity to an old sandbox.
Warning — GATOR-3e464893-05 · Ambiguous PID reuse can assign new ETW events to the old sandbox
Summary: When Windows rapidly reuses a wxc-exec PID before the old monitor records retirement, the new sandbox can emit its first ETW record before registration. Current code closes the old generation at the new registration time, so that record falls inside the synthetic old interval. It is emitted under the old sandbox and can bind the new identity/activity/CV keys there, corrupting durable JSONL and per-sandbox audit streams.
Fix: Fail closed for PID-only records across an ambiguous replacement. Do not create historical ownership from a replaced or merely observed-dead registration without an exact generation-end boundary; retain a reuse tombstone and allow delayed records only through already-established strong keys.
Verify: Register sandbox A on PID 1000, create a B-identity event before retiring A, then register sandbox B on the same PID. The event and a later B-identity-only event must not resolve to A. Retain the inverse old-record-after-reuse coverage too.
Agent context
- Location:
crates/openshell-driver-mxc/src/etw_consumer.rs:1293 - Sibling sites:
etw_consumer.rs:1450,etw_consumer.rs:1481 - Ownership: Commit
0ba0dfeareplaces the earlier fail-closed tombstone with the synthetic historical interval.
Gator metadata
- Validation: Project-valid because verified maintainer @drew explicitly confirmed this cross-cutting scope.
- Docs: Fern gateway configuration and OCSF export documentation are updated.
- Checks: Current-head bootstrap checks are green; required E2E is not dispatched while review feedback remains.
- E2E:
test:e2eis required after the carried finding is resolved. - Head SHA:
f443a0a46135f2710ccfb8166656e861a04655f6 - Base SHA:
0357daee316f32a4d5c312174d68672cb0f4d389 - Merge base SHA:
0357daee316f32a4d5c312174d68672cb0f4d389 - Patch ID:
c5f7d24989535bf0860f2bd818e7e4a1b240c15d - Gator payload:
8 - Review mode:
follow_up - Previous reviewed SHA:
3e4648932dbe1e823ac190560a3e507c7dd97ad2 - Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:in-review
johntmyers
added
gator:in-review
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
|
/ok to test f6c029f |
Maintainer Convergence DecisionThe third finding-bearing review round is complete. The merge-main update integrates the OCSF Root-cause findings:
Scope growth:
Reviewer-quality signals:
Action required: @NVIDIA/openshell-maintainers, decide specifically whether Gator metadata
|
johntmyers
added
gator:blocked
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
johntmyers
left a comment
johntmyers
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @drew, I checked the PID-reuse remediation you added at the latest head. It removes the synthetic old lifetime and adds both displacement-direction tests, but the fixed grace period still allows a new process event to resolve through the stale old PID registration after two seconds. The carried audit-integrity finding therefore remains open, and this is the third finding-bearing round.
Action required: @NVIDIA/openshell-maintainers, decide whether GATOR-3e464893-05 must be completed with generation-backed PID evidence before test dispatch or is explicitly waived. If it remains required, @araza008 or @drew should address the carried finding below and push a new head.
Blocking findings:
- No new findings.
Carried findings:
GATOR-3e464893-05: elapsed time is still treated as proof that a stale PID registration owns an event.
Gator metadata
- Validation: Project-valid because verified maintainer @drew confirmed this cross-cutting Windows audit scope.
- Docs: Fern gateway configuration and OCSF export documentation are updated.
- Checks: Current-head Branch Checks and Helm Lint are pending; DCO and Trivy Changes are green.
- E2E:
test:e2eis required but not dispatched while the carried review obligation remains unresolved. - Head SHA:
333c114d45ce89525536fd0387946603acd21485 - Base SHA:
38f2aef930140a74ecc573811ab7f3ac15d14b8a - Merge base SHA:
38f2aef930140a74ecc573811ab7f3ac15d14b8a - Patch ID:
259293644227b575a718ed3e131944efa2a0718d - Gator payload:
8 - Review mode:
follow_up - Previous reviewed SHA:
f443a0a46135f2710ccfb8166656e861a04655f6 - Review budget exhausted:
yes - Maintainer decision required:
yes — disposition of GATOR-3e464893-05 after the third finding-bearing round - Next state:
gator:blocked - Blocked reason:
review_convergence_decision_required
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
|
Label |
|
/ok to test ac559fe |
PR Review StatusThanks @drew. I checked your process-generation remediation on the latest head and your maintainer approval. PID attribution now requires the ETW process start key to match the key queried from the driver-owned child handle, retired PID evidence is discarded, and the added regressions cover both directions of PID reuse. This resolves Blocking findings:
Carried findings:
Gator metadata
|
johntmyers
added
gator:watch-pipeline
Monitoring CompleteMonitoring is complete because this PR has merged. Final status: The current head I removed the active |
johntmyers
removed
the
gator:in-review
4 participants
Summary
Adds a Windows MXC audit trail that consumes Sandboxing ETW events, attributes them to OpenShell sandboxes, and emits structured OCSF events to a durable JSONL log. This provides Windows audit coverage and output parity with the Linux OCSF path.
Changes
ProcessTracefailures accurately.Testing
mise run --skip-tools windows:cisuccessfully on an x64 Windows host.openshell-gateway.exeandopenshell.exe.Checklist