DAIS 2026: operational demo consolidation and platform hardening#71
Open
oleksandrabovkun wants to merge 22 commits into
Open
DAIS 2026: operational demo consolidation and platform hardening#71oleksandrabovkun wants to merge 22 commits into
oleksandrabovkun wants to merge 22 commits into
Conversation
…reams/agents, ship Casper's Ops dashboard
This squashes the entire `demo/dais-2026` branch into a single commit on top of
origin. Three themes:
1. Repo restructure (ceo → operational, unified `all` target)
- Rename ceo-demo → operational-dashboard-demo and ceo-dashboard app →
caspers-ops-dashboard; catalog-scope app names so two devs on one
workspace don't collide.
- Collapse Knowledge Assistants, Genie spaces, PDF ingestion, and
Multi-Agent Supervisor wiring into single shared stages reused across
targets; keep `default` focused on the refund demo and add a
comprehensive `all` target for DAIS.
- Migrate Knowledge Assistants to the v2.1 API.
- Bake 4 US + 4 EMEA locations into the canonical dataset.
2. Stability & rate-limit hardening
- Stream self-healing on DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE
(recovery skips the historical backfill).
- Per-batch inference caps on complaint_agent_stream and
complaint_generator (foreachBatch + MAX_INFERENCES_PER_BATCH + first-run
templated backfill) so a backlog cannot blow the 10-min task budget.
- timeout_seconds=600 and queue=disabled on every stream task so a stuck
run cannot tie up the cron queue.
- 30s per-call UDF timeouts on the OpenAI clients in the refund / support
/ complaint agent streams.
- Switch default LLM_MODEL from llama-3-3-70b to claude-sonnet-4-5 (separate
per-endpoint QPS bucket). Add SKIP_EVAL=true default and gate
mlflow.genai.evaluate behind both SKIP_EVAL and an
_endpoint_already_serving idempotency check so redeploys don't burn LM
quota.
- Idempotent get-or-create for Lakebase instances (default, complaint,
support) and the streaming jobs they back; storage_catalog/_schema
pointed at the real catalog instead of the hardcoded placeholders.
- Robust Postgres-role lookup for the refund manager app: match by app SP
id with fallbacks for "role already exists" races.
- Genie spaces: register in uc_state only on create, recreate when the
bound warehouse is stale, and populate Instructions / Sample SQL /
Joins via the API.
- Refunder agent: sys.path resolution + eval rate-limit retry; complaint
agent waits for delivered events before sampling.
3. New: AI/BI Operational dashboard as a DABs resource (`all` target only)
- resources/dashboards/caspers_kitchen_operations.lvdash.json checked in
with hardcoded `oleksandra.` catalog refs stripped and the 3
pipeline-id-scoped `__materialization_mat_*` table refs rewritten to
the stable DLT names (silver_order_items, gold_location_sales_hourly).
The same JSON works in any catalog via DABs `dataset_catalog`.
- Add `resources.sql_warehouses.caspers_ops_warehouse` so DABs owns the
${CATALOG}-ops-warehouse — gives `bundle deploy` a stable warehouse id
to bind the dashboard to (no chicken-and-egg with task-run-time
creation). Refactor stages/operational_app.ipynb to find-only and
stop registering the warehouse with uc_state (DABs handles cleanup).
…eholder
- destroy.ipynb: belt-and-suspenders sweep before DROP CATALOG that deletes
any serving endpoint named `<CATALOG>_*` and any KA/MAS tile named
`<CATALOG>-*`, using the same path precedence as uc_state.clear_all().
Catches orphans the state-table-driven cleanup misses when a previous
destroy left the table empty but the agent endpoints/tiles alive.
- jobs/complaint_agent_stream.ipynb: stop the recurring 615s task timeouts.
Lower MAX_INFERENCES_PER_BATCH 20 -> 10 and remove the 3-attempt retry
loop in process_complaint (measured warm latency is 16-21s/call, so one
timed-out retry blew the 600s task budget every tick). Fix-fast on
failure; the next availableNow tick re-processes anything escalated.
- databricks.yml + stages/environment_helpers.ipynb: new no-op stage
`Environment_Helpers` in the `all` target, slotting after Canonical_Data,
as a parking spot for small post-ingestion setup steps (seed lookups,
helper UDFs, demo grants) that are too small to warrant their own stage.
- stages/{lakebase,complaint_lakebase,support_lakebase,genie_spaces}.ipynb:
minor refactors + notebook-format normalization carried in alongside.
Spark Structured Streaming creates offsets/, commits/, metadata/, sources/ subdirectories under the checkpoint path *before* the first call to process_batch. The old heuristic `os.listdir(CHECKPOINT_PATH) == 0` therefore always returned False on the very first batch, silently disabling the fake-it-till-up backfill path and forcing the entire historical backlog through real inference (or ai_gen) in a single micro-batch. Symptom on a fresh deploy: Complaint Agent Stream timed out every cron tick at ~615s (above the 600s task budget). Zero rows were ever written to complaint_responses; checkpoint contained offsets/0 but an empty commits/, so the next tick replanned the same too-large batch and the backlog grew. Fix: check the commits/ subdirectory for any non-hidden file. A committed batch is the unambiguous "this is not the first run" signal. Applied to all 4 streams that share this pattern: - jobs/complaint_agent_stream.ipynb - jobs/complaint_generator.ipynb - jobs/refund_recommender_stream.ipynb - jobs/support_request_agent_stream.ipynb
1. complaint_agent_stream: idempotent CDC enablement Replaced the unconditional `ALTER TABLE ... SET TBLPROPERTIES (delta.enableChangeDataFeed = true)` with a SHOW TBLPROPERTIES check, matching the pattern already used in refund + support streams. Without this, every cron tick produced a no-op SET TBLPROPERTIES commit (observed 4+ versions accumulating in Delta history). 2. refund_recommender_stream + support_request_agent_stream: drop 3-retry loop Both UDFs wrapped the agent call in `for _ in range(3): ... continue`, meaning worst-case per-row wall time was 3 × _CALL_TIMEOUT_S = 90s. With MAX_INFERENCES_PER_BATCH = 50 this can blow the 600s task budget under endpoint stress — exactly the failure mode complaint_agent_stream already learned and removed. Failure policy preserved (refund still falls back to fake, support still falls back to fake), just single-shot. Updated the misleading "30s leaves headroom for 3 retries × 50 calls" comment on _CALL_TIMEOUT_S to reflect the new (correct) math. 3. complaint_agent_stream + complaint_generator + support_request_agent_stream: expand checkpoint-recovery markers to match refund_recommender_stream. The other three only caught DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE; they now also auto-recover from STREAMING_CHECKPOINT_METADATA_ERROR and MISSING_METADATA_FILE (raised when `bundle run cleanup` partially wipes the checkpoint dir).
…hardening Dashboards (all wired into the `all` target, sharing caspers_ops_warehouse): - Delivery Performance & SLA - AI Agent Performance - Menu Intelligence & Nutrition environment_helpers: replaces the stub with an AI SQL demo over the food_safety inspection PDFs. Chains ai_parse_document → ai_classify → ai_extract → ai_summarize over PDF_SAMPLE_SIZE=6 rows and writes four food_safety.ai_*_inspections tables. Gated on Document_Data; cells are idempotent (CREATE OR REPLACE) and skip gracefully if the volume is missing or empty. databricks.yml: register the 3 new dashboards, add Document_Data as a depends_on for Environment_Helpers, refresh the Env_Helpers comment. Lakebase stages (lakebase, complaint_lakebase, support_lakebase): rewrite to add psycopg2-binary to the install line, harden synced-table teardown to also clear stale Delta under _lakebase_storage, and document the storage-path convention used by the synced pipeline.
…r-component DBs
Single `<CATALOG>-caspers` Autoscale Lakebase project now hosts the refund,
complaint, and ops workloads as isolated logical databases (caspers-refund,
caspers-complaint, caspers-ops) — replacing the previous mix of three
Provisioned instances + one ad-hoc Autoscale project. Note: DB names are
RFC-1123 (hyphens, no underscores) per the Lakebase Autoscale create_database
contract; the support app intentionally stays on its Provisioned instance.
- stages/lakebase_project.ipynb (new): provisions the shared Autoscale project
and blocks until its endpoint is RUNNING; registered with uc_state.
- stages/{lakebase,complaint_lakebase,operational_lakebase}.ipynb: rewritten to
create their logical DB inside the shared project + register synced tables.
- utils/lakebase_autoscale.py (new): SDK wrappers (w.postgres.{get,create}_database)
for branch-scoped resources; auto-discovers the deploying user's role for
spec.role; idempotent get_or_create for both DBs and synced tables.
- utils/uc_state/state_manager.py: tracks `autoscale_synced_tables` and deletes
them before their parent `postgres_projects` so cleanup never orphans.
- apps/refund-manager/{app/db.py,requirements.txt}: per-endpoint credential
pattern (mirroring ops dashboard) + pinned databricks-sdk>=0.81.0 so the App
container has w.postgres available.
- stages/apps.ipynb: dropped AppResourceDatabase (Provisioned-only); writes
LAKEBASE_ENDPOINT_PATH/LAKEBASE_DATABASE_NAME env vars; simplified Postgres
role promotion to the Autoscale superuser pattern.
- stages/operational_app.ipynb + apps/caspers-ops-dashboard/app.yaml: point at
the shared Autoscale endpoint + caspers-ops database.
- databricks.yml: Lakebase_Project task wired into default/complaints/all
targets as a dependency of the per-component Lakebase stages; sync.exclude
consolidated into glob patterns.
Also folds in pre-existing code-review fixes:
- token-refresh hardening with exponential backoff in caspers-ops-dashboard/db.py
- lazy WorkspaceClient init + removed unused imports in uc_state/state_manager.py
- new utils/status.py for consistent emoji-aware status output (+ exported)
- removed stale init.ipynb
Verified end-to-end: bundle destroy + redeploy + run -t all against catalog
oleksandra succeeded with all 21 tasks green (run 234191798743190).
Genie.ipynb: full presenter flow — single-space chat across 3 spaces, universal Genie (/one) cross-space payoff, mobile, benchmark generation prompt, and "what else Genie can do" reference. AgentBricks.ipynb: IPD/AI SQL functions, Knowledge Assistants, Supervisor Agent, MLflow tracing and evaluation flow. Co-authored-by: Isaac
…t recovery
Embedded AI/BI Operations dashboard now follows the app's light/dark theme
via a two-variant publish strategy. Databricks embedded dashboards always
render their `light` theme slot (the `?theme=` URL parameter is ignored),
so we ship a second `lvdash.json` with dark-palette tokens baked into the
`light` slot and let the frontend swap the iframe `src` when the user
toggles themes. The Revenue card on the always-visible main view stays
on Chart.js (the embed's other limitations — fixed canvas width, separate
workspace login on first view — were deal-breakers for an always-visible
panel but acceptable for an opt-in tab).
- New `resources/dashboards/caspers_kitchen_operations_dark.lvdash.json`
is structurally identical to the light variant; only the `uiSettings`
theme tokens differ.
- `databricks.yml` registers it as `caspers_ops_dark`.
- `stages/operational_app.ipynb` looks up both variants, publishes each
with `embed_credentials=True` (so query execution doesn't require a
separate workspace login), and writes `OPS_DASHBOARD_ID` +
`OPS_DASHBOARD_ID_DARK` to `app.yaml`.
- App backend (`main.py`) exposes `ops_dashboard_embed_url_{light,dark}`
from `/api/tech-info`; frontend (`index.html`) reads `data-theme` at
mount time and on toggle to pick the right iframe URL.
Other hardening included:
- `operational_supervisor.ipynb` now PATCHes the supervisor on every run
when its wired Genie space IDs / KA endpoints have drifted from
`uc_state` (the `genie_spaces` stage is non-idempotent, so reusing an
existing supervisor previously pointed it at trashed resources, which
surfaced as "permissions issue accessing the revenue analytics system"
in chat).
- `main.py` `/api/revenue` now derives `cancel_rate_pct` and
`cancelled_orders` from `lakeflow.all_events` (anti-join of
`order_created` minus `delivered`) so the location map's risk_level
reflects real data, and the SQL connector uses
`_sdk_config.authenticate` as the credentials provider for proper
OAuth-M2M refresh.
- `/api/locations` now blends absolute cancel-rate thresholds with
relative revenue-change percentiles to flag high/medium-risk locations.
- `index()` now ships `Cache-Control: no-store` so deploys land
immediately on subsequent loads without a manual hard-refresh.
- Removed dead CSS from the inline-iframe pivot (`#ops-dashboard-iframe`,
`.dashboard-embed-container`, `.dashboard-embed-fallback`).
- Ops dashboard: add /api/refund, /api/complaint, /api/complaint-decisions endpoints; add Refund modal + Review Complaints flow; surface custom agents in sub-agent list; expose REFUND_/COMPLAINT_AGENT_ENDPOINT and REFUND_MANAGER_/SUPPORT_CONSOLE_APP_URL env vars. - Stages (refunder/complaint/support/canonical/raw/document): add GRANT USE CATALOG + EXECUTE so serving endpoints' auto-generated SP can resolve UC functions at model-load time (fixes first-deploy permission errors). - Refunder agent: pin langgraph >=0.3.5,<0.4.0 with import fallback; add RefundDecision pydantic schema for structured output. - Add DAIS 2026 LakebaseApps runbook stub (plus empty MLflow runbook placeholder).
…, H4) H1+H2: Agent-stream tasks now wait for their serving endpoint to deploy before firing the first `run_now()`. Previously `Refund_Recommender_Stream` and `Support_Request_Agent_Stream` depended only on the data-pipeline (and generator-stream) task, so the agent endpoint and stream task could run in parallel and the stream's inference UDF could hit a missing endpoint on a fresh deploy. Complaints stream already had the correct pattern; refund (default + all targets) and support (support target) now follow suit. H4: Documented and clarified that `databricks bundle run cleanup` is a bundle SCRIPT (not a job task), so the catalog override must use `--var catalog=...` (or `BUNDLE_VAR_catalog` env var). `--params "CATALOG=..."` is silently ignored by scripts — the CLI only forwards `--params` to job tasks — and the script falls back to the default catalog (`caspersdev`), which is rarely what the user wants. Updated AGENTS.md (3 sites), README.md, and the cleanup-script header comment to make the contract explicit, and the script now echoes a hint about the override flag on every invocation.
…d caches + Lakebase pool
Four demo-stability fixes from the post-code-review triage:
1. operational_supervisor: drift-recovery PATCH now retries 3x with
exponential backoff (1/2/4 s) and raises on exhaustion. Previously
logged a warning and continued, which left the supervisor wired to
trashed Genie/KA resources and surfaced as "permissions issue
accessing the revenue analytics system" in chat — an expensive
failure mode to diagnose at demo time.
2. /api/revenue: removed the silent mock-data fallback. When the
warehouse query fails AND no cached snapshot exists, the route now
returns 503 with a structured {error, message, underlying} body
instead of fabricated numbers. The frontend shows an explicit
"Revenue analytics temporarily unavailable" banner and paints
placeholder cards rather than demoing fake figures. _mockRevenue()
is deleted (both backend and frontend).
3. /api/tech-info: wrapped in a 60 s in-memory TTL cache, coalesced
under a lock. Eliminates 4-6 SDK round-trips per page load
(Lakebase endpoint, project list, workspace metadata, MLflow
experiment search, app deployment lookup). Stale-cache fallback
on transient SDK failures keeps the page snappy and rendering.
4. Lakebase: ops-dashboard now uses psycopg_pool.ConnectionPool
(min=2 max=10, max_idle=30min) instead of psycopg.connect() per
request. Token freshness preserved via a _TokenInjectingConnection
subclass that mints a fresh credential on every new socket, and
the pool's check hook (pool_pre_ping equivalent) recycles stale
connections automatically. Mirrors the proven pattern from
refund-manager/app/db.py.
…ream batch sizing
Ops dashboard slow-load fix (permanent / survives bundle destroy+deploy):
* pipelines/order_items: new gold_location_order_status_daily MV
pre-aggregates created/delivered/cancelled per location per day.
Replaces a 30-day SELECT DISTINCT anti-join over raw all_events.
* apps/caspers-ops-dashboard: 60s TTL cache shared between /api/revenue
and /api/locations; locations no longer triggers a duplicate warehouse
query. SQL reads gold tables only.
* stages/operational_app: explicit SELECT grant for the new MV on the
app SP — UC USE_SCHEMA does NOT cascade SELECT to materialized views
the way it does to streaming tables.
* databricks.yml: ops warehouse auto_stop_mins 10 -> 60 to kill the
30-60s cold-start every time the presenter steps away.
Measured end-to-end on the live oleksandra deployment: warm page-load
~0.66s per endpoint vs ~4-8s × 2 endpoints before; the second endpoint
is now a pure cache hit.
Other batched-in fixes from the same debugging session:
* jobs/{refund_recommender,complaint_agent,support_request_agent}_stream:
MAX_INFERENCES_PER_BATCH sized to actual healthy agent latency
(refund 50->15, complaint 10->5, support 50->15) — old caps were
calibrated for fast-failing endpoints and started timing out the
600s task wall once the agents became responsive.
* apps/refund-manager: split startup DDL into per-statement try/except
so a CREATE INDEX IF NOT EXISTS on an already-owned table doesn't
crash startup with "must be owner of table …" under Lakebase's
DATABRICKS_SUPERUSER (which does not bypass per-table ownership).
* stages/{canonical_data,refunder_agent,operational_app}: belt-and-
braces GRANT USE_CATALOG patterns and other UC grant hardening.
* demos/dais2026-runbooks/LakebaseApps.ipynb: runbook updates.
…deploy permission failures
Switch all three agent stages (refunder, complaint, support_request) from
mlflow.pyfunc.log_model(resources=...) to log_model(auth_policy=AuthPolicy(
system_auth_policy=SystemAuthPolicy([LLM endpoint only]),
user_auth_policy=UserAuthPolicy(api_scopes=[
"sql.statement-execution",
"serving.serving-endpoints",
]),
)).
Why
---
agents.deploy() previously created endpoints with EMBEDDED_CREDENTIALS, which
made the auto-generated endpoint service principal the runtime identity. UC
EXECUTE on the tool functions was granted to that SP via its `account users`
group membership, but that membership propagation has a multi-minute delay.
Result: the first deploy of <catalog>_refund_agent consistently failed with
PERMISSION_DENIED: User does not have EXECUTE on Routine '<catalog>.ai.<fn>'
during model load, and the first request to the complaint/support endpoints
hit the same error before propagation completed. Subsequent retries
"worked" only because the propagation eventually caught up.
With UserAuthPolicy the endpoint is created in ON_BEHALF_OF_USER mode and
forwards the caller's OAuth token at request time. UC ACL checks evaluate
against the caller's identity — the streams run as the catalog owner, who
already has EXECUTE — so there is no SP-membership propagation to wait on.
Also refactor refunder_agent's agent.py to drop UCFunctionToolkit (which
eagerly calls client.get_function() at model load) in favour of three
@tool-decorated wrappers around uc_client.execute_function() — the same
lazy pattern complaint_agent and support_request_agent already use. Lazy
construction means model load itself no longer touches UC at all.
Together these changes make the agent endpoints deploy correctly on the
first attempt, with no retry logic needed.
…rs (match complaint_agent) Only refunder_agent.ipynb was actually broken at first deploy: it used UCFunctionToolkit, which eagerly calls client.get_function() at model LOAD time to introspect tool signatures. When the auto-created endpoint SP had not yet propagated into `account users`, that call hit PERMISSION_DENIED: User does not have EXECUTE on Routine '<...>' and the first deploy failed. Refactor agent.py to use three @tool-decorated wrappers around a lazily constructed uc_client.execute_function() — exactly the same pattern complaint_agent.py and support_request_agent.py already use. Model load no longer touches UC at all, so first-deploy completes regardless of SP propagation state. Also add a wait_get_serving_endpoint_not_updating + READY check after agents.deploy() in refunder_agent so the stage fails fast on deploy errors instead of silently returning SUCCESS while the container is still building (also matches complaint_agent). Reverts the previous OBO / UserAuthPolicy attempt — it was the wrong fix. The other two agents already work without OBO, so refunder just needs to stop introspecting UC at model load.
…l' runtime SPs
Root cause
----------
Model serving endpoints created by agents.deploy() run their inference
container as a workspace-level SCIM SP whose displayName is literally
"System Service Principal". Each agent endpoint typically gets its own
such SP (e.g. refund and complaint endpoints had different SPs in audit
logs). Crucially these SPs are NOT members of `account users`, so the
USE_CATALOG / USE_SCHEMA / EXECUTE grants we make to `account users`
elsewhere in the bundle do not apply to them.
Result: every freshly-deployed agent endpoint failed on its first tool
call with
PERMISSION_DENIED: User does not have USE CATALOG on Catalog '<cat>'
or
PERMISSION_DENIED: User does not have EXECUTE on Routine '<...>'
even though `account users` had every privilege. Streams that called
the endpoints "succeeded" because the agents caught the tool errors and
gracefully escalated, masking the bug.
Fix
---
Add utils/agent_runtime_grants.py: a small helper that discovers every
"System Service Principal" in the workspace via SCIM and grants them
USE_CATALOG / USE_SCHEMA / EXECUTE on `<catalog>.ai` plus
USE_SCHEMA / SELECT on `<catalog>.lakeflow` and `<catalog>.simulator`.
EXECUTE is granted at schema level so it cascades to all current and
future functions (Unity Catalog inheritance), making the fix
deploy-order-independent.
Call grant_agent_runtime_perms() from each agent stage immediately after
agents.deploy() + wait_get_serving_endpoint_not_updating, so perms are
in place before any external traffic hits the endpoint. Wired up in
refunder_agent, complaint_agent, and support_request_agent.
Grants are idempotent (re-granting an existing privilege in UC is a
no-op), so reruns of the stages are safe.
Verified
--------
- oleksandra_refund_agent now returns proper SLA-based refund decisions
on first attempt (e.g. "Order delivered in 26.2 min, within P75 SLA").
- oleksandra_complaint_agent now actually uses get_order_overview /
get_order_timing tools and produces evidence-based escalations
instead of "Unable to access order data due to permission errors"
rationales.
… runbook polish Bundle / infra -------------- - databricks.yml: Environment_Helpers now depends_on Spark_Declarative_Pipeline in addition to Canonical_Data + Document_Data, so the new ABAC cells can tag columns on lakeflow.silver_*/gold_* after the pipeline materialises them. Environment_Helpers (DAIS 2026 governance beat) ----------------------------------------------- - ABAC governance: governance schema, four governed tags (caspers.pii, geo_region, value_class, sensitivity), four UDFs, and four CREATE POLICY statements covering PII column mask, US/EU regional row filter, high-value order revenue gate, and table-level sensitivity policy on AI-extracted inspections. - Data Quality monitors via w.data_quality (new SDK service) where available; each cell is best-effort + idempotent (older runtimes / SDKs degrade gracefully without failing the stage). Apps ---- - stages/apps.ipynb + stages/operational_app.ipynb: upload app thumbnails (💳 refund manager, ops dashboard) via w.apps.update_app_thumbnail when the SDK exposes it; pure visual polish for the Discover/Workspace cards so missing APIs never fail the stage. - utils/app_thumbnails.py: small helper that renders an emoji to a base64 PNG thumbnail. Ops dashboard frontend ---------------------- - apps/caspers-ops-dashboard/index.html: rename action button classes (hire→briefing, fire→risk, ai→board) so the styling matches the actual supervisor agent workflow. - apps/caspers-ops-dashboard/app/main.py: tweak docstring (CEO→ops supervisor) to match the rename. Runbooks -------- - demos/dais2026-runbooks/SETUP.ipynb: new setup runbook. - demos/dais2026-runbooks/LakebaseApps.ipynb: split the monolithic demo markdown into per-section cells for easier presentation flow. - demos/dais2026-runbooks/Genie.ipynb: add "Discover Domains" section.
UCState.add() builds the INSERT INTO ... VALUES (..., '{resource_data}', ...)
statement with raw f-string interpolation, where `resource_data` is the
JSON-serialized representation of the Databricks SDK object being tracked
(app, endpoint, job, etc.).
JSON does NOT escape single quotes — `json.dumps({"description": "Casper's
Ops Dashboard"})` returns the literal `{"description": "Casper's Ops
Dashboard"}` with the apostrophe intact. When that is then interpolated
into the SQL `VALUES ('...{"description": "Casper's Ops Dashboard"}...')`,
the apostrophe in `Casper's` terminates the SQL string literal early and
Spark fails to parse the rest:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'Ops':
extra input 'Ops'. SQLSTATE: 42601
The failure mode bit us when stages/operational_app.ipynb started passing
the ops dashboard's app_status to uc_state.add() — the app description
("Casper's Ops Dashboard ...") flows through unchanged and breaks the
INSERT, failing the Operational_App stage and cascading to every
downstream task (Evaluation, Readiness_Check, etc.).
Fix: double-quote any embedded apostrophe (SQL-standard escape) before
interpolation. Safe to apply unconditionally — re-escaping an already-
escaped value is a no-op when the value never contained one to begin
with. No schema or API change.
…lly work ABAC was failing on every workspace: - CREATE TAG IF NOT EXISTS isn't real SQL (ParseException) - UC tag keys reject dots (caspers.pii -> INVALID_PARAMETER_VALUE) - CREATE POLICY validates EXCEPT principals at create time, so policies referencing the seven demo groups hit PRINCIPAL_DOES_NOT_EXIST on any account that hadn't pre-created them - silver_order_items.customer_addr was never projected from bronze, so the PII tag target failed with COLUMN_NOT_FOUND_IN_TABLE Fixes: - Drop the fake CREATE TAG block; rename all ABAC keys to underscored form (caspers_pii, caspers_geo_region, caspers_value_class, caspers_sensitivity) so application and policy match shapes align - Remove EXCEPT clauses from all 4 policies; bypass logic now lives only in the UDFs via is_account_group_member, which returns false for missing groups. Demo groups become truly optional. - Project customer_addr from bronze body_obj into silver_order_items so the PII column mask has a real column to bind to DQ monitor was silently lying: - Import error swallowed; success message printed regardless of outcome - Misleading "Common causes:" hint listed four possible causes for one error Fixes: - %pip install --upgrade databricks-sdk at top of the notebook so the dataquality module is always available - Each failure mode diagnosed and printed separately - get_monitor verification round-trip after create_monitor - Honest final summary reflecting real state Discover Domain tagging (original ask): three governed tag policies (caspers_domain_operations/revenue/compliance) applied to UC securables, AI/BI dashboards, Genie spaces, and apps via the per-surface tagging API. Shared helpers in utils/domain_tags.py. Underscored keys because the workspace-entity tag API rejects the same reserved characters UC does. SETUP.ipynb cells 1/4/5 rewritten to match reality (removed the "UC SQL DDL accepts dots fine" lie, the "skipped silently" lie about the DQ monitor, and clarified that demo groups are genuinely optional).
Squashes three local commits (057e465, c923378, 65f218d) plus this session's fixes into one for the demo branch. MLflow scorers (across 8 agents) -------------------------------- - 5+ scorers per agent registered at 100% sampling on the managed dev experiment, with idempotent register-or-restart helpers (handles re-runs without ValueError on already-registered scorers). - First-deploy hardening: tolerate eventual-consistency window between register() and start() with retry loop. - Custom @scorer fixes for the supervisor: * import re inlined inside cites_specific_data — module-level imports don't survive pickling for production-monitoring runs, so the registered scorer raised NameError: name 're' not defined. * Both routing_accuracy and cites_specific_data now coerce outputs to text via inline _as_text helper that handles ChatAgent (messages[]), Chat Completions (choices[0].message.content), Responses-API (output[]), final_response, and bare-string shapes. Previously they assumed outputs was a string and crashed in production with 'dict object has no attribute lower'. Managed-agent eval ------------------ - Consolidate per-agent eval flow into demos/operational-dashboard-demo/ evaluation.ipynb so the dev-eval and prod-monitoring scorers are registered from the same notebook against the same managed experiments. ABAC governance / Environment_Helpers ------------------------------------- - Move the geo-region row filter off the live simulator.locations table onto a non-pipeline _security.locations_snapshot. The live table was being filtered to 0 rows for the ops-dashboard app SP (no membership in caspers_geo_admins/_us_users/_emea_users since those account groups don't exist in fresh accounts), which broke the dashboard's revenue query. Demo narrative still works on the snapshot; the ops dashboard now reads the unfiltered live table. - Cleanup statements at top of stage drop the legacy tag/policy on simulator.locations.location_code so re-runs converge cleanly. - PII column mask, high-value-order gate, and regulated-doc deny are unchanged. Ops dashboard frontend ---------------------- - Fix regression from 80a0022: the 4 always-visible action buttons (Executive briefing / Highest combined risk / Top 3 for the board / Review Complaints) were disappearing after 'New session'. clearChat() still had a stale line that hid the entire action-buttons container; now hides only the conditional btn-refund and leaves the container display:flex. New stages / assets ------------------- - stages/ai_gateway_demo.ipynb: new AI gateway demo stage. - utils/fix_agent_perms.ipynb: helper for repairing agent endpoint permissions out-of-band. - utils/casperslogo_green.png: brand asset. - demos/dais2026-runbooks/{Omnigent,SETUP}.ipynb: new runbooks. - demos/dais2026-runbooks/assets/: runbook screenshots and diagrams. - pipelines/order_items: transformation tweaks for the gold layer the dashboards depend on. - Various stage hardening (apps, complaint_*, refund_*, knowledge_*, operational_*, support_request_*) and runbook polish (AgentBricks/Genie/LakebaseApps/MLflow notebooks). Removed ------- - utils/app_thumbnails.py: emoji-thumbnail helper folded into stages/operational_app.ipynb directly.
…y, with auth refresh + ops drift fixes
`all` target only — `default` and `complaints` keep their legacy
direct-FMAPI behaviour.
Unity AI Gateway (v2 Beta) routing
----------------------------------
When the new `AI_GATEWAY_ENDPOINT_NAME` job parameter is non-empty, the
Refund and Complaint agents route their internal LLM calls through the
named gateway endpoint instead of calling the foundation-model endpoint
named by `LLM_MODEL` directly:
- refunder_agent: `ChatDatabricks(endpoint=...)` →
`langchain_openai.ChatOpenAI(base_url=<host>/ai-gateway/mlflow/v1,
api_key=<bearer>)`. langchain-openai added to %pip and to log_model
pip_requirements.
- complaint_agent: `dspy.LM('databricks/<model>')` →
`dspy.LM('openai/<gateway>', api_base=..., api_key=<bearer>)`.
- Bearer is pulled from `WorkspaceClient().config.authenticate()` so it
works under the OAuth M2M credentials Model Serving rotates
(`config.token` returns None in that mode).
- `DatabricksServingEndpoint(endpoint_name=AI_GATEWAY_ENDPOINT_NAME)`
intentionally NOT added to the `resources=` list at log_model time:
v2 Beta gateway endpoints live on a separate API surface and aren't
discoverable via the regular `/api/2.0/serving-endpoints/*` API the
resource type validates against — listing it produces
`NOT_FOUND: Dependent serving endpoint <gateway> does not exist` at
`agents.deploy()` time. CAN_QUERY on the gateway has to be granted
manually post-deploy — SETUP.ipynb step 7 walks through it.
- demos/dais2026-runbooks/SETUP.ipynb step 5: documents manual UI flow
for creating the gateway endpoint (preview enable, inference tables,
usage tracking, guardrails PII=Block + Jailbreak + Unsafe Content,
rate limits).
- demos/dais2026-runbooks/MLflow.ipynb: fills the previously empty
Unity AI Gateway section with demo beats — inference-table query,
usage-tracking query, PII guardrail block via raw `requests.post`.
Correctness fixes for the gateway path
--------------------------------------
- Per-invocation LLM client rebuild. In OAuth M2M mode the SDK
rotates the bearer token ~hourly, but langchain-openai and dspy.LM
cache `api_key` at construction time, so a single client built at
module load goes stale and surfaces as HTTP 400 / "Invalid Token"
from the gateway on warm endpoints. Both agents now rebuild via
`_build_gateway_llm()` / `_build_gateway_lm()` on every request,
pulling fresh bearer from `_w.config.authenticate()`. Direct
`databricks/<model>` mode handles auth refresh internally and is
unaffected.
- DSPy thread-affinity. `dspy.configure(lm=...)` only allows the
thread that first configured it to change settings. Model Serving
invokes `predict()` on a worker thread, so calling `dspy.configure()`
there raised `dspy.settings can only be changed by the thread that
initially configured it` on every gateway-mode request.
`DSPyComplaintAgent.predict()` now uses `with dspy.context(lm=...)`,
the thread-safe per-request override.
operational_app endpoint-name drift fix
---------------------------------------
stages/operational_app.ipynb hardcoded agent endpoint names from the
runtime `CATALOG` widget in two places (SP-grant loop, app.yaml env
construction). Every other consumer in the codebase already reads
from `REFUND_AGENT_ENDPOINT_NAME` / `COMPLAINT_AGENT_ENDPOINT_NAME`
job parameters whose defaults bake at deploy time from `${var.catalog}`.
When `--var catalog` (deploy-time) and `--params CATALOG` (run-time)
disagreed, the agent stages deployed endpoints under the deploy-time
name and operational_app embedded the run-time name into the deployed
App's app.yaml, producing a 404 at user-interaction time:
`.../serving-endpoints/dais2026_refund_agent/invocations` 404 when the
actual endpoint was `caspersdev_refund_agent`.
Both cells now read the widget with a fallback to the legacy
`f"{CATALOG}_{role}_agent"` form so standalone-notebook usage keeps
working.
ops warehouse name pinning
--------------------------
Same `--var` vs `--params` drift bit the ops warehouse: DABs creates
`${var.catalog}-ops-warehouse` at deploy time, but the stage was
reconstructing the name from the runtime CATALOG widget. New
`OPS_WAREHOUSE_NAME` job parameter with default
`${var.catalog}-ops-warehouse` (resolved at deploy time and baked into
the parameter default) — `stages/operational_app.ipynb` reads from it,
with the legacy reconstruction kept as a fallback.
Docs
----
- README.md + AGENTS.md: document the `--var catalog` (deploy-time)
vs `--params CATALOG` (run-time) distinction for the `all` target
and the requirement that they match.
Incidental
----------
- stages/apps.ipynb + demos/operational-dashboard-demo/evaluation.ipynb:
workspace re-export rewrote cell `source` from single concatenated
strings to multi-line arrays; no functional change. evaluation.ipynb
additionally drops one stale guidelines-shape diagnostic cell.
Fix bundle cleanup to honor BUNDLE_VAR_catalog and document that --var does not reach script subprocesses. Refresh README/AGENTS with accurate target descriptions and rebuild steps. Move operational evaluation into stages/, remove legacy generator/raw_data assets, and drop scratch docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
alltarget that unifies refund, complaints, and the operational dashboard (3 Genies, 6 KAs, MAS, Lakebase app, 5 AI/BI dashboards) with DAIS 2026 runbooksSKIP_EVAL=true)BUNDLE_VAR_catalog(not--varonbundle run cleanup)@nkarpov I know it's a lot, but more FYI I guess