diff --git a/console/doc/design/20260706_dataflow_visualizer_rebuild.md b/console/doc/design/20260706_dataflow_visualizer_rebuild.md new file mode 100644 index 0000000000000..1ba63653ce507 --- /dev/null +++ b/console/doc/design/20260706_dataflow_visualizer_rebuild.md @@ -0,0 +1,262 @@ +# Dataflow visualizer rebuild + +- Associated: [CNS-108](https://linear.app/materializeinc/issue/CNS-108), [CNS-109](https://linear.app/materializeinc/issue/CNS-109) + +## The Problem + +The dataflow visualizer (`src/platform/clusters/DataflowVisualizer.tsx`) builds a Graphviz DOT string by hand and renders it with `d3-graphviz`, which pulls in a WASM Graphviz build. +This design has produced two open bugs. +CNS-108: `scopeToGv` concatenates catalog strings into DOT and only escapes double quotes, allowing XSS through crafted object names. +CNS-109: `useSqlApiRequest` never clears previous results, so a failed refetch leaves the stale graph on screen. +Beyond the bugs, the visualizer lacks features operators need: no dataflow selection (one object, one dataflow), no whole-graph overview (drill-down replaces the canvas), no filtering, and no way to find hotspots in large dataflows. + +## Success Criteria + +* A user can pick any dataflow running on a replica, including transient ones, and see its operator graph. +* A user drills into one region at a time, with aggregated stats on regions and jump navigation across scope boundaries. +* Nodes show scheduling time and arrangement size, edges show message counts and container type. +* Filter tools let a user locate operators by name, hide idle elements, heat-color by metric (including per-worker skew), and filter edges by container type. +* Rendering strings from the catalog cannot inject markup (closes CNS-108 by construction). +* A failed refetch always replaces the graph with an error state (closes CNS-109, with a regression test). +* No `d3-graphviz` or WASM dependency remains. + +## Out of Scope + +* Backend changes. + All required data exists in `mz_introspection` relations already queried today. +* Live-updating stats via `SUBSCRIBE`. + The data layer separates structure from stats so this can be added later, but v1 ships manual refresh only. +* URL-persisted filters (search, heatmap mode, hide-idle). + The drill-down scope itself is URL-persisted (see Graph model and drill-down), since back/forward and copyable links depend on it; filters are lower-value to persist and stay in component state. +* New end-to-end tests. + +## Solution Proposal + +Rebuild the visualizer on React Flow (`@xyflow/react`) for rendering and interaction, with elkjs (`elk.layered`) for hierarchical layout in a web worker. +Nodes become React components styled with Chakra, so all catalog strings render as text nodes and the DOT injection class disappears. +React Flow provides pan, zoom, minimap, selection, nested parent nodes, and viewport culling (`onlyRenderVisibleElements`), which matters for dataflows with over a thousand operators. +elkjs is the only mainstream JS layout engine with real compound-graph support: nested regions and edges routed across hierarchy boundaries. +elkjs is GWT-compiled JavaScript, not WASM. + +### Product surface + +Two entry points share one graph implementation. + +* A new "Dataflows" tab on cluster detail: replica selector, then a dataflow list from `mz_dataflows` joined with `mz_records_per_dataflow` and summed `mz_scheduling_elapsed` (name, records, size, elapsed). Clicking a row opens the graph view. This covers transient dataflows such as peeks and subscribes. +* The existing "Visualize" tab on index and materialized view detail deep-links into the cluster page with the object's dataflow preselected, resolved via `mz_compute_exports`. + +Both remain behind the `visualization-features` LaunchDarkly flag. +The old `DataflowVisualizer.tsx` and the `d3-graphviz` dependency are deleted once the new view lands, since the visualizer is its only consumer. +No e2e test references `DataflowVisualizer`, so deletion needs no test migration. + +### Data layer + +The three structure queries from `useDataflowStructure.ts` (operators, channels, LIR operators) are kept, but keyed by dataflow id instead of export id so transient dataflows work. +Channels need no extra join for this: the first element of an operator address is the dataflow id, so the channel query filters on `from_operator_address[1]`. +`mz_lir_mapping` is keyed by `global_id`, so the LIR query joins `mz_compute_exports` on `dataflow_id` to find the dataflow's export ids. +Transient dataflows (peeks, subscribes) are logged in both relations while alive (`CollectionLogging::new` runs for every export id in `compute_state.rs`, `log_lir_mapping` for every built object in `render.rs`), so LIR data works for them too. +A dataflow can have multiple exports, so the LIR panel groups entries by export id. +A new cheap query feeds the dataflow selector list. +A fourth query, `perWorkerStats`, feeds the skew heatmaps: per-worker elapsed time from `mz_scheduling_elapsed_per_worker` and per-worker arrangement size from `mz_arrangement_sizes_per_worker`, full-outer-joined on `(id, worker_id)` since an operator can appear in one source and not the other, and a worker absent from a source never touched that side of it (must not be coalesced to a false zero, since that would understate skew). + +The full structure is fetched client-side in one shot. +Assumption: dataflows stay within tens of thousands of operators and channels, which fits comfortably in memory and within the existing 30 second query timeout, which v1 keeps. + +A new hook returns results tagged with the parameters that produced them. +The render layer discards results whose tag does not match the current selection, and an error always wins over retained data. +This closes CNS-109 without depending on the broken `requestIdRef` logic in `useSqlApiRequest`. + +Refresh is manual: a refresh button plus a "last fetched" timestamp. +Layout, `focusedScope`, and viewport survive a refresh. +Stats update in place, and relayout happens only if the structure changed. +The structure/stats split makes a later `SUBSCRIBE`-driven live mode a drop-in: stats stream into node badges without relayout. + +### Graph model and drill-down + +Pure functions in a new `dataflowGraph.ts` build the region tree from `mz_dataflow_addresses` alone, replacing `collateOperators`. +An operator's parent is the operator whose address is its own minus the last element, so `mz_dataflow_operator_parents` (itself derived from addresses) is dropped to avoid two sources of truth that can disagree across query times. +Every node carries own and transitive stats: arrangement records, size, elapsed, and CPU/memory skew (worst worker over the average, matching `EXPLAIN ANALYZE ... WITH SKEW`). +Transitive stats and skew are computed once when the structure is built, from per-worker vectors merged up the tree; skew itself is recomputed from the merged vector at each level rather than summed. + +Rather than expanding and collapsing regions in place, the view shows exactly one scope's direct children at a time, addressed by a single `focusedScope: NodeId`. +Double-clicking a region navigates into it (`focusedScope` becomes that region); a breadcrumb trail navigates back out. +This avoids the in-place model's edge-remapping and aggregation machinery entirely: a scope's boundary-crossing channels become synthetic port nodes (`kind: "port"`) on the boundary, each carrying a `peers: PortPeer[]` list describing where the crossing goes outside the current view. +A peer records the address on the far side and, when that peer is itself a region, the exact port inside it the crossing lands on (`peerPortId`), so a jump can drill straight to the matching port instead of parking on the peer's outer box. +Jumping is click-to-select-then-jump, or double-click direct when a port has exactly one peer; a fanned-out port (one output feeding several inputs) has no single unambiguous target, so double-click just selects it. +The derivation `deriveVisibleGraph(structure, focusedScope)` is a pure function from a structure and a scope to the visible nodes, edges, and ports; there is no persisted collapse state, so a stale scope from a previous structure resolves at render time by falling back to the new structure's root rather than crashing. + +`focusedScope` lives in the URL, not component state, as the dataflow address it resolves to (a `scope` search param, e.g. `?scope=5.1.2`; omitted at the root). +Every navigation goes through `setSearchParams`, which pushes rather than replaces, so browser back/forward walks the drill-down history and a copied link reopens at the same scope. +The same stale-scope tolerance above covers a scope param that doesn't resolve in the current structure, whether from a stale link, a reshaped dataflow, or no param at all. + +Regions and LIR spans are two different hierarchies. +LIR information appears as a badge and operator text on each node, plus a side panel listing the dataflow's LIR operators from `mz_lir_mapping` as a tree, with member highlighting and multi-pin. +Clicking a panel entry navigates to the scope containing its operators and highlights them. + +```mermaid +flowchart LR + sql[SQL results] --> structure[dataflowGraph.ts: region tree + stats + skew] + structure --> visible["(structure, focusedScope) -> visible nodes/edges/ports"] + visible --> elk[elkjs worker layout] + elk --> rf[React Flow canvas] + rf -- navigate into region, jump to peer, filter --> visible +``` + +### Type contracts + +The units communicate through these shapes, pinned up front so the graph model, the layout worker, and the rendering layer can be built independently. + +```typescript +type Address = number[]; +type NodeId = string; // JSON.stringify(address), as today + +interface NodeStats { + arrangementRecords: bigint; + arrangementSize: bigint; + elapsedNs: bigint; +} + +// Worst worker's value over the average, matching EXPLAIN ANALYZE ... WITH SKEW. +interface SkewStats { + cpuSkew: number; + memorySkew: number; +} + +interface DataflowNode { + id: NodeId; + address: Address; + name: string; + parent: NodeId | null; + children: NodeId[]; // empty for leaf operators + own: NodeStats; + transitive: NodeStats; // own + subtree, precomputed + ownSkew: SkewStats; + transitiveSkew: SkewStats; // recomputed from merged per-worker vectors, not summed + lir: { exportId: string; lirId: string; operator: string }[]; +} + +interface DataflowStructure { + nodes: Map; + root: NodeId; + channels: Channel[]; // as fetched, operator-level +} + +// A boundary crossing out of the current view. peerPortId names the exact +// port inside the peer this crossing lands on, when the peer is a region; +// null for a leaf peer, which has no inside to drill into. +interface PortPeer { + address: Address; + label: string; + messagesSent: bigint; + batchesSent: bigint; + channelTypes: string[]; + peerPortId: NodeId | null; +} + +interface VisibleNode { + id: NodeId; + kind: "operator" | "region" | "port"; + // ... name, stats, transitiveSkew, address, etc. + peers: PortPeer[]; // non-empty only for kind: "port" +} + +interface VisibleGraph { + nodes: VisibleNode[]; // exactly focusedScope's direct children, plus ports + edges: VisibleEdge[]; // directed, scoped to the current view only +} + +// Worker protocol. One in-flight request, stale responses dropped by id. +interface LayoutRequest { + requestId: number; + graph: VisibleGraph; +} +interface LayoutResponse { + requestId: number; + positions: Map; +} +``` + +Filter state and `focusedScope` are owned by the top-level visualizer page component and passed down. +The visible-graph derivation is `deriveVisibleGraph(structure, focusedScope): VisibleGraph`; filters are a separate `decorateGraph(graph, filters)` pass over the result (dimming, heat coloring, highlighting) rather than part of the derivation itself. + +### Rendering and layout + +elkjs runs `elk.layered` in a web worker, direction top to bottom, over exactly the current scope's visible graph. +The console builds with Vite, so layout runs in a Vite module worker (`new Worker(new URL("./layout.worker.ts", import.meta.url), { type: "module" })`) that imports `elkjs/lib/elk.bundled.js` and implements the layout request protocol below. +Phase 1 verified this in both `vite dev` and the production build. +If the module worker fails to bundle or run `elk.bundled.js`, the fallback is running elkjs on the main thread in a lazily loaded chunk, accepting UI stalls during layout. +Navigating into or out of a region recomputes layout for the new scope. +A small "layouting" overlay shows during computation while the canvas stays interactive. +Since elk lays each scope out independently near its own origin, the viewport's pan/zoom from a previous, unrelated scope can land somewhere with nothing from the new one in it. +Once a scope's layout lands, if none of its nodes are actually on screen, the view refits; this is checked once per scope rather than on every render, so it doesn't fight deliberate panning once landed. + +The node component shows name, arranged records, size, and elapsed time. +Default colors keep a two-by-two palette (region versus operator, has arrangement versus not). +Heatmap mode overrides node color. +Edges are labeled with records and batches, dashed when zero messages were sent, with the container type as an edge badge and tooltip. + +Perf guard: drill-down bounds the visible node count structurally, since a view only ever renders one scope's direct children, not the whole graph; viewport culling handles scopes with many children. + +### Filters and interactions + +A toolbar above the canvas offers: + +* Name search: matching operators highlighted, others dimmed, next/previous jump that navigates to the scope containing the current match. +* Hide idle: hides zero-message edges and zero-elapsed operators. +* Heatmap: mode select (off, elapsed, arrangement size, CPU skew, memory skew), sequential color scale, threshold slider that dims nodes below the cutoff, with a legend. + The scale domain is `[0, max]` over the visible nodes' transitive metric. + If the max is zero, all nodes get the neutral base color and the slider is disabled. + +Filters are pure derivations over the visible graph (`decorateGraph`) and compose. +Filter state lives in component state. +Clicking a node or port opens a detail side panel (full name, all stats, LIR operator, address, and for ports, the peers on the far side of the boundary with a jump-to-peer action). +Double-clicking a region navigates into it; double-clicking a port with exactly one peer jumps directly to it, drilling into the peer's matching port when the peer is a region. + +### Error handling + +The first fetch for a selection shows a centered spinner, as today. +An error always replaces the graph. +The CNS-109 regression test from the issue lands, adapted to the new component. +`INSUFFICIENT_PRIVILEGE` keeps the existing USAGE-privilege alert. +A timeout shows an error box with retry. +A replica or dataflow that disappeared on refresh shows a "dataflow no longer exists" empty state and refreshes the selector. + +## Minimal Viable Prototype + +The riskiest assumption is elkjs layout quality and speed on real dataflow shapes. +Phase 1 of the implementation plan is a thin vertical slice: fetch one dataflow, build the region tree, lay out the default collapsed view with elkjs, and render it in React Flow with expand/collapse only. + +The slice runs against a local environmentd with a reference dataflow: an index on a five-way join with at least two aggregations, producing at least 300 operators. +The exact SQL is fixed in the implementation plan. +The gate, all parts required to proceed: + +* Worker loading works in `vite dev` and the production build (see rendering section for the pinned mechanism and fallback). +* Default collapsed view lays out in under 500 ms. +* A fully expanded view of 1500 visible nodes lays out in under 5 s in the worker, UI responsive throughout. +* No overlapping nodes, edges follow top-to-bottom flow. +* Manual sign-off on visual readability by the requester, from screenshots of the reference dataflow collapsed and expanded. + +If the gate fails, work stops and the decision returns to this document: tune ELK options, or fall back to the in-house Canvas alternative, decided with the reviewer rather than unilaterally. + +## Alternatives + +* Extend the in-house `src/components/Graph` Canvas plus dagre stack (used by WorkflowGraph, RoleGraph, CriticalPathGraph). + No new dependencies, but dagre's compound-graph support is weak, in-place collapse and boundary edge rerouting would be hand-built, and the plain SVG canvas has no viewport culling for graphs with over a thousand nodes. + The hard part of this feature is exactly the part dagre is bad at. +* In-house Canvas rendering with elkjs layout. + Keeps console rendering conventions and solves hierarchy, but rebuilds interaction machinery React Flow ships (nested nodes, culling, minimap, selection). +* Keep Graphviz but sanitize DOT properly. + Fixes CNS-108 but keeps the WASM dependency, string-based pipeline, and none of the requested interaction features. + +## Open questions + +* None blocking. + The `SUBSCRIBE` live mode is a deliberate follow-up, not an open design point. + +## Testing + +* Unit (vitest): `dataflowGraph.ts` pure functions. + Region tree construction, skew computation, drill-down derivation, peer-port resolution, filter derivations, LIR membership. + Most logic lives here, so most coverage lands here. +* Component (vitest plus msw): selector flow, error-on-refetch (the CNS-109 case), the stale-scope crash regression, port jump (click and double-click, 1:1 and fan-out), permission alert, empty states. + React Flow renders with the elk worker stubbed to deterministic positions. diff --git a/console/package.json b/console/package.json index 4b2a72ae49490..69cec039294f7 100644 --- a/console/package.json +++ b/console/package.json @@ -59,7 +59,6 @@ "@tanstack/react-query-devtools": "^5.95.0", "@tanstack/react-table": "^8.21.3", "@types/d3": "^7.4.3", - "@types/d3-graphviz": "^2.6.10", "@types/debug": "^4.1.12", "@types/react-table": "^7.7.20", "@visx/axis": "^3.12.0", @@ -72,16 +71,17 @@ "@visx/shape": "^3.12.0", "@visx/tooltip": "^3.12.0", "@xstate/fsm": "^2.1.0", + "@xyflow/react": "^12.11.1", "base32-encoding": "^1.0.0", "buffer": "^6.0.3", "codemirror": "^6.0.1", "copy-to-clipboard": "^3.3.3", "d3": "^7.9.0", - "d3-graphviz": "^5.6.0", "d3-scale-chromatic": "^3.1.0", "date-fns": "^4.1.0", "date-fns-tz": "^3.2.0", "downshift": "^9.0.8", + "elkjs": "^0.11.1", "fast-deep-equal": "^3.1.3", "framer-motion": "^12.38.0", "jotai": "^2.9.3", diff --git a/console/src/api/materialize/dataflow/useDataflowGraphData.test.ts b/console/src/api/materialize/dataflow/useDataflowGraphData.test.ts new file mode 100644 index 0000000000000..66a5198782726 --- /dev/null +++ b/console/src/api/materialize/dataflow/useDataflowGraphData.test.ts @@ -0,0 +1,176 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { renderHook, waitFor } from "@testing-library/react"; +import { http, HttpResponse } from "msw"; + +import { ErrorCode } from "~/api/materialize/types"; +import server from "~/api/mocks/server"; +import { createProviderWrapper } from "~/test/utils"; + +import { useDataflowGraphData } from "./useDataflowGraphData"; + +const FAILING_REPLICA = "r2"; +const okResult = { + desc: { columns: [] }, + rows: [], +}; +// operators result with one root row so buildDataflowStructure succeeds +const operatorsResult = { + desc: { + columns: [ + { name: "id" }, + { name: "address" }, + { name: "name" }, + { name: "arrangementRecords" }, + { name: "arrangementSize" }, + { name: "elapsedNs" }, + ], + }, + rows: [["10", ["7"], "Dataflow", "0", "0", "0"]], +}; + +beforeEach(() => { + server.use( + http.post("*/api/sql", async ({ request }) => { + const options = JSON.parse( + new URL(request.url).searchParams.get("options") ?? "{}", + ); + if (options.cluster_replica === FAILING_REPLICA) { + return HttpResponse.json({ + results: [ + { + error: { + message: "no such replica", + code: ErrorCode.INTERNAL_ERROR, + }, + }, + ], + }); + } + return HttpResponse.json({ + results: [operatorsResult, okResult, okResult, okResult, okResult], + }); + }), + ); +}); + +describe("useDataflowGraphData", () => { + it("clears data when a refetch for new params fails", async () => { + // The hook reads the current environment off a jotai atom that resolves + // asynchronously, so it needs the same Suspense + JotaiProvider wrapper + // the app renders under. Without it the atom never resolves against the + // fake environment set up in vitest.setup.ts and the query never fires. + const Wrapper = await createProviderWrapper(); + const { result, rerender } = renderHook( + (params) => useDataflowGraphData(params), + { + wrapper: Wrapper, + initialProps: { + clusterName: "c", + replicaName: "r1", + dataflowId: "7", + } as + | { clusterName: string; replicaName: string; dataflowId: string } + | undefined, + }, + ); + await waitFor(() => expect(result.current.data).not.toBeNull()); + + rerender({ + clusterName: "c", + replicaName: FAILING_REPLICA, + dataflowId: "7", + }); + await waitFor(() => expect(result.current.error).toBeTruthy()); + // The retained previous result must not surface for the new params. + expect(result.current.data).toBeNull(); + }); + + it("hides the previous graph while a new selection is loading", async () => { + const Wrapper = await createProviderWrapper(); + const { result, rerender } = renderHook( + (params) => useDataflowGraphData(params), + { + wrapper: Wrapper, + initialProps: { + clusterName: "c", + replicaName: "r1", + dataflowId: "7", + } as + | { clusterName: string; replicaName: string; dataflowId: string } + | undefined, + }, + ); + await waitFor(() => expect(result.current.data).not.toBeNull()); + const firstStructure = result.current.data?.structure; + + // Switch to a different valid dataflow whose fetch also succeeds. The + // previous fetch's results are still resident, so a naive guard would tag + // them with the new key and render the old graph. React flushes the + // render-phase reset synchronously, so data must already be null here. + rerender({ clusterName: "c", replicaName: "r1", dataflowId: "8" }); + expect(result.current.data).toBeNull(); + + await waitFor(() => expect(result.current.data).not.toBeNull()); + expect(result.current.data?.structure).not.toBe(firstStructure); + }); + + it("threads the replica's worker count through from workers * processes", async () => { + server.use( + http.post("*/api/sql", async () => { + return HttpResponse.json({ + results: [ + operatorsResult, + okResult, + okResult, + okResult, + { + desc: { columns: [{ name: "workerCount" }] }, + rows: [["8"]], + }, + ], + }); + }), + ); + const Wrapper = await createProviderWrapper(); + const { result } = renderHook( + () => + useDataflowGraphData({ + clusterName: "c", + replicaName: "r1", + dataflowId: "7", + }), + { wrapper: Wrapper }, + ); + await waitFor(() => expect(result.current.data).not.toBeNull()); + // 8, not the 1-worker fallback: proves the query result actually flows + // through, not just that the field exists with its default value. + expect(result.current.data?.workerCount).toEqual(8); + }); + + it("rejects a non-numeric dataflow id without ever compiling a query for it", async () => { + const Wrapper = await createProviderWrapper(); + // A malformed id is only reachable via a hand-edited URL, so this must + // report an error the page can render, not throw and crash to the error + // boundary over a URL typo. + const { result } = renderHook( + () => + useDataflowGraphData({ + clusterName: "c", + replicaName: "r1", + dataflowId: "7; DROP", + }), + { wrapper: Wrapper }, + ); + expect(result.current.error).toEqual("invalid dataflow id: 7; DROP"); + expect(result.current.loading).toEqual(false); + expect(result.current.data).toBeNull(); + }); +}); diff --git a/console/src/api/materialize/dataflow/useDataflowGraphData.ts b/console/src/api/materialize/dataflow/useDataflowGraphData.ts new file mode 100644 index 0000000000000..0f3d55841a3f7 --- /dev/null +++ b/console/src/api/materialize/dataflow/useDataflowGraphData.ts @@ -0,0 +1,249 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { sql } from "kysely"; +import React from "react"; + +import { escapedLiteral as lit, useSqlManyTyped } from "~/api/materialize"; +import { useLastGoodByKey } from "~/api/materialize/useLastGoodByKey"; +import { + buildDataflowStructure, + type ChannelRow, + type LirSpanRow, + type OperatorRow, + type PerWorkerStatRow, +} from "~/platform/dataflows/dataflowGraph"; + +import { queryBuilder } from "../db"; + +export interface DataflowGraphParams { + clusterName: string; + replicaName: string; + dataflowId: string; +} + +interface ReplicaWorkerCountRow { + workerCount: bigint | number | string; +} + +// A malformed id can only reach this hook via a hand-edited URL (the route +// itself only ever supplies real dataflow ids), so this returns undefined +// for the caller to render as a not-found state, rather than throwing +// inside useMemo and crashing to the error boundary over a typo in the +// address bar. +function dataflowIdLiteral(dataflowId: string) { + if (!/^\d+$/.test(dataflowId)) return undefined; + return sql`${lit(dataflowId)}::uint8`; +} + +export function useDataflowGraphData(params?: DataflowGraphParams) { + // Read primitives so an inline params object from a caller does not + // recompile the queries (and refetch) on every render. Only the dataflow id + // is interpolated into the SQL. Cluster and replica reach the four + // dataflow-scoped queries below through the request options, so their text + // doesn't depend on them; replicaWorkers is the one exception, since + // mz_cluster_replicas/mz_cluster_replica_sizes are plain catalog tables, + // not per-replica-scoped introspection relations that the session context + // resolves automatically. + const dataflowId = params?.dataflowId; + const clusterName = params?.clusterName; + const replicaName = params?.replicaName; + const queries = React.useMemo(() => { + if ( + dataflowId === undefined || + clusterName === undefined || + replicaName === undefined + ) + return null; + const id = dataflowIdLiteral(dataflowId); + if (id === undefined) return null; + return { + operators: sql` + SELECT + mdod.id, + mda.address, + mdod.name, + coalesce(mas.records, 0) AS "arrangementRecords", + coalesce(mas.size, 0) AS "arrangementSize", + coalesce(mse.elapsed_ns, 0) AS "elapsedNs", + coalesce(mcodh.count, 0) AS "scheduleCount" + FROM mz_dataflow_operator_dataflows AS mdod + JOIN mz_dataflow_addresses AS mda ON mda.id = mdod.id + LEFT JOIN mz_arrangement_sizes AS mas ON mas.operator_id = mdod.id + LEFT JOIN mz_scheduling_elapsed AS mse ON mse.id = mdod.id + LEFT JOIN ( + SELECT id, sum(count) AS count + FROM mz_compute_operator_durations_histogram + GROUP BY id + ) AS mcodh ON mcodh.id = mdod.id + WHERE mdod.dataflow_id = ${id}` + .$castTo() + .compile(queryBuilder), + channels: sql` + SELECT + mdco.id, + from_operator_address AS "fromOperatorAddress", + mdc.from_port AS "fromPort", + to_operator_address AS "toOperatorAddress", + mdc.to_port AS "toPort", + COALESCE(sum(mmc.sent), 0) AS "messagesSent", + COALESCE(sum(mmc.batch_sent), 0) AS "batchesSent", + mdco.type AS "channelType" + FROM mz_dataflow_channel_operators AS mdco + JOIN mz_dataflow_channels AS mdc ON mdc.id = mdco.id + LEFT JOIN mz_message_counts AS mmc ON mdco.id = mmc.channel_id + WHERE from_operator_address[1] = ${id} + GROUP BY + mdco.id, "fromOperatorAddress", mdc.from_port, + "toOperatorAddress", mdc.to_port, mdco.type` + .$castTo() + .compile(queryBuilder), + // mz_lir_mapping is keyed by the built object's global id (often a + // transient id), which mz_compute_exports does not bridge to a dataflow. + // Operator id ranges do: a span belongs to this dataflow when one of the + // dataflow's operators falls inside its [start, end) range. + lirSpans: sql` + SELECT DISTINCT + mlm.global_id AS "exportId", + mlm.lir_id::text AS "lirId", + mlm.parent_lir_id::text AS "parentLirId", + mlm.nesting::int4 AS nesting, + mlm.operator, + mlm.operator_id_start AS "operatorIdStart", + mlm.operator_id_end AS "operatorIdEnd" + FROM mz_lir_mapping AS mlm + WHERE EXISTS ( + SELECT 1 + FROM mz_dataflow_operator_dataflows AS dod + WHERE dod.dataflow_id = ${id} + AND dod.id >= mlm.operator_id_start + AND dod.id < mlm.operator_id_end + )` + .$castTo() + .compile(queryBuilder), + // Per-worker CPU/memory/schedule count, for the skew heatmap (worst + // worker over the average). Three independent per-worker sources + // full-outer-joined on (id, worker_id): an operator can show up in + // one and not the others (e.g. it schedules but never arranges + // anything), and a worker with no row in a source never touched that + // side of it at all, which matters for skew (see dataflowGraph.ts) + // and must not be coalesced to a false zero here. + perWorkerStats: sql` + WITH cpu AS ( + SELECT mdod.id, mse.worker_id, mse.elapsed_ns + FROM mz_dataflow_operator_dataflows AS mdod + JOIN mz_scheduling_elapsed_per_worker AS mse ON mse.id = mdod.id + WHERE mdod.dataflow_id = ${id} + ), + memory AS ( + SELECT mdod.id, mas.worker_id, mas.size + FROM mz_dataflow_operator_dataflows AS mdod + JOIN mz_arrangement_sizes_per_worker AS mas ON mas.operator_id = mdod.id + WHERE mdod.dataflow_id = ${id} + ), + schedules AS ( + SELECT mdod.id, mcodhpw.worker_id, sum(mcodhpw.count) AS count + FROM mz_dataflow_operator_dataflows AS mdod + JOIN mz_compute_operator_durations_histogram_per_worker AS mcodhpw + ON mcodhpw.id = mdod.id + WHERE mdod.dataflow_id = ${id} + GROUP BY mdod.id, mcodhpw.worker_id + ) + SELECT + coalesce(cpu.id, memory.id, schedules.id) AS id, + coalesce(cpu.worker_id, memory.worker_id, schedules.worker_id) + AS "workerId", + cpu.elapsed_ns AS "elapsedNs", + memory.size AS "arrangementSize", + schedules.count AS "scheduleCount" + FROM cpu + FULL OUTER JOIN memory + ON memory.id = cpu.id AND memory.worker_id = cpu.worker_id + FULL OUTER JOIN schedules + ON schedules.id = coalesce(cpu.id, memory.id) + AND schedules.worker_id = coalesce(cpu.worker_id, memory.worker_id)` + .$castTo() + .compile(queryBuilder), + // The replica's total worker count (workers per process times + // processes), the fixed ceiling the skew heatmap normalizes against + // (see decorateGraph). Plain catalog tables, not introspection, so + // this is the one query here that needs cluster/replica in its text + // rather than relying on the request's session context. + replicaWorkers: sql` + SELECT crs.workers * crs.processes AS "workerCount" + FROM mz_cluster_replicas AS cr + JOIN mz_clusters AS c ON c.id = cr.cluster_id + JOIN mz_cluster_replica_sizes AS crs ON crs.size = cr.size + WHERE c.name = ${lit(clusterName)} AND cr.name = ${lit(replicaName)}` + .$castTo() + .compile(queryBuilder), + }; + }, [dataflowId, clusterName, replicaName]); + + const { + results, + error: queryError, + databaseError, + loading: queryLoading, + refetch, + } = useSqlManyTyped(queries, { + cluster: params?.clusterName, + replica: params?.replicaName, + // This query can be slow for large dataflows. + timeout: 30_000, + }); + + // A malformed dataflow id (only reachable via a hand-edited URL) never + // compiles a query, so `loading` would otherwise stay false forever with + // no error to show. Surfacing it here, rather than throwing inside the + // queries useMemo above, lets the page render a real error message + // instead of crashing to the error boundary over a URL typo. + const invalidDataflowId = + dataflowId !== undefined && dataflowIdLiteral(dataflowId) === undefined; + const error = invalidDataflowId + ? `invalid dataflow id: ${dataflowId}` + : queryError; + const loading = !invalidDataflowId && queryLoading; + + // JSON tuple, not a delimiter-joined string. Cluster and replica names are + // identifiers that can themselves contain any delimiter we might pick. + const key = params + ? JSON.stringify([ + params.clusterName, + params.replicaName, + params.dataflowId, + ]) + : null; + + const compute = React.useCallback( + (r: NonNullable) => ({ + structure: buildDataflowStructure( + r.operators ?? [], + r.channels ?? [], + r.lirSpans ?? [], + r.perWorkerStats ?? [], + ), + // A replica always has at least 1 worker; falling back to 1 (a heatmap + // ceiling of log2(1) = 0, disabling skew coloring entirely) only + // matters if the query somehow returns no row. + workerCount: Number(r.replicaWorkers?.[0]?.workerCount ?? 1), + fetchedAt: new Date(), + }), + [], + ); + const data = useLastGoodByKey({ + key, + results, + error, + loading, + compute, + }); + + return { data, error, databaseError, loading, refetch }; +} diff --git a/console/src/api/materialize/dataflow/useDataflowIdForExport.ts b/console/src/api/materialize/dataflow/useDataflowIdForExport.ts new file mode 100644 index 0000000000000..4bcd8929e9330 --- /dev/null +++ b/console/src/api/materialize/dataflow/useDataflowIdForExport.ts @@ -0,0 +1,51 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { sql } from "kysely"; +import React from "react"; + +import { escapedLiteral as lit, useSqlManyTyped } from "~/api/materialize"; + +import { queryBuilder } from "../db"; + +export interface DataflowIdForExportParams { + clusterName: string; + replicaName: string; + exportId: string; +} + +interface DataflowIdRow { + dataflowId: string; +} + +export function useDataflowIdForExport(params?: DataflowIdForExportParams) { + const exportId = params?.exportId; + const queries = React.useMemo(() => { + // Export ids are of the form s/t/u. Reject anything else before + // interpolating so a malformed param can never reach the query. + if (exportId === undefined || !/^[stu]\d+$/.test(exportId)) return null; + return { + row: sql` + SELECT dataflow_id::text AS "dataflowId" + FROM mz_compute_exports + WHERE export_id = ${lit(exportId)}` + .$castTo() + .compile(queryBuilder), + }; + }, [exportId]); + + const { results, error, databaseError, loading } = useSqlManyTyped(queries, { + cluster: params?.clusterName, + replica: params?.replicaName, + }); + + const dataflowId = + !error && results ? (results.row?.[0]?.dataflowId ?? null) : null; + return { dataflowId, loading, error, databaseError }; +} diff --git a/console/src/api/materialize/dataflow/useDataflowList.test.ts b/console/src/api/materialize/dataflow/useDataflowList.test.ts new file mode 100644 index 0000000000000..250315c2c1d24 --- /dev/null +++ b/console/src/api/materialize/dataflow/useDataflowList.test.ts @@ -0,0 +1,114 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { renderHook, waitFor } from "@testing-library/react"; +import { http, HttpResponse } from "msw"; + +import { ErrorCode } from "~/api/materialize/types"; +import server from "~/api/mocks/server"; +import { createProviderWrapper } from "~/test/utils"; + +import { useDataflowList } from "./useDataflowList"; + +const FAILING_REPLICA = "r2"; +const COLUMNS = [ + { name: "id" }, + { name: "name" }, + { name: "records" }, + { name: "size" }, + { name: "elapsedNs" }, +]; +const listResult = { + desc: { columns: COLUMNS }, + rows: [["7", "Dataflow: mv", "100", "4096", "12345"]], +}; +// Distinct ids from listResult: dataflow ids are per-replica, so a stale row +// from the wrong replica reads as an entirely different, meaningless entry +// rather than merely out of date. +const otherReplicaResult = { + desc: { columns: COLUMNS }, + rows: [["9", "Dataflow: other_mv", "200", "8192", "54321"]], +}; + +beforeEach(() => { + server.use( + http.post("*/api/sql", async ({ request }) => { + const options = JSON.parse( + new URL(request.url).searchParams.get("options") ?? "{}", + ); + if (options.cluster_replica === FAILING_REPLICA) { + return HttpResponse.json({ + results: [ + { + error: { + message: "no such replica", + code: ErrorCode.INTERNAL_ERROR, + }, + }, + ], + }); + } + if (options.cluster_replica === "r3") { + return HttpResponse.json({ results: [otherReplicaResult] }); + } + return HttpResponse.json({ results: [listResult] }); + }), + ); +}); + +describe("useDataflowList", () => { + it("maps rows to DataflowListEntry with bigint fields", async () => { + const Wrapper = await createProviderWrapper(); + const { result } = renderHook( + () => useDataflowList({ clusterName: "c", replicaName: "r1" }), + { wrapper: Wrapper }, + ); + await waitFor(() => expect(result.current.data).not.toBeNull()); + expect(result.current.data?.[0]).toEqual({ + id: "7", + name: "Dataflow: mv", + records: 100n, + size: 4096n, + elapsedNs: 12345n, + }); + }); + + it("hides the previous replica's list while the new one is loading", async () => { + const Wrapper = await createProviderWrapper(); + const { result, rerender } = renderHook( + (replicaName: string) => + useDataflowList({ clusterName: "c", replicaName }), + { wrapper: Wrapper, initialProps: "r1" }, + ); + await waitFor(() => expect(result.current.data).not.toBeNull()); + expect(result.current.data?.[0].id).toBe("7"); + + // Switching replicas must not show r1's still-resident rows tagged as + // r3's: dataflow ids are per-replica, so navigating one would be + // meaningless, and a transient dataflow gone on r3 would crash instead. + rerender("r3"); + expect(result.current.data).toBeNull(); + + await waitFor(() => expect(result.current.data).not.toBeNull()); + expect(result.current.data?.[0].id).toBe("9"); + }); + + it("clears data when the fetch fails", async () => { + // The hook reads the current environment off a jotai atom that resolves + // asynchronously, so it needs the same Suspense + JotaiProvider wrapper the + // app renders under. + const Wrapper = await createProviderWrapper(); + const { result } = renderHook( + () => useDataflowList({ clusterName: "c", replicaName: FAILING_REPLICA }), + { wrapper: Wrapper }, + ); + await waitFor(() => expect(result.current.error).toBeTruthy()); + expect(result.current.data).toBeNull(); + }); +}); diff --git a/console/src/api/materialize/dataflow/useDataflowList.ts b/console/src/api/materialize/dataflow/useDataflowList.ts new file mode 100644 index 0000000000000..168c3c5426f95 --- /dev/null +++ b/console/src/api/materialize/dataflow/useDataflowList.ts @@ -0,0 +1,100 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { sql } from "kysely"; +import React from "react"; + +import { useSqlManyTyped } from "~/api/materialize"; +import { useLastGoodByKey } from "~/api/materialize/useLastGoodByKey"; + +import { queryBuilder } from "../db"; + +export interface DataflowListParams { + clusterName: string; + replicaName: string; +} + +export interface DataflowListEntry { + id: string; // uint8 as string + name: string; + records: bigint; + size: bigint; + elapsedNs: bigint; +} + +// Raw row shape as returned by the query. The numeric columns arrive as +// strings and are widened to bigint in normalize. +interface DataflowListRow { + id: string; + name: string; + records: string; + size: string; + elapsedNs: string; +} + +function normalize(rows: DataflowListRow[]): DataflowListEntry[] { + return rows.map((row) => ({ + id: row.id, + name: row.name, + records: BigInt(row.records), + size: BigInt(row.size), + elapsedNs: BigInt(row.elapsedNs), + })); +} + +export function useDataflowList(params?: DataflowListParams) { + // The query text is constant. Cluster and replica reach the request through + // the options below, so recompiling only depends on whether params exist. + const enabled = params !== undefined; + const queries = React.useMemo(() => { + if (!enabled) return null; + return { + list: sql` + SELECT + md.id::text AS id, + md.name, + COALESCE(mrpd.records, 0) AS records, + COALESCE(mrpd.size, 0) AS size, + COALESCE(el.elapsed_ns, 0) AS "elapsedNs" + FROM mz_dataflows AS md + LEFT JOIN mz_records_per_dataflow AS mrpd ON mrpd.id = md.id + LEFT JOIN ( + SELECT mdod.dataflow_id, sum(mse.elapsed_ns) AS elapsed_ns + FROM mz_dataflow_operator_dataflows AS mdod + JOIN mz_scheduling_elapsed AS mse ON mse.id = mdod.id + GROUP BY mdod.dataflow_id + ) AS el ON el.dataflow_id = md.id + ORDER BY size DESC` + .$castTo() + .compile(queryBuilder), + }; + }, [enabled]); + + const { results, error, databaseError, loading, refetch } = useSqlManyTyped( + queries, + { + cluster: params?.clusterName, + replica: params?.replicaName, + }, + ); + + // JSON tuple, not a delimiter-joined string: identifiers can themselves + // contain any delimiter picked. + const key = params + ? JSON.stringify([params.clusterName, params.replicaName]) + : null; + + const compute = React.useCallback( + (r: NonNullable) => normalize(r.list ?? []), + [], + ); + const data = useLastGoodByKey({ key, results, error, loading, compute }); + + return { data, error, databaseError, loading, refetch }; +} diff --git a/console/src/api/materialize/executeSql.tsx b/console/src/api/materialize/executeSql.tsx index 21d6588f98e6f..d2a8d49a433bc 100644 --- a/console/src/api/materialize/executeSql.tsx +++ b/console/src/api/materialize/executeSql.tsx @@ -62,6 +62,17 @@ export function isExecuteSqlError(error: unknown): error is ExecuteSqlError { return error != null && typeof error === "object" && "errorMessage" in error; } +/** True for a `databaseError` denied by a missing USAGE/SELECT grant. */ +export function isInsufficientPrivilegeError( + error: ExecuteSqlError | null | undefined, +): boolean { + return ( + !!error && + "code" in error && + error.code === ErrorCode.INSUFFICIENT_PRIVILEGE + ); +} + export function buildExecuteSqlUrl(environmentdHttpAddress: string) { return new URL( `${apiClient.mzHttpUrlScheme}://${environmentdHttpAddress}/api/sql`, diff --git a/console/src/api/materialize/useDataflowStructure.ts b/console/src/api/materialize/useDataflowStructure.ts deleted file mode 100644 index a5cc9fe897f86..0000000000000 --- a/console/src/api/materialize/useDataflowStructure.ts +++ /dev/null @@ -1,162 +0,0 @@ -// Copyright Materialize, Inc. and contributors. All rights reserved. -// -// Use of this software is governed by the Business Source License -// included in the LICENSE file. -// -// As of the Change Date specified in that file, in accordance with -// the Business Source License, use of this software will be governed -// by the Apache License, Version 2.0. - -import { sql } from "kysely"; -import React from "react"; - -import { escapedLiteral as lit, useSqlManyTyped } from "~/api/materialize"; - -import { queryBuilder } from "./db"; - -export interface DataflowStructureParams { - clusterName: string; - replicaName: string; - objectId: string; -} - -export type DataflowStructureResult = ReturnType< - typeof useDataflowStructure ->["results"]; - -export type Operator = { - id: bigint; - address: string[]; - name: string; - parentId: bigint | null; - arrangementRecords: bigint | null; - arrangementSizes: bigint | null; - elapsedNs: number; - lirId: string | null; - lirOperator: string | null; -}; - -export type Channel = { - id: number; - fromOperatorId: number; - fromOperatorAddress: string[]; - fromPort: number; - toOperatorId: number; - toOperatorAddress: string[]; - toPort: number; - messagesSent: number; - batchesSent: number; - channelType: string; -}; - -export type LirOperator = { - lir_id: string; - operator: string; - addresses: string[][]; -}; - -export function useDataflowStructure(params?: DataflowStructureParams) { - const queries = React.useMemo(() => { - if (!params) { - return null; - } - const { objectId } = params; - - return { - channels: sql` - SELECT - mdco.id, - from_operator_id AS "fromOperatorId", - from_operator_address AS "fromOperatorAddress", - from_port AS "fromPort", - to_operator_id AS "toOperatorId", - to_operator_address AS "toOperatorAddress", - to_port AS "toPort", - COALESCE(sum(sent), 0) AS "messagesSent", - COALESCE(sum(batch_sent), 0) AS "batchesSent", - mdc.type as "channelType" - FROM - mz_dataflow_channel_operators AS mdco - JOIN mz_dataflow_channels AS mdc ON - mdc.id = mdco.id - LEFT JOIN mz_message_counts AS mmc ON - mdco.id = mmc.channel_id - JOIN mz_compute_exports mce ON mce.dataflow_id = from_operator_address[1] - WHERE mce.export_id = ${lit(objectId)} - GROUP BY - mdco.id, - "fromOperatorId", - "fromOperatorAddress", - "toOperatorId", - "toOperatorAddress", - "fromPort", - "toPort", - mdc."type"` - .$castTo() - .compile(queryBuilder), - - operators: sql` - WITH - export_to_dataflow AS ( - SELECT export_id, id FROM mz_compute_exports AS mce JOIN mz_dataflows AS md ON mce.dataflow_id = md.id - ), - all_ops AS ( - SELECT - e2d.export_id, mdod.id, mda.address, mdod.name, mdop.parent_id AS "parentId", - coalesce(mas.records, 0) AS "arrangementRecords", - coalesce(mas.size, 0) AS "arrangementSizes", - coalesce(mse.elapsed_ns, 0) AS "elapsedNs", - mlm.lir_id::text AS "lirId", - mlm.operator AS "lirOperator" - FROM - export_to_dataflow AS e2d - JOIN mz_dataflow_operator_dataflows - AS mdod ON e2d.id = mdod.dataflow_id - LEFT JOIN mz_scheduling_elapsed - AS mse ON mdod.id = mse.id - LEFT JOIN mz_arrangement_sizes - AS mas ON mdod.id = mas.operator_id - LEFT JOIN mz_dataflow_operator_parents - AS mdop ON mdod.id = mdop.id - LEFT JOIN mz_dataflow_addresses - AS mda ON mdod.id = mda.id - LEFT JOIN mz_lir_mapping - AS mlm ON mlm.operator_id_start <= mdod.id AND mdod.id < mlm.operator_id_end AND mlm.global_id = export_id - ) - SELECT - id, - address, - name, - "parentId", - "arrangementRecords", - "arrangementSizes", - "elapsedNs", - "lirId", - "lirOperator" - FROM all_ops WHERE export_id = ${lit(objectId)}` - .$castTo() - .compile(queryBuilder), - - lir_operators: sql` - SELECT lir_id, operator, list_agg(mda.address) AS addresses - FROM - mz_lir_mapping AS mlm - LEFT JOIN mz_dataflow_operators AS mdo - ON (mlm.operator_id_start <= mdo.id - AND mdo.id < mlm.operator_id_end) - JOIN mz_dataflow_addresses AS mda - ON mdo.id = mda.id - WHERE mlm.global_id = ${lit(objectId)} - GROUP BY lir_id, operator` - .$castTo() - .compile(queryBuilder), - }; - }, [params]); - const { clusterName, replicaName } = params ?? {}; - return useSqlManyTyped(queries, { - cluster: clusterName, - replica: replicaName, - // This query can be slow for large dataflows - timeout: 30_000, - }); -} diff --git a/console/src/api/materialize/useLastGoodByKey.test.ts b/console/src/api/materialize/useLastGoodByKey.test.ts new file mode 100644 index 0000000000000..6422150d02783 --- /dev/null +++ b/console/src/api/materialize/useLastGoodByKey.test.ts @@ -0,0 +1,130 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { renderHook } from "@testing-library/react"; + +import { + useLastGoodByKey, + type UseLastGoodByKeyParams, +} from "./useLastGoodByKey"; + +type Params = UseLastGoodByKeyParams; + +function setup(initial: Params) { + return renderHook((props: Params) => useLastGoodByKey(props), { + initialProps: initial, + }); +} + +const compute = (r: string) => `computed:${r}`; + +// A real query hook always starts loading on first mount for a fresh key, +// only later flipping to `loading: false` with a result. Mounting directly +// in the already-loaded state (skipping that transition) would never set +// sawLoadingRef, so this drives a hook through the realistic sequence +// before a test builds further scenarios on top of an accepted result. +function primeAccepted( + key: string, + results: string, + fn: (r: string) => string, +) { + const rendered = setup({ + key, + results: null, + error: null, + loading: true, + compute: fn, + }); + rendered.rerender({ key, results, error: null, loading: false, compute: fn }); + return rendered; +} + +describe("useLastGoodByKey", () => { + it("returns null until a result arrives", () => { + const { result } = setup({ + key: "a", + results: null, + error: null, + loading: true, + compute, + }); + expect(result.current).toBeNull(); + }); + + it("accepts a result once loading finishes for the same key", () => { + const { result, rerender } = setup({ + key: "a", + results: null, + error: null, + loading: true, + compute, + }); + rerender({ key: "a", results: "r1", error: null, loading: false, compute }); + expect(result.current).toEqual("computed:r1"); + }); + + it("keeps showing the previous key's data while a new key is loading, not stale data mislabeled under the new key", () => { + const { result, rerender } = primeAccepted("a", "r1", compute); + expect(result.current).toEqual("computed:r1"); + + // Key changes; a fetch hasn't started yet (loading still false from the + // previous render). Without CNS-109's guard, the still-resident "r1" + // result would get relabeled as belonging to key "b". + rerender({ key: "b", results: "r1", error: null, loading: false, compute }); + expect(result.current).toBeNull(); + + // The new key's fetch starts (loading flips true), still with no new + // result yet. + rerender({ key: "b", results: "r1", error: null, loading: true, compute }); + expect(result.current).toBeNull(); + + // The new key's own result arrives. + rerender({ key: "b", results: "r2", error: null, loading: false, compute }); + expect(result.current).toEqual("computed:r2"); + }); + + it("clears data immediately on error, even before a refetch resolves", () => { + const { result, rerender } = primeAccepted("a", "r1", compute); + expect(result.current).toEqual("computed:r1"); + + rerender({ + key: "a", + results: "r1", + error: "boom", + loading: false, + compute, + }); + expect(result.current).toBeNull(); + }); + + it("recomputes only when compute is referentially stable across renders with the same results", () => { + let calls = 0; + const countingCompute = (r: string) => { + calls++; + return `computed:${r}`; + }; + const { rerender } = primeAccepted("a", "r1", countingCompute); + expect(calls).toEqual(1); + rerender({ + key: "a", + results: "r1", + error: null, + loading: false, + compute: countingCompute, + }); + rerender({ + key: "a", + results: "r1", + error: null, + loading: false, + compute: countingCompute, + }); + expect(calls).toEqual(1); + }); +}); diff --git a/console/src/api/materialize/useLastGoodByKey.ts b/console/src/api/materialize/useLastGoodByKey.ts new file mode 100644 index 0000000000000..eed6c83ede00c --- /dev/null +++ b/console/src/api/materialize/useLastGoodByKey.ts @@ -0,0 +1,72 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import React from "react"; + +export interface UseLastGoodByKeyParams { + // Null when there's nothing to fetch yet (e.g. required params missing). + // Identifies the request a result belongs to (e.g. a JSON tuple of the + // params that vary the query), not the query text itself. + key: string | null; + results: TResult | null | undefined; + error: unknown; + loading: boolean; + // Must be referentially stable (e.g. wrapped in useCallback with an empty + // or narrow dependency list) or this recomputes on every render even when + // `results` hasn't actually changed. + compute: (results: TResult) => TData; +} + +/** + * Retains the last successfully computed value for a given key, so a caller + * keeps showing prior data while a new fetch for a *different* key is in + * flight, without briefly flashing the previous key's data under the new + * one. A result only replaces `lastGood` once a fetch has actually started + * under the current key (see `sawLoadingRef` below): without that guard, a + * key change followed by data that hasn't refetched yet would tag the old + * fetch's still-resident result with the new key in the render before the + * underlying query's own effect flips `loading` (CNS-109). + */ +export function useLastGoodByKey({ + key, + results, + error, + loading, + compute, +}: UseLastGoodByKeyParams): TData | null { + const [lastGood, setLastGood] = React.useState<{ + key: string; + data: TData; + } | null>(null); + + const sawLoadingRef = React.useRef(false); + + // Reset acceptance synchronously when the key changes, using the + // render-phase state-adjustment pattern so the reset lands before any + // effect runs, not one render later. + const [trackedKey, setTrackedKey] = React.useState(key); + if (key !== trackedKey) { + setTrackedKey(key); + // Deliberate render-phase write, not a side effect: it must land before + // this render commits, so a stale sawLoadingRef can't be read by the + // effect below in the same commit (one render later would be too late). + // eslint-disable-next-line react-compiler/react-compiler + sawLoadingRef.current = false; + } + + React.useEffect(() => { + if (loading) sawLoadingRef.current = true; + if (key && results != null && !error && !loading && sawLoadingRef.current) { + setLastGood({ key, data: compute(results) }); + } + }, [key, results, error, loading, compute]); + + // Error always wins, and data for other keys never renders. + return !error && lastGood && lastGood.key === key ? lastGood.data : null; +} diff --git a/console/src/platform/clusters/ClusterDetail.tsx b/console/src/platform/clusters/ClusterDetail.tsx index a729379cb17d3..a8dfb25188851 100644 --- a/console/src/platform/clusters/ClusterDetail.tsx +++ b/console/src/platform/clusters/ClusterDetail.tsx @@ -15,6 +15,7 @@ import { isSystemCluster } from "~/api/materialize"; import { ClusterWithOwnership } from "~/api/materialize/cluster/clusterList"; import DeleteObjectMenuItem from "~/components/DeleteObjectMenuItem"; import OverflowMenu from "~/components/OverflowMenu"; +import { useFlags } from "~/hooks/useFlags"; import { Breadcrumb, PageBreadcrumbs, @@ -40,6 +41,13 @@ import Sinks from "./Sinks"; import Sources from "./Sources"; import { useShowSystemObjects } from "./useShowSystemObjects"; +const DataflowDetailPage = React.lazy( + () => import("~/platform/dataflows/DataflowDetailPage"), +); +const DataflowsPage = React.lazy( + () => import("~/platform/dataflows/DataflowsPage"), +); + const ClusterDetailBreadcrumbs = (props: { crumbs: Breadcrumb[] }) => { const [showSystemObjects] = useShowSystemObjects(); const { clusterId, clusterName } = useParams(); @@ -124,17 +132,21 @@ const ClusterDetailPage = () => { ], [clusterName], ); - const subnavItems: Tab[] = React.useMemo( - () => [ + const flags = useFlags(); + const subnavItems: Tab[] = React.useMemo(() => { + const tabs: Tab[] = [ { label: "Overview", href: "..", end: true }, { label: "Replicas", href: "../replicas" }, { label: "Materialized Views", href: "../materialized-views" }, { label: "Indexes", href: "../indexes" }, { label: "Sources", href: "../sources" }, { label: "Sinks", href: "../sinks" }, - ], - [], - ); + ]; + if (flags["visualization-features"]) { + tabs.push({ label: "Dataflows", href: "../dataflows" }); + } + return tabs; + }, [flags]); // Setting key on the route elements prevents any weird jank when you use the context // menu to switch between clusters. @@ -159,6 +171,18 @@ const ClusterDetailPage = () => { } /> } /> } /> + {flags["visualization-features"] && ( + <> + } + /> + } + /> + + )} ); diff --git a/console/src/platform/clusters/DataflowVisualizer.tsx b/console/src/platform/clusters/DataflowVisualizer.tsx deleted file mode 100644 index a228be7e4885b..0000000000000 --- a/console/src/platform/clusters/DataflowVisualizer.tsx +++ /dev/null @@ -1,389 +0,0 @@ -// Copyright Materialize, Inc. and contributors. All rights reserved. -// -// Use of this software is governed by the Business Source License -// included in the LICENSE file. -// -// As of the Change Date specified in that file, in accordance with -// the Business Source License, use of this software will be governed -// by the Apache License, Version 2.0. - -import { - Alert, - AlertIcon, - Box, - Button, - HStack, - Spinner, - Text, - VStack, -} from "@chakra-ui/react"; -import * as d3 from "d3"; -import { graphviz } from "d3-graphviz"; -import React from "react"; -import { useParams } from "react-router-dom"; - -import { Replica } from "~/api/materialize/cluster/clusterList"; -import { ErrorCode } from "~/api/materialize/types"; -import { - Channel, - LirOperator, - Operator, - useDataflowStructure, -} from "~/api/materialize/useDataflowStructure"; -import ErrorBox from "~/components/ErrorBox"; -import LabeledSelect from "~/components/LabeledSelect"; -import { MainContentContainer } from "~/layouts/BaseLayout"; -import { useAllClusters } from "~/store/allClusters"; -import { useAllObjects } from "~/store/allObjects"; -import { assert } from "~/util"; -import { formatBytesShort } from "~/utils/format"; - -interface EnrichedOperator extends Operator { - // First-level. - channelsInScope: Channel[]; - children: EnrichedOperator[]; - transitiveArrangementRecords: bigint | null; - transitiveArrangementSizes: bigint | null; -} - -type EnrichedLirOperator = { - lir_id: string; - operator: string; - addresses: string[]; -}; - -function groupBy(values: T[], group: (item: T) => K): Map { - const output = new Map(); - for (const v of values) { - const k = group(v); - if (!output.has(k)) { - output.set(k, []); - } - output.get(k)!.push(v); - } - return output; -} - -// Returns a map of (stringified) operator address to corresponding operators, -// as well as a designated root. -function collateOperators( - operators: Operator[], - channels: Channel[], - lirOperators: LirOperator[], -): [Map, EnrichedOperator, EnrichedLirOperator[]] { - const scopes = groupBy(operators, (o) => o.parentId); - const channelsByParentScope = groupBy(channels, (ch) => - stringifyAddress(ch.fromOperatorAddress.slice(0, -1)), - ); - - const roots = scopes.get(null) || []; - - function walk( - op: Operator, - m: Map, - ): EnrichedOperator { - assert(!m.has(stringifyAddress(op.address))); - const children = (scopes.get(op.id) || []).map((ch) => walk(ch, m)); - const channelsInScope = - channelsByParentScope.get(stringifyAddress(op.address)) || []; - const ret = { - ...op, - children, - channelsInScope, - transitiveArrangementRecords: - children - .map((child) => child.transitiveArrangementRecords || 0n) - .reduce((a, b) => a + b, 0n) + (op.arrangementRecords || 0n), - transitiveArrangementSizes: - children - .map((child) => child.transitiveArrangementSizes || 0n) - .reduce((a, b) => a + b, 0n) + (op.arrangementSizes || 0n), - }; - m.set(stringifyAddress(ret.address), ret); - return ret; - } - - const enrichedLirOperators: EnrichedLirOperator[] = lirOperators.map( - (lirOp) => ({ - lir_id: lirOp.lir_id, - operator: lirOp.operator, - addresses: lirOp.addresses.map((addr) => stringifyAddress(addr)), - }), - ); - - const m = new Map(); - - const enrichedRoots = roots.map((r) => walk(r, m)); - assert(enrichedRoots.length == 1); - return [m, enrichedRoots[0], enrichedLirOperators]; -} - -const noArrangementRegionColor = "#12b886"; -const noArrangementOperatorColor = "#ffffff"; -const arrangementRegionColor = "#7950f2"; -const arrangementOperatorColor = "#fab005"; - -function stringifyAddress(address: string[]) { - return JSON.stringify(address.map((val) => parseInt(val))); -} - -function scopeToGv( - scope: EnrichedOperator, - lir_operators: EnrichedLirOperator[], -): string { - const chunks = ["digraph {", 'node [style="filled",shape=box];']; - const addresses = new Set(); - for (const op of scope.children) { - const isRegion = op.children.length !== 0; - const hasArrangedData = (op.transitiveArrangementRecords || 0n) > 0n; - let fillColor; - if (isRegion) { - if (hasArrangedData) { - fillColor = arrangementRegionColor; - } else { - fillColor = noArrangementRegionColor; - } - } else { - if (hasArrangedData) { - fillColor = arrangementOperatorColor; - } else { - fillColor = noArrangementOperatorColor; - } - } - const nodeLabelFields = [op.name]; - if (hasArrangedData) { - nodeLabelFields.push( - `${op.transitiveArrangementRecords} arranged records`, - ); - nodeLabelFields.push( - formatBytesShort(BigInt(op.transitiveArrangementSizes || 0)), - ); - } - if (op.elapsedNs > 0) { - nodeLabelFields.push( - `scheduled ${Math.round(op.elapsedNs / 1_000_000_000)}s`, - ); - } - - const tooltip = `Lir ID ${op.lirId}: ${op.lirOperator}`.replace( - /"/g, - '\\"', - ); - - const opAddressString = stringifyAddress(op.address); - addresses.add(opAddressString); - - const nodeGv = `"${opAddressString}" [fillcolor="${fillColor}",tooltip="${tooltip}",id="${opAddressString}",label="${nodeLabelFields.join("\n").replace(/"/g, '\\"')}",class="${isRegion ? "region" : ""}"];`; - chunks.push(nodeGv); - } - const pseudoOperators = new Map(); - for (const ch of scope.channelsInScope) { - let fromAddressKey = stringifyAddress(ch.fromOperatorAddress); - let toAddressKey = stringifyAddress(ch.toOperatorAddress); - - if (ch.fromOperatorAddress[ch.fromOperatorAddress.length - 1] === "0") { - fromAddressKey = `${fromAddressKey}:${ch.fromPort}:FROM`; - pseudoOperators.set(fromAddressKey, `input ${ch.fromPort}`); - } - if (ch.toOperatorAddress[ch.toOperatorAddress.length - 1] === "0") { - toAddressKey = `${toAddressKey}:${ch.toPort}:TO`; - pseudoOperators.set(toAddressKey, `output ${ch.toPort}`); - } - const chanAttr = ch.messagesSent === 0 ? `,style="dashed"` : ""; - const chanLabel = `${ch.messagesSent > 0 ? `${ch.messagesSent} records` : ""}${ch.batchesSent > 0 ? `\n${ch.batchesSent} batches` : ""}`; - const tooltip = ch.channelType || "unknown channel type"; - const chanGv = `"${fromAddressKey}" -> "${toAddressKey}" [label="${chanLabel}",tooltip="${tooltip.replace(/"/g, '\\"')}"${chanAttr}];`; - chunks.push(chanGv); - } - - for (const [k, v] of pseudoOperators) { - chunks.push( - `"${k}" [fillcolor="lightgrey",id="${k}",label="${v.replace(/"/g, '\\"')}"]`, - ); - } - - for (const lir_operator of lir_operators) { - if (!lir_operator.addresses.some((addr) => addresses.has(addr))) continue; - chunks.push("subgraph cluster_" + lir_operator.lir_id + " {"); - chunks.push( - `label="LIR ID ${lir_operator.lir_id}: ${lir_operator.operator.replace(/"/g, '\\"')}";`, - ); - for (const addr of lir_operator.addresses) { - chunks.push(`"${addr}";`); - } - chunks.push("}"); - } - - chunks.push("}"); - const ret = chunks.join("\n"); - return ret; -} - -interface DotVizProps { - dot?: string; - onClickedNode: (id: string) => void; -} - -const DotViz = ({ dot, onClickedNode }: DotVizProps) => { - const d3Container = React.useRef(null); - React.useEffect(() => { - if (d3Container.current && dot) { - const gv = graphviz(d3Container.current) - .scale(0.5) - .attributer(function (d) { - if (d.tag === "svg") { - d.attributes.width = "100%"; - d.attributes.height = "100%"; - } - }); - gv.on("initEnd", () => { - gv.renderDot(dot, function () { - gv.resetZoom(); - - const regions = d3.selectAll(".region"); - regions.on("dblclick", function (event) { - const clickedId = event.currentTarget.getAttribute("id")!; - if (clickedId) { - event.stopPropagation(); - onClickedNode(clickedId); - } - }); - }); - }); - } - }, [dot, onClickedNode]); - return ; -}; - -const defaultReplicas: Replica[] = []; - -const DataflowVisualizer = () => { - const { getClusterById } = useAllClusters(); - const params = useParams(); - const { data: allObjects } = useAllObjects(); - const object = allObjects.find((o) => o.id === params.id); - assert(object && object.clusterId); - const cluster = getClusterById(object.clusterId); - const replicas = cluster?.replicas ?? defaultReplicas; - const [scopeBreadcrumb, setScopeBreadcrumb] = React.useState([]); - const [replicaName, setReplicaName] = React.useState(null); - - React.useEffect(() => { - if (replicas.length > 0 && replicaName === null) { - setReplicaName(replicas[0].name); - } - }, [replicaName, replicas]); - - // Reset scope if props changed. - // TODO - If we track scopes by - // address, rather than by operator ID, we can avoid resetting it - // when replica name changes (addresses other than the initial component - // are the same across replicas, whereas operator IDs aren't - React.useEffect(() => { - setScopeBreadcrumb([]); - }, [params.id, replicaName]); - const dfStructureParams = React.useMemo( - () => - cluster && replicaName - ? { clusterName: cluster.name, replicaName, objectId: object.id } - : undefined, - [cluster, object.id, replicaName], - ); - const { - results: structure, - failedToLoad, - databaseError, - loading, - } = useDataflowStructure(dfStructureParams); - const [allEnriched, root, lirOperators] = React.useMemo(() => { - return structure && structure.operators.length > 0 - ? collateOperators( - structure.operators, - structure.channels, - structure.lir_operators, - ) - : [null, null, []]; - }, [structure]); - const dot = React.useMemo(() => { - const scopeOperator = - scopeBreadcrumb.length > 0 && allEnriched - ? allEnriched.get(scopeBreadcrumb[scopeBreadcrumb.length - 1])! - : root; - return scopeOperator ? scopeToGv(scopeOperator, lirOperators) : undefined; - }, [allEnriched, root, lirOperators, scopeBreadcrumb]); - - const pushScope = React.useCallback( - (s: string) => { - const newBreadcrumb = [...scopeBreadcrumb, s]; - setScopeBreadcrumb(newBreadcrumb); - }, - [scopeBreadcrumb, setScopeBreadcrumb], - ); - - if (!cluster) return null; - - const permissionError = - databaseError && - "code" in databaseError && - databaseError.code === ErrorCode.INSUFFICIENT_PRIVILEGE; - return ( - - {permissionError ? ( - - - - You'll need{" "} - - USAGE - {" "} - privilege on this cluster to visualize this dataflow. - - - ) : failedToLoad ? ( - - ) : loading ? ( - - ) : ( - - {replicaName && ( - setReplicaName(e.target.value)} - flexShrink={0} - > - {cluster.replicas.map((r) => ( - - ))} - - )} - - {" "} - - - {dot === undefined ? ( - This dataflow contains no operators. - ) : ( - - )} - - )} - - ); -}; - -export default DataflowVisualizer; diff --git a/console/src/platform/clusters/IndexList.tsx b/console/src/platform/clusters/IndexList.tsx index 70daf6e7a4354..aa89ca2b7b8ae 100644 --- a/console/src/platform/clusters/IndexList.tsx +++ b/console/src/platform/clusters/IndexList.tsx @@ -31,7 +31,7 @@ import { isSystemCluster, isSystemId, } from "~/api/materialize"; -import { Replica } from "~/api/materialize/cluster/clusterList"; +import { Cluster, Replica } from "~/api/materialize/cluster/clusterList"; import { Index } from "~/api/materialize/cluster/indexesList"; import { AppErrorBoundary } from "~/components/AppErrorBoundary"; import IndexListEmptyState from "~/components/IndexListEmptyState"; @@ -50,8 +50,12 @@ import { useFlags } from "~/hooks/useFlags"; import useLocalStorage from "~/hooks/useLocalStorage"; import { InfoIcon } from "~/icons"; import { MainContentContainer } from "~/layouts/BaseLayout"; -import { useBuildIndexPath } from "~/platform/routeHelpers"; +import { + absoluteClusterPath, + useBuildIndexPath, +} from "~/platform/routeHelpers"; import { useAllClusters } from "~/store/allClusters"; +import { useRegionSlug } from "~/store/environments"; import { MaterializeTheme } from "~/theme"; import { truncateMaxWidth } from "~/theme/components/Table"; import { assert } from "~/util"; @@ -187,6 +191,7 @@ const IndexListInner = ({ replicas={cluster?.replicas ?? []} memoryUsageMap={memoryUsageById} lagMap={lagMap} + cluster={cluster} /> ); }; @@ -196,14 +201,17 @@ interface IndexTableProps { replicas: Replica[]; memoryUsageMap: ArrangmentsMemoryUsageMap | undefined; lagMap?: LagMap; + cluster: Cluster | undefined; } const IndexTable = (props: IndexTableProps) => { const navigate = useNavigate(); const flags = useFlags(); const indexPath = useBuildIndexPath(); + const regionSlug = useRegionSlug(); const dataflowVisualizerEnabled = flags["visualization-features"]; const { colors } = useTheme(); + const { cluster } = props; return ( <> @@ -275,7 +283,7 @@ const IndexTable = (props: IndexTableProps) => { )} {formattedLag} - {dataflowVisualizerEnabled && ( + {dataflowVisualizerEnabled && cluster && ( { { e.stopPropagation(); }} diff --git a/console/src/platform/clusters/MaterializedViewsList.tsx b/console/src/platform/clusters/MaterializedViewsList.tsx index 4f5697fcc7d85..7afbb03a8a07c 100644 --- a/console/src/platform/clusters/MaterializedViewsList.tsx +++ b/console/src/platform/clusters/MaterializedViewsList.tsx @@ -25,7 +25,7 @@ import React from "react"; import { Link, useNavigate, useParams } from "react-router-dom"; import { createNamespace } from "~/api/materialize"; -import { Replica } from "~/api/materialize/cluster/clusterList"; +import { Cluster, Replica } from "~/api/materialize/cluster/clusterList"; import { MaterializedView, MaterializedViewsResponse, @@ -51,8 +51,12 @@ import { SampleCodeBoxWrapper, } from "~/layouts/listPageComponents"; import docUrls from "~/mz-doc-urls.json"; -import { useBuildMaterializedViewPath } from "~/platform/routeHelpers"; +import { + absoluteClusterPath, + useBuildMaterializedViewPath, +} from "~/platform/routeHelpers"; import { useAllClusters } from "~/store/allClusters"; +import { useRegionSlug } from "~/store/environments"; import { MaterializeTheme } from "~/theme"; import { truncateMaxWidth } from "~/theme/components/Table"; import { assert } from "~/util"; @@ -205,6 +209,7 @@ const MaterializedViewListInner = ({ replicas={cluster?.replicas ?? []} memoryUsageMap={memoryUsageById} lagMap={lagMap} + cluster={cluster} /> @@ -216,13 +221,16 @@ interface MaterializedViewTableProps { replicas: Replica[]; memoryUsageMap: ArrangmentsMemoryUsageMap | undefined; lagMap?: LagMap; + cluster: Cluster | undefined; } const MaterializedViewTable = (props: MaterializedViewTableProps) => { const navigate = useNavigate(); const flags = useFlags(); const materializedViewPath = useBuildMaterializedViewPath(); + const regionSlug = useRegionSlug(); const dataflowVisualizerEnabled = flags["visualization-features"]; + const { cluster } = props; const { colors } = useTheme(); @@ -279,7 +287,7 @@ const MaterializedViewTable = (props: MaterializedViewTableProps) => { )} {formattedLag} - {dataflowVisualizerEnabled && ( + {dataflowVisualizerEnabled && cluster && ( { { e.stopPropagation(); }} diff --git a/console/src/platform/dataflows/ChannelEdge.test.tsx b/console/src/platform/dataflows/ChannelEdge.test.tsx new file mode 100644 index 0000000000000..e2d6779add733 --- /dev/null +++ b/console/src/platform/dataflows/ChannelEdge.test.tsx @@ -0,0 +1,144 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { ThemeProvider } from "@chakra-ui/react"; +import { fireEvent, render, screen } from "@testing-library/react"; +import { Position, ReactFlowProvider, useStoreApi } from "@xyflow/react"; +import React from "react"; + +import { lightTheme } from "~/theme"; + +import { ChannelEdge, type ChannelEdgeData } from "./ChannelEdge"; +import type { PortPeer } from "./dataflowGraph"; +import { HIGHLIGHT_COLORS } from "./nodeStyle"; + +const BASE_POSITION = { + id: "e1", + source: "a", + target: "b", + sourceX: 0, + sourceY: 0, + targetX: 100, + targetY: 100, + sourcePosition: Position.Bottom, + targetPosition: Position.Top, +}; + +// EdgeLabelRenderer portals into `.react-flow__edgelabel-renderer` inside the +// store's `domNode`, which the real wrapper sets on mount; a bare +// under ReactFlowProvider never gets one, so the portal (and anything +// inside it, like the label or jump buttons) silently renders nothing. This +// stands in for that wrapper, wiring domNode to a real container so the +// portal has somewhere to render into. +const DomNodeSetter = ({ children }: { children: React.ReactNode }) => { + const store = useStoreApi(); + const ref = React.useRef(null); + React.useLayoutEffect(() => { + store.setState({ domNode: ref.current }); + }, [store]); + return ( +
+
+ {children} +
+ ); +}; + +function renderEdge(data: ChannelEdgeData) { + const { container } = render( + + + + + + + + + , + ); + return container.querySelector("path")!; +} + +const DATA: ChannelEdgeData = { + messagesSent: 1n, + batchesSent: 1n, + channelTypes: [], + dimmed: false, + selected: false, + connected: false, + sourceLandings: [], + targetLandings: [], +}; + +const LANDING: PortPeer = { + address: [1, 2], + label: "Inner", + messagesSent: 1n, + batchesSent: 1n, + channelTypes: [], + peerPortId: null, +}; + +describe("ChannelEdge", () => { + it("plain edge has no highlight stroke", () => { + const path = renderEdge(DATA); + expect(path.style.stroke).toBe(""); + }); + + // Touching the selected node (but not being the clicked thing itself) + // gets a lighter highlight than the directly selected edge does. + it("a connected edge gets the lighter highlight", () => { + const path = renderEdge({ ...DATA, connected: true }); + expect(path.style.stroke).toBe(HIGHLIGHT_COLORS.connected); + expect(path.style.strokeWidth).toBe("1.75"); + }); + + it("the directly selected edge gets the stronger highlight, not the connected one", () => { + const path = renderEdge({ ...DATA, selected: true, connected: true }); + expect(path.style.stroke).toBe(HIGHLIGHT_COLORS.selected); + expect(path.style.strokeWidth).toBe("2.5"); + }); + + it("shows no jump buttons when nothing is selected, even with a landing", () => { + renderEdge({ ...DATA, sourceLandings: [LANDING] }); + expect(screen.queryByRole("button")).toBeNull(); + }); + + it("shows no jump buttons for an unambiguous edge with nothing to drill into", () => { + renderEdge({ ...DATA, selected: true }); + expect(screen.queryByRole("button")).toBeNull(); + }); + + it("selecting an edge with exactly one landing per side shows a jump button per side", () => { + const onJumpTo = vi.fn(); + const targetLanding: PortPeer = { ...LANDING, label: "Other" }; + renderEdge({ + ...DATA, + selected: true, + sourceLandings: [LANDING], + targetLandings: [targetLanding], + onJumpTo, + }); + fireEvent.click(screen.getByRole("button", { name: /Inner/ })); + expect(onJumpTo).toHaveBeenCalledWith(LANDING); + fireEvent.click(screen.getByRole("button", { name: /Other/ })); + expect(onJumpTo).toHaveBeenCalledWith(targetLanding); + }); + + it("skips the jump button on a side with more than one landing (ambiguous)", () => { + const onJumpTo = vi.fn(); + renderEdge({ + ...DATA, + selected: true, + sourceLandings: [LANDING, { ...LANDING, label: "Other" }], + onJumpTo, + }); + expect(screen.queryByRole("button")).toBeNull(); + }); +}); diff --git a/console/src/platform/dataflows/ChannelEdge.tsx b/console/src/platform/dataflows/ChannelEdge.tsx new file mode 100644 index 0000000000000..bd6e952882ab0 --- /dev/null +++ b/console/src/platform/dataflows/ChannelEdge.tsx @@ -0,0 +1,176 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { Button, HStack, Text, Tooltip, useTheme } from "@chakra-ui/react"; +import { + BaseEdge, + EdgeLabelRenderer, + type EdgeProps, + getBezierPath, +} from "@xyflow/react"; +import React from "react"; + +import { MaterializeTheme } from "~/theme"; + +import type { PortPeer } from "./dataflowGraph"; +import { HIGHLIGHT_COLORS, prettyPrintChannelType } from "./nodeStyle"; + +export type ChannelEdgeData = { + messagesSent: bigint; + batchesSent: bigint; + channelTypes: string[]; + dimmed: boolean; + selected: boolean; + // Touches the selected node without being the clicked thing itself (an + // edge whose own id is selected gets `selected` instead), so it reads as + // a lighter, secondary emphasis. + connected: boolean; + // Where this edge's real channel(s) land inside a collapsed source/target + // region, mirroring VisibleEdge's fields of the same name. Only ever + // shown as an inline canvas button when there's exactly one: with several + // (a merge fanned out to multiple real landings) a single click can't say + // which one, so the picker stays in the detail panel's landing list. + sourceLandings: PortPeer[]; + targetLandings: PortPeer[]; + onJumpTo?: (peer: PortPeer) => void; +}; + +const compactCount = Intl.NumberFormat("default", { + notation: "compact", + maximumFractionDigits: 1, +}); + +const compact = (n: bigint) => compactCount.format(n); + +export const ChannelEdge = (props: EdgeProps & { data: ChannelEdgeData }) => { + const { colors } = useTheme(); + const [path, labelX, labelY] = getBezierPath(props); + const { + messagesSent, + batchesSent, + channelTypes, + dimmed, + selected, + connected, + sourceLandings, + targetLandings, + onJumpTo, + } = props.data; + const idle = messagesSent === 0n; + // Inline label stays terse: compact record/batch counts only. Exact figures + // and the full container type names live in the tooltip, since those Rust + // type strings are long enough to swamp the canvas. Selecting an idle edge + // still shows its (zero) counts, confirming the click landed. + const label = + idle && !selected + ? "" + : `${compact(messagesSent)} / ${compact(batchesSent)}`; + const prettyTypes = channelTypes.map(prettyPrintChannelType); + const tooltip = idle + ? prettyTypes.join(", ") || "unknown channel type" + : `${messagesSent} records / ${batchesSent} batches` + + (prettyTypes.length > 0 ? ` · ${prettyTypes.join(", ")}` : ""); + // Only an unambiguous (single-landing) side gets a one-click button here; + // see the sourceLandings/targetLandings doc comment for why a fan-out + // doesn't. + const sourceLanding = + selected && sourceLandings.length === 1 ? sourceLandings[0] : null; + const targetLanding = + selected && targetLandings.length === 1 ? targetLandings[0] : null; + return ( + <> + + {label && ( + + + + {label} + + + + )} + {onJumpTo && (sourceLanding || targetLanding) && ( + + + {sourceLanding && ( + + + + )} + {targetLanding && ( + + + + )} + + + )} + + ); +}; diff --git a/console/src/platform/dataflows/DataflowBreadcrumbs.tsx b/console/src/platform/dataflows/DataflowBreadcrumbs.tsx new file mode 100644 index 0000000000000..3d658d59e8a5b --- /dev/null +++ b/console/src/platform/dataflows/DataflowBreadcrumbs.tsx @@ -0,0 +1,77 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { HStack, Text, useTheme } from "@chakra-ui/react"; +import React from "react"; + +import TextLink from "~/components/TextLink"; +import { ChevronRightIcon } from "~/icons"; +import { MaterializeTheme } from "~/theme"; + +import { + type DataflowNode, + type DataflowStructure, + type NodeId, +} from "./dataflowGraph"; + +export interface DataflowBreadcrumbsProps { + structure: DataflowStructure; + focusedScope: NodeId; + onNavigate: (scope: NodeId) => void; +} + +export const DataflowBreadcrumbs = ({ + structure, + focusedScope, + onNavigate, +}: DataflowBreadcrumbsProps) => { + const { colors } = useTheme(); + const path: DataflowNode[] = []; + for (let id: NodeId | null = focusedScope; id; ) { + const node = structure.nodes.get(id); + if (!node) { + throw new Error(`missing dataflow node for id ${id} in breadcrumb path`); + } + path.unshift(node); + id = node.parent; + } + return ( + + {path.map((node, i) => + i === path.length - 1 ? ( + + {node.name} + + ) : ( + + onNavigate(node.id)} + > + {node.name} + + + + ), + )} + + ); +}; diff --git a/console/src/platform/dataflows/DataflowDetailPage.test.tsx b/console/src/platform/dataflows/DataflowDetailPage.test.tsx new file mode 100644 index 0000000000000..e3979499ca281 --- /dev/null +++ b/console/src/platform/dataflows/DataflowDetailPage.test.tsx @@ -0,0 +1,666 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { screen, within } from "@testing-library/react"; +import userEvent from "@testing-library/user-event"; +import { http, HttpResponse } from "msw"; +import React from "react"; +import { Route, Routes, useLocation } from "react-router-dom"; + +import type { Cluster } from "~/api/materialize/cluster/clusterList"; +import { ErrorCode } from "~/api/materialize/types"; +import server from "~/api/mocks/server"; +import { getStore } from "~/jotai"; +import { allClusters } from "~/store/allClusters"; +import { mockSubscribeState } from "~/test/mockSubscribe"; +import { renderComponent } from "~/test/utils"; + +import DataflowDetailPage from "./DataflowDetailPage"; +import { nodeIdOf } from "./dataflowGraph"; + +// Declared via vi.hoisted so the vi.mock factory below (itself hoisted +// above regular imports) can close over it. +const fitViewSpy = vi.hoisted(() => vi.fn()); + +// jsdom lacks ResizeObserver/DOMMatrixReadOnly, so replace @xyflow/react with a +// flat renderer that preserves the decision under test (whether the canvas and +// its nodes exist). This mirrors src/test/mockReactFlow.tsx but also stubs +// useReactFlow, which DataflowDetailPage's centering helper reads and the +// shared helper does not provide. Click/double-click are wired through so +// selection and drill-down navigation can be driven from tests too. +vi.mock("@xyflow/react", () => ({ + ReactFlow: ({ + nodes, + children, + onNodeClick, + onNodeDoubleClick, + }: { + nodes: { + id: string; + type?: string; + data: { node?: { label: string }; label?: string }; + }[]; + children?: React.ReactNode; + onNodeClick?: (e: unknown, node: unknown) => void; + onNodeDoubleClick?: (e: unknown, node: unknown) => void; + }) => ( +
+ {nodes.map((n) => ( +
onNodeClick?.(null, n)} + onDoubleClick={() => onNodeDoubleClick?.(null, n)} + > + {n.data.node ? n.data.node.label : n.data.label} +
+ ))} + {children} +
+ ), + Background: () => null, + Controls: () => null, + MiniMap: () => null, + Handle: () => null, + Position: { Left: "left", Right: "right", Top: "top", Bottom: "bottom" }, + BaseEdge: () => null, + EdgeLabelRenderer: ({ children }: { children: React.ReactNode }) => ( + <>{children} + ), + getBezierPath: () => ["", 0, 0], + useReactFlow: () => ({ + getInternalNode: () => undefined, + setCenter: () => {}, + fitView: fitViewSpy, + getViewport: () => ({ x: 0, y: 0, zoom: 1 }), + }), +})); + +// elkjs runs in a web worker that jsdom cannot load, so replace the layout hook +// with a synchronous stub. The Proxy resolves any node id to a fixed box, so +// every visible node lands on the mocked React Flow canvas. +vi.mock("./useElkLayout", () => ({ + useElkLayout: () => ({ + positions: new Proxy( + {}, + { get: () => ({ x: 0, y: 0, width: 100, height: 40 }) }, + ), + layouting: false, + error: null, + }), +})); + +const FAILING_REPLICA = "r2"; + +const cluster = { + id: "u5", + name: "test_cluster", + replicas: [{ name: "r1" }, { name: "r2" }], +} as unknown as Cluster; + +const okResult = { + desc: { columns: [] }, + rows: [], +}; +// Two operators (a root and one child) so buildDataflowStructure yields a +// graph with more than one node and DataflowDetailPage renders the canvas. +const operatorsResult = { + desc: { + columns: [ + { name: "id" }, + { name: "address" }, + { name: "name" }, + { name: "arrangementRecords" }, + { name: "arrangementSize" }, + { name: "elapsedNs" }, + ], + }, + rows: [ + ["10", ["7"], "Dataflow", "0", "0", "0"], + ["11", ["7", "1"], "Map", "0", "0", "1"], + ], +}; +// The page also renders a dataflow switcher (useDataflowList), a separate +// request from the graph data's three queries above. +const dataflowListResult = { + desc: { + columns: [ + { name: "id" }, + { name: "name" }, + { name: "records" }, + { name: "size" }, + { name: "elapsedNs" }, + ], + }, + rows: [["7", "Dataflow: mv", "0", "0", "0"]], +}; + +// Dataflow 8: root [8], sibling regions RegionA [8,1] (leaf LeafA [8,1,1]) and +// RegionB [8,2] (leaf LeafB [8,2,1]), with RegionA's output feeding RegionB's +// input directly (a plain sibling-to-sibling crossing, not nested). +const portJumpOperatorsResult = { + ...operatorsResult, + rows: [ + ["20", ["8"], "Dataflow", "0", "0", "0"], + ["21", ["8", "1"], "RegionA", "0", "0", "0"], + ["22", ["8", "1", "1"], "LeafA", "0", "0", "1"], + ["23", ["8", "2"], "RegionB", "0", "0", "0"], + ["24", ["8", "2", "1"], "LeafB", "0", "0", "1"], + ], +}; +const portJumpChannelsResult = { + desc: { + columns: [ + { name: "id" }, + { name: "fromOperatorAddress" }, + { name: "fromPort" }, + { name: "toOperatorAddress" }, + { name: "toPort" }, + { name: "messagesSent" }, + { name: "batchesSent" }, + { name: "channelType" }, + ], + }, + rows: [["1", ["8", "1"], "0", ["8", "2"], "0", "5", "2", "rows"]], +}; + +// Dataflow 9: RegionA [9,1]'s single output port (0) fans out to two +// siblings, RegionB [9,2] and RegionC [9,3] (1:n, not 1:1). +const fanOutOperatorsResult = { + ...operatorsResult, + rows: [ + ["30", ["9"], "Dataflow", "0", "0", "0"], + ["31", ["9", "1"], "RegionA", "0", "0", "0"], + ["32", ["9", "1", "1"], "LeafA", "0", "0", "1"], + ["33", ["9", "2"], "RegionB", "0", "0", "0"], + ["34", ["9", "2", "1"], "LeafB", "0", "0", "1"], + ["35", ["9", "3"], "RegionC", "0", "0", "0"], + ["36", ["9", "3", "1"], "LeafC", "0", "0", "1"], + ], +}; +const fanOutChannelsResult = { + ...portJumpChannelsResult, + rows: [ + ["1", ["9", "1"], "0", ["9", "2"], "0", "5", "2", "rows"], + ["2", ["9", "1"], "0", ["9", "3"], "0", "3", "1", "rows"], + ], +}; + +// Dataflow 40: root [40], child [40,1] "Join" (operator id 41), covered by +// one LIR span, to exercise the "Show LIR groups" toggle in isolation from +// every other test's dataflow 7 fixture, which has no LIR spans at all. +const lirOperatorsResult = { + ...operatorsResult, + rows: [ + ["40", ["40"], "Dataflow", "0", "0", "0"], + ["41", ["40", "1"], "Join", "0", "0", "0"], + ], +}; +const lirSpansResult = { + desc: { + columns: [ + { name: "exportId" }, + { name: "lirId" }, + { name: "parentLirId" }, + { name: "nesting" }, + { name: "operator" }, + { name: "operatorIdStart" }, + { name: "operatorIdEnd" }, + ], + }, + rows: [["u7", "1", null, "0", "Join::Differential", "41", "42"]], +}; + +beforeEach(() => { + fitViewSpy.mockClear(); + const store = getStore(); + store.set(allClusters, mockSubscribeState({ data: [cluster] })); + server.use( + http.post("*/api/sql", async ({ request }) => { + const options = JSON.parse( + new URL(request.url).searchParams.get("options") ?? "{}", + ); + if (options.cluster_replica === FAILING_REPLICA) { + return HttpResponse.json({ + results: [ + { + error: { + message: "no such replica", + code: ErrorCode.INTERNAL_ERROR, + }, + }, + ], + }); + } + const body = await request.text(); + // useDataflowList's query targets mz_dataflows directly; none of + // useDataflowGraphData's three queries reference that exact table name. + if (body.includes("mz_dataflows")) { + return HttpResponse.json({ results: [dataflowListResult] }); + } + // The dataflow id is interpolated as a SQL literal, so it shows up + // verbatim in the compiled query text. + if (body.includes("'8'")) { + return HttpResponse.json({ + results: [ + portJumpOperatorsResult, + portJumpChannelsResult, + okResult, + okResult, + okResult, + ], + }); + } + if (body.includes("'9'")) { + return HttpResponse.json({ + results: [ + fanOutOperatorsResult, + fanOutChannelsResult, + okResult, + okResult, + okResult, + ], + }); + } + if (body.includes("'40'")) { + return HttpResponse.json({ + results: [ + lirOperatorsResult, + okResult, + lirSpansResult, + okResult, + okResult, + ], + }); + } + return HttpResponse.json({ + results: [operatorsResult, okResult, okResult, okResult, okResult], + }); + }), + ); +}); + +// Exposes the current URL search string so a test can assert the +// drill-down scope actually landed in the URL, not just in the rendered +// graph (which would still look right if it were only in component state). +const LocationProbe = () => ( +
{useLocation().search}
+); + +describe("DataflowDetailPage", () => { + // CNS-109: switching to a replica whose query fails must surface the error + // state and drop the graph from the previously successful replica. + it("shows the error state and hides the stale graph when the new replica fails", async () => { + await renderComponent( + + } + /> + , + { + initialRouterEntries: ["/clusters/u5/test_cluster/dataflows/7"], + }, + ); + + // r1 succeeds, so the graph renders first. + expect(await screen.findByTestId("react-flow")).toBeVisible(); + + // Switch to r2, whose query fails. The toolbar renders its own selects, so + // pick the replica select by its options rather than the ambiguous role. + const replicaSelect = screen + .getAllByRole("combobox") + .find((select) => + Array.from((select as HTMLSelectElement).options).some( + (option) => option.value === FAILING_REPLICA, + ), + ); + expect(replicaSelect).toBeDefined(); + await userEvent.selectOptions(replicaSelect!, FAILING_REPLICA); + + expect( + await screen.findByText("There was an error visualizing your dataflow"), + ).toBeVisible(); + // The stale graph from r1 must be gone. + expect(screen.queryByTestId("react-flow")).toBeNull(); + }); + + // A port's "Jump" button should work the same regardless of which + // direction it represents: this drills into RegionA, whose only channel + // is its own output feeding sibling RegionB's input, and checks that + // jumping from the resulting "out" port's peer actually navigates. + it("jumps to an output port's peer, same as an input port's peer", async () => { + await renderComponent( + + } + /> + , + { + initialRouterEntries: ["/clusters/u5/test_cluster/dataflows/8"], + }, + ); + + const regionAId = nodeIdOf([8, 1]); + const regionBId = nodeIdOf([8, 2]); + expect(await screen.findByTestId(`node-${regionAId}`)).toHaveTextContent( + "RegionA", + ); + + // Drill into RegionA: its only channel is its own output (port 0) + // reaching sibling RegionB, which isn't part of this view, so it + // should surface as a dangling "out" port. + await userEvent.dblClick(screen.getByTestId(`node-${regionAId}`)); + const outPortId = `${regionAId}:out:0`; + const outPortNode = await screen.findByTestId(`node-${outPortId}`); + + // Select the port to open its detail panel, then jump its peer. + await userEvent.click(outPortNode); + expect(await screen.findByText(/RegionB/)).toBeVisible(); + const jumpButton = await screen.findByRole("button", { name: "Jump" }); + await userEvent.click(jumpButton); + + // RegionB is itself a region, so jumping drills straight into it and + // selects the matching "in" port there (port 0, matching the channel's + // toPort), rather than just landing outside it as an unlabeled box. + const regionBInPortId = `${regionBId}:in:0`; + expect(await screen.findByTestId(`node-${regionBInPortId}`)).toBeVisible(); + // Its detail panel is open, selected: a second "input 0" text node (the + // panel title) alongside the port box's own label. + expect(await screen.findAllByText("input 0")).toHaveLength(2); + }); + + // A 1:1 port doesn't need the detail panel detour: double-clicking it + // jumps directly, the same destination as clicking through to its Jump + // button. + it("double-clicking a port with exactly one peer jumps directly", async () => { + await renderComponent( + + } + /> + , + { + initialRouterEntries: ["/clusters/u5/test_cluster/dataflows/8"], + }, + ); + + const regionAId = nodeIdOf([8, 1]); + const regionBId = nodeIdOf([8, 2]); + await screen.findByTestId(`node-${regionAId}`); + await userEvent.dblClick(screen.getByTestId(`node-${regionAId}`)); + + const outPortId = `${regionAId}:out:0`; + const outPortNode = await screen.findByTestId(`node-${outPortId}`); + await userEvent.dblClick(outPortNode); + + const regionBInPortId = `${regionBId}:in:0`; + expect(await screen.findByTestId(`node-${regionBInPortId}`)).toBeVisible(); + expect(await screen.findAllByText("input 0")).toHaveLength(2); + }); + + // The drill-down scope has to actually be in the URL, not just reflected + // in the rendered graph, for a copied link or the back button to work. + it("puts the drill-down scope in the URL, so a link into it reopens there directly", async () => { + const { unmount } = await renderComponent( + + + + + + } + /> + , + { + initialRouterEntries: ["/clusters/u5/test_cluster/dataflows/8"], + }, + ); + + const regionAId = nodeIdOf([8, 1]); + await screen.findByTestId(`node-${regionAId}`); + // A fresh, undrilled link carries no scope param at all. + expect(screen.getByTestId("location-search").textContent).toBe(""); + + await userEvent.dblClick(screen.getByTestId(`node-${regionAId}`)); + await screen.findByTestId(`node-${nodeIdOf([8, 1, 1])}`); + const search = screen.getByTestId("location-search").textContent; + expect(search).toBe("?scope=8.1"); + + unmount(); + + // Reopening at that exact URL, with no clicks, lands drilled in already. + await renderComponent( + + } + /> + , + { + initialRouterEntries: [ + `/clusters/u5/test_cluster/dataflows/8${search}`, + ], + }, + ); + expect( + await screen.findByTestId(`node-${nodeIdOf([8, 1, 1])}`), + ).toBeVisible(); + }); + + // Selection lives in the URL too, the same as scope: opening a link + // straight at `select=...` (indistinguishable, from react-router's + // perspective, from landing there via the back/forward buttons) must + // restore the panel without any click, and fit the viewport to it since + // nothing guarantees it's already onscreen. + it("restores selection from a direct link and fits the viewport to it", async () => { + await renderComponent( + + } + /> + , + { + initialRouterEntries: [ + `/clusters/u5/test_cluster/dataflows/8?scope=8.1&select=${encodeURIComponent(`node:${nodeIdOf([8, 1, 1])}`)}`, + ], + }, + ); + + // The panel title and the node's own label both render "LeafA". + expect(await screen.findAllByText("LeafA")).toHaveLength(2); + // Present only in the detail panel's own rendering, confirming the + // panel (not just the node) is open. + expect(screen.getByText("Kind")).toBeInTheDocument(); + expect(fitViewSpy).toHaveBeenCalledTimes(1); + }); + + // elk lays each scope out from scratch, so a viewport left over from + // wherever the user was in the previous scope can easily miss the new + // one entirely. Navigating into a scope should recenter for that reason, + // but an unrelated re-render within the same scope (e.g. selecting a + // node) must not refit again, or it would fight the user's own panning. + it("recenters after navigating into a scope, but not again within it", async () => { + await renderComponent( + + } + /> + , + { + initialRouterEntries: ["/clusters/u5/test_cluster/dataflows/8"], + }, + ); + + const regionAId = nodeIdOf([8, 1]); + await screen.findByTestId(`node-${regionAId}`); + // The initial scope is skipped: React Flow's own `fitView` prop already + // covers first load. + expect(fitViewSpy).not.toHaveBeenCalled(); + + await userEvent.dblClick(screen.getByTestId(`node-${regionAId}`)); + const leafAId = nodeIdOf([8, 1, 1]); + await screen.findByTestId(`node-${leafAId}`); + expect(fitViewSpy).toHaveBeenCalledTimes(1); + + await userEvent.click(screen.getByTestId(`node-${leafAId}`)); + expect(fitViewSpy).toHaveBeenCalledTimes(1); + }); + + // A 1:n port (RegionA's single output feeding both RegionB and RegionC) + // can't pick a jump target on its own; double-clicking it should behave + // like a plain click (open its own detail panel) rather than guess. + it("double-clicking a fanned-out port selects it instead of guessing a peer", async () => { + await renderComponent( + + } + /> + , + { + initialRouterEntries: ["/clusters/u5/test_cluster/dataflows/9"], + }, + ); + + const regionAId = nodeIdOf([9, 1]); + await screen.findByTestId(`node-${regionAId}`); + await userEvent.dblClick(screen.getByTestId(`node-${regionAId}`)); + + const outPortId = `${regionAId}:out:0`; + const outPortNode = await screen.findByTestId(`node-${outPortId}`); + await userEvent.dblClick(outPortNode); + + // Still inside RegionA: neither RegionB nor RegionC drilled into. + expect(screen.queryByTestId(`node-${nodeIdOf([9, 2])}:in:0`)).toBeNull(); + expect(screen.queryByTestId(`node-${nodeIdOf([9, 3])}:in:0`)).toBeNull(); + // The port itself is selected, showing both peers to choose from. + expect(await screen.findAllByText("output 0")).toHaveLength(2); + expect(screen.getByText(/RegionB/)).toBeVisible(); + expect(screen.getByText(/RegionC/)).toBeVisible(); + }); + + // A scope drilled into on one dataflow is a node id that (almost + // certainly) doesn't exist in a different dataflow's structure. Switching + // must fall back to the new structure's root instead of crashing on a + // `nodes.get(focusedScope)!` lookup for an id the new data doesn't have. + it("falls back to the new root when switching dataflows while drilled into a scope", async () => { + await renderComponent( + + } + /> + {/* The dataflow switcher navigates through absoluteClusterPath, + which prefixes the region slug; the initial entry below doesn't + (matching every other test in this file), so both shapes need a + matching route. */} + } + /> + , + { + initialRouterEntries: ["/clusters/u5/test_cluster/dataflows/8"], + }, + ); + + const regionAId = nodeIdOf([8, 1]); + await screen.findByTestId(`node-${regionAId}`); + await userEvent.dblClick(screen.getByTestId(`node-${regionAId}`)); + // Drilled in: RegionA's own id no longer names anything in dataflow 7. + await screen.findByTestId(`node-${regionAId}:out:0`); + + const dataflowSelect = screen + .getAllByRole("combobox") + .find((select) => + Array.from((select as HTMLSelectElement).options).some( + (option) => option.value === "7", + ), + ); + expect(dataflowSelect).toBeDefined(); + await userEvent.selectOptions(dataflowSelect!, "7"); + + // No crash, and the new dataflow's own root renders. + expect( + await screen.findByTestId(`node-${nodeIdOf([7, 1])}`), + ).toHaveTextContent("Map"); + }); + + it("shows a LIR group box by default, and hides it when the toggle is turned off", async () => { + await renderComponent( + + } + /> + , + { + initialRouterEntries: ["/clusters/u5/test_cluster/dataflows/40"], + }, + ); + + expect( + await screen.findByTestId(`node-${nodeIdOf([40, 1])}`), + ).toBeInTheDocument(); + // On by default. + const groupNode = screen.getByTestId("node-u7/1"); + expect(groupNode).toHaveTextContent("Join::Differential"); + + // Regression check for the onNodeDoubleClick guard added in Step 4: + // double-clicking a group must not throw (it read `.data.node.kind` + // unconditionally before the guard, which crashes on a group node) and + // must not navigate anywhere, since groups aren't scopes. + await userEvent.dblClick(groupNode); + expect(screen.getByTestId("node-u7/1")).toBeInTheDocument(); + + await userEvent.click(screen.getByLabelText("Show LIR groups")); + expect( + within(screen.getByTestId("react-flow")).queryByText( + /Join::Differential/, + ), + ).not.toBeInTheDocument(); + }); + + it("opens the detail panel with LIR summary info when a group header is clicked", async () => { + await renderComponent( + + } + /> + , + { + initialRouterEntries: ["/clusters/u5/test_cluster/dataflows/40"], + }, + ); + + await screen.findByTestId(`node-${nodeIdOf([40, 1])}`); + await userEvent.click(screen.getByTestId("node-u7/1")); + + // The panel title and the LIR sidebar's own row both render "LIR 1: + // Join::Differential", so assert at least one match rather than a + // single getByText, which throws on more than one hit. + expect( + screen.getAllByText("LIR 1: Join::Differential").length, + ).toBeGreaterThan(0); + // Rendered the same Row-shaped way as a regular node's details, not a + // visually distinct card. + expect(screen.getByText("Export")).toBeInTheDocument(); + expect(screen.getByText("Members")).toBeInTheDocument(); + expect(screen.getByText("Records")).toBeInTheDocument(); + expect(screen.getByText("Memory")).toBeInTheDocument(); + }); +}); diff --git a/console/src/platform/dataflows/DataflowDetailPage.tsx b/console/src/platform/dataflows/DataflowDetailPage.tsx new file mode 100644 index 0000000000000..656f2d1c5474f --- /dev/null +++ b/console/src/platform/dataflows/DataflowDetailPage.tsx @@ -0,0 +1,701 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { Box, Button, HStack, Spinner, Text, VStack } from "@chakra-ui/react"; +import { interpolateYlOrRd } from "d3-scale-chromatic"; +import React from "react"; +import { + Link, + useNavigate, + useNavigationType, + useParams, + useSearchParams, +} from "react-router-dom"; + +import { useDataflowGraphData } from "~/api/materialize/dataflow/useDataflowGraphData"; +import { useDataflowList } from "~/api/materialize/dataflow/useDataflowList"; +import { isInsufficientPrivilegeError } from "~/api/materialize/executeSql"; +import ErrorBox from "~/components/ErrorBox"; +import LabeledSelect from "~/components/LabeledSelect"; +import { MainContentContainer } from "~/layouts/BaseLayout"; +import { absoluteClusterPath } from "~/platform/routeHelpers"; +import { useAllClusters } from "~/store/allClusters"; +import { useRegionSlug } from "~/store/environments"; + +import { DataflowBreadcrumbs } from "./DataflowBreadcrumbs"; +import { + type Address, + allSearchMatches, + commonAncestorScope, + decorateGraph, + DEFAULT_FILTERS, + deriveVisibleGraph, + type Filters, + lirIndex, + lirSummary, + mustGet, + type NodeId, + nodeIdOf, + type PortPeer, + representativeInView, + subtreeSearchMatches, +} from "./dataflowGraph"; +import { DataflowGraphView } from "./DataflowGraphView"; +import { DataflowToolbar } from "./DataflowToolbar"; +import { LirPanel } from "./LirPanel"; +import { NodeDetailPanel, type Selection } from "./NodeDetailPanel"; +import { hashString } from "./nodeStyle"; +import { UsagePrivilegeAlert } from "./UsagePrivilegeAlert"; + +// Injected into decorateGraph so the pure graph module stays d3-free. +const heatColor = (t: number) => interpolateYlOrRd(0.15 + 0.85 * t); + +const DataflowDetailPage = () => { + const { clusterId, dataflowId } = useParams(); + const navigate = useNavigate(); + const regionSlug = useRegionSlug(); + const { getClusterById } = useAllClusters(); + const cluster = clusterId ? getClusterById(clusterId) : undefined; + const [searchParams, setSearchParams] = useSearchParams(); + const replicaName = searchParams.get("replica") ?? cluster?.replicas[0]?.name; + + const params = React.useMemo( + () => + cluster && replicaName && dataflowId + ? { clusterName: cluster.name, replicaName, dataflowId } + : undefined, + [cluster, replicaName, dataflowId], + ); + const { data, error, databaseError, loading, refetch } = + useDataflowGraphData(params); + + // Lets the toolbar switch dataflows in place instead of forcing a trip + // back to the list page. + const listParams = React.useMemo( + () => + cluster && replicaName + ? { clusterName: cluster.name, replicaName } + : undefined, + [cluster, replicaName], + ); + const { data: dataflowList } = useDataflowList(listParams); + + // The drill-down scope lives in the URL (as the dataflow address it + // resolves to, e.g. "5.1.2"), not component state: back/forward then walk + // the drill-down history, and a copied link reopens at the same scope. + // Read fresh every render rather than synchronized via effect, so a scope + // that doesn't resolve in the current structure (a stale link, a dataflow + // whose shape changed, or simply no scope param) falls back to the root + // rather than crashing every direct `nodes.get(focusedScope)!` lookup + // downstream. + const scopeParam = searchParams.get("scope"); + const rawFocusedScope = scopeParam + ? nodeIdOf(scopeParam.split(".").map(Number)) + : null; + const selectParam = searchParams.get("select"); + const focusedScope = data + ? rawFocusedScope && data.structure.nodes.has(rawFocusedScope) + ? rawFocusedScope + : data.structure.root + : null; + // Every URL-param write funnels through here: calling setSearchParams + // more than once in the same synchronous handler is unsafe (the second + // call's `prev` doesn't see the first call's pending change, so it wins + // and silently discards the first), so anything that needs to touch more + // than one param at once (see focusOn, onSelectLir below) must do it in a + // single mutate callback rather than composing two setters. + const updateSearchParams = React.useCallback( + (mutate: (urlParams: URLSearchParams) => void) => { + setSearchParams((prev) => { + const next = new URLSearchParams(prev); + mutate(next); + return next; + }); + }, + [setSearchParams], + ); + // The root has no scope of its own (it's exactly a link's default), so + // leave the URL as the plain dataflow link instead of carrying a + // redundant, always-implied param. + const applyScope = (urlParams: URLSearchParams, address: Address) => { + if (address.length > 1) urlParams.set("scope", address.join(".")); + else urlParams.delete("scope"); + }; + const setFocusedScope = React.useCallback( + (scope: NodeId) => { + const address = data?.structure.nodes.get(scope)?.address; + if (!address) return; + // A selection from the old scope is almost never still valid in the + // new one (a node's id generally isn't shared across scopes, and a + // click immediately preceding a double-click-to-navigate already set + // one for the region just navigated into, which won't be a listed + // child of itself). focusOn and onSelectLir bypass this by writing + // scope and selection together in their own updateSearchParams call, + // so a real "navigate to X and select Y" still lands intact. + updateSearchParams((urlParams) => { + applyScope(urlParams, address); + urlParams.delete("select"); + }); + }, + [data, updateSearchParams], + ); + // Selection lives in the URL too (same reasoning as scope above): a + // copied link, a reload, or back/forward all reopen at the same node, + // edge, or LIR group, not just the same scope. Encoded as + // `:` in one param; kind is always one of the three literal + // strings below, none of which contain ":", so splitting on the first + // ":" recovers id intact even though ids themselves can (port and edge + // ids do). + const selectionParamValue = (next: Selection): string => { + const id = + next.kind === "node" + ? next.node.id + : next.kind === "edge" + ? next.edge.id + : next.node.key; + return `${next.kind}:${id}`; + }; + const setSelection = React.useCallback( + (next: Selection | null) => { + updateSearchParams((urlParams) => { + if (next) urlParams.set("select", selectionParamValue(next)); + else urlParams.delete("select"); + }); + }, + [updateSearchParams], + ); + const [filters, setFilters] = React.useState(DEFAULT_FILTERS); + const [matchIndex, setMatchIndex] = React.useState(0); + const [lirHighlight, setLirHighlight] = + React.useState | null>(null); + const [pendingFitIds, setPendingFitIds] = React.useState( + null, + ); + // Persist independently of `selection`: closing (X) clears the selection + // itself, so the next click naturally reopens the panel, but collapsing + // should keep the panel out of the way across subsequent clicks too. + const [lirPanelCollapsed, setLirPanelCollapsed] = React.useState(false); + const [detailPanelCollapsed, setDetailPanelCollapsed] = React.useState(false); + + // The route only remounts on a clusterId change, so switching dataflow or + // replica in place (via the dropdowns) keeps this component instance, its + // filters alive. Selection and scope need no entry here: switching + // dataflow or replica always lands on a fresh URL built from scratch (the + // dropdowns pass only `replica`), which already carries neither a `scope` + // nor a `select` param, so both already read back as their defaults (root, + // nothing selected). Reset render-phase, not via effect, so there is no + // render in between showing the new graph under the old filters. Calling + // setSelection (a setSearchParams wrapper) here instead would be a + // different matter: mutating router state mid-render, unlike this filters + // state, which is local and always safe to update during render. + const resetKey = JSON.stringify([replicaName, dataflowId]); + const [trackedResetKey, setTrackedResetKey] = React.useState(resetKey); + if (resetKey !== trackedResetKey) { + setTrackedResetKey(resetKey); + setFilters(DEFAULT_FILTERS); + setMatchIndex(0); + setLirHighlight(null); + } + + const centerRef = React.useRef<((id: string) => void) | null>(null); + const fitRef = React.useRef<((ids: string[]) => void) | null>(null); + + // Navigates to the scope that makes every given address directly visible + // (as itself, or as the box that rolls it up), then fits the view to + // whichever representative boxes result once that scope's layout lands. + const navigateAndFit = React.useCallback( + (addresses: Address[]) => { + if (!data || addresses.length === 0) return; + const scope = commonAncestorScope(data.structure, addresses); + setFocusedScope(scope); + const scopeAddress = mustGet(data.structure.nodes, scope).address; + const targets = [ + ...new Set( + addresses + .map((a) => representativeInView(a, scopeAddress)) + .filter((id): id is NodeId => id !== null), + ), + ]; + setPendingFitIds(targets); + }, + [data, setFocusedScope], + ); + + const onLirSelect = React.useCallback( + (memberIds: NodeId[]) => { + if (!data) return; + const addresses = memberIds + .map((id) => data.structure.nodes.get(id)?.address) + // The dataflow root is never a direct child of any scope, so it + // can never be shown or highlighted; a LIR span that happens to + // cover it simply can't anchor navigation on it. + .filter((a): a is Address => a !== undefined && a.length > 1); + navigateAndFit(addresses); + }, + [data, navigateAndFit], + ); + + // Navigates to a scope and selects one specific box in it. Deriving the + // destination's graph to find that box doesn't need to wait for anything + // async (unlike fitting, which needs real layout positions from elk), so + // this can select synchronously, right here, in the same call that + // navigates — landing at a scope with many siblings (e.g. the root) never + // leaves it ambiguous which one a jump actually reached. + const focusOn = React.useCallback( + (scope: NodeId, targetId: NodeId) => { + if (!data) return; + const address = data.structure.nodes.get(scope)?.address; + if (!address) return; + setPendingFitIds([targetId]); + const node = deriveVisibleGraph(data.structure, scope).nodes.find( + (n) => n.id === targetId, + ); + updateSearchParams((urlParams) => { + applyScope(urlParams, address); + if (node) { + urlParams.set("select", selectionParamValue({ kind: "node", node })); + } else { + urlParams.delete("select"); + } + }); + }, + [data, updateSearchParams], + ); + + // A port's peer lives outside the current view. When the peer is itself a + // region, peerPortId names the exact port this crossing lands on from + // inside it, so the jump drills straight there instead of parking outside + // as an unlabeled box. A leaf peer has no inside to drill into, so the + // fallback lands on the peer itself, in its own containing scope. + const onJumpToPeer = React.useCallback( + (peer: PortPeer) => { + if (!data) return; + if (peer.peerPortId) { + focusOn(nodeIdOf(peer.address), peer.peerPortId); + return; + } + const scope = commonAncestorScope(data.structure, [peer.address]); + const scopeAddress = mustGet(data.structure.nodes, scope).address; + const targetId = representativeInView(peer.address, scopeAddress); + if (targetId) focusOn(scope, targetId); + }, + [data, focusOn], + ); + + const selectLirGroup = React.useCallback( + (key: string) => { + if (!data) return; + const entry = lirIndex(data.structure).get(key); + if (!entry) return; + setSelection({ + kind: "lirGroup", + node: { + key, + info: entry.info, + memberIds: entry.memberIds, + children: [], + summary: lirSummary(data.structure, entry.memberIds), + }, + }); + }, + [data, setSelection], + ); + + const onLirGroupClick = React.useCallback( + (group: { id: string }) => selectLirGroup(group.id), + [selectLirGroup], + ); + + // A node's own LIR entries link back to that LIR's group, wherever it + // actually is: unlike clicking the group's box directly on the graph + // (already visible, no navigation needed), the group here isn't + // necessarily in view yet. + const onSelectLir = React.useCallback( + (exportId: string, lirId: string) => { + if (!data) return; + const key = `${exportId}/${lirId}`; + const entry = lirIndex(data.structure).get(key); + if (!entry) return; + const addresses = entry.memberIds + .map((id) => data.structure.nodes.get(id)?.address) + .filter((a): a is Address => a !== undefined && a.length > 1); + if (addresses.length === 0) return; + // Inlines navigateAndFit's scope computation and selectLirGroup's + // param value, rather than calling both, to land scope and + // selection in one updateSearchParams call: two separate calls in + // this same handler would have the second silently discard the + // first (see updateSearchParams's doc comment above). + const scope = commonAncestorScope(data.structure, addresses); + const scopeAddress = mustGet(data.structure.nodes, scope).address; + const targets = [ + ...new Set( + addresses + .map((a) => representativeInView(a, scopeAddress)) + .filter((id): id is NodeId => id !== null), + ), + ]; + setPendingFitIds(targets); + const lirSelection: Selection = { + kind: "lirGroup", + node: { + key, + info: entry.info, + memberIds: entry.memberIds, + children: [], + summary: lirSummary(data.structure, entry.memberIds), + }, + }; + updateSearchParams((urlParams) => { + applyScope(urlParams, scopeAddress); + urlParams.set("select", selectionParamValue(lirSelection)); + }); + }, + [data, updateSearchParams], + ); + + const allMatches = React.useMemo( + () => (data ? allSearchMatches(data.structure, filters.search) : []), + [data, filters.search], + ); + // matchIndex resets to 0 only when the search text changes (see the effect + // below). A refetch that shrinks the match set for the same search text, + // or the render between a search-text change and that effect running, + // otherwise leaves it pointing past the end of allMatches. + const activeMatchIndex = Math.min(matchIndex, allMatches.length - 1); + + // Shared with DataflowGraphView via the visible prop below, so the current + // scope's graph is derived once per render instead of once here and again + // there. + const visibleGraph = React.useMemo( + () => + data && focusedScope + ? deriveVisibleGraph(data.structure, focusedScope) + : null, + [data, focusedScope], + ); + + // Mirrors DataflowGraphView's own labelById: restoring a URL-driven + // selection needs edge endpoint labels independently of a click event. + const labelById = React.useMemo( + () => new Map((visibleGraph?.nodes ?? []).map((n) => [n.id, n.label])), + [visibleGraph], + ); + + // Re-derived from the current graph on every render rather than trusted + // verbatim, the same tolerance focusedScope gets above: a stale link, or + // a dataflow whose shape changed, just fails to resolve and this reads + // back as null instead of crashing. + const selection = React.useMemo(() => { + if (!data || !visibleGraph || !selectParam) return null; + const sep = selectParam.indexOf(":"); + if (sep === -1) return null; + const kind = selectParam.slice(0, sep); + const id = selectParam.slice(sep + 1); + if (kind === "node") { + const node = visibleGraph.nodes.find((n) => n.id === id); + if (!node) return null; + const connectedEdges = + node.kind === "port" + ? visibleGraph.edges + .filter((e) => e.source === node.id || e.target === node.id) + .map((e) => ({ + ...e, + sourceLabel: labelById.get(e.source) ?? e.source, + targetLabel: labelById.get(e.target) ?? e.target, + })) + : undefined; + return { kind: "node", node, connectedEdges }; + } + if (kind === "edge") { + const edge = visibleGraph.edges.find((e) => e.id === id); + if (!edge) return null; + return { + kind: "edge", + edge: { + ...edge, + sourceLabel: labelById.get(edge.source) ?? edge.source, + targetLabel: labelById.get(edge.target) ?? edge.target, + }, + }; + } + if (kind === "lirGroup") { + const entry = lirIndex(data.structure).get(id); + if (!entry) return null; + return { + kind: "lirGroup", + node: { + key: id, + info: entry.info, + memberIds: entry.memberIds, + children: [], + summary: lirSummary(data.structure, entry.memberIds), + }, + }; + } + return null; + }, [data, visibleGraph, selectParam, labelById]); + + // Back/forward (and opening a shared link directly, which react-router + // also reports as "POP") can restore a selection the current viewport + // isn't anywhere near — a click, by contrast, always selects something + // already onscreen, so it doesn't need this. Re-fits only then, and only + // for node/edge (a lirGroup's members can be scattered outside a single + // fit's worth of space, so it's left to whatever the panel alone shows). + const navigationType = useNavigationType(); + React.useEffect(() => { + if (navigationType !== "POP" || !selection) return; + if (selection.kind === "node") setPendingFitIds([selection.node.id]); + else if (selection.kind === "edge") { + setPendingFitIds([selection.edge.source, selection.edge.target]); + } + }, [navigationType, selection]); + + const decorations = React.useMemo(() => { + if (!data || !focusedScope || !visibleGraph) return undefined; + const visible = visibleGraph; + const searchInfo = filters.search + ? subtreeSearchMatches(data.structure, filters.search) + : null; + const d = decorateGraph( + visible, + filters, + heatColor, + searchInfo, + data.workerCount, + ); + // Hovering a LIR row in the sidebar dims everything outside its members. + // A member outside the current scope, or rolled up into one of its + // boxes, highlights that box instead of being silently dropped. + if (lirHighlight) { + const focusedScopeAddress = mustGet( + data.structure.nodes, + focusedScope, + ).address; + const highlighted = new Set(); + for (const id of lirHighlight) { + const address = data.structure.nodes.get(id)?.address; + const rep = + address && representativeInView(address, focusedScopeAddress); + if (rep) highlighted.add(rep); + } + for (const n of visible.nodes) { + if (!highlighted.has(n.id)) d.dimmedNodeIds.add(n.id); + } + } + return d; + }, [data, focusedScope, visibleGraph, filters, lirHighlight]); + + React.useEffect(() => { + setMatchIndex(0); + }, [filters.search]); + + const onJump = React.useCallback( + (delta: 1 | -1) => { + if (allMatches.length === 0) return; + const next = + (activeMatchIndex + delta + allMatches.length) % allMatches.length; + setMatchIndex(next); + navigateAndFit([allMatches[next].address]); + }, + [allMatches, activeMatchIndex, navigateAndFit], + ); + // Digest of the sorted node ids: identical structure across a stats-only + // refresh yields the same key, so layout and collapse state are preserved. + // Memoized since sorting and hashing every node id is real work on a + // dataflow with tens of thousands of operators, and this otherwise reran + // on every render, not just on a new structure. + const structureKey = React.useMemo(() => { + if (!data) return null; + const ids = [...data.structure.nodes.keys()].sort(); + return `${params?.dataflowId}/${params?.replicaName}/${ids.length}-${hashString(ids.join(","))}`; + }, [data, params?.dataflowId, params?.replicaName]); + if (!cluster) return null; + if (cluster.replicas.length === 0) { + return ( + + This cluster has no replicas. + + ); + } + const permissionError = isInsufficientPrivilegeError(databaseError); + + return ( + // minH=0: MainContentContainer is a flex column item of BaseLayout's + //
, which defaults to min-height:auto and so refuses to shrink + // below its content's natural size. Harmless for pages that just scroll, + // but this page pins height=100% end-to-end so the graph area can fill + // whatever's left; without this override, one more row of content above + // the graph (e.g. the breadcrumb trail) pushes the whole page taller + // than the viewport instead of the graph area shrinking to absorb it. + + + + setSearchParams({ replica: e.target.value })} + flexShrink={0} + > + {cluster.replicas.map((r) => ( + + ))} + + + navigate( + `${absoluteClusterPath(regionSlug, cluster)}/dataflows/${e.target.value}?replica=${replicaName}`, + ) + } + flex="1" + minWidth="320px" + > + {dataflowId && !dataflowList?.some((d) => d.id === dataflowId) && ( + // The list query hasn't resolved yet (or this dataflow just + // disappeared from it); keep the select populated with + // something so it isn't blank while data is in flight. + + )} + {(dataflowList ?? []).map((d) => ( + + ))} + + + {data && ( + + + + Last fetched {data.fetchedAt.toLocaleTimeString()} + + + )} + {permissionError ? ( + + ) : error ? ( + + + + + ) : !data || !focusedScope ? ( + + ) : data.structure.nodes.size <= 1 ? ( + + This dataflow no longer exists on this replica.{" "} + Back to dataflows + + ) : ( + + + + + {filters.heatmap !== "off" && ( + + + low + + + + high + + + )} + + + setLirPanelCollapsed((prev) => !prev)} + onHighlight={setLirHighlight} + onSelect={onLirSelect} + /> + setPendingFitIds(null)} + selectedId={ + selection?.kind === "node" + ? selection.node.id + : selection?.kind === "edge" + ? selection.edge.id + : undefined + } + activeMatchId={ + allMatches.length > 0 + ? nodeIdOf(allMatches[activeMatchIndex].address) + : undefined + } + onNodeClick={(node, connectedEdges) => + setSelection({ + kind: "node", + node, + connectedEdges: + node.kind === "port" ? connectedEdges : undefined, + }) + } + onEdgeClick={(edge) => setSelection({ kind: "edge", edge })} + onPaneClick={() => setSelection(null)} + onJumpToPeer={onJumpToPeer} + onLirGroupClick={onLirGroupClick} + /> + {selection && ( + + setDetailPanelCollapsed((prev) => !prev) + } + onClose={() => setSelection(null)} + onJumpTo={onJumpToPeer} + onSelectLir={onSelectLir} + /> + )} + + + )} + + + ); +}; + +export default DataflowDetailPage; diff --git a/console/src/platform/dataflows/DataflowGraphView.css b/console/src/platform/dataflows/DataflowGraphView.css new file mode 100644 index 0000000000000..56dbe19e0464b --- /dev/null +++ b/console/src/platform/dataflows/DataflowGraphView.css @@ -0,0 +1,20 @@ +/* Copyright Materialize, Inc. and contributors. All rights reserved. + * + * Use of this software is governed by the Business Source License + * included in the LICENSE file. + * + * As of the Change Date specified in that file, in accordance with + * the Business Source License, use of this software will be governed + * by the Apache License, Version 2.0. + */ + +/* Expanding/collapsing a region re-runs elk layout from scratch, moving + * every node to a new position. Without a transition this reads as a jump + * cut; nodes are non-draggable, so animating transform/size here never + * fights with drag interaction. */ +.react-flow__node { + transition: + transform 200ms ease, + width 200ms ease, + height 200ms ease; +} diff --git a/console/src/platform/dataflows/DataflowGraphView.tsx b/console/src/platform/dataflows/DataflowGraphView.tsx new file mode 100644 index 0000000000000..eedc96e979001 --- /dev/null +++ b/console/src/platform/dataflows/DataflowGraphView.tsx @@ -0,0 +1,539 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import "@xyflow/react/dist/style.css"; +import "./DataflowGraphView.css"; + +import { Box, Button, Spinner, useTheme, VStack } from "@chakra-ui/react"; +import { + Background, + Controls, + type Edge, + MiniMap, + type Node, + Position, + ReactFlow, + useReactFlow, +} from "@xyflow/react"; +import React from "react"; + +import ErrorBox from "~/components/ErrorBox"; +import { MaterializeTheme } from "~/theme"; + +import { ChannelEdge } from "./ChannelEdge"; +import { + type GraphDecorations, + groupByLir, + type LirGroupNode as LirGroupNodeData, + type NodeId, + type PortPeer, + rerouteHiddenNodes, + type VisibleEdge, + type VisibleGraph, + type VisibleNode, +} from "./dataflowGraph"; +import { + NODE_DIMENSIONS, + type Positions, + resolveAbsolutePositions, +} from "./elkGraph"; +import { + LirGroupNode as LirGroupNodeComponent, + OperatorNode, + PortNode, + RegionNode, +} from "./nodes"; +import { lirGroupColor, nodeFillColor } from "./nodeStyle"; +import { useElkLayout } from "./useElkLayout"; + +const edgeTypes = { channel: ChannelEdge }; +const nodeTypes = { + operator: OperatorNode, + region: RegionNode, + port: PortNode, + lirGroup: LirGroupNodeComponent, +}; + +// An edge plus resolved endpoint labels, since VisibleEdge only carries ids. +export type SelectedEdge = VisibleEdge & { + sourceLabel: string; + targetLabel: string; +}; + +export interface DataflowGraphViewProps { + // The current scope's graph, already derived by the caller (which also + // needs it, to decorate). Computing it again here would repeat the same + // work every render. + visible: VisibleGraph; + // The scope whose direct children this view renders. Double-clicking a + // region box navigates to a new view rooted there rather than expanding it + // in place. + focusedScope: NodeId; + onNavigate: (scope: NodeId) => void; + cacheKey: string; + decorations?: GraphDecorations; + selectedId?: string; + activeMatchId?: string; + // Edges connected to the clicked node, with endpoint labels resolved, so a + // clicked port can show what flows through it without the caller + // recomputing the (already decorated, rerouted) visible graph. + onNodeClick?: (node: VisibleNode, connectedEdges: SelectedEdge[]) => void; + onEdgeClick?: (edge: SelectedEdge) => void; + onPaneClick?: () => void; + // Double-clicking a port with exactly one peer jumps straight there. With + // zero or several peers it's ambiguous (or there's nothing to jump to), so + // the preceding click/click of the double-click has already opened the + // port's own detail panel instead, same as a single click would. + onJumpToPeer?: (peer: PortPeer) => void; + centerRef?: React.MutableRefObject<((id: string) => void) | null>; + // A node to center on once its layout position exists (e.g. right after an + // expand triggered by jumping to it). Centering an id whose ancestors were + // just expanded can't happen synchronously: layout runs in a worker, so + // this id is retried across renders until its position shows up. + centerOnId?: string | null; + onCentered?: () => void; + // Fits the viewport around a whole set of nodes (e.g. every operator + // belonging to one LIR id), as opposed to centerRef's single-node zoom. + fitRef?: React.MutableRefObject<((ids: string[]) => void) | null>; + fitOnIds?: string[] | null; + onFit?: () => void; + // Toggled from the toolbar; grouping is computed here (from the same + // post-hideIdle, post-reroute node list elk actually lays out) rather than + // by the caller, so a hidden node can never end up as a group's only + // member. + showLirGroups?: boolean; + onLirGroupClick?: (group: LirGroupNodeData) => void; +} + +// Exposes centering/fitting callbacks through refs once React Flow context +// exists. +const CenterHelper = ({ + centerRef, + fitRef, +}: { + centerRef: React.MutableRefObject<((id: string) => void) | null>; + fitRef?: React.MutableRefObject<((ids: string[]) => void) | null>; +}) => { + const reactFlow = useReactFlow(); + React.useEffect(() => { + // Assigning to the ref inside an effect is safe: it never runs during + // render, so the parent's ref just receives the latest centering callback. + // eslint-disable-next-line react-compiler/react-compiler + centerRef.current = (id: string) => { + const internal = reactFlow.getInternalNode(id); + if (!internal) return; + const { x, y } = internal.internals.positionAbsolute; + const width = internal.measured?.width ?? 0; + const height = internal.measured?.height ?? 0; + reactFlow.setCenter(x + width / 2, y + height / 2, { + zoom: 1, + duration: 300, + }); + }; + if (fitRef) { + // eslint-disable-next-line react-compiler/react-compiler + fitRef.current = (ids: string[]) => { + void reactFlow.fitView({ + nodes: ids.map((id) => ({ id })), + duration: 300, + padding: 0.3, + maxZoom: 1, + }); + }; + } + return () => { + centerRef.current = null; + if (fitRef) fitRef.current = null; + }; + }, [reactFlow, centerRef, fitRef]); + return null; +}; + +// elk lays each scope out independently, starting near its own origin, so a +// viewport pan/zoom left over from a previous, unrelated scope (or from the +// user having panned away) can land on a spot with nothing from the new +// scope in it at all. Once this scope's layout lands, if none of its nodes +// are actually on screen, refit rather than leaving the canvas looking +// empty. Checked once per scope (skipping the initial one, which React +// Flow's own `fitView` prop already covers), not on every render, so +// panning away deliberately after landing doesn't get fought. +const ViewportGuard = ({ + focusedScope, + nodes, + positions, + paneRef, +}: { + focusedScope: NodeId; + nodes: VisibleNode[]; + positions: Positions | null; + paneRef: React.RefObject; +}) => { + const reactFlow = useReactFlow(); + const checkedScopeRef = React.useRef(focusedScope); + React.useEffect(() => { + if (!positions || nodes.length === 0) return; + if (checkedScopeRef.current === focusedScope) return; + checkedScopeRef.current = focusedScope; + const pane = paneRef.current; + if (!pane) return; + const rect = pane.getBoundingClientRect(); + const { x, y, zoom } = reactFlow.getViewport(); + const visMinX = -x / zoom; + const visMinY = -y / zoom; + const visMaxX = visMinX + rect.width / zoom; + const visMaxY = visMinY + rect.height / zoom; + const anyVisible = nodes.some((n) => { + const p = positions[n.id]; + return ( + p && + p.x < visMaxX && + p.x + p.width > visMinX && + p.y < visMaxY && + p.y + p.height > visMinY + ); + }); + if (!anyVisible) void reactFlow.fitView({ padding: 0.3, duration: 300 }); + }, [focusedScope, nodes, positions, paneRef, reactFlow]); + return null; +}; + +export const DataflowGraphView = ({ + visible: rawVisible, + focusedScope, + onNavigate, + cacheKey, + decorations, + selectedId, + activeMatchId, + onNodeClick, + onEdgeClick, + onPaneClick, + onJumpToPeer, + centerRef, + centerOnId, + onCentered, + fitRef, + fitOnIds, + onFit, + showLirGroups, + onLirGroupClick, +}: DataflowGraphViewProps) => { + const { colors, shadows } = useTheme(); + + const visible = React.useMemo(() => { + // Hidden nodes get spliced out with connectivity preserved (a hidden idle + // run still shows a pass-through edge). Hidden edges (independently + // zero-message) are a plain removal: nothing to reroute since both + // endpoints stay visible. + const rerouted = decorations?.hiddenNodeIds + ? rerouteHiddenNodes(rawVisible, decorations.hiddenNodeIds) + : rawVisible; + const hiddenEdgeIds = decorations?.hiddenEdgeIds; + if (!hiddenEdgeIds?.size) return rerouted; + return { + nodes: rerouted.nodes, + edges: rerouted.edges.filter((e) => !hiddenEdgeIds.has(e.id)), + }; + }, [rawVisible, decorations?.hiddenNodeIds, decorations?.hiddenEdgeIds]); + + const grouping = React.useMemo( + () => (showLirGroups ? groupByLir(visible.nodes) : null), + [showLirGroups, visible.nodes], + ); + + const layoutKey = `${cacheKey}|${focusedScope}|${showLirGroups ?? false}|${ + decorations?.hiddenNodeIds + ? [...decorations.hiddenNodeIds].sort().join(",") + : "" + }|${ + decorations?.hiddenEdgeIds + ? [...decorations.hiddenEdgeIds].sort().join(",") + : "" + }`; + const { positions, layouting, error, retry } = useElkLayout( + visible, + layoutKey, + grouping ?? undefined, + ); + + // Only ViewportGuard needs this: everything else either reads positions + // relative to parentId (React Flow's own convention, matching elk's) or + // resolves absolute position through React Flow's internal APIs. + const absolutePositions = React.useMemo( + () => + positions + ? resolveAbsolutePositions(positions, grouping?.parentOf) + : null, + [positions, grouping], + ); + + React.useEffect(() => { + if (!centerOnId || !positions?.[centerOnId]) return; + // CenterHelper assigns centerRef.current in its own mount effect; on the + // rare commit where that hasn't run yet, skip without marking the + // request consumed (onCentered stays uncalled) so the next positions or + // centerOnId change gets another chance, rather than silently dropping + // the jump forever. + if (!centerRef?.current) return; + centerRef.current(centerOnId); + onCentered?.(); + }, [centerOnId, positions, centerRef, onCentered]); + + React.useEffect(() => { + if (!fitOnIds || fitOnIds.length === 0) return; + // Fires as soon as any targets have a position rather than waiting for + // every one, so a partially-expanded set (e.g. one member still hidden + // behind a filter) doesn't block the fit indefinitely. + const present = fitOnIds.filter((id) => positions?.[id]); + if (present.length === 0) return; + if (!fitRef?.current) return; + fitRef.current(present); + onFit?.(); + }, [fitOnIds, positions, fitRef, onFit]); + + const nodes: Node[] = React.useMemo(() => { + if (!positions) return []; + const parentOf = grouping?.parentOf; + const groupNodes: Node[] = (grouping?.groups ?? []).map((g) => { + const pos = positions[g.id] ?? { x: 0, y: 0, width: 0, height: 0 }; + return { + id: g.id, + type: "lirGroup", + position: { x: pos.x, y: pos.y }, + parentId: parentOf?.get(g.id), + width: pos.width, + height: pos.height, + // pointerEvents: "none" on React Flow's own node wrapper (not just + // inside LirGroupNode's own JSX) is the other half of the + // click-through contract LirGroupNode's comment documents: the + // wrapper defaults to pointer-events:all and LirGroupNode can't + // override an ancestor it doesn't render. With the wrapper opted + // out, LirGroupNode's header (pointerEvents:"auto") still receives + // clicks (a descendant can always re-enable itself), and a click on + // the group's empty body falls through to a member's own wrapper + // when one is there, or further to the pane when none is. + style: { width: pos.width, height: pos.height, pointerEvents: "none" }, + draggable: false, + connectable: false, + // Below its members, which draw over it; React Flow requires a + // group node to precede its children in this array, which the + // groupNodes-then-memberNodes concat below already guarantees, but + // an explicit zIndex also protects against member nodes elsewhere + // in the array being reordered by a future change. + zIndex: -1, + data: { + group: g, + label: g.operator, + color: lirGroupColor(g.lirId, colors.lineGraph), + }, + }; + }); + const memberNodes: Node[] = visible.nodes.map((n) => { + const pos = positions[n.id] ?? { x: 0, y: 0, ...NODE_DIMENSIONS[n.kind] }; + return { + id: n.id, + type: n.kind, + position: { x: pos.x, y: pos.y }, + parentId: grouping?.parentOf.get(n.id), + // Match the Top/Bottom handles so bezier control points meet the + // node edges. Without this the edge curves toward the default + // Bottom/Top and appears detached across region boundaries. + targetPosition: Position.Top, + sourcePosition: Position.Bottom, + // Set as explicit node fields, not just CSS style: the MiniMap and + // culled (onlyRenderVisibleElements) off-screen nodes both need a + // known size without waiting for DOM measurement. + width: pos.width, + height: pos.height, + style: { width: pos.width, height: pos.height }, + draggable: false, + connectable: false, + data: { + node: n, + dimmed: decorations?.dimmedNodeIds?.has(n.id) ?? false, + color: + decorations?.nodeColors?.get(n.id) ?? + nodeFillColor(n, colors.lineGraph, { + arranged: colors.accent.purple, + notArranged: colors.accent.green, + }), + selected: n.id === selectedId, + activeMatch: n.id === activeMatchId, + }, + }; + }); + return [...groupNodes, ...memberNodes]; + }, [ + visible, + positions, + grouping, + decorations?.dimmedNodeIds, + decorations?.nodeColors, + selectedId, + activeMatchId, + colors, + ]); + + const edges: Edge[] = React.useMemo( + () => + visible.edges.map((e) => ({ + id: e.id, + source: e.source, + target: e.target, + type: "channel", + data: { + messagesSent: e.messagesSent, + batchesSent: e.batchesSent, + channelTypes: e.channelTypes, + dimmed: + (decorations?.dimmedNodeIds?.has(e.source) ?? false) || + (decorations?.dimmedNodeIds?.has(e.target) ?? false), + selected: e.id === selectedId, + // selectedId is a node id when a node (rather than an edge) is + // selected; an edge's source/target are always node ids too, so + // this naturally never matches while an edge itself is selected, + // with no need to separately track which kind selectedId is. + connected: e.source === selectedId || e.target === selectedId, + sourceLandings: e.sourceLandings, + targetLandings: e.targetLandings, + onJumpTo: onJumpToPeer, + }, + })), + [visible, decorations?.dimmedNodeIds, selectedId, onJumpToPeer], + ); + + // Built once per visible-graph change instead of a .find() per connected + // edge in the click handlers below, which is O(nodes) per edge and adds up + // for a hub node in a large scope. + const labelById = React.useMemo( + () => new Map(visible.nodes.map((n) => [n.id, n.label])), + [visible], + ); + + const paneRef = React.useRef(null); + + if (error) { + return ( + + + + + ); + } + return ( + + {layouting && ( + + + + )} + { + if (node.type === "lirGroup") { + onLirGroupClick?.((node.data as { group: LirGroupNodeData }).group); + return; + } + const visibleNode = (node.data as { node: VisibleNode }).node; + const connectedEdges = visible.edges + .filter( + (e) => e.source === visibleNode.id || e.target === visibleNode.id, + ) + .map((e) => ({ + ...e, + sourceLabel: labelById.get(e.source) ?? e.source, + targetLabel: labelById.get(e.target) ?? e.target, + })); + onNodeClick?.(visibleNode, connectedEdges); + }} + onEdgeClick={(_, edge) => { + const e = visible.edges.find((ve) => ve.id === edge.id); + if (!e) return; + onEdgeClick?.({ + ...e, + sourceLabel: labelById.get(e.source) ?? e.source, + targetLabel: labelById.get(e.target) ?? e.target, + }); + }} + onPaneClick={onPaneClick} + onNodeDoubleClick={(_, node) => { + // Groups aren't scopes: no drill-down, no jump, nothing happens. + if (node.type === "lirGroup") return; + const visibleNode = (node.data as { node: VisibleNode }).node; + if (visibleNode.kind === "region") { + onNavigate(visibleNode.id); + } else if ( + visibleNode.kind === "port" && + visibleNode.peers.length === 1 + ) { + onJumpToPeer?.(visibleNode.peers[0]); + } + }} + proOptions={{ hideAttribution: true }} + > + + + + (node.data as { color?: string }).color ?? + colors.background.tertiary + } + // Node fills are light (operator colors and the cold end of the + // heatmap gradient both skew pale) in light mode, and dark in + // dark mode, so without an explicit stroke a scope of same-toned + // nodes can render as an almost-blank minimap. + nodeStrokeColor={colors.border.secondary} + /> + {centerRef && } + + + + ); +}; diff --git a/console/src/platform/dataflows/DataflowToolbar.test.tsx b/console/src/platform/dataflows/DataflowToolbar.test.tsx new file mode 100644 index 0000000000000..6481c6921ad66 --- /dev/null +++ b/console/src/platform/dataflows/DataflowToolbar.test.tsx @@ -0,0 +1,58 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { ThemeProvider } from "@chakra-ui/react"; +import { render, screen } from "@testing-library/react"; +import React from "react"; + +import { lightTheme } from "~/theme"; + +import { DEFAULT_FILTERS } from "./dataflowGraph"; +import { DataflowToolbar } from "./DataflowToolbar"; + +function renderToolbar(workerCount: number) { + return render( + + {}} + matchCount={0} + matchIndex={0} + onJump={() => {}} + workerCount={workerCount} + /> + , + ); +} + +describe("DataflowToolbar", () => { + it("disables the skew heatmap options on a single-worker replica, where skew is undefined", () => { + renderToolbar(1); + for (const label of [ + "Heat: CPU skew", + "Heat: memory skew", + "Heat: schedule skew", + ]) { + expect(screen.getByRole("option", { name: label })).toBeDisabled(); + } + // Non-skew heatmap modes still work fine on a single worker. + expect(screen.getByRole("option", { name: "Heat: elapsed" })).toBeEnabled(); + }); + + it("enables the skew heatmap options once there's more than one worker to compare", () => { + renderToolbar(4); + for (const label of [ + "Heat: CPU skew", + "Heat: memory skew", + "Heat: schedule skew", + ]) { + expect(screen.getByRole("option", { name: label })).toBeEnabled(); + } + }); +}); diff --git a/console/src/platform/dataflows/DataflowToolbar.tsx b/console/src/platform/dataflows/DataflowToolbar.tsx new file mode 100644 index 0000000000000..d5376ef7a8fd0 --- /dev/null +++ b/console/src/platform/dataflows/DataflowToolbar.tsx @@ -0,0 +1,168 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { + Button, + FormControl, + FormLabel, + HStack, + Slider, + SliderFilledTrack, + SliderThumb, + SliderTrack, + Switch, + Text, +} from "@chakra-ui/react"; +import React from "react"; + +import SearchInput from "~/components/SearchInput"; +import SimpleSelect from "~/components/SimpleSelect"; + +import { type Filters } from "./dataflowGraph"; + +export interface DataflowToolbarProps { + filters: Filters; + onFiltersChange: (next: Filters) => void; + matchCount: number; + matchIndex: number; + onJump: (delta: 1 | -1) => void; + // Skew is a worst-worker-over-average ratio: on a single-worker replica + // there's nothing to compare against, so every node's ratio is trivially + // 1 (or the "no data" 0 sentinel), and the heatmap would silently render + // no color at all, indistinguishable from being off. Disabling the option + // instead of shipping a heatmap mode that quietly does nothing. + workerCount: number; +} + +export const DataflowToolbar = ({ + filters, + onFiltersChange, + matchCount, + matchIndex, + onJump, + workerCount, +}: DataflowToolbarProps) => { + const skewDisabled = workerCount <= 1; + const [search, setSearch] = React.useState(filters.search); + // Debounce search input into the filters object. + React.useEffect(() => { + if (search === filters.search) return; + const timeout = setTimeout( + () => onFiltersChange({ ...filters, search }), + 300, + ); + return () => clearTimeout(timeout); + }, [search, filters, onFiltersChange]); + return ( + + setSearch(e.target.value)} + /> + {filters.search && ( + + + + + {matchCount === 0 ? "0/0" : `${matchIndex + 1}/${matchCount}`} + + + )} + + + Hide idle + + + onFiltersChange({ ...filters, hideIdle: e.target.checked }) + } + /> + + + + Show LIR groups + + + onFiltersChange({ ...filters, showLirGroups: e.target.checked }) + } + /> + + + onFiltersChange({ + ...filters, + heatmap: e.target.value as Filters["heatmap"], + }) + } + > + + + + + + + + + onFiltersChange({ ...filters, heatmapThreshold: v })} + > + + + + + + + ); +}; diff --git a/console/src/platform/dataflows/DataflowsPage.tsx b/console/src/platform/dataflows/DataflowsPage.tsx new file mode 100644 index 0000000000000..4d872ea65d605 --- /dev/null +++ b/console/src/platform/dataflows/DataflowsPage.tsx @@ -0,0 +1,139 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { + Spinner, + Table, + Tbody, + Td, + Th, + Thead, + Tr, + VStack, +} from "@chakra-ui/react"; +import React from "react"; +import { Link, Navigate, useParams, useSearchParams } from "react-router-dom"; + +import { useDataflowIdForExport } from "~/api/materialize/dataflow/useDataflowIdForExport"; +import { useDataflowList } from "~/api/materialize/dataflow/useDataflowList"; +import { isInsufficientPrivilegeError } from "~/api/materialize/executeSql"; +import ErrorBox from "~/components/ErrorBox"; +import LabeledSelect from "~/components/LabeledSelect"; +import { MainContentContainer } from "~/layouts/BaseLayout"; +import { useAllClusters } from "~/store/allClusters"; +import { formatBytesShort, formatElapsedNs } from "~/utils/format"; + +import { UsagePrivilegeAlert } from "./UsagePrivilegeAlert"; + +const DataflowsPage = () => { + const { clusterId } = useParams(); + const { getClusterById } = useAllClusters(); + const cluster = clusterId ? getClusterById(clusterId) : undefined; + const [searchParams, setSearchParams] = useSearchParams(); + const replicaName = searchParams.get("replica") ?? cluster?.replicas[0]?.name; + const params = React.useMemo( + () => + cluster && replicaName + ? { clusterName: cluster.name, replicaName } + : undefined, + [cluster, replicaName], + ); + const { data, error, databaseError, loading } = useDataflowList(params); + + // When an object is deep-linked via ?export=, resolve its running + // dataflow on the chosen replica and redirect to that dataflow's page. The + // hook is called unconditionally and no-ops when there is no export to + // resolve. + const exportId = searchParams.get("export") ?? undefined; + const exportParams = React.useMemo( + () => + cluster && replicaName && exportId + ? { clusterName: cluster.name, replicaName, exportId } + : undefined, + [cluster, replicaName, exportId], + ); + const { + dataflowId, + loading: exportLoading, + error: exportError, + databaseError: exportDatabaseError, + } = useDataflowIdForExport(exportParams); + + if (!cluster) return null; + + if (exportParams) { + if (exportLoading) return ; + if (dataflowId !== null) { + return ; + } + } + const permissionError = + isInsufficientPrivilegeError(databaseError) || + isInsufficientPrivilegeError(exportDatabaseError); + + // The export resolved cleanly but no dataflow is running for it on this + // replica. Surface that above the list, which still renders below. + const exportHasNoDataflow = + exportParams !== undefined && + !exportLoading && + !exportError && + dataflowId === null; + return ( + + + {exportHasNoDataflow && ( + + )} + setSearchParams({ replica: e.target.value })} + > + {cluster.replicas.map((r) => ( + + ))} + + {permissionError ? ( + + ) : error ? ( + + ) : !data && loading ? ( + + ) : ( + + + + + + + + + + + {(data ?? []).map((d) => ( + + + + + + + ))} + +
NameRecordsSizeScheduled
+ {d.name} + {d.records.toString()}{formatBytesShort(d.size)}{formatElapsedNs(d.elapsedNs)}
+ )} +
+
+ ); +}; + +export default DataflowsPage; diff --git a/console/src/platform/dataflows/LirPanel.tsx b/console/src/platform/dataflows/LirPanel.tsx new file mode 100644 index 0000000000000..fe194108bcf7f --- /dev/null +++ b/console/src/platform/dataflows/LirPanel.tsx @@ -0,0 +1,139 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { Box, Button, HStack, Text, useTheme } from "@chakra-ui/react"; +import React from "react"; + +import { ChevronLeftIcon, ChevronRightIcon } from "~/icons"; +import { MaterializeTheme } from "~/theme"; + +import { + type DataflowStructure, + lirIndex, + lirTree, + type LirTreeNode, + type NodeId, +} from "./dataflowGraph"; + +interface LirRowProps { + node: LirTreeNode; + onHighlight: (ids: ReadonlySet | null) => void; + onSelect: (memberIds: NodeId[]) => void; +} + +const LirRow = ({ node, onHighlight, onSelect }: LirRowProps) => { + const { colors } = useTheme(); + return ( + <> + onHighlight(new Set(node.memberIds))} + onMouseLeave={() => onHighlight(null)} + onClick={() => onSelect(node.memberIds)} + > + LIR {node.info.lirId}: {node.info.operator} + + {node.children.map((child) => ( + + ))} + + ); +}; + +export interface LirPanelProps { + structure: DataflowStructure; + collapsed: boolean; + onToggleCollapsed: () => void; + onHighlight: (ids: ReadonlySet | null) => void; + onSelect: (memberIds: NodeId[]) => void; +} + +export const LirPanel = ({ + structure, + collapsed, + onToggleCollapsed, + onHighlight, + onSelect, +}: LirPanelProps) => { + const { colors } = useTheme(); + const index = React.useMemo(() => lirIndex(structure), [structure]); + const tree = React.useMemo( + () => lirTree(structure, index), + [structure, index], + ); + + if (index.size === 0) return null; + if (collapsed) { + return ( + + + + ); + } + return ( + + + + LIR + + + + {[...tree.entries()].map(([exportId, roots]) => ( + + + {exportId} + + {roots.map((root) => ( + + ))} + + ))} + + ); +}; diff --git a/console/src/platform/dataflows/NodeDetailPanel.tsx b/console/src/platform/dataflows/NodeDetailPanel.tsx new file mode 100644 index 0000000000000..df54baf2ae48e --- /dev/null +++ b/console/src/platform/dataflows/NodeDetailPanel.tsx @@ -0,0 +1,403 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { + Box, + Button, + CloseButton, + HStack, + Text, + useTheme, +} from "@chakra-ui/react"; +import React from "react"; + +import TextLink from "~/components/TextLink"; +import { ArrowRightIcon, ChevronLeftIcon, ChevronRightIcon } from "~/icons"; +import { MaterializeTheme } from "~/theme"; +import { formatBytesShort, formatElapsedNs } from "~/utils/format"; + +import { + type LirTreeNode, + type PortPeer, + type VisibleNode, +} from "./dataflowGraph"; +import type { SelectedEdge } from "./DataflowGraphView"; +import { formatSkew, prettyPrintChannelType } from "./nodeStyle"; + +export type Selection = + | { kind: "node"; node: VisibleNode; connectedEdges?: SelectedEdge[] } + | { kind: "edge"; edge: SelectedEdge } + | { kind: "lirGroup"; node: LirTreeNode }; + +interface RowProps { + label: string; + value: string; +} + +const Row = ({ label, value }: RowProps) => { + const { colors } = useTheme(); + return ( + + + {label} + + + {value} + + + ); +}; + +// A border-top divider between stat groups (own vs subtree vs skew, or a +// titled block like LIR/Connected edges), matching the workflow graph +// sidebar's SidebarSection dividers, without that component's collapse +// behavior, which this panel's shorter, denser content doesn't need. +const StatGroup = ({ children }: { children: React.ReactNode }) => { + const { colors } = useTheme(); + return ( + + {children} + + ); +}; + +const SectionLabel = (props: React.PropsWithChildren) => ( + +); + +interface TypeRowProps { + channelTypes: string[]; +} + +// Channel types are Rust container type signatures (pretty-printed, but +// still nested-generic shaped, e.g. "[Rc>]") long enough +// that they get their own full-width stacked row instead of Row's +// space-between line. +const TypeRow = ({ channelTypes }: TypeRowProps) => { + const { colors } = useTheme(); + return ( + + + Type + + + {channelTypes.map(prettyPrintChannelType).join(", ") || "unknown"} + + + ); +}; + +interface EdgeRowsProps { + edge: SelectedEdge; + // Omitted for the compact form used inside a node's "Connected edges" + // list (which can be long for a hub node): landing lists there would + // swamp the panel. The dedicated edge-selection view always passes it, + // to show where a merged edge's real channels land inside a collapsed + // region. + onJumpTo?: (peer: PortPeer) => void; +} + +const EdgeRows = ({ edge, onJumpTo }: EdgeRowsProps) => ( + <> + + + + {onJumpTo && edge.sourceLandings.length > 0 && ( + + Inside {edge.sourceLabel} + {edge.sourceLandings.map((p) => ( + + ))} + + )} + {onJumpTo && edge.targetLandings.length > 0 && ( + + Inside {edge.targetLabel} + {edge.targetLandings.map((p) => ( + + ))} + + )} + +); + +interface PeerRowProps { + peer: PortPeer; + onJumpTo: (peer: PortPeer) => void; +} + +// A port's peer lives outside the current view (that's why it's a port and +// not a drawn edge), so its only affordance is a jump: navigate to wherever +// the peer actually is instead of tracing a line to it. +const PeerRow = ({ peer, onJumpTo }: PeerRowProps) => { + const { colors } = useTheme(); + return ( + + + + + + {peer.label} + + + + + + + + + ); +}; + +interface LirGroupDetailProps { + node: LirTreeNode; +} + +// Same Row-based shape as NodeDetail, rather than a visually distinct card, +// so a LIR group's details read like every other selection in this panel. +const LirGroupDetail = ({ node }: LirGroupDetailProps) => ( + <> + + + + + + + + +); + +interface NodeDetailProps { + node: VisibleNode; + connectedEdges?: SelectedEdge[]; + onJumpTo: (peer: PortPeer) => void; + onSelectLir: (exportId: string, lirId: string) => void; +} + +const NodeDetail = ({ + node, + connectedEdges, + onJumpTo, + onSelectLir, +}: NodeDetailProps) => { + const { colors } = useTheme(); + return ( + <> + + {node.address && } + {node.operatorId !== null && ( + + )} + {node.childCount > 0 && ( + + )} + {node.own && ( + + + + + + + )} + {node.childCount > 0 && ( + + )} + {node.transitive && node.childCount > 0 && ( + + + + + + + )} + {node.ownSkew && ( + + + + + + )} + {node.transitiveSkew && node.childCount > 0 && ( + + + + + + )} + {node.lir.length > 0 && ( + + LIR + {node.lir.map((l) => ( + onSelectLir(l.exportId, l.lirId)} + > + LIR {l.lirId} ({l.exportId}): {l.operator} + + ))} + + )} + {connectedEdges && connectedEdges.length > 0 && ( + + Connected edges + {connectedEdges.map((e) => ( + + + {e.sourceLabel} → {e.targetLabel} + + + + ))} + + )} + {node.peers.length > 0 && ( + + Connects outside this view + {node.peers.map((p) => ( + + ))} + + )} + + ); +}; + +export interface NodeDetailPanelProps { + selection: Selection; + collapsed: boolean; + onToggleCollapsed: () => void; + onClose: () => void; + onJumpTo: (peer: PortPeer) => void; + onSelectLir: (exportId: string, lirId: string) => void; +} + +export const NodeDetailPanel = ({ + selection, + collapsed, + onToggleCollapsed, + onClose, + onJumpTo, + onSelectLir, +}: NodeDetailPanelProps) => { + if (collapsed) { + return ( + + + + ); + } + const title = + selection.kind === "node" + ? selection.node.label + : selection.kind === "edge" + ? `${selection.edge.sourceLabel} → ${selection.edge.targetLabel}` + : `LIR ${selection.node.info.lirId}: ${selection.node.info.operator}`; + return ( + + + + {title} + + + + + + + {selection.kind === "node" ? ( + + ) : selection.kind === "edge" ? ( + + ) : ( + + )} + + ); +}; diff --git a/console/src/platform/dataflows/UsagePrivilegeAlert.tsx b/console/src/platform/dataflows/UsagePrivilegeAlert.tsx new file mode 100644 index 0000000000000..f044fe12be312 --- /dev/null +++ b/console/src/platform/dataflows/UsagePrivilegeAlert.tsx @@ -0,0 +1,32 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { Text } from "@chakra-ui/react"; +import React from "react"; + +import Alert from "~/components/Alert"; + +export interface UsagePrivilegeAlertProps { + action: string; +} + +export const UsagePrivilegeAlert = ({ action }: UsagePrivilegeAlertProps) => ( + + You'll need{" "} + + USAGE + {" "} + privilege on this cluster to {action}. + + } + /> +); diff --git a/console/src/platform/dataflows/dataflowGraph.test.ts b/console/src/platform/dataflows/dataflowGraph.test.ts new file mode 100644 index 0000000000000..e56921d63ae76 --- /dev/null +++ b/console/src/platform/dataflows/dataflowGraph.test.ts @@ -0,0 +1,1510 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { describe, expect, it } from "vitest"; + +import { + allSearchMatches, + buildDataflowStructure, + type ChannelRow, + commonAncestorScope, + decorateGraph, + DEFAULT_FILTERS, + deriveVisibleGraph, + groupByLir, + lirIndex, + type LirInfo, + type LirSpanRow, + lirTree, + nodeIdOf, + type OperatorRow, + type PerWorkerStatRow, + representativeInView, + rerouteHiddenNodes, + subtreeSearchMatches, + type VisibleEdge, + type VisibleNode, +} from "./dataflowGraph"; + +// Dataflow 5: root [5], region [5,1] with children [5,1,1], [5,1,2], leaf [5,2]. +export const OPS: OperatorRow[] = [ + { + id: "10", + address: ["5"], + name: "Dataflow", + arrangementRecords: null, + arrangementSize: null, + elapsedNs: "0", + }, + { + id: "11", + address: ["5", "1"], + name: "Region", + arrangementRecords: "0", + arrangementSize: "0", + elapsedNs: "5", + }, + { + id: "12", + address: ["5", "1", "1"], + name: "Join", + arrangementRecords: "100", + arrangementSize: "4096", + elapsedNs: "7", + }, + { + id: "13", + address: ["5", "1", "2"], + name: "Map", + arrangementRecords: "0", + arrangementSize: "0", + elapsedNs: "1", + }, + { + id: "14", + address: ["5", "2"], + name: "Sink", + arrangementRecords: "0", + arrangementSize: "0", + elapsedNs: "2", + }, +]; +export const CHANNELS: ChannelRow[] = [ + // into the region: leaf [5,2] <- region output; region input port -> child + { + id: "1", + fromOperatorAddress: ["5", "1", "0"], + fromPort: "0", + toOperatorAddress: ["5", "1", "1"], + toPort: "0", + messagesSent: "3", + batchesSent: "1", + channelType: "rows", + }, + { + id: "2", + fromOperatorAddress: ["5", "1", "1"], + fromPort: "0", + toOperatorAddress: ["5", "1", "2"], + toPort: "0", + messagesSent: "5", + batchesSent: "2", + channelType: "rows", + }, + { + id: "3", + fromOperatorAddress: ["5", "1"], + fromPort: "0", + toOperatorAddress: ["5", "2"], + toPort: "0", + messagesSent: "5", + batchesSent: "2", + channelType: "batches", + }, +]; +export const LIR_SPANS: LirSpanRow[] = [ + { + exportId: "u42", + lirId: "1", + parentLirId: null, + nesting: 0, + operator: "Join::Differential", + operatorIdStart: "11", + operatorIdEnd: "13", + }, +]; + +describe("buildDataflowStructure", () => { + it("builds the region tree from address prefixes", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS); + expect(s.root).toEqual(nodeIdOf([5])); + const root = s.nodes.get(s.root)!; + expect(root.children).toEqual([nodeIdOf([5, 1]), nodeIdOf([5, 2])]); + const region = s.nodes.get(nodeIdOf([5, 1]))!; + expect(region.parent).toEqual(s.root); + expect(region.children).toEqual([nodeIdOf([5, 1, 1]), nodeIdOf([5, 1, 2])]); + expect(region.operatorId).toEqual(11n); + }); + + it("precomputes transitive stats", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS); + const region = s.nodes.get(nodeIdOf([5, 1]))!; + expect(region.own.elapsedNs).toEqual(5n); + expect(region.transitive.elapsedNs).toEqual(13n); + expect(region.transitive.arrangementRecords).toEqual(100n); + expect(region.transitive.arrangementSize).toEqual(4096n); + }); + + it("maps LIR spans to operators by id range, as an array", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS); + expect(s.nodes.get(nodeIdOf([5, 1, 1]))!.lir).toEqual([ + { + exportId: "u42", + lirId: "1", + parentLirId: null, + nesting: 0, + operator: "Join::Differential", + }, + ]); + // end is exclusive + expect(s.nodes.get(nodeIdOf([5, 2]))!.lir).toEqual([]); + }); + + it("throws with more than one root", () => { + const twoRoots: OperatorRow[] = [ + { ...OPS[0], id: "10", address: ["5"] }, + { ...OPS[0], id: "20", address: ["6"] }, + ]; + expect(() => buildDataflowStructure(twoRoots, [], [])).toThrow(); + }); + + it("returns a single placeholder node for a dropped dataflow, instead of throwing on zero operators", () => { + // A dropped or transient dataflow returns zero operator rows: a valid + // empty result, not an error. nodes.size === 1 here is what routes it + // through the same "no longer exists" branch as any other empty + // dataflow, rather than crashing mid-fetch. + const s = buildDataflowStructure([], [], []); + expect(s.nodes.size).toBe(1); + expect(s.nodes.get(s.root)).toBeDefined(); + }); + + it("normalizes channels", () => { + const s = buildDataflowStructure(OPS, CHANNELS, []); + expect(s.channels[0]).toEqual({ + id: 1, + fromAddress: [5, 1, 0], + fromPort: 0, + toAddress: [5, 1, 1], + toPort: 0, + messagesSent: 3n, + batchesSent: 1n, + channelType: "rows", + }); + }); + + it("computes own and transitive schedule counts alongside elapsed time", () => { + const s = buildDataflowStructure( + OPS.map((o) => + o.id === "12" + ? { ...o, scheduleCount: "100" } + : o.id === "13" + ? { ...o, scheduleCount: "50" } + : o, + ), + CHANNELS, + LIR_SPANS, + ); + const join = s.nodes.get(nodeIdOf([5, 1, 1]))!; + expect(join.own.scheduleCount).toEqual(100n); + const region = s.nodes.get(nodeIdOf([5, 1]))!; + // Region's own row sets no scheduleCount (defaults 0); transitive sums + // Join's and Map's. + expect(region.transitive.scheduleCount).toEqual(150n); + }); + + it("computes a region's overhead as its own elapsed time minus its direct children's own elapsed time", () => { + const overheadOps = OPS.map((o) => + o.id === "11" + ? { ...o, elapsedNs: "100" } + : o.id === "12" + ? { ...o, elapsedNs: "30" } + : o.id === "13" + ? { ...o, elapsedNs: "20" } + : o, + ); + const s = buildDataflowStructure(overheadOps, CHANNELS, LIR_SPANS); + const region = s.nodes.get(nodeIdOf([5, 1]))!; + // 100 - (30 + 20) = 50: time in the region's own scheduling loop, not + // attributable to either child. + expect(region.overheadNs).toEqual(50n); + // A leaf has no children to subtract, so this is just its own elapsed + // time -- not a meaningful "overhead" reading on its own. + const join = s.nodes.get(nodeIdOf([5, 1, 1]))!; + expect(join.overheadNs).toEqual(30n); + }); + + it("compares a region's own time against its direct children's own time, not their transitive rollup", () => { + // Timely's own-elapsed accounting is exclusive self time (verified + // against live introspection data: a real region's own.elapsedNs can + // sit well below its direct children's own.elapsedNs summed, which is + // only possible if "own" never included child execution to begin + // with). Join here has its own grandchild (a nested region inside it) + // whose time dwarfs Join's own row; overhead must compare Region + // against Join's own 5ns, not Join's transitive total. + const nested = buildDataflowStructure( + [ + { ...OPS[0], id: "20", address: ["6"], name: "Dataflow" }, + { + ...OPS[0], + id: "21", + address: ["6", "1"], + name: "Region", + elapsedNs: "10", + }, + { + ...OPS[0], + id: "22", + address: ["6", "1", "1"], + name: "Join", + elapsedNs: "5", + }, + { + ...OPS[0], + id: "23", + address: ["6", "1", "1", "1"], + name: "GrandchildLeaf", + elapsedNs: "1000", + }, + ], + [], + [], + ); + const region = nested.nodes.get(nodeIdOf([6, 1]))!; + // Join's transitive elapsed (5 + 1000 = 1005) would make this wildly + // negative if used instead of Join's own (5): 10 - 5 = 5. + expect(region.overheadNs).toEqual(5n); + }); + + describe("skew", () => { + // Join (id 12, [5,1,1]): worker 0 does 10ns/1000B, worker 1 does + // 30ns/3000B -> 2x every direction. Map (id 13, [5,1,2]): only worker 0 + // ever scheduled it (5ns) and it never arranges anything -> perfectly + // even CPU (a single data point can't be skewed), no memory data at all. + const PER_WORKER: PerWorkerStatRow[] = [ + { id: "12", workerId: "0", elapsedNs: "10", arrangementSize: "1000" }, + { id: "12", workerId: "1", elapsedNs: "30", arrangementSize: "3000" }, + { id: "13", workerId: "0", elapsedNs: "5", arrangementSize: null }, + ]; + + it("computes own skew as worst-worker over average, over participating workers only", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS, PER_WORKER); + const join = s.nodes.get(nodeIdOf([5, 1, 1]))!; + expect(join.ownSkew.cpuSkew).toBeCloseTo(1.5); // 30 / ((10+30)/2) + expect(join.ownSkew.memorySkew).toBeCloseTo(1.5); // 3000 / 2000 + + const map = s.nodes.get(nodeIdOf([5, 1, 2]))!; + expect(map.ownSkew.cpuSkew).toBeCloseTo(1); // one worker, one data point + expect(map.ownSkew.memorySkew).toEqual(0); // no memory data at all + + const sink = s.nodes.get(nodeIdOf([5, 2]))!; + expect(sink.ownSkew).toEqual({ + cpuSkew: 0, + memorySkew: 0, + scheduleSkew: 0, + }); + }); + + it("computes schedule-count skew the same way as CPU/memory skew", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS, [ + { + id: "12", + workerId: "0", + elapsedNs: null, + arrangementSize: null, + scheduleCount: "100", + }, + { + id: "12", + workerId: "1", + elapsedNs: null, + arrangementSize: null, + scheduleCount: "300", + }, + ]); + const join = s.nodes.get(nodeIdOf([5, 1, 1]))!; + expect(join.ownSkew.scheduleSkew).toBeCloseTo(1.5); // 300 / ((100+300)/2) + }); + + it("computes transitive skew as the max of its own and its subtree's skew, matching EXPLAIN ANALYZE's rollup, not a merged-and-then-ratioed vector", () => { + // Join is skewed toward worker 0 (1000 vs 0); Map is skewed toward + // worker 1 (0 vs 1000) by the same amount, in the opposite + // direction. Summed per worker, the totals are 1000/1000 across the + // whole region -- perfectly even -- even though both operators are + // individually badly skewed (2x each): a merged-vector rollup would + // report the region as perfectly balanced, hiding both bad actors. + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS, [ + { id: "12", workerId: "0", elapsedNs: "1000", arrangementSize: "0" }, + { id: "12", workerId: "1", elapsedNs: "0", arrangementSize: "0" }, + { id: "13", workerId: "0", elapsedNs: "0", arrangementSize: "0" }, + { id: "13", workerId: "1", elapsedNs: "1000", arrangementSize: "0" }, + ]); + const region = s.nodes.get(nodeIdOf([5, 1]))!; + expect(region.transitiveSkew.cpuSkew).toBeCloseTo(2); + }); + + it("ignores an operator's own skew if its work is negligible next to the busiest node in the dataflow, however extreme its raw ratio, and keeps it from polluting an ancestor's rolled-up skew", () => { + // Join is genuinely busy (1ms) and perfectly balanced. Map's own + // elapsed is 1ns (0.0001% of Join's magnitude), but its per-worker + // split (1/0/0/0) would post the theoretical 4-worker ceiling ratio + // if taken at face value, despite contributing nothing real. + const skewedOps = OPS.map((o) => + o.id === "12" + ? { ...o, elapsedNs: "1000000" } + : o.id === "13" + ? { ...o, elapsedNs: "1" } + : o, + ); + const s = buildDataflowStructure(skewedOps, CHANNELS, LIR_SPANS, [ + { id: "12", workerId: "0", elapsedNs: "500000", arrangementSize: "0" }, + { id: "12", workerId: "1", elapsedNs: "500000", arrangementSize: "0" }, + { id: "13", workerId: "0", elapsedNs: "1", arrangementSize: "0" }, + { id: "13", workerId: "1", elapsedNs: "0", arrangementSize: "0" }, + { id: "13", workerId: "2", elapsedNs: "0", arrangementSize: "0" }, + { id: "13", workerId: "3", elapsedNs: "0", arrangementSize: "0" }, + ]); + const map = s.nodes.get(nodeIdOf([5, 1, 2]))!; + expect(map.ownSkew.cpuSkew).toEqual(0); + // The region's worst real skew is Join's honest 1.0 (perfectly + // even), not Map's suppressed near-ceiling ratio. + const region = s.nodes.get(nodeIdOf([5, 1]))!; + expect(region.transitiveSkew.cpuSkew).toEqual(1); + }); + + it("keeps the real own skew available unfiltered, for display, even when the magnitude floor suppresses it for rollup", () => { + const skewedOps = OPS.map((o) => + o.id === "12" + ? { ...o, elapsedNs: "1000000" } + : o.id === "13" + ? { ...o, elapsedNs: "1" } + : o, + ); + const s = buildDataflowStructure(skewedOps, CHANNELS, LIR_SPANS, [ + { id: "12", workerId: "0", elapsedNs: "500000", arrangementSize: "0" }, + { id: "12", workerId: "1", elapsedNs: "500000", arrangementSize: "0" }, + { id: "13", workerId: "0", elapsedNs: "1", arrangementSize: "0" }, + { id: "13", workerId: "1", elapsedNs: "0", arrangementSize: "0" }, + { id: "13", workerId: "2", elapsedNs: "0", arrangementSize: "0" }, + { id: "13", workerId: "3", elapsedNs: "0", arrangementSize: "0" }, + ]); + const map = s.nodes.get(nodeIdOf([5, 1, 2]))!; + expect(map.ownSkew.cpuSkew).toEqual(0); // suppressed, for rollup safety + expect(map.ownSkewRaw.cpuSkew).toBeCloseTo(4); // 1 / ((1+0+0+0)/4) + }); + }); +}); + +describe("deriveVisibleGraph", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS); + const regionId = nodeIdOf([5, 1]); + + it("shows the region collapsed and the sibling leaf from the root view", () => { + const g = deriveVisibleGraph(s, s.root); + expect(g.nodes.map((n) => [n.id, n.kind])).toEqual([ + [regionId, "region"], + [nodeIdOf([5, 2]), "operator"], + ]); + // channel 3 region -> sink survives, channels 1 and 2 are internal to + // the region and never surface at this level. + expect(g.edges).toEqual([ + { + id: `${regionId}=>${nodeIdOf([5, 2])}`, + source: regionId, + target: nodeIdOf([5, 2]), + messagesSent: 5n, + batchesSent: 2n, + channelTypes: ["batches"], + // channel 3 touches the region at exactly its own address (port 0), + // so this edge's source can drill straight to that inner port. + sourceLandings: [ + { + address: [5, 1], + label: "output 0", + messagesSent: 5n, + batchesSent: 2n, + channelTypes: ["batches"], + peerPortId: `${regionId}:out:0`, + }, + ], + targetLandings: [], + }, + ]); + const region = g.nodes[0]; + expect(region.stats).toEqual(s.nodes.get(regionId)!.transitive); + expect(region.childCount).toEqual(2); + // `stats` collapses to the subtree total for a quick canvas-level read, + // but the detail panel needs the region's own activity (5ns) kept + // distinct from that total (5+7+1=13ns), or a region's own dispatch + // cost becomes invisible next to its children's. + expect(region.own).toEqual(s.nodes.get(regionId)!.own); + expect(region.own!.elapsedNs).toEqual(5n); + expect(region.transitive!.elapsedNs).toEqual(13n); + }); + + it("drilling into the region shows its children and an inbound port", () => { + const g = deriveVisibleGraph(s, regionId); + expect(g.nodes.map((n) => [n.id, n.kind]).sort()).toEqual( + [ + [nodeIdOf([5, 1, 1]), "operator"], + [nodeIdOf([5, 1, 2]), "operator"], + [`${regionId}:in:0`, "port"], + [`${regionId}:out:0`, "port"], + ].sort(), + ); + const inPort = g.nodes.find((n) => n.id === `${regionId}:in:0`)!; + expect(inPort.label).toEqual("input 0"); + expect(inPort.operatorId).toBeNull(); + // channel 1 (port fan-in) and channel 2 (internal) both survive. + expect(g.edges.map((e) => e.id).sort()).toEqual( + [ + `${regionId}:in:0=>${nodeIdOf([5, 1, 1])}`, + `${nodeIdOf([5, 1, 1])}=>${nodeIdOf([5, 1, 2])}`, + ].sort(), + ); + // channel 3's outer half (region's own address -> sibling [5,2]) has no + // representable target from inside the region (the sibling isn't part + // of this view), so its edge is dropped, but the "out" port it resolved + // to registers anyway: the box's boundary traffic isn't silently erased. + expect( + g.edges.some( + (e) => + e.source === `${regionId}:out:0` || e.target === `${regionId}:out:0`, + ), + ).toBe(false); + // ...and the sibling it can't show directly is recorded as a jump target. + const outPort = g.nodes.find((n) => n.id === `${regionId}:out:0`)!; + expect(outPort.peers).toEqual([ + { + address: [5, 2], + label: "Sink", + messagesSent: 5n, + batchesSent: 2n, + channelTypes: ["batches"], + // Sink is a leaf: nothing to drill into, so the peer box itself is + // the target. + peerPortId: null, + }, + ]); + // channel 1's other end (Join) is already visible in this same view, so + // the "in" port needs no jump link for it. + expect(inPort.peers).toEqual([]); + }); + + it("aggregates parallel channels between the same visible pair", () => { + const extra = buildDataflowStructure( + OPS, + [ + ...CHANNELS, + { + id: "4", + fromOperatorAddress: ["5", "1"], + fromPort: "1", + toOperatorAddress: ["5", "2"], + toPort: "1", + messagesSent: "7", + batchesSent: "1", + channelType: "rows", + }, + ], + [], + ); + const g = deriveVisibleGraph(extra, extra.root); + const e = g.edges.find((edge) => edge.target === nodeIdOf([5, 2]))!; + expect(e.messagesSent).toEqual(12n); + expect(e.channelTypes).toEqual(["batches", "rows"]); + }); + + it("routes an external-to-region channel through a port node, keyed by port number", () => { + // Materialize logs a scope boundary crossing as two channels sharing a + // port number but addressed differently: the outer half (external + // producer/consumer <-> scope) targets the scope's OWN operator address + // directly (no [..., 0] suffix), the same address the region's own + // operator row uses. Verified against live introspection data: an + // external channel `{5,4}->{5,2}` (to_port=0) pairs with the internal + // fan-out channel `{5,2,0}->{5,2,1}` (from_port=0). + const withExternal = buildDataflowStructure( + [ + ...OPS, + { + id: "20", + address: ["5", "3"], + name: "External", + arrangementRecords: "0", + arrangementSize: "0", + elapsedNs: "1", + }, + ], + [ + ...CHANNELS, + { + id: "98", + fromOperatorAddress: ["5", "3"], + fromPort: "0", + toOperatorAddress: ["5", "1"], + toPort: "0", + messagesSent: "9", + batchesSent: "3", + channelType: "rows", + }, + ], + LIR_SPANS, + ); + const externalId = nodeIdOf([5, 3]); + const portId = `${regionId}:in:0`; + + // Viewed from inside the region, the crossing's inner half (channel 1) + // already produced this port; the outer half's external source isn't + // part of this view, so it contributes no new edge or node. + const drilledIn = deriveVisibleGraph(withExternal, regionId); + const inPort = drilledIn.nodes.find((n) => n.id === portId)!; + expect(inPort.kind).toEqual("port"); + expect( + drilledIn.edges.some( + (e) => e.source === externalId || e.target === externalId, + ), + ).toBe(false); + // ...but the external source is still reachable as a jump target off + // the port itself. + expect(inPort.peers).toEqual([ + { + address: [5, 3], + label: "External", + messagesSent: 9n, + batchesSent: 3n, + channelTypes: ["rows"], + // External is a leaf too. + peerPortId: null, + }, + ]); + + // From the root, the crossing lands on the region's own (collapsed) + // node, matching how every other boundary crossing into a collapsed + // region behaves: the whole region is one box. + const root = deriveVisibleGraph(withExternal, withExternal.root); + const rootEdge = root.edges.find((e) => e.source === externalId)!; + expect(rootEdge.target).toEqual(regionId); + }); + + it("fans a port out to multiple peers (one output can feed several inputs)", () => { + const fannedOut = buildDataflowStructure( + [ + ...OPS, + { + id: "21", + address: ["5", "4"], + name: "Sink2", + arrangementRecords: "0", + arrangementSize: "0", + elapsedNs: "1", + }, + ], + [ + ...CHANNELS, + // Same output (region's port 0) also feeds a second sibling. + { + id: "5", + fromOperatorAddress: ["5", "1"], + fromPort: "0", + toOperatorAddress: ["5", "4"], + toPort: "0", + messagesSent: "2", + batchesSent: "1", + channelType: "rows", + }, + ], + [], + ); + const g = deriveVisibleGraph(fannedOut, regionId); + const outPort = g.nodes.find((n) => n.id === `${regionId}:out:0`)!; + expect(outPort.peers.map((p) => p.label).sort()).toEqual(["Sink", "Sink2"]); + expect(outPort.peers.find((p) => p.label === "Sink2")).toEqual({ + address: [5, 4], + label: "Sink2", + messagesSent: 2n, + batchesSent: 1n, + channelTypes: ["rows"], + peerPortId: null, + }); + }); + + it("records which of a region's own ports a merged edge's real channels land on", () => { + // Timely gates every scope crossing through the box's own address, one + // hop at a time (verified against live introspection data: real + // channels never skip straight past a region to something deeper + // inside it), so a channel touching RegionA always does so at exactly + // address [8,1], never deeper. Two of RegionA's own output ports (0 and + // 1) both happen to feed the same downstream leaf, Sink: from the + // root, both collapse onto the same visible edge (RegionA -> Sink), + // discarding which of RegionA's ports is which unless something + // records it separately. + const withPorts = buildDataflowStructure( + [ + { ...OPS[0], id: "60", address: ["8"], name: "Dataflow" }, + { ...OPS[0], id: "61", address: ["8", "1"], name: "RegionA" }, + { ...OPS[0], id: "62", address: ["8", "1", "1"], name: "LeafInner" }, + { ...OPS[0], id: "64", address: ["8", "2"], name: "Sink" }, + ], + [ + { + id: "50", + fromOperatorAddress: ["8", "1"], + fromPort: "0", + toOperatorAddress: ["8", "2"], + toPort: "0", + messagesSent: "3", + batchesSent: "1", + channelType: "rows", + }, + { + id: "51", + fromOperatorAddress: ["8", "1"], + fromPort: "1", + toOperatorAddress: ["8", "2"], + toPort: "0", + messagesSent: "2", + batchesSent: "1", + channelType: "rows", + }, + ], + [], + ); + const g = deriveVisibleGraph(withPorts, withPorts.root); + const regionAId = nodeIdOf([8, 1]); + const edge = g.edges.find( + (e) => e.id === `${regionAId}=>${nodeIdOf([8, 2])}`, + )!; + expect( + edge.sourceLandings.sort((a, b) => a.label.localeCompare(b.label)), + ).toEqual([ + { + address: [8, 1], + label: "output 0", + messagesSent: 3n, + batchesSent: 1n, + channelTypes: ["rows"], + peerPortId: `${regionAId}:out:0`, + }, + { + address: [8, 1], + label: "output 1", + messagesSent: 2n, + batchesSent: 1n, + channelTypes: ["rows"], + peerPortId: `${regionAId}:out:1`, + }, + ]); + // Sink is a leaf: it has no inner ports of its own to land on. + expect(edge.targetLandings).toEqual([]); + }); + + it("resolves a region peer's own port, so a jump can drill straight to it", () => { + // Two sibling regions, A's output feeding B's input directly (not + // nested in each other): from inside A, B is a region peer with a + // real port to land on inside B; from inside B, A is the same. + const s2 = buildDataflowStructure( + [ + { ...OPS[0], id: "1", address: ["9"], name: "Dataflow" }, + { ...OPS[0], id: "2", address: ["9", "1"], name: "RegionA" }, + { ...OPS[0], id: "3", address: ["9", "1", "1"], name: "LeafA" }, + { ...OPS[0], id: "4", address: ["9", "2"], name: "RegionB" }, + { ...OPS[0], id: "5", address: ["9", "2", "1"], name: "LeafB" }, + ], + [ + { + id: "1", + fromOperatorAddress: ["9", "1"], + fromPort: "3", + toOperatorAddress: ["9", "2"], + toPort: "7", + messagesSent: "1", + batchesSent: "1", + channelType: "rows", + }, + ], + [], + ); + const aId = nodeIdOf([9, 1]); + const bId = nodeIdOf([9, 2]); + + const fromA = deriveVisibleGraph(s2, aId); + const outPort = fromA.nodes.find((n) => n.id === `${aId}:out:3`)!; + expect(outPort.peers).toEqual([ + expect.objectContaining({ address: [9, 2], peerPortId: `${bId}:in:7` }), + ]); + + const fromB = deriveVisibleGraph(s2, bId); + const inPort = fromB.nodes.find((n) => n.id === `${bId}:in:7`)!; + expect(inPort.peers).toEqual([ + expect.objectContaining({ address: [9, 1], peerPortId: `${aId}:out:3` }), + ]); + }); + + it("resolves a peer past a nested scope's boundary pseudo-vertex", () => { + // A scope's own boundary is logged as a pseudo-vertex [scope, 0], which + // never gets an operator row (only real children do). When a region is + // nested two scopes deep, its outer boundary channel targets its + // *enclosing* scope's pseudo-vertex directly (mirroring the [5,1,0] + // pattern in CHANNELS one level up), e.g. [6,1,0] -> [6,1,1] for a region + // at [6,1,1] inside [6,1]. [6,1,0] has no operator row, so naively + // resolving the peer address finds nothing; the fix is to recognize the + // pseudo-vertex and peel back to its enclosing scope [6,1], which does. + const nested = buildDataflowStructure( + [ + { ...OPS[0], id: "30", address: ["6"], name: "Dataflow" }, + { ...OPS[0], id: "31", address: ["6", "1"], name: "RegionOuter" }, + { ...OPS[0], id: "32", address: ["6", "1", "1"], name: "RegionInner" }, + { + ...OPS[0], + id: "33", + address: ["6", "1", "1", "1"], + name: "LeafInner", + }, + ], + [ + { + id: "40", + fromOperatorAddress: ["6", "1", "0"], + fromPort: "0", + toOperatorAddress: ["6", "1", "1"], + toPort: "0", + messagesSent: "4", + batchesSent: "1", + channelType: "rows", + }, + { + id: "41", + fromOperatorAddress: ["6", "1", "1", "0"], + fromPort: "0", + toOperatorAddress: ["6", "1", "1", "1"], + toPort: "0", + messagesSent: "4", + batchesSent: "1", + channelType: "rows", + }, + ], + [], + ); + const outerId = nodeIdOf([6, 1]); + const innerId = nodeIdOf([6, 1, 1]); + + const fromInner = deriveVisibleGraph(nested, innerId); + const inPort = fromInner.nodes.find((n) => n.id === `${innerId}:in:0`)!; + expect(inPort.peers).toEqual([ + expect.objectContaining({ + address: [6, 1], + label: "RegionOuter", + peerPortId: `${outerId}:in:0`, + }), + ]); + }); + + it("drops channels that reference an address with no operator row", () => { + // Real dataflows can log a channel touching a scope address that never + // got its own operator row (e.g. an elided region). [5, 5] does not + // appear in OPS. + const withDangling = buildDataflowStructure( + OPS, + [ + ...CHANNELS, + { + id: "99", + fromOperatorAddress: ["5", "5"], + fromPort: "0", + toOperatorAddress: ["5", "2"], + toPort: "0", + messagesSent: "1", + batchesSent: "1", + channelType: "rows", + }, + ], + LIR_SPANS, + ); + expect(() => + deriveVisibleGraph(withDangling, withDangling.root), + ).not.toThrow(); + const g = deriveVisibleGraph(withDangling, withDangling.root); + const nodeIds = new Set(g.nodes.map((n) => n.id)); + for (const e of g.edges) { + expect(nodeIds.has(e.source)).toBe(true); + expect(nodeIds.has(e.target)).toBe(true); + } + }); +}); + +describe("commonAncestorScope / representativeInView", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS); + const regionId = nodeIdOf([5, 1]); + + it("a single leaf address navigates to its immediate parent scope", () => { + expect(commonAncestorScope(s, [[5, 1, 1]])).toEqual(regionId); + expect(representativeInView([5, 1, 1], [5, 1])).toEqual( + nodeIdOf([5, 1, 1]), + ); + }); + + it("backs off past an address that IS the raw common prefix", () => { + // [5,1] is itself a prefix of [5,1,1], so it can't be the view root + // (nothing could highlight it as its own child); back off to its parent. + expect( + commonAncestorScope(s, [ + [5, 1], + [5, 1, 1], + ]), + ).toEqual(s.root); + // From the root, [5,1] represents itself and [5,1,1] rolls up into it. + expect(representativeInView([5, 1], [5])).toEqual(regionId); + expect(representativeInView([5, 1, 1], [5])).toEqual(regionId); + }); + + it("returns null for an address outside the view root's subtree", () => { + expect(representativeInView([5, 2], [5, 1])).toBeNull(); + }); +}); + +describe("allSearchMatches / subtreeSearchMatches", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS); + const regionId = nodeIdOf([5, 1]); + + it("allSearchMatches finds matches anywhere, in address order, excluding the root", () => { + expect(allSearchMatches(s, "i").map((n) => n.id)).toEqual([ + regionId, // "Region" + nodeIdOf([5, 1, 1]), // "Join" + nodeIdOf([5, 2]), // "Sink" + ]); + expect(allSearchMatches(s, "nonexistent")).toEqual([]); + expect(allSearchMatches(s, "")).toEqual([]); + }); + + it("subtreeSearchMatches flags a region as containing a match without matching itself", () => { + const { matches, containsMatch } = subtreeSearchMatches(s, "join"); + expect(matches.has(nodeIdOf([5, 1, 1]))).toBe(true); + expect(matches.has(regionId)).toBe(false); + // The region doesn't match "join" itself, but a descendant does. + expect(containsMatch.has(regionId)).toBe(true); + expect(containsMatch.has(nodeIdOf([5, 2]))).toBe(false); + }); +}); + +describe("decorateGraph", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS); + const regionId = nodeIdOf([5, 1]); + const root = deriveVisibleGraph(s, s.root); + const drilledIn = deriveVisibleGraph(s, regionId); + const heat = (t: number) => `heat(${t.toFixed(2)})`; + + it("a region containing a match (but not matching itself) stays undimmed and uncounted", () => { + const searchInfo = subtreeSearchMatches(s, "join"); + const d = decorateGraph( + root, + { ...DEFAULT_FILTERS, search: "join" }, + heat, + searchInfo, + 1, + ); + // "Join" is inside the region, not visible at the root; the region + // doesn't match "join" by name either, so it's neither a listed match + // nor dimmed, while the sibling sink (no match anywhere inside it) is. + expect(d.searchMatches).toEqual([]); + expect(d.dimmedNodeIds.has(regionId)).toBe(false); + expect(d.dimmedNodeIds.has(nodeIdOf([5, 2]))).toBe(true); + }); + + it("a directly visible match is listed and left undimmed", () => { + const searchInfo = subtreeSearchMatches(s, "join"); + const d = decorateGraph( + drilledIn, + { ...DEFAULT_FILTERS, search: "join" }, + heat, + searchInfo, + 1, + ); + expect(d.searchMatches).toEqual([nodeIdOf([5, 1, 1])]); + expect(d.dimmedNodeIds.has(nodeIdOf([5, 1, 1]))).toBe(false); + expect(d.dimmedNodeIds.has(nodeIdOf([5, 1, 2]))).toBe(true); + }); + + it("hideIdle hides zero-activity operators and zero-message edges, never regions or ports", () => { + const d = decorateGraph( + root, + { ...DEFAULT_FILTERS, hideIdle: true }, + heat, + null, + 1, + ); + // every fixture operator has elapsed > 0 except none; hide check via a synthetic idle op + const idle = buildDataflowStructure( + [ + ...OPS, + { + id: "15", + address: ["5", "3"], + name: "Idle", + arrangementRecords: "0", + arrangementSize: "0", + elapsedNs: "0", + }, + ], + CHANNELS, + [], + ); + const d2 = decorateGraph( + deriveVisibleGraph(idle, idle.root), + { ...DEFAULT_FILTERS, hideIdle: true }, + heat, + null, + 1, + ); + expect(d2.hiddenNodeIds.has(nodeIdOf([5, 3]))).toBe(true); + expect(d2.hiddenNodeIds.has(nodeIdOf([5, 1]))).toBe(false); + expect(d.hiddenEdgeIds.size).toBe(0); // all fixture edges have messages + }); + + it("heatmap colors by transitive metric and dims below threshold", () => { + const d = decorateGraph( + root, + { ...DEFAULT_FILTERS, heatmap: "elapsed", heatmapThreshold: 0.5 }, + heat, + null, + 1, + ); + // max transitive elapsed among visible non-ports is the region (13n) + expect(d.nodeColors.get(regionId)).toEqual("heat(1.00)"); + expect(d.dimmedNodeIds.has(nodeIdOf([5, 2]))).toBe(true); // 2/13 < 0.5 + }); + + it("heatmap with all-zero metric sets no colors", () => { + const zero = buildDataflowStructure( + OPS.map((o) => ({ ...o, elapsedNs: "0" })), + CHANNELS, + [], + ); + const d = decorateGraph( + deriveVisibleGraph(zero, zero.root), + { ...DEFAULT_FILTERS, heatmap: "elapsed" }, + heat, + null, + 1, + ); + expect(d.nodeColors.size).toBe(0); + }); + + it("cpuSkew/memorySkew heatmap colors by transitive skew", () => { + const skewed = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS, [ + { id: "12", workerId: "0", elapsedNs: "10", arrangementSize: "1000" }, + { id: "12", workerId: "1", elapsedNs: "30", arrangementSize: "3000" }, + ]); + const skewedRoot = deriveVisibleGraph(skewed, skewed.root); + const d = decorateGraph( + skewedRoot, + { ...DEFAULT_FILTERS, heatmap: "cpuSkew" }, + heat, + null, + 4, + ); + // The region's subtree (containing Join) is the only skewed thing (1.5 + // ratio); the sibling sink has no per-worker data at all, so it's the + // coolest, and neither depends on the other being in view. + expect(d.nodeColors.get(regionId)).toEqual( + `heat(${(Math.log2(1.5) / Math.log2(4)).toFixed(2)})`, + ); + expect(d.nodeColors.get(nodeIdOf([5, 2]))).toEqual("heat(0.00)"); + }); + + it("scheduleSkew heatmap colors by transitive schedule-count skew, same as cpu/memory", () => { + const skewed = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS, [ + { + id: "12", + workerId: "0", + elapsedNs: null, + arrangementSize: null, + scheduleCount: "100", + }, + { + id: "12", + workerId: "1", + elapsedNs: null, + arrangementSize: null, + scheduleCount: "300", + }, + ]); + const skewedRoot = deriveVisibleGraph(skewed, skewed.root); + const d = decorateGraph( + skewedRoot, + { ...DEFAULT_FILTERS, heatmap: "scheduleSkew" }, + heat, + null, + 4, + ); + expect(d.nodeColors.get(regionId)).toEqual( + `heat(${(Math.log2(1.5) / Math.log2(4)).toFixed(2)})`, + ); + expect(d.nodeColors.get(nodeIdOf([5, 2]))).toEqual("heat(0.00)"); + }); + + it("skew heatmap floor is exactly ratio 1, ceiling is exactly the worker count", () => { + const perfectlyEven = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS, [ + { id: "12", workerId: "0", elapsedNs: "100", arrangementSize: "0" }, + { id: "12", workerId: "1", elapsedNs: "100", arrangementSize: "0" }, + ]); + const dAtFloor = decorateGraph( + deriveVisibleGraph(perfectlyEven, regionId), + { ...DEFAULT_FILTERS, heatmap: "cpuSkew" }, + heat, + null, + 4, + ); + expect(dAtFloor.nodeColors.get(nodeIdOf([5, 1, 1]))).toEqual("heat(0.00)"); + + // One worker does everything, the other three do nothing (but are + // present, not absent): ratio hits exactly the 4-worker ceiling + // (400 / (400/4) = 4), so this must read as fully hot. + const atCeiling = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS, [ + { id: "12", workerId: "0", elapsedNs: "400", arrangementSize: "0" }, + { id: "12", workerId: "1", elapsedNs: "0", arrangementSize: "0" }, + { id: "12", workerId: "2", elapsedNs: "0", arrangementSize: "0" }, + { id: "12", workerId: "3", elapsedNs: "0", arrangementSize: "0" }, + ]); + const dAtCeiling = decorateGraph( + deriveVisibleGraph(atCeiling, regionId), + { ...DEFAULT_FILTERS, heatmap: "cpuSkew" }, + heat, + null, + 4, + ); + expect(dAtCeiling.nodeColors.get(nodeIdOf([5, 1, 1]))).toEqual( + "heat(1.00)", + ); + }); + + it("skew heatmap color for a fixed ratio does not depend on what else is in view", () => { + const buildWithMapData = (mapRows: PerWorkerStatRow[]) => + decorateGraph( + deriveVisibleGraph( + buildDataflowStructure(OPS, CHANNELS, LIR_SPANS, [ + { id: "12", workerId: "0", elapsedNs: "100", arrangementSize: "0" }, + { id: "12", workerId: "1", elapsedNs: "200", arrangementSize: "0" }, + ...mapRows, + ]), + regionId, + ), + { ...DEFAULT_FILTERS, heatmap: "cpuSkew" }, + heat, + null, + 4, + ); + const mildCompanion = buildWithMapData([ + { id: "13", workerId: "0", elapsedNs: "100", arrangementSize: "0" }, + { id: "13", workerId: "1", elapsedNs: "105", arrangementSize: "0" }, + ]); + const extremeCompanion = buildWithMapData([ + { id: "13", workerId: "0", elapsedNs: "1", arrangementSize: "0" }, + { id: "13", workerId: "1", elapsedNs: "1000", arrangementSize: "0" }, + ]); + // Join's own ratio (200/150) never changes; its color must be identical + // regardless of Map's ratio, since the domain is fixed at + // [1, workerCount] rather than derived from whatever else is in view. + expect(mildCompanion.nodeColors.get(nodeIdOf([5, 1, 1]))).toEqual( + extremeCompanion.nodeColors.get(nodeIdOf([5, 1, 1])), + ); + }); + + // A negligible-magnitude operator's raw skew is suppressed to the "no + // data" sentinel at the source (buildDataflowStructure's ownSkew, see + // dataflowGraph.test.ts's "skew" describe block) rather than filtered + // here, so decorateGraph itself no longer needs, or has, any + // magnitude-aware logic of its own to test. +}); + +describe("lirIndex", () => { + it("buckets member nodes by export/lir id", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS); + const index = lirIndex(s); + expect([...index.keys()]).toEqual(["u42/1"]); + // span 11..13 covers operator ids 11 ([5,1]) and 12 ([5,1,1]) + expect(index.get("u42/1")!.memberIds.sort()).toEqual( + [nodeIdOf([5, 1]), nodeIdOf([5, 1, 1])].sort(), + ); + }); +}); + +describe("lirTree", () => { + it("nests children under their parent via parentLirId, siblings descending by lir id", () => { + const ops: OperatorRow[] = [ + { + id: "1", + address: ["9"], + name: "Dataflow", + arrangementRecords: "0", + arrangementSize: "0", + elapsedNs: "0", + }, + { + id: "2", + address: ["9", "1"], + name: "OpA", + arrangementRecords: "100", + arrangementSize: "0", + elapsedNs: "10", + }, + { + id: "3", + address: ["9", "2"], + name: "OpB", + arrangementRecords: "200", + arrangementSize: "0", + elapsedNs: "20", + }, + ]; + const spans: LirSpanRow[] = [ + { + exportId: "u1", + lirId: "10", + parentLirId: null, + nesting: 0, + operator: "Root", + operatorIdStart: "2", + operatorIdEnd: "4", + }, + { + exportId: "u1", + lirId: "11", + parentLirId: "10", + nesting: 1, + operator: "OpA", + operatorIdStart: "2", + operatorIdEnd: "3", + }, + { + exportId: "u1", + lirId: "12", + parentLirId: "10", + nesting: 1, + operator: "OpB", + operatorIdStart: "3", + operatorIdEnd: "4", + }, + ]; + const s = buildDataflowStructure(ops, [], spans); + const tree = lirTree(s, lirIndex(s)); + const roots = tree.get("u1")!; + expect(roots.map((n) => n.key)).toEqual(["u1/10"]); + expect(roots[0].children.map((n) => n.key)).toEqual(["u1/12", "u1/11"]); + expect(roots[0].memberIds.sort()).toEqual( + [nodeIdOf([9, 1]), nodeIdOf([9, 2])].sort(), + ); + expect(roots[0].children.find((n) => n.key === "u1/11")!.memberIds).toEqual( + [nodeIdOf([9, 1])], + ); + }); + + it("sums own stats over members, matching EXPLAIN ANALYZE's semantics: a parent's summary includes its children's", () => { + const ops: OperatorRow[] = [ + { + id: "1", + address: ["9"], + name: "Dataflow", + arrangementRecords: "0", + arrangementSize: "0", + elapsedNs: "0", + }, + { + id: "2", + address: ["9", "1"], + name: "OpA", + arrangementRecords: "100", + arrangementSize: "0", + elapsedNs: "10", + }, + { + id: "3", + address: ["9", "2"], + name: "OpB", + arrangementRecords: "200", + arrangementSize: "0", + elapsedNs: "20", + }, + ]; + const spans: LirSpanRow[] = [ + { + exportId: "u1", + lirId: "10", + parentLirId: null, + nesting: 0, + operator: "Root", + operatorIdStart: "2", + operatorIdEnd: "4", + }, + { + exportId: "u1", + lirId: "11", + parentLirId: "10", + nesting: 1, + operator: "OpA", + operatorIdStart: "2", + operatorIdEnd: "3", + }, + ]; + const s = buildDataflowStructure(ops, [], spans); + const tree = lirTree(s, lirIndex(s)); + const root = tree.get("u1")![0]; + expect(root.summary).toEqual({ + arrangementRecords: 300n, + arrangementSize: 0n, + elapsedNs: 30n, + scheduleCount: 0n, + }); + expect(root.children[0].summary).toEqual({ + arrangementRecords: 100n, + arrangementSize: 0n, + elapsedNs: 10n, + scheduleCount: 0n, + }); + }); +}); + +describe("rerouteHiddenNodes", () => { + const node = (id: string): VisibleNode => ({ + id, + kind: "operator", + label: id, + stats: null, + transitive: null, + own: null, + ownSkew: null, + transitiveSkew: null, + overheadNs: null, + childCount: 0, + lir: [], + address: null, + operatorId: null, + peers: [], + }); + const edge = ( + id: string, + source: string, + target: string, + messagesSent = 0n, + batchesSent = 0n, + channelTypes: string[] = [], + ): VisibleEdge => ({ + id, + source, + target, + messagesSent, + batchesSent, + channelTypes, + sourceLandings: [], + targetLandings: [], + }); + + it("returns the graph unchanged when nothing is hidden", () => { + const g = { + nodes: [node("a"), node("b")], + edges: [edge("a=>b", "a", "b")], + }; + expect(rerouteHiddenNodes(g, new Set())).toBe(g); + }); + + it("splices a pass-through edge across a single hidden node", () => { + const g = { + nodes: [node("a"), node("b"), node("c")], + edges: [ + edge("a=>b", "a", "b", 3n, 1n, ["rows"]), + edge("b=>c", "b", "c", 3n, 1n, ["rows"]), + ], + }; + const r = rerouteHiddenNodes(g, new Set(["b"])); + expect(r.nodes.map((n) => n.id)).toEqual(["a", "c"]); + expect(r.edges).toEqual([ + { + id: "a=>c", + source: "a", + target: "c", + messagesSent: 6n, + batchesSent: 2n, + channelTypes: ["rows"], + sourceLandings: [], + targetLandings: [], + }, + ]); + }); + + it("fans out through a hidden node with multiple successors", () => { + const g = { + nodes: [node("a"), node("b"), node("c"), node("d")], + edges: [ + edge("a=>b", "a", "b"), + edge("b=>c", "b", "c"), + edge("b=>d", "b", "d"), + ], + }; + const r = rerouteHiddenNodes(g, new Set(["b"])); + expect(new Set(r.edges.map((e) => e.id))).toEqual( + new Set(["a=>c", "a=>d"]), + ); + }); + + it("fans in from multiple sources through a hidden node", () => { + const g = { + nodes: [node("a1"), node("a2"), node("b"), node("c")], + edges: [ + edge("a1=>b", "a1", "b"), + edge("a2=>b", "a2", "b"), + edge("b=>c", "b", "c"), + ], + }; + const r = rerouteHiddenNodes(g, new Set(["b"])); + expect(new Set(r.edges.map((e) => e.id))).toEqual( + new Set(["a1=>c", "a2=>c"]), + ); + }); + + it("does not hang on a hidden feedback cycle and still bridges past it", () => { + // Timely feedback loops mean two hidden nodes can point at each other. + const g = { + nodes: [node("a"), node("l"), node("m"), node("c")], + edges: [ + edge("a=>l", "a", "l"), + edge("l=>m", "l", "m"), + edge("m=>l", "m", "l"), // cycle + edge("m=>c", "m", "c"), + ], + }; + const r = rerouteHiddenNodes(g, new Set(["l", "m"])); + expect(r.nodes.map((n) => n.id)).toEqual(["a", "c"]); + expect(r.edges.map((e) => e.id)).toEqual(["a=>c"]); + }); + + it("drops an entirely hidden component with no visible endpoint", () => { + const g = { + nodes: [node("a"), node("b")], + edges: [edge("a=>b", "a", "b")], + }; + const r = rerouteHiddenNodes(g, new Set(["a", "b"])); + expect(r.nodes).toEqual([]); + expect(r.edges).toEqual([]); + }); + + it("keeps ordinary edges between two surviving visible nodes untouched", () => { + const g = { + nodes: [node("a"), node("b"), node("hidden")], + edges: [edge("a=>b", "a", "b", 5n, 1n, ["rows"])], + }; + const r = rerouteHiddenNodes(g, new Set(["hidden"])); + expect(r.edges).toEqual(g.edges); + }); +}); + +describe("groupByLir", () => { + const node = ( + id: string, + operatorId: number, + lir: LirInfo[] = [], + ): VisibleNode => ({ + id, + kind: "operator", + label: id, + stats: null, + transitive: null, + own: null, + ownSkew: null, + transitiveSkew: null, + overheadNs: null, + childCount: 0, + lir, + address: null, + operatorId: BigInt(operatorId), + peers: [], + }); + const port = (id: string): VisibleNode => ({ + id, + kind: "port", + label: id, + stats: null, + transitive: null, + own: null, + ownSkew: null, + transitiveSkew: null, + overheadNs: null, + childCount: 0, + lir: [], + address: null, + operatorId: null, + peers: [], + }); + const span = ( + exportId: string, + lirId: string, + nesting: number, + parentLirId: string | null, + ): LirInfo => ({ exportId, lirId, parentLirId, nesting, operator: lirId }); + + it("groups nothing when no node has lir data", () => { + const g = groupByLir([node("a", 1), node("b", 2)]); + expect(g.groups).toEqual([]); + expect(g.parentOf.size).toEqual(0); + }); + + it("groups a flat run under one lir id", () => { + const s = span("u1", "10", 0, null); + const g = groupByLir([ + node("a", 1, [s]), + node("b", 2, [s]), + node("c", 3, [s]), + ]); + expect(g.groups.map((x) => x.id)).toEqual(["u1/10"]); + expect(g.parentOf.get("a")).toEqual("u1/10"); + expect(g.parentOf.get("b")).toEqual("u1/10"); + expect(g.parentOf.get("c")).toEqual("u1/10"); + }); + + it("nests an inner span inside an outer span", () => { + const outer = span("u1", "10", 0, null); + const inner = span("u1", "11", 1, "10"); + const g = groupByLir([ + node("a", 1, [outer]), + node("b", 2, [outer, inner]), + node("c", 3, [outer, inner]), + node("d", 4, [outer]), + ]); + expect(g.groups.map((x) => x.id)).toEqual(["u1/10", "u1/11"]); + expect(g.parentOf.get("u1/11")).toEqual("u1/10"); + expect(g.parentOf.get("a")).toEqual("u1/10"); + expect(g.parentOf.get("b")).toEqual("u1/11"); + expect(g.parentOf.get("c")).toEqual("u1/11"); + expect(g.parentOf.get("d")).toEqual("u1/10"); + }); + + it("leaves ports and lir-less nodes ungrouped alongside a group", () => { + const s = span("u1", "10", 0, null); + const g = groupByLir([node("a", 1, [s]), port("p"), node("b", 2)]); + expect(g.groups.map((x) => x.id)).toEqual(["u1/10"]); + expect(g.parentOf.get("a")).toEqual("u1/10"); + expect(g.parentOf.has("p")).toEqual(false); + expect(g.parentOf.has("b")).toEqual(false); + }); + + it("picks the lowest exportId, leaving other-export-only nodes ungrouped", () => { + const sA = span("u1", "5", 0, null); + const sB = span("u2", "3", 0, null); + const g = groupByLir([ + node("a", 1, [sA]), + node("b", 2, [sB]), + node("c", 3, [sA, sB]), + ]); + expect(g.groups.map((x) => x.id)).toEqual(["u1/5"]); + expect(g.parentOf.get("a")).toEqual("u1/5"); + expect(g.parentOf.has("b")).toEqual(false); + expect(g.parentOf.get("c")).toEqual("u1/5"); + }); + + it("sorts input by operatorId regardless of array order", () => { + const outer = span("u1", "10", 0, null); + const inner = span("u1", "11", 1, "10"); + // Deliberately out of operatorId order. + const g = groupByLir([ + node("d", 4, [outer]), + node("b", 2, [outer, inner]), + node("a", 1, [outer]), + node("c", 3, [outer, inner]), + ]); + expect(g.groups.map((x) => x.id)).toEqual(["u1/10", "u1/11"]); + expect(g.parentOf.get("b")).toEqual("u1/11"); + expect(g.parentOf.get("c")).toEqual("u1/11"); + }); +}); diff --git a/console/src/platform/dataflows/dataflowGraph.ts b/console/src/platform/dataflows/dataflowGraph.ts new file mode 100644 index 0000000000000..8a65bda53ba54 --- /dev/null +++ b/console/src/platform/dataflows/dataflowGraph.ts @@ -0,0 +1,1476 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +export type Address = number[]; +export type NodeId = string; +export function nodeIdOf(address: Address): NodeId { + return JSON.stringify(address); +} + +/** + * Looks up a key expected to be present by construction (e.g. a child id + * drawn from the same map it indexes into). Throws with the key in the + * message instead of a bare non-null assertion, so a broken invariant fails + * loudly at the lookup site rather than as a generic "undefined" downstream. + */ +export function mustGet(map: ReadonlyMap, key: K): V { + const value = map.get(key); + if (value === undefined) { + throw new Error(`missing expected map entry for key ${String(key)}`); + } + return value; +} + +export interface NodeStats { + arrangementRecords: bigint; + arrangementSize: bigint; + elapsedNs: bigint; + // Total times this operator was scheduled (summed over the scheduling + // duration histogram's buckets). A high count relative to elapsed time + // means the operator yields often. Each activation does little work, so + // per-activation overhead (waking the worker, progress tracking) + // dominates, and raw elapsed time alone doesn't surface this. + scheduleCount: bigint; +} + +/** + * How unevenly work for this node is spread across workers: worst worker's + * value over the average, matching EXPLAIN ANALYZE ... WITH SKEW. Avg is + * over workers that show up at all for this node, not the full replica + * worker count. A worker with zero rows anywhere for a node is invisible to + * both, same convention. 1 means perfectly even, higher means more skewed, + * 0 means no data. + */ +export interface SkewStats { + cpuSkew: number; + memorySkew: number; + scheduleSkew: number; +} + +export interface LirInfo { + exportId: string; + lirId: string; + parentLirId: string | null; + nesting: number; + operator: string; +} + +/** + * One node of a dataflow's internal address tree: pure introspection data + * (own/transitive stats and skew), independent of any particular scope's + * view. `deriveVisibleGraph` projects this into `VisibleNode`s for a given + * view; this type carries no view-specific computation itself. + */ +export interface DataflowNode { + id: NodeId; + operatorId: bigint; + address: Address; + name: string; + parent: NodeId | null; + children: NodeId[]; + own: NodeStats; + transitive: NodeStats; // own + subtree, precomputed + ownSkew: SkewStats; + // Same ratios as ownSkew, but never suppressed by the magnitude floor. + // ownSkew exists to seed transitiveSkew's MAX rollup (see its comment for + // why negligible nodes must not pollute an ancestor's reported skew). + // ownSkewRaw is this node's actual measured skew, for display when this + // exact node is selected, where a floor built for ancestor-safety would + // otherwise misreport a real (if individually low-stakes) imbalance as + // "no data". + ownSkewRaw: SkewStats; + // The worst skew anywhere in the subtree (this node's own, or any + // descendant's), matching EXPLAIN ANALYZE's own MAX-based rollup. Not a + // merged-and-then-ratioed per-worker vector: two operators skewed + // toward different workers (or the same worker in opposite directions) + // can sum to a nearly-even total even though each is individually bad, + // which a merge would hide and a max surfaces. + transitiveSkew: SkewStats; + lir: LirInfo[]; // one entry per export span covering this operator id + // Timely's own-elapsed accounting is exclusive self time: an operator's + // own.elapsedNs already excludes time spent inside any nested/child + // operator's own activation, even though scheduling a region + // synchronously runs its children first (verified against live + // introspection data: a region's own.elapsedNs can come in well below + // its direct children's own.elapsedNs summed, which an inclusive + // wall-clock reading never could). Comparing self time to self time (own + // minus direct children's own, not their transitive) isolates the + // region's own dispatch overhead: pointstamp/progress tracking, + // activation logic, everything beyond simply handing off to each direct + // child once. Equals own.elapsedNs for a leaf (no children to subtract), + // which isn't a meaningful "overhead" reading on its own. + overheadNs: bigint; +} + +export interface Channel { + id: number; + fromAddress: Address; + fromPort: number; + toAddress: Address; + toPort: number; + messagesSent: bigint; + batchesSent: bigint; + channelType: string | null; +} + +export interface DataflowStructure { + nodes: Map; + root: NodeId; + channels: Channel[]; +} + +/** + * A boundary crossing this view can't show directly (the far side isn't part + * of it): kept so a port can still be jumped to, even fanning out to several + * peers (one output can feed multiple inputs). + */ +export interface PortPeer { + address: Address; + label: string; + messagesSent: bigint; + batchesSent: bigint; + channelTypes: string[]; + // The exact port this crossing lands on from inside the peer, so jumping + // can drill straight into it rather than parking outside as an unlabeled + // box. Null when the peer is a leaf (nothing to drill into, so the peer + // itself is the target). + peerPortId: NodeId | null; +} + +export interface VisibleNode { + id: string; + kind: "operator" | "region" | "port"; + label: string; + // For a region, the rolled-up subtree total (same as transitive): a + // quick canvas-level preview, not this node's own activity. See `own` + // for that. + stats: NodeStats | null; + transitive: NodeStats | null; + // This node's own stats, never rolled up: for a region, its own + // dispatch/activation cost distinct from anything its children did (see + // DataflowNode.overheadNs). `stats` collapses this into the subtree + // total for regions, which is the right quick read on the canvas but + // hides this node's own contribution entirely, which the detail panel + // needs to show on its own. + own: NodeStats | null; + // This node's own skew, unfiltered by the magnitude floor that guards + // transitiveSkew's ancestor rollup (see DataflowNode.ownSkewRaw). The + // real number for this exact node, not a heatmap-safe one. + ownSkew: SkewStats | null; + transitiveSkew: SkewStats | null; + // null for synthetic port nodes, same as transitive/transitiveSkew (no + // operator row of their own to compute it from). + overheadNs: bigint | null; + childCount: number; + lir: LirInfo[]; + address: Address | null; + // null for synthetic port nodes, which have no operator row of their own. + operatorId: bigint | null; + // Non-empty only for ports: the out-of-view side of every crossing that + // resolved to this port and wasn't already reachable some other way. + peers: PortPeer[]; +} + +/** + * A rendered edge in one scope's view: a real `Channel`'s raw endpoints, + * resolved to whatever's actually visible at this scope (an exact node, or + * a collapsed region/port peeled back to it). Several real channels can + * merge onto one `VisibleEdge` when they share the same visible endpoints. + */ +export interface VisibleEdge { + id: string; + source: string; + target: string; + messagesSent: bigint; + batchesSent: bigint; + channelTypes: string[]; + // When source/target is a collapsed region, which of its own inner ports + // this edge's real channel(s) land on (Timely always gates a scope + // crossing through the box's own address, one hop at a time, so it's + // never any deeper than that). Reuses PortPeer's shape: address is the + // box's own address (what onJumpToPeer navigates into) and peerPortId the + // resolved inner port (what it selects there). A merge of several real + // channels onto one visible edge (same collapse that merges their stats, + // e.g. two of a region's outputs feeding the same downstream sink) can + // name more than one. Empty when the endpoint is already exact (a leaf + // operator or port, or a box with no matching inner port for this hop). + sourceLandings: PortPeer[]; + targetLandings: PortPeer[]; +} + +export interface VisibleGraph { + nodes: VisibleNode[]; + edges: VisibleEdge[]; +} + +/** + * SQL row shapes (values arrive as strings or numbers, normalized here) + */ +export interface OperatorRow { + id: bigint | number | string; + address: string[]; + name: string; + arrangementRecords: bigint | number | string | null; + arrangementSize: bigint | number | string | null; + elapsedNs: bigint | number | string | null; + // Optional (unlike the fields above) so fixtures unrelated to schedule + // counting don't all need updating: real query rows always provide it + // (see useDataflowGraphData.ts's COALESCE), omitting it just reads as 0 + // via toBigInt's undefined handling, same as an explicit null would. + scheduleCount?: bigint | number | string | null; +} +export interface ChannelRow { + id: number | string; + fromOperatorAddress: string[]; + fromPort: number | string; + toOperatorAddress: string[]; + toPort: number | string; + messagesSent: bigint | number | string; + batchesSent: bigint | number | string; + channelType: string | null; +} +export interface LirSpanRow { + exportId: string; + lirId: string; + parentLirId: string | null; + nesting: number; + operator: string; + operatorIdStart: bigint | number | string; + operatorIdEnd: bigint | number | string; +} +export interface PerWorkerStatRow { + id: bigint | number | string; + workerId: bigint | number | string; + elapsedNs: bigint | number | string | null; + arrangementSize: bigint | number | string | null; + // See OperatorRow.scheduleCount for why this is optional. + scheduleCount?: bigint | number | string | null; +} + +const toBigInt = (v: bigint | number | string | null | undefined): bigint => + v == null ? 0n : BigInt(v); + +// workerId -> value. A worker absent from a node's vector never scheduled or +// arranged anything for it (no row at all, per the introspection tables' +// convention), which the caller treats as "doesn't count towards average" +// rather than "counts as zero" (matching EXPLAIN ANALYZE ... WITH SKEW). +type WorkerVector = Map; + +function skewRatio(vector: WorkerVector): number { + if (vector.size === 0) return 0; + let sum = 0n; + let max = 0n; + for (const v of vector.values()) { + sum += v; + if (v > max) max = v; + } + const avg = Number(sum) / vector.size; + return avg === 0 ? 0 : Number(max) / avg; +} + +// A node's own skew is only meaningful when the underlying work is real: an +// operator that ran once for a few nanoseconds while another worker did +// nothing can post an enormous max/avg ratio despite contributing nothing +// to the dataflow's actual cost, and since transitiveSkew rolls up as a +// MAX (see DataflowNode), that single negligible node's inflated ratio +// would otherwise become every ancestor's reported skew all the way to +// the root. A node's own magnitude (elapsed for cpu, arrangement size for +// memory) must be at least this fraction of the busiest single node's own +// magnitude anywhere in the dataflow to count. Below that floor, ownSkew +// reports the same 0 ("no data") sentinel a node with no per-worker rows at +// all would. +const SKEW_MAGNITUDE_FLOOR_FRACTION = 0.01; + +function passesMagnitudeFloor(own: bigint, maxOwn: bigint): boolean { + return ( + maxOwn === 0n || + Number(own) >= Number(maxOwn) * SKEW_MAGNITUDE_FLOOR_FRACTION + ); +} + +/** + * Builds the address tree and own/transitive stats from raw introspection + * rows. `operators` must contain at most one root (an address of length 1); + * more than one throws. An empty `operators` list (a dropped or transient + * dataflow) is valid input, not an error: it returns a single-node + * placeholder structure instead. + */ +export function buildDataflowStructure( + operators: OperatorRow[], + channels: ChannelRow[], + lirSpans: LirSpanRow[], + perWorkerStats: PerWorkerStatRow[] = [], +): DataflowStructure { + // A dropped or transient dataflow that has already gone away returns zero + // operator rows, a valid query result rather than an error. Represent it + // as a single-node placeholder structure instead of throwing below, so it + // renders through the same nodes.size <= 1 "no longer exists" branch as + // any other empty dataflow. + if (operators.length === 0) { + const placeholderId = nodeIdOf([]); + return { + nodes: new Map([ + [ + placeholderId, + { + id: placeholderId, + operatorId: 0n, + address: [], + name: "", + parent: null, + children: [], + own: { + arrangementRecords: 0n, + arrangementSize: 0n, + elapsedNs: 0n, + scheduleCount: 0n, + }, + transitive: { + arrangementRecords: 0n, + arrangementSize: 0n, + elapsedNs: 0n, + scheduleCount: 0n, + }, + ownSkew: { cpuSkew: 0, memorySkew: 0, scheduleSkew: 0 }, + ownSkewRaw: { cpuSkew: 0, memorySkew: 0, scheduleSkew: 0 }, + transitiveSkew: { cpuSkew: 0, memorySkew: 0, scheduleSkew: 0 }, + lir: [], + overheadNs: 0n, + }, + ], + ]), + root: placeholderId, + channels: [], + }; + } + const spans = lirSpans.map((s) => ({ + exportId: s.exportId, + lirId: s.lirId, + parentLirId: s.parentLirId, + nesting: s.nesting, + operator: s.operator, + start: toBigInt(s.operatorIdStart), + end: toBigInt(s.operatorIdEnd), + })); + const nodes = new Map(); + const nodeIdByOperatorId = new Map(); + for (const row of operators) { + const address = row.address.map(Number); + const id = nodeIdOf(address); + const opId = toBigInt(row.id); + const own: NodeStats = { + arrangementRecords: toBigInt(row.arrangementRecords), + arrangementSize: toBigInt(row.arrangementSize), + elapsedNs: toBigInt(row.elapsedNs), + scheduleCount: toBigInt(row.scheduleCount), + }; + nodeIdByOperatorId.set(opId.toString(), id); + nodes.set(id, { + id, + operatorId: opId, + address, + name: row.name, + parent: address.length > 1 ? nodeIdOf(address.slice(0, -1)) : null, + children: [], + own, + transitive: own, // replaced below + ownSkew: { cpuSkew: 0, memorySkew: 0, scheduleSkew: 0 }, // replaced below + ownSkewRaw: { cpuSkew: 0, memorySkew: 0, scheduleSkew: 0 }, // replaced below + transitiveSkew: { cpuSkew: 0, memorySkew: 0, scheduleSkew: 0 }, // replaced below + lir: spans + .filter((s) => s.start <= opId && opId < s.end) + .map(({ exportId, lirId, parentLirId, nesting, operator }) => ({ + exportId, + lirId, + parentLirId, + nesting, + operator, + })), + overheadNs: 0n, // replaced below + }); + } + const roots: NodeId[] = []; + for (const node of nodes.values()) { + if (node.parent === null) roots.push(node.id); + else nodes.get(node.parent)?.children.push(node.id); + } + if (roots.length !== 1) { + throw new Error( + `expected exactly one root operator, found ${roots.length}`, + ); + } + // Children in address order so layout and tests are deterministic. + for (const node of nodes.values()) { + node.children.sort((a, b) => { + const [x, y] = [mustGet(nodes, a).address, mustGet(nodes, b).address]; + return x[x.length - 1] - y[y.length - 1]; + }); + } + + const ownCpuByNode = new Map(); + const ownMemoryByNode = new Map(); + const ownScheduleByNode = new Map(); + for (const row of perWorkerStats) { + const id = nodeIdByOperatorId.get(toBigInt(row.id).toString()); + if (!id) continue; // operator outside this dataflow's own address tree + const workerId = Number(row.workerId); + if (row.elapsedNs != null) { + const vec = ownCpuByNode.get(id) ?? new Map(); + vec.set(workerId, (vec.get(workerId) ?? 0n) + toBigInt(row.elapsedNs)); + ownCpuByNode.set(id, vec); + } + if (row.arrangementSize != null) { + const vec = ownMemoryByNode.get(id) ?? new Map(); + vec.set( + workerId, + (vec.get(workerId) ?? 0n) + toBigInt(row.arrangementSize), + ); + ownMemoryByNode.set(id, vec); + } + if (row.scheduleCount != null) { + const vec = ownScheduleByNode.get(id) ?? new Map(); + vec.set( + workerId, + (vec.get(workerId) ?? 0n) + toBigInt(row.scheduleCount), + ); + ownScheduleByNode.set(id, vec); + } + } + + let maxOwnElapsedNs = 0n; + let maxOwnArrangementSize = 0n; + let maxOwnScheduleCount = 0n; + for (const node of nodes.values()) { + if (node.own.elapsedNs > maxOwnElapsedNs) { + maxOwnElapsedNs = node.own.elapsedNs; + } + if (node.own.arrangementSize > maxOwnArrangementSize) { + maxOwnArrangementSize = node.own.arrangementSize; + } + if (node.own.scheduleCount > maxOwnScheduleCount) { + maxOwnScheduleCount = node.own.scheduleCount; + } + } + + const fillTransitive = (id: NodeId): void => { + const node = mustGet(nodes, id); + const ownCpuVec = ownCpuByNode.get(id) ?? new Map(); + const ownMemoryVec = ownMemoryByNode.get(id) ?? new Map(); + const ownScheduleVec = ownScheduleByNode.get(id) ?? new Map(); + node.ownSkew = { + cpuSkew: passesMagnitudeFloor(node.own.elapsedNs, maxOwnElapsedNs) + ? skewRatio(ownCpuVec) + : 0, + memorySkew: passesMagnitudeFloor( + node.own.arrangementSize, + maxOwnArrangementSize, + ) + ? skewRatio(ownMemoryVec) + : 0, + scheduleSkew: passesMagnitudeFloor( + node.own.scheduleCount, + maxOwnScheduleCount, + ) + ? skewRatio(ownScheduleVec) + : 0, + }; + node.ownSkewRaw = { + cpuSkew: skewRatio(ownCpuVec), + memorySkew: skewRatio(ownMemoryVec), + scheduleSkew: skewRatio(ownScheduleVec), + }; + const t = { ...node.own }; + let cpuSkew = node.ownSkew.cpuSkew; + let memorySkew = node.ownSkew.memorySkew; + let scheduleSkew = node.ownSkew.scheduleSkew; + let childrenOwnElapsedNs = 0n; + for (const c of node.children) { + fillTransitive(c); + const child = mustGet(nodes, c); + t.arrangementRecords += child.transitive.arrangementRecords; + t.arrangementSize += child.transitive.arrangementSize; + t.elapsedNs += child.transitive.elapsedNs; + t.scheduleCount += child.transitive.scheduleCount; + childrenOwnElapsedNs += child.own.elapsedNs; + cpuSkew = Math.max(cpuSkew, child.transitiveSkew.cpuSkew); + memorySkew = Math.max(memorySkew, child.transitiveSkew.memorySkew); + scheduleSkew = Math.max(scheduleSkew, child.transitiveSkew.scheduleSkew); + } + node.transitive = t; + node.transitiveSkew = { cpuSkew, memorySkew, scheduleSkew }; + node.overheadNs = node.own.elapsedNs - childrenOwnElapsedNs; + }; + fillTransitive(roots[0]); + return { + nodes, + root: roots[0], + channels: channels.map((c) => ({ + id: Number(c.id), + fromAddress: c.fromOperatorAddress.map(Number), + fromPort: Number(c.fromPort), + toAddress: c.toOperatorAddress.map(Number), + toPort: Number(c.toPort), + messagesSent: toBigInt(c.messagesSent), + batchesSent: toBigInt(c.batchesSent), + channelType: c.channelType, + })), + }; +} + +/** + * A view shows exactly one scope's direct children. Anything deeper is + * rolled up into its child's box, addressed at that box's depth (viewRoot's + * depth + 1). representativeInView is the general form of that projection, + * exported for translating arbitrary structure addresses (search matches, + * LIR members) into the box that would represent them in a given view, or + * null if the address isn't inside viewRootAddress's subtree at all. + */ +export function representativeInView( + address: Address, + viewRootAddress: Address, +): NodeId | null { + if ( + address.length > viewRootAddress.length && + viewRootAddress.every((v, i) => address[i] === v) + ) { + return nodeIdOf(address.slice(0, viewRootAddress.length + 1)); + } + return null; +} + +function addressesEqual(a: Address, b: Address): boolean { + return a.length === b.length && a.every((v, i) => v === b[i]); +} + +function representative(address: Address, viewRootAddress: Address): NodeId { + return representativeInView(address, viewRootAddress) ?? nodeIdOf(address); +} + +// A scope boundary crossing is logged as two channels that share a port +// number but use different addressing for each half: +// - The outer half (external <-> scope) addresses the scope by its own +// operator address, e.g. an external producer's channel row targets +// address [5,2] directly, the same address BuildRegion's own operator +// row uses. +// - The inner half (scope <-> child) addresses the scope's internal +// fan-out point one level deeper, e.g. [5,2,0]. +// Both halves are mapped to the same synthetic port id, keyed by (scope, +// direction, port number), so they visually chain through one port node +// instead of the outer half landing on the region's own node (indistinguishable +// from every other port sharing that scope) or the inner half dangling. +// +// A port only ever renders for viewRoot's own boundary: every other scope in +// a view is shown as a collapsed box (never "expanded"), so any crossing at +// its boundary rolls up into that box instead, exactly like a normal channel +// endpoint landing inside it would. +function endpointId( + structure: DataflowStructure, + address: Address, + port: number, + side: "from" | "to", + viewRoot: NodeId, + viewRootAddress: Address, +): { + id: string; + port?: { scope: NodeId; direction: "input" | "output"; port: number }; +} { + if (address[address.length - 1] === 0) { + const scope = address.slice(0, -1); + const scopeId = nodeIdOf(scope); + const rep = + scope.length === 0 ? scopeId : representative(scope, viewRootAddress); + if (rep !== scopeId || scopeId !== viewRoot) return { id: rep }; + const direction = side === "from" ? "input" : "output"; + return { + id: `${scopeId}:${direction === "input" ? "in" : "out"}:${port}`, + port: { scope: scopeId, direction, port }, + }; + } + const scopeId = nodeIdOf(address); + const scopeNode = structure.nodes.get(scopeId); + if (scopeNode && scopeNode.children.length > 0) { + const rep = representative(address, viewRootAddress); + if (rep === scopeId && scopeId === viewRoot) { + const direction = side === "to" ? "input" : "output"; + return { + id: `${scopeId}:${direction === "input" ? "in" : "out"}:${port}`, + port: { scope: scopeId, direction, port }, + }; + } + return { id: rep }; + } + return { id: representative(address, viewRootAddress) }; +} + +/** + * Renders exactly one scope's direct children, each either a leaf operator or + * a box summarizing a whole nested subtree (never expanded in place). + * Double-clicking a box navigates to a new view rooted there instead. + */ +export function deriveVisibleGraph( + structure: DataflowStructure, + viewRoot: NodeId, +): VisibleGraph { + const viewRootNode = mustGet(structure.nodes, viewRoot); + const viewRootAddress = viewRootNode.address; + const nodes: VisibleNode[] = viewRootNode.children.map((id) => { + const node = mustGet(structure.nodes, id); + const kind = node.children.length === 0 ? "operator" : "region"; + return { + id, + kind, + label: node.name, + stats: kind === "region" ? node.transitive : node.own, + transitive: node.transitive, + own: node.own, + ownSkew: node.ownSkewRaw, + transitiveSkew: node.transitiveSkew, + overheadNs: node.overheadNs, + childCount: node.children.length, + lir: node.lir, + address: node.address, + operatorId: node.operatorId, + peers: [], + }; + }); + const visibleIds = new Set(nodes.map((n) => n.id)); + + // sourceLandings/targetLandings are computed separately (see + // sourceLandingsByEdge/targetLandingsByEdge below) and attached once, in + // the final map at the bottom, rather than threaded through here. + const edgesById = new Map< + string, + Omit & { + typeSet: Set; + } + >(); + const ports = new Map(); + const portId = (p: { + scope: NodeId; + direction: "input" | "output"; + port: number; + }) => `${p.scope}:${p.direction === "input" ? "in" : "out"}:${p.port}`; + // Records the out-of-view side of a crossing on its port, so a port that + // can't show its peer directly can still be jumped to. One output can feed + // several inputs (and vice versa), so this can add up across channels + // rather than replace. The same peer address showing up twice (e.g. two + // channel rows for the same logical pair) just accumulates its stats. + const addPeer = ( + p: { scope: NodeId; direction: "input" | "output"; port: number }, + peerAddress: Address, + peerPort: number, + peerSide: "from" | "to", + messagesSent: bigint, + batchesSent: bigint, + channelType: string | null, + ) => { + // A scope's own boundary is logged as a pseudo-vertex [scope, 0], which + // never gets an operator row of its own (only real children do). A + // region nested two or more scopes deep has its outer half target that + // pseudo-vertex directly, one level further up than the region itself. + // Peel back to the enclosing scope, which does have a row, so the jump + // still lands somewhere instead of being dropped as if elided. + let resolvedAddress = peerAddress; + while ( + resolvedAddress.length > 0 && + resolvedAddress[resolvedAddress.length - 1] === 0 && + !structure.nodes.has(nodeIdOf(resolvedAddress)) + ) { + resolvedAddress = resolvedAddress.slice(0, -1); + } + const peerId = nodeIdOf(resolvedAddress); + if (!structure.nodes.has(peerId)) return; // elided scope, nothing to jump to + const port = mustGet(ports, portId(p)); + const existing = port.peers.find( + (peer) => nodeIdOf(peer.address) === peerId, + ); + if (existing) { + existing.messagesSent += messagesSent; + existing.batchesSent += batchesSent; + if (channelType && !existing.channelTypes.includes(channelType)) { + existing.channelTypes = [...existing.channelTypes, channelType].sort(); + } + } else { + // Re-resolve this same crossing as if viewRoot were the peer itself, + // to find the exact port a jump should land on inside it. A leaf peer + // has nothing to drill into, so this naturally comes back port-less. + const fromPeer = endpointId( + structure, + peerAddress, + peerPort, + peerSide, + peerId, + resolvedAddress, + ); + port.peers.push({ + address: resolvedAddress, + label: mustGet(structure.nodes, peerId).name, + messagesSent, + batchesSent, + channelTypes: channelType ? [channelType] : [], + peerPortId: fromPeer.port ? fromPeer.id : null, + }); + } + }; + // A channel touching a collapsed region box always does so at exactly the + // box's own address (Timely gates every scope crossing through a single + // boundary vertex, one hop at a time. A plain, non-port box reference is + // therefore never deeper than its own address, unlike addPeer's peers, + // which land outside viewRoot's subtree entirely). What can still be + // ambiguous is which of the box's own inner ports a given hop's port + // number lands on, and a merge of several such hops onto one visible + // edge (same collapse that merges their stats, e.g. two of a region's + // outputs feeding the same downstream sink) can name more than one. + // Recorded per side, keyed by edge id. + const sourceLandingsByEdge = new Map>(); + const targetLandingsByEdge = new Map>(); + const addLanding = ( + byEdge: Map>, + edgeId: string, + boxAddress: Address, + innerPortId: NodeId, + label: string, + messagesSent: bigint, + batchesSent: bigint, + channelType: string | null, + ) => { + let landings = byEdge.get(edgeId); + if (!landings) { + landings = new Map(); + byEdge.set(edgeId, landings); + } + const existing = landings.get(innerPortId); + if (existing) { + existing.messagesSent += messagesSent; + existing.batchesSent += batchesSent; + if (channelType && !existing.channelTypes.includes(channelType)) { + existing.channelTypes = [...existing.channelTypes, channelType].sort(); + } + } else { + landings.set(innerPortId, { + address: boxAddress, + label, + messagesSent, + batchesSent, + channelTypes: channelType ? [channelType] : [], + peerPortId: innerPortId, + }); + } + }; + for (const ch of structure.channels) { + const from = endpointId( + structure, + ch.fromAddress, + ch.fromPort, + "from", + viewRoot, + viewRootAddress, + ); + const to = endpointId( + structure, + ch.toAddress, + ch.toPort, + "to", + viewRoot, + viewRootAddress, + ); + // A port node is registered whenever either side resolves to one, even + // if the edge itself is about to be dropped: viewRoot's own boundary can + // have traffic to a sibling entirely outside this view (nothing on the + // far side to draw an edge to), and the port stub is the only visible + // trace that the crossing exists at all. + for (const p of [from.port, to.port]) { + if (p && !ports.has(portId(p))) { + ports.set(portId(p), { + id: portId(p), + kind: "port", + label: `${p.direction} ${p.port}`, + stats: null, + transitive: null, + own: null, + ownSkew: null, + transitiveSkew: null, + overheadNs: null, + childCount: 0, + lir: [], + address: null, + operatorId: null, + peers: [], + }); + } + } + // The far side of a port crossing is worth recording as a jump target + // only when this view doesn't already show it some other way (e.g. the + // inner half of a boundary pairing lands on a node already visible here, + // which needs no jump link since there's already a drawn edge to it). + if (from.port && !to.port && !visibleIds.has(to.id)) { + addPeer( + from.port, + ch.toAddress, + ch.toPort, + "to", + ch.messagesSent, + ch.batchesSent, + ch.channelType, + ); + } + if (to.port && !from.port && !visibleIds.has(from.id)) { + addPeer( + to.port, + ch.fromAddress, + ch.fromPort, + "from", + ch.messagesSent, + ch.batchesSent, + ch.channelType, + ); + } + // A non-port endpoint id must name one of this view's own boxes. Real + // dataflows can log channels touching an address with no corresponding + // operator row (e.g. an elided scope), and any address outside viewRoot's + // subtree resolves to an id this view never emits either. Drop the + // edge (but keep any port registered above) in both cases rather than + // hand elk a dangling reference. + if ( + (!from.port && !visibleIds.has(from.id)) || + (!to.port && !visibleIds.has(to.id)) + ) { + continue; + } + if (from.id === to.id) continue; + const id = `${from.id}=>${to.id}`; + // A landing only adds information when the channel touches the box at + // exactly its own address (the boundary-crossing pattern endpointId's + // ends-in-0 pairing relies on): only then does it name one of the box's + // own inner ports to resolve. Anything else (a leaf collapsing into an + // ancestor box, an address that isn't this box's boundary at all) has + // no more precise a target than the box itself, already shown. + if (!from.port) { + const box = mustGet(structure.nodes, from.id); + if (addressesEqual(ch.fromAddress, box.address)) { + const inner = endpointId( + structure, + ch.fromAddress, + ch.fromPort, + "from", + from.id, + box.address, + ); + if (inner.port) { + addLanding( + sourceLandingsByEdge, + id, + box.address, + inner.id, + `${inner.port.direction} ${inner.port.port}`, + ch.messagesSent, + ch.batchesSent, + ch.channelType, + ); + } + } + } + if (!to.port) { + const box = mustGet(structure.nodes, to.id); + if (addressesEqual(ch.toAddress, box.address)) { + const inner = endpointId( + structure, + ch.toAddress, + ch.toPort, + "to", + to.id, + box.address, + ); + if (inner.port) { + addLanding( + targetLandingsByEdge, + id, + box.address, + inner.id, + `${inner.port.direction} ${inner.port.port}`, + ch.messagesSent, + ch.batchesSent, + ch.channelType, + ); + } + } + } + const existing = edgesById.get(id); + if (existing) { + existing.messagesSent += ch.messagesSent; + existing.batchesSent += ch.batchesSent; + if (ch.channelType) existing.typeSet.add(ch.channelType); + } else { + edgesById.set(id, { + id, + source: from.id, + target: to.id, + messagesSent: ch.messagesSent, + batchesSent: ch.batchesSent, + channelTypes: [], + typeSet: new Set(ch.channelType ? [ch.channelType] : []), + }); + } + } + const edges = [...edgesById.values()].map(({ typeSet, ...e }) => ({ + ...e, + channelTypes: [...typeSet].sort(), + sourceLandings: [...(sourceLandingsByEdge.get(e.id)?.values() ?? [])], + targetLandings: [...(targetLandingsByEdge.get(e.id)?.values() ?? [])], + })); + return { nodes: [...nodes, ...ports.values()], edges }; +} + +/** + * The scope to navigate to so that every given address is directly visible, + * each as either itself (if it's already a direct child of the result) or + * the box that rolls it up. Backs off past any address that IS the raw + * common prefix (so every input lands strictly below the result, never + * equal to it) and past any prefix with no operator row of its own (an + * elided scope can't be navigated to, since it never appears in a view). + */ +export function commonAncestorScope( + structure: DataflowStructure, + addresses: readonly Address[], +): NodeId { + let prefix = addresses[0]; + for (const a of addresses.slice(1)) { + let len = 0; + while (len < prefix.length && len < a.length && prefix[len] === a[len]) { + len++; + } + prefix = prefix.slice(0, len); + } + const shortest = Math.min(...addresses.map((a) => a.length)); + while (prefix.length >= shortest || !structure.nodes.has(nodeIdOf(prefix))) { + if (prefix.length === 0) return structure.root; + prefix = prefix.slice(0, -1); + } + return nodeIdOf(prefix); +} + +/** + * Hiding a run of idle operators would otherwise sever every path through + * them, splitting the graph into disconnected islands even though a real + * (idle) path exists. Splices a pass-through edge across each hidden run, + * so hiding idle nodes never removes connectivity, only the boxes in + * between. The spliced edge sums messages/batches/types along the run, + * which is almost always 0/0 (that is why the run was hidden) and so + * renders with the same dashed "idle" styling as an ordinary quiet edge. + * + * Cycles (timely feedback loops) are real and must not hang this: `path` + * tracks nodes on the current walk, and re-entering one simply stops that + * branch rather than looping forever. + */ +export function rerouteHiddenNodes( + graph: VisibleGraph, + hiddenNodeIds: ReadonlySet, +): VisibleGraph { + if (hiddenNodeIds.size === 0) return graph; + const nodes = graph.nodes.filter((n) => !hiddenNodeIds.has(n.id)); + + const outAdj = new Map(); + for (const e of graph.edges) { + const list = outAdj.get(e.source); + if (list) list.push(e); + else outAdj.set(e.source, [e]); + } + + interface Agg { + messagesSent: bigint; + batchesSent: bigint; + types: Set; + } + + // Every visible node reachable by walking forward from `from` through only + // hidden nodes, with edge stats accumulated along the way. + function reachableVisible(from: string, path: Set): Map { + const result = new Map(); + if (path.has(from)) return result; + path.add(from); + for (const e of outAdj.get(from) ?? []) { + if (!hiddenNodeIds.has(e.target)) { + merge(result, e.target, e.messagesSent, e.batchesSent, e.channelTypes); + } else { + for (const [target, agg] of reachableVisible(e.target, path)) { + merge( + result, + target, + agg.messagesSent + e.messagesSent, + agg.batchesSent + e.batchesSent, + [...agg.types, ...e.channelTypes], + ); + } + } + } + path.delete(from); + return result; + } + + function merge( + into: Map, + target: string, + messagesSent: bigint, + batchesSent: bigint, + types: string[], + ) { + const existing = into.get(target); + if (existing) { + existing.messagesSent += messagesSent; + existing.batchesSent += batchesSent; + for (const t of types) existing.types.add(t); + } else { + into.set(target, { messagesSent, batchesSent, types: new Set(types) }); + } + } + + const edgesById = new Map(); + for (const e of graph.edges) { + if (!hiddenNodeIds.has(e.source) && !hiddenNodeIds.has(e.target)) { + edgesById.set(e.id, e); + } + } + for (const e of graph.edges) { + if (hiddenNodeIds.has(e.source) || !hiddenNodeIds.has(e.target)) continue; + for (const [target, agg] of reachableVisible( + e.target, + new Set([e.source]), + )) { + const id = `${e.source}=>${target}`; + const messagesSent = e.messagesSent + agg.messagesSent; + const batchesSent = e.batchesSent + agg.batchesSent; + const types = new Set([...e.channelTypes, ...agg.types]); + const existing = edgesById.get(id); + if (existing) { + edgesById.set(id, { + ...existing, + messagesSent: existing.messagesSent + messagesSent, + batchesSent: existing.batchesSent + batchesSent, + channelTypes: [ + ...new Set([...existing.channelTypes, ...types]), + ].sort(), + }); + } else { + edgesById.set(id, { + id, + source: e.source, + target, + messagesSent, + batchesSent, + channelTypes: [...types].sort(), + // A spliced edge crosses one or more hidden nodes: it no longer + // corresponds to a single real channel, so there's no landing + // list to carry over. + sourceLandings: [], + targetLandings: [], + }); + } + } + } + return { nodes, edges: [...edgesById.values()] }; +} + +export interface Filters { + search: string; + hideIdle: boolean; + heatmap: + | "off" + | "elapsed" + | "size" + | "schedules" + | "cpuSkew" + | "memorySkew" + | "scheduleSkew"; + heatmapThreshold: number; // 0..1 fraction of max + showLirGroups: boolean; +} + +export const DEFAULT_FILTERS: Filters = { + search: "", + hideIdle: false, + heatmap: "off", + heatmapThreshold: 0, + showLirGroups: true, +}; + +/** + * decorateGraph returns every field set. + */ +export interface GraphDecorations { + dimmedNodeIds: Set; + hiddenNodeIds: Set; + hiddenEdgeIds: Set; + nodeColors: Map; + searchMatches: string[]; // visible node ids, document order +} + +/** + * Search matches anywhere in the whole dataflow, not just the current view + * (search is structure-wide so it stays useful once a graph is too big to + * show in one screen). searchInfo carries that lookup in, precomputed once + * per search string rather than re-walked per view. A box that doesn't + * itself match but contains a match deeper in its rolled-up subtree is left + * undimmed rather than excluded, so the match's path stays visible without + * forcing a navigation on every keystroke. + */ +export function decorateGraph( + graph: VisibleGraph, + filters: Filters, + heatColor: (t: number) => string, + searchInfo: SubtreeSearchMatches | null, + // The replica's total worker count: skewRatio (max-worker-over-average) + // is mathematically bounded above by it (one worker doing 100% of a + // fixed amount of work, the rest 0%, is the most skewed that work can + // ever be split across this many workers). Used only by the cpuSkew/ + // memorySkew/scheduleSkew heatmap modes, as a fixed color-scale ceiling. + workerCount: number, +): GraphDecorations { + const d: GraphDecorations = { + dimmedNodeIds: new Set(), + hiddenNodeIds: new Set(), + hiddenEdgeIds: new Set(), + nodeColors: new Map(), + searchMatches: [], + }; + for (const n of graph.nodes) { + if (searchInfo && n.kind !== "port") { + if (searchInfo.matches.has(n.id)) d.searchMatches.push(n.id); + else if (!searchInfo.containsMatch.has(n.id)) d.dimmedNodeIds.add(n.id); + } + if ( + filters.hideIdle && + n.kind === "operator" && + n.stats !== null && + n.stats.elapsedNs === 0n && + n.stats.arrangementRecords === 0n + ) { + d.hiddenNodeIds.add(n.id); + } + } + for (const e of graph.edges) { + if (filters.hideIdle && e.messagesSent === 0n) d.hiddenEdgeIds.add(e.id); + } + if (filters.heatmap !== "off") { + const heatmap = filters.heatmap; + const metric = (n: VisibleNode): number => { + switch (heatmap) { + case "elapsed": + return Number(n.transitive?.elapsedNs ?? 0n); + case "size": + return Number(n.transitive?.arrangementSize ?? 0n); + case "schedules": + return Number(n.transitive?.scheduleCount ?? 0n); + case "cpuSkew": + return n.transitiveSkew?.cpuSkew ?? 0; + case "memorySkew": + return n.transitiveSkew?.memorySkew ?? 0; + case "scheduleSkew": + return n.transitiveSkew?.scheduleSkew ?? 0; + } + }; + const isSkew = + heatmap === "cpuSkew" || + heatmap === "memorySkew" || + heatmap === "scheduleSkew"; + const candidates = graph.nodes.filter((n) => n.kind !== "port"); + if (isSkew) { + // A skew ratio is bounded above by workerCount (see decorateGraph's + // own doc comment), and real skew clusters near 1 in practice: a + // linear [1, workerCount] scale would crush a genuinely bad 2x-3x + // skew near the cold end on any replica with more than a handful of + // workers. log2 spreads each doubling of skew evenly across the + // color range instead, matching the ratio's multiplicative nature + // (the same reason decibels/Richter use log scales for other + // ratio-based measurements), and is fixed rather than derived from + // whatever else happens to be in the current view. + const domainMax = Math.log2(Math.max(1, workerCount)); + if (domainMax > 0) { + for (const n of candidates) { + const t = Math.max( + 0, + Math.min(1, Math.log2(Math.max(1, metric(n))) / domainMax), + ); + d.nodeColors.set(n.id, heatColor(t)); + if (t < filters.heatmapThreshold) d.dimmedNodeIds.add(n.id); + } + } + } else { + const max = candidates.reduce((m, n) => Math.max(m, metric(n)), 0); + if (max > 0) { + for (const n of candidates) { + const t = metric(n) / max; + d.nodeColors.set(n.id, heatColor(t)); + if (t < filters.heatmapThreshold) d.dimmedNodeIds.add(n.id); + } + } + } + } + return d; +} + +export interface SubtreeSearchMatches { + matches: Set; // nodes whose own name matches + containsMatch: Set; // matches, plus any ancestor of one +} + +/** + * Computed once over the whole structure (not the current view), so a box is + * never dimmed just because the current scope hasn't drilled down to the + * match nested inside it yet. + */ +export function subtreeSearchMatches( + structure: DataflowStructure, + search: string, +): SubtreeSearchMatches { + const needle = search.trim().toLowerCase(); + const matches = new Set(); + const containsMatch = new Set(); + if (!needle) return { matches, containsMatch }; + const visit = (id: NodeId): boolean => { + const node = mustGet(structure.nodes, id); + let hit = node.name.toLowerCase().includes(needle); + if (hit) matches.add(id); + for (const c of node.children) { + if (visit(c)) hit = true; + } + if (hit) containsMatch.add(id); + return hit; + }; + for (const id of mustGet(structure.nodes, structure.root).children) { + visit(id); + } + return { matches, containsMatch }; +} + +function addressCompare(a: Address, b: Address): number { + for (let i = 0; i < Math.min(a.length, b.length); i++) { + if (a[i] !== b[i]) return a[i] - b[i]; + } + return a.length - b.length; +} + +/** + * The ordered, structure-wide match list prev/next cycles through: jumping + * to a match may navigate the view (unlike decorateGraph's searchMatches, + * which only ever lists what's already visible in the current scope). + */ +export function allSearchMatches( + structure: DataflowStructure, + search: string, +): DataflowNode[] { + const needle = search.trim().toLowerCase(); + if (!needle) return []; + return [...structure.nodes.values()] + .filter( + (node) => + node.id !== structure.root && node.name.toLowerCase().includes(needle), + ) + .sort((a, b) => addressCompare(a.address, b.address)); +} + +/** + * Groups operator nodes by the LIR span covering them, keyed + * `${exportId}/${lirId}`. One dataflow can back several exports, so entries + * are not unique per lir id across exports. + */ +export function lirIndex( + structure: DataflowStructure, +): Map { + const index = new Map(); + for (const node of structure.nodes.values()) { + for (const info of node.lir) { + const key = `${info.exportId}/${info.lirId}`; + const entry = index.get(key) ?? { info, memberIds: [] }; + entry.memberIds.push(node.id); + index.set(key, entry); + } + } + return index; +} + +export interface LirTreeNode { + key: string; + info: LirInfo; + memberIds: NodeId[]; + children: LirTreeNode[]; + summary: NodeStats; +} + +/** + * Sums each member's OWN stats (not transitive): every operator belonging to + * a LIR id appears in memberIds exactly once, including a parent LIR id's + * descendants (buildDataflowStructure assigns an operator to every ancestor + * span that contains it), so this matches EXPLAIN ANALYZE's own total_memory/ + * total_records/total_elapsed exactly without a separate query: the same + * operator rows are already fetched for the graph itself. + */ +export function lirSummary( + structure: DataflowStructure, + memberIds: readonly NodeId[], +): NodeStats { + const summary: NodeStats = { + arrangementRecords: 0n, + arrangementSize: 0n, + elapsedNs: 0n, + scheduleCount: 0n, + }; + for (const id of memberIds) { + const own = structure.nodes.get(id)?.own; + if (!own) continue; + summary.arrangementRecords += own.arrangementRecords; + summary.arrangementSize += own.arrangementSize; + summary.elapsedNs += own.elapsedNs; + summary.scheduleCount += own.scheduleCount; + } + return summary; +} + +/** + * Arranges lirIndex's flat entries into a tree per export, following + * parentLirId. A lir id is assigned once its own build (and so every + * descendant's) completes, so within one export a higher lir id is never an + * ancestor of a lower one. Sorting each level descending by lir id therefore + * matches construction order without needing it as an explicit input, the + * same convention EXPLAIN ANALYZE's own tree rendering uses. + */ +export function lirTree( + structure: DataflowStructure, + index: ReadonlyMap, +): Map { + const byExport = new Map>(); + for (const { info, memberIds } of index.values()) { + const byLirId = byExport.get(info.exportId) ?? new Map(); + byLirId.set(info.lirId, { + key: `${info.exportId}/${info.lirId}`, + info, + memberIds, + children: [], + summary: lirSummary(structure, memberIds), + }); + byExport.set(info.exportId, byLirId); + } + const sortDesc = (nodes: LirTreeNode[]) => { + nodes.sort((a, b) => Number(b.info.lirId) - Number(a.info.lirId)); + for (const n of nodes) sortDesc(n.children); + }; + const roots = new Map(); + for (const [exportId, byLirId] of byExport) { + const exportRoots: LirTreeNode[] = []; + for (const node of byLirId.values()) { + const parent = node.info.parentLirId + ? byLirId.get(node.info.parentLirId) + : undefined; + if (parent) parent.children.push(node); + else exportRoots.push(node); + } + sortDesc(exportRoots); + roots.set(exportId, exportRoots); + } + return roots; +} + +export interface LirGroupNode { + id: NodeId; // `${exportId}/${lirId}`, matching lirIndex's key + exportId: string; + lirId: string; + operator: string; + nesting: number; +} + +/** + * A grouping layer over an already-derived VisibleGraph, the same way + * decorateGraph layers dimming and color over it: nothing here replaces + * `nodes`, so search, dimming, and click handling keep working against the + * flat list untouched. + */ +export interface LirGrouping { + groups: LirGroupNode[]; // outer groups appear before groups nested inside them + parentOf: Map; // a VisibleNode or LirGroupNode id -> its immediate enclosing group's id +} + +/** + * Groups a scope's direct children into the LIR tree that encloses them, for + * drawing nested boxes on the canvas. LIR spans are contiguous ranges over + * operator ids and properly nested: a region is always fully enclosed by a + * single LIR node's range. A dataflow can back several exports, and a shared + * subplan can carry lir entries from more than one of them at once; a + * compound-node tree needs one parent chain per node, so this groups by the + * lowest exportId (string-compared) present among the given nodes only. A + * node whose only lir entries belong to a different export renders + * ungrouped, alongside nodes with no lir data at all (e.g. ports). + */ +export function groupByLir(nodes: readonly VisibleNode[]): LirGrouping { + const groups: LirGroupNode[] = []; + const parentOf = new Map(); + const ordered = nodes + .filter( + (n): n is VisibleNode & { operatorId: bigint } => n.operatorId !== null, + ) + .slice() + .sort((a, b) => { + const x = a.operatorId; + const y = b.operatorId; + return x < y ? -1 : x > y ? 1 : 0; + }); + if (ordered.length === 0) return { groups, parentOf }; + + let chosenExport: string | null = null; + for (const n of ordered) { + for (const l of n.lir) { + if (chosenExport === null || l.exportId < chosenExport) { + chosenExport = l.exportId; + } + } + } + if (chosenExport === null) return { groups, parentOf }; + + const groupIndex = new Map(); + const stack: LirGroupNode[] = []; // open groups, outer to inner + for (const node of ordered) { + const stackKeys = node.lir + .filter((l) => l.exportId === chosenExport) + .sort((a, b) => a.nesting - b.nesting) + .map((l) => `${l.exportId}/${l.lirId}`); + let commonLen = 0; + while ( + commonLen < stack.length && + commonLen < stackKeys.length && + stack[commonLen].id === stackKeys[commonLen] + ) { + commonLen++; + } + stack.length = commonLen; // pop back to the shared prefix + for (let depth = commonLen; depth < stackKeys.length; depth++) { + const key = stackKeys[depth]; + let group = groupIndex.get(key); + if (!group) { + const info = node.lir.find((l) => `${l.exportId}/${l.lirId}` === key); + if (!info) { + // key was derived from this same node.lir list two lines up + // (filtered to chosenExport, mapped to "exportId/lirId"), so the + // matching entry always exists. This only guards the invariant. + throw new Error(`missing lir info for key ${key}`); + } + group = { + id: key, + exportId: info.exportId, + lirId: info.lirId, + operator: info.operator, + nesting: info.nesting, + }; + groupIndex.set(key, group); + groups.push(group); + } + if (depth > 0) parentOf.set(group.id, stack[depth - 1].id); + stack.push(group); + } + if (stack.length > 0) parentOf.set(node.id, stack[stack.length - 1].id); + } + return { groups, parentOf }; +} diff --git a/console/src/platform/dataflows/elkGraph.test.ts b/console/src/platform/dataflows/elkGraph.test.ts new file mode 100644 index 0000000000000..8cb462669abf7 --- /dev/null +++ b/console/src/platform/dataflows/elkGraph.test.ts @@ -0,0 +1,235 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import ELK from "elkjs"; +import { describe, expect, it } from "vitest"; + +import { + buildDataflowStructure, + deriveVisibleGraph, + groupByLir, + nodeIdOf, +} from "./dataflowGraph"; +import { CHANNELS, LIR_SPANS, OPS } from "./dataflowGraph.test"; +import { + extractPositions, + NODE_DIMENSIONS, + type Positions, + resolveAbsolutePositions, + toElkGraph, +} from "./elkGraph"; + +describe("toElkGraph", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS); + + it("lists every visible node flat, at its kind's dimensions", () => { + const elk = toElkGraph(deriveVisibleGraph(s, nodeIdOf([5, 1]))); + const ids = elk.children!.map((c) => c.id); + expect(ids).toEqual( + expect.arrayContaining([ + nodeIdOf([5, 1, 1]), + nodeIdOf([5, 1, 2]), + `${nodeIdOf([5, 1])}:in:0`, + ]), + ); + const leaf = elk.children!.find((c) => c.id === nodeIdOf([5, 1, 1]))!; + expect(leaf.width).toEqual(NODE_DIMENSIONS.operator.width); + // Flat: no node carries children of its own. + for (const c of elk.children!) expect(c.children).toBeUndefined(); + }); + + it("round-trips positions", () => { + const elk = toElkGraph(deriveVisibleGraph(s, s.root)); + // simulate a layout result + elk.children![0].x = 10; + elk.children![0].y = 20; + const positions = extractPositions(elk); + expect(positions[elk.children![0].id]).toMatchObject({ x: 10, y: 20 }); + }); +}); + +describe("toElkGraph with a LIR grouping", () => { + const s = buildDataflowStructure(OPS, CHANNELS, LIR_SPANS); + + it("nests a group's members as elk children, sized only by content", () => { + const visible = deriveVisibleGraph(s, s.root); + const grouping = groupByLir(visible.nodes); + const elk = toElkGraph(visible, grouping); + const group = elk.children!.find((c) => c.id === grouping.groups[0].id)!; + expect(group.children!.map((c) => c.id)).toEqual( + expect.arrayContaining([nodeIdOf([5, 1])]), + ); + // Auto-sized by elk from its content, not one of NODE_DIMENSIONS's fixed sizes. + expect(group.width).toBeUndefined(); + expect(group.height).toBeUndefined(); + // Flat sibling stays a direct child of the root, not absorbed into the group. + expect(elk.children!.map((c) => c.id)).toEqual( + expect.arrayContaining([nodeIdOf([5, 2])]), + ); + }); + + it("keeps every edge at the root level regardless of nesting depth", () => { + const visible = deriveVisibleGraph(s, s.root); + const grouping = groupByLir(visible.nodes); + const elk = toElkGraph(visible, grouping); + expect(elk.edges!.map((e) => e.id)).toEqual(visible.edges.map((e) => e.id)); + for (const child of elk.children!) expect(child.edges).toBeUndefined(); + }); + + it("round-trips nested positions unchanged (parent-relative, no conversion)", () => { + const visible = deriveVisibleGraph(s, s.root); + const grouping = groupByLir(visible.nodes); + const elk = toElkGraph(visible, grouping); + const group = elk.children!.find((c) => c.id === grouping.groups[0].id)!; + group.x = 12; + group.y = 87; + group.width = 132; + group.height = 259; + const member = group.children!.find((c) => c.id === nodeIdOf([5, 1]))!; + member.x = 16; + member.y = 24; + const positions = extractPositions(elk); + // The member's extracted position is exactly its own x/y, not offset by + // the group's: elk already reports it relative to the group. + expect(positions[member.id]).toMatchObject({ x: 16, y: 24 }); + expect(positions[group.id]).toMatchObject({ + x: 12, + y: 87, + width: 132, + height: 259, + }); + }); + + it("real elk lays out nested groups with auto-sized bounds and positions relative to parent", async () => { + // Empirical check: elk auto-sizes a compound node from its children, + // reports child positions relative to their own parent (matching React + // Flow's parentId convention with no coordinate conversion needed), and + // processes edges declared at the root even with nested containers present, + // given hierarchyHandling: INCLUDE_CHILDREN. + const elk = new ELK(); + const graph = { + id: "root", + layoutOptions: { + "elk.algorithm": "layered", + "elk.direction": "DOWN", + "elk.hierarchyHandling": "INCLUDE_CHILDREN", + }, + children: [ + { id: "ungrouped", width: 100, height: 50 }, + { + id: "groupA", + layoutOptions: { "elk.padding": "[top=24,left=8,bottom=8,right=8]" }, + children: [ + { id: "a1", width: 100, height: 50 }, + { + id: "groupA_inner", + layoutOptions: { + "elk.padding": "[top=24,left=8,bottom=8,right=8]", + }, + children: [ + { id: "a2", width: 100, height: 50 }, + { id: "a3", width: 100, height: 50 }, + ], + }, + ], + }, + ], + edges: [ + { id: "e1", sources: ["ungrouped"], targets: ["a1"] }, + { id: "e2", sources: ["a1"], targets: ["a2"] }, + { id: "e3", sources: ["a2"], targets: ["a3"] }, + ], + }; + const result = await elk.layout(graph); + const groupA = result.children!.find((c) => c.id === "groupA")!; + const groupAInner = groupA.children!.find((c) => c.id === "groupA_inner")!; + // Auto-sized larger than any single member: real content-driven sizing. + expect(groupA.width!).toBeGreaterThan(100); + expect(groupA.height!).toBeGreaterThan(50); + // All edges remain at root even though nested containers are present; + // they layout correctly with hierarchyHandling: INCLUDE_CHILDREN. + expect((result.edges ?? []).map((e) => e.id)).toEqual( + expect.arrayContaining(["e1", "e2", "e3"]), + ); + // Nested child positions are relative to their immediate parent (groupA_inner). + // hierarchyHandling: INCLUDE_CHILDREN is the discriminating option that changes + // elk's nested layout: without it, a3 lands at x:128, y:24 instead. + // Pinning this exact position discriminates the option's presence in production. + const a3 = groupAInner.children!.find((c) => c.id === "a3")!; + expect(a3).toMatchObject({ x: 8, y: 94 }); + }); +}); + +// The fix for ViewportGuard reading elk's group-relative positions as if +// they were already absolute canvas coordinates (DataflowGraphView.tsx). +// Exercised directly here, rather than through DataflowGraphView or +// DataflowDetailPage, because the mocked useElkLayout those component tests +// rely on returns the same fixed {x: 0, y: 0, ...} box for every node id via +// a Proxy, so a group and its member always land on the same position +// regardless of whether the offset gets resolved, which can't discriminate +// this bug from a fix. +describe("resolveAbsolutePositions", () => { + it("returns positions unchanged when there is no grouping", () => { + const positions: Positions = { + a: { x: 5, y: 5, width: 10, height: 10 }, + }; + expect(resolveAbsolutePositions(positions, undefined)).toBe(positions); + expect(resolveAbsolutePositions(positions, new Map())).toBe(positions); + }); + + it("offsets a group member by its group's own position", () => { + const positions: Positions = { + group: { x: 100, y: 200, width: 50, height: 50 }, + member: { x: 8, y: 24, width: 10, height: 10 }, + }; + const parentOf = new Map([["member", "group"]]); + const resolved = resolveAbsolutePositions(positions, parentOf); + // The group itself has no parent, so its own position is already absolute. + expect(resolved.group).toMatchObject({ x: 100, y: 200 }); + // The member's elk-reported position (8, 24) is relative to the group; + // its absolute position is the group's origin plus that offset. + expect(resolved.member).toMatchObject({ x: 108, y: 224 }); + }); + + it("sums offsets across arbitrarily many levels of nesting", () => { + const positions: Positions = { + outer: { x: 100, y: 100, width: 300, height: 300 }, + inner: { x: 20, y: 30, width: 200, height: 200 }, + leaf: { x: 5, y: 7, width: 10, height: 10 }, + }; + const parentOf = new Map([ + ["inner", "outer"], + ["leaf", "inner"], + ]); + const resolved = resolveAbsolutePositions(positions, parentOf); + expect(resolved.outer).toMatchObject({ x: 100, y: 100 }); + // inner: outer's origin plus inner's own offset. + expect(resolved.inner).toMatchObject({ x: 120, y: 130 }); + // leaf: inner's *absolute* origin (120, 130), not inner's raw + // group-relative offset (20, 30), plus leaf's own offset. Summing + // against the wrong (non-recursive) base would give (25, 37) instead + // of (125, 137), so this is the case that would fail if the resolution + // only walked up a single level. + expect(resolved.leaf).toMatchObject({ x: 125, y: 137 }); + }); + + it("leaves a node with no resolvable parent position undefined", () => { + const positions: Positions = { + lone: { x: 42, y: 43, width: 1, height: 1 }, + member: { x: 1, y: 1, width: 1, height: 1 }, + }; + const parentOf = new Map([["member", "group-not-in-positions"]]); + const resolved = resolveAbsolutePositions(positions, parentOf); + expect(resolved.lone).toMatchObject({ x: 42, y: 43 }); + // The member's parent has no known position (e.g. hidden or not yet + // laid out), so its absolute position can't be resolved either, rather + // than silently treating the unresolved offset as already absolute. + expect(resolved.member).toBeUndefined(); + }); +}); diff --git a/console/src/platform/dataflows/elkGraph.ts b/console/src/platform/dataflows/elkGraph.ts new file mode 100644 index 0000000000000..c16716ae59008 --- /dev/null +++ b/console/src/platform/dataflows/elkGraph.ts @@ -0,0 +1,157 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import type { ElkNode } from "elkjs/lib/elk-api"; + +import type { LirGrouping, VisibleGraph, VisibleNode } from "./dataflowGraph"; + +export interface NodePosition { + x: number; + y: number; + width: number; + height: number; +} + +export type Positions = Record; + +export const NODE_DIMENSIONS: Record< + VisibleNode["kind"], + { width: number; height: number } +> = { + operator: { width: 240, height: 72 }, + region: { width: 260, height: 96 }, + port: { width: 90, height: 24 }, +}; + +const LAYOUT_OPTIONS: Record = { + "elk.algorithm": "layered", + "elk.direction": "DOWN", + "elk.padding": "[top=48,left=16,bottom=16,right=16]", + "elk.spacing.nodeNode": "24", + "elk.layered.spacing.nodeNodeBetweenLayers": "48", + // Lets an edge crossing a LIR group's boundary be declared uniformly at + // the root: elk relocates it to the correct container itself. Harmless + // when there's no grouping at all (today's flat views). + "elk.hierarchyHandling": "INCLUDE_CHILDREN", +}; + +// A LIR group is a label-only wrapper, sized purely from its content (no +// fixed width/height), with top padding reserved for the header strip +// LirGroupNode renders. +const GROUP_LAYOUT_OPTIONS: Record = { + "elk.algorithm": "layered", + "elk.direction": "DOWN", + "elk.padding": "[top=24,left=8,bottom=8,right=8]", + "elk.spacing.nodeNode": "24", + "elk.layered.spacing.nodeNodeBetweenLayers": "48", +}; + +// A view is always one scope's direct children, so without a grouping this +// is flat: nothing is nested inside anything else. With a grouping, a +// group's members (VisibleNodes or other groups) become its elk children; +// everything else stays a direct child of the root, exactly like today. +export function toElkGraph( + graph: VisibleGraph, + grouping?: LirGrouping, +): ElkNode { + const nodeById = new Map(graph.nodes.map((n) => [n.id, n])); + const groupById = new Map((grouping?.groups ?? []).map((g) => [g.id, g])); + const childrenOf = new Map(); // parent id ("" = root) -> ordered child ids + const addChild = (parentKey: string, id: string) => { + const list = childrenOf.get(parentKey); + if (list) list.push(id); + else childrenOf.set(parentKey, [id]); + }; + for (const n of graph.nodes) + addChild(grouping?.parentOf.get(n.id) ?? "", n.id); + for (const g of groupById.values()) + addChild(grouping?.parentOf.get(g.id) ?? "", g.id); + + const buildNode = (id: string): ElkNode => { + const group = groupById.get(id); + if (group) { + return { + id, + layoutOptions: GROUP_LAYOUT_OPTIONS, + children: (childrenOf.get(id) ?? []).map(buildNode), + }; + } + const node = nodeById.get(id); + if (!node) { + throw new Error(`missing visible node for id ${id} building elk graph`); + } + return { id, ...NODE_DIMENSIONS[node.kind] }; + }; + + return { + id: "__root__", + layoutOptions: LAYOUT_OPTIONS, + children: (childrenOf.get("") ?? []).map(buildNode), + edges: graph.edges.map((e) => ({ + id: e.id, + sources: [e.source], + targets: [e.target], + })), + }; +} + +// Recurses into nested children: elk reports every node's x/y already +// relative to its own immediate parent, the same convention React Flow uses +// for a node with parentId set, so no coordinate conversion happens here — +// this only flattens the tree into one lookup map, keyed by id at any depth. +export function extractPositions(layouted: ElkNode): Positions { + const positions: Positions = {}; + const visit = (node: ElkNode) => { + for (const child of node.children ?? []) { + positions[child.id] = { + x: child.x ?? 0, + y: child.y ?? 0, + width: child.width ?? 0, + height: child.height ?? 0, + }; + visit(child); + } + }; + visit(layouted); + return positions; +} + +// extractPositions above deliberately leaves nested positions parent-relative +// (elk's convention, matching React Flow's parentId semantics), which is +// exactly right for feeding React Flow but wrong for any consumer that +// compares positions against absolute canvas coordinates instead (e.g. a +// viewport-visibility check reading reactFlow.getViewport()). This resolves +// each node up its parent chain, summing offsets, before such a comparison. +// Nesting can go arbitrarily deep, so it recurses rather than assuming one +// level. +export function resolveAbsolutePositions( + positions: Positions, + parentOf: LirGrouping["parentOf"] | undefined, +): Positions { + if (!parentOf || parentOf.size === 0) return positions; + const resolved: Positions = {}; + const resolve = (id: string): NodePosition | undefined => { + const cached = resolved[id]; + if (cached) return cached; + const p = positions[id]; + if (!p) return undefined; + const parentId = parentOf.get(id); + if (!parentId) { + resolved[id] = p; + return p; + } + const parentPos = resolve(parentId); + if (!parentPos) return undefined; + const abs = { ...p, x: p.x + parentPos.x, y: p.y + parentPos.y }; + resolved[id] = abs; + return abs; + }; + for (const id of Object.keys(positions)) resolve(id); + return resolved; +} diff --git a/console/src/platform/dataflows/nodeStyle.test.ts b/console/src/platform/dataflows/nodeStyle.test.ts new file mode 100644 index 0000000000000..95356149acb99 --- /dev/null +++ b/console/src/platform/dataflows/nodeStyle.test.ts @@ -0,0 +1,187 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { describe, expect, it } from "vitest"; + +import { + formatSkew, + hashString, + lirGroupColor, + nodeFillColor, + operatorColor, + prettyPrintChannelType, + textColorFor, +} from "./nodeStyle"; + +const PALETTE = ["#111111", "#222222", "#333333", "#444444"]; +const ACCENT = { arranged: "#aaaaaa", notArranged: "#bbbbbb" }; + +describe("hashString", () => { + it("is deterministic for the same string", () => { + expect(hashString("abc")).toEqual(hashString("abc")); + }); + + it("differs for different strings often enough to be useful", () => { + const hashes = new Set(["1", "2", "3", "4", "5", "6"].map(hashString)); + expect(hashes.size).toBeGreaterThan(1); + }); +}); + +describe("lirGroupColor", () => { + it("is deterministic for the same lirId", () => { + expect(lirGroupColor("42", PALETTE)).toEqual(lirGroupColor("42", PALETTE)); + }); + + it("differs for different lirIds often enough to be useful", () => { + const picked = new Set( + ["1", "2", "3", "4", "5", "6"].map((id) => lirGroupColor(id, PALETTE)), + ); + expect(picked.size).toBeGreaterThan(1); + }); + + it("only ever picks from the given palette", () => { + expect(PALETTE).toContain(lirGroupColor("some-lir-id", PALETTE)); + }); +}); + +describe("operatorColor", () => { + it("is deterministic for the same operator name", () => { + expect(operatorColor("Reduce", PALETTE)).toEqual( + operatorColor("Reduce", PALETTE), + ); + }); + + it("differs for different operator names often enough to be useful", () => { + const picked = new Set( + ["Map", "Filter", "Reduce", "Join", "FlatMap", "Arrange"].map((name) => + operatorColor(name, PALETTE), + ), + ); + expect(picked.size).toBeGreaterThan(1); + }); + + it("only ever picks from the given palette", () => { + expect(PALETTE).toContain(operatorColor("Reduce", PALETTE)); + }); +}); + +describe("nodeFillColor", () => { + const baseNode = { + id: "1", + label: "Reduce", + stats: null, + transitive: null, + own: null, + ownSkew: null, + transitiveSkew: null, + overheadNs: null, + childCount: 0, + lir: [], + address: null, + operatorId: null, + peers: [], + }; + + it("colors an operator by its name, not by whether it's arranged", () => { + const unarranged = nodeFillColor( + { ...baseNode, kind: "operator" }, + PALETTE, + ACCENT, + ); + const arranged = nodeFillColor( + { + ...baseNode, + kind: "operator", + transitive: { + arrangementRecords: 100n, + arrangementSize: 100n, + elapsedNs: 0n, + scheduleCount: 0n, + }, + }, + PALETTE, + ACCENT, + ); + expect(unarranged).toEqual(operatorColor("Reduce", PALETTE)); + expect(arranged).toEqual(operatorColor("Reduce", PALETTE)); + }); + + it("still colors a region by whether it's arranged, unaffected by name", () => { + const noArrangement = nodeFillColor( + { ...baseNode, kind: "region" }, + PALETTE, + ACCENT, + ); + const arranged = nodeFillColor( + { + ...baseNode, + kind: "region", + transitive: { + arrangementRecords: 100n, + arrangementSize: 100n, + elapsedNs: 0n, + scheduleCount: 0n, + }, + }, + PALETTE, + ACCENT, + ); + expect(noArrangement).toEqual(ACCENT.notArranged); + expect(arranged).toEqual(ACCENT.arranged); + }); +}); + +describe("formatSkew", () => { + it("formats a real ratio to 2 decimal places", () => { + expect(formatSkew(1.5)).toEqual("1.50"); + expect(formatSkew(1.0244)).toEqual("1.02"); + }); + + it("labels the no-data sentinel (0) distinctly from a real ratio", () => { + expect(formatSkew(0)).toEqual("no data"); + }); +}); + +describe("textColorFor", () => { + it("picks light text on a dark background", () => { + // colors.lineGraph's purple[700], the color from the reported contrast bug + expect(textColorFor("#391D7E")).toEqual("#FFF"); + }); + + it("picks dark text on a light background", () => { + // colors.lineGraph's purple[200] + expect(textColorFor("#C8B5FF")).toEqual("#111"); + }); +}); + +describe("prettyPrintChannelType", () => { + it("strips module paths, aliases Diff/Error, and brackets Vec", () => { + expect( + prettyPrintChannelType( + "alloc::vec::Vec<(mz_repr::row::Row, mz_repr::timestamp::Timestamp, mz_ore::overflowing::Overflowing)>", + ), + ).toEqual("[(Row, Timestamp, Diff)]"); + }); + + it("handles the error-channel variant", () => { + expect( + prettyPrintChannelType( + "alloc::vec::Vec<(mz_compute::render::errors::DataflowErrorSer, mz_repr::timestamp::Timestamp, mz_ore::overflowing::Overflowing)>", + ), + ).toEqual("[(Error, Timestamp, Diff)]"); + }); + + it("unwraps nested Vec and unrecognized generics without mangling them", () => { + expect( + prettyPrintChannelType( + "alloc::vec::Vec)>>>>", + ), + ).toEqual("[Rc>>]"); + }); +}); diff --git a/console/src/platform/dataflows/nodeStyle.ts b/console/src/platform/dataflows/nodeStyle.ts new file mode 100644 index 0000000000000..a35f6e24d5fd6 --- /dev/null +++ b/console/src/platform/dataflows/nodeStyle.ts @@ -0,0 +1,153 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import colors from "~/theme/colors"; + +import type { LirGroupNode, VisibleNode } from "./dataflowGraph"; + +// Selection/match highlights read the same in light and dark mode (a bright +// ring reads fine against either canvas background), so these pin to the +// raw swatch module instead of a light/dark-aware theme token, only to keep +// them from drifting out of sync with the rest of the palette. +export const HIGHLIGHT_COLORS = { + selected: colors.blue[500], + activeMatch: colors.orange[400], + // Same hue as `selected`, for an edge that merely touches the selected + // node rather than being the clicked thing itself; ChannelEdge pairs this + // with a thinner stroke than `selected` gets, so the two read as a clear + // emphasis/de-emphasis pair rather than two unrelated colors. No exact + // swatch matches this shade; chosen for the pairing, not a token. + connected: "#8fd2f5", +}; + +/** Deterministic 32-bit string hash, for cycling a fixed color palette. */ +export function hashString(s: string): number { + let h = 0; + for (let i = 0; i < s.length; i++) + h = (Math.imul(h, 31) + s.charCodeAt(i)) | 0; + return h; +} + +// Cycled by a hash of lirId/operator name so the same id or name always +// gets the same color within one render (and across re-renders, since both +// are stable). Groups and operators share one palette: LirGroupNode's own +// border/label already marks a box as a group, so the fill color doesn't +// need to carry that distinction too. +export function lirGroupColor( + lirId: string, + palette: readonly string[], +): string { + return palette[Math.abs(hashString(lirId)) % palette.length]; +} + +export function operatorColor( + name: string, + palette: readonly string[], +): string { + return palette[Math.abs(hashString(name)) % palette.length]; +} + +// WCAG relative luminance of a "#RRGGBB" color, for picking readable text. +function relativeLuminance(hex: string): number { + const channels = [ + parseInt(hex.slice(1, 3), 16), + parseInt(hex.slice(3, 5), 16), + parseInt(hex.slice(5, 7), 16), + ].map((c) => { + const s = c / 255; + return s <= 0.03928 ? s / 12.92 : Math.pow((s + 0.055) / 1.055, 2.4); + }); + return 0.2126 * channels[0] + 0.7152 * channels[1] + 0.0722 * channels[2]; +} + +// colors.lineGraph mixes dark and light swatches in both themes (it's +// indexed by hash, not by theme), so a node's fill color carries no +// relationship to the current color mode. Text on top of it needs a +// contrast decision from the fill's actual luminance, not a theme token. +// Pins to raw black/white (not theme foreground tokens) since those exist +// specifically for text on the app's own background, not on an arbitrary +// hashed fill. +export function textColorFor(background: string): string { + const luminance = relativeLuminance(background); + // WCAG contrast ratio, (lighter + 0.05) / (darker + 0.05), against each + // candidate; picking the higher one is equivalent to maximizing contrast. + const contrastWithWhite = 1.05 / (luminance + 0.05); + const contrastWithBlack = (luminance + 0.05) / 0.05; + return contrastWithBlack > contrastWithWhite ? colors.black : colors.white; +} + +export function nodeFillColor( + node: VisibleNode, + palette: readonly string[], + accent: { arranged: string; notArranged: string }, +): string { + const region = node.kind !== "operator" && node.kind !== "port"; + if (!region) return operatorColor(node.label, palette); + const arranged = (node.transitive?.arrangementRecords ?? 0n) > 0n; + return arranged ? accent.arranged : accent.notArranged; +} + +// skewRatio's 0 is a "no per-worker data at all" sentinel, not a real ratio +// (a real ratio is always >= 1). Labeling it distinctly avoids it reading +// as an implausibly perfect score. +export function formatSkew(ratio: number): string { + return ratio === 0 ? "no data" : ratio.toFixed(2); +} + +export function formatElapsed(ns: bigint): string { + return `scheduled ${Math.round(Number(ns) / 1e9)}s`; +} + +// A channel's `type` column is the raw monomorphized Rust container type +// (e.g. "alloc::vec::Vec<(mz_repr::row::Row, mz_repr::timestamp::Timestamp, +// mz_ore::overflowing::Overflowing)>"). Path-qualified and generics-heavy, +// so this drops module paths down to the final segment, aliases the two +// generics that always mean the same thing in this position (the diff/count +// column, the decoded error variant), and turns `Vec` into `[X]` to match +// how we'd write the type by hand. Anything it doesn't recognize (batch/trace +// internals) still gets the path-stripping and Vec treatment, which is +// already most of the noise, even without a dedicated alias. +export function prettyPrintChannelType(raw: string): string { + let s = raw.replace(/(?:[A-Za-z_][A-Za-z0-9_]*::)+/g, ""); + s = s.replace(/\bOverflowing/g, "Diff"); + s = s.replace(/\bDataflowErrorSer\b/g, "Error"); + // `Vec<...>` -> `[...]`, repeatedly: each pass strips one level of nesting, + // so a Vec inside a Vec's contents surfaces on the next iteration. + for (;;) { + const start = s.indexOf("Vec<"); + if (start === -1) break; + let depth = 0; + let end = start + 3; + for (; end < s.length; end++) { + if (s[end] === "<") depth++; + else if (s[end] === ">" && --depth === 0) break; + } + s = + s.slice(0, start) + + "[" + + s.slice(start + 4, end) + + "]" + + s.slice(end + 1); + } + return s; +} + +export type FlowNodeData = { + node: VisibleNode; + dimmed: boolean; + color: string; + selected: boolean; + activeMatch: boolean; +}; + +export type FlowGroupData = { + group: LirGroupNode; + label: string; + color: string; +}; diff --git a/console/src/platform/dataflows/nodes.tsx b/console/src/platform/dataflows/nodes.tsx new file mode 100644 index 0000000000000..36a5a8b76aa08 --- /dev/null +++ b/console/src/platform/dataflows/nodes.tsx @@ -0,0 +1,173 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import { Badge, Box, HStack, Text, Tooltip, useTheme } from "@chakra-ui/react"; +import { Handle, type NodeProps, Position } from "@xyflow/react"; +import React from "react"; + +import { MaterializeTheme } from "~/theme"; +import { formatBytesShort } from "~/utils/format"; + +import type { VisibleNode } from "./dataflowGraph"; +import { + type FlowGroupData, + type FlowNodeData, + formatElapsed, + HIGHLIGHT_COLORS, + textColorFor, +} from "./nodeStyle"; + +// Records and size share one line, not two: the fixed node height only has +// room for name + a couple of stat lines before text spills out of the box. +const statLines = (node: VisibleNode) => { + const lines: string[] = []; + const stats = node.stats; + if (!stats) return lines; + if (stats.arrangementRecords > 0n) { + lines.push( + `${stats.arrangementRecords} rec · ${formatBytesShort(stats.arrangementSize)}`, + ); + } + if (stats.elapsedNs > 0n) lines.push(formatElapsed(stats.elapsedNs)); + return lines; +}; + +// Selection (clicked) wins over the active search match when both apply to +// the same node; both are visually distinct from ordinary dimming. +const highlightShadow = (data: FlowNodeData): string | undefined => { + if (data.selected) return `0 0 0 2px ${HIGHLIGHT_COLORS.selected}`; + if (data.activeMatch) return `0 0 0 2px ${HIGHLIGHT_COLORS.activeMatch}`; + return undefined; +}; + +// A subtle resting shadow (matching the app's other node-graph view) so +// cards read as raised surfaces even when unselected; highlightShadow +// overrides it with a colored ring for selected/matched nodes. +const RESTING_SHADOW = "0px 0.5px 2.5px 0 rgba(0, 0, 0, 0.08)"; + +const CardShell = ({ + data, + children, +}: { + data: FlowNodeData; + children: React.ReactNode; +}) => ( + + {children} + + + +); + +export const OperatorNode = ({ data }: NodeProps & { data: FlowNodeData }) => ( + + `LIR ${l.lirId}: ${l.operator}`) + .join(", ")} + isDisabled={data.node.lir.length === 0} + > + + + {data.node.label} + + {statLines(data.node).map((line) => ( + + {line} + + ))} + + + +); + +// Always shown collapsed (a scope is never expanded in place); double-click +// navigates into it instead of unfolding it here. +export const RegionNode = ({ data }: NodeProps & { data: FlowNodeData }) => ( + + + + {data.node.label} + + + {data.node.childCount} + + + {statLines(data.node).map((line) => ( + + {line} + + ))} + +); + +export const PortNode = ({ data }: NodeProps & { data: FlowNodeData }) => { + const { colors } = useTheme(); + return ( + + {data.node.label} + + + + ); +}; + +// A label-only wrapper around its members. Header is clickable +// (pointerEvents="auto"), body is not (pointerEvents="none"). For clicks on +// the body to pass through to underlying members, the node object this +// component is instantiated with must also set style={{pointerEvents:"none"}}, +// since React Flow's own node wrapper defaults to pointer-events:all and +// this component's root cannot override an ancestor. Its width/height come +// from elk's auto-sized bounds, same as every other node in this file. +export const LirGroupNode = ({ data }: NodeProps & { data: FlowGroupData }) => ( + + + + + {data.label} + + + +); diff --git a/console/src/platform/dataflows/useElkLayout.ts b/console/src/platform/dataflows/useElkLayout.ts new file mode 100644 index 0000000000000..f2530efcece86 --- /dev/null +++ b/console/src/platform/dataflows/useElkLayout.ts @@ -0,0 +1,81 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import ELK, { type ElkNode } from "elkjs/lib/elk-api.js"; +// The elk engine detects a worker context at load time and registers itself +// as the worker's message handler, so it must be the worker entry point +// itself. A hand-written worker wrapping it cannot work: imported into +// another worker the engine hijacks that worker's onmessage and exports +// nothing. elk-api on the main thread speaks to it and correlates +// request/response pairs internally. +import ElkWorker from "elkjs/lib/elk-worker.min.js?worker"; +import React from "react"; + +import type { LirGrouping, VisibleGraph } from "./dataflowGraph"; +import { extractPositions, type Positions, toElkGraph } from "./elkGraph"; + +export function useElkLayout( + graph: VisibleGraph | null, + cacheKey: string, + grouping?: LirGrouping, +) { + const elkRef = React.useRef | null>(null); + const requestIdRef = React.useRef(0); + const cacheRef = React.useRef(new Map()); + const [state, setState] = React.useState<{ + key: string | null; + positions: Positions | null; + error: string | null; + }>({ key: null, positions: null, error: null }); + // Bumped by retry() to force the layout effect to run again for the same + // (graph, cacheKey): a failed layout is never cached, so re-running it is + // enough to retry, but the effect's own dependencies wouldn't otherwise + // change. + const [retryNonce, setRetryNonce] = React.useState(0); + + React.useEffect(() => { + const elk = new ELK({ workerFactory: () => new ElkWorker() }); + elkRef.current = elk; + return () => { + elk.terminateWorker(); + }; + }, []); + + React.useEffect(() => { + if (!graph) return; + const cached = cacheRef.current.get(cacheKey); + if (cached) { + setState({ key: cacheKey, positions: cached, error: null }); + return; + } + const elk = elkRef.current; + if (!elk) return; + const requestId = ++requestIdRef.current; + elk.layout(toElkGraph(graph, grouping)).then( + (layouted: ElkNode) => { + // Drop responses for superseded requests. + if (requestId !== requestIdRef.current) return; + const positions = extractPositions(layouted); + cacheRef.current.set(cacheKey, positions); + setState({ key: cacheKey, positions, error: null }); + }, + (error: unknown) => { + if (requestId !== requestIdRef.current) return; + setState({ key: cacheKey, positions: null, error: String(error) }); + }, + ); + }, [graph, cacheKey, retryNonce, grouping]); + + return { + positions: state.key === cacheKey ? state.positions : null, + layouting: graph !== null && state.key !== cacheKey, + error: state.key === cacheKey ? state.error : null, + retry: () => setRetryNonce((n) => n + 1), + }; +} diff --git a/console/src/platform/object-explorer/SimpleObjectDetailRoutes.tsx b/console/src/platform/object-explorer/SimpleObjectDetailRoutes.tsx index c93baafffb064..74653e3427ec4 100644 --- a/console/src/platform/object-explorer/SimpleObjectDetailRoutes.tsx +++ b/console/src/platform/object-explorer/SimpleObjectDetailRoutes.tsx @@ -22,8 +22,11 @@ import { ShowCreateBlock } from "~/components/ShowCreateBlock"; import WorkflowGraph from "~/components/WorkflowGraph/WorkflowGraph"; import { useFlags } from "~/hooks/useFlags"; import { MainContentContainer, Tab } from "~/layouts/BaseLayout"; +import { absoluteClusterPath } from "~/platform/routeHelpers"; import { SentryRoutes } from "~/sentry"; +import { useAllClusters } from "~/store/allClusters"; import { useAllObjects } from "~/store/allObjects"; +import { useRegionSlug } from "~/store/environments"; import { ObjectDetailsContainer, ObjectDetailsStrip } from "./detailComponents"; import { ObjectColumns } from "./ObjectColumns"; @@ -34,10 +37,6 @@ import { SourceOverviewContainer } from "./SourceDetailRoutes"; import { useSchemaObjectParams } from "./useSchemaObjectParams"; import { useToastIfObjectNotExtant } from "./useToastIfObjectNotExtant"; -const DataflowVisualizer = React.lazy( - () => import("~/platform/clusters/DataflowVisualizer"), -); - const SIMPLE_OBJECTS_WITH_INDEXES = ["materialized-view", "view", "table"]; const DATAFLOW_VISUALIZER_OBJECTS = ["materialized-view", "index"]; @@ -102,6 +101,8 @@ export const SimpleObjectDetailRoutes = () => { useToastIfObjectNotExtant(); const { data: objects } = useAllObjects(); const object = objects.find((o) => o.id === id); + const regionSlug = useRegionSlug(); + const { getClusterById } = useAllClusters(); const dataflowVisualizerEnabled = flags["visualization-features"]; @@ -158,10 +159,21 @@ export const SimpleObjectDetailRoutes = () => { }); } if (shouldShowDataflowVisualizer) { - tabStripItems.push({ - label: "Visualize", - href: `../dataflow-visualizer`, - }); + // The visualizer lives on the owning cluster's dataflows page, which + // resolves the object's running dataflow from the ?export param. Skip the + // tab when the cluster is not resolvable rather than link somewhere broken. + const cluster = object.clusterId + ? getClusterById(object.clusterId) + : undefined; + if (cluster) { + tabStripItems.push({ + label: "Visualize", + href: `${absoluteClusterPath(regionSlug, { + id: cluster.id, + name: cluster.name, + })}/dataflows?export=${object.id}`, + }); + } } return ( @@ -200,9 +212,6 @@ export const SimpleObjectDetailRoutes = () => { {shouldShowIndexes && ( } /> )} - {shouldShowDataflowVisualizer && ( - } /> - )} } /> diff --git a/console/src/test/mockReactFlow.tsx b/console/src/test/mockReactFlow.tsx new file mode 100644 index 0000000000000..e1d3343503fbb --- /dev/null +++ b/console/src/test/mockReactFlow.tsx @@ -0,0 +1,45 @@ +// Copyright Materialize, Inc. and contributors. All rights reserved. +// +// Use of this software is governed by the Business Source License +// included in the LICENSE file. +// +// As of the Change Date specified in that file, in accordance with +// the Business Source License, use of this software will be governed +// by the Apache License, Version 2.0. + +import React from "react"; +import { vi } from "vitest"; + +// jsdom lacks ResizeObserver/DOMMatrixReadOnly, so component tests mock +// @xyflow/react to a flat renderer that preserves the decision logic under +// test (which nodes and labels exist) instead of exercising real layout. +export function mockReactFlow() { + vi.mock("@xyflow/react", () => ({ + ReactFlow: ({ + nodes, + children, + }: { + nodes: { id: string; data: { node: { label: string } } }[]; + children?: React.ReactNode; + }) => ( +
+ {nodes.map((n) => ( +
+ {n.data.node.label} +
+ ))} + {children} +
+ ), + Background: () => null, + Controls: () => null, + MiniMap: () => null, + Handle: () => null, + Position: { Left: "left", Right: "right" }, + BaseEdge: () => null, + EdgeLabelRenderer: ({ children }: { children: React.ReactNode }) => ( + <>{children} + ), + getBezierPath: () => ["", 0, 0], + })); +} diff --git a/console/src/utils/format.test.ts b/console/src/utils/format.test.ts index b72a989ba5876..12cd9f52fcff5 100644 --- a/console/src/utils/format.test.ts +++ b/console/src/utils/format.test.ts @@ -10,12 +10,32 @@ import parse from "postgres-interval"; import { + formatElapsedNs, formatInterval, formatIntervalShort, fromSeconds, toSeconds, } from "./format"; +describe("formatElapsedNs", () => { + it("formats zero as a plain 0s", () => { + expect(formatElapsedNs(0n)).toEqual("0s"); + }); + + it("shows sub-second durations in milliseconds instead of collapsing to 0s", () => { + expect(formatElapsedNs(257_000_000n)).toEqual("257ms"); + }); + + it("shows single-digit-second durations with one decimal place", () => { + expect(formatElapsedNs(1_400_000_000n)).toEqual("1.4s"); + expect(formatElapsedNs(1_600_000_000n)).toEqual("1.6s"); + }); + + it("rounds durations of 10s or more to whole seconds", () => { + expect(formatElapsedNs(543_400_000_000n)).toEqual("543s"); + }); +}); + describe("formatIntervalShort", () => { it("only returns years if present", () => { const interval = parse("1 years 2 mons 3 days 04:05:06.789"); diff --git a/console/src/utils/format.ts b/console/src/utils/format.ts index a65076cc62944..2fea2d04a61af 100644 --- a/console/src/utils/format.ts +++ b/console/src/utils/format.ts @@ -60,6 +60,21 @@ export const formatBytesShort = (bytes: bigint) => { return `${numberFormatter.format(value)} ${unit}`; }; +/** + * Formats a nanosecond duration for display. Rounding straight to whole + * seconds collapses every sub-second duration to "0s" and can't + * distinguish e.g. 1.4s from 1.6s, which matters for a dataflow operator's + * own (as opposed to subtree) elapsed time -- often sub-second even when + * its ancestors run for minutes. + */ +export function formatElapsedNs(ns: bigint): string { + const seconds = Number(ns) / 1e9; + if (seconds === 0) return "0s"; + if (seconds < 1) return `${Math.round(seconds * 1000)}ms`; + if (seconds < 10) return `${seconds.toFixed(1)}s`; + return `${Math.round(seconds)}s`; +} + export function convertBytes(bytes: number, unit: ByteUnit) { switch (unit) { case "B": diff --git a/console/yarn.lock b/console/yarn.lock index bda1f83709b4f..eb488e48eb09e 100644 --- a/console/yarn.lock +++ b/console/yarn.lock @@ -2769,17 +2769,6 @@ __metadata: languageName: node linkType: hard -"@hpcc-js/wasm@npm:^2.20.0": - version: 2.20.0 - resolution: "@hpcc-js/wasm@npm:2.20.0" - dependencies: - yargs: "npm:17.7.2" - bin: - dot-wasm: node ./node_modules/@hpcc-js/wasm-graphviz-cli/bin/index.js - checksum: 10c0/925ce14d47389a99837b4fc7d97d270cd401a72e089c55d5fac62212ae652830c4f3f54475e46685a4fed0658c39598a82dd182468bfa603d29d647c419f1ea5 - languageName: node - linkType: hard - "@humanwhocodes/config-array@npm:^0.11.14": version: 0.11.14 resolution: "@humanwhocodes/config-array@npm:0.11.14" @@ -4451,13 +4440,6 @@ __metadata: languageName: node linkType: hard -"@types/d3-color@npm:^1": - version: 1.4.2 - resolution: "@types/d3-color@npm:1.4.2" - checksum: 10c0/f1c70d7deabe2b30e337361c3fa26b7ce4e9c875dfd1ad75e17379c6a5596f116fb21ca0e595c2f2220e04866bca6bdb2623e772cfb5f8bf4d20ffe6ba5c72ab - languageName: node - linkType: hard - "@types/d3-contour@npm:*": version: 3.0.2 resolution: "@types/d3-contour@npm:3.0.2" @@ -4491,6 +4473,15 @@ __metadata: languageName: node linkType: hard +"@types/d3-drag@npm:^3.0.7": + version: 3.0.7 + resolution: "@types/d3-drag@npm:3.0.7" + dependencies: + "@types/d3-selection": "npm:*" + checksum: 10c0/65e29fa32a87c72d26c44b5e2df3bf15af21cd128386bcc05bcacca255927c0397d0cd7e6062aed5f0abd623490544a9d061c195f5ed9f018fe0b698d99c079d + languageName: node + linkType: hard + "@types/d3-dsv@npm:*": version: 3.0.1 resolution: "@types/d3-dsv@npm:3.0.1" @@ -4546,17 +4537,6 @@ __metadata: languageName: node linkType: hard -"@types/d3-graphviz@npm:^2.6.10": - version: 2.6.10 - resolution: "@types/d3-graphviz@npm:2.6.10" - dependencies: - "@types/d3-selection": "npm:^1" - "@types/d3-transition": "npm:^1" - "@types/d3-zoom": "npm:^1" - checksum: 10c0/bbb4143dcef1ebac612f93e448a53291cec3f708489261dc54c657db0441ed0a87c3bdf5e902e6d695918b7a39ace15bcf07e2c828d6568611eee8b5c15fe6c4 - languageName: node - linkType: hard - "@types/d3-hierarchy@npm:*": version: 3.1.2 resolution: "@types/d3-hierarchy@npm:3.1.2" @@ -4573,12 +4553,12 @@ __metadata: languageName: node linkType: hard -"@types/d3-interpolate@npm:^1": - version: 1.4.2 - resolution: "@types/d3-interpolate@npm:1.4.2" +"@types/d3-interpolate@npm:^3.0.4": + version: 3.0.4 + resolution: "@types/d3-interpolate@npm:3.0.4" dependencies: - "@types/d3-color": "npm:^1" - checksum: 10c0/98bff93ce4d94485a4f6117e554854ec69072382910008e785d2c960b50e643093a8cfa2e0875b3d1dff19f3603b6e16a4eea8122c7c8ead3623daf3044cd22e + "@types/d3-color": "npm:*" + checksum: 10c0/066ebb8da570b518dd332df6b12ae3b1eaa0a7f4f0c702e3c57f812cf529cc3500ec2aac8dc094f31897790346c6b1ebd8cd7a077176727f4860c2b181a65ca4 languageName: node linkType: hard @@ -4649,10 +4629,10 @@ __metadata: languageName: node linkType: hard -"@types/d3-selection@npm:^1": - version: 1.4.3 - resolution: "@types/d3-selection@npm:1.4.3" - checksum: 10c0/47c181f8362ade4df151e01737816356c939bc5728ff87c3a29bd43aaa0413296170119949eb3aa0ef8d9c10fac4463eb1d154b6fd0e89617b45eeb06bdefb8b +"@types/d3-selection@npm:^3.0.10": + version: 3.0.11 + resolution: "@types/d3-selection@npm:3.0.11" + checksum: 10c0/0c512956c7503ff5def4bb32e0c568cc757b9a2cc400a104fc0f4cfe5e56d83ebde2a97821b6f2cb26a7148079d3b86a2f28e11d68324ed311cf35c2ed980d1d languageName: node linkType: hard @@ -4711,12 +4691,12 @@ __metadata: languageName: node linkType: hard -"@types/d3-transition@npm:^1": - version: 1.3.2 - resolution: "@types/d3-transition@npm:1.3.2" +"@types/d3-transition@npm:^3.0.8": + version: 3.0.9 + resolution: "@types/d3-transition@npm:3.0.9" dependencies: - "@types/d3-selection": "npm:^1" - checksum: 10c0/9e1340c2840fde63f224550cbde5531b8b2493218d421f97de138c1e7ca9b1f481dfc9e4c91cb5ce84df9df671e2d95468556be1f8cf1e28f0c33929322115a4 + "@types/d3-selection": "npm:*" + checksum: 10c0/4f68b9df7ac745b3491216c54203cbbfa0f117ae4c60e2609cdef2db963582152035407fdff995b10ee383bae2f05b7743493f48e1b8e46df54faa836a8fb7b5 languageName: node linkType: hard @@ -4730,13 +4710,13 @@ __metadata: languageName: node linkType: hard -"@types/d3-zoom@npm:^1": - version: 1.8.3 - resolution: "@types/d3-zoom@npm:1.8.3" +"@types/d3-zoom@npm:^3.0.8": + version: 3.0.8 + resolution: "@types/d3-zoom@npm:3.0.8" dependencies: - "@types/d3-interpolate": "npm:^1" - "@types/d3-selection": "npm:^1" - checksum: 10c0/d93ab0b610a68c02926c8388ed839fcc31ac0ee2ed19c69c59b5ea0cb87afd70515d507fc10c063eba5fc400389b3216f1bcdfe97ad84b75eac6c30e67cbccdf + "@types/d3-interpolate": "npm:*" + "@types/d3-selection": "npm:*" + checksum: 10c0/1dbdbcafddcae12efb5beb6948546963f29599e18bc7f2a91fb69cc617c2299a65354f2d47e282dfb86fec0968406cd4fb7f76ba2d2fb67baa8e8d146eb4a547 languageName: node linkType: hard @@ -5556,6 +5536,44 @@ __metadata: languageName: node linkType: hard +"@xyflow/react@npm:^12.11.1": + version: 12.11.1 + resolution: "@xyflow/react@npm:12.11.1" + dependencies: + "@xyflow/system": "npm:0.0.78" + classcat: "npm:^5.0.3" + zustand: "npm:^4.4.0" + peerDependencies: + "@types/react": ">=17" + "@types/react-dom": ">=17" + react: ">=17" + react-dom: ">=17" + peerDependenciesMeta: + "@types/react": + optional: true + "@types/react-dom": + optional: true + checksum: 10c0/5774a01a28289effd8f5f3427265e1ddb34911557efbd2bc82d3648641369e5a68d56d992596980bfc2c95ea64252ac983b82412dad1cb2f2175a38b9cefeb68 + languageName: node + linkType: hard + +"@xyflow/system@npm:0.0.78": + version: 0.0.78 + resolution: "@xyflow/system@npm:0.0.78" + dependencies: + "@types/d3-drag": "npm:^3.0.7" + "@types/d3-interpolate": "npm:^3.0.4" + "@types/d3-selection": "npm:^3.0.10" + "@types/d3-transition": "npm:^3.0.8" + "@types/d3-zoom": "npm:^3.0.8" + d3-drag: "npm:^3.0.0" + d3-interpolate: "npm:^3.0.1" + d3-selection: "npm:^3.0.0" + d3-zoom: "npm:^3.0.0" + checksum: 10c0/1c4b497b5fcc86457a5b5120371223131036bbc3aa0ed0b0de5dac7f5db9f6198312839f2363a4a18c2bb0ba0dfe6ea4bf92701913a2a289d74da59da07b6a96 + languageName: node + linkType: hard + "@zag-js/dom-query@npm:0.16.0": version: 0.16.0 resolution: "@zag-js/dom-query@npm:0.16.0" @@ -6352,6 +6370,13 @@ __metadata: languageName: node linkType: hard +"classcat@npm:^5.0.3": + version: 5.0.5 + resolution: "classcat@npm:5.0.5" + checksum: 10c0/ff8d273055ef9b518529cfe80fd0486f7057a9917373807ff802d75ceb46e8f8e148f41fa094ee7625c8f34642cfaa98395ff182d9519898da7cbf383d4a210d + languageName: node + linkType: hard + "classnames@npm:^2.3.1": version: 2.3.2 resolution: "classnames@npm:2.3.2" @@ -6855,14 +6880,14 @@ __metadata: languageName: node linkType: hard -"d3-dispatch@npm:1 - 3, d3-dispatch@npm:3, d3-dispatch@npm:^3.0.1": +"d3-dispatch@npm:1 - 3, d3-dispatch@npm:3": version: 3.0.1 resolution: "d3-dispatch@npm:3.0.1" checksum: 10c0/6eca77008ce2dc33380e45d4410c67d150941df7ab45b91d116dbe6d0a3092c0f6ac184dd4602c796dc9e790222bad3ff7142025f5fd22694efe088d1d941753 languageName: node linkType: hard -"d3-drag@npm:2 - 3, d3-drag@npm:3": +"d3-drag@npm:2 - 3, d3-drag@npm:3, d3-drag@npm:^3.0.0": version: 3.0.0 resolution: "d3-drag@npm:3.0.0" dependencies: @@ -6920,7 +6945,7 @@ __metadata: languageName: node linkType: hard -"d3-format@npm:1 - 3, d3-format@npm:3, d3-format@npm:3.1.0, d3-format@npm:^3.1.0": +"d3-format@npm:1 - 3, d3-format@npm:3, d3-format@npm:3.1.0": version: 3.1.0 resolution: "d3-format@npm:3.1.0" checksum: 10c0/049f5c0871ebce9859fc5e2f07f336b3c5bfff52a2540e0bac7e703fce567cd9346f4ad1079dd18d6f1e0eaa0599941c1810898926f10ac21a31fd0a34b4aa75 @@ -6936,24 +6961,6 @@ __metadata: languageName: node linkType: hard -"d3-graphviz@npm:^5.6.0": - version: 5.6.0 - resolution: "d3-graphviz@npm:5.6.0" - dependencies: - "@hpcc-js/wasm": "npm:^2.20.0" - d3-dispatch: "npm:^3.0.1" - d3-format: "npm:^3.1.0" - d3-interpolate: "npm:^3.0.1" - d3-path: "npm:^3.1.0" - d3-timer: "npm:^3.0.1" - d3-transition: "npm:^3.0.1" - d3-zoom: "npm:^3.0.0" - peerDependencies: - d3-selection: ^3.0.0 - checksum: 10c0/ffbe5facb35cfe80b9bc1985acbd1facdab180c0eb0cfaccaf6f662e9f1d6be6bd8d20539a5fa04f555075b0c511154ebb567e96335b9d3283874f9f3b5b2744 - languageName: node - linkType: hard - "d3-hierarchy@npm:3": version: 3.1.2 resolution: "d3-hierarchy@npm:3.1.2" @@ -7028,7 +7035,7 @@ __metadata: languageName: node linkType: hard -"d3-selection@npm:2 - 3, d3-selection@npm:3": +"d3-selection@npm:2 - 3, d3-selection@npm:3, d3-selection@npm:^3.0.0": version: 3.0.0 resolution: "d3-selection@npm:3.0.0" checksum: 10c0/e59096bbe8f0cb0daa1001d9bdd6dbc93a688019abc97d1d8b37f85cd3c286a6875b22adea0931b0c88410d025563e1643019161a883c516acf50c190a11b56b @@ -7071,14 +7078,14 @@ __metadata: languageName: node linkType: hard -"d3-timer@npm:1 - 3, d3-timer@npm:3, d3-timer@npm:^3.0.1": +"d3-timer@npm:1 - 3, d3-timer@npm:3": version: 3.0.1 resolution: "d3-timer@npm:3.0.1" checksum: 10c0/d4c63cb4bb5461d7038aac561b097cd1c5673969b27cbdd0e87fa48d9300a538b9e6f39b4a7f0e3592ef4f963d858c8a9f0e92754db73116770856f2fc04561a languageName: node linkType: hard -"d3-transition@npm:2 - 3, d3-transition@npm:3, d3-transition@npm:^3.0.1": +"d3-transition@npm:2 - 3, d3-transition@npm:3": version: 3.0.1 resolution: "d3-transition@npm:3.0.1" dependencies: @@ -7591,6 +7598,13 @@ __metadata: languageName: node linkType: hard +"elkjs@npm:^0.11.1": + version: 0.11.1 + resolution: "elkjs@npm:0.11.1" + checksum: 10c0/e2216a7439c9dc65413e5f8d9e46bf829c32dacbfd869585bae8f8775b3cd5802d0ac9ccf0f45ad14decfa622244a092533263cc3a596e6d82e8247bf6bf84d7 + languageName: node + linkType: hard + "emoji-regex@npm:^10.3.0": version: 10.3.0 resolution: "emoji-regex@npm:10.3.0" @@ -10958,7 +10972,6 @@ __metadata: "@testing-library/user-event": "npm:^14.5.2" "@types/base32-encoding": "npm:^1.0.2" "@types/d3": "npm:^7.4.3" - "@types/d3-graphviz": "npm:^2.6.10" "@types/dagre": "npm:^0.7.54" "@types/debug": "npm:^4.1.12" "@types/jsonwebtoken": "npm:^9" @@ -10986,6 +10999,7 @@ __metadata: "@visx/tooltip": "npm:^3.12.0" "@vitejs/plugin-react": "npm:^4.5.0" "@xstate/fsm": "npm:^2.1.0" + "@xyflow/react": "npm:^12.11.1" base32-encoding: "npm:^1.0.0" browserslist-to-esbuild: "npm:^2.1.1" buffer: "npm:^6.0.3" @@ -10994,13 +11008,13 @@ __metadata: copy-to-clipboard: "npm:^3.3.3" core-js: "npm:^3.38.1" d3: "npm:^7.9.0" - d3-graphviz: "npm:^5.6.0" d3-scale-chromatic: "npm:^3.1.0" date-fns: "npm:^4.1.0" date-fns-tz: "npm:^3.2.0" debug: "npm:^4.3.7" dedent: "npm:^1.7.2" downshift: "npm:^9.0.8" + elkjs: "npm:^0.11.1" eslint: "npm:^8.57.0" eslint-config-prettier: "npm:^9.1.0" eslint-plugin-import: "npm:^2.31.0" @@ -15620,7 +15634,7 @@ __metadata: languageName: node linkType: hard -"yargs@npm:17.7.2, yargs@npm:^17.7.2": +"yargs@npm:^17.7.2": version: 17.7.2 resolution: "yargs@npm:17.7.2" dependencies: @@ -15681,3 +15695,23 @@ __metadata: checksum: 10c0/8f14c87d6b1b53c944c25ce7a28616896319d95bc46a9660fe441adc0ed0a81253b02b5abdaeffedbeb23bdd25a0bf1c29d2c12dd919aef6447652dd295e3e69 languageName: node linkType: hard + +"zustand@npm:^4.4.0": + version: 4.5.7 + resolution: "zustand@npm:4.5.7" + dependencies: + use-sync-external-store: "npm:^1.2.2" + peerDependencies: + "@types/react": ">=16.8" + immer: ">=9.0.6" + react: ">=16.8" + peerDependenciesMeta: + "@types/react": + optional: true + immer: + optional: true + react: + optional: true + checksum: 10c0/55559e37a82f0c06cadc61cb08f08314c0fe05d6a93815e41e3376130c13db22a5017cbb0cd1f018c82f2dad0051afe3592561d40f980bd4082e32005e8a950c + languageName: node + linkType: hard