Skip to content

feat(computer-use): keep overlay cursor elevated when the aim point is covered - #5052

Open
wutongyuonce wants to merge 2 commits into
apache:mainfrom
wutongyuonce:feat/cu-overlay-occlusion
Open

feat(computer-use): keep overlay cursor elevated when the aim point is covered#5052
wutongyuonce wants to merge 2 commits into
apache:mainfrom
wutongyuonce:feat/cu-overlay-occlusion

Conversation

@wutongyuonce

Copy link
Copy Markdown

Summary

obscuringRects already travel from maka-cu through the observation record, and CuObservation says the agent cursor should test the point it is about to draw. keepElevated still sank whenever a window id existed, so a covered aim point could disappear under the covering window.

This PR:

  • adds obscuringRects to overlay context
  • copies them from the session observation record
  • stays elevated only when the aim point hits a rect
  • keeps today's sink behaviour when rects are missing or the point is clear

Does not change the desktop sink z-order implementation. Elevation is per point, not per whole window.

Fixes #5048

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

Verification

  • npm --workspace @maka/runtime run build
  • npm --workspace @maka/computer-use run build
  • npm --workspace @maka/computer-use run test:dist — 122 pass, including hook-only point-in-rect cases and driveRealTool wiring (covered centre stays up, uncovered centre still sinks)
  • Did not run root lint / format:check / full typecheck
  • No overlay recording attached (behaviour is asserted in tests; desktop sink is unchanged)

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope:

pi-coding-agent drafted the overlay context wiring, point-in-rect keepElevated, tests, and this description. Commits carry Generated-by: pi-coding-agent.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

…s covered

obscuringRects already travel from maka-cu through the observation record,
and CuObservation says the agent cursor should test the point it is about
to draw. keepElevated still sank whenever a window id existed. Pass the
rects on overlay context and stay elevated only when that point hits one.

Generated-by: pi-coding-agent
Hook-only cases cannot see computer-use-tools copy the observation record
onto overlay context. Drive the real tool so a covered element centre
stays elevated and an uncovered one still sinks.

Generated-by: pi-coding-agent
@github-actions github-actions Bot added the effort/M Under 500 readable lines label Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/M Under 500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent cursor sinks behind covering windows during Computer Use

1 participant