Skip to content

Add a PR change risk indicator to the PR panel #833

@nicoalbanese

Description

@nicoalbanese

Summary

Add a change risk signal to the PR panel that classifies the current PR as Low, Medium, or High risk.

This is for design/spec discussion only for now — no implementation yet.

Desired behavior

  • Only analyze risk after a PR exists
  • Recompute when the PR head SHA changes
  • Reuse the cached result when the PR head SHA is unchanged
  • Show the result in the open PR panel
  • Initial output should be:
    • risk level: Low | Medium | High
    • short reason

Risk rubric

Risk should weight critical surface area more heavily than raw diff size.

Examples:

  • High: auth, permissions, secrets, billing, data writes/deletes, migrations, infra/config, security-sensitive code
  • Medium: data fetching, business logic, state management, routing, non-trivial backend behavior
  • Low: copy, styling, isolated presentational UI, tests-only, small local refactors

Recommended v1 approach

Use a hybrid approach instead of sending the full raw diff by default:

  1. Build a PR file manifest with:
    • path
    • status
    • additions/deletions
    • file type
  2. Derive local tags such as:
    • auth
    • api
    • migration
    • infra
    • ui
    • tests
  3. Use heuristics first for a baseline risk score
  4. Optionally send the manifest and small sanitized snippets for sensitive files to GPT-5.4 for the final classification

Security / privacy constraints

  • Treat diff contents as untrusted input
  • Use structured output only
  • Redact likely secrets before any model call
  • Skip binaries and generated files
  • Cap input size
  • Cache by PR head SHA to avoid repeated calls
  • Persist only the final risk result, not raw diff content used for prompting

Open questions

  • Should v1 use only file metadata/tags, or include small sanitized snippets for sensitive files?
  • Should the cached result live on the session record keyed by PR head SHA?
  • Should the UI eventually include impacted areas in addition to the short reason?

Acceptance criteria

  • Analysis timing is defined clearly
  • Low/medium/high rubric is documented
  • Model input is minimized and sanitized
  • Cache invalidation is based on PR head SHA
  • No implementation is started as part of this issue

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions