Skip to content

LLMs should not need grep or CLI execute permissions to use ql-mcp tools #120

@data-douser

Description

@data-douser

Problem

LLMs calling the ql-mcp server currently rely on grep, find, and other CLI commands for tasks that should be handled by MCP tools. This forces users to grant blanket CLI execute permissions, which is:

  • Slow — the user must manually approve each CLI invocation ("allow / allow / allow")
  • Insecure — blanket CLI access is an overly broad permission grant
  • Unreliable — LLMs construct ad-hoc shell commands that may not work cross-platform

The MCP server should provide sufficient tools so that a calling LLM can complete any CodeQL development task using only MCP tool calls and standard read/write primitives (e.g. read_file, edit_file) — never shelling out to grep, find, codeql CLI, or any other executable directly.

Use Cases Requiring CLI Today

  1. Searching QL source code — LLMs grep library pack .qll files for class/predicate definitions
  2. Finding QL files by name/pattern — LLMs use find or ls + grep to locate files like *RemoteFlowSource*.qll in library packs
  3. Analyzing evaluator logs — LLMs grep large JSON evaluator logs for pipeline timing, tuple counts, and RA operations
  4. Finding module definitionsfind_class_position only finds class, not module definitions

Solution (PR #119)

PR #119 addresses these gaps with:

New Tools

  • search_ql_code — text/regex search across .ql/.qll files with structured JSON results (file paths, line numbers, context lines). Replaces grep -rn for QL code search.
  • codeql_resolve_files — find files by extension and glob patterns in directory trees, wrapping codeql resolve files. Replaces find + grep for file discovery.

Improved Tools

  • profile_codeql_query_from_logs — rewritten with a two-tier response design:
    • Tier 1 (inline): compact JSON with per-predicate metrics (name, duration, resultSize, evalOrder, strategy, dependency count). Always small, always consumable.
    • Tier 2 (detail file): line-indexed text file with full RA operations, pipeline-stage tuple progressions, and dependency lists per predicate. Each predicate in the inline response includes {startLine, endLine} for targeted read_file access.
    • Parser enhanced to capture RA steps and per-pipeline timing/tuple counts from raw evaluator logs.
  • codeql_resolve_database — now probes child directories when given a parent path (e.g. a vscode-codeql storage directory), resolving to the actual database subdirectory automatically.

Prompt/Resource Updates

  • All grep and CLI command references removed from prompts
  • codeql_generate_log-summary de-emphasized in favor of profile_codeql_query_from_logs as the primary evaluator log analysis tool
  • New "Discover and Search QL Code" workflow section in server-tools.md

Cross-Platform

  • Line ending normalization (\r\n\n) in evaluator log parser and search tool
  • Platform-native path delimiter for environment variable parsing

Acceptance Criteria

  • An LLM can search QL library code, find files, and analyze evaluator logs using only MCP tools
  • No prompt or resource file recommends or requires grep, find, or direct CLI access
  • All tools produce cross-platform compatible output (Windows + POSIX)
  • profile_codeql_query_from_logs response fits within MCP message size limits for real-world evaluator logs (35MB+)

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentationenhancementNew feature or requestserver

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions