name	validate-ql-mcp-server-tools-queries
description	Validate that the CodeQL Development MCP Server reliably returns non-empty AST, CFG, and CallGraph data from the bundled tools queries. Use this skill to ensure the `codeql_query_run` tool properly executes PrintAST, PrintCFG, CallGraphFrom, and CallGraphTo queries for any supported language.

Validate QL MCP Server Tools Queries

This skill provides a systematic approach to validating that the CodeQL Development MCP Server's tools queries (PrintAST, PrintCFG, CallGraphFrom, CallGraphTo) return meaningful, non-empty output for test code across all supported languages.

Why This Skill Exists

⚠️ CRITICAL: The most common failure mode in MCP server tool validation is tests that "go through the motions" without verifying that query output is substantive.

A test that:

✅ Invokes codeql_query_run with PrintAST.ql
✅ Receives a successful HTTP response
❌ Does NOT verify the output contains actual AST nodes

...is a false positive that masks real bugs.

Supported Languages

The tools queries are available for all CodeQL-supported languages:

Language	Tools Pack Location	Test Code Extension
actions	`server/ql/actions/tools/`	`.yml`
cpp	`server/ql/cpp/tools/`	`.cpp`
csharp	`server/ql/csharp/tools/`	`.cs`
go	`server/ql/go/tools/`	`.go`
java	`server/ql/java/tools/`	`.java`
javascript	`server/ql/javascript/tools/`	`.js`
python	`server/ql/python/tools/`	`.py`
ruby	`server/ql/ruby/tools/`	`.rb`
rust	`server/ql/rust/tools/`	`.rs`
swift	`server/ql/swift/tools/`	`.swift`

Tools Queries Overview

Each language pack includes four standard tools queries:

PrintAST

Purpose: Outputs a hierarchical representation of the Abstract Syntax Tree.

Expected Output: A graph with labeled nodes representing code elements (functions, classes, statements, expressions).

Validation Check: Output must contain multiple semmle.label properties with actual node names.

PrintCFG

Purpose: Outputs the Control Flow Graph showing execution paths.

Expected Output: A graph with nodes (CFG nodes) and edges (control flow transitions).

Validation Check: Output must contain both nodes and edges predicates with non-zero results.

CallGraphFrom

Purpose: Shows outbound function calls from a specified function.

Expected Output: Problem-style results showing call sites and their targets.

Validation Check: Output must contain call relationships when test code has function calls.

CallGraphTo

Purpose: Shows inbound function calls to a specified function.

Expected Output: Problem-style results showing callers of the target function.

Validation Check: Output must contain caller relationships when test code has function definitions that are called.

Validation Workflow

Step 1: Select Language and Test Code

Choose a language and ensure valid test code exists:

server/ql/<language>/tools/test/PrintAST/
├── Example1.<ext>       # Test source code
├── PrintAST.expected    # Expected results
└── PrintAST.qlref       # Reference to query

Step 2: Extract Test Database

Use codeql_test_extract to create the test database:

{
  "testPath": "server/ql/<language>/tools/test/PrintAST",
  "searchPath": ["server/ql/<language>/tools/src", "server/ql/<language>/tools/test"]
}

Step 3: Run PrintAST Query

Use codeql_query_run:

{
  "queryPath": "server/ql/<language>/tools/src/PrintAST/PrintAST.ql",
  "database": "server/ql/<language>/tools/test/PrintAST/PrintAST.testproj",
  "outputFormat": "bqrs"
}

Step 4: Interpret Results

Use codeql_bqrs_interpret with graphtext format:

{
  "file": "<path-to-results.bqrs>",
  "format": "graphtext",
  "output": "<path-to-output.txt>"
}

Step 5: Validate Non-Empty Output

CRITICAL: Parse the output and verify it contains substantive data:

For PrintAST: Count nodes with semmle.label > 0
For PrintCFG: Verify both nodes and edges sections exist
For CallGraphFrom/To: Count result rows > 0 (when calls exist)

Language-Specific Examples

JavaScript Example

Test Code (server/ql/javascript/tools/test/PrintAST/Example1.js):

class Example1 {
  constructor(name = 'World') {
    this.name = name;
  }

  static main(args = []) {
    const example = new Example1();
    example.demo();
  }

  demo() {
    for (let i = 0; i < 3; i++) {
      console.log(i);
    }
  }
}

Expected AST Output (non-empty validation):

Must contain ClassDefinition node for Example1
Must contain MethodDefinition nodes for constructor, main, demo
Must contain ForStatement node in demo method

Expected CFG Output:

Must contain nodes for function entry/exit
Must contain edges for loop control flow

Expected CallGraph Output:

CallGraphFrom(main) → should find call to Example1() and demo()
CallGraphTo(demo) → should find caller main

C++ Example

Test Code (server/ql/cpp/tools/test/PrintAST/Example1.cpp):

class someClass {
public:
    void f(void);
    int g(int i, int j);
};

void fun3(someClass *sc) {
    int i;
    sc->f();
    i = sc->g(1, 2);
}

Expected AST Output:

Must contain Class node for someClass
Must contain MemberFunction nodes for f and g
Must contain Function node for fun3

Expected CallGraph Output:

CallGraphFrom(fun3) → should find calls to f() and g()
CallGraphTo(f) → should find caller fun3

Python Example

Test Code (server/ql/python/tools/test/PrintAST/Example1.py):

class Example1:
    def __init__(self, name="World"):
        self.name = name

    def demo(self):
        for i in range(3):
            print(i)

def main():
    example = Example1()
    example.demo()

Expected AST Output:

Must contain ClassDef node for Example1
Must contain FunctionDef nodes for __init__, demo, main
Must contain For node in demo

Java Example

Test Code (server/ql/java/tools/test/PrintAST/Example1.java):

public class Example1 {
    private String name;

    public Example1(String name) {
        this.name = name;
    }

    public void demo() {
        for (int i = 0; i < 3; i++) {
            System.out.println(i);
        }
    }

    public static void main(String[] args) {
        Example1 example = new Example1("World");
        example.demo();
    }
}

Expected AST Output:

Must contain ClassOrInterface node for Example1
Must contain Method nodes for constructor, demo, main
Must contain ForStmt node in demo

Validation Failure Indicators

Symptom	Likely Cause	Resolution
Empty AST output	Database not extracted properly	Re-run `codeql_test_extract`
Only headers in output	Query ran but no results matched	Check query's source file selection logic
Empty CFG nodes/edges	Test code has no control flow	Add functions with loops/conditionals
Empty CallGraph	No function calls in test code	Add function calls or verify `targetFunction` param
BQRS decode error	Incompatible CodeQL version	Update CodeQL CLI and re-run
Query compilation error	Missing pack dependencies	Run `codeql pack install` in tools pack
"Nothing to extract"	Missing test source files	Run workshop's `initialize-qltests.sh` script
Pack not found	Older pack version not cached	Run `codeql pack install` in solutions directory

Handling External Workshops

When validating against external workshops (outside the current workspace), follow these guidelines:

Pre-Validation Setup

Install pack dependencies: External workshops often use older CodeQL pack versions

cd /path/to/external/workshop
codeql pack install solutions
codeql pack install solutions-tests

Run initialization scripts: Many workshops have scripts that copy test files from shared directories
```
./initialize-qltests.sh  # Copies from tests-common/ to test directories
```
Verify test files exist: Check that test source files are present before running tests
```
ls solutions-tests/Exercise*/test.*
```

File Access Patterns

When the external workshop is outside your workspace:

Use terminal commands (cat, ls, head) to read files
Use MCP tools (codeql_query_run, codeql_test_run) with absolute paths
Pre-built databases in the workshop root can be used for tools queries

Integration with Prompts and Agents

This skill is designed to be used with:

Prompt: validate-ql-mcp-server-tools-via-workshop - Uses this skill as part of workshop creation validation
Agent: mcp-enabled-ql-workshop-developer - Orchestrates validation during workshop creation
Agent: ql-mcp-tool-tester - Tests MCP tools including tools query execution

MCP Tools Used

This skill exercises the following MCP server tools:

Tool	Purpose
`codeql_test_extract`	Create test database from test code
`codeql_query_run`	Execute tools queries
`codeql_bqrs_decode`	Decode raw BQRS results
`codeql_bqrs_interpret`	Convert results to readable formats
`codeql_pack_install`	Install pack dependencies
`codeql_pack_ls`	List available packs

Important Implementation Notes

Server Logger Output Goes to stderr

All logger.info/warn/error/debug methods write to stderr (console.error), not stdout. This is required because in stdio transport mode, stdout is reserved exclusively for the MCP JSON-RPC protocol wire format. When validating server startup logs (e.g., confirming CODEQL_PATH resolution), always check stderr.

CODEQL_PATH Environment Variable

The MCP server resolves the CodeQL CLI binary at startup via resolveCodeQLBinary() in server/src/lib/cli-executor.ts. When CODEQL_PATH is set to an absolute path pointing to a valid codeql binary, the server uses that binary for all CodeQL CLI operations instead of searching PATH. This is validated per-OS in .github/workflows/build-and-test-client.yml (codeql-path-tests job).

STDIO Transport and stdin EOF

When the STDIO transport receives an immediate EOF on stdin (e.g., via </dev/null), the server exits cleanly with code 0. To keep the server alive for testing, use a named pipe (mkfifo) with a persistent background writer (sleep 300 > fifo &).

npm Package Includes Tool Query Source Packs

The published npm package (codeql-development-mcp-server) bundles all tool query source packs under ql/*/tools/src/. These are the same .ql, .qll, .md, codeql-pack.yml, and codeql-pack.lock.yml files — but never compiled .qlx bytecode (excluded by server/.npmignore).

Success Criteria

Validation passes when ALL of the following are true:

PrintAST returns ≥10 labeled AST nodes for test code
PrintCFG returns both nodes and edges with ≥5 entries each
CallGraphFrom returns ≥1 result when test code contains function calls
CallGraphTo returns ≥1 result when test code contains called functions
No MCP tool errors or timeouts occurred
Results are consistent across multiple runs

Related Resources

Server Documentation
create-codeql-query-tdd-generic - TDD workflow that uses AST/CFG analysis
create-codeql-query-development-workshop - Workshop creation that depends on tools queries

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validate QL MCP Server Tools Queries

Why This Skill Exists

Supported Languages

Tools Queries Overview

PrintAST

PrintCFG

CallGraphFrom

CallGraphTo

Validation Workflow

Step 1: Select Language and Test Code

Step 2: Extract Test Database

Step 3: Run PrintAST Query

Step 4: Interpret Results

Step 5: Validate Non-Empty Output

Language-Specific Examples

JavaScript Example

C++ Example

Python Example

Java Example

Validation Failure Indicators

Handling External Workshops

Pre-Validation Setup

File Access Patterns

Integration with Prompts and Agents

MCP Tools Used

Important Implementation Notes

Server Logger Output Goes to stderr

CODEQL_PATH Environment Variable

STDIO Transport and stdin EOF

npm Package Includes Tool Query Source Packs

Success Criteria

Related Resources

FilesExpand file tree

SKILL.md

Latest commit

History

SKILL.md

File metadata and controls

Validate QL MCP Server Tools Queries

Why This Skill Exists

Supported Languages

Tools Queries Overview

PrintAST

PrintCFG

CallGraphFrom

CallGraphTo

Validation Workflow

Step 1: Select Language and Test Code

Step 2: Extract Test Database

Step 3: Run PrintAST Query

Step 4: Interpret Results

Step 5: Validate Non-Empty Output

Language-Specific Examples

JavaScript Example

C++ Example

Python Example

Java Example

Validation Failure Indicators

Handling External Workshops

Pre-Validation Setup

File Access Patterns

Integration with Prompts and Agents

MCP Tools Used

Important Implementation Notes

Server Logger Output Goes to stderr

CODEQL_PATH Environment Variable

STDIO Transport and stdin EOF

npm Package Includes Tool Query Source Packs

Success Criteria

Related Resources