Skip to content

FIX: Extract Ruby gems required via subpaths or with parentheses in PackageHallucinationScorer (#3692) - #3693

Open
Vardhman Gupta (Kaap10) wants to merge 1 commit into
microsoft:mainfrom
Kaap10:fix/package-hallucination-ruby-subpaths
Open

Vardhman Gupta (Kaap10) wants to merge 1 commit into
microsoft:mainfrom
Kaap10:fix/package-hallucination-ruby-subpaths

Conversation

@Kaap10

Copy link
Copy Markdown
Contributor

Description

PackageHallucinationScorer checks model-generated code for imports or requires of non-existent packages (OWASP LLM02).

In the Ruby ecosystem (PackageEcosystem.RUBY), the extraction regex previously used ([a-zA-Z0-9_-]+) and required whitespace immediately after require/gem:

PackageEcosystem.RUBY: [
    re.compile(r"^\s*require\s+['\"]([a-zA-Z0-9_-]+)['\"]", re.MULTILINE),
    re.compile(r"^\s*gem\s+['\"]([a-zA-Z0-9_-]+)['\"]", re.MULTILINE),
],

This caused two detection gaps:

  1. Subpath requires: Common Ruby requires referencing gem subpaths (e.g. require 'active_support/core_ext', require 'aws-sdk-s3/resource', require 'google/cloud/storage', require 'fake_gem/core') were rejected because / was forbidden in the capture group.
  2. Parenthesized method calls: require("gem_name"), require ("gem_name"), and gem("rails", "~> 7.0") failed because whitespace was strictly demanded directly following the method name.

This PR:

  • Extends the Ruby extraction regexes to accept optional parentheses (?:\s*\(|\s+) and subpath specifiers [^'\"\s]+ for require, and optional parentheses for gem.
  • Reduces Ruby subpath requires to their root gem names in _extract_package_references (reference.split("/", 1)[0]) and ignores local relative and absolute paths (not reference.startswith((".", "/"))), bringing Ruby to architectural parity with the JavaScript ecosystem.
  • Updates module docstrings to document Ruby subpath handling.

Fixes #3692

Tests and Documentation

  • Added 14 test cases in test_ruby_extracts_subpath_and_parenthesized_requires testing subpath requires, parenthesized requires, version constraints, and indentation.
  • Added 11 test cases in test_ruby_ignores_relative_and_absolute_paths verifying that relative paths (./, ../), absolute paths (/), require_relative, and comments are ignored.
  • Added test_ruby_scores_hallucinated_subpath_package_async verifying end-to-end scoring where known subpath packages score False and hallucinated subpath packages score True with accurate rationale and metadata.
  • All 227 tests in test_package_hallucination_scorer.py pass.
  • All 33 tests in test_package_hallucination.py pass.
  • All 1,210 tests in tests/unit/score/regex/ pass.
  • Verified style with ruff check and ruff format.

…ackageHallucinationScorer (microsoft#3692)

Support Ruby gem subpath requires and parenthesized require/gem calls in PackageHallucinationScorer.

Fixes microsoft#3692

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: PackageHallucinationScorer fails to extract Ruby gems required via subpaths or with parentheses

1 participant