Repository navigation
FIX: Extract Ruby gems required via subpaths or with parentheses in PackageHallucinationScorer (#3692) - #3693
Open
Vardhman Gupta (Kaap10) wants to merge 1 commit into
Conversation
…ackageHallucinationScorer (microsoft#3692) Support Ruby gem subpath requires and parenthesized require/gem calls in PackageHallucinationScorer. Fixes microsoft#3692
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PackageHallucinationScorerchecks model-generated code for imports or requires of non-existent packages (OWASP LLM02).In the Ruby ecosystem (
PackageEcosystem.RUBY), the extraction regex previously used([a-zA-Z0-9_-]+)and required whitespace immediately afterrequire/gem:This caused two detection gaps:
require 'active_support/core_ext',require 'aws-sdk-s3/resource',require 'google/cloud/storage',require 'fake_gem/core') were rejected because/was forbidden in the capture group.require("gem_name"),require ("gem_name"), andgem("rails", "~> 7.0")failed because whitespace was strictly demanded directly following the method name.This PR:
(?:\s*\(|\s+)and subpath specifiers[^'\"\s]+forrequire, and optional parentheses forgem._extract_package_references(reference.split("/", 1)[0]) and ignores local relative and absolute paths (not reference.startswith((".", "/"))), bringing Ruby to architectural parity with the JavaScript ecosystem.Fixes #3692
Tests and Documentation
test_ruby_extracts_subpath_and_parenthesized_requirestesting subpath requires, parenthesized requires, version constraints, and indentation.test_ruby_ignores_relative_and_absolute_pathsverifying that relative paths (./,../), absolute paths (/),require_relative, and comments are ignored.test_ruby_scores_hallucinated_subpath_package_asyncverifying end-to-end scoring where known subpath packages scoreFalseand hallucinated subpath packages scoreTruewith accurate rationale and metadata.test_package_hallucination_scorer.pypass.test_package_hallucination.pypass.tests/unit/score/regex/pass.ruff checkandruff format.