chore: add script to audit changelog labels on merged PRs

admin/README.md

@@ -4,33 +4,33 @@ This folder contains tools or docs useful for project maintainers.
 
 ## Repositories inside https://github.com/codeigniter4
 
--   **CodeIgniter4** is the main development repository.  
-    It supports issues and pull requests, and has a rule to enforce GPG-signed commits.  
-    In addition to the framework source, it includes unit testing and documentation source.  
-    The three repositories following are built from this one as part of the release workflow.  
+-   **CodeIgniter4** is the main development repository.
+    It supports issues and pull requests, and has a rule to enforce GPG-signed commits.
+    In addition to the framework source, it includes unit testing and documentation source.
+    The three repositories following are built from this one as part of the release workflow.
     This repo is meant to be forked by contributors.
--   **framework** is the released developer repository.  
+-   **framework** is the released developer repository.
     It contains all the main pieces of the framework that developers would use to
-    build their apps, but not the framework unit testing or the user guide source.  
-    It is meant to be downloaded by developers, or composer-installed.  
+    build their apps, but not the framework unit testing or the user guide source.
+    It is meant to be downloaded by developers, or composer-installed.
     This is a read-only repository.
--   **appstarter** is the released application starter repository.  
+-   **appstarter** is the released application starter repository.
     It is derived from the framework's `app` and `public` folders, with
-    a composer requirement dependency to pull in the framework itself.  
-    It is meant to be downloaded or composer-installed.  
+    a composer requirement dependency to pull in the framework itself.
+    It is meant to be downloaded or composer-installed.
     This is a read-only repository.
--   **userguide** is released documentation publishing repository.  
+-   **userguide** is released documentation publishing repository.
     It contains built versions of the user guide, corresponding to the
-    framework releases.  
-    It could be downloaded, forked or potentially composer-installed.  
+    framework releases.
+    It could be downloaded, forked or potentially composer-installed.
     This is a read-only repository.
--   **coding-standard** <https://github.com/CodeIgniter/coding-standard> is the coding style standards repository.  
+-   **coding-standard** <https://github.com/CodeIgniter/coding-standard> is the coding style standards repository.
     It contains PHP-CS-Fixer rules to ensure consistent code style
-    within the framework itself.  
+    within the framework itself.
     It is meant to be composer-installed.
 -   **translations** is the repository holding official translations of
-    the locale-dependent system messages.  
-    It is community-maintained, and accepts issues and pull requests.  
+    the locale-dependent system messages.
+    It is community-maintained, and accepts issues and pull requests.
     It could be downloaded, forked or composer-installed.
 
 ## Contributor Scripts
@@ -52,18 +52,22 @@ are used as part of that process:
     Usage: `php admin/generate-changelog.php 4.x.x [--dry-run]`
 -   **prepare-release.php** creates the `release-4.x.x` branch and updates
     version references in the framework source, the user guide, and the
-    distribution build script.  
+    distribution build script.
     Usage: `php admin/prepare-release.php 4.x.x`
 -   **create-new-changelog.php** creates the changelog and upgrade guide
-    stubs for the next version and adds them to their index files.  
+    stubs for the next version and adds them to their index files.
     Usage: `php admin/create-new-changelog.php <current_version> <new_version>`
+-   **check-pr-labels.php** flags PRs merged after a release (default: the
+    latest) that appear to be missing the labels used to generate the
+    changelog.
+    Usage: `php admin/check-pr-labels.php [<version>]`
 
 ## Other Stuff
 
 -   The **framework** and **starter** subfolders contain files that will over-ride
     those from the development repository, when the distribution repositories
     are built.
--   The subfolders inside `admin` contain "next release" files in the case of 
+-   The subfolders inside `admin` contain "next release" files in the case of
     `codeigniter4` and over-written distribution files in the other cases.
 -   The CHANGELOG.md file is generated from GitHub's auto-generated release
     notes, as described in [RELEASE.md](./RELEASE.md).

admin/RELEASE.md

@@ -74,6 +74,11 @@ the following [labels](https://github.com/codeigniter4/CodeIgniter4/labels):
 PRs with breaking changes must have the following additional label:
 - **breaking change** ... PRs that may break existing functionalities
 
+At any time, missing labels can be checked for with
+`php admin/check-pr-labels.php`, which flags PRs merged since the last release
+that appear to lack a changelog label. It requires the authenticated
+[GitHub CLI](https://cli.github.com/).
+
 ### Generate Changelog
 
 The changelog is generated from GitHub's auto-generated release notes with a

admin/check-pr-labels.php

@@ -0,0 +1,203 @@
+<?php
+
+declare(strict_types=1);
+
+/**
+ * Checks that PRs merged since the last release carry the labels used to
+ * generate the changelog, suggesting a label from the PR title type.
+ *
+ * Requires the authenticated GitHub CLI (`gh`).
+ *
+ * Usage: php admin/check-pr-labels.php [version]
+ */
+function color(string $text, string $code, bool $ansi): string
+{
+    return $ansi ? sprintf("\033[%sm%s\033[0m", $code, $text) : $text;
+}
+
+function hyperlink(string $text, string $url, bool $ansi): string
+{
+    return $ansi ? sprintf("\033]8;;%s\033\\%s\033]8;;\033\\", $url, $text) : $text;
+}
+
+function format_pull_line(array $pull, bool $ansi): string
+{
+    return sprintf(
+        '* %s %s %s',
+        hyperlink(color(sprintf('#%d', $pull['number']), '36', $ansi), $pull['url'], $ansi),
+        $pull['title'],
+        color(sprintf('(%s)', $pull['base']), '35', $ansi),
+    );
+}
+
+function touches_system_dir(string $repo, int $number): bool
+{
+    exec(sprintf("gh api repos/%s/pulls/%d/files --paginate --jq '.[].filename' 2>&1", $repo, $number), $files, $exitCode);
+
+    if ($exitCode !== 0) {
+        return true;
+    }
+
+    foreach ($files as $file) {
+        if (str_starts_with($file, 'system/')) {
+            return true;
+        }
+    }
+
+    return false;
+}
+
+chdir(__DIR__ . '/..');
+
+$ansi = stream_isatty(STDOUT);
+$repo = 'codeigniter4/CodeIgniter4';
+
+// Labels used by the changelog categories. See ".github/release.yml".
+$changelogLabels = ['breaking change', 'bug', 'enhancement', 'new feature', 'refactor'];
+
+// PR title types that map to a changelog label.
+$typeToLabel = [
+    'fix'      => 'bug',
+    'feat'     => 'new feature',
+    'perf'     => 'enhancement',
+    'refactor' => 'refactor',
+];
+
+// PR title types that need no changelog label.
+$typesWithoutLabel = ['chore', 'ci', 'docs', 'style', 'test'];
+
+// Release process PRs carry no labels.
+$releaseTitles = '/\A(?:Prep for \d+\.\d+\.\d+ release|\d+\.\d+\.\d+ (?:Ready|Merge) code)\z/';
+
+$tag = null;
+
+foreach (array_slice($argv, 1) as $arg) {
+    if (preg_match('/\Av?(\d+\.\d+\.\d+)\z/', $arg, $matches) === 1) {
+        $tag = "v{$matches[1]}";
+
+        continue;
+    }
+
+    echo sprintf("Usage: php %s [version]\n", $argv[0]);
+    echo sprintf("E.g.,: php %s 4.7.3\n", $argv[0]);
+    echo "Checks the PRs merged after the given release. Defaults to the latest release.\n";
+
+    exit(1);
+}
+
+$endpoint = $tag === null ? sprintf('repos/%s/releases/latest', $repo) : sprintf('repos/%s/releases/tags/%s', $repo, $tag);
+exec(sprintf('gh api %s 2>&1', $endpoint), $output, $exitCode);
+
+if ($exitCode !== 0) {
+    echo sprintf("Failed to fetch the release %s from GitHub:\n", $tag ?? '(latest)');
+    echo implode("\n", $output) . "\n";
+
+    exit(1);
+}
+
+$release = json_decode(implode("\n", $output), true);
+$since   = $release['published_at'] ?? '';
+
+echo sprintf("Checking PRs merged since %s (%s).\n", $release['tag_name'], $since);
+
+$command = sprintf(
+    'gh pr list --repo %s --search %s --json number,title,labels,baseRefName,url --limit 300 2>&1',
+    $repo,
+    escapeshellarg(sprintf('is:merged -base:master merged:>%s', $since)),
+);
+exec($command, $prOutput, $exitCode);
+
+if ($exitCode !== 0) {
+    echo "Failed to fetch the merged PRs from GitHub:\n";
+    echo implode("\n", $prOutput) . "\n";
+
+    exit(1);
+}
+
+$pulls = json_decode(implode("\n", $prOutput), true);
+
+if (! is_array($pulls)) {
+    echo "Unexpected response from GitHub.\n";
+
+    exit(1);
+}
+
+echo sprintf("Found %d merged PRs.\n\n", count($pulls));
+
+$missingLabel   = [];
+$needManualLook = [];
+
+foreach ($pulls as $pull) {
+    $labels   = array_column($pull['labels'], 'name');
+    $hasLabel = array_intersect($changelogLabels, $labels) !== [];
+    $type     = null;
+
+    if (preg_match('/\A(\w+)(?:\([^)]*\))?:/', $pull['title'], $matches) === 1) {
+        $type = $matches[1];
+    }
+
+    if ($hasLabel || in_array($type, $typesWithoutLabel, true)) {
+        continue;
+    }
+
+    if (preg_match($releaseTitles, $pull['title']) === 1) {
+        continue;
+    }
+
+    if ($type !== null && isset($typeToLabel[$type])) {
+        // The refactor label applies only to refactoring in system/. Test-only
+        // refactors may use the testing label, which is not in the changelog.
+        if ($type === 'refactor' && ! touches_system_dir($repo, (int) $pull['number'])) {
+            continue;
+        }
+
+        $missingLabel[] = [
+            'number' => $pull['number'],
+            'title'  => $pull['title'],
+            'base'   => $pull['baseRefName'],
+            'url'    => $pull['url'],
+            'label'  => $typeToLabel[$type],
+        ];
+    } else {
+        $needManualLook[] = [
+            'number' => $pull['number'],
+            'title'  => $pull['title'],
+            'base'   => $pull['baseRefName'],
+            'url'    => $pull['url'],
+        ];
+    }
+}
+
+if ($missingLabel !== []) {
+    echo color('PRs that appear to be missing a changelog label:', '1', $ansi) . "\n";
+
+    foreach ($missingLabel as $pull) {
+        echo sprintf("%s [suggested: %s]\n", format_pull_line($pull, $ansi), color($pull['label'], '33', $ansi));
+    }
+
+    echo "\nTo add the suggested labels, run the following commands (drop any that are not warranted):\n";
+
+    foreach ($missingLabel as $pull) {
+        echo sprintf("gh pr edit %d --repo %s --add-label \"%s\"\n", $pull['number'], $repo, $pull['label']);
+    }
+
+    echo "\n";
+}
+
+if ($needManualLook !== []) {
+    echo color('PRs with no changelog label and no recognized title type (check manually):', '1', $ansi) . "\n";
+
+    foreach ($needManualLook as $pull) {
+        echo format_pull_line($pull, $ansi) . "\n";
+    }
+
+    echo "\n";
+}
+
+if ($missingLabel === []) {
+    echo color('No PRs are missing changelog labels.', '32', $ansi) . "\n";
+
+    exit(0);
+}
+
+exit(1);

tests/system/AutoReview/CheckPrLabelsTest.php

@@ -0,0 +1,95 @@
+<?php
+
+declare(strict_types=1);
+
+/**
+ * This file is part of CodeIgniter 4 framework.
+ *
+ * (c) CodeIgniter Foundation <admin@codeigniter.com>
+ *
+ * For the full copyright and license information, please view
+ * the LICENSE file that was distributed with this source code.
+ */
+
+namespace CodeIgniter\AutoReview;
+
+use Nexus\PHPUnit\Tachycardia\Attribute\TimeLimit;
+use PHPUnit\Framework\Attributes\CoversNothing;
+use PHPUnit\Framework\Attributes\Group;
+use PHPUnit\Framework\TestCase;
+
+/**
+ * @internal
+ */
+#[CoversNothing]
+#[Group('AutoReview')]
+final class CheckPrLabelsTest extends TestCase
+{
+    public function testUsageErrorWithUnknownArgument(): void
+    {
+        exec('php ./admin/check-pr-labels.php foo 2>&1', $output, $exitCode);
+
+        $this->assertSame(1, $exitCode);
+        $this->assertStringContainsString('Usage:', implode("\n", $output));
+    }
+
+    public function testUsageErrorWithInvalidVersion(): void
+    {
+        exec('php ./admin/check-pr-labels.php 4.7 2>&1', $output, $exitCode);
+
+        $this->assertSame(1, $exitCode);
+        $this->assertStringContainsString('Usage:', implode("\n", $output));
+    }
+
+    #[TimeLimit(10.0)]
+    public function testChecksAgainstLatestRelease(): void
+    {
+        $this->skipUnlessGhIsAuthenticated();
+
+        exec('php ./admin/check-pr-labels.php 2>&1', $output, $exitCode);
+        $outputString = implode("\n", $output);
+
+        $this->assertContains($exitCode, [0, 1], "Script exited with code {$exitCode}. Output: {$outputString}");
+        $this->assertMatchesRegularExpression('/Checking PRs merged since v\d+\.\d+\.\d+/', $outputString);
+        $this->assertMatchesRegularExpression('/Found \d+ merged PRs\./', $outputString);
+    }
+
+    #[TimeLimit(10.0)]
+    public function testChecksSinceGivenRelease(): void
+    {
+        $this->skipUnlessGhIsAuthenticated();
+
+        $version = $this->latestChangelogVersion();
+
+        exec("php ./admin/check-pr-labels.php {$version} 2>&1", $output, $exitCode);
+        $outputString = implode("\n", $output);
+
+        $this->assertContains($exitCode, [0, 1], "Script exited with code {$exitCode}. Output: {$outputString}");
+        $this->assertStringContainsString("Checking PRs merged since v{$version} (", $outputString);
+        $this->assertMatchesRegularExpression('/Found \d+ merged PRs\./', $outputString);
+    }
+
+    private function latestChangelogVersion(): string
+    {
+        $changelog = (string) file_get_contents('./CHANGELOG.md');
+
+        if (preg_match('/^## \[v(\d+\.\d+\.\d+)\]/m', $changelog, $matches) !== 1) {
+            $this->fail('Could not find a version entry in CHANGELOG.md.');
+        }
+
+        return $matches[1];
+    }
+
+    private function skipUnlessGhIsAuthenticated(): void
+    {
+        exec('gh auth status 2>&1', $output, $exitCode);
+
+        if ($exitCode !== 0) {
+            $this->markTestSkipped(
+                'The GitHub CLI is not available or not authenticated. This test is expected '
+                . 'to be skipped in CI and runs only where `gh` is authenticated, such as on '
+                . 'a maintainer\'s machine.',
+            );
+        }
+    }
+}