Add MAUI iOS Inner Loop measurements for CI

[davidnguyen-tech]

Summary

Adds MAUI iOS Inner Loop performance measurements to CI, supporting both iOS simulators and physical devices.

What's included:

• iOSInnerLoopParser.cs — Binlog parser extracting iOS build task/target timings
• Startup.cs/Reporter.cs — Wiring and null-safety fix
• ioshelper.py — iOS simulator and physical device management
• runner.py — IOSINNERLOOP execution branch (build → deploy → measure → incremental)
• Scenario scripts — pre.py, test.py, post.py, setup_helix.py
• maui_scenarios_ios_innerloop.proj — Helix workitem definition (simulator + physical device)
• Pipeline YAML — Job entries in sdk-perf-jobs.yml and routing in run_performance_job.py

Measurements:

• First deploy: full build + install + launch timing via binlog parsing
• Incremental deploy: source edit → rebuild + reinstall + relaunch timing

Targets:

• iOS Simulator (iossimulator-arm64) on macOS Helix machines
• Physical iPhone (ios-arm64) on Mac.iPhone.17.Perf queue

Based on:

• Existing Android Inner Loop CI scenario (working reference)
• Local iOS implementation from feature/measure-maui-ios-deploy branch

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (8)

src/scenarios/mauiiosinnerloop/post.py:18

• The cleanup script uninstalls using com.companyname.<exename> but the scenario sets the MAUI app bundle id to net.dot.mauiiosinnerloop (see pre.py / HelixWorkItem args). With the current value, uninstall is a no-op and can leave the app installed between runs.

    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

• This on-agent driver passes --bundle-id com.companyname.mauiiosinnerloop, but the scenario csproj/Helix work item uses net.dot.mauiiosinnerloop. Using the wrong bundle id will break install/launch and cleanup targeting.

  --bundle-id com.companyname.mauiiosinnerloop \

eng/pipelines/sdk-perf-jobs.yml:20

• Hard-coding ${{ if false }} disables all public correctness/scenario jobs. This should remain gated by the existing runPublicJobs parameter (or another explicit feature flag), otherwise PRs lose public CI coverage entirely.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

• This second ${{ if false }} also disables the rest of the public-job section. It should be conditioned on parameters.runPublicJobs so public scenario coverage isn't silently turned off.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

• Hard-coding ${{ if false }} here disables all scheduled private jobs. This should be gated by parameters.runScheduledPrivateJobs (or a dedicated WIP flag) so scheduled runs still execute their intended coverage.

- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:367

• This ${{ if false }} disables the entire private scenario benchmark set (and many other blocks below are similarly disabled). If the intent is to land iOS inner loop without turning off existing CI coverage, these blocks should not be hard-disabled in the checked-in pipeline YAML.

  - ${{ if false }}: # disabled: iOS-only WIP test run

src/scenarios/shared/runner.py:233

• --csproj-path is treated as mandatory later (the scenario raises if missing), but argparse doesn't mark it required. Making it required improves UX (fails fast with standard help) and matches the Android inner loop argument style.

        iosinnerloopparser.add_argument('--csproj-path', help='Path to .csproj file to build', dest='csprojpath')

src/scenarios/shared/runner.py:239

• --bundle-id is required for the iOS inner loop run (and validated later), but argparse doesn't enforce it. Mark it required so missing args are caught by argparse consistently.

        iosinnerloopparser.add_argument('--bundle-id', help='iOS bundle identifier', dest='bundleid')

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (5)

src/scenarios/mauiiosinnerloop/post.py:18

• post.py uninstalls the wrong bundle id (com.companyname.) but pre.py/test invocation uses net.dot.mauiiosinnerloop. This will leave the app installed and can leak state across Helix workitems.

    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/sdk-perf-jobs.yml:20

• This hard-codes the entire public correctness job block off (if false), which will stop public CI from running tooling tests/validation unrelated to the iOS inner loop work.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

• This also hard-codes the public scenario benchmark block off (if false), effectively disabling the public scenario CI matrix.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

• This hard-codes scheduled private jobs off (if false), preventing scheduled perf runs from executing.

- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

• The on-agent driver invokes the inner loop with bundle id com.companyname.mauiiosinnerloop, but the scenario’s pre.py overrides ApplicationId to net.dot.mauiiosinnerloop and the Helix .proj passes that bundle id too. If runOnAgent is ever enabled, install/startup measurement will target the wrong app id.

  --bundle-id com.companyname.mauiiosinnerloop \

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (7)

src/scenarios/shared/ioshelper.py:96

• _EXPIRED_WWDR_SHA1 is documented as a SHA-1 fingerprint but the literal has 42 hex characters (SHA-1 should be 40). As written, _remove_expired_wwdr() will never match/delete the expired WWDR intermediate, so the codesign chain-building mitigation is ineffective.

_EXPIRED_WWDR_SHA1 = "FF677979793A3CD798DC5B2ABEF56F73EDC9F83A64"

src/scenarios/mauiiosinnerloop/post.py:18

• post.py uninstalls using bundle_id com.companyname., but this scenario sets ApplicationId/bundle id to net.dot.mauiiosinnerloop (see pre.py and maui_scenarios_ios.proj). With the current value, uninstall will target the wrong app and leave the measured app installed on devices/simulators across runs.

    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

• The on-agent driver passes --bundle-id com.companyname.mauiiosinnerloop, but the scenario sets ApplicationId to net.dot.mauiiosinnerloop and Helix jobs pass that bundle id. This mismatch will make install/startup measurement fail (and post.py uninstall the wrong app).

  --bundle-id com.companyname.mauiiosinnerloop \

src/scenarios/shared/runner.py:242

• iosinnerloop argparse options that are required for the scenario to run (--csproj-path, --edit-src, --edit-dest, --framework, --bundle-id) are not marked required, so failures surface later as generic Exceptions instead of argparse validation. This makes CI failures harder to diagnose and diverges from the Android inner loop CLI contract in the same file.

        iosinnerloopparser.add_argument('--csproj-path', help='Path to .csproj file to build', dest='csprojpath')
        iosinnerloopparser.add_argument('--edit-src', help='Modified source file paths, semicolon-separated', dest='editsrc')
        iosinnerloopparser.add_argument('--edit-dest', help='Destination paths for modified files, semicolon-separated', dest='editdest')
        iosinnerloopparser.add_argument('--framework', '-f', help='Target framework (e.g., net11.0-ios)', dest='framework')
        iosinnerloopparser.add_argument('--configuration', '-c', help='Build configuration', dest='configuration', default='Debug')
        iosinnerloopparser.add_argument('--msbuild-args', help='Additional MSBuild arguments', dest='msbuildargs', default='')
        iosinnerloopparser.add_argument('--bundle-id', help='iOS bundle identifier', dest='bundleid')
        iosinnerloopparser.add_argument('--device-id', help='iOS device ID (UDID for physical device, simulator ID or "booted" for simulator)', dest='deviceid', default='booted')
        iosinnerloopparser.add_argument('--device-type', choices=['simulator', 'device'], help='Target device type: simulator (default) or physical device. Auto-detected from RuntimeIdentifier if not set.', dest='devicetype', default=None)
        iosinnerloopparser.add_argument('--inner-loop-iterations', help='Number of incremental build+deploy+startup iterations (1+)', type=int, default=10, dest='innerloopiterations')

eng/pipelines/sdk-perf-jobs.yml:20

• This hard-codes the public jobs block to never run (if false), ignoring the runPublicJobs pipeline parameter. That effectively disables public CI coverage for unrelated areas and looks like an accidental WIP gate that should not land as-is.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

• This second public-jobs guard is also hard-coded to if false, so the public scenario benchmark jobs are disabled regardless of the runPublicJobs parameter. If you need to disable only iOS inner loop temporarily, gate just those jobs instead of the entire public section.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

• This hard-codes scheduled private jobs to never run (if false), ignoring the runScheduledPrivateJobs parameter. That disables scheduled perf coverage across the pipeline, which is a significant operational change unrelated to adding iOS inner loop measurements.

- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (9)

src/scenarios/mauiiosinnerloop/post.py:20

• post.py computes bundle_id as com.companyname., but this scenario explicitly sets ApplicationId to net.dot.mauiiosinnerloop (and passes that to runner via --bundle-id). With the current value, uninstall/cleanup will target the wrong app and leave the real app installed on simulator/device.

try:
    bundle_id = f'com.companyname.{EXENAME.lower()}'
    ios_rid = os.environ.get('IOS_RID', 'iossimulator-arm64')
    is_physical = (ios_rid == 'ios-arm64')

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:176

• The on-agent driver passes --bundle-id com.companyname.mauiiosinnerloop, but pre.py overwrites the project's to net.dot.mauiiosinnerloop. This mismatch will cause install/launch/uninstall to operate on the wrong bundle id.

python3 test.py iosinnerloop \
  --csproj-path app/MauiiOSInnerLoop.csproj \
  --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" \
  --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" \
  --bundle-id com.companyname.mauiiosinnerloop \
  -f "$framework-ios" -c Debug \

src/scenarios/shared/runner.py:1542

• StartupWrapper.parsetraces() defaults to copying the entire TRACEDIR to the Helix upload directory each call. For the inner-loop scenario this is invoked repeatedly and TRACEDIR grows each iteration, leading to O(N^2) I/O. The wrapper already supports copy_traces=False and this code later performs a single final copy/upload.

                                       tracename=f'{binlog_prefix}first-build-and-deploy.binlog',
                                       scenarioname=scenarioprefix + " - First Build and Deploy",
                                       upload_to_perflab_container=False)
                startup.parsetraces(self.traits)

src/scenarios/shared/runner.py:1620

• Same as above: calling parsetraces() inside the incremental loop triggers an expensive TRACEDIR copy each iteration. Since this scenario aggregates results and later uploads/copies once, pass copy_traces=False here as well.

                        if os.path.exists(traces_upload):
                            rmtree(traces_upload)

                    startup.parsetraces(self.traits)

src/scenarios/shared/ioshelper.py:97

• _EXPIRED_WWDR_SHA1 is documented as a SHA-1 fingerprint, but the current value is 42 hex characters (21 bytes), not 40. security delete-certificate -Z expects a valid fingerprint; with the current value the delete step will fail and the expired WWDR cert may remain, undermining the intended codesign-chain fix.

# The old Apple WWDR intermediate (no G-series), expired 2023-02-07. Its lingering
# presence makes codesign build the chain through it (candidate matched by issuer
# name) and fail "unable to build chain to self-signed root" even though the valid
# WWDR G3 is installed. The documented fix is to delete it. SHA-1:
_EXPIRED_WWDR_SHA1 = "FF677979793A3CD798DC5B2ABEF56F73EDC9F83A64"

eng/pipelines/sdk-perf-jobs.yml:22

• Hardcoding ${{ if false }} here disables all public correctness jobs and scenario coverage regardless of the pipeline parameters. If this is intended to be a temporary WIP gate, it should be controlled via the existing runPublicJobs parameter (or a new feature flag parameter), not a permanent false.

######################################################

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP
  - job: Tooling_Tests_Windows
    displayName: Tooling Tests (Windows)

eng/pipelines/sdk-perf-jobs.yml:61

• This second ${{ if false }} similarly disables the entire public scenario job block (scenarios.proj) regardless of runPublicJobs. It should be keyed off the runPublicJobs parameter (or another explicit flag) so public CI retains baseline coverage.

        displayName: Run tooling tests

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

  # Scenario benchmarks
  - template: /eng/pipelines/templates/build-machine-matrix.yml
    parameters:

eng/pipelines/sdk-perf-jobs.yml:369

• Within the runPrivateJobs block, multiple job groups are wrapped in ${{ if false }} which effectively disables the normal private scenario coverage when runPrivateJobs=true. This should be removed or controlled by a dedicated feature flag, otherwise enabling private jobs won't actually run the expected scenarios.

- ${{ if parameters.runPrivateJobs }}:

  # Scenario benchmarks
  - ${{ if false }}: # disabled: iOS-only WIP test run
    - template: /eng/pipelines/templates/build-machine-matrix.yml
      parameters:

eng/pipelines/sdk-perf-jobs.yml:829

• Hardcoding ${{ if false }} here disables scheduled private runs entirely, regardless of the runScheduledPrivateJobs parameter. This will prevent scheduled CI performance coverage from running at all.

# Scheduled runs will run all of the jobs on the PerfTigers, as well as the Arm64 job
- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

  # SDK scenario benchmarks
  - template: /eng/pipelines/templates/build-machine-matrix.yml

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (7)

src/scenarios/shared/ioshelper.py:96

• _EXPIRED_WWDR_SHA1 is not a valid SHA-1 fingerprint length (40 hex chars). security delete-certificate -Z expects a SHA-1 hash; with the current 42-character value this cleanup will never match and will always fail, leaving the expired WWDR intermediate in place.

# The old Apple WWDR intermediate (no G-series), expired 2023-02-07. Its lingering
# presence makes codesign build the chain through it (candidate matched by issuer
# name) and fail "unable to build chain to self-signed root" even though the valid
# WWDR G3 is installed. The documented fix is to delete it. SHA-1:
_EXPIRED_WWDR_SHA1 = "FF677979793A3CD798DC5B2ABEF56F73EDC9F83A64"

src/scenarios/mauiiosinnerloop/post.py:18

• Bundle id used by post-cleanup does not match the ApplicationId set in pre.py (net.dot.mauiiosinnerloop). This will cause uninstall to target the wrong app, leaving the test app installed on simulators/devices across runs.

    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:176

• This on-agent runner passes a bundle id that doesn't match the ApplicationId enforced by the scenario's pre.py (net.dot.mauiiosinnerloop). The install/launch and timing measurements will fail because the app on disk has a different bundle id.

python3 test.py iosinnerloop \
  --csproj-path app/MauiiOSInnerLoop.csproj \
  --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" \
  --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" \
  --bundle-id com.companyname.mauiiosinnerloop \
  -f "$framework-ios" -c Debug \

eng/pipelines/sdk-perf-jobs.yml:20

• This change hard-disables all public correctness jobs by replacing parameters.runPublicJobs with false. That prevents CI from running the public validation suite even when requested.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

• This block also hard-disables the public scenario benchmark jobs (if false). If the intention is only to gate iOS inner loop jobs, this should remain controlled by parameters.runPublicJobs (or a dedicated flag) rather than being permanently off.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

• Scheduled private runs are hard-disabled (if false), so periodic perf coverage will never execute. This should remain controlled by parameters.runScheduledPrivateJobs so scheduled pipelines can opt in as before.

- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:367

• Large portions of the private job matrix are currently wrapped in if false blocks (e.g. scenario benchmarks), effectively disabling most perf coverage when runPrivateJobs is true. If this is only intended for temporary WIP iteration, it should not be merged as-is; consider gating via a dedicated parameter instead of hard-coding false.

  - ${{ if false }}: # disabled: iOS-only WIP test run

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (4)

src/scenarios/mauiiosinnerloop/post.py:20

• post.py uninstalls using bundle id com.companyname.<exename> but the scenario explicitly sets the app’s ApplicationId to net.dot.mauiiosinnerloop (pre.py) and the Helix work item runs with --bundle-id net.dot.mauiiosinnerloop. As written, cleanup will target the wrong app and can leave the measured app installed, which can skew “first deploy” timings across runs.

    bundle_id = f'com.companyname.{EXENAME.lower()}'
    ios_rid = os.environ.get('IOS_RID', 'iossimulator-arm64')
    is_physical = (ios_rid == 'ios-arm64')

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:176

• run-ios-innerloop-on-agent.sh invokes the scenario with --bundle-id com.companyname.mauiiosinnerloop, but the scenario’s csproj is rewritten to ApplicationId=net.dot.mauiiosinnerloop (pre.py) and the Helix .proj also passes --bundle-id net.dot.mauiiosinnerloop. This mismatch will cause install/launch/uninstall to fail or target the wrong app when the on-agent path is used.

python3 test.py iosinnerloop \
  --csproj-path app/MauiiOSInnerLoop.csproj \
  --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" \
  --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" \
  --bundle-id com.companyname.mauiiosinnerloop \
  -f "$framework-ios" -c Debug \

eng/pipelines/sdk-perf-jobs.yml:22

• This file hard-disables public correctness jobs by changing the condition to ${{ if false }}. That will prevent tooling tests and public scenario coverage from running at all, which is a functional CI regression unrelated to adding iOS inner loop measurements. If you need to gate iOS inner loop WIP, gate only the new jobs or use the existing parameters.runPublicJobs switch.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP
  - job: Tooling_Tests_Windows
    displayName: Tooling Tests (Windows)

eng/pipelines/sdk-perf-jobs.yml:829

• Scheduled private runs are also hard-disabled via ${{ if false }}. This prevents the normal scheduled performance signal from running and should not be merged as-is; use parameters.runScheduledPrivateJobs (or gate only the new iOS inner-loop scheduled jobs) instead of disabling the entire block.

# Scheduled runs will run all of the jobs on the PerfTigers, as well as the Arm64 job
- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

  # SDK scenario benchmarks
  - template: /eng/pipelines/templates/build-machine-matrix.yml

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (7)

src/scenarios/shared/runner.py:1541

• StartupWrapper.parsetraces() copies TRACEDIR to the Helix upload directory by default; calling it here for the first build causes unnecessary I/O, and the subsequent incremental loop already does a final upload/copy at the end. Pass copy_traces=False for this parse-only invocation to avoid repeated trace copying (and potential O(N^2) behavior as binlogs accumulate).

                self.traits.add_traits(overwrite=True, apptorun="app", startupmetric=const.IOSINNERLOOP,
                                       tracename=f'{binlog_prefix}first-build-and-deploy.binlog',
                                       scenarioname=scenarioprefix + " - First Build and Deploy",
                                       upload_to_perflab_container=False)
                startup.parsetraces(self.traits)

src/scenarios/shared/runner.py:1619

• Same as the first-build parse: parsetraces() will copy TRACEDIR into the Helix upload dir every iteration. That repeatedly recopies all prior binlogs and is much more expensive than needed. Use copy_traces=False during each iteration parse and rely on the single final copy/upload at the end of the scenario.

                        traces_upload = os.path.join(helix_upload_dir, 'traces')
                        if os.path.exists(traces_upload):
                            rmtree(traces_upload)

                    startup.parsetraces(self.traits)

src/scenarios/shared/runner.py:240

• --csproj-path and --bundle-id are validated later via runtime exceptions, but they are required for this subcommand (and androidinnerloop already marks analogous args as required). Marking them as required=True improves CLI help/validation and avoids late failures.

        iosinnerloopparser = subparsers.add_parser(const.IOSINNERLOOP,
                                                 description='measure first and incremental build+deploy time via binlogs (iOS)')
        iosinnerloopparser.add_argument('--csproj-path', help='Path to .csproj file to build', dest='csprojpath')
        iosinnerloopparser.add_argument('--edit-src', help='Modified source file paths, semicolon-separated', dest='editsrc')
        iosinnerloopparser.add_argument('--edit-dest', help='Destination paths for modified files, semicolon-separated', dest='editdest')
        iosinnerloopparser.add_argument('--framework', '-f', help='Target framework (e.g., net11.0-ios)', dest='framework')
        iosinnerloopparser.add_argument('--configuration', '-c', help='Build configuration', dest='configuration', default='Debug')
        iosinnerloopparser.add_argument('--msbuild-args', help='Additional MSBuild arguments', dest='msbuildargs', default='')
        iosinnerloopparser.add_argument('--bundle-id', help='iOS bundle identifier', dest='bundleid')
        iosinnerloopparser.add_argument('--device-id', help='iOS device ID (UDID for physical device, simulator ID or "booted" for simulator)', dest='deviceid', default='booted')

src/scenarios/mauiiosinnerloop/post.py:20

• The scenario sets ApplicationId to net.dot.mauiiosinnerloop (pre.py) and the Helix work item uses --bundle-id net.dot.mauiiosinnerloop, but post.py uninstalls com.companyname.<exename>. This likely leaves the tested app installed and can interfere with subsequent runs / device state.

try:
    bundle_id = f'com.companyname.{EXENAME.lower()}'
    ios_rid = os.environ.get('IOS_RID', 'iossimulator-arm64')
    is_physical = (ios_rid == 'ios-arm64')

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:176

• This on-agent runner uses --bundle-id com.companyname.mauiiosinnerloop, but the scenario’s csproj is rewritten to use net.dot.mauiiosinnerloop (pre.py) and Helix work items pass that bundle id. Using a different bundle id here will fail install/launch/uninstall consistency and makes results incomparable.

python3 test.py iosinnerloop \
  --csproj-path app/MauiiOSInnerLoop.csproj \
  --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" \
  --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" \
  --bundle-id com.companyname.mauiiosinnerloop \
  -f "$framework-ios" -c Debug \

eng/pipelines/sdk-perf-jobs.yml:24

• Hard-coding large parts of the pipeline behind ${{ if false }} disables public correctness jobs and many private/scheduled jobs. If this file is merged as-is, CI coverage for unrelated areas will be silently turned off. Prefer keeping the original parameter-gated conditions (e.g. parameters.runPublicJobs, parameters.runScheduledPrivateJobs) and gate only the new iOS inner loop jobs, or use a dedicated feature branch for WIP pipeline experiments.

######################################################

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP
  - job: Tooling_Tests_Windows
    displayName: Tooling Tests (Windows)
    timeoutInMinutes: 30
    condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/main'))

src/scenarios/shared/ioshelper.py:97

• security delete-certificate -Z expects a SHA-1 fingerprint (40 hex characters / 20 bytes). _EXPIRED_WWDR_SHA1 is 42 hex characters, so the delete command will never match anything and the expired WWDR intermediate will remain (potentially reintroducing the chain-building failure this code is trying to fix). Recommend capturing the correct SHA-1 from security find-certificate -a -Z -c "Apple Worldwide Developer Relations Certification Authority" and updating this constant accordingly (or deriving it dynamically).

# The old Apple WWDR intermediate (no G-series), expired 2023-02-07. Its lingering
# presence makes codesign build the chain through it (candidate matched by issuer
# name) and fail "unable to build chain to self-signed root" even though the valid
# WWDR G3 is installed. The documented fix is to delete it. SHA-1:
_EXPIRED_WWDR_SHA1 = "FF677979793A3CD798DC5B2ABEF56F73EDC9F83A64"

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (8)

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

• The bundle ID passed to the scenario doesn't match the ApplicationId configured by pre.py and the Helix .proj (net.dot.mauiiosinnerloop). With the current value, the launch/measurement will fail because the installed app's bundle identifier won't match.

  --bundle-id com.companyname.mauiiosinnerloop \

src/scenarios/mauiiosinnerloop/post.py:18

• post.py uses com.companyname. as the bundle id, but the scenario sets ApplicationId/bundle id to net.dot.mauiiosinnerloop. This mismatch can cause uninstall/terminate to target the wrong app (leaving the measured app installed or running).

    bundle_id = f'com.companyname.{EXENAME.lower()}'

src/scenarios/shared/ioshelper.py:97

• _EXPIRED_WWDR_SHA1 is not a valid SHA-1 fingerprint length (SHA-1 must be 40 hex characters). As written, security delete-certificate -Z will never match the expired intermediate, so the cleanup won't work.

_EXPIRED_WWDR_SHA1 = "FF677979793A3CD798DC5B2ABEF56F73EDC9F83A64"

src/scenarios/shared/ioshelper.py:509

• This runs sudo security authorizationdb write com.apple.trust-settings.admin allow, which is a persistent system-wide policy change and can weaken machine security across work items. This should be opt-in (or handled by queue provisioning) rather than performed automatically by the scenario.

            auth = subprocess.run(
                ["sudo", "-n", "security", "authorizationdb", "write",
                 "com.apple.trust-settings.admin", "allow"],
                capture_output=True, text=True)
            getLogger().info(
                "authorizationdb write trust-settings.admin allow -> %d: %s",
                auth.returncode, (auth.stderr or auth.stdout or "").strip()[:200])

eng/pipelines/sdk-perf-jobs.yml:20

• This hard-disables all public jobs regardless of the runPublicJobs parameter, which can unintentionally remove PR correctness coverage. If this is meant to be temporary WIP gating, it should be controlled via parameters rather than if false.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

• This second if false block also disables public scenario jobs unconditionally. Restoring the parameter-based condition avoids silently skipping CI coverage.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

• This hard-disables scheduled private jobs regardless of the runScheduledPrivateJobs parameter, which will stop scheduled perf coverage entirely.

- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:367

• This if false block disables the standard private scenario job matrix even when runPrivateJobs is true, leaving only the iOS-innerloop jobs enabled. If this is intended as a temporary WIP reduction, it should be parameterized (or isolated in a separate pipeline) so normal private perf coverage isn't accidentally dropped when enabling private jobs.

  - ${{ if false }}: # disabled: iOS-only WIP test run

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (9)

src/scenarios/mauiiosinnerloop/post.py:18

• post.py uninstalls a different bundle id (com.companyname.*) than the scenario builds/launches (net.dot.mauiiosinnerloop from pre.py and maui_scenarios_ios.proj). This will leave the app installed between runs and can also uninstall the wrong app if present.

    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

• The on-agent driver passes --bundle-id com.companyname.mauiiosinnerloop, but the scenario csproj is forced to ApplicationId net.dot.mauiiosinnerloop in pre.py. This mismatch will make install/launch/uninstall target the wrong app id.

  --bundle-id com.companyname.mauiiosinnerloop \

src/scenarios/shared/ioshelper.py:97

• _EXPIRED_WWDR_SHA1 is used with security delete-certificate -Z, which expects a SHA-1 fingerprint (40 hex chars). The current value is 42 hex chars, so the delete step will never match/remove the intended expired WWDR cert.

# The old Apple WWDR intermediate (no G-series), expired 2023-02-07. Its lingering
# presence makes codesign build the chain through it (candidate matched by issuer
# name) and fail "unable to build chain to self-signed root" even though the valid
# WWDR G3 is installed. The documented fix is to delete it. SHA-1:
_EXPIRED_WWDR_SHA1 = "FF677979793A3CD798DC5B2ABEF56F73EDC9F83A64"

src/scenarios/shared/runner.py:234

• --csproj-path is validated later via a runtime exception, but argparse can enforce it up-front (as AndroidInnerLoop does). Marking it required improves CLI UX and avoids failing late in the run.

        iosinnerloopparser = subparsers.add_parser(const.IOSINNERLOOP,
                                                 description='measure first and incremental build+deploy time via binlogs (iOS)')
        iosinnerloopparser.add_argument('--csproj-path', help='Path to .csproj file to build', dest='csprojpath')
        iosinnerloopparser.add_argument('--edit-src', help='Modified source file paths, semicolon-separated', dest='editsrc')

eng/pipelines/sdk-perf-jobs.yml:20

• This change hard-disables the entire public correctness job block by switching the condition to ${{ if false }}. That will prevent Tooling_Tests_Windows (and other public PR validation jobs under this block) from running at all.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

• This second ${{ if false }} disables the public scenario benchmark jobs block entirely (win-x64/ubuntu-x64). If this is meant to be temporary, it should stay parameter-driven so PR validation coverage isn't silently removed.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:369

• Private CI coverage is being turned off via ${{ if false }} wrappers (starting here), which disables the standard private scenario jobs. This is a large reduction in coverage unrelated to adding iOS inner-loop, and is likely to mask regressions.

  - ${{ if false }}: # disabled: iOS-only WIP test run
    - template: /eng/pipelines/templates/build-machine-matrix.yml
      parameters:

eng/pipelines/sdk-perf-jobs.yml:826

• Scheduled private jobs are fully disabled by changing the condition to ${{ if false }}. This prevents scheduled coverage from running even when parameters.runScheduledPrivateJobs is true.

- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:738

• These comments describe device signing as requiring an embedded.mobileprovision file and a sign tool, but ioshelper/setup_helix.py in this PR explicitly state the provisioning profile is downloaded and signing is done via codesign/keychain (and that no sign tool exists on perf Macs). Keeping this comment incorrect will mislead future triage.

  # Device variants need the Helix Mac device queue's physical iPhone + code-signing infra
  # (embedded.mobileprovision + sign tool). Those live on Mac.iPhone.17.Perf, but the queue
  # config currently points at Mac.iPhone.13.Perf (the only queue with Xcode 26.5); device
  # signing therefore needs provisioning on .13 (or Xcode 26.5 on .17). See build-machine-matrix.yml.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (6)

src/scenarios/mauiiosinnerloop/post.py:18

• pre.py sets the MAUI app's ApplicationId/bundle id to net.dot.mauiiosinnerloop, but post-cleanup derives com.companyname.<exename>. This mismatch prevents uninstall/terminate from working and can leave stale apps installed on shared machines.

    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:175

• The app bundle id is set to net.dot.mauiiosinnerloop by pre.py and used by the Helix work item. This on-agent runner uses com.companyname.mauiiosinnerloop, so install/launch will target the wrong app id.

  --bundle-id com.companyname.mauiiosinnerloop \

eng/pipelines/sdk-perf-jobs.yml:20

• This hard-codes the runPublicJobs gating to false, so public correctness and PR validation jobs will never run even when parameters.runPublicJobs is true.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

• This hard-codes the scenario benchmark gating to false, so the public scenario jobs will never run even when parameters.runPublicJobs is true.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

• This hard-codes the scheduled private jobs gating to false, so scheduled perf runs will never execute even when parameters.runScheduledPrivateJobs is true.

- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP

src/scenarios/shared/ioshelper.py:516

• Writing com.apple.trust-settings.admin to allow via security authorizationdb write makes a persistent, machine-wide policy change and weakens the host's security posture. This should be avoided on shared CI machines; prefer queue pre-provisioning or a narrowly-scoped, opt-in mechanism with clear rollback.

            auth = subprocess.run(
                ["sudo", "-n", "security", "authorizationdb", "write",
                 "com.apple.trust-settings.admin", "allow"],
                capture_output=True, text=True)

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (6)

src/scenarios/mauiiosinnerloop/post.py:18

• The cleanup script uninstalls a different bundle id than the scenario builds/installs (pre.py sets ApplicationId to net.dot.mauiiosinnerloop, and the Helix work item passes --bundle-id net.dot.mauiiosinnerloop). With the current com.companyname... value, simulator/device cleanup will often be a no-op, leaving the app installed and potentially affecting subsequent runs.

    bundle_id = f'com.companyname.{EXENAME.lower()}'

eng/pipelines/templates/run-ios-innerloop-on-agent.sh:176

• This on-agent runner passes --bundle-id com.companyname.mauiiosinnerloop, but the scenario’s pre.py rewrites the project ApplicationId to net.dot.mauiiosinnerloop. If this dormant path is enabled, install/launch will fail because the bundle id won’t match the built app’s Info.plist.

python3 test.py iosinnerloop \
  --csproj-path app/MauiiOSInnerLoop.csproj \
  --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" \
  --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" \
  --bundle-id com.companyname.mauiiosinnerloop \
  -f "$framework-ios" -c Debug \

src/scenarios/shared/ioshelper.py:523

• security authorizationdb write com.apple.trust-settings.admin allow modifies the machine’s global authorization database and persists beyond the work item. On shared/macOS hosted agents this is a high-risk side effect that can affect other jobs and weaken security posture. Consider gating this behavior behind an explicit opt-in env var (or remove it) and rely on best-effort add-trusted-cert when not opted-in.

            auth = subprocess.run(
                ["sudo", "-n", "security", "authorizationdb", "write",
                 "com.apple.trust-settings.admin", "allow"],
                capture_output=True, text=True)

eng/pipelines/sdk-perf-jobs.yml:20

• Public jobs are currently hard-disabled with if false, which bypasses the runPublicJobs parameter and changes pipeline behavior (public correctness checks and scenario jobs never run even when requested). This should be reverted to the original parameter gate so CI can be configured predictably.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:57

• This second if false block also hard-disables the runPublicJobs section and leaves subsequent public jobs outside any gate, which can unintentionally run jobs even when runPublicJobs is false. Restoring the parameter condition avoids both disabling and accidental always-on jobs.

- ${{ if false }}: # parameters.runPublicJobs — disabled for iOS Inner Loop WIP

eng/pipelines/sdk-perf-jobs.yml:826

• Scheduled private jobs are hard-disabled with if false, so runScheduledPrivateJobs can no longer enable them. This effectively turns off all scheduled perf validation and should be restored to use the pipeline parameter gate.

- ${{ if false }}: # parameters.runScheduledPrivateJobs — disabled for iOS Inner Loop WIP