Skip to content

test: compare the Sauce Labs iOS app checks with committed baselines - #1299

Merged
plum117 merged 3 commits into
mainfrom
test/sauce-ios-baselines
Oct 9, 2026
Merged

plum117 merged 3 commits into
mainfrom
test/sauce-ios-baselines

Conversation

@plum117

@plum117 plum117 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The iOS checks of the Sauce Labs app job had no committed baselines: tests/sauceLabsBaseline/ had only the Android emulator (Google Pixel 8a). Because autoSaveBaseline was on (the default) in the Sauce config, every run saved a new baseline for the iOS sessions, and those 4 checks per session passed without a comparison.

This PR adds real iOS baselines collected from Sauce Labs and turns autoSaveBaseline off in CI, like #1274 did for LambdaTest.

Changes

  • tests/sauceLabsBaseline/: 24 iOS baselines: iPhone 8, iPhone 8 Plus (iOS 14.5), iPhone XS, iPhone 11 (15.5), iPhone 12, iPhone 12 Pro Max (16.2) simulators × 4 checks (app-forms, app-login-button, app-login-button-resized, web-app).
  • tests/configs/wdio.saucelabs.shared.conf.ts: autoSaveBaseline: !process.env.CI. In CI a missing baseline now fails; local runs can still create baselines.
  • A temporary commit uploaded the baselines from CI; it is reverted, so the workflows do not change.

How the baselines were collected

  • Run 37873418437 (round 1) and a re-run of the Sauce job (round 2) on this branch, with a temporary upload of tests/sauceLabsBaseline/.
  • The 24 iOS images of both rounds are identical (0 different pixels). I looked at the images: complete screens (login form, webview page, login button), no blank or black screen.
  • The 12 committed Android baselines did not change.

Test

  • Final run 37879291323 (no temporary step, autosave off in CI): all 5 cloud jobs pass. Sauce app: 9/9 spec files, the 6 iOS sessions now compare with the committed baselines (4 passing each), no "Baseline image not found".

Note (not part of this PR)

In round 1, the LambdaTest Android web job failed once: Android 15 portrait in the -vp426x823 state, "screen with ignore elements", 0.066 % (tolerance 0.05 %). That baseline was made with the old test order before #1274; the difference is only at the edges of the ignored button regions. It passed in the final run (another display state). A separate PR will refresh that baseline.

🤖 Generated with Claude Code

Remove before merge. It collects the iOS app baselines, which are not
committed yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 21e5501

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@plum117
plum117 marked this pull request as ready for review October 9, 2026 02:20
@plum117
plum117 marked this pull request as draft October 9, 2026 02:20
@greptile-apps

greptile-apps Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Low impact] The PR appears safe to merge; no blocking issue was found.

Summary

Adds 24 iOS reference images and turns off autoSaveBaseline in Sauce Labs CI runs.

  • Sauce iPhone checks compare screenshots with committed baselines in CI.

Reviews (2) · Last reviewed commit: "Revert "ci: TEMPORARY upload the Sauce L..." · Reviewed by Greptile

dprevost-LMI and others added 2 commits October 8, 2026 23:26
The iOS sessions of the Sauce Labs app job (iPhone 8, 8 Plus, XS, 11,
12, 12 Pro Max simulators on iOS 14.5, 15.5 and 16.2) had no committed
baselines, so every run saved a new baseline and the 4 iOS checks passed
without a comparison. The 24 baselines come from two CI runs of this
branch, which gave identical images.

With the baselines in place, autoSaveBaseline is now off in CI for the
Sauce Labs configs, like for LambdaTest: a missing baseline fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The iOS baselines are collected, so the temporary upload step is not
needed any more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@plum117
plum117 marked this pull request as ready for review October 9, 2026 09:40
@plum117
plum117 merged commit f51b015 into main Oct 9, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants