The iOS sessions of the Sauce Labs app job (iPhone 8, 8 Plus, XS, 11,
12, 12 Pro Max simulators on iOS 14.5, 15.5 and 16.2) had no committed
baselines, so every run saved a new baseline and the 4 iOS checks passed
without a comparison. The 24 baselines come from two CI runs of this
branch, which gave identical images.
With the baselines in place, autoSaveBaseline is now off in CI for the
Sauce Labs configs, like for LambdaTest: a missing baseline fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
The iOS checks of the Sauce Labs app job had no committed baselines:
tests/sauceLabsBaseline/had only the Android emulator (Google Pixel 8a). BecauseautoSaveBaselinewas on (the default) in the Sauce config, every run saved a new baseline for the iOS sessions, and those 4 checks per session passed without a comparison.This PR adds real iOS baselines collected from Sauce Labs and turns
autoSaveBaselineoff in CI, like #1274 did for LambdaTest.Changes
tests/sauceLabsBaseline/: 24 iOS baselines: iPhone 8, iPhone 8 Plus (iOS 14.5), iPhone XS, iPhone 11 (15.5), iPhone 12, iPhone 12 Pro Max (16.2) simulators × 4 checks (app-forms,app-login-button,app-login-button-resized,web-app).tests/configs/wdio.saucelabs.shared.conf.ts:autoSaveBaseline: !process.env.CI. In CI a missing baseline now fails; local runs can still create baselines.How the baselines were collected
tests/sauceLabsBaseline/.Test
Note (not part of this PR)
In round 1, the LambdaTest Android web job failed once: Android 15 portrait in the
-vp426x823state, "screen with ignore elements", 0.066 % (tolerance 0.05 %). That baseline was made with the old test order before #1274; the difference is only at the edges of the ignored button regions. It passed in the final run (another display state). A separate PR will refresh that baseline.🤖 Generated with Claude Code