Skip to content

ci: run the Android mobile web e2e tests on an emulator - #1306

Merged
plum117 merged 7 commits into
webdriverio:mainfrom
plum117:ci/mobile-emulator-suites
Oct 9, 2026
Merged

plum117 merged 7 commits into
webdriverio:mainfrom
plum117:ci/mobile-emulator-suites

Conversation

@plum117

@plum117 plum117 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What

  • New workflow android emulator: an Android 15 emulator (API 35, google_apis, x86_64, KVM) in the GitHub runner, with reactivecircus/android-emulator-runner v2.38.0, Appium 3.8.0 and UiAutomator2 8.7.0. It runs pnpm test.local.emus.web 3 times: a warm-up (the full page test only; its result and files are not kept), a setup run that saves the baselines, and a compare run that compares with them (a missing baseline fails). If the setup run fails, the job stops and the compare run does not happen. The image is google_apis: with a Play Store image, Chrome can update itself during the run; without a Play Store account, Chrome stays the same.
  • The local Android configs take the AVD name and the Android version from ANDROID_AVD / ANDROID_PLATFORM_VERSION (the defaults do not change).
  • AGENTS.md and the verify skill describe the new workflow and the env vars.
  • The LambdaTest Android jobs stay.

Test

  • Run 37947405491: warm-up 1 passing, setup run 7 passing, compare run 7 passing; the job took 7 min 40 s.
  • Why the warm-up: in the first session after the boot, a native screenshot can still show the frame before a scroll, so the full page baseline was stitched wrong (the scroll positions were exact). In 6 CI jobs, 2 of 3 failed without a warm-up and 0 of 3 with one.

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 67ef5f4

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@plum117
plum117 marked this pull request as ready for review October 9, 2026 12:42
@plum117
plum117 marked this pull request as draft October 9, 2026 12:42
@plum117 plum117 closed this Oct 9, 2026
@plum117 plum117 reopened this Oct 9, 2026
dprevost-LMI and others added 3 commits October 9, 2026 09:15
… vars

ANDROID_AVD and ANDROID_PLATFORM_VERSION replace the fixed AVD names
(Pixel_8_Pro_Android_15_API_35, Pixel_7_Pro_Android_14_API_34), which
exist only on one machine. The defaults do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new workflow starts an Android 15 emulator (google_apis, x86_64, KVM) in
the GitHub runner with reactivecircus/android-emulator-runner, starts
Appium 3 with the UiAutomator2 driver, and runs the mobile web suite
twice: the first run saves the baselines, the second compares. It runs
for pull requests that change the core, the service, the mobile web spec
or its configs, and by hand. The LambdaTest Android jobs stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rkflow

The cmdline-tools of the runner image do not know pixel_8_pro.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@plum117
plum117 force-pushed the ci/mobile-emulator-suites branch from a639eae to 03c106b Compare October 9, 2026 13:16
@plum117
plum117 marked this pull request as ready for review October 9, 2026 13:33
@greptile-apps

greptile-apps Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the previous cleanup finding is fixed and no new blocking issues were found.

Summary

Adds Android mobile web tests on a GitHub-hosted emulator.

  • Android mobile web tests now run on a GitHub emulator.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Build and install Appium] --> B[Start Android emulator]
  B --> C[Start Appium and warm up]
  C --> D[Discard warm-up images]
  D --> E[Save baselines]
  E --> F[Compare screenshots]
  F --> G[Stop Appium and clean up crash handlers]
Loading

Reviews (4) · Last reviewed commit: "ci: match crashpad_handler by its comman..." · Reviewed by Greptile

Comment thread .github/scripts/android-emulator-e2e.sh
Comment thread .github/workflows/android-emulator.yml
dprevost-LMI and others added 2 commits October 9, 2026 09:50
…endency changes

- In CI only the setup run (BASELINE_SETUP=true) saves baselines with the
  local Appium configs, so the compare run fails on a missing baseline.
- The workflow also runs when the root or package manifests, the lockfile,
  the workspace file or the pnpm patches change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e job log

The job hung for 20+ minutes after the tests passed: when the action
stops the emulator, it waits for the crashpad_handler processes that the
emulator leaves running (ReactiveCircus/android-emulator-runner#385).
The script now stops Appium and kills those processes when it ends.
The Appium output (6000 lines) goes to logs/appium.log instead of the
job log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread .github/scripts/android-emulator-e2e.sh Outdated
dprevost-LMI and others added 2 commits October 9, 2026 10:52
In the first minutes after the boot the CI emulator is busy, and a native
screenshot can still show the frame before a scroll, so the full page
baseline of the setup run was stitched wrong (the scroll positions were
exact). In an experiment with 6 CI jobs, 2 of 3 failed without a warm-up
and 0 of 3 with one. The warm-up runs the full page test once and does
not keep its files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Linux cuts process names to 15 characters ("crashpad_handle"), so
`pgrep -x crashpad_handler` never matched and the cleanup did not kill
those processes. Use -f with the path of the executable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@plum117
plum117 merged commit 1a00c43 into webdriverio:main Oct 9, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants