Add automated browser-based tests to check examples on web editor for console errors

[nbogie]

Addresses #84

What is this?

Adds playwright-based config and tests to check the p5.sound examples in place on the web editor, checking they load and don't have any errors in the console.

How to run the tests (with test-runner UI)

This is recommended as it allows easier exploration of test failures:

1. once, to install playwright

npm install

2. launch the test runner's UI

npm run test:integration:ui

3. in the gui that opens, click "projects" and ensure "firefox" and "chromium" are checked. (see image)

4. Also in the gui, press the play button on the top test or above it. (see image)

Alternative: How to run the tests (unattended):

npm run test:integration 

This will present a html report when it finishes.

What are the additions and changes?

• Adds playwright as a dev dependency in package.json
  • (playwright also downloads browsers (chromium and firefox) but will not do so until the user runs the tests for the first time)
• Adds playwright.config.js, specifying, for example, where the tests are (./tests/) and which browsers to use.
• Adds tests/integration/test-examples-on-web-editor.spec.js This is the main test file that instructs playwright what web pages to load , what to click, and what to check for.
• Adds tests/integration/about-these-tests.md documenting the testing (and some problems faced in automated testing of the web editor).
• Adds the playwright-generated .github/workflows/playwright.yml.DISABLED - this is untested.

Limitations:

• It runs the tests against firefox and chromium but not safari (webkit) nor on any mobile browsers. Research is needed in how to have webkit granted microphone and webcam permissions and to allow audio context start without user interaction.
• Because I've only written ONE standardised test to exercise ALL p5.sound examples, certain conditions have to be met for an example to be compatible:
  • Example must render a canvas.
  • Example must not require user permissions beyond microphone and camera.
• No CI setup has been tested. It is envisaged this tool will be plenty useful when used on an ad-hoc basis. Certainly once that proves true it would be worth hooking up the CI. The playwright-generated workflow for github actions has been placed in the repo but disabled: .github/workflows/playwright.yml.DISABLED

AI usage disclosure

Claude Code did most of the implementation on this, including research into chrome and firefox support for pre-emptive grants of microphone + video, troubleshooting the main issue with testing the web editor, and the bulk of the write-up about-these-tests.md. That said, I've reworked the code to how I want it, I have read and understood each line, and the PR write-up is all mine.

[nbogie]

update: silenced the headless firefox test run to match chrome.

[ksen0]

Minor extra notes from trying this with @ogbabydiesal :

1. Fresh install will require npx playwright install after npm install as well to get the browser drivers
2. Tommy got 25 passed and 19 failed (yours was 29 and 15) which is a bit surprising, however

As next steps Tommy will work on creating an examples directory in the repository, which should be used instead. Would that require much of a change to this PR?

Thanks so much for getting this started!

[nbogie]

I am happy to alter it to work from examples/ this weekend. It should generally be a simplification, I think.

(copied from my discord response:)

If the examples are going to continue to be published on the web editor (I hope so - I love the availability and the subsequent discoverability in the collection), then I'd probably argue mildly that we might try to have the tests be capable of running against sketches in both locations, assuming it doesn't add a lot of maintenance.

In the end the actual test process doesn't change wherever they are, only some implementation details for each step.

1. access the sketch
2. start sketch if necessary
3. interact minimally with the canvas
4. wait a second
5. check no errors in console

Pros of running the automated tests against the editor:

• Tests what the large majority of users will actually be interacting with
• Ensures, after a library or example change, that all examples have been successfully uploaded / updated / created, including assets
  • unless the publication step from examples/ is automated these transfers/updates will be a significant source of errors

Cons of running the automated tests against the editor:

• Changes to the web editor can break the tests
• Testing framework is slightly more complex, slower, and less obvious ("why does the test wait here before pressing play?")
  • maintenance, troubleshooting, and other contribution is therefore harder
• Test machine needs to be online to run
• (Hypothetical) eventual CI run of the tests would be flakier due to web editor dependency (outages, quirks, weather)
  • if we engineered the tests to be able to run in both locations, I'd suggest CI runs them only locally

[ogbabydiesal]

hey @nbogie I created a repo with collection in the examples folder, there's only one for now but this would be the folder hierarchy if you'd like to test with that. I'll add the rest soon

https://github.com/processing/p5.sound.js.git

I hear you about running the tests in the editor as that is how many people will interact with them first, if your into the idea of running them both places, and it isn't too much work you can use that repo!

[ksen0]

@ogbabydiesal not sure if that's the intended link but can you make the examples dir be inthis repo please, like in p5.sound.js? Totally ok if there's just 1 example! Thanks so much both

[nbogie]

Hi @ksen0 @ogbabydiesal
In case it's helpful i've made a PR with all the examples downloaded verbatim into examples/ and a tracking issue.

(Sorry if this work is already underway elsewhere).