Proposal: a noise-tolerant compare mode with pixelmatch 8's windowSize #1311

@wswebcreation, this is a proposal for v11, and I would like your approval (or your changes) before any code is written.

Problem

Today a check counts all different pixels and compares the percentage with a tolerance. This mixes 2 different things:

• Noise: anti-aliasing, GPU dithering or font rendering change many single pixels all over the screenshot. The total can be larger than a small tolerance, so the check fails, and users increase the tolerance or use ignoreAntialiasing.
• A real change in a small area (an icon, a badge, a few letters) changes few pixels in total, so a tolerance can hide it.

What pixelmatch 8 gives us

Since #1289 we use pixelmatch 8. It has a new option: with windowSize: N, pixelmatch returns the highest number of different pixels in any N×N square instead of the total (README). Scattered noise never fills one small square; a real change usually does.

Measured with pixelmatch 8.0.0 on a 1366×768 image, threshold: 0.1:

With a tolerance of 0.05 %, today A fails and B passes. With windows, A is clearly noise and B is clearly a change.

Proposal

A new, optional compare option. Nothing changes when it is not set.

await expect(browser).toMatchScreenSnapshot('home', {
    windowSize: 16,          // compare in 16×16 windows
})

When windowSize is set:

• The check returns the worst window as a percentage of the window area (6 / 256 = 2.3 % for A, 256 / 256 = 100 % for B). The existing tolerance semantics and matchers stay the same: "fail above X %", but X now means "of one window".
• One pixelmatch pass is enough: pixelmatch still writes the diff image, and we already read the different pixels from it for the diff bounding boxes.
• Ignore regions keep working: since refactor: skip ignored regions with pixelmatch's ignoreMask #1301 they are a pixelmatch ignoreMask, so masked pixels are not in any window.
• The diff image does not change (all different pixels are still marked).
• With returnAllCompareData, the result also has the total, so users can see both numbers.

Questions for you

1. Do you agree with this feature for v11?
2. Name and place: windowSize as a compare option (method and service level), like the other compare options?
3. Result: the worst window as a percentage of the window area (proposal above), or the pixel count (with a separate option for the limit)?
4. Defaults: off by default (my proposal), or a recommended value in the docs (pixelmatch's example uses 16)?
5. Reporter: show "worst window" next to the total in the visual reporter now, or later?

Scope

• In: the option in @wdio/image-comparison-core and @wdio/visual-service, unit tests, a local e2e check with known images, docs (compare options on webdriver.io), the v11 guide, a minor changeset.
• Out (later): the reporter display, if you prefer to keep it separate.