CI: test/firestore.test.tsx flakes intermittently on both Node 22 and Node 24

test/firestore.test.tsx fails intermittently in CI and passes on re-run with no code change. It has now fired five times in three days, every time on a PR that cannot plausibly cause it, including a lockfile-only dependency bump and a README-only docs change.

Every occurrence costs a full CI cycle and a manual re-run, and it trains reviewers to ignore red runs.

⚠️ Two corrections, 2026-08-05

This issue previously said the flake was Node 24 only and that the RESOURCE_EXHAUSTED error was ruled out. Both were wrong, and the earlier investigation comment is superseded on those two points.

It is not Node-24-specific. On docs: replace the concurrent mode note with a Suspense section #778, a README-only diff, Test Node.js 22 failed twice consecutively while Test Node.js 24 passed in the same runs. The earlier "Node 22 passes every time" claim rested on a local sample of 6 runs, which was underpowered rather than clean.

RESOURCE_EXHAUSTED is not ruled out. It is absent from every local reproduction, which is all the evidence supported. In CI it is present, including on the docs: replace the concurrent mode note with a Suspense section #778 failures. This raises the possibility that the local repro and the CI failure are not the same bug, which matters because the fix below was measured against the local one.

Signature

AssertionError: expected 'loading' to deeply equal 'success'
Expected: "success"
Received: "loading"

In CI, alongside it on the same run:

@firebase/firestore: Firestore (11.10.0): GrpcConnection RPC 'Listen' stream error.
Code: 8 Message: 8 RESOURCE_EXHAUSTED: Received message larger than max (795107700 vs 4194304)

A ~795 MB (and in an earlier run ~3.5 GB) claimed frame against a 4 MB cap is a gRPC framing desync: the reader is interpreting payload bytes as a length header. That is a @grpc/grpc-js-layer failure, not an application one.

Locally the failure looks different: waitFor hitting its 1000ms default, at 1058ms and 1142ms, plus one 120s hang, with no gRPC error at all.

Occurrences

Date	PR	What it touched	Which Node
2026-08-05	#778	`README.md` only	22 (twice in a row; 24 passed)
2026-08-04	#774	`package-lock.json` only (postcss bump)	24
2026-08-03	#770	`src/` import specifiers only	24
2026-08-03	(earlier run)	no source changes	24

⚠️ The #778 pair is the first time it did not clear on a single re-run, and the first time Node 22 failed. Prior occurrences all passed on the first re-run.

The two tests that fail

Both are in test/firestore.test.tsx, and the same two every time:

useFirestoreDocData > returns undefined if document does not exist (~line 100)
useFirestoreDocOnce > works when the document does not exist, and does not update when it is created (~line 174)

They are the two tests that read a document that does not exist. That is the shared property, and it is more specific than "these tests are slow". A non-existent document cannot be answered from the local cache, so the hook stays in loading until the Listen stream confirms absence. That round trip is what exceeds waitFor's 1000ms budget, so these two fail first whenever the stream is slow or broken.

Suspected cause

The current suspect is @grpc/grpc-js 1.9.16, which @firebase/firestore pins via ~1.9.0. A framing desync is exactly the class of bug that lives at that layer, which is what the CI-side RESOURCE_EXHAUSTED points at.

Measured locally, on Node 24:

Arm	Failures
Baseline (`@grpc/grpc-js` 1.9.16)	6 / 44 (13.6%)
`overrides` forcing 1.14.4	0 / 38

This is suggestive, not proven, and the caveats have grown:

The pre-registered 30-runs-per-arm comparison came back p = 0.12, which is not significant. Pooling all batches gives p = 0.02, but that pooling was chosen after seeing the results.
The measurement was against the local failure mode, which has no gRPC error. If the CI failure is a different bug, these numbers do not speak to it.
The Node 22 result that made Node 24 look special has since been contradicted in CI.

Two constraints on any fix:

Upgrading firebase does not help. Every published @firebase/firestore through 4.17.0 still pins ~1.9.0. An overrides entry is the only lever in this repo.
An override makes CI test a @grpc/grpc-js that consumers do not get. If 1.9.16 is genuinely unwell, the bug still reaches anyone running reactfire server-side, and the real fix belongs upstream in firebase-js-sdk. An override here buys a quiet CI, not a fixed product.

Suggested scope

Apply the overrides bump as an uncontrolled fix, and say so plainly rather than claiming it is verified.

⚠️ Merging it means CI never runs 1.9.16 again, so no further baseline accumulates. It is a before-and-after with no control, not an experiment. The honest reading afterwards is:

flake stops → consistent with the theory, but not proof
flake continues → the theory is wrong, and that is genuinely informative

If it continues, the 120s hang is the next thread. Note that raising the waitFor timeout would not have saved that run, so "just bump the timeout" is not a sufficient fallback.

Note for reviewers

Until this is fixed: a red Test Node.js 22 or Test Node.js 24 on a PR touching no source is probably this. Re-run before investigating, and say so in the PR thread rather than leaving a red run unexplained. Do not conclude anything from a single run in either direction, since one pass does not clear a branch and one failure does not condemn it. As of #778, assume it may take more than one re-run.

Reproduced this locally and ran a controlled comparison. Two results: the RESOURCE_EXHAUSTED lead in the description is a dead end, and the @grpc/grpc-js version is a strong but not conclusive suspect.

Correction to the description

The Received message larger than max error appeared in none of the local failures. The actual signature is waitFor hitting its 1000ms default, failing at 1058ms and 1142ms, plus one run where the test hung for 120s.

So the 3.5 GB frame co-occurs with the flake rather than causing it. Anyone picking this up should not start from the "corrupted length prefix" theory, which is what the description currently points at.

What was measured

Each run is the full firestore suite against the emulator. All Node 24 runs on the same machine, same suite, back to back.

Config	Target failures
Node 22, `@grpc/grpc-js` 1.9.16 (what we ship)	0 / 6
Node 24, `@grpc/grpc-js` 1.9.16	6 / 44 (13.6%)
Node 24, `@grpc/grpc-js` 1.14.4 (forced via `overrides`)	0 / 38

Two additional failures were excluded as unrelated infrastructure noise (emulator startup crashes, distinct exit codes and no test assertion).

How strong is this

Weaker than the table suggests, and worth stating plainly:

The pre-registered comparison was 30 runs per arm on Node 24: 3/30 vs 0/30, Fisher exact p = 0.12. Not significant.
Pooling every batch gives 6/44 vs 0/38, p = 0.02, but that pooling was chosen after seeing the results, so it is weaker evidence than the number looks.
Put the other way: if the override arm had the baseline arm's failure rate, the chance of seeing zero failures across 30 runs is about 1.2%.

The correlation is real and the mechanism is plausible, since 1.9.16 predates Node 24 by a wide margin. It is not proven, and it should not be repeated as settled.

The constraint on any fix

@firebase/firestore pins @grpc/grpc-js at ~1.9.0. Every published version through the current 4.17.0 still does, so this cannot be fixed by upgrading the firebase dependency. An overrides entry is the only lever available in this repo.

That has a cost worth naming: an override here means CI tests a @grpc/grpc-js that consumers do not get. If 1.9.16 really is unwell on Node 24, the bug still reaches anyone running reactfire server-side on Node 24, and the real fix belongs in firebase-js-sdk.

Suggested next step

Try the override and let CI accumulate evidence, rather than spending more local runs chasing significance. If the flake stops across the next few weeks of PRs, that is a stronger signal than any batch run here. If it does not, the grpc-js theory is wrong and the 120s hang is the thread to pull next, since a raised waitFor timeout would not have saved that run.

ci: address review feedback on the firestore flake probe

ci: add a manual firestore flake probe for #776 (#780)

First controlled measurement in CI: 120 runs, zero failures

The probe from #780 is on main and ran today at its defaults, 30 iterations per arm, both arms sequentially on one runner: run 31204989659.

Node	Arm	grpc-js	Pass
22	baseline	1.9.16	30
22	override	1.14.4	30
24	baseline	1.9.16	30
24	override	1.14.4	30

Read this as a negative result, not as good news. Three things follow.

The 13.6% local rate does not reproduce in CI when the firestore suite runs alone. Zero failures in 60 baseline runs; at p = 0.12 that has probability 0.88^60, about 0.05%. By the rule of three the true rate for this configuration is under 5% at 95% confidence.
The override arm is uninformative. It cannot improve on a baseline that never failed, so there is still no controlled evidence either way on the @grpc/grpc-js bump. The comparison this issue wants has not been made yet.
Zero RESOURCE_EXHAUSTED in 120 CI runs, which is the part that changes the diagnosis below.

The 08-05 correction was right to fire, but "local vs CI" is the wrong axis

That correction framed the discriminator as environment: absent locally, present in CI. These 120 runs were in CI, on CI hardware, and the gRPC error did not appear once.

So the variable is the workload, not the environment. The failing runs are full npm run test: five emulators, the whole suite, parallel vitest workers. The probe is one emulator and one file. Everything that distinguishes a failing CI run from a passing probe run is on that axis: emulator contention, concurrent streams, total memory and CPU pressure.

That is a more useful place to look than the client library in isolation, and it also means the local repro and the CI failure being "different bugs" may be understating it: the isolated firestore suite may simply not be capable of producing either one at a measurable rate.

New occurrence, and a diagnostic worth having

Add to the occurrence table:

Date	PR	What it touched	Which Node
2026-08-06	#781	`example/` comments only	24

Same two tests, same assertion, alongside RESOURCE_EXHAUSTED: Received message larger than max (1633837427 vs 4194304). It cleared on a single re-run today, so the "#778 needed two" pattern has not repeated.

The bogus frame lengths are ASCII, and that is direct evidence for the framing-desync reading. Decoding the two exact values on record as big-endian bytes:

Claimed length	Hex	As bytes
795107700	`0x2f646174`	`/dat`
1633837427	`0x61626173`	`abas`

Both are printable, and both are 4-byte windows into the same string: /databases, at offset 0 and offset 4. That is the Firestore resource path, projects/<project>/databases/(default)/documents.

So the reader is not producing a garbage number from corrupted binary. It is reading four bytes of the resource path out of the message body and using them as the length prefix. That upgrades the framing-desync diagnosis from an inference about implausible sizes to something directly observed, and it means the desync offset varies between runs rather than being a fixed misalignment.

What this means for the suggested scope

The proposal above is to apply the overrides bump as an uncontrolled fix and say so plainly. That still holds, and the honest reading of the aftermath is unchanged. But one caveat gets sharper: we now know the isolated suite cannot serve as a control at all, so merging the override forecloses the comparison entirely rather than merely making it harder.

Before doing that, the cheaper move is to widen the probe to run the actual CI workload. That is tracked in #783 along with the timeout arithmetic it forces (a full-suite iteration is roughly 57s against 8s here, so the job timeout has to rise, and the arms must stay on one runner).

Correction to the timing figure in my comment above

That comment closed by saying a full-suite probe iteration is "roughly 57s against 8s here, so the job timeout has to rise."

The 57s was the whole Test Node.js 22 job, including checkout, npm ci, Java setup, functions install and artifact download. The Run tests step alone is 23s. Corrected figures:

Mode	Per iteration	30 per arm (60 runs)	50 per arm (100 runs)
firestore-only	~8s	~10 min	~16 min
full suite	~23s	~28 min	~43 min

Both fit inside the existing 60 minute job timeout, so widening the probe to the real CI workload is cheaper than I said. Nothing else in that comment changes: the 120/120 clean result, the exclusion of the 13.6% rate for the isolated suite, the workload-not-environment reframing, and the ASCII decoding of the bogus frame lengths all stand.

The widened probe is now open as #785.

One finding from building it that belongs on this issue, because it affects how any future measurement should be read: test/auth.test.tsx, test/firestore.test.tsx and test/database.test.tsx each open with a double check - emulator is running health check, and those fail by timing out. So an emulator that never came up looks, in a log, exactly like a test hanging. Anyone triaging a red CI run here should check whether the health check failed before concluding anything about a hang, and that includes the 120s hang recorded in this issue.

New occurrence, and it is the most informative one yet

#785, which adds one workflow_dispatch-only workflow file and touches nothing test.yaml runs. Both test jobs failed; both passed on re-run with no code change.

Date	PR	What it touched	Which Node
2026-08-07	#785	one `workflow_dispatch`-only workflow file	22 and 24, same run

Same two tests as always, useFirestoreDocData > returns undefined if document does not exist and useFirestoreDocOnce > works when the document does not exist, and does not update when it is created, with RESOURCE_EXHAUSTED: Received message larger than max (1650553701 vs 4194304).

This is the first time both Node versions failed in the same run. The previous worst case was #778, where Node 22 failed twice consecutively while Node 24 passed. Two versions failing simultaneously on the same run is evidence against anything Node-specific, and points at conditions shared by the whole run rather than by an interpreter version. That lines up with today's probe result, where the isolated firestore suite came back 120/120 clean and the distinguishing variable turned out to be the workload rather than the environment.

The frame lengths are now three for three

Adding today's value to the two decoded earlier:

Claimed length	Hex	As bytes	Offset in `/databases/`
795107700	`0x2f646174`	`/dat`	0
1633837427	`0x61626173`	`abas`	4
1650553701	`0x62617365`	`base`	5

Three independent samples, every one printable ASCII, every one a four-byte window into the same string, at three different offsets.

Two things follow that a single sample could not establish:

This is not coincidence. One printable value could be luck. Three, all landing inside the same short substring, is the reader taking four bytes of the Firestore resource path (projects/<project>/databases/(default)/documents) and using them as a length prefix.
The misalignment is not a fixed offset. Offsets 0, 4 and 5 mean the desync amount varies per occurrence, so this is not a constant off-by-N in a header parse. It is consistent with the reader resynchronizing at an arbitrary point in the stream, which is what makes stream-level pressure a plausible trigger.

The practical consequence for the fix: the size in the error message is meaningless as a diagnostic. 795107700 and 1650553701 are not "a huge message", they are text. Nobody should chase the magnitude.

ci(flake-probe): fix the errexit abort and scope counting to firestore

ci(flake-probe): add a full-suite workload mode (#785)

The full suite reproduces it, and the `@grpc/grpc-js` override does not fix it

Run 31223456796, full-suite mode, 30 iterations per arm, both Node versions, both arms sequential on one runner. 120 runs total.

Node	Arm	grpc-js	Pass	Flake	Runs with `RESOURCE_EXHAUSTED`	Flakes that also had it
22	baseline	1.9.16	27	3	11	3
22	override	1.14.4	29	1	4	1
24	baseline	1.9.16	28	2	7	2
24	override	1.14.4	27	3	9	3

Zero infra failures, zero hangs and zero unrecognized failures across all 120 runs, so nothing was silently miscounted in either direction.

1. It is the workload, not the environment

9 flakes in 120 full-suite runs (7.5%, 95% CI 4.0 to 13.6). The firestore-only mode returned 0 in 120 earlier the same day, on the same hardware.

That settles the question the 2026-08-05 correction left open. It framed the discriminator as local versus CI, because the gRPC error appeared only in CI. The isolated suite running in CI produces neither the flake nor the gRPC error, so the variable is five emulators and a parallel suite versus one emulator and one file: contention, concurrent streams, resource pressure.

2. `RESOURCE_EXHAUSTED` only exists under the real workload

31 of 120 runs (25.8%), against zero in any firestore-only run ever recorded, local or CI. The signal that motivated this whole investigation is invisible unless the full suite is running.

3. Every flake co-occurred with the gRPC desync, but most desyncs are harmless

The overlap column equals the flake count in all four arms: 3/3, 1/1, 2/2, 3/3. All 9 flakes had RESOURCE_EXHAUSTED in the same run.

The converse does not hold, and it is the more interesting half: 31 desyncs produced only 9 flakes. So the desync looks necessary but is clearly not sufficient. Roughly two thirds of the time the reader resynchronizes without a test noticing.

4. ⚠️ The override does not fix the flake

Arm	Flakes	Rate	`RESOURCE_EXHAUSTED`
baseline (1.9.16)	5 / 60	8.3%	18 / 60
override (1.14.4)	4 / 60	6.7%	13 / 60

Fisher exact, two-sided: p = 1.0 on the flake counts and p = 0.40 on RESOURCE_EXHAUSTED. No detectable effect on either.

The sharp version. The override was proposed on a local 0 out of 38. If it eliminated the flake, 60 override runs should have produced approximately zero. They produced four, with a 95% CI of 2.6% to 15.9% that comfortably excludes zero. P(0 flakes in 60 | the observed 7.5% rate) is about 0.9%. So the local result does not replicate, and "the override fixes it" is ruled out.

⚠️ What this does NOT say. 60 runs per arm can only exclude a large effect. Detecting a halving of an 8% rate would need several hundred per arm. So the honest claim is "does not eliminate the flake", not "does nothing at all". If there is a modest improvement, this run cannot see it.

What follows

The "suggested scope" above should be revisited. It proposed applying the override as an uncontrolled fix and saying so plainly, because no control was available. A control is now available and it says no. Merging the override would buy a quiet conscience rather than a quiet CI, and would still mean CI never runs 1.9.16 again.
The 120 second hang did not reproduce. Zero hangs in 120 runs, and the classifier has a dedicated bucket for exactly that signature, so this is a real absence rather than a miscount. It remains a single recorded observation.
The next thread is the timing one. The firestore suite's own health check, a bare addDoc, took 2245ms in docs(example): drop the concurrent mode framing from the demo entry #781's failing run while every waitFor in that file uses testing-library's 1000ms default. The two tests that fail are the two that read a document that does not exist, which cannot be served from cache and must complete a server round trip. That is consistent with the local failures at 1058ms and 1142ms. The next probe change is to record the firestore health check duration per iteration and test whether slow rounds correlate with flakes.

Reproducing this: workflow_dispatch on Firestore flake probe, defaults. Roughly 22 minutes per job. ⚠️ Note that any table produced before #785 merged is unreliable: the probe aborted on the first failing iteration, so it could only ever report a clean run.

ci(probe): record emulator health-check durations per iteration

Correcting something I posted above: the frame lengths are not ASCII. That was five hand-decoded samples, and they happened to be the printable ones.

I decoded all 31 occurrences from the probe run's artifacts. 25 distinct values, of which 5 are fully printable (abas, atab, data, good, ses/). The other 20 are binary, several with the shape of protobuf varints.

What holds up: 20% printable against a 1.9% chance baseline, so text is over-represented roughly tenfold, and the reader is taking four bytes from inside the message body.

What doesn't: 25 distinct values across 31 events means the resync point is essentially arbitrary. The "slips by exactly one 5-byte gRPC header" reading was an artifact of looking only at the printable ones.

So this thread doesn't discriminate between a grpc-js bug, a Node http2 bug, and the emulator writing a bad frame. Worth flagging that the emulator side has never been tested, and the override experiment couldn't have tested it. The timing lead looks like the better thread, and #787 adds per-iteration health-check durations so we can check it against real data.

ci(probe): record emulator health-check durations per iteration

Following the note at the end of your last comment, that the next probe change is to record the firestore health check per iteration and see whether slow rounds correlate with flakes. I went looking for that data and it turns out most of it is already sitting in run 31223456796's artifacts.

The durations are recoverable from the run you already did

Vitest's default reporter prints a per-test line in two cases: any test slower than slowTestThreshold, which defaults to 300ms and is not overridden in vite.config.ts, and every test in a file that failed, whatever its duration. Every firestore and database health check in that run cleared 300ms, so all 120 printed on the first case alone, and they are in probe-logs-node22 and probe-logs-node24.

To pull them out:

Strip the ANSI codes.
Track the current file heading, and take the double check - emulator is running line that follows it.
Match both heading markers. A passing file is ✓ and a failing one is ❯, so matching only ✓ silently drops every flake iteration. That is the one trap in doing this.

Two checks that the extraction is sound:

Re-classifying all 120 logs with the workflow's own grep patterns reproduces your published tallies exactly: 3 and 1 on Node 22, 2 and 3 on Node 24, zero infra, zero hang.
A firestore duration comes back for 120 of 120 iterations, so nothing is censored by the 300ms threshold and the sample is the whole run.

What they show

Firestore health check, in ms:

Outcome	n	min	q1	median	q3	max
pass	111	1509	1657	1701	1748	1899
flake	9	1562	1661	1721	1732	1777

The flake range sits entirely inside the pass range, and so does the flake interquartile range.
47 of the 111 passing iterations had a slower health check than the flake median.
Database is the same shape, pass median 587 against flake median 574, recovered for 120 of 120 as well.
Auth is the one column not recoverable this way, and it is the one that needs ci(probe): record emulator health-check durations per iteration #787. No test in auth.test.tsx printed a line in any of the 120 iterations, and since that file never failed, that puts every test in it under the 300ms threshold. The whole file runs in 315ms to 556ms, against 1509ms to 1899ms for firestore's health check on its own.

With n = 9 this can only exclude a large effect, the same limit you put on the override comparison. But there is no separation visible at all rather than a small one, and it does answer half the split you set up: two different emulators were answering at their normal speed on the iterations that failed, so runner-wide contention does not look like the trigger.

Why I think it comes out flat: the health check is on the wrong stream

The health check is addDoc, so it times connection setup plus a write round trip.
Both failing waitFor calls wait on the Listen stream. test/firestore.test.tsx:111 observes useFirestoreDocData and :180 observes useFirestoreDoc, both of which resolve to rxfire's doc/docData. That rxfire module imports only onSnapshot, refEqual and getCountFromServer from firebase/firestore, so even the Once variants are a listener with first() applied, not a one-shot read.
Across the same 120 iterations the desync lines split 49 RPC 'Listen' stream to 8 RPC 'Write' stream, with no other RPC appearing. That is lines rather than runs, since one run can log several.

So the health check is a good detector for the runner-wide half of your split and a blind one for the other half, which is the half the evidence points at.

If the recovered numbers look wrong to you I would like to know, since the whole table rests on parsing durations out of a reporter that was never meant to be parsed.

The mechanism is closed end to end, from artifacts run 31223456796 already produced plus SDK source. A flake is the frame desync landing on the Listen stream before the two cache-miss tests run, after which the Firestore SDK parks the stream at maximum backoff for 60 seconds. No dispatch was needed.

1. The failing tests are not slow, they are stalled. Following your recovery method: neither flaking test printed a per-test line on any of the 111 passing iterations, so both always finished under the 300ms slow threshold (control: 1600 ✓ … Nms lines across the artifacts). On all 9 failures both printed, at 1040-1083ms and 1015-1022ms, which is the 1000ms waitFor default plus overhead. At least 700ms of headroom on every pass, pinned at the timeout on every failure.

2. The desync fires at a fixed suite position. In all 31 desync runs it lands right after test/functions.test.tsx completes (~1.6s in). It is not random noise across the run.

3. Position relative to the two vulnerable tests decides the outcome. vitest attributes the stderr to the firestore test in flight: all 9 flakes had it during useFirestoreDoc > can get a Firestore document, before both vulnerable tests; the passing desyncs sat mostly in the collection tests, after them.

4. The control: only RESOURCE_EXHAUSTED kills the run. The same logs hold 19 Listen-stream deaths with codes 13/14/1, including 5 at that same early position. Every one passed. The difference is in the SDK: error.code === RESOURCE_EXHAUSTED → backoff.resetToMax() with DEFAULT_BACKOFF_MAX_DELAY_MS = 60 * 1000 (@firebase/firestore 11.10.0, index.node.cjs.js:23090-23094, :22767), logged as "Using maximum backoff delay to prevent overloading the backend", visible in the flake logs. Codes 13/14/1 get normal backoff and recover in well under a second.

5. Why only those two tests: every other firestore test writes first and is served by latency compensation from the local cache. Only the two nonexistent-doc reads need a live server round trip.

This retires the timing lead I left in comment 5222993659: the 1058ms/1142ms local failures were the same timeout, not round trips finishing just over the line. It also explains why no timeout increase can help (60s ≫ any budget), and the recorded 120s hang is plausibly the same mechanism, though that is inference, not verified.

Still open: what corrupts the frame. CI-only (0 in 240 local iterations, 0 in 120 firestore-only), at a fixed suite offset that coincides with a vitest worker freeing up and starting the next file. That fixed-position signature is a strong repro report for upstream (emulator or grpc-js), and it is narrower than anything we have had on this.

Proposed mitigation, locally verified for no-regression: initializeFirestore(app, { experimentalForceLongPolling: true }) in the test harness. It bypasses the grpc path entirely, which removes the desync class regardless of where the corruption lives. Local runs with it: full suite green on React 18 and 19 (environmental functions failures aside), and 6 of 7 firestore-only runs clean, with the one 2-failure run being the first, cold-start suspected, log not kept, so it stays unexplained rather than explained away. What local runs cannot show is the fix itself, since the desync has never reproduced locally (0 in 240 iterations). The proof is a probe dispatch after it lands: at the historical 26% per-run desync rate, 30 clean iterations is p ≈ 1×10⁻⁴ against no-effect, assuming runs are independent. Two costs stated plainly: the suite stops exercising the grpc transport production Node users get, and the upstream bug stays unfixed, which is what the fixed-position repro signature above is for.

Plan from here: merge #787 (the probe with baseline-only default and per-iteration recording), open the long-polling change as its own PR, dispatch 30 iterations once it lands, and close this on a clean count. #783 then removes the probe, and the corruption signature goes upstream as a report rather than staying our bug to chase.

ci(probe): record emulator health-check durations per iteration (#787)

Correction to the mitigation above: experimentalForceLongPolling does nothing in this test suite. Before pushing it I ran a transport probe with debug logging against the emulator, and the connection is GrpcConnection with the flag on or off. The source confirms it: the Node build's newConnection returns GrpcConnection unconditionally (index.node.cjs.js:22685-88), and no WebChannel implementation exists in that bundle, so the setting parses and then has nothing to act on. My "verified locally" above was a check that could not fail: a green suite with an inert flag proves nothing, and the tests demonstrably run the Node build, which is why CI's desync lines say GrpcConnection.

Replacement, same goal by a different route: fileParallelism: false in the vitest config. The evidence for it is already in this thread: 0 desyncs in 120 firestore-only runs vs 31 in 120 full-suite on the same hardware, and the desync fires exactly when a worker frees up and starts the next file mid-Listen. Serializing files removes the trigger rather than the transport.

Three local serial runs are clean apart from the known Node-version functions failures, and unlike the flag this cannot be silently inert: wall time moves from 42s to ~50s locally, so expect +10-20s per CI test job. One more datapoint from timing that baseline: the two tests failed a parallel local run for the first time, pinned at the timeout with no desync in the log. So contention alone is a second, rarer path to the same symptom, serialization plausibly helps both, and the probe readout should count desyncs and test failures separately rather than assuming one implies the other.

PR coming; the probe dispatch stays the proof, same 30-iteration bar.

test(firestore): let the offline fallback rescue two cache-miss tests

test(firestore): let the offline fallback rescue two cache-miss tests (…

Closing this. #791 landed as 23da94e and the proof run is clean.

Probe run 32061237569 on the merge commit, 30 full-suite iterations per Node version, same workload as the 2026-08-11 baseline:

	08-11 baseline	08-17 with fix	Fisher p
Flakes	11/60	0/60	0.0006
Desyncs	19/60	13/60	0.30
Desyncs that flaked	11/19	0/13	0.0006

The desync rate is statistically unchanged, which is the point: the emulator still corrupts the Listen frame and the SDK still parks the stream on maximum backoff. What stopped is the conversion into a test failure.

Five of those 13 desyncs landed on returns undefined if document does not exist, which passed at 9820, 9857, 9818, 9858 and 9861ms. Under the old 1000ms waitFor default every one would have failed, and vitest would have killed them at its 5000ms default regardless.

The ~9.8s clustering is the offline fallback at ONLINE_STATE_TIMEOUT_MS, not a stream reconnect. An offline client raises the pending snapshot from the local cache, and these two tests assert that a document is absent, which is exactly what an empty cache reports.

The upstream bug (firebase-tools#8654) is untouched, so this survives it rather than fixes it. The cost is that a genuine regression in those two tests now takes up to 120s to surface instead of 1s.

#783 tracks removing the probe workflow.

chore(ci): remove the firestore flake probe workflow (#795)

ci: run the test suite against React 19 (FirebaseExtended#789)

Navigation Menu

Repository navigation

CI: test/firestore.test.tsx flakes intermittently on both Node 22 and Node 24 #776

⚠️ Two corrections, 2026-08-05

Signature

Occurrences

The two tests that fail

Suspected cause

Suggested scope

Note for reviewers

Correction to the description

What was measured

How strong is this

The constraint on any fix

Suggested next step

First controlled measurement in CI: 120 runs, zero failures

The 08-05 correction was right to fire, but "local vs CI" is the wrong axis

New occurrence, and a diagnostic worth having

What this means for the suggested scope

Correction to the timing figure in my comment above

New occurrence, and it is the most informative one yet

The frame lengths are now three for three

The full suite reproduces it, and the `@grpc/grpc-js` override does not fix it

1. It is the workload, not the environment

2. `RESOURCE_EXHAUSTED` only exists under the real workload

3. Every flake co-occurred with the gRPC desync, but most desyncs are harmless

4. ⚠️ The override does not fix the flake

What follows

The durations are recoverable from the run you already did

What they show

Why I think it comes out flat: the health check is on the wrong stream

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CI: test/firestore.test.tsx flakes intermittently on both Node 22 and Node 24 #776

Description

⚠️ Two corrections, 2026-08-05

Signature

Occurrences

The two tests that fail

Suspected cause

Suggested scope

Note for reviewers

Activity

tyler-reitz commented on Aug 4, 2026

Correction to the description

What was measured

How strong is this

The constraint on any fix

Suggested next step

tyler-reitz commented on Aug 7, 2026

First controlled measurement in CI: 120 runs, zero failures

The 08-05 correction was right to fire, but "local vs CI" is the wrong axis

New occurrence, and a diagnostic worth having

What this means for the suggested scope

tyler-reitz commented on Aug 7, 2026

Correction to the timing figure in my comment above

tyler-reitz commented on Aug 7, 2026

New occurrence, and it is the most informative one yet

The frame lengths are now three for three

tyler-reitz commented on Aug 7, 2026

The full suite reproduces it, and the @grpc/grpc-js override does not fix it

1. It is the workload, not the environment

2. RESOURCE_EXHAUSTED only exists under the real workload

3. Every flake co-occurred with the gRPC desync, but most desyncs are harmless

4. ⚠️ The override does not fix the flake

What follows

tyler-reitz commented on Aug 10, 2026

armando-navarro commented on Aug 11, 2026

The durations are recoverable from the run you already did

What they show

Why I think it comes out flat: the health check is on the wrong stream

tyler-reitz commented on Aug 11, 2026

tyler-reitz commented on Aug 11, 2026

tyler-reitz commented on Aug 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The full suite reproduces it, and the `@grpc/grpc-js` override does not fix it

2. `RESOURCE_EXHAUSTED` only exists under the real workload