Debugging Flaky Observer Tests in CI

Observer tests fail intermittently for a small number of reasons, and all of them come down to the same root cause: the test asserted before the browser or the runtime had finished doing something asynchronous. A slower machine simply widens the window in which that is visible.

Problem / Scenario Context

The pattern is familiar. A lazy-loading test passes a hundred times locally and fails once in every fifteen CI runs, always on the same assertion, always with a message that reads as if the feature is broken:

expect(received).toBe(expected)
Expected: "/hero.jpg"
Received: null

Retrying the job makes it pass, which is the worst possible outcome — it teaches the team that the test is unreliable rather than that the wait is missing, and the flake gets an @flaky tag instead of a fix. Meanwhile the same missing wait is quietly present in production code, where nothing retries.

Mechanics Explanation

Four mechanisms produce nearly all observer flakes.

A missing microtask flush. Observer delivery is asynchronous by contract. A test that triggers an entry and asserts on the next line is racing the callback. Locally the microtask queue drains almost instantly and the assertion usually wins by luck of scheduling; under load it does not.

Real timers in a dwell test. Impression tracking pairs an observer with a timer. If the test uses real timers and a 1000 ms dwell, it passes when the runner is idle and fails when the event loop is busy, because the timer fires late. Fake timers remove the machine from the equation entirely.

requestAnimationFrame in a headless browser with no compositor. Code that coalesces work into an animation frame — the rAF batching pattern — depends on frames being produced. A backgrounded or offscreen headless page may throttle them heavily, so work that "always" runs within 16 ms takes hundreds.

Shared module state across test files. A pool or a static instances array that is not reset lets one file's observer be picked up by the next file's helper. This one is not timing-related at all, but it presents identically: intermittent, order-dependent, unreproducible locally where files often run in a different order.

Comparison Table: Symptom to Cause

Minimal Reproducible Example

// Reproduce the flake deterministically by making the queue busy
it('is racy', async () => {
  render(<LazyImage dataSrc="/hero.jpg" />);
  const img = screen.getByRole('img');
  lastObserver().trigger(img, true);
  for (let i = 0; i < 5; i++) queueMicrotask(() => {});   // simulate a loaded queue
  expect(img.getAttribute('src')).toBe('/hero.jpg');      // now fails every time
});

If a test fails with those five extra microtasks in place, it was always a race — the CI machine was simply providing them for you.

Production-Safe Solution

Flush explicitly, in a helper the whole suite shares.

export const flushObservers = () => new Promise<void>((r) => queueMicrotask(() => r()));
export const flushFrames = (n = 1) =>
  new Promise<void>((r) => {
    let left = n;
    const step = () => (--left <= 0 ? r() : requestAnimationFrame(step));
    requestAnimationFrame(step);
  });

Use fake timers for anything with a dwell.

beforeEach(() => vi.useFakeTimers());
afterEach(() => vi.useRealTimers());

it('records an impression after one second in view', async () => {
  render(<Impression id="a" />);
  await flushObservers();
  lastObserver().trigger(screen.getByTestId('a'), true, 0.6);
  await flushObservers();          // callback runs and starts the timer
  vi.advanceTimersByTime(1000);    // ONLY now does the dwell complete
  expect(beacon).toHaveBeenCalledTimes(1);
});

The order is the part people get wrong: advancing the clock before the flush does nothing, because the timer has not been created yet.

Reset shared state centrally.

afterEach(() => {
  MockIntersectionObserver.instances.length = 0;
  disposeAll();                    // the shared observer pool
  document.body.innerHTML = '';
});

Prefer consequence-based waits in browser tests, as described in testing ResizeObserver callbacks with Playwright, and never paper over a race with a longer global timeout.

Verification Steps

• Prove it is a race, not a bug: insert the five extra microtasks from the reproduction above. A genuine race becomes a hundred-per-cent failure.
• Run the suite in a random file order (--sequence.shuffle in Vitest, --randomize in Jest) locally. Order-dependent state leaks surface immediately.
• Run with reduced parallelism on CI once to confirm the failure rate tracks machine load rather than a specific test.
• Grep for waitForTimeout and bare setTimeout in tests. Each one is a candidate cause.
• Check the retry count is zero. A suite configured with retries hides exactly the failures this page is about.

Common Mistakes to Avoid

• Raising the global test timeout. That makes a slow failure into a slower failure; it does not remove the race.
• Adding await new Promise(r => setTimeout(r, 50)). It works locally for the same reason the bug is invisible locally.
• Enabling test retries to get the pipeline green. The flake is now permanently invisible, and the underlying missing wait usually exists in the production code path too.
• Assuming headless equals headed. Frame scheduling genuinely differs; anything that waits on requestAnimationFrame needs a consequence-based wait instead.

FAQ

Related

• Mocking IntersectionObserver in Jest and Vitest — the flush helper these fixes depend on
• Testing ResizeObserver Callbacks with Playwright — consequence-based waiting in a real browser
• Simulating Viewport Intersection in Unit Tests — building sequences that a browser could actually deliver

↑ Back to Testing Observers in JSDOM & Real Browsers