Reducing Lighthouse CI Variance in Staging

Staging is where most flaky gates are born: shared host CPU, cold CDN caches, and live-changing seed data push LCP and TBT around far more than the production-like numbers you actually want to assert against. This guide is a practical walkthrough within the Statistical Noise & Flakiness Reduction reference — it isolates the specific variance sources that staging adds, measures them, removes what it can, and shows how to gate around the residual instead of fighting it.

The goal is concrete: get the coefficient of variation (CV) of your gated metrics below roughly 3% for LCP and 5% for TBT, measured at the median of five runs on a high-end mobile profile throttled to a 4G-class network (150 ms RTT, 1638 Kbps), so that a 10% regression is unmistakably larger than the noise. That synthetic median is not your user-facing number — you still gate the field P75 LCP against a production target such as 2500 ms on real mid-range mobile over 4G — but a stable staging signal is what lets a pull request block before the regression ever reaches those users.

Why Variance Breaks a Gate

A performance gate is a hypothesis test in disguise: it asserts that today's build is no slower than a ceiling, and it fails when the measured metric crosses that line. If the metric wobbles by more noise than the regression you care about, the gate can no longer tell the two apart. Concretely, if run-to-run LCP jitter on a high-end mobile / 4G emulated profile is ±9%, then a genuine 8% regression is invisible — it lives inside the noise — while an innocent build can trip the ceiling purely by chance. You get red builds nobody trusts, and engineers start re-running until green, which defeats the gate entirely.

The fix is not a looser ceiling. A looser ceiling raises the P75 LCP target you gate against — say from 2900 ms to 3400 ms on the mobile profile — and quietly ships the regressions you were trying to catch. The fix is to shrink the noise until the tolerance band around your target is a fraction of the smallest regression worth blocking. Everything below is in service of that: measure the CV, remove the controllable variance, then size the ceiling to the residual.

Variance Sources in Staging

Read the table as a priority order, not a menu. Cold caches and CPU contention together account for most of the swing, and both are fully controllable, so fixing them first buys the largest CV reduction for the least effort. Live seed data is sneakier: a staging database that grows through the day changes the number of rows a template renders, which changes the largest contentful element's paint time even though no front-end code moved. Pinning a fixed snapshot removes that drift at the source. The last two rows — network throttling and single-run sampling — are not really about staging at all; they are measurement hygiene you would apply anywhere, and throttlingMethod: simulate plus five runs handle them.

Diagnostic Steps

First, measure the noise floor on the unchanged staging target so you know what you are dealing with before touching thresholds. Ten runs against a single URL, then compute the CV directly from the raw JSON.

npx lhci collect --url=https://staging.example.com/ --numberOfRuns=10
node -e "const fs=require('fs');const v=fs.readdirSync('.lighthouseci').filter(f=>f.endsWith('.json')).map(f=>JSON.parse(fs.readFileSync('.lighthouseci/'+f)).audits['largest-contentful-paint'].numericValue);const m=v.reduce((a,b)=>a+b)/v.length;const sd=Math.sqrt(v.reduce((a,b)=>a+(b-m)**2,0)/v.length);console.log('LCP mean',m.toFixed(0),'ms  CV%',(100*sd/m).toFixed(1))"

Expected output before mitigation looks like LCP mean 2480 ms CV% 9.7 — too noisy to gate. A CV that high on a high-end mobile / 4G emulated profile means your ±2σ band is roughly ±480 ms around the mean, wide enough to swallow a real 15% regression. Next, confirm whether the first run is a cold-cache outlier by inspecting the per-run spread rather than the summary statistic.

node -e "const fs=require('fs');const v=fs.readdirSync('.lighthouseci').filter(f=>f.endsWith('.json')).sort().map(f=>JSON.parse(fs.readFileSync('.lighthouseci/'+f)).audits['largest-contentful-paint'].numericValue);console.log(v.map(x=>x.toFixed(0)).join('  '))"

If the output is something like 3910 2410 2380 2350 2400, the first run is cold — the four warm runs cluster tightly and a warm-up step will recover most of your CV. If instead the numbers scatter evenly with no single outlier, your problem is CPU contention or seed data, not cold cache, and warming the CDN will not help.

Implementation

Apply the four mitigations together: warm the cache, pin the dataset, simulate throttling, and collect five runs. The lighthouserc.js below resolves the staging URL from CI context and bakes in the deterministic settings.

// lighthouserc.js
module.exports = {
  ci: {
    collect: {
      url: [process.env.STAGING_URL || "https://staging.example.com/"],
      numberOfRuns: 5,
      settings: {
        preset: "perf",
        formFactor: "mobile",
        throttlingMethod: "simulate",
        throttling: {
          cpuSlowdownMultiplier: 4,
          rttMs: 150,
          throughputKbps: 1638,
        },
        chromeFlags:
          "--no-sandbox --disable-dev-shm-usage --disable-background-networking --disable-extensions --disable-sync",
      },
    },
    assert: {
      aggregationMethod: "median-run",
      assertions: {
        "metric-lcp": ["error", { maxNumericValue: 2900 }],
        "metric-cls": ["error", { maxNumericValue: 0.1 }],
        "metric-tbt": ["error", { maxNumericValue: 300 }],
      },
    },
  },
};

Every flag here earns its place. throttlingMethod: "simulate" computes the throttled timings from a single unthrottled trace instead of shaping live packets, which removes the runner's variable bandwidth from the measurement — the biggest lever for a stable TTFB. The chromeFlags disable background chatter (sync, extensions, background networking) that would otherwise compete for the same shared CPU. --disable-dev-shm-usage matters specifically on containerised CI runners where a small /dev/shm causes Chrome to fall back to disk and injects random stalls.

Wire the warm-up and data pin into the job. Seeding a fixed snapshot before collection removes the data-driven swing that no Chrome flag can fix.

- name: Pin staging dataset
  run: ./scripts/seed-staging.sh --snapshot fixtures/perf-baseline.sql
- name: Warm cache (discard cold run)
  run: |
    curl -s -o /dev/null https://staging.example.com/
    sleep 2
- name: Collect and assert (5 runs, median)
  run: npx lhci autorun

The CPU multiplier of 4 here is a starting point; calibrate it to your real device target with Calibrating CPU Throttling for CI Runners so the emulated machine matches the hardware your users actually carry. If you run this gate on every pull request, the same deterministic config drops straight into the workflow described in Running Lighthouse CI on Every Pull Request.

CI Gating Assertion

The assertion ceilings are set above the calibrated median to absorb the CV that survives mitigation. With a 2480 ms median LCP on the high-end mobile / 4G emulated profile and a post-mitigation CV near 3%, two standard deviations is about 150 ms, so a P75 LCP ceiling of 2900 ms leaves comfortable headroom while still catching any regression larger than ~15%. Hold the same logic for the other two: a P75 CLS ceiling of 0.1 and a TBT ceiling of 300 ms on that mobile profile both sit well outside their residual noise once the environment is pinned.

{
  "ci": {
    "assert": {
      "aggregationMethod": "median-run",
      "assertions": {
        "metric-lcp": ["error", { "maxNumericValue": 2900 }],
        "metric-tbt": ["warn", { "maxNumericValue": 300 }],
        "metric-cls": ["error", { "maxNumericValue": 0.1 }]
      }
    }
  }
}

Keep TBT at warn until its CV holds under 5% for two weeks, then promote to error. The general rule for choosing which side of the ceiling a metric sits on is the decision below: measure the CV, then let it pick the gate mode.

Sizing the Tolerance Band

Once the CV is low, the ceiling is not a guess — it is the calibrated median plus a margin scaled to the residual noise. A common rule is target-plus-2σ: with a 2480 ms median and a 3% CV on the high-end mobile / 4G profile, σ ≈ 74 ms, so a 2σ margin of ~150 ms yields the 2900 ms P75 LCP ceiling used above. That band fails on any shift larger than roughly two standard deviations, which is where a change stops looking like luck.

If two builds sit close together and you cannot tell from the medians alone whether the difference is real, do not widen the ceiling to make the red go away. Run a significance test instead. Statistical Significance Testing for Noisy CI covers the decision framing, and for the specific case of comparing two run sets, Welch's t-Test for Performance Regressions handles unequal variances cleanly. When you want an interval around the metric rather than a pass/fail, Bootstrap Confidence Intervals for LCP turns your five samples into a defensible range you can gate against.

Verification

Re-run the diagnostic after applying the mitigations and confirm the CV dropped into the gateable band.

npx lhci collect --url=https://staging.example.com/ --numberOfRuns=10

A passing result shows LCP mean 2460 ms CV% 2.8 — under 3% on the high-end mobile / 4G emulated profile, with no single run more than ~6% from the median. The assertion summary should then read All results processed! with no metric within its noise margin of the ceiling. If the first run is still an outlier, your warm-up did not take; check that the curl target matches the audited URL exactly, including trailing slash and any redirect, because a 301 to a canonical URL will warm the wrong cache entry. If the spread is even but still wide, the residual is CPU contention — move to a dedicated runner or a self-hosted runner with pinned cores before you conclude the number is irreducible.

Frequently Asked Questions