Statistical Noise & Flakiness Reduction

A performance gate that fails one build in five for no code reason is worse than no gate at all — engineers learn to re-run until green, and the signal is gone. This is the variance-control layer of the Threshold Calibration & Baseline Management reference: it turns a jittery lab measurement into a stable number you can assert against, by collecting multiple runs, reducing them to a median, controlling the runner environment, and sizing tolerances to the residual noise you cannot remove.

The core problem is that a single Lighthouse or WebPageTest run is a sample, not a measurement. V8 garbage-collection timing, CPU contention from noisy neighbours, cold CDN caches, and background network activity all push individual metrics around by 10–30% even when the code under test is byte-identical. On a shared GitHub-hosted runner, a single LCP reading for a page whose true P75 on mid-range mobile at Fast 3G is 2600 ms can land anywhere from 2200 ms to 3400 ms across consecutive runs. The job here is to shrink that spread until it is smaller than the regressions you care about, then set the gate just above the residual.

How Variance Becomes a Stable Signal

Every noise source feeds the raw spread of a metric. Mitigations attack those sources, and what survives is collapsed by a median-of-N into a single gateable value. The diagram traces that flow from cause to a narrowed distribution.

The mental model to hold onto is that you are never trying to eliminate variance — that is impossible on shared infrastructure — you are trying to push the measurement noise well below the smallest regression you would ever want to block. If your team cares about a 150 ms LCP regression on mid-range mobile at Fast 3G, and your median-of-5 has a standard error of 40 ms, you have comfortable headroom. If that standard error is 120 ms, the gate cannot tell a real 150 ms regression from a lucky-versus-unlucky pair of runs, and it will either flake or leak. Everything below is in service of shrinking that standard error.

Prerequisites & Environment

Variance reduction starts with a runner you control. The default GitHub-hosted runner is acceptable for simulate throttling but is a shared 2-vCPU box, so never use real-network or provided throttling on it. The settings below are the deterministic baseline; tighten the CPU multiplier against your real device target using Device & Network Emulation Weighting.

• @lhci/cli ≥ 0.13 with the version pinned in package-lock.json so scoring weights do not shift mid-quarter.
• Pinned Chrome major version — a Chromium bump can move LCP by 100–200 ms on its own; bake the binary into a container image when reproducibility matters.
• A warmable target URL — a staging deploy whose cache and database you can prime before collection, so the first run is not penalised for a cold start.
• A baseline sample — at least 20 historical runs of the unchanged target so you can measure the coefficient of variation (CV) before choosing a gate tolerance.
• A quiet collection window — schedule long baseline captures away from your busiest CI hours so noisy-neighbour contention on shared runners does not inflate the measured CV and trick you into a looser gate than you need.

Treat the environment as the first thing to fix and the last thing to change. Once you have measured a CV and calibrated a gate on top of it, any later change to the runner image, the throttling profile, or the Chrome version invalidates that calibration — the noise floor you tuned against no longer describes the machine. Pin aggressively, and when a pin has to move, rebuild the baseline before you trust the gate again.

Configuration Reference

The block below is the authoritative noise-control configuration. numberOfRuns raises the sample size, throttlingMethod: simulate models the network in software so timings do not depend on the runner's real bandwidth, and the aggregation method tells Lighthouse CI which run to keep.

{
  "ci": {
    "collect": {
      "url": ["https://staging.example.com/"],
      "numberOfRuns": 5,
      "settings": {
        "preset": "desktop",
        "throttlingMethod": "simulate",
        "throttling": { "cpuSlowdownMultiplier": 4, "requestLatencyMs": 150 },
        "disableStorageReset": false,
        "chromeFlags": "--no-sandbox --disable-dev-shm-usage --disable-background-networking --disable-extensions"
      }
    },
    "assert": {
      "aggregationMethod": "median-run",
      "assertions": {
        "metric-lcp": ["error", { "maxNumericValue": 2600 }],
        "metric-cls": ["error", { "maxNumericValue": 0.1 }],
        "metric-tbt": ["error", { "maxNumericValue": 220 }]
      }
    }
  }
}

numberOfRuns: 5 is the comfortable default for noisy shared runners; 3 is the floor. aggregationMethod: median-run keeps a single internally consistent report rather than mixing the best LCP from one run with the best TBT from another. The --disable-background-networking flag removes a frequent source of late-run jitter. Note that the assertion ceilings are deliberately set a little above the calibrated P75 target to absorb residual noise — sizing that gap is the calibration step below.

One subtlety worth calling out: metric-lcp here is a lab metric on the desktop preset, ceiling 2600 ms, which is a different quantity from the P75 field LCP your RUM pipeline reports. The lab gate is a fast, deterministic proxy that catches large regressions before merge; the field P75 on mid-range mobile at Fast 3G is the number your users actually experience. Keep them distinct in your head and in your dashboards — a lab gate passing does not prove the field P75 held, only that the pull request did not introduce a lab-visible regression.

Anatomy of a Noisy Metric

Before you can size a tolerance you have to see the shape of your own noise. Collect ten or more runs of the unchanged target and plot the metric per run. Two things jump out on almost every real staging target: the bulk of runs cluster tightly, and one or two runs sit far above the pack. Those outliers are almost always the cold-cache first run or a garbage-collection pause, and they are exactly why the mean is the wrong summary statistic. The chart below plots ten LCP samples with one cold-start outlier removed and the median drawn through the dense band of runs.

The chart makes the case for two decisions at once. First, discard or warm away the cold-start run so it never enters the sample — a single 4180 ms reading among nine ~2300 ms readings pulls the mean up by roughly 190 ms and inflates the CV from 1.1% to 24%, which would either widen your gate uselessly or flake it constantly. Second, prefer the median as the summary statistic precisely because it ignores that outlier's magnitude and only counts its rank. When you interpolate a percentile from a small run set rather than reading a raw median, the sparse-sample techniques in Interpolating Percentiles From Sparse Samples keep the estimate honest.

Step-by-Step Implementation

1. Collect a baseline sample of the unchanged target so you can measure its noise floor.

   npx lhci collect --url=https://staging.example.com/ --numberOfRuns=10

   Expected tail: Done running Lighthouse! ten times, with ten reports written to .lighthouseci/.

2. Compute the coefficient of variation for the metric you intend to gate. CV is the standard deviation divided by the mean, expressed as a percentage.

   npx lhci collect --url=https://staging.example.com/ --numberOfRuns=10
   node -e "const fs=require('fs');const v=fs.readdirSync('.lighthouseci').filter(f=>f.endsWith('.json')).map(f=>JSON.parse(fs.readFileSync('.lighthouseci/'+f)).audits['largest-contentful-paint'].numericValue);const m=v.reduce((a,b)=>a+b)/v.length;const sd=Math.sqrt(v.reduce((a,b)=>a+(b-m)**2,0)/v.length);console.log('mean',m.toFixed(0),'CV%',(100*sd/m).toFixed(1))"

   Expected output resembles mean 2310 CV% 4.2. A CV above 8% means the environment is too noisy to gate tightly — fix the runner before lowering thresholds.

3. Trim the cold-start outlier so it does not distort the CV you just measured. This snippet drops the single slowest run before recomputing, which mirrors what a warm-up curl achieves at collection time.

   node -e "const fs=require('fs');let v=fs.readdirSync('.lighthouseci').filter(f=>f.endsWith('.json')).map(f=>JSON.parse(fs.readFileSync('.lighthouseci/'+f)).audits['largest-contentful-paint'].numericValue).sort((a,b)=>a-b);v=v.slice(0,-1);const m=v.reduce((a,b)=>a+b)/v.length;const sd=Math.sqrt(v.reduce((a,b)=>a+(b-m)**2,0)/v.length);console.log('trimmed mean',m.toFixed(0),'trimmed CV%',(100*sd/m).toFixed(1),'median',v[Math.floor(v.length/2)])"

   Expected output resembles trimmed mean 2312 trimmed CV% 1.1 median 2305. If trimming one run drops the CV from double digits to low single digits, your problem is a cold cache, not a noisy machine.

4. Apply mitigations and re-measure. Switch to simulate throttling, warm the cache, pin Chrome, and re-run step 2. A well-controlled run should land under 3% CV for LCP and under 5% for TBT.

5. Set the assertion ceiling at the median-of-5 target plus a margin equal to two standard deviations, then commit lighthouserc.json.

Threshold Calibration

How many runs you need depends on the CV you measured and how tight a gate you want. The relationship is that the standard error of the median shrinks roughly with the square root of the run count, so doubling precision costs four times the runs. The matrix gives practical starting points by measured noise level.

Keep a metric at warn while its CV is still above target and promote it to error only after the noise floor holds for two consecutive weeks of baselines, mirroring the promotion discipline in Percentile-Based Threshold Tuning. For the field-driven side of choosing the underlying target, see the staging-specific walkthrough in Reducing Lighthouse CI Variance in Staging.

Why More Runs Buy Less Each Time

The square-root law behind the table has a blunt consequence: the second run helps a lot, the ninth barely helps at all. The relative standard error of an N-run median falls as one over the square root of N, so going from one run to three roughly halves the noise, three to five shaves another chunk, and everything past seven is a rounding error paid for in CI minutes. The curve below plots that decay so you can see where the elbow sits for your budget.

This is why the configuration defaults to five and the table only reaches for seven or nine when the raw CV is genuinely bad. If your measured CV forces you into the seven-to-nine band to gate at all, the right move is almost never to keep buying runs — it is to fix the environment so the noise floor drops and five runs suffice again. Runs are a way to average out noise you could not prevent, not a substitute for preventing it.

CI Enforcement Snippet

This GitHub Actions job warms the target, runs five collections, and asserts against the median — the warm-up step removes the cold-cache outlier that otherwise dominates the first run.

name: Performance Gating
on:
  pull_request:
    branches: [main]

jobs:
  lighthouse-ci:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    concurrency:
      group: lhci-${{ github.ref }}
      cancel-in-progress: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
      - run: npm ci
      - run: npm run build
      - name: Warm the target cache
        run: curl -s -o /dev/null https://staging.example.com/ || true
      - name: Run Lighthouse CI (5 runs, median)
        run: npx lhci autorun
        env:
          LHCI_TOKEN: ${{ secrets.LHCI_TOKEN }}
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: lighthouse-reports
          path: .lighthouseci/

When a residual breach is still ambiguous after five runs, decide whether it is real before failing the build using Statistical Significance Testing for Noisy CI, and route confirmed shifts into your trend store via Automated Regression Detection.

Deciding When a Breach Is Real

Even a well-controlled gate will occasionally show a median that sits a few milliseconds over the ceiling with no code change behind it. The discipline that keeps the gate trustworthy is refusing to treat a single over-ceiling median as proof of a regression. Instead, ask whether the new sample of runs is statistically distinguishable from the baseline sample, not merely numerically larger. Two techniques do this cheaply and belong in your toolkit once medians alone stop being decisive.

A two-sample test compares the distribution of the pull request's runs against the distribution of the baseline's runs and returns a probability that the difference is chance. Because performance samples rarely share equal variance, the unequal-variance form is the safe default — walk through it in Welch's t-Test for Performance Regressions. When your sample is small or visibly skewed and you would rather not assume any distribution shape at all, resample the runs you have to build an empirical interval around the median — the method in Bootstrap Confidence Intervals for LCP turns five or ten runs into a defensible range. In both cases the rule is the same: fail the build only when the new distribution is separable from the old one, and let overlapping distributions pass with a warning rather than a red X.

The pattern that ties it together is a two-tier gate. Tier one is the fast numeric ceiling from the configuration above, which catches gross regressions instantly. Tier two, invoked only when tier one is marginally breached, runs the significance check against the stored baseline and either confirms the regression or clears it as noise. This keeps the common case fast and the ambiguous case honest, and it is the single most effective way to stop engineers from learning to re-run until green.

Troubleshooting & Edge Cases

• One run is always 2× slower than the rest → it is the cold-cache first run; warm the target with a curl before lhci collect or discard the first sample explicitly.
• CV good locally, terrible in CI → the runner is the noise source; switch from provided/devtools throttling to simulate, which does not depend on real bandwidth.
• LCP stable but TBT swings wildly → CPU contention from a noisy neighbour; isolate to a dedicated runner or container with a guaranteed CPU quota.
• Variance crept up after a dependency bump → an unpinned Chromium or Lighthouse version changed scoring; pin both in the lockfile and rebuild the baseline.
• Median still drifts week to week with no code change → real baseline movement, not noise; recalibrate against your historical store rather than widening tolerances.
• Background networking spikes late runs → add --disable-background-networking and --disable-sync to chromeFlags.
• Gate flakes only on large pull requests → the extra build artifacts are evicting the warm cache between the warm-up curl and the first Lighthouse run; move the warm-up immediately before lhci autorun or increase numberOfRuns to dilute the cold sample.
• CV is fine but the gate still catches nothing → the ceiling is set too far above the median; a margin of two standard deviations on a 1% CV is only ~50 ms, so recompute the margin from the current noise floor rather than an old one.

Frequently Asked Questions