Statistical Significance Testing for Noisy CI

A median LCP that ticks up 80 ms on a pull request might be a real regression or it might be the same noise that moves it 80 ms on a no-op commit. Failing the build on the raw delta makes the gate flaky; ignoring it lets regressions through. The fix is to ask a sharper question — is this change larger than the run-to-run noise predicts? — and answer it with a significance test. This guide is part of the Statistical Noise & Flakiness Reduction reference and shows how to compare a baseline sample against a PR sample and only fail when the difference is statistically real.

Every measurement in this guide is collected under one fixed profile: an emulated mid-range mobile CPU throttle on a Fast 3G connection inside the CI runner, the same profile the P75 LCP budget of 3500 ms for mid-range mobile on Fast 3G is calibrated against. Holding the profile constant is what lets a significance test attribute a shift to the code rather than to the environment. If your runner's throttle drifts between runs, no test can rescue the signal — calibrate it first through Calibrating CPU Throttling for CI Runners.

Choosing a Test Method

You are comparing two small samples — N runs from the baseline and N from the PR — and deciding whether their central tendency differs. The right test depends on the distribution shape and on what you want out of the answer.

For Lighthouse metrics with five or more runs, Welch's t-test paired with a confidence interval on the difference is the pragmatic default; reach for Mann-Whitney when a metric is visibly skewed or an outlier survives your median aggregation. When N is tiny or the metric is bimodal — common for LCP when a lazy-loaded hero sometimes wins and sometimes loses the race — a resampling approach like bootstrap confidence intervals for LCP makes no distributional assumption at all. The key principle across all four: never fail on a point estimate alone — require that the confidence interval of the regression excludes zero and that the effect exceeds a minimum size you care about.

Diagnostic Steps

Collect matched samples from the baseline branch and the PR branch under identical settings, then look at the raw spread before testing.

git checkout main && npm run build && npx lhci collect --url=http://localhost:8080/ --numberOfRuns=8
node -e "const fs=require('fs');console.log(fs.readdirSync('.lighthouseci').filter(f=>f.endsWith('.json')).map(f=>JSON.parse(fs.readFileSync('.lighthouseci/'+f)).audits['largest-contentful-paint'].numericValue.toFixed(0)).join(' '))"

Example baseline output: 2310 2280 2350 2300 2330 2290 2360 2320. Repeat on the PR branch into a separate directory; an example PR sample might be 2390 2410 2380 2440 2400 2370 2420 2400. The two sample means sit near 2318 ms and 2401 ms — a gap of about 84 ms — but whether that gap is signal or spread is exactly what the test decides. Eyeballing the numbers, the baseline cluster and the PR cluster barely overlap, which is a promising sign; a comparison where the two clouds of points interleave heavily almost never clears significance no matter how the medians land.

Implementation

This runnable script takes two arrays of metric values, computes Welch's t-test and a 95% confidence interval on the difference of means, and prints a verdict. It uses only Node's standard library.

// significance.js — node significance.js
// Decide if a PR metric sample is a significant regression vs baseline.

const BASELINE = [2310, 2280, 2350, 2300, 2330, 2290, 2360, 2320];
const PR = [2390, 2410, 2380, 2440, 2400, 2370, 2420, 2400];
const MIN_EFFECT_MS = 50; // ignore differences smaller than this
const ALPHA = 0.05; // 95% confidence

const mean = (a) => a.reduce((s, x) => s + x, 0) / a.length;
const variance = (a) => {
  const m = mean(a);
  return a.reduce((s, x) => s + (x - m) ** 2, 0) / (a.length - 1);
};

function welch(a, b) {
  const ma = mean(a), mb = mean(b);
  const va = variance(a), vb = variance(b);
  const se = Math.sqrt(va / a.length + vb / b.length);
  const t = (mb - ma) / se;
  // Welch–Satterthwaite degrees of freedom
  const df =
    (va / a.length + vb / b.length) ** 2 /
    ((va / a.length) ** 2 / (a.length - 1) +
      (vb / b.length) ** 2 / (b.length - 1));
  return { diff: mb - ma, se, t, df };
}

// Two-sided p-value via a t-distribution survival approximation.
function tPValue(t, df) {
  const x = df / (df + t * t);
  // Regularized incomplete beta I_x(df/2, 1/2) gives the tail.
  const ib = incBeta(x, df / 2, 0.5);
  return Math.min(1, ib); // two-sided
}

function incBeta(x, a, b) {
  if (x <= 0) return 0;
  if (x >= 1) return 1;
  const lbeta =
    lgamma(a) + lgamma(b) - lgamma(a + b);
  const front =
    Math.exp(Math.log(x) * a + Math.log(1 - x) * b - lbeta) / a;
  let c = 1, d = 0, f = 1;
  for (let i = 0; i <= 200; i++) {
    const m = Math.floor(i / 2);
    let num;
    if (i === 0) num = 1;
    else if (i % 2 === 0)
      num = (m * (b - m) * x) / ((a + 2 * m - 1) * (a + 2 * m));
    else
      num =
        -((a + m) * (a + b + m) * x) /
        ((a + 2 * m) * (a + 2 * m + 1));
    d = 1 + num * d;
    if (Math.abs(d) < 1e-30) d = 1e-30;
    d = 1 / d;
    c = 1 + num / c;
    if (Math.abs(c) < 1e-30) c = 1e-30;
    f *= d * c;
  }
  return front * (f - 1);
}

function lgamma(z) {
  const g = [
    676.5203681218851, -1259.1392167224028, 771.32342877765313,
    -176.61502916214059, 12.507343278686905, -0.13857109526572012,
    9.9843695780195716e-6, 1.5056327351493116e-7,
  ];
  z -= 1;
  let x = 0.99999999999980993;
  for (let i = 0; i < g.length; i++) x += g[i] / (z + i + 1);
  const t = z + g.length - 0.5;
  return (
    0.5 * Math.log(2 * Math.PI) +
    (z + 0.5) * Math.log(t) -
    t +
    Math.log(x)
  );
}

const { diff, se, t, df } = welch(BASELINE, PR);
const p = tPValue(t, df);
const tCrit = 1.96 + 2.4 / df; // approx two-sided 95% critical t
const ciLow = diff - tCrit * se;
const ciHigh = diff + tCrit * se;
const significant = p < ALPHA && Math.abs(diff) >= MIN_EFFECT_MS;

console.log(`Δ mean        : ${diff.toFixed(1)} ms`);
console.log(`95% CI        : [${ciLow.toFixed(1)}, ${ciHigh.toFixed(1)}] ms`);
console.log(`p-value       : ${p.toFixed(4)}`);
console.log(`min effect    : ${MIN_EFFECT_MS} ms`);
console.log(`verdict       : ${significant ? "REGRESSION (fail)" : "noise (pass)"}`);
process.exit(significant && diff > 0 ? 1 : 0);

Run it after collecting both samples. With the example data it prints a Δ near +84 ms, a CI that excludes zero, a p-value well under 0.05, and exits 1 — a confirmed regression. Two knobs govern its behaviour. ALPHA is the false-positive rate you tolerate on unchanged code: at 0.05, one no-op PR in twenty will trip the gate by chance, so teams that merge dozens of PRs a day often tighten it to 0.01. MIN_EFFECT_MS is the smallest shift worth a red build; setting it well above the run-to-run standard error is what stops a statistically-real-but-trivial 12 ms move from blocking a merge.

Interpreting the Confidence Interval

The p-value tells you whether the difference is distinguishable from zero; the confidence interval tells you how big it plausibly is, which is the number reviewers actually reason about. A CI of [58, 110] ms says the true regression is almost certainly between 58 and 110 ms — small but real. A CI of [-15, 190] ms straddles zero, so even though the point estimate is +87 ms you cannot rule out no change at all, and the gate must pass. Reporting the interval, not just a pass or fail, turns the gate from an opaque gatekeeper into a review artifact: a reviewer can weigh a firm 60 ms LCP regression against the feature that caused it, where the same profile's P75 LCP budget of 3500 ms for mid-range mobile on Fast 3G still has headroom. Pair this with Choosing Between P75 and P90 Budget Targets when you decide which percentile the interval should be judged against.

Sizing the Sample and Setting Alpha

The width of that interval shrinks with the square root of the run count, so how many runs you collect per side sets the smallest regression you can reliably catch. With a coefficient of variation near 4% — typical for a well-isolated LCP measurement on the mid-range-mobile Fast 3G profile — the relationship looks like the chart below. Eight runs per side is the usual sweet spot: it detects a roughly 5% regression at 95% confidence without doubling CI wall-clock time. Dropping to four runs only catches gross 8%-plus regressions; pushing to sixteen buys you sub-3% sensitivity but rarely pays for the minutes it costs.

CI Gating Assertion

Wire the script as the gate so the build fails only on a statistically significant, large-enough regression. The baseline sample is fetched from your trend store rather than recollected, which is faster and more stable.

- name: Collect PR sample
  run: |
    npm run build
    npx lhci collect --url=http://localhost:8080/ --numberOfRuns=8
    node scripts/extract-lcp.js .lighthouseci > pr-sample.json
- name: Fetch baseline sample
  run: curl -s "$BASELINE_API/lcp?branch=main&n=8" -o baseline-sample.json
- name: Significance gate
  run: node scripts/significance.js baseline-sample.json pr-sample.json

Because the gate only fails when the regression's confidence interval clears zero by the minimum effect size, a PR that merely catches a bad noise day passes — exactly the flakiness this removes. Feed confirmed failures into your trend history through Automated Regression Detection so a slow drift across many small-but-insignificant PRs is still caught at the baseline level, and route the alerts through Configuring Statistical Regression Alerts so a real regression pages the right owner instead of dying in a log.

Verification

Validate the gate against two known cases before trusting it. First, run the script with the PR sample set equal to the baseline — it must print a Δ near zero and noise (pass), exiting 0. Second, inject a deliberate +300 ms regression into the PR sample — it must print a CI that excludes zero, a p-value under 0.05, and REGRESSION (fail), exiting 1.

node scripts/significance.js   # with PR === BASELINE → "noise (pass)", exit 0

A correctly tuned gate produces no failures across a week of no-op commits — the false-positive rate should sit under the chosen alpha of 5% — while still failing the injected regression. If no-op commits trip it, raise the run count or the minimum effect size; if the injected regression passes, the minimum effect is set too high. A useful third check is a replay harness: pull the last hundred merged PRs from your trend store, run the gate against each, and count how many would have blocked. If more than a handful of known-good merges turn red, your alpha or minimum effect is miscalibrated for this metric's real noise floor, and tightening the runner isolation buys back more than tuning the test will.

Frequently Asked Questions