Welch's t-Test for Performance Regressions

A build that reports Largest Contentful Paint at the 75th percentile of 2,591 ms against a baseline of 2,425 ms on a mid-range Android phone throttled to Fast 3G looks like a regression — but a single pair of numbers cannot tell you whether the 166 ms gap is a real slowdown or the same jitter that moves a no-op commit by a hundred milliseconds. This how-to is part of the Statistical Noise & Flakiness Reduction reference, and it shows how to answer that question precisely with Welch's t-test: collect several runs per build, treat each build as a distribution rather than a point, and only fail the gate when the difference between the two distributions is larger than their run-to-run noise would plausibly produce.

Welch's t-test is the right default for continuous performance metrics because it does not assume the two builds have the same variance — and in practice they never do. The sibling guide on Statistical Significance Testing for Noisy CI surveys the full menu of tests; this page zooms all the way into the one you will reach for first and walks the arithmetic, the code, and the gate end to end.

Why You Compare Distributions, Not Single Numbers

Every synthetic run is a random draw. Cold caches, a noisy neighbour on the CI runner, garbage-collection timing, and TCP retransmits all nudge a metric up or down between otherwise identical runs. When you record only one number per build you are comparing one sample from Build A against one sample from Build B, and the difference you see is the true change plus the difference in their random noise. On a fleet of shared GitHub-hosted runners it is common to see LCP at the 75th percentile on a mid-range mobile profile over Fast 3G swing by 100–150 ms between back-to-back runs of the exact same commit.

The fix is to stop asking "did the number go up?" and start asking "did the distribution move by more than its own spread?" If you draw the two builds as bell-shaped clouds of possible run outcomes, a real regression shifts the whole cloud to the right; noise just widens it. When the clouds overlap heavily, a single run from either build could have come from the other, and no honest gate can separate them. When they barely overlap, the shift is real. Welch's t-test formalises exactly this overlap into a single number.

Welch's t-Test for Unequal Variances

The classic Student's t-test assumes both samples share one variance. Performance data violates that constantly: a build that adds a lazy-loaded hero image can make LCP both slower and noisier, so its spread differs from the baseline's. Welch's t-test drops the equal-variance assumption. It divides the difference in means by the standard error of that difference, where each sample contributes its own variance:

The statistic is t = (mean_B − mean_A) / sqrt(s1²/n1 + s2²/n2), where s1² and s2² are the two sample variances and n1, n2 the run counts. Because the variances are unequal, the test also adjusts the degrees of freedom with the Welch–Satterthwaite formula, which blends both variances and sample sizes into an effective df that is usually smaller than n1 + n2 − 2. A larger absolute t means the shift between builds is large relative to their combined noise, and the df tells you how heavy-tailed the reference distribution is when you convert t into a p-value.

Collecting N Runs Per Build

The test needs a sample from each build, not a single measurement. Collect the runs under identical conditions — same URL, same throttling, same runner class — so the only difference between the two samples is the code. Five runs is the practical minimum; eight to ten gives the degrees of freedom enough room that a modest true regression clears the significance bar. Fewer runs means wider standard errors, which means only large regressions register as significant.

The table below shows how sample size trades off against sensitivity for LCP at the 75th percentile on a mid-range mobile profile over Fast 3G, assuming a per-run standard deviation near 35 ms. "Detectable shift" is roughly the smallest true regression that reaches significance at the 5% level with that many runs.

Run collection is where variance control pays off: throttling drift and background load on shared runners inflate s1² and s2² and blunt the test. The techniques in Reducing Lighthouse CI Variance in Staging — pinning CPU throttling, warming caches, and isolating the runner — shrink the denominator so a smaller real shift becomes detectable at the same sample size.

Computing t and the p-value

Work a concrete example. Suppose you collect eight runs of LCP at the 75th percentile on a mid-range Android phone over Fast 3G from each build, in milliseconds:

baseline: 2420 2380 2460 2410 2440 2390 2470 2430
pr:       2560 2610 2540 2650 2580 2600 2620 2570

The baseline mean is 2,425 ms with sample variance s1² = 1000 (standard deviation about 32 ms). The PR mean is 2,591.25 ms with sample variance s2² = 1269.6 (standard deviation about 36 ms). The difference of means is 166.25 ms. The standard error is sqrt(1000/8 + 1269.6/8) = sqrt(125 + 158.7) = sqrt(283.7) = 16.84, so t = 166.25 / 16.84 = 9.87. The Welch–Satterthwaite degrees of freedom work out to about 13.8. Converting a t of 9.87 on ~14 degrees of freedom to a two-sided p-value gives a number far below 0.0001 — the shift is enormously larger than the noise, so this is a real regression, not jitter.

The p-value answers a precise question: if the two builds were truly identical, how often would pure noise produce a |t| at least this large? A p-value of 0.05 means "once in twenty" — your chosen tolerance for a false alarm. You compare p against a significance threshold (usually 0.05) and fail the gate only when p falls below it.

A JavaScript Implementation

This script computes Welch's t-test with no dependencies. It implements the Student's t distribution through the regularized incomplete beta function, so it returns an exact two-sided p-value rather than a normal approximation — which matters at the small degrees of freedom you get from a handful of runs. It reads two whitespace-separated files of numbers when you pass them as arguments and otherwise falls back to the worked sample above.

// welch.js — node welch.js [baseline.txt] [pr.txt]
// Welch's two-sample t-test for performance regressions.
const fs = require("fs");

function loadOr(path, fallback) {
  if (path && fs.existsSync(path)) {
    return fs.readFileSync(path, "utf8").trim().split(/\s+/).map(Number);
  }
  return fallback;
}

const BASELINE = loadOr(process.argv[2], [2420, 2380, 2460, 2410, 2440, 2390, 2470, 2430]);
const PR = loadOr(process.argv[3], [2560, 2610, 2540, 2650, 2580, 2600, 2620, 2570]);
const ALPHA = 0.05; // two-sided significance level
const MIN_EFFECT_MS = 50; // ignore regressions smaller than this on LCP P75

const mean = (a) => a.reduce((s, x) => s + x, 0) / a.length;
const variance = (a) => {
  const m = mean(a);
  return a.reduce((s, x) => s + (x - m) ** 2, 0) / (a.length - 1);
};

// Log-gamma via the Lanczos approximation.
function logGamma(x) {
  const c = [
    76.18009172947146, -86.50532032941677, 24.01409824083091,
    -1.231739572450155, 0.1208650973866179e-2, -0.5395239384953e-5,
  ];
  let a = x;
  let tmp = x + 5.5;
  tmp -= (x + 0.5) * Math.log(tmp);
  let ser = 1.000000000190015;
  for (let j = 0; j < 6; j++) {
    a += 1;
    ser += c[j] / a;
  }
  return -tmp + Math.log((2.5066282746310005 * ser) / x);
}

// Continued fraction for the incomplete beta function.
function betacf(a, b, x) {
  const FPMIN = 1e-30;
  const qab = a + b;
  const qap = a + 1;
  const qam = a - 1;
  let c = 1;
  let d = 1 - (qab * x) / qap;
  if (Math.abs(d) < FPMIN) d = FPMIN;
  d = 1 / d;
  let h = d;
  for (let m = 1; m <= 200; m++) {
    const m2 = 2 * m;
    let aa = (m * (b - m) * x) / ((qam + m2) * (a + m2));
    d = 1 + aa * d;
    if (Math.abs(d) < FPMIN) d = FPMIN;
    c = 1 + aa / c;
    if (Math.abs(c) < FPMIN) c = FPMIN;
    d = 1 / d;
    h *= d * c;
    aa = (-(a + m) * (qab + m) * x) / ((a + m2) * (qap + m2));
    d = 1 + aa * d;
    if (Math.abs(d) < FPMIN) d = FPMIN;
    c = 1 + aa / c;
    if (Math.abs(c) < FPMIN) c = FPMIN;
    d = 1 / d;
    const del = d * c;
    h *= del;
    if (Math.abs(del - 1) < 3e-7) break;
  }
  return h;
}

// Regularized incomplete beta I_x(a, b).
function betai(a, b, x) {
  if (x <= 0) return 0;
  if (x >= 1) return 1;
  const front = Math.exp(
    logGamma(a + b) - logGamma(a) - logGamma(b) + a * Math.log(x) + b * Math.log(1 - x)
  );
  if (x < (a + 1) / (a + b + 2)) return (front * betacf(a, b, x)) / a;
  return 1 - (front * betacf(b, a, 1 - x)) / b;
}

// Two-sided p-value for Student's t with df degrees of freedom.
function tTwoSidedP(t, df) {
  return betai(df / 2, 0.5, df / (df + t * t));
}

function welch(a, b) {
  const ma = mean(a);
  const mb = mean(b);
  const va = variance(a);
  const vb = variance(b);
  const se = Math.sqrt(va / a.length + vb / b.length);
  const t = (mb - ma) / se;
  const num = (va / a.length + vb / b.length) ** 2;
  const den =
    (va / a.length) ** 2 / (a.length - 1) + (vb / b.length) ** 2 / (b.length - 1);
  const df = num / den;
  return { ma, mb, delta: mb - ma, t, df, p: tTwoSidedP(t, df) };
}

const r = welch(BASELINE, PR);
const regressed = r.delta > MIN_EFFECT_MS && r.p < ALPHA;
console.log(`baseline mean : ${r.ma.toFixed(1)} ms`);
console.log(`pr mean       : ${r.mb.toFixed(1)} ms`);
console.log(`delta         : ${r.delta.toFixed(1)} ms`);
console.log(`t statistic   : ${r.t.toFixed(2)}`);
console.log(`welch df      : ${r.df.toFixed(1)}`);
console.log(`p-value       : ${r.p.toExponential(2)}`);
console.log(regressed ? "FAIL: significant regression" : "PASS: within noise");
process.exit(regressed ? 1 : 0);

Run it with no arguments and it prints t = 9.87, df ≈ 13.8, a p-value near 4e-06, and FAIL. Notice the two independent gates in the verdict: the difference must clear a minimum effect size (here 50 ms on LCP at the 75th percentile for mid-range mobile over Fast 3G) and be statistically significant. Requiring both keeps you from failing on a 6 ms shift that happens to be significant because the runner was unusually quiet.

Wiring the CI Gate

To gate a pull request, collect matched samples from the base branch and the PR branch, extract the metric from each Lighthouse run, and feed the two files to the script. The exit code drives the pass or fail. The extractor below turns a .lighthouseci directory into a line of numbers.

// extract-lcp.js <lhci-dir> — prints one LCP value per run, in milliseconds.
const fs = require("fs");
const dir = process.argv[2];
const values = fs
  .readdirSync(dir)
  .filter((f) => f.startsWith("lhr-") && f.endsWith(".json"))
  .map((f) => JSON.parse(fs.readFileSync(`${dir}/${f}`, "utf8")))
  .map((lhr) => Math.round(lhr.audits["largest-contentful-paint"].numericValue));
process.stdout.write(values.join(" ") + "\n");

The workflow checks out the base branch first, collects eight runs, extracts the sample, then repeats on the PR commit before running the gate.

name: perf-regression-gate
on: pull_request
jobs:
  welch-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Collect baseline sample
        run: |
          git checkout origin/${{ github.base_ref }}
          npm ci && npm run build
          npx lhci collect --url=http://localhost:8080/ --numberOfRuns=8
          node scripts/extract-lcp.js .lighthouseci > baseline.txt
      - name: Collect PR sample
        run: |
          git checkout ${{ github.sha }}
          npm ci && npm run build
          rm -rf .lighthouseci
          npx lhci collect --url=http://localhost:8080/ --numberOfRuns=8
          node scripts/extract-lcp.js .lighthouseci > pr.txt
      - name: Welch gate
        run: node scripts/welch.js baseline.txt pr.txt

Because the base sample is re-collected on the same runner in the same job, it inherits the runner's current speed — so a slow runner shifts both samples together and the test still measures only the code difference. That is more robust than comparing against a stored historical mean, though a rolling baseline is worth adding once your runner fleet is stable; see how Configuring Statistical Regression Alerts formalises the alerting side, and how Percentile-Based Threshold Tuning sets the ceilings the significance test defends.

Caveats When Data Is Non-Normal

Welch's t-test assumes each sample is drawn from a roughly normal distribution. For LCP, TBT, and Total Bytes on well-controlled runners that assumption holds well enough, especially after you aggregate to a per-run percentile. But it breaks in three situations. First, heavy skew: interaction metrics like INP often have a long right tail from occasional slow frames, and a couple of large runs can inflate the mean and the variance together, weakening the test. Second, tiny samples: with three runs the normality assumption is untestable and the p-value is fragile. Third, multi-modal data: if a route sometimes serves a cached hero and sometimes a cold one, the metric has two clusters and no single mean describes it.

When any of these apply, stop leaning on the mean. A distribution-free alternative such as the Mann-Whitney U test compares ranks and shrugs off outliers, and a resampling approach makes no shape assumption at all. For confidence intervals on a skewed metric like LCP the resampling route is the safest — walk through it in Bootstrap Confidence Intervals for LCP, which builds an interval by repeatedly resampling your runs rather than trusting a t distribution. A good rule of thumb: use Welch as the default for LCP and byte-weight metrics at the 75th percentile on mid-range mobile over Fast 3G, and switch to bootstrap or Mann-Whitney the moment a metric's runs look visibly skewed or bimodal.

Frequently Asked Questions