Automated Regression Detection

A fixed threshold answers one question — "is this run above the line?" — and answers it badly, because a noisy metric crosses any line by chance and a slowly drifting one never does. Statistical regression detection asks a better question: "is this run's distribution different from the baseline's, beyond what noise explains?" That distinction is the difference between a gate engineers trust and one they learn to re-run until it goes green. This is the change-detection layer of the Threshold Calibration & Baseline Management reference: it compares a candidate run against the distribution captured in your rolling baseline and gates only when the shift is statistically significant.

Detection has three coupled parts — choosing a method (what statistic decides "significant"), sizing the window and sensitivity (how much evidence is enough), and enforcing the decision in CI without drowning the team in false alarms. Tune the method too hot and every wobble blocks a merge; too cold and a real 10% regression sails through. This page is the authoritative spec for all three, and it assumes your candidate runs are already collected under a stable emulation profile so the comparison is apples to apples.

Core Concept: Change Detection vs Fixed Thresholds

A fixed threshold treats each run as a single point against a constant. Change detection treats the candidate as a sample drawn from a distribution and asks whether that distribution has moved relative to the baseline window. Concretely, a mid-range mobile LCP baseline might sit at a P75 of 3500 ms on Fast 3G with a run-to-run standard deviation of 180 ms. A single run at 3650 ms is well inside the noise band and means nothing; five runs averaging 3720 ms with tight spread is a real 6% shift the gate should catch. A fixed 3800 ms ceiling would pass both cases and then fail the day noise alone pushed one run to 3810 ms — exactly the false alarm that erodes trust. The flow below shows the decision: a new run is compared against the baseline distribution, a significance test runs, and only a significant shift alerts or gates.

The practical payoff is that detection separates two failure modes a fixed line conflates. Variance failures — the run that just happened to be slow — are absorbed, because the test weighs the difference against the observed spread. Drift failures — a metric creeping up 1% per week — are caught by an accumulating detector even while every individual run stays under any static ceiling. A budget program that only ships fixed thresholds catches neither cleanly; adding change detection turns the gate from a blunt tripwire into a calibrated instrument.

Prerequisites & Environment

Change detection consumes the rolling baseline distribution, so it inherits everything the baseline needs first. Establish the baseline series and its outlier filtering through Historical Baseline Calibration before enabling detection, and clean the input samples per Statistical Noise & Flakiness Reduction — a detector fed raw noise produces either constant false alarms or a band so wide it is blind. The percentile you baseline against matters too; if you gate on the P75 of mid-range mobile LCP, the detector must compare candidate P75 against baseline P75, a decision covered in Percentile-Based Threshold Tuning.

• A baseline window of samples, not just a number — the detector needs the full set of recent values per metric to estimate the baseline mean and spread, not a single median.
• Deterministic collection — throttlingMethod: simulate, a fixed numberOfRuns, and a pinned device profile (for example mid-range mobile on Fast 3G), so the candidate's spread is comparable to the baseline's rather than reflecting a runner that happened to be busier.
• A defined per-series identity — detection runs per (metric, device, route); never compare a mobile candidate against a desktop baseline, and never pool the P75 of a marketing landing page with a logged-in dashboard whose distribution is genuinely different.
• Enough baseline samples to estimate spread — a Welch t-test on three baseline points has almost no power; aim for 30 or more clean samples per series before you trust a verdict.

Map the inputs through environment variables: DETECT_BASELINE_URL for the baseline window source, DETECT_METHOD to select the test, and DETECT_ALPHA for the significance level. Keeping these in the environment rather than hardcoded lets you dry-run a stricter alpha in a scratch branch without editing committed config.

Choosing a Detection Method

The method decides which statistic separates signal from noise, and no single test is right for every metric. Timing metrics like LCP and TBT on desktop over cable are roughly symmetric and suit a parametric test; INP and mid-range mobile timings are right-skewed with a heavy upper tail and violate the normality a t-test assumes; byte-size metrics rarely spike but drift, and drift needs a memory of past runs a single-shot test does not have. The matrix below maps the three workhorse methods to the shape of data each was built for.

In practice most teams run a hybrid: a Welch or Mann-Whitney test per metric for step regressions, plus a CUSUM detector on the same series to catch creep the single-run tests miss. The two are complementary, not competing — the t-test fires on the deploy that added 400 ms of LCP overnight, and CUSUM fires on the ten deploys that each added 40 ms. For the deeper parametric mechanics, including degrees-of-freedom handling and pooled-variance pitfalls, see Welch's t-Test for Performance Regressions; for the accumulating side, Changepoint Detection for Performance Time Series covers segmenting a series where the shift point itself is unknown.

Configuration Reference

The detection config below is the authoritative spec. It selects the statistical method, the comparison window, and the sensitivity that trades false positives against missed regressions. Every field is explained inline.

{
  "detection": {
    "method": "welch",
    "window": { "baselineSize": 60, "candidateRuns": 5, "minBaseline": 30 },
    "sensitivity": { "alpha": 0.01, "minEffect": { "type": "rel", "value": 0.05 } },
    "direction": "regression-only",
    "metrics": {
      "lcp": { "enabled": true, "level": "error" },
      "inp": { "enabled": true, "method": "mannwhitney", "level": "error" },
      "cls": { "enabled": true, "level": "warn" },
      "script_bytes": { "method": "cusum", "level": "error" }
    }
  }
}

method chooses the statistic: welch (Welch's t-test) for normally distributed timings, mannwhitney for skewed metrics like INP, cusum for catching slow drift that a single-run test misses. alpha is the false-positive rate — 0.01 means a 1% chance of flagging pure noise. minEffect is the floor on practical significance: a difference must be both statistically significant and at least 5% to gate, which suppresses tiny-but-real shifts nobody cares about. On a mid-range mobile LCP baseline of P75 3500 ms on Fast 3G, a 5% floor means the detector ignores anything under 175 ms of movement even if the statistics are technically significant. direction: regression-only ignores improvements so a faster run never fails the build. Note that per-metric overrides — the method on inp and script_bytes — beat the top-level default, letting one config route each metric to the right test.

Step-by-Step Implementation

1. Pull the baseline window and the candidate. Fetch the recent baseline samples per metric and the candidate's runs.

   node scripts/detect-fetch.js --branch main --metric lcp --out baseline_lcp.json

   Expected output: baseline lcp: 60 samples, mean=2180ms sd=140ms confirming enough samples and a usable spread. If the sample count is below minBaseline, the script exits early and the gate should fall back to a static ceiling rather than testing on thin data.

2. Run the significance test. Compare candidate against baseline with the configured method, applying both alpha and minEffect.

   node scripts/detect-run.js --config detection.json --baseline baseline_lcp.json --run .lighthouseci/

   Expected tail: lcp: candidate mean=2460ms Δ=+12.8% p=0.004 → REGRESSION (error) — one line per metric, with the verdict, the observed effect size, and the p-value so a reviewer can see why it fired, not just that it did.

3. Wire the exit code into the gate. A REGRESSION at error level exits non-zero; warn annotates without blocking. Calibrate sensitivity against your own false-positive rate before promoting any metric to error.

The core of the significance step is small enough to read in full. A Welch two-sample test in plain Node, with a one-sided regression check and a minimum-effect gate, looks like this:

function mean(a) { return a.reduce((s, x) => s + x, 0) / a.length; }
function variance(a, m) { return a.reduce((s, x) => s + (x - m) ** 2, 0) / (a.length - 1); }

function welchRegression(baseline, candidate, alpha, minEffectRel) {
  const mb = mean(baseline), mc = mean(candidate);
  const vb = variance(baseline, mb), vc = variance(candidate, mc);
  const nb = baseline.length, nc = candidate.length;
  const se = Math.sqrt(vb / nb + vc / nc);
  const t = (mc - mb) / se;
  const df = (vb / nb + vc / nc) ** 2 /
    ((vb / nb) ** 2 / (nb - 1) + (vc / nc) ** 2 / (nc - 1));
  const pOneSided = 1 - studentCdf(t, df);
  const relEffect = (mc - mb) / mb;
  const regression = pOneSided < alpha && relEffect >= minEffectRel;
  return { mean_baseline: mb, mean_candidate: mc, effect: relEffect, p: pOneSided, regression };
}

The studentCdf(t, df) helper is the cumulative Student's t distribution; ship it from a small stats package or an incomplete-beta implementation. The one-sided 1 - cdf is what makes this regression-only — a faster candidate produces a negative t and a p-value near 1, so it can never trip the gate.

Threshold Calibration

The single dial that matters is sensitivity: lower alpha and higher minEffect mean fewer false alarms but slower detection of real regressions; the reverse catches small shifts fast at the cost of noise. Calibrate by replaying the detector over your last few weeks of known-good runs and counting how often it would have fired — that empirical false-positive rate, not theory, sets the dial. The chart below shows why the choice is not linear: alarms from pure noise climb steeply as alpha loosens, so the jump from 0.01 to 0.05 costs far more than the numbers suggest.

The matrix gives defensible starting points by environment. Mid-range mobile metrics are skewed and noisier, so a rank-based test and a looser alpha keep the false-positive rate tolerable.

Keep every metric at warn until its replayed false-positive rate sits below roughly one alarm per two weeks, then promote to error. The detailed method-by-method tuning lives in Configuring Statistical Regression Alerts. A useful discipline: never change alpha and minEffect in the same commit. Move one, replay the history, read the new false-positive count, then move the other — otherwise you cannot attribute a change in alarm rate to the dial you turned.

Catching Slow Drift With CUSUM

Single-run tests have a blind spot: they compare the candidate against the baseline once, so a metric that gains 30 ms of mid-range mobile LCP every week — never enough to be significant against the immediately preceding window, because the window itself drifts along with it — is invisible to them. CUSUM (cumulative sum) closes that gap by keeping a running total of small deviations from a fixed target. Each run adds (value − target − slack) to an accumulator; while runs hover near target the accumulator stays flat, but a persistent one-sided bias makes it climb until it crosses a decision threshold h and fires. The diagram traces ten runs that each drift a little slow, and the point where the accumulated sum finally trips.

The two knobs are slack (how much per-run deviation to forgive before counting it, usually half the smallest shift you care about) and h (how far the accumulator may climb before firing). A tight h catches drift in a handful of runs but reacts to short benign wobbles; a loose h waits for sustained bias. Because CUSUM has memory, reset the accumulator to zero whenever you promote a new baseline, or it will carry a stale total across the very shift you just accepted as the new normal.

CI Enforcement Snippet

This GitHub Actions job runs detection against the baseline window and gates the merge on a significant regression. It is copy-paste ready and exposes a required status check.

name: Regression Detection
on:
  pull_request:
    branches: [main]

jobs:
  detect:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: "20", cache: "npm" }
      - run: npm ci
      - name: Fetch baseline window
        run: node scripts/detect-fetch.js --branch main --out baseline_window.json
        env: { DETECT_BASELINE_URL: "${{ secrets.DETECT_BASELINE_URL }}" }
      - name: Collect candidate
        run: npx lhci collect && npx lhci upload --target=filesystem
      - name: Run change detection
        run: node scripts/detect-run.js --config detection.json --baseline baseline_window.json --run .lighthouseci/
      - name: Publish verdict
        if: always()
        run: node scripts/detect-comment.js --out detect_report.md

The detect-run.js step exits non-zero only on a significant, large-enough regression, so the job is safe to make a required check. The if: always() on the final step ensures the human-readable report — effect size and p-value per metric — posts to the PR even when the gate fails, so a reviewer sees the evidence without digging through logs. For the alert side — firing on real shifts without paging on noise — see Configuring Statistical Regression Alerts.

Troubleshooting & Edge Cases

• Constant false positives → alpha too high or the baseline window too noisy. Lower alpha to 0.01, raise minEffect, and verify outliers are trimmed upstream per Statistical Noise & Flakiness Reduction.
• Real regressions slip through → minEffect set above the regression size, or too few candidateRuns to reach significance. Raise the run count and lower the effect floor.
• Skewed metric flagged constantly by a t-test → Welch assumes roughly normal data; switch that metric to mannwhitney. INP on mid-range mobile over Fast 3G is the classic offender.
• Slow drift never fires → single-run tests compare one point against the window; add a cusum detector that accumulates small shifts over successive runs, and remember to reset it on baseline promotion.
• Baseline window too small after a reset → fewer than minBaseline samples. Fall back to a static ceiling until the window refills.
• Improvement fails the build → direction not set to regression-only; a two-sided test flags faster runs too.
• Every metric fires at once after an infra change → a runner image or Node upgrade shifted the whole distribution. This is a real change, not a bug in detection; re-baseline deliberately rather than loosening alpha to hide it.

Frequently Asked Questions