Configuring Statistical Regression Alerts

An alert that fires on every noisy run is worse than no alert — the team mutes the channel and the one real regression a month goes unread. The fix is to make the alert statistical: it fires only when a candidate run differs from the baseline distribution beyond what run-to-run variance explains. This guide, part of the Automated Regression Detection reference, shows how to choose between a z-score, a CUSUM, and a Mann–Whitney test, compute the significance, and wire the result into a CI alert that stays quiet until something actually moved.

The choice of test is not cosmetic — each answers a different question about how the metric is allowed to move, and picking the wrong one is the most common reason an alert is either deaf or hysterical. The worked examples below all use one running baseline: a window of P75 LCP samples collected on a mid-range mobile profile (a Moto G-class device throttled to Fast 3G, 4x CPU slowdown) with a mean of 2180 ms and a standard deviation of 140 ms. Every threshold in this guide is stated against that percentile and that device-and-connection context, because a z-score tuned for desktop broadband will misfire the moment you point it at a noisier mobile lane.

Method Comparison

The three tests cover the situations a budget metric actually presents: a single suspicious run, slow accumulating drift, and a skewed metric that breaks normality assumptions. A fourth option, the Welch two-sample test, sits between the z-score and Mann–Whitney when you have two small samples of unequal variance; it is covered in Welch's t-Test for Performance Regressions.

A practical default: z-score for desktop timings, Mann–Whitney for mobile and INP, and a CUSUM running alongside on bundle size to catch the drift that single-run tests miss. The full per-environment dials are in Automated Regression Detection. The decision tree below collapses that guidance into a single glance: read the baseline's skew, pick the per-run test, and layer CUSUM underneath regardless.

Diagnostic Steps

Before wiring an alert, check that the baseline has enough samples and a stable spread, and see what each test would say about the latest candidate.

# 1. Inspect the baseline distribution for the metric
node scripts/detect-fetch.js --branch main --metric lcp --stats
# → lcp  n=60  mean=2180ms  sd=140ms  skew=0.31  (normal-ish → z-score ok)

# 2. Dry-run all three tests against the candidate without alerting
node scripts/alert-eval.js --metric lcp --baseline baseline_lcp.json --run .lighthouseci/ --dry
# → z=3.1 (p=0.002)  cusum=+0.4σ (below h)  mw p=0.01  → z-score & MW agree: REGRESSION

When two independent tests agree, the signal is solid. A low skew (under ~0.5) confirms the z-score is appropriate; a higher skew is the cue to lean on Mann–Whitney instead. The chart below plots that exact diagnostic: the baseline band spans two standard deviations either side of the 2180 ms mean, and the candidate run at 2600 ms lands more than three standard deviations to the right — a P75 LCP shift on mid-range mobile at Fast 3G that no honest reading of variance can explain away.

Tuning CUSUM for Slow Drift

The per-run tests answer "is this run bad?" They are blind to the failure mode where every run is individually innocent but the mean creeps up 3 ms per merge until, six weeks later, the P75 LCP baseline on mid-range mobile at Fast 3G has quietly slid from 2180 ms to 2450 ms. That is the domain of the cumulative sum, or CUSUM. It standardizes each run against the baseline mean and adds the excess above a slack k to a running total; when the total crosses a decision threshold h, it alerts and resets. The slack absorbs ordinary jitter so the sum only climbs when deviations are consistently one-directional.

// scripts/cusum.js  —  detect slow upward drift the per-run tests miss
function cusum(series, mu0, sigma, k = 0.5, h = 4) {
  // k = slack in sd units (ignore drift smaller than half a sigma)
  // h = decision threshold in sd units (alert once accumulated drift exceeds it)
  let sHi = 0;
  const alerts = [];
  series.forEach((x, i) => {
    const z = (x - mu0) / sigma;          // standardize each run
    sHi = Math.max(0, sHi + z - k);       // accumulate one-sided upward drift
    if (sHi > h) { alerts.push({ run: i + 1, sHi: +sHi.toFixed(2) }); sHi = 0; }
  });
  return alerts;
}

With k = 0.5 and h = 4, a drift smaller than half a standard deviation per run never accumulates, but a steady quarter-sigma bias compounds and trips the threshold within about ten runs. Tightening h to 3 makes CUSUM twitchier and shortens the detection lag; loosening it to 5 buys quiet at the cost of a later catch. CUSUM is one of a family of trend detectors — if you need to pin the exact merge where the level shifted rather than just knowing that it did, reach for Changepoint Detection for Performance Time Series. The timeline below shows the running sum staying flat through normal jitter and then climbing once a persistent bias sets in.

Implementation

The snippet below is runnable and self-contained: it computes a z-score and a Mann–Whitney U significance for a candidate against a baseline array, applies the alpha and minimum-effect gates, and emits an alert payload only when a real shift is found.

// scripts/alert-eval.js  —  fire only on statistically real regressions
const ALPHA = 0.01, MIN_EFFECT = 0.05; // 1% false-positive rate, 5% practical floor

const mean = a => a.reduce((s, x) => s + x, 0) / a.length;
const sd = a => { const m = mean(a); return Math.sqrt(mean(a.map(x => (x - m) ** 2))); };

// z-score of the candidate median against the baseline distribution
function zTest(baseline, candidate) {
  const z = (mean(candidate) - mean(baseline)) / (sd(baseline) || 1);
  const p = 0.5 * (1 - erf(Math.abs(z) / Math.SQRT2)); // one-sided
  return { z: +z.toFixed(2), p: +p.toFixed(4) };
}

// Mann–Whitney U: distribution-free, robust to skew
function mannWhitney(a, b) {
  const all = [...a.map(v => [v, "a"]), ...b.map(v => [v, "b"])].sort((x, y) => x[0] - y[0]);
  let rank = 0, Ua = 0;
  all.forEach(([, g], i) => { rank = i + 1; if (g === "a") Ua += rank; });
  const U = Ua - (a.length * (a.length + 1)) / 2;
  const mu = (a.length * b.length) / 2;
  const sigma = Math.sqrt((a.length * b.length * (a.length + b.length + 1)) / 12);
  const z = (U - mu) / sigma;
  return { p: +(0.5 * (1 - erf(Math.abs(z) / Math.SQRT2))).toFixed(4) };
}

function erf(x) { // Abramowitz–Stegun approximation
  const t = 1 / (1 + 0.3275911 * x);
  const y = 1 - (((((1.061405429 * t - 1.453152027) * t) + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t * Math.exp(-x * x);
  return y;
}

export function evaluate(baseline, candidate) {
  const effect = (mean(candidate) - mean(baseline)) / mean(baseline);
  const z = zTest(baseline, candidate);
  const mw = mannWhitney(baseline, candidate);
  const significant = z.p < ALPHA && mw.p < ALPHA && effect >= MIN_EFFECT;
  return { significant, effect: +(effect * 100).toFixed(1), z: z.z, pZ: z.p, pMW: mw.p };
}

A regression is reported only when both tests clear alpha and the effect is at least 5% — requiring agreement plus a practical floor is what keeps the alert quiet on noise. The gate is a logical AND across three independent conditions, and the run is flagged only if all three hold at once; the diagram makes the flow explicit.

CI Gating Assertion

This GitHub Actions step runs the evaluator and posts an alert plus a non-zero exit only on a real regression. The continue-on-error: false is the assertion: a significant shift fails the required check.

- name: Statistical regression alert
  continue-on-error: false
  run: |
    node -e '
      import("./scripts/alert-eval.js").then(async ({ evaluate }) => {
        const fs = await import("node:fs");
        const base = JSON.parse(fs.readFileSync("baseline_lcp.json"));
        const cand = JSON.parse(fs.readFileSync("candidate_lcp.json"));
        const r = evaluate(base.samples, cand.samples);
        console.log(`lcp Δ=${r.effect}%  pZ=${r.pZ}  pMW=${r.pMW}`);
        if (r.significant) {
          fs.writeFileSync("alert.json", JSON.stringify({ metric: "lcp", ...r }));
          process.exit(1);
        }
      });
    '
- name: Notify on regression
  if: failure()
  run: node scripts/alert-notify.js --in alert.json --channel "#perf-alerts"

Choosing Alpha and the Effect Floor

The two knobs that decide how loud the alert is are alpha (the tolerated false-positive rate per evaluation) and MIN_EFFECT (the smallest relative shift worth acting on). They trade off against each other and against how noisy the lane is. A quiet desktop-broadband lane can afford a stricter alpha because its variance is low; a mid-range mobile lane at Fast 3G, where P75 LCP naturally jitters ±120 ms run to run, needs a higher effect floor so ordinary jitter never clears the bar. The table gives sane starting points per lane; whether you anchor the floor to a P75 or a P90 target is itself a decision covered in Choosing Between P75 and P90 Budget Targets.

If clean history still produces the odd false alert on a lane, lower alpha before you touch the effect floor — dropping alpha from 0.02 to 0.01 halves the false-positive rate without blinding the alert to genuine regressions. The baseline these tests compare against should itself be a rolling window rather than a single frozen run; see Rolling Median Baseline Windows for how to keep it current without letting a slow regression bake itself into the reference.

Verification

Confirm the alert is calibrated by replaying it over known-good and known-bad runs. A correctly tuned alert stays silent on clean history and fires on the seeded regression.

# Replay 30 known-good main runs: expect zero alerts
node scripts/alert-replay.js --metric lcp --window known-good/ --expect 0
# → 30 runs evaluated, 0 alerts  ✓ false-positive rate within budget

# Replay a run with a seeded +12% regression: expect one alert
node scripts/alert-replay.js --metric lcp --run seeded-regression.json --expect 1
# → REGRESSION lcp Δ=+12.0% pZ=0.001 pMW=0.004  ✓ alert fired

Zero alerts over clean history and a clean fire on the seeded case means the alpha and effect floor are right for this metric. If clean history produces even one or two alerts, lower alpha or raise MIN_EFFECT. The deeper significance-testing methodology behind these choices is covered in Statistical Significance Testing for Noisy CI.

Frequently Asked Questions