Rolling Median Baseline Windows

A single pinned baseline is a brittle reference. Pin the gate to one blessed run and every measurement afterward is judged against a snapshot that may have been slightly fast, slightly slow, or captured on a noisy runner. This how-to, part of the Historical Baseline Calibration reference, replaces that single point with a rolling window: the baseline for any build is the median of the last N builds, and the tolerance band is derived from that same window's spread. The result is a reference that breathes with normal variance yet still fails hard when performance drifts in one direction and stays there.

The core idea is that both halves of the gate — the center line and the band around it — should be computed from recent history, not hand-set once. We use the median for the center because it ignores outliers, and the median absolute deviation (MAD) for the band because it measures spread without letting one bad run inflate it. Everything below is how to build, size, and enforce that window.

Why a Rolling Window Beats a Pinned Point

Synthetic performance runs are noisy. The same commit measured ten times on the same CI runner will produce a spread of Largest Contentful Paint values — commonly a range of 150–300ms of jitter for an LCP P75 around 2400ms on an emulated mid-range mobile (Moto G4-class CPU at 4x throttle) over Fast 3G. A pinned baseline captures one draw from that noise and freezes it. If the pin happened to land on a lucky fast run, every honest run afterward looks like a regression; if it landed on a slow run, real regressions hide under the inflated ceiling.

A rolling window solves both problems at once. Because the center is the median of the last N runs, a single anomalous build barely moves it, and the band automatically reflects how noisy the environment actually is this week rather than how noisy it was on one historical day. When performance genuinely drifts — a heavier hero image, a new blocking script — the sustained shift eventually pulls the median with it, but only after several consecutive builds confirm the change is real rather than a blip.

Median and MAD Over the Window

Two robust statistics define the window. The center is the median of the last N build values for a metric. The spread is the median absolute deviation: take the absolute distance of each sample from the median, then take the median of those distances. To put MAD on the same scale as a standard deviation for roughly normal data, multiply it by the constant 1.4826 — this scaled MAD is a drop-in, outlier-resistant estimate of sigma.

The tolerance band is then median ± k · (1.4826 · MAD). A factor of k = 3 gives a band that a well-behaved run clears comfortably while a genuine regression breaches. Concretely, for an INP P75 window centered at 190ms with a scaled MAD of 18ms, measured on a mid-range mobile (4x CPU throttle) over Fast 3G, the upper bound sits at 190 + 3·18 = 244ms. Any build whose own INP P75 exceeds 244ms is flagged. Because the band is computed from the window itself, a quiet week tightens it and a noisy week widens it — the gate calibrates its own sensitivity. This robustness is what lets the window coexist with the variance-reduction techniques covered under Statistical Noise and Flakiness Reduction rather than fighting them.

Choosing the Window Size

Window size N is the one dial that trades responsiveness against stability. A short window (N = 5) adapts fast — a real improvement is reflected within a few builds — but a couple of noisy runs can jerk the median and band around. A long window (N = 30) is rock-steady and rejects transient noise, but it is slow to acknowledge a legitimate change and can carry stale samples from before a deliberate optimization. Most teams land between these poles and tune by watching false-positive and false-negative rates on replayed history.

The table below shows how the same INP P75 stream, measured on a mid-range mobile (4x CPU throttle) over Fast 3G, behaves under three window sizes. Pick the smallest N whose false-alarm rate your team can tolerate, because smaller windows catch real drift sooner.

A practical default is N = 10 to 15 for a repository that merges several times a day, dropping to N = 20 for a low-traffic project where builds are infrequent and each one carries more weight. Always trim the window to clean samples only — failed runs, timeouts, and known-bad SHAs must be excluded before the median is taken, or the baseline inherits their noise.

Comparing a New Build to the Window

A build under test contributes its own metric value — itself the median of the repeated runs in that CI job, taken at P75 across the emulated sessions — and that value is compared to the window band. The gating logic has three outcomes. If the build value sits inside the band, it passes and (on main) becomes eligible to enter the window. If it exceeds the upper bound, the gate fails: performance has regressed beyond what recent noise explains. If it falls below the lower bound, it also passes, but it is flagged as a candidate improvement worth confirming before it pulls the median down.

Crucially, the build under test is compared to the window before being added to it, so a regression cannot mask itself by contaminating its own reference. Only after a clean pass on main does the value join the window, which ties this technique directly to the promotion mechanics in Automating Baseline Promotion Workflows. For catching drift that stays inside the band but trends steadily, pair the window with the sequential methods in Automated Regression Detection.

Implementation

The module below computes the window statistics and gates a candidate value. It sorts a clean sample array, derives the median and scaled MAD, builds the band at a configurable k, and returns a verdict plus the numbers behind it. It is dependency-free and copy-paste runnable under Node 20.

// rolling-window.js — robust rolling-median baseline gate
export function median(xs) {
  const s = [...xs].sort((a, b) => a - b);
  const m = Math.floor(s.length / 2);
  return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2;
}

export function scaledMad(xs, med = median(xs)) {
  const deviations = xs.map((x) => Math.abs(x - med));
  return 1.4826 * median(deviations);
}

export function buildBand(samples, { k = 3 } = {}) {
  if (samples.length < 5) throw new Error(`window too thin: ${samples.length}`);
  const center = median(samples);
  const spread = scaledMad(samples, center);
  return {
    center: Math.round(center),
    lower: Math.round(center - k * spread),
    upper: Math.round(center + k * spread),
    mad: Math.round(spread),
  };
}

export function gate(candidate, samples, opts = {}) {
  const band = buildBand(samples, opts);
  let status = "pass";
  if (candidate > band.upper) status = "fail";
  else if (candidate < band.lower) status = "improvement";
  return { status, candidate, ...band };
}

Drive it from your CI step with the last N clean samples pulled from the baseline store. The window is trimmed to the most recent N entries so it rolls forward as history grows.

// gate-step.js — invoked in CI after collecting the run
import { readFileSync } from "node:fs";
import { gate } from "./rolling-window.js";

const N = 12; // window size
const history = JSON.parse(readFileSync("history.json", "utf8")); // { inp: [..clean values..] }
const candidate = Number(process.argv[2]); // this build's INP P75, ms

const window = history.inp.slice(-N);
const result = gate(candidate, window, { k: 3 });

console.log(JSON.stringify(result, null, 2));
if (result.status === "fail") {
  console.error(`REGRESSION: INP ${result.candidate}ms > upper ${result.upper}ms`);
  process.exit(1);
}

A non-zero exit fails the CI job. On a green main push you append candidate to history.inp so the next build compares against a window that now includes this one.

Verification

Confirm the gate behaves as designed before trusting it. Feed it three synthetic candidates against a known window and check each verdict. With a window whose INP P75 values (mid-range mobile, 4x CPU throttle, Fast 3G) are [188, 192, 190, 195, 187, 191, 189, 193, 190, 188, 194, 190], the median is 190ms and the upper bound at k = 3 lands near 244ms.

node gate-step.js 205   # inside band → status "pass"
node gate-step.js 260   # above upper → status "fail", exit 1
node gate-step.js 150   # below lower → status "improvement"

The first call passes because 205ms sits comfortably inside a band sized by the window's own small spread. The second exits non-zero because 260ms exceeds the 244ms ceiling that recent noise cannot explain — a real regression. The third is flagged as an improvement candidate rather than silently accepted, so a suspiciously fast run (often a broken page that never rendered) is reviewed before it drags the median down. Replay a few weeks of real history through the gate and count false alarms against the table above; if the rate is too high, raise k or lengthen N, and if regressions slip through, do the reverse. Once the window is stable, layer statistical alerting on top of it via Automated Regression Detection to catch slow trends the band alone would miss.

Frequently Asked Questions