Historical Baseline Calibration

A budget threshold copied from a blog post is a guess; a threshold derived from your own last ninety days of runs is a measurement. Historical baseline calibration replaces arbitrary static limits with thresholds computed from the longitudinal distribution of your own metrics, so the gate reflects how the site actually behaves rather than how someone hoped it would. This is the baseline layer of the Threshold Calibration & Baseline Management reference: it turns a stream of past Lighthouse and RUM samples into a rolling baseline, wraps that baseline in a tolerance band, and fails the build only when a new run breaks out of the band.

The work splits into three coupled concerns — ingesting clean historical samples (windowing and outlier removal), deriving a baseline and tolerance from that window (percentiles plus deltas), and enforcing a new run against the baseline in CI. Get the first wrong and the baseline inherits the noise; get the second wrong and the gate either nags constantly or never fires. This page is the authoritative spec for all three, and it carries the worked scripts, schemas, and defensible starting numbers you need to stand a calibrated gate up in an afternoon.

Core Concept: The Rolling Baseline

A rolling baseline is not a single number frozen at a point in time. It is a value recomputed continuously over a sliding history window, surrounded by a tolerance band that absorbs normal run-to-run variance. A new run is compared against that band, not against a hand-picked constant. The distinction matters because a static ceiling can only answer "is this run under an absolute limit?" while a rolling baseline answers the far more useful question: "did this change make the site measurably slower than it has been?" A page can be comfortably under a 2500 ms LCP ceiling on mid-range mobile at P75 over Fast 3G and still have quietly regressed 400 ms in a week — the rolling baseline catches that; the static ceiling never does.

The diagram below shows how a history window collapses into a baseline, how the tolerance band widens it, and how each incoming run lands inside or outside.

The band is deliberately two-sided even though most teams only fear the upper edge. A run that suddenly lands far below the band is not automatically good news — it usually means the measurement changed, not the site: a route started returning a cached error page, an image failed to load so LCP fired on a placeholder, or a throttling profile silently reset. Treating a large downward break as a signal worth investigating, not silently celebrating, is what keeps the baseline honest.

Prerequisites & Environment

Calibration needs a durable store of past runs and a deterministic collection pipeline feeding it. Without determinism the band has to be so wide it never catches anything, so pin your collection settings first against the Lighthouse CI Configuration & Storage reference before you trust any baseline derived from the data.

• A time-series or artifact store — TimescaleDB, InfluxDB, or even a versioned JSON artifact in object storage. It must retain at least one full window of runs keyed by Git SHA and branch, partitioned by day for fast window queries.
• Deterministic collection — throttlingMethod: simulate and a fixed numberOfRuns >= 3 so each stored sample is a stable median, not a single noisy draw. On mid-range mobile emulation over Fast 3G, a single un-medianed run of LCP can swing 500 ms between draws; the median of three collapses most of that.
• Per-environment separation — never blend mobile and desktop, or staging and production, into one baseline. Each (device class, connection profile, route) tuple gets its own baseline series. When routes share a template but not an audience, split further with Segmenting Baselines by Page Type.

Map the moving parts through environment variables so nothing is hardcoded: BASELINE_STORE_URL for the store endpoint, BASELINE_BRANCH for the series key (usually main), and BASELINE_WINDOW for the rolling window size in runs or days. The gate reads these at run time so the same workflow file can service every environment without edits.

Configuration Reference

The baseline store schema below is the authoritative spec. Each metric series carries the derived baseline, the tolerance that defines its band, the window it was computed over, and the provenance needed to audit it. Storing provenance — the Git SHA and sample count — is what lets you later trust or reject a baseline.

{
  "schemaVersion": 2,
  "key": { "branch": "main", "device": "mobile", "route": "/checkout" },
  "window": { "type": "rolling", "size": 90, "unit": "runs", "minSamples": 30 },
  "outlierFilter": { "method": "iqr", "k": 1.5 },
  "metrics": {
    "lcp":  { "baseline": "p75", "tolerance": { "type": "abs", "value": 200 }, "unit": "ms" },
    "inp":  { "baseline": "p75", "tolerance": { "type": "abs", "value": 50 },  "unit": "ms" },
    "cls":  { "baseline": "p75", "tolerance": { "type": "rel", "value": 0.10 } },
    "tbt":  { "baseline": "p75", "tolerance": { "type": "abs", "value": 75 },  "unit": "ms" },
    "script_bytes": { "baseline": "p90", "tolerance": { "type": "rel", "value": 0.05 } }
  },
  "provenance": { "computedAt": "2026-06-20T02:00:00Z", "sampleCount": 87, "gitSha": "4f1c9ad" }
}

window.minSamples is the safety valve: if the trimmed window holds fewer than 30 clean samples the baseline is marked stale and the gate falls back to warn rather than blocking on thin data. The interquartile-range filter (k: 1.5) removes any sample outside Q1 - 1.5*IQR to Q3 + 1.5*IQR so a transient network spike cannot drag the baseline. tolerance is either absolute (abs, in metric units) or relative (rel, a fraction of the baseline) — use absolute for metrics with a meaningful floor like LCP and relative for metrics that scale, like script bytes.

Note the deliberate split of baseline percentiles inside one file: LCP, INP, CLS, and TBT anchor to P75 because that is the percentile Core Web Vitals is judged at in the field, while script_bytes anchors to P90. Bytes have far less run-to-run variance than timings, so a higher percentile still yields a stable number, and pinning to P90 means a payload spike has to affect nearly every build before the baseline moves. When your sample count is thin enough that even P75 is jumpy, interpolate it rather than snapping to the nearest observed run, following Interpolating Percentiles From Sparse Samples.

Choosing the Window Shape

"Rolling window" is not one thing. There are three common shapes, and the choice changes how fast the baseline reacts and how much a single noisy day can move it. A count window keeps the last N runs regardless of calendar time — simple, but a burst of retries can pack the window with an hour of one bad afternoon. A time window keeps every run in the last D days — stable against bursts, but its sample count swells and shrinks with commit volume. A rolling median of daily medians first collapses each day to one median, then takes the median (or P75) across days — the most robust against a single bad build, at the cost of one day of lag. The panels below contrast the three.

Most teams should start with a count window for its simplicity and move to a rolling median of daily medians only once run volume is high enough that a bad hour can dominate the window. The full mechanics of the daily-median approach — including how to backfill days with no runs and how to weight partial days — live in Rolling Median Baseline Windows.

Step-by-Step Implementation

1. Ingest and trim the window. Pull the last N runs for each series and remove outliers with an interquartile-range filter so a single bad runner cannot shift the baseline. This preprocessing is the same discipline covered in Statistical Noise & Flakiness Reduction.

   node scripts/baseline-ingest.js --branch main --window 90 --filter iqr

   Expected output: window=90 raw=90 trimmed=87 (3 outliers removed) confirming the filter ran and enough clean samples remain.

2. Derive the baseline and band. Compute the configured percentile per metric and attach the tolerance to form the band. Derive the percentile with the methodology in Percentile-Based Threshold Tuning so the baseline reflects the experience you actually want to hold.

   node scripts/baseline-derive.js --in trimmed.json --out baseline_store.json

   Expected tail: lcp p75=2180ms band=[1980,2380] inp p75=164ms band=[114,214] — one line per gated metric.

3. Compare a candidate run. Run the PR build, then diff its median against the band. Commit baseline_store.json only from the promotion job, never from a feature branch, so the baseline never absorbs an unmerged regression.

The comparison logic is small enough to read in full. The function below is the exact core of baseline-compare.js: it computes each metric's band from the stored baseline and tolerance, checks the candidate median against it, and returns a non-zero exit intent when any gated metric breaks out. Reading it removes any doubt about what "outside the band" means.

// baseline-compare.js — core comparison, returns { failed, rows }
function bandFor(metric) {
  const t = metric.tolerance;
  const half = t.type === "abs" ? t.value : metric.value * t.value;
  return { lower: metric.value - half, upper: metric.value + half };
}

function compare(baseline, candidate) {
  const rows = [];
  let failed = false;
  for (const [name, metric] of Object.entries(baseline.metrics)) {
    const observed = candidate[name];
    if (observed == null) continue;
    const { lower, upper } = bandFor({ ...metric, value: metric.baselineValue });
    const outside = observed > upper || observed < lower;
    const gating = metric.mode !== "warn";
    if (outside && gating) failed = true;
    rows.push({ name, observed, lower, upper, outside, gating });
  }
  return { failed, rows };
}

module.exports = { compare, bandFor };

Notice that a warn metric still records outside in its row — it just does not flip failed. That is deliberate: the pull-request comment can surface a warning breach so a human sees drift building, while the required status check stays green until the metric has earned promotion to error. Notice too that the comparison flags both an upper and a lower breach, wiring the two-sided band from the core concept straight into the gate.

Threshold Calibration

Two knobs govern whether the gate is useful: the window size and the tolerance. Too short a window tracks every wobble and the baseline chases noise; too long and it lags real improvements for weeks. Too tight a tolerance fires on normal variance; too loose and a real 15% regression slips through. The matrix below shows defensible starting points by environment — calibrate, then hold for two weeks before tightening.

Size the tolerance from the measured run-to-run standard deviation of each metric, not by feel: a band of roughly two standard deviations around the baseline catches real shifts while tolerating normal jitter. Keep new metrics at warn until the band has held for two consecutive weeks, then promote them to error so the gate earns trust before it blocks merges.

The percentile column widens on mid-range mobile deliberately. On a mid-range mobile device over Fast 3G, LCP at P90 can sit 700 ms above the same page's P75 because tail latency on constrained CPUs is fat, so anchoring the baseline at P90 there stops the gate firing every time a slow tail sample lands. On desktop over Cable, where the P75-to-P90 gap is often under 150 ms, P75 is tight enough. Match the baseline percentile to the shape of the distribution, not to a house style.

To turn the two standard-deviation rule into concrete numbers, compute the deviation from the same trimmed window you derive the baseline from. The snippet below reads a metric's samples and prints the baseline, the standard deviation, and the two-sigma band in one pass, so the tolerance you commit to the schema is measured rather than guessed.

function twoSigmaBand(samples, percentile = 0.75) {
  const sorted = [...samples].sort((a, b) => a - b);
  const idx = Math.min(sorted.length - 1, Math.floor(percentile * sorted.length));
  const baseline = sorted[idx];
  const mean = samples.reduce((s, v) => s + v, 0) / samples.length;
  const variance = samples.reduce((s, v) => s + (v - mean) ** 2, 0) / samples.length;
  const sigma = Math.sqrt(variance);
  return { baseline, sigma, lower: baseline - 2 * sigma, upper: baseline + 2 * sigma };
}

Deciding Tolerance Type and Escalation

Every metric faces two orthogonal decisions: whether its tolerance is absolute or relative, and whether it gates as warn or error. The first depends on whether the metric has a meaningful floor; the second depends on how much the band has proven itself. The decision tree below routes each metric to the right combination.

CI Enforcement Snippet

This GitHub Actions job fetches the current baseline, runs the candidate, and fails the build when any gated metric breaks out of its band. It is copy-paste ready and surfaces a required status check that branch protection can gate on.

name: Baseline Gate
on:
  pull_request:
    branches: [main]

jobs:
  baseline-compare:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
      - run: npm ci
      - name: Fetch current baseline
        run: node scripts/baseline-fetch.js --branch main --out baseline_store.json
        env:
          BASELINE_STORE_URL: ${{ secrets.BASELINE_STORE_URL }}
      - name: Collect candidate run
        run: npx lhci collect && npx lhci upload --target=filesystem
      - name: Compare against band
        run: node scripts/baseline-compare.js --baseline baseline_store.json --run .lighthouseci/

The baseline-compare.js step exits non-zero when a metric's median falls outside [baseline - tolerance, baseline + tolerance], so the job becomes a required check. To close the loop and keep the baseline current, promote a fresh baseline after every clean main-branch run with Automating Baseline Promotion Workflows, and gate that promotion itself on a green build following Promoting Baselines After Green Main Builds. For statistical change detection that adapts to the distribution rather than a fixed band, pair this with Automated Regression Detection.

Guarding Against Drift and Poisoning

The most dangerous failure of a rolling baseline is silent: because the baseline learns from recent runs, a regression that merges can teach the band to accept it. Two patterns cause this. Baseline poisoning is a single large regression that merges and is folded into the next promotion, jumping the band up so the regression reads as normal from then on. Slow drift is subtler — a string of tiny regressions each stay inside the band, but each nudges the next baseline up, so over weeks the gate ratchets hundreds of milliseconds worse without ever firing. The chart below traces both against an absolute ceiling that caps how far the rolling baseline is ever allowed to travel.

The defense is a hybrid gate: keep the rolling band for sensitivity, but layer an absolute ceiling derived from your field P75 that the baseline may never exceed. If a promotion would push the baseline past the ceiling, the promotion is rejected and a human is paged. That single guard rail converts both drift and poisoning from silent into loud, because the ceiling does not learn — it is set from the experience you have committed to hold for real users, not from what recent builds happened to produce.

Troubleshooting & Edge Cases

• Baseline poisoning → a regression that merged to main gets folded into the next baseline, ratcheting the band up so the regression looks normal. Gate promotion on a clean run (see the promotion workflow) and store the Git SHA so a poisoned baseline can be rolled back.
• Slow drift → tiny per-run regressions each stay inside the band but accumulate. Add an absolute ceiling derived from your field P75 alongside the rolling band, so the baseline can never drift past a hard limit.
• Cold start / thin window → fewer than minSamples clean runs after trimming. Fall back to warn and a static ceiling until the window fills, rather than blocking on noise.
• Bimodal metrics → a route with two code paths produces two clusters; a single percentile sits in the empty middle. Split the series by path or segment before deriving the baseline.
• Window discontinuity after a deploy → an intentional architecture change shifts the true baseline. Reset the window (clear history before the change) so the band re-forms around the new normal instead of straddling both.
• Cross-environment bleed → mobile samples leaking into a desktop series widen the band uselessly. Verify the series key (device, route, connection) on ingest.
• Timezone-skewed daily medians → collapsing runs to a "day" without a fixed timezone splits a single busy evening across two calendar days on the UTC boundary, halving each day's sample count. Pin the day boundary to one timezone in the ingest step so daily medians are computed over consistent buckets.
• Percentile snapping on sparse days → on a day with only four runs, P75 snaps to the third-slowest sample and jumps erratically. Interpolate the percentile rather than snapping, and raise minSamples per day so thin days fall back to warn.

Frequently Asked Questions