Alerting on Synthetic Performance Drift

Scheduled synthetic runs catch slow decay that a single pull-request gate never sees: a third-party tag that grew 40 KB over a quarter, a hero image that quietly lost its fetchpriority, a framework upgrade that added 120 ms to hydration. This how-to is part of the Continuous Performance Monitoring guide within the broader Lighthouse CI & WebPageTest Integration reference, and it assumes you already have nightly Lighthouse runs producing timestamped results. The goal here is narrow and practical: turn that stream of nightly numbers into an alert that fires on genuine drift and stays silent on noise.

The trap with synthetic alerting is a fixed threshold. If you page on-call every time P75 LCP crosses 2,800 ms on a mid-range mobile profile (Moto G-class device, 4x CPU throttle, throttled 4G at 1.6 Mbps down / 750 Kbps up / 150 ms RTT), you will either set the line so high it never catches slow decay or so low it screams on every noisy night. A rolling baseline plus a percentage-delta rule fixes both problems: the line moves with your real performance, and the delta absorbs the flakiness inherent in emulated runs.

Define a Rolling Baseline

A rolling baseline is the reference number that today's run is measured against. Use the median of the last N completed nightly runs, not the mean — the median ignores a single catastrophic outlier (a run that landed on a cold CI runner, a CDN hiccup) that would otherwise drag a mean upward and poison your comparison. A window of N = 14 nights is a good default for a daily schedule: two weeks is long enough to smooth weekday/weekend infrastructure differences yet short enough to track a genuine, intended improvement within a sprint. If you deploy less often, widen to N = 28; if you run several times a day, keep the window in number of runs, not days.

Compute one baseline per metric per page template. P75 LCP, P75 INP, P75 CLS, and total JavaScript transfer bytes each get their own rolling median, and each URL or page-type segment gets its own series. Mixing a product-detail page and a search-results page into one baseline hides drift on both. The same rolling-median technique underpins rolling median baseline windows used for promotion decisions; here we reuse it purely as the alert reference.

Write the Drift Rule

A drift rule has two parts: a magnitude (how far above the baseline counts as drift) and a persistence (how many consecutive nights it must hold). Breach only when today's value is more than x% above the rolling median for k consecutive nights. A single night above the line is noise more often than not; requiring persistence is what separates real drift from a bad emulation run.

Sensible starting values for P75 LCP on the mid-range mobile profile (Moto G-class, 4x CPU throttle, throttled 4G) are x = 10% and k = 3 consecutive nights. In other words, if the rolling median P75 LCP is 2,500 ms, the alert arms only when three nights in a row land above 2,750 ms. For P75 INP on that same mobile profile, tighten the magnitude to x = 8% because interaction latency is less noisy run-to-run than LCP; for P75 CLS, which is bounded and jumpy, prefer an absolute floor (breach above 0.02 over baseline) rather than a percentage, since a 10% swing on a 0.01 baseline is meaningless. For JavaScript transfer bytes measured on the same throttled-4G profile, x = 5% for k = 2 nights works well because bundle size barely moves without a real code change, so even a small persistent rise is signal.

The reason persistence matters is visible in a night-by-night chart. A lone spike pokes above the threshold and falls back; drift is a run of nights that stay above it. The rule fires on the run, not the spike.

Deduplicate the Noise

Even with a persistence rule, a firing alert can re-notify every night the drift continues, burying on-call in duplicates. Deduplicate on a stable fingerprint — hash the tuple of page-template + metric + rule-id — and treat repeated fires with the same fingerprint as updates to one open incident, not new incidents. Send one notification when the alert transitions from OK to FIRING, update the existing thread while it stays FIRING, and send exactly one RESOLVED notification when the metric falls back under the threshold for the same k-night persistence in the healthy direction.

Two more dampeners keep the signal clean. First, gate the whole evaluation on a minimum sample count: if fewer than 5 of the last 14 nights completed successfully (runner failures, timeouts), skip evaluation rather than compare against a thin, unreliable window. Second, borrow from statistical noise and flakiness reduction and require that the delta exceed run-to-run variance — if the interquartile range of the window is already ±6% for P75 LCP on the throttled-4G mobile profile, a 10% threshold is comfortably outside the noise band, but a 4% threshold would not be, and you should widen x rather than chase phantom drift.

Route to Slack and PagerDuty

Not every drift deserves a page at 3 a.m. Route by severity computed from the same delta. A moderate breach (P75 LCP 10–20% over baseline on the mid-range mobile 4G profile) is a warning posted to a Slack channel for the owning team to triage next business day. A severe breach (more than 20% over baseline, or any breach that also crosses your hard performance budget ceiling) is a critical that pages on-call through PagerDuty. The dedupe fingerprint travels with both so a warning that escalates to critical updates the same incident rather than opening a second one.

Configure the Rule

Keep the rule in version control next to the nightly workflow so a change to x, k, or a channel is reviewed like any other code. The YAML below declares one series per metric with its own magnitude and persistence, plus routing.

# drift-alerts.yml
window:
  runs: 14            # rolling window length (nights)
  min_samples: 5      # skip evaluation below this many good runs
profile:
  device: moto-g-class
  cpu_throttle: 4x
  network: throttled-4g   # 1.6 Mbps down / 750 Kbps up / 150 ms RTT
series:
  - metric: lcp_p75
    baseline: rolling_median
    magnitude_pct: 10
    persistence_nights: 3
    warn_pct: 10
    page_pct: 20
  - metric: inp_p75
    baseline: rolling_median
    magnitude_pct: 8
    persistence_nights: 3
    warn_pct: 8
    page_pct: 18
  - metric: cls_p75
    baseline: rolling_median
    magnitude_abs: 0.02   # absolute, not percentage
    persistence_nights: 3
  - metric: js_transfer_bytes
    baseline: rolling_median
    magnitude_pct: 5
    persistence_nights: 2
    warn_pct: 5
    page_pct: 12
routing:
  warn:  { type: slack, channel: "#web-perf" }
  page:  { type: pagerduty, service: "frontend-oncall" }
  dedupe_key: "{template}:{metric}:drift"

The evaluator that reads a metric's recent history and applies the rule is small enough to run inside the nightly job. It computes the rolling median, checks the last k nights against the threshold, and returns a routing decision that a thin webhook layer turns into a Slack or PagerDuty call.

// evaluateDrift.js — returns { state, severity, delta } for one metric series
function median(values) {
  const s = [...values].sort((a, b) => a - b);
  const mid = Math.floor(s.length / 2);
  return s.length % 2 ? s[mid] : (s[mid - 1] + s[mid]) / 2;
}

function evaluateDrift(history, cfg) {
  // history: [{ night: '2026-07-23', value: 2620 }, ...] newest last
  const good = history.filter((h) => Number.isFinite(h.value));
  if (good.length < cfg.min_samples) {
    return { state: 'SKIPPED', reason: 'insufficient-samples' };
  }
  const window = good.slice(-cfg.window_runs);
  const baseline = median(window.map((h) => h.value));
  const threshold = cfg.magnitude_abs != null
    ? baseline + cfg.magnitude_abs
    : baseline * (1 + cfg.magnitude_pct / 100);

  const recent = good.slice(-cfg.persistence_nights);
  const allBreached = recent.length === cfg.persistence_nights
    && recent.every((h) => h.value > threshold);

  const today = good[good.length - 1].value;
  const deltaPct = ((today - baseline) / baseline) * 100;

  if (!allBreached) {
    return { state: 'OK', baseline, delta: deltaPct };
  }
  const severity = deltaPct > (cfg.page_pct ?? Infinity) ? 'page' : 'warn';
  return { state: 'FIRING', severity, baseline, threshold, delta: deltaPct };
}

module.exports = { evaluateDrift, median };

Verify Before You Trust It

An alert you have never seen fire is an alert you cannot trust. Verify the rule end to end before it guards anything real. The lifecycle you are confirming is OK to WARN to FIRING to RESOLVED, with exactly one notification at each real transition.

Run the verification as three deliberate injections against a staging copy of the evaluator, then diff the notifications produced against what you expect. This is the same discipline described in automated regression detection, applied to the alert plumbing itself rather than to the code under test.

Once all five rows pass, wire the evaluator into the nightly job, point routing at the real Slack channel and PagerDuty service, and let it run in shadow for one week — logging what it would have sent without paging anyone. If the shadow week is quiet on healthy nights and would have caught your last known regression, promote it to live.

Frequently Asked Questions