Changepoint Detection for Performance Time Series

A nightly synthetic run produces one number per build. Over a month that is a noisy line that wanders by tens of milliseconds every night, and buried somewhere in that wander is the one deploy that quietly moved your P75 LCP from 2.4 s to 2.9 s on a mid-range Android phone throttled to Fast 3G. A fixed threshold never fired because the new level still sat under the 3500 ms mobile budget ceiling. This page, part of the Automated Regression Detection reference, shows how to find that exact build with changepoint detection instead of waiting for a hard limit to be breached.

Changepoint detection answers a different question than threshold alerting. Threshold alerting asks "is today's value bad?"; changepoint detection asks "did the whole series shift to a new level, and if so, on which build?" The second question is the one that maps a regression back to a single pull request. Below we contrast the two approaches, walk through a tabular CUSUM algorithm, implement it in JavaScript against a real LCP series, pinpoint the offending deploy, and verify the result before we page anyone.

Why Threshold Alerting Misses Step-Changes

A threshold is a horizontal line. It fires the moment a sample lands above it and stays silent otherwise. That works when a regression is large enough to punch through the ceiling in one night, but most real regressions do not. A third-party tag, an unshipped font-display, or a hydration change adds 300–500 ms to P75 LCP on a mid-range mobile device on Fast 3G — enough to hurt users and blow a budget you set a headroom margin under, but not enough to cross an absolute alert line pinned near the failing budget.

The failure mode is that the mean of the series moves while every individual point stays legal. Night after night the metric reads "green" against the threshold, yet the baseline has permanently degraded. The chart below plots twenty nightly builds where the P75 LCP steps from roughly 2400 ms to roughly 2900 ms at build 11, all while remaining well below a 3500 ms threshold line. No threshold alert ever fires, but a step-change is plainly there.

If you have already stood up statistical regression alerts that compare each build against a rolling baseline, changepoint detection is the natural next layer: rather than testing one build against recent history, it scans the whole window and reports the single index where the level changed.

How CUSUM Detects a Shift

The cumulative sum (CUSUM) algorithm is the simplest changepoint detector that is robust enough for CI. Instead of looking at each sample in isolation, it accumulates how far each sample deviates from an expected mean. Small random deviations cancel out over time and the running sum hovers near zero; a genuine shift makes the deviations share a sign, so the sum climbs steadily until it crosses a decision boundary.

We use a one-sided upper CUSUM because for LCP, INP, or CLS we only care about regressions — metrics getting worse. Given a target mean mu (the baseline level) and a slack value k (half the smallest shift we want to catch, expressed in the metric's units), the recursion is:

S_0 = 0
S_i = max(0, S_(i-1) + (x_i - mu - k))

Each night we add the current sample's excess over mu + k to the running statistic and clamp it at zero so quiet periods reset it. When S_i exceeds a decision interval h, we declare a changepoint. The build where the alarm fires is not the changepoint itself — the shift began earlier, at the last build where S was zero before it started climbing. That reset index is what we report as the offending build. The next diagram shows the statistic staying flat through the noisy stretch and then ramping past h a few builds after the true shift.

Two parameters govern sensitivity. The slack k sets the smallest shift you treat as real: pick k at half the shift you refuse to miss, so if a +300 ms move in P75 LCP on a mid-range mobile on Fast 3G matters, use k = 150. The decision interval h trades detection speed against false alarms: a larger h needs more consecutive excess to fire, so it is slower but quieter. A common starting point is h around four to five times the night-to-night standard deviation of the stable series.

Applying CUSUM to a Nightly LCP Series

Here is a self-contained implementation you can drop into a Node step after your nightly synthetic job. It takes an array of { build, commit, lcp } samples ordered oldest-first, estimates the baseline mean from a stable prefix, runs the one-sided CUSUM, and returns the changepoint build plus the commit range to blame.

function detectChangepoint(series, options = {}) {
  const { k = 150, hMultiplier = 4.5, baselineCount = 10 } = options;
  if (series.length <= baselineCount) {
    return { detected: false, reason: "not enough history" };
  }
  const baseline = series.slice(0, baselineCount).map((s) => s.lcp);
  const mu = baseline.reduce((a, b) => a + b, 0) / baseline.length;
  const variance =
    baseline.reduce((a, b) => a + (b - mu) ** 2, 0) / baseline.length;
  const sd = Math.sqrt(variance);
  const h = Math.max(hMultiplier * sd, 4 * k);

  let s = 0;
  let resetIndex = 0;
  for (let i = 0; i < series.length; i++) {
    const excess = series[i].lcp - mu - k;
    if (s + excess <= 0) {
      s = 0;
      resetIndex = i;
    } else {
      s += excess;
    }
    if (s > h) {
      const start = series[resetIndex];
      const alarm = series[i];
      return {
        detected: true,
        changepointBuild: start.build,
        alarmBuild: alarm.build,
        commitRange: `${start.commit}..${alarm.commit}`,
        baselineMean: Math.round(mu),
        statistic: Math.round(s),
        h: Math.round(h),
      };
    }
  }
  return { detected: false, baselineMean: Math.round(mu), h: Math.round(h) };
}

Feed it the twenty-build series from the first chart and it estimates mu around 2400 ms from builds 1–10, sets h from the baseline spread, and returns a changepoint at the build 10 reset with an alarm a few builds later. The commitRange field is the payload that matters: it hands your on-call engineer a git log start..alarm range instead of a shrug. The table below shows how the parameters translate into behaviour for three metrics measured on a mid-range mobile device on Fast 3G.

Run the detector on a trailing window rather than the entire history — thirty to sixty nightly builds is plenty. A window keeps the baseline mean current so that an intentional, promoted improvement does not permanently bias mu, and it aligns the detector with the same window your rolling median baseline windows already maintain for promotion.

Pinpointing the Offending Deploy

The detector gives you a build index; a build index is only useful once it maps to a deploy. Nightly runs are numbered by CI, not by release, so you need a lookup from the changepoint build back to the set of commits that shipped between the last clean night and the first shifted one. The flow below is the whole chain from a flagged nightly run to a single suspect change.

In practice the lookup is a small shell step. Store each nightly run's head commit alongside its metrics, then when the detector fires, resolve the range and filter it to changes that touch the render path:

START=$(jq -r '.changepointBuild' cusum-result.json)
ALARM=$(jq -r '.alarmBuild' cusum-result.json)
START_SHA=$(jq -r --arg b "$START" '.[] | select(.build==($b|tonumber)) | .commit' runs.json)
ALARM_SHA=$(jq -r --arg b "$ALARM" '.[] | select(.build==($b|tonumber)) | .commit' runs.json)
git log --oneline "$START_SHA..$ALARM_SHA" -- src/ public/ next.config.js

That command usually shrinks a night's worth of merges to a handful touching the bundle or the critical render path. Attach the range to the alert so the person triaging starts with suspects, not with an empty git blame. Wiring this into your nightly job pairs cleanly with scheduling nightly Lighthouse runs so the changepoint scan runs as a post-processing step on every fresh series.

Verifying the Changepoint

A CUSUM alarm is a hypothesis, not a verdict. Before you page anyone, confirm the shift is real and persistent rather than a one-night spike or a bimodal artifact of a flaky test environment. Three cheap checks separate signal from noise.

First, require persistence: the shifted level must hold for at least three consecutive builds after the changepoint, not a single outlier that the next night reverts. A lone spike trips the statistic once and then the reset clamps it back — but if you loosened h too far you can still fire on noise, so gate on persistence explicitly. Second, confirm the magnitude clears your slack with margin: a shift of k + a few ms is within measurement jitter, whereas a shift of 2k or more on a mid-range mobile on Fast 3G is a genuine regression. Third, corroborate with a significance test on the two segments — running a Welch's t-test for performance regressions on the pre- and post-changepoint samples tells you whether the two means differ beyond what the variance alone would produce.

Combining changepoint detection with a significance gate is what keeps the alert channel trustworthy. The changepoint step tells you where and the significance step tells you whether, and running both means your team stops muting alerts. For deeper treatment of the noise side of that equation, see statistical significance testing for noisy CI, which formalizes when a two-sample difference is worth acting on.

Frequently Asked Questions