Continuous Performance Monitoring

A pull-request gate is the sharpest tool you own for stopping a bad change before it ships, and it belongs at the centre of the Lighthouse CI & WebPageTest Integration reference. But a gate only fires when a commit moves through it. The moment your code stops changing, the gate goes quiet — and your production performance keeps changing anyway. Third-party tags rewrite themselves overnight, a CDN reroutes traffic to a slower edge, a marketing team swaps a hero image, a product catalogue grows until a query that returned 40 rows now returns 4,000. None of that touches your repository, so none of it trips a pre-merge check. Continuous performance monitoring is the discipline of watching the deployed site on a fixed schedule, comparing each run to a rolling baseline, and alerting a human when the numbers drift — independent of whether anyone opened a pull request.

This guide treats continuous monitoring as the second half of a two-part enforcement model. The gate protects the merge; the monitor protects everything that happens after. We will cover what monitoring actually catches that gating cannot, a concrete scheduled-synthetic architecture built on cron, how to store trends so a baseline means something, how to alert on drift without drowning in false positives, how to reconcile synthetic runs against real field data, how to benchmark competitors so your budgets stay honest, and finally how to wire the whole stream into Grafana and Datadog so the trend lives where your team already looks.

What Continuous Monitoring Adds Over PR Gating

A PR gate answers exactly one question: did this diff make the page slower than the baseline on the runner? That is a code question. It is answered on an emulated device, in a clean environment, against the exact revision under review. It is deliberately blind to four large classes of regression that never appear in a diff.

The first is third-party drift. Your tag manager loads an analytics script by URL, and the vendor ships a new version behind that URL without telling you. Your bundle is byte-for-byte identical, but the page now blocks the main thread for another 180 ms at P75 on a mid-range mobile device throttled to Fast 3G. No commit, no gate, no warning. The second is data-driven regression: the template is unchanged but the data flowing through it grew. A category page that rendered 12 products now renders 96, and LCP at P75 on desktop over cable slides from 1.9 s to 3.1 s purely because the payload changed. The third is CDN and infrastructure change: a cache purge, an origin failover, a TLS renegotiation added by a new WAF rule. The fourth is gradual creep — the slow accumulation of a few kilobytes per sprint that no single PR would ever flag because each increment sits comfortably inside the noise band, yet the sum over a quarter blows the budget.

The diagram below maps each regression class against the tool that can actually see it. Notice that the gate column is nearly empty for everything except code.

The practical consequence is that a mature team runs both, and never asks one to do the other's job. Your Lighthouse CI Configuration & Storage setup already produces the assertion config, the throttling profile, and the storage target you need — continuous monitoring reuses all three and simply invokes them on a timer against the live URL instead of a pull-request preview.

A Scheduled Synthetic Architecture

The core of continuous monitoring is a job that runs on a fixed cadence, collects a synthetic measurement of your production URLs, and writes the result somewhere durable. Cadence is a budget decision, not a technical one. A nightly run at a low-traffic hour is enough to catch overnight third-party and data drift; an hourly run catches CDN failovers and marketing pushes within the same shift; a run every fifteen minutes is reserved for a handful of revenue-critical templates where a two-hour blind spot is unacceptable. Most teams settle on nightly for the full route set and hourly for a short critical-path list.

The scheduler itself can be a GitHub Actions schedule trigger, a plain crontab on a monitoring box, or a cloud scheduler firing a container. The important property is that the run is decoupled from deploys — it fires whether or not anyone shipped. A minimal GitHub Actions workflow that runs Lighthouse against three production URLs every night and uploads the result looks like this.

name: nightly-synthetic
on:
  schedule:
    - cron: "17 3 * * *"
  workflow_dispatch: {}
jobs:
  measure:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - run: npm install -g @lhci/[email protected]
      - name: Collect against production
        run: |
          lhci collect \
            --url=https://www.example.com/ \
            --url=https://www.example.com/category/shoes \
            --url=https://www.example.com/product/atlas-runner \
            --numberOfRuns=5 \
            --settings.preset=desktop
      - name: Upload to the LHCI server
        env:
          LHCI_TOKEN: ${{ secrets.LHCI_BUILD_TOKEN }}
        run: |
          lhci upload \
            --target=lhci \
            --serverBaseUrl=https://lhci.internal.example.com \
            --token="$LHCI_TOKEN"

Five runs per URL is the floor for a stable synthetic median; a single run is worthless because run-to-run variance on an emulated device routinely swings LCP by 300–500 ms at P75 even with nothing changed. The --settings.preset=desktop flag pins the emulation profile so that every nightly run is measured under the same device and connection context — here, an emulated desktop over a simulated cable link — and a second matrix leg should repeat the same URLs under the default mobile preset throttled to a Slow 4G-equivalent link. Never compare a mobile run to a desktop run; they are different populations. For the heavier, filmstrip-grade measurements you feed into competitor benchmarking, the same scheduled job can call out to a WebPageTest Private Instance Setup so you control the test location, browser build, and traffic-shaping profile.

The architecture below shows the full loop: scheduler, runner, collector, trend store, and the two consumers — the alerting evaluator and the dashboards.

Storing Trends and Building a Rolling Baseline

A single measurement tells you nothing; a baseline tells you everything. The trend store exists so that tonight's number has a yesterday and a last week to be compared against. You can use the Lighthouse CI server's own Postgres store, a purpose-built time-series database, or a flat table you write yourself. The schema only needs to answer one query cheaply: give me the last N medians for this URL, metric, and device profile. A minimal relational table is enough to start.

CREATE TABLE synthetic_runs (
  id            BIGSERIAL PRIMARY KEY,
  captured_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  url           TEXT        NOT NULL,
  device        TEXT        NOT NULL,   -- 'mobile' | 'desktop'
  metric        TEXT        NOT NULL,   -- 'lcp' | 'inp' | 'cls' | 'tbt'
  value_ms      NUMERIC     NOT NULL,   -- median of the run batch
  run_count     INT         NOT NULL,
  git_sha       TEXT
);
CREATE INDEX idx_runs_lookup
  ON synthetic_runs (url, device, metric, captured_at DESC);

The baseline itself is a rolling aggregate over that history, not a single frozen number. Freezing a baseline the day you launch guarantees it drifts out of relevance; a rolling window keeps it honest while still resisting a single bad night. The workhorse is a rolling median over a trailing window — typically the last 14 measurements for a nightly cadence, which is two weeks of history. The median is chosen over the mean deliberately: synthetic runs produce occasional outliers when a CI runner is noisy, and a 14-point median simply ignores a single 2× spike that would drag a mean upward. The query that computes the current baseline for one series is a windowed median.

SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY value_ms) AS baseline_ms
FROM (
  SELECT value_ms
  FROM synthetic_runs
  WHERE url = $1 AND device = $2 AND metric = $3
  ORDER BY captured_at DESC
  LIMIT 14
) recent;

This is the same rolling-median discipline that underpins broader Automated Regression Detection: the baseline moves slowly with genuine, sustained change and holds firm against one-off noise. When a green build lands on your main branch and its numbers are healthy, you let that measurement flow into the window like any other, so the baseline promotes itself continuously rather than through a manual gate.

Drift Alerting Against a Rolling Baseline

Drift alerting is the payoff. On every scheduled run, the evaluator pulls the rolling baseline, compares tonight's median, and decides whether the gap is large enough and persistent enough to page a human. Two mistakes dominate here. The first is alerting on a single run — synthetic noise alone will fire a same-night alert several times a week and train your team to ignore it. The second is a fixed absolute threshold that ignores the metric's natural variance. The robust rule combines a relative delta with a persistence requirement: alert only when the median exceeds the baseline by more than a metric-specific margin and stays there for at least two consecutive runs.

Concretely, for LCP measured on an emulated mid-range mobile device over a Slow 4G-equivalent link, a sensible drift trigger is a P75 median more than 12% above the 14-run rolling baseline sustained across two nights. For INP under the same device and connection context, use a 15% margin at P75 because interaction latency is noisier. For CLS, which is unitless and small, prefer an absolute floor — alert when the median crosses 0.02 above baseline at P75 — since a percentage on a near-zero number is meaningless. The chart below shows a real drift event: a flat baseline band, a nightly series that jumps on day 6, and the alert firing only once the breach persists into day 7.

Every alert should carry context that makes triage fast: the metric, the URL, the device and connection profile, the baseline value, tonight's value, the percent delta, and a link to the run artifact so an engineer can open the trace without hunting. An evaluator that emits a structured payload keeps the on-call rotation productive.

function evaluateDrift(series, baseline, cfg) {
  // series: newest-last medians; baseline: rolling median ms
  const latest = series[series.length - 1];
  const prev = series[series.length - 2];
  const margin = baseline * cfg.relativeMargin; // e.g. 0.12 for LCP
  const breachedNow = latest - baseline > margin;
  const breachedPrev = prev - baseline > margin;
  if (breachedNow && breachedPrev) {
    return {
      fire: true,
      metric: cfg.metric,
      url: cfg.url,
      device: cfg.device,
      baselineMs: Math.round(baseline),
      latestMs: Math.round(latest),
      deltaPct: Math.round(((latest - baseline) / baseline) * 100),
      artifact: cfg.artifactUrl,
    };
  }
  return { fire: false };
}

You can lift the margins straight into a config table so each metric and device profile carries its own threshold, and route the resulting payload to the child guide on Alerting on Synthetic Performance Drift for the notification-channel wiring.

Reconciling Synthetic and Field Data

Synthetic monitoring and real-user monitoring measure different things, and a mature program keeps both honest by reconciling them. A synthetic run is a controlled lab measurement on one emulated device from one location; field data is the P75 across your entire real audience with all their devices, networks, and cache states. They will never be identical, and that is fine — what matters is that they track together. When synthetic drifts up but field stays flat, suspect your lab profile has diverged from reality. When field drifts up but synthetic stays flat, suspect a population you are not testing: a device tier, a geography, or a logged-in state your synthetic run never exercises.

The table below shows a healthy reconciliation and two unhealthy divergences for a product page, all at P75.

A useful rule of thumb: keep the synthetic median within roughly 15–25% of the field P75 for the same page and device class, and treat a widening gap as its own alert. Field data typically runs a little slower than a clean lab run because real devices carry background load and colder caches, so a synthetic number that is faster than field by a stable margin is expected and healthy. The reconciliation itself belongs in your dashboard as two overlaid series, sourced from the same beacon and aggregation pipeline that feeds your RUM.

Competitor Benchmarking With WebPageTest

A budget defined only against your own history can slowly normalise mediocrity — you stay 10% faster than last quarter while a competitor laps you. Scheduled competitor benchmarking keeps the target external. Because you are testing sites you do not control, you run these through WebPageTest rather than Lighthouse CI: WebPageTest gives you a consistent test location, a real browser build, and a filmstrip you can eyeball, and its API is designed for exactly this kind of scripted third-party measurement. Run the same critical templates — home, a category page, a product page — against two or three named competitors on the same weekly cadence, same location, and same connection profile, so the comparison is apples to apples.

#!/usr/bin/env bash
set -euo pipefail
WPT_HOST="https://wpt.internal.example.com"
API_KEY="${WPT_API_KEY:?set WPT_API_KEY}"
LOCATION="ec2-us-east-1:Chrome.4G"
declare -a TARGETS=(
  "https://www.example.com/product/atlas-runner"
  "https://competitor-a.example/product/trail-x"
  "https://competitor-b.example/shop/road-2"
)
for url in "${TARGETS[@]}"; do
  test_id=$(curl -s "${WPT_HOST}/runtest.php" \
    --data-urlencode "url=${url}" \
    --data "k=${API_KEY}&location=${LOCATION}&runs=5&fvonly=1&f=json" \
    | jq -r '.data.testId')
  echo "queued ${url} -> ${test_id}"
  printf '%s\t%s\n' "${test_id}" "${url}" >> pending_tests.tsv
done

Once the runs complete, pull the median first-view LCP and total blocking time for each target and store them beside your own series so the dashboard can rank you. The one discipline that matters: hold the connection profile constant. A Chrome.4G location approximates a mid-range mobile device on a Slow 4G-equivalent link, and a competitor's P75 LCP of 2.4 s under that profile is only meaningful next to your P75 LCP measured under the identical profile. The dedicated child guide on Benchmarking Competitors With WebPageTest walks through parsing the result JSON and building the ranking view.

Wiring Results to Grafana and Datadog

Monitoring that nobody sees decays. The final step is pushing every scheduled median into the dashboard your team already watches, so the trend is ambient rather than something you go looking for after an incident. Whether that home is Grafana or Datadog, the pattern is the same: emit one metric point per run, tagged with the URL, device profile, and metric name, and let the dashboard draw the rolling baseline and the alert threshold as reference lines.

For a Prometheus-backed Grafana, expose the latest medians on a small /metrics endpoint that your collector updates after each run, tagged so a single panel can filter by URL and device.

import client from "prom-client";
const registry = new client.Registry();
const lcpGauge = new client.Gauge({
  name: "synthetic_lcp_ms",
  help: "Median synthetic LCP in milliseconds",
  labelNames: ["url", "device", "percentile"],
  registers: [registry],
});
export function recordRun(run) {
  lcpGauge
    .labels(run.url, run.device, "p75")
    .set(run.medianLcpMs);
}
export async function metricsHandler(_req, res) {
  res.set("Content-Type", registry.contentType);
  res.end(await registry.metrics());
}

For Datadog, post the same medians straight to the metrics intake as a gauge with matching tags, which lets a Datadog monitor own the drift alert if you would rather keep alerting in one platform.

curl -s -X POST "https://api.datadoghq.com/api/v2/series" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "series": [{
      "metric": "synthetic.lcp.ms",
      "type": 3,
      "points": [{ "timestamp": 1753324620, "value": 2810 }],
      "tags": ["url:product_atlas", "device:mobile", "percentile:p75"]
    }]
  }'

With the medians flowing, build one panel per metric that overlays the raw nightly series, the 14-run rolling baseline, and the drift threshold band from the earlier chart. That single view answers the two questions a performance owner asks every morning — are we drifting, and if so since when — without opening a single trace. From here, the child guide on Scheduling Nightly Lighthouse Runs covers hardening the cron job itself, retries, and keeping the emulation profile pinned across runner upgrades.

Frequently Asked Questions