Alerting on Performance Budget Regressions

A dashboard catches regressions only when someone is looking at it; an alert catches them at 2 a.m. when a deploy quietly pushes P75 LCP past budget. This guide configures Grafana unified alerting on the budget threshold lines from Visualizing Budget Trends with Grafana so a sustained breach pages the right team — without flooding the channel every time a single noisy bucket clips the line.

The hard part of performance alerting is not detecting a breach, it is not detecting a non-breach. Field percentiles wobble bucket to bucket — a P75 LCP measured on mid-range mobile over Fast 3G can swing 300 ms between two adjacent five-minute windows purely from sample composition — and a naive "alert when LCP > 2500 ms" rule fires and resolves dozens of times an hour. The fix is a for duration that requires the breach to persist, an evaluation window wide enough to smooth single-bucket noise, and notification routing that groups and deduplicates before anyone is paged.

Alert Rule Plan

Every condition in the plan is a P75 field value measured on the mid-range-mobile-over-Fast-3G segment, matching the budget lines the dashboard draws. CLS gets a longer for because layout-shift spikes are often a single bad deploy or a mis-sized ad slot that self-corrects; LCP gets a second, tighter rule at the "poor" boundary (4000 ms) that escalates to critical and pages immediately. If you have not yet decided which percentile to gate on, read Choosing Between P75 and P90 Budget Targets first — the alert percentile must equal the budget percentile or the two signals disagree. Pair both rules with the statistical methods in Automated Regression Detection so the rule fires on a real shift rather than on variance.

How the For Duration Suppresses Noise

The for duration is the single most important fatigue control, so it is worth understanding exactly what it does to a wobbling field series. When the reduced P75 first crosses 2500 ms the rule enters Pending, not Firing. It only transitions to Firing if every subsequent evaluation across the full for window also breaches. One good bucket resets the clock. The chart below plots a realistic five-minute-bucket LCP series for a checkout route on mid-range mobile over Fast 3G: two transient spikes poke above the 2500 ms budget line and immediately recover, while a genuine regression sits above it for five consecutive buckets.

Two levers control how much noise the rule absorbs. The first is the evaluation query window: querying now() - interval '15 minutes' and reducing to P75 blends three five-minute buckets into every evaluation, so a single anomalous session cannot dominate. The second is the for value relative to the evaluation interval. With interval: 5m and for: 15m, the breach must hold across three consecutive evaluations — roughly nine buckets of underlying field data once the query window overlap is counted — before the rule fires. That is deliberately conservative: a warning that a checkout LCP has crept from a 2200 ms baseline to a steady 2700 ms on mid-range mobile over Fast 3G is worth a page, but a lone 2700 ms bucket driven by one slow session is not.

Diagnostic Steps

Before wiring an alert, confirm the query returns the same value the panel shows, evaluated as Grafana's alerting engine will see it (a single reduced number, not a series).

curl -s -u admin:$GRAFANA_PW -X POST http://grafana.internal/api/v1/eval \
  -H 'Content-Type: application/json' \
  -d '{"expr":"SELECT percentile_cont(0.75) WITHIN GROUP (ORDER BY value) FROM web_vitals WHERE metric='LCP' AND ts > now() - interval '15 minutes'"}'

Expected output is a single numeric reduction, e.g. 2310, matching the latest LCP panel value within rounding. If it returns a time series instead of one number, add a reduce (Last) expression to the alert query so the threshold compares against a scalar. A common failure here is an evaluation that silently returns NoData because the fifteen-minute window straddles a low-traffic period with too few samples to compute a stable P75; handle that explicitly in the rule (below) rather than letting it flap.

Implementation

Provision the rule as code. The block below is a Grafana unified-alerting rule (LCP warning) with a query stage, a reduce stage, and a threshold condition, plus the for debounce. The noDataState: OK keeps a quiet overnight window from paging on absence of traffic rather than on a real regression.

# /etc/grafana/provisioning/alerting/cwv-rules.yaml
apiVersion: 1
groups:
  - orgId: 1
    name: core-web-vitals
    folder: Performance
    interval: 5m
    rules:
      - uid: lcp-budget-warning
        title: LCP P75 over budget
        condition: C
        for: 15m
        noDataState: OK
        execErrState: Error
        labels:
          severity: warning
          team: frontend
        annotations:
          summary: "LCP P75 is {{ $values.B }}ms, budget 2500ms"
        data:
          - refId: A
            datasourceUid: PerfTSDB
            model:
              format: time_series
              rawSql: "SELECT ts AS time, value FROM web_vitals WHERE metric='LCP' AND ts > now() - interval '15 minutes'"
          - refId: B
            datasourceUid: __expr__
            model: { type: reduce, expression: A, reducer: p75 }
          - refId: C
            datasourceUid: __expr__
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator: { type: gt, params: [2500] }

Wire the contact point and a notification policy that groups by metric and route, waits before the first send, and repeats sparingly so a single ongoing regression is one message, not a stream:

contactPoints:
  - orgId: 1
    name: frontend-perf
    receivers:
      - uid: slack-perf
        type: slack
        settings:
          recipient: "#perf-alerts"
          title: "{{ .CommonLabels.severity }}: budget regression"
policies:
  - orgId: 1
    receiver: frontend-perf
    group_by: ['alertname', 'route']
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h

Routing, Grouping, and Deduplication

The rule decides whether to fire; the notification policy decides how loudly. Grafana's Alertmanager batches everything sharing the group_by key into one notification, holds it for group_wait so near-simultaneous breaches (LCP and INP both slipping after the same deploy) arrive as a single message, and then re-sends only every repeat_interval while the group stays active. The pipeline below is what turns a firing rule into exactly one Slack message.

Choose group_by deliberately. Grouping by ['alertname', 'route'] means a checkout regression and a homepage regression arrive as two distinct messages — usually what you want, because they route to different owners. Grouping by ['alertname'] alone would merge them, which is quieter but hides that two independent routes broke. Set repeat_interval to the shortest interval on which someone will actually take fresh action; four hours suits an off-hours warning, whereas a critical page might use one hour. If Slack is not your team's system of record, mirror the same signal into your APM tool — Datadog Monitors for Budget Regressions covers wiring the identical P75 breach condition into a Datadog monitor so on-call sees one coherent story.

CI Gating Assertion

Alerting watches the field; the gate still blocks the merge. Keep the alert threshold and the build assertion on the same value so an engineer is never paged for a breach the gate should have caught:

{
  "ci": {
    "assert": {
      "assertions": {
        "metric-lcp": ["error", { "maxNumericValue": 2500 }]
      }
    }
  }
}

The lab gate and the field alert are complementary, not redundant. The gate stops a regression a synthetic Lighthouse run can reproduce before it merges; the field alert catches the regressions synthetic runs miss — a third-party tag that only slows real devices, a CDN edge that degrades in one region, a cache-hit ratio that decays over a week. If you also run scheduled synthetic checks between deploys, keep them consistent with these rules by following Alerting on Synthetic Performance Drift.

Verification

Fire a test alert without waiting for a real regression. Insert a sustained over-budget sample series, then check the rule state:

psql "$PERF_DB_URL" -c "INSERT INTO web_vitals(ts, metric, route, session_id, value, source)
  SELECT now() - (g || ' minute')::interval, 'LCP', '/checkout', 'verify-'||g, 4300, 'test'
  FROM generate_series(0,16) g;"
curl -s -u admin:$GRAFANA_PW http://grafana.internal/api/alertmanager/grafana/api/v2/alerts | jq '.[].labels.alertname'

After the next two evaluation cycles the rule moves Pending → Firing and the contact point receives one grouped message reading LCP P75 is 4300ms, budget 2500ms. Confirm the critical rule also fired (since 4300 > 4000), then delete the test rows (WHERE source='test') and verify the alert resolves on the following evaluation. Run this drill after every change to the rule group; a provisioning typo that leaves a rule permanently in NoData is silent until the day you actually need it.

Frequently Asked Questions