Datadog Monitors for Budget Regressions

Once your field Web Vitals land in Datadog, the next job is to make the platform tell you when a route slips past its budget instead of waiting for a human to notice a drooping graph. This page is a step-by-step build for a metric monitor on P75 LCP by route, and it is part of the Integrating Performance Budgets With Datadog reference. It assumes you have already followed Sending Web Vitals to Datadog RUM so that an lcp measure and a route (or view.name) attribute exist on your RUM events.

The whole design turns on one tension. Field data is noisy: a single slow session, a device rebooting mid-page, or a thin sample window can spike a percentile for a few minutes. If your monitor reacts to every twitch it becomes a flapping nuisance that the team mutes within a week. Tune it too slack and a real regression ships to production for hours before anyone hears about it. Everything below — the evaluation window, the recovery threshold, the per-segment split — exists to hold that line.

Choose the metric and the aggregation

Budgets are percentile statements, never averages. A mean LCP hides the tail where real users suffer, so the monitor queries the 75th percentile of the lcp measure, matching the target you set when using the 75th percentile for real-world targets. In Datadog RUM the query is built from the @view.loading_time or a custom @lcp attribute; if you forward vitals as custom metrics instead, you compute a distribution metric and read p75: off it.

For a mid-range mobile device on a Fast 3G-equivalent connection, a "good" LCP P75 sits at or below 2500 ms, and our production budget for the marketing landing route is 2800 ms P75 on that same device+connection class. Desktop on cable gets a tighter 1800 ms P75 ceiling. Keep those numbers segmented — never blend mobile and desktop into one series, or the healthier desktop mass drags the percentile down and masks a mobile regression.

Set the threshold and the evaluation window

A metric monitor has two numbers that govern its temperament: the alert threshold and the evaluation window. The threshold is your budget — 2800 ms P75 for the mobile landing route. The evaluation window is how long the query looks back each time it runs. This is the single most important anti-flap control you have.

A short window (say 5 minutes) recomputes the percentile off a thin slice of sessions. On a low-traffic route that slice might hold twenty samples, and the P75 of twenty samples jumps around wildly. Widen the window to 30 minutes and the same route accumulates enough sessions that the percentile stabilizes, so a lone slow outlier no longer drags the aggregate over the line. The trade is latency: a 30-minute window means a real regression takes up to half an hour of sustained badness before it trips. For a page whose budget is a P75 field target rather than a synthetic gate, that latency is acceptable — you are catching production drift, not blocking a pull request.

Read the table by traffic. A busy route can run a tight 5-minute window and stay honest; a quiet route needs 30 to 60 minutes to gather enough samples for a trustworthy P75. When in doubt, widen the window rather than loosen the threshold — a wider window preserves the budget number while buying stability, whereas a looser threshold silently forgives real slowness.

Add a recovery threshold to stop flapping

If a monitor alerts the instant P75 crosses 2800 ms and recovers the instant it dips back under 2800 ms, a metric that hovers right on the budget will toggle open and closed repeatedly — the classic flap. The fix is hysteresis: alert at 2800 ms P75 but only mark the monitor recovered once P75 falls back below 2500 ms. The 300 ms gap means a route that is genuinely borderline stays in the alerting state instead of spamming resolve/re-alert notifications, and it only clears when performance has meaningfully improved rather than merely brushed the line.

Datadog exposes this as recovery_threshold in the monitor options, distinct from the primary threshold. Pair it with a renotify_interval so that a still-firing budget breach re-pings the channel every, say, 60 minutes rather than going quiet after the first alert. This mirrors the broader approach in Alerting on Performance Budget Regressions, which covers the same hysteresis idea for Grafana-fed alert rules.

Split by segment with a multi-alert monitor

One monitor for the whole site is a trap: it either aggregates every route into one meaningless percentile, or it forces you to hand-build a monitor per route and drift out of sync. A multi-alert monitor solves this. You group the query by {route, device_type} and Datadog evaluates the threshold independently for each group, opening and closing a separate alert per combination. Add a new route to your RUM and it is covered automatically the moment traffic appears; retire a route and its group simply stops evaluating.

Multi-alert also lets the notification carry the offending segment. The message template can interpolate {{route.name}} and {{device_type.name}} so on-call sees "P75 LCP on /checkout (mobile) is 3100 ms, budget 2800 ms" instead of a vague site-wide flag. Because each group carries its own device+connection context, you can keep the desktop groups on their tighter 1800 ms P75 ceiling and the mobile groups on 2800 ms without a second monitor — Datadog does not vary the numeric threshold per group, so where ceilings genuinely differ by device you run one monitor per device_type, each scoped by a device_type filter and grouped by {route}.

Define the monitor as code

Click-ops monitors rot: someone tweaks a threshold in the UI, nobody records why, and the budget quietly drifts. Define the monitor in Terraform so the threshold, window, and recovery value live in version control next to the budget policy they enforce. Here is the mobile LCP monitor with a 30-minute evaluation window and hysteresis.

resource "datadog_monitor" "lcp_p75_mobile_by_route" {
  name    = "P75 LCP budget breach — mobile — {{route.name}}"
  type    = "query alert"
  message = <<-EOT
    P75 LCP on {{route.name}} (mobile / Fast 3G class) is {{value}} ms,
    over the 2800 ms budget. Check recent deploys and third-party tags.
    @slack-web-perf-alerts
  EOT

  query = "percentile(last_30m):p75:rum.view.lcp{device_type:mobile} by {route} > 2800"

  monitor_thresholds {
    critical = 2800
    critical_recovery = 2500
  }

  notify_no_data    = false
  renotify_interval = 60
  require_full_window = false
  evaluation_delay  = 300

  tags = ["team:web-perf", "budget:lcp", "device:mobile"]
}

Two options deserve a note. evaluation_delay of 300 seconds tells Datadog to wait five minutes before evaluating a window, which lets late-arriving RUM beacons settle so the percentile is computed on a complete slice — RUM ingestion is not instantaneous. require_full_window = false allows the monitor to evaluate before a full 30 minutes of history exists, which matters right after you create it; set it to true only if partial-window evaluations produce noisy early alerts.

If you drive Datadog through its HTTP API instead of Terraform, the same monitor is a POST to /api/v1/monitor with an equivalent JSON body.

{
  "name": "P75 LCP budget breach - desktop - {{route.name}}",
  "type": "query alert",
  "query": "percentile(last_30m):p75:rum.view.lcp{device_type:desktop} by {route} > 1800",
  "message": "P75 LCP on {{route.name}} (desktop / cable) is {{value}} ms, over the 1800 ms budget. @slack-web-perf-alerts",
  "options": {
    "thresholds": { "critical": 1800, "critical_recovery": 1600 },
    "notify_no_data": false,
    "renotify_interval": 60,
    "require_full_window": false,
    "evaluation_delay": 300
  },
  "tags": ["team:web-perf", "budget:lcp", "device:desktop"]
}

Note the desktop body uses the tighter 1800 ms P75 ceiling with a 1600 ms recovery line, and filters device_type:desktop — one monitor per device class, each with its own budget, exactly as the segmentation section described.

Verify the monitor before you trust it

A monitor you have never seen fire is a monitor you cannot trust. Verify it three ways before you consider the budget enforced.

First, apply the config and confirm it exists and parses:

terraform apply -target=datadog_monitor.lcp_p75_mobile_by_route
curl -s -X GET "https://api.datadoghq.com/api/v1/monitor?tags=budget:lcp" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" | jq '.[].name'

Second, dry-run the alert logic against real history with the monitor validate/test endpoint, which replays the query over the past window and reports whether it would have alerted — no need to actually degrade production:

curl -s -X POST "https://api.datadoghq.com/api/v1/monitor/validate" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"type":"query alert","query":"percentile(last_30m):p75:rum.view.lcp{device_type:mobile} by {route} > 2800","options":{"thresholds":{"critical":2800,"critical_recovery":2500}}}'

Third, force a transient breach in a controlled way — ship a deliberate 500 ms delay to a low-traffic staging route, watch the P75 for that group climb over 2800 ms, confirm the alert opens after roughly one full window, then revert and confirm it clears only once P75 falls under 2500 ms. That end-to-end test proves the threshold, the window, and the recovery gap all behave together. Record the deploy that caused it so you can correlate deploys with Datadog events and confirm the alert timeline lines up with the change. From here, treat every real firing as an incident with a root cause, not a graph to squint at.

Frequently Asked Questions