Correlating Deploys With Datadog Events

When an LCP line climbs on a dashboard, the first question is always the same: what changed? This how-to, part of the Integrating Performance Budgets With Datadog reference, shows you how to post a deploy event to the Datadog API from your CI pipeline, tag it with the commit SHA, overlay those markers on your Web Vitals graphs, and use event correlation to name the exact release that moved a percentile. The payoff is that a regression stops being a mystery and becomes a diff.

Without deploy markers, a performance investigation is archaeology. You screenshot a graph, guess a rough timestamp, cross-reference the release calendar, and hope nobody shipped a hotfix you forgot about. With markers, the culprit is the vertical line sitting under the knee of the curve. The whole technique costs one HTTP request per deploy and a few tags, and it pairs directly with the Datadog monitors for budget regressions that page you when a percentile breaks its ceiling.

Why Deploy Events Belong on Your Web Vitals Graphs

A Datadog event is a discrete, timestamped record with a title, body, tags, and an optional aggregation key. Unlike a metric, it does not carry a value over time; it marks a moment. That is exactly what a deploy is. Once the event exists, Datadog can render it as an overlay band on any timeseries widget, so the deploy line and the LCP line share one x-axis. When LCP P75 jumps from 2.4 s to 3.9 s on mid-range mobile over Fast 3G at 14:22 UTC, and a deploy marker sits at 14:20 UTC tagged with commit 88f3a1c, the correlation is visual and immediate.

The data flow is small. Your deploy job runs, it finishes promoting the build, and as its last step it fires a single POST to the Datadog Events API. Datadog stores the event and indexes its tags. Every dashboard that has event overlays enabled — including the Web Vitals Grafana dashboard if you also mirror the marker there — draws it without any further work.

Posting a Deploy Event From CI

The event endpoint is POST https://api.datadoghq.com/api/v1/events (use datadoghq.eu if your org is on the EU site). It takes a JSON body and authenticates with a DD-API-KEY header. The minimum useful body sets a title, a text body, an alert_type of info, and a tags array. Add source_type_name set to your CI vendor so Datadog picks the right icon, and set date_happened to the actual deploy Unix timestamp rather than letting the API stamp receipt time, which matters when the POST is delayed by a slow runner.

Here is a self-contained step you can drop into a GitHub Actions job. It runs after the deploy has succeeded and pulls the commit SHA, actor, and environment from the runner context.

#!/usr/bin/env bash
set -euo pipefail

DD_SITE="${DD_SITE:-datadoghq.com}"
SHA_SHORT="$(git rev-parse --short HEAD)"
DEPLOY_TS="$(date +%s)"

curl -sf -X POST "https://api.${DD_SITE}/api/v1/events" \
  -H "Content-Type: application/json" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -d @- <<JSON
{
  "title": "Deploy ${SHA_SHORT} to production",
  "text": "Service checkout-web released by ${GITHUB_ACTOR} from ${GITHUB_REF_NAME}. %%% [run](${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}) %%%",
  "date_happened": ${DEPLOY_TS},
  "priority": "normal",
  "alert_type": "info",
  "source_type_name": "github actions",
  "aggregation_key": "deploy-checkout-web",
  "tags": [
    "env:production",
    "service:checkout-web",
    "event_type:deploy",
    "git.commit.sha:${GITHUB_SHA}",
    "git.commit.sha_short:${SHA_SHORT}",
    "deployed_by:${GITHUB_ACTOR}"
  ]
}
JSON

echo "Posted deploy event for ${SHA_SHORT}"

Wire it into a workflow so it only fires on a real production release, and pass the secret through the environment rather than interpolating it into the command line where it could leak into logs.

name: deploy
on:
  push:
    branches: [main]
jobs:
  release:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Deploy build
        run: ./scripts/deploy.sh
      - name: Mark deploy in Datadog
        env:
          DD_API_KEY: ${{ secrets.DD_API_KEY }}
          DD_SITE: datadoghq.com
        run: ./scripts/datadog-deploy-event.sh

The -sf flags on curl matter: -f makes curl return a non-zero exit on an HTTP 4xx or 5xx, so a rejected event fails the step loudly instead of silently swallowing a typo in your API key. Because this is the last step, a failure here does not roll back a good deploy, but it does turn red in the run so you notice the marker is missing.

Tagging Events With the Commit SHA and Environment

Tags are what make an event queryable and what let you filter the overlay down to a single service. The three that earn their keep are env, service, and the commit SHA. Store the SHA twice — once full (git.commit.sha) and once short (git.commit.sha_short) — because the full form joins cleanly to your RUM data and traces while the short form is what a human types into a filter box. If you follow Datadog's unified service tagging conventions, the same env and service tags already flow from your Datadog RUM Web Vitals stream, so the deploy event and the metric share a filter and line up on the same graph.

The aggregation_key deserves attention. Events sharing an aggregation key collapse into a single thread in the event stream, which keeps a chatty pipeline — say twelve deploys of the same service in a day — from burying every other event. Keep the key per service, not per commit, so the thread groups the deploy history of one service together.

Overlaying Events on Web Vitals Dashboards

On a Datadog timeseries widget, open the widget editor, and under the graph's display options enable event overlays with a search query such as event_type:deploy service:checkout-web env:production. Datadog then draws a shaded vertical band at each matching event's date_happened, and hovering the band shows the title and tags. Set the query narrowly: an unfiltered overlay that also catches monitor-alert events and config changes turns the graph into a picket fence and defeats the purpose.

The visual result is that your LCP, INP, and CLS lines carry release annotations. When LCP P75 for mid-range mobile on Fast 3G sits flat at roughly 2.4 s across several deploys and then steps up to 3.9 s immediately after one band, that band names your suspect. The step is far easier to read than the raw series because the eye anchors on the marker.

Finding the Culprit Deploy Through Event Correlation

Reading a graph by eye works for a single obvious step, but a busy service ships several times an hour, and two deploys can land minutes apart. To pin the culprit, query the events that fall inside the regression onset window rather than scanning the whole day. When a monitor detects that LCP P75 crossed its ceiling at, say, 14:22 UTC, ask Datadog for event_type:deploy service:checkout-web events with a date_happened between the last clean sample and the first bad one. Usually one deploy falls in that window; that is your prime suspect, and its SHA is one click from the diff.

The correlation window should be a few minutes wider than a single RUM aggregation bucket, because field data lags — a real-user LCP sample is only reported after the page finishes loading and the session ends. Treat any deploy whose marker sits inside the shaded onset band as a candidate, then rank candidates by how much of the regressing traffic ran the new build. This is the same reasoning that automated regression detection uses when it flags a shift, and it dovetails with changepoint detection for performance time series that locates the exact bucket where the level changed.

Verifying the Integration

Do not wait for a real regression to discover the marker never posted. Verify end to end. First, confirm the API accepts the payload by watching the curl step exit 0 and echoing the SHA. Second, open the event stream in Datadog and search event_type:deploy service:checkout-web — your latest release should appear within a few seconds with the right tags. Third, open a Web Vitals widget, enable the overlay query, and confirm a band renders at the deploy time. A quick programmatic check reads the event back and asserts the SHA tag is present.

#!/usr/bin/env bash
set -euo pipefail
NOW="$(date +%s)"
FROM=$(( NOW - 900 ))

curl -sf -G "https://api.datadoghq.com/api/v1/events" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
  --data-urlencode "start=${FROM}" \
  --data-urlencode "end=${NOW}" \
  --data-urlencode "tags=event_type:deploy,service:checkout-web" \
| python3 -c 'import sys,json; d=json.load(sys.stdin); e=d.get("events",[]); print("events found:", len(e)); sys.exit(0 if e else 1)'

If that returns at least one event, the pipeline is healthy. For a stronger guarantee, add a scheduled synthetic that posts a canary deploy event tagged event_type:deploy_canary and alerts if none arrives in a day, which catches a silently rotated API key. Finally, tie the loop shut: once markers are reliable, the on-call runbook for a fired budget monitor becomes read the onset time, filter deploys to the window, open the diff — a workflow that also feeds cleaner baseline promotion after green main builds because you know which release owns each shift.

Frequently Asked Questions