Building P75/P99 Aggregation Pipelines

A raw beacon stream is millions of individual numbers; a budget gate needs one number — the P75 — and one tripwire — the P99. The pipeline between them must compute percentiles that are accurate at the tail, cheap to update incrementally, and mergeable across time windows. This guide is part of the Custom Performance Beacons & RUM reference and covers the percentile-method tradeoff, how a sketch keeps the tail honest, rolling windows, storage, and the final CI assertion.

The naive approach — store every value and sort on read — is exact but does not scale and cannot be rolled up. Production pipelines use a mergeable sketch (t-digest or fixed-bucket histograms) so each ingestion window produces a small structure that can be combined into any larger window without re-reading raw events. Get that shape right once and the same stored digests answer a 1-day regression question, a 7-day trend, and the 28-day reading you actually gate on — the same window Chrome uses to assess Core Web Vitals in the field.

Percentile Method Comparison

The right method depends on tail accuracy, mergeability, and storage. The table compares the three you will choose between.

For Core Web Vitals, t-digest is the default: it spends its resolution where the percentiles you gate on live (the upper tail), stays around a few KB per window regardless of session count, and merges cleanly so a daily digest rolls up into a 28-day reading without touching raw events. Fixed-bucket histograms are the pragmatic runner-up when your metric has a well-known range and you already run Prometheus — for LCP you might allocate 100 ms buckets from 0 to 8000 ms, which caps the P75 estimate error for mid-range mobile on Fast 3G at half a bucket, roughly 50 ms. Exact sorting is reserved for a single small window during verification, never for the live gate.

How t-Digest Concentrates Resolution at the Tail

The reason a t-digest beats a uniform histogram for percentile gating is where it puts its accuracy. A t-digest groups nearby values into centroids whose allowed weight is largest near the median and shrinks toward the extremes. That means the median region is summarised coarsely — you do not gate on the exact median — while the P90-to-P99 band is described by many small centroids, so the number you actually enforce is faithful. A budget that fails a build when LCP P75 crosses 2500 ms for mid-range mobile on Fast 3G needs that reading stable to within a few milliseconds run-to-run, and the sketch delivers it at a fraction of the storage.

Diagnostic Steps

Confirm the aggregation is faithful before you gate on it — a histogram whose buckets are too wide, or a digest fed unsorted garbage, produces a confident wrong number.

1. Spot-check a computed percentile against a brute-force sort on one window of raw data:

   curl -fsSL "$RUM_QUERY_URL?window=1d&metric=LCP&raw=true" | \
     jq -s 'sort_by(.v) | .[(length*0.75)|floor].v'
   # 2380  -> compare against the pipeline's reported P75 for the same window

2. Verify the digest is mergeable by checking a 7-day rollup equals the merge of seven daily digests, not a re-sort:

   curl -fsSL "$RUM_QUERY_URL?window=7d&metric=LCP&pct=99"   # rolled-up
   # { "LCP": 4480 }

3. Confirm the tail has enough samples to trust. A P99 computed from a segment with only a few hundred kept sessions is dominated by a handful of outliers and will jitter run-to-run. When a segment is thin, widen the window or interpolate — see Interpolating Percentiles From Sparse Samples — rather than gating on a noisy number.

If the sketch and the brute-force sort disagree by more than a few percent on the same window, the compression or the merge is wrong; do not ship the gate until they converge.

Implementation

This pipeline ingests a window of raw beacons into a t-digest per metric and segment, persists the serialized digest, and answers percentile queries by merging the digests covering the requested window. It uses a t-digest library so the sketch math is correct and mergeable.

// aggregate.js — run per ingestion window (e.g. hourly) per metric+segment
import TDigest from "tdigest";

// 1. Build a digest from this window's raw beacon values.
export function buildDigest(values) {
  const td = new TDigest();
  for (const v of values) td.push(v);
  td.compress();
  return td.toArray();          // serialized centroids, ~1-5 KB
}

// 2. Answer a percentile query by merging the digests in the window.
export function percentile(digestRows, p) {
  const merged = new TDigest();
  for (const row of digestRows) merged.push_centroid(row); // mergeable rollup
  merged.compress();
  return Math.round(merged.percentile(p)); // p in [0,1], e.g. 0.75 or 0.99
}

Segment on the dimensions your budgets are written against — at minimum metric, device_class, and route — because a single blended P75 hides a mobile regression behind fast desktop traffic. Keep the segment cardinality bounded: one digest per metric per device class per route template is fine, but never key on a raw URL or a user id or the digest count explodes. For storage, a columnar store makes the rollup a one-line aggregate.

-- rollup query: read the digests covering the gate window, not raw events.
-- (Engines like ClickHouse expose quantileTDigest natively; this is the shape.)
SELECT metric,
       device_class,
       route,
       quantileTDigestMerge(0.75)(digest) AS p75,
       quantileTDigestMerge(0.99)(digest) AS p99
FROM rum_digests
WHERE ts >= now() - INTERVAL 28 DAY
GROUP BY metric, device_class, route;

Window by ingestion time and roll up on read so a single slow hour does not permanently skew the gate, and so you can answer 1-day, 7-day, and 28-day questions from the same stored digests. Retain raw beacons for a short window (for example, 7 days) for the spot-check above, then keep only the digests long-term — they are orders of magnitude smaller.

CI Gating Assertion

The gate reads the rolled-up P75 and P99 straight from the pipeline and fails the build when either breaches budget — P75 as the hard gate, P99 as a tail tripwire surfaced in the log. The budgets below are expressed for mobile field traffic: LCP P75 at 2500 ms and P99 at 4500 ms, INP P75 at 200 ms and P99 at 500 ms, and CLS P75 at 0.10 (encoded as 100 milli-units) — the ceilings you set in Percentile-Based Threshold Tuning.

# .github/workflows/percentile-gate.yml
name: P75/P99 Budget Gate
on:
  schedule:
    - cron: "0 6 * * *"
  workflow_dispatch:
jobs:
  percentile-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - name: Assert P75/P99 from aggregation pipeline
        env:
          RUM_QUERY_URL: ${{ secrets.RUM_QUERY_URL }}
        run: |
          curl -fsSL "$RUM_QUERY_URL?window=28d&pct=75,99&metrics=LCP,INP,CLS" -o pct.json
          node -e '
            const d = require("./pct.json");
            const p75Budget = { LCP: 2500, INP: 200, CLS: 100 };
            const p99Budget = { LCP: 4500, INP: 500, CLS: 250 };
            let failed = false;
            for (const m of Object.keys(p75Budget)) {
              const okP75 = d[m].p75 <= p75Budget[m];
              const okP99 = d[m].p99 <= p99Budget[m];
              console.log(`[PctGate] ${m} P75=${d[m].p75}/${p75Budget[m]} ${okP75?"PASS":"FAIL"} | P99=${d[m].p99}/${p99Budget[m]} ${okP99?"OK":"WATCH"}`);
              if (!okP75) failed = true; // P75 gates; P99 only warns
            }
            process.exit(failed ? 1 : 0);
          '

Verification

The passing signal is every metric under its P75 budget, with P99 shown for tail awareness:

[PctGate] LCP P75=2310/2500 PASS | P99=4280/4500 OK
[PctGate] INP P75=180/200 PASS | P99=470/500 OK
[PctGate] CLS P75=80/100 PASS | P99=240/250 WATCH

A WATCH on P99 with a passing P75 is the early signal of a tail regression — investigate before it pulls the median up and trips the gate. It usually means a slice of the tail is drifting: a slow third-party on a subset of sessions, or a device class you would only see if you slice the field traffic the way Head-Based vs Tail-Based RUM Sampling describes, keeping the slow sessions the digest needs. If the pipeline's P75 disagrees with the brute-force sort from the diagnostic step by more than a few percent, the digest compression or merge is wrong; rebuild from raw before trusting the gate. Calibrate the budgets themselves against Percentile-Based Threshold Tuning.

Frequently Asked Questions