Percentile-Based Threshold Tuning

A mean hides the failure that matters: it absorbs a handful of fast loads and a handful of slow outliers into one number that no real user ever experiences, so a budget gated on the average passes while a quarter of your sessions degrade. Percentile-driven gating replaces brittle mean and median baselines with distribution-aware thresholds that track the tail behaviour users actually feel. This is the distribution layer of the Threshold Calibration & Baseline Management reference: it specifies how to compute P75, P90, and P99 from your runs, decide which percentile belongs to which metric, and assert on that percentile in CI so a tail regression is unmergeable.

The work splits into three coupled concerns — which percentile expresses the contract for a given metric, how you compute that percentile stably from a finite sample of runs, and what assertion fails the build when the percentile drifts. Choose the percentile too low and you ship tail pain; compute it from too few runs and the gate flaps; assert on a noisy percentile and the team mutes it. This page is the authoritative spec for all three, and it connects to Interpolating Percentiles From Sparse Samples for the arithmetic that keeps a low-N estimate honest.

Core Concept: From a Distribution to a Gate

Every metric you collect — LCP, INP, CLS, a custom beacon — is a distribution, not a point. A percentile is the value below which that share of observations fall: P75 is the slowest experience of the fastest three-quarters of users, P90 the slowest of the fastest nine-tenths. The budget line is a single horizontal threshold; the gate fails when the chosen percentile of the run distribution crosses it. Because web performance distributions are right-skewed — a long thin tail of slow sessions dragging away from a dense cluster of fast ones — the mean sits well to the right of the median and to the left of the tail, describing no one. The percentile, by contrast, is a real observation: some user actually had that experience. The diagram below shows where each marker sits and how the budget line relates to them.

The practical consequence is that the percentile you pick is a policy decision about how many users the budget defends, not merely a statistical detail. Gating LCP at P75 on mid-range mobile over Fast 3G says "three in four users must load the hero within the ceiling"; moving to P90 promises the same to nine in ten and pulls the assertion further into the expensive tail, where each additional point of coverage costs disproportionately more engineering. That trade is the heart of Choosing Between P75 and P90 Budget Targets, and it is why the same 2,500 ms number can be either comfortable or brutal depending on the percentile bolted to it.

Prerequisites & Environment

Percentile tuning consumes a sample of runs, not a single audit. You need enough collection to estimate the percentile and enough storage to track it over time.

• A multi-run collector — Lighthouse CI with numberOfRuns ≥ 5 per URL, or a RUM stream feeding an aggregation store. A single run cannot produce a percentile. Pin collection determinism per the Lighthouse CI Configuration & Storage reference so the spread you measure is real, not runner noise. When the percentile comes from field data rather than lab runs, build the rollup through a dedicated P75/P99 aggregation pipeline so the numbers you gate on are computed once and consistently.
• Node.js ≥ 18 for the percentile evaluation script, plus jq for quick CLI inspection of result manifests.
• A field or lab dataset with a known sample size. Record N alongside every percentile; a P90 from 8 samples is not a P90. Below ~20 samples for lab and ~1,000 sessions for field, percentile estimates are too unstable to gate on — see the sample-size section and Troubleshooting below.
• Environment pinning. Always tag each percentile with its device class and connection profile. A P75 LCP of 2,500 ms on mid-range mobile / Fast 3G is a different contract from a P75 LCP of 1,200 ms on desktop / cable, and mixing them silently corrupts the budget. Calibrate emulation through Device & Network Emulation Weighting.

Configuration Reference

Express percentile budgets as data, not code, so the gate is auditable and diff-able. The annotated thresholds.json below is the authoritative spec — each route declares the metric, the percentile that expresses its contract, the ceiling, and the minimum sample size required before the assertion is allowed to fail rather than warn.

{
  "minimumSamples": 20,
  "routes": {
    "/checkout": {
      "lcp":  { "percentile": 75, "maxMs": 2500, "level": "error" },
      "inp":  { "percentile": 75, "maxMs": 200,  "level": "error" },
      "cls":  { "percentile": 90, "max":   0.10, "level": "error" }
    },
    "/landing": {
      "lcp":  { "percentile": 75, "maxMs": 2800, "level": "error" },
      "inp":  { "percentile": 90, "maxMs": 300,  "level": "warn"  }
    }
  },
  "gating": { "failBufferPercent": 5 }
}

percentile names which point of the distribution is the contract — P75 for typical-user metrics, P90 for stricter flows, escalating to P95/P99 only for revenue-critical paths. minimumSamples blocks the gate from asserting on an under-sampled percentile: below the floor it downgrades to warn. failBufferPercent adds a small tolerance so a percentile sitting exactly on the line does not flap the build. Note that each ceiling is device-and-connection specific: the /checkout LCP ceiling of 2,500 ms assumes mid-range mobile on Fast 3G, and a desktop profile would carry its own file with a tighter number such as 1,500 ms at the same P75. Keep one thresholds file per device class rather than pooling classes into a single distribution — pooling is the most common way a percentile budget quietly becomes meaningless.

Step-by-Step Implementation

1. Collect a sample. Run the collector with at least five runs per URL so each metric has a distribution to percentile over.

   npx lhci collect --numberOfRuns=5 --url=https://staging.example.com/checkout

   Expected tail: Run #5 ... Done running Lighthouse! and a .lighthouseci/ directory holding five JSON reports.

2. Compute percentiles from the runs. The script below reads every numeric value for a metric, sorts, and interpolates the requested percentile — the same nearest-rank-with-interpolation method CrUX uses.

   // scripts/percentile.js
   function percentile(values, p) {
     const sorted = [...values].sort((a, b) => a - b);
     if (sorted.length === 0) return NaN;
     const rank = (p / 100) * (sorted.length - 1);
     const lo = Math.floor(rank);
     const hi = Math.ceil(rank);
     if (lo === hi) return sorted[lo];
     return sorted[lo] + (rank - lo) * (sorted[hi] - sorted[lo]);
   }
   
   module.exports = { percentile };

   node -e "const {percentile}=require('./scripts/percentile');\
   console.log(percentile([180,190,205,210,260,195,200],75))"

   Expected output: 207.5 — the interpolated P75 INP in milliseconds across those seven runs.

3. Assert against the budget. Feed the computed percentile and the thresholds.json contract into an evaluator that exits non-zero on a breach, then commit both thresholds.json and the evaluator. A minimal but complete evaluator reads the reports, applies the minimumSamples and failBufferPercent rules, and prints a per-metric verdict.

   // scripts/evaluate-percentiles.js
   const fs = require("fs");
   const path = require("path");
   const { percentile } = require("./percentile");
   
   const cfg = JSON.parse(fs.readFileSync("./config/thresholds.json", "utf8"));
   const dir = ".lighthouseci";
   
   function collectValues(metricKey) {
     return fs.readdirSync(dir)
       .filter((f) => f.startsWith("lhr-") && f.endsWith(".json"))
       .map((f) => JSON.parse(fs.readFileSync(path.join(dir, f), "utf8")))
       .map((lhr) => lhr.audits[metricKey].numericValue)
       .filter((v) => typeof v === "number");
   }
   
   const auditKey = { lcp: "largest-contentful-paint", inp: "interaction-to-next-paint", cls: "cumulative-layout-shift" };
   let failed = false;
   
   for (const [route, metrics] of Object.entries(cfg.routes)) {
     for (const [metric, rule] of Object.entries(metrics)) {
       const values = collectValues(auditKey[metric]);
       const ceiling = rule.maxMs ?? rule.max;
       const buffered = ceiling * (1 + cfg.gating.failBufferPercent / 100);
       const value = percentile(values, rule.percentile);
       const underSampled = values.length < cfg.minimumSamples;
       const effectiveLevel = underSampled ? "warn" : rule.level;
       const over = value > buffered;
       const status = !over ? "PASS" : effectiveLevel === "error" ? "FAIL" : "WARN";
       if (status === "FAIL") failed = true;
       console.log(`${route} ${metric} P${rule.percentile}=${value.toFixed(1)} ceiling=${ceiling} n=${values.length} -> ${status}`);
     }
   }
   
   process.exit(failed ? 1 : 0);

   Running this against a green /checkout sample prints lines such as /checkout lcp P75=2380.0 ceiling=2500 n=5 -> PASS and exits 0; a tail regression that pushes P75 past the buffered ceiling flips the line to FAIL and exits 1, which is exactly what a required status check consumes.

How Many Samples a Percentile Needs

The single most common way a percentile gate loses trust is asserting on an estimate computed from too few observations. A percentile is an order statistic — it is literally the value at a rank — so the higher the percentile and the smaller the sample, the more one unlucky run swings it. With five runs, the P90 is effectively the second-slowest run: change that one observation and the whole gate moves. The relationship is not linear; each step up the tail demands disproportionately more samples to pin the estimate within a tolerable band. The chart below shows a defensible minimum sample count per percentile for gating, spanning lab runs at the bottom of the tail and field sessions at the top.

The operational rule that falls out of this is simple: gate lab runs at P75 and P90, and reserve P95 and P99 for field datasets where the session count naturally reaches the thousands. When you have no choice but to estimate a high percentile from a thin sample — a new route with little traffic, say — treat the number as a warn-level signal and lean on the interpolation and confidence-interval techniques in Bootstrap Confidence Intervals for LCP to know how wide the uncertainty band is before you let the estimate block a merge. A percentile you cannot bound is a percentile you should not gate on.

Threshold Calibration

Pick the percentile per metric from how the metric behaves and how much risk a slow tail carries, not from habit. Layout shift is near-binary and rare-but-severe, so it earns a stricter percentile than a metric that degrades gracefully. The matrix below is a representative starting point by metric and context; derive the actual ceiling from your own field P75 and set the lab assertion 10–15% tighter to absorb lab-to-field drift.

Set the assertion level to warn for any percentile still being calibrated and promote to error only after the threshold has held for two consecutive weekly baselines, so the gate earns trust before it can block a merge. The decision of which percentile a metric deserves is mechanical enough to draw as a tree — walk it top to bottom and stop at the first branch that matches the route.

CI Enforcement

This GitHub Actions job collects five runs, computes the configured percentile per metric, and fails the required status check when any error-level percentile breaches its ceiling. The pipeline it encodes is deliberately linear — collect, sort, interpolate, compare, gate — and each stage owns exactly one responsibility so a failure is easy to localise.

name: Percentile Performance Gate
on:
  pull_request:
    branches: [main]

jobs:
  percentile-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
      - run: npm ci
      - run: npm run build
      - name: Collect runs
        run: npx lhci collect --numberOfRuns=5 --url=http://localhost:8080/checkout
      - name: Evaluate percentile budgets
        run: node ./scripts/evaluate-percentiles.js
              --reports .lighthouseci
              --thresholds ./config/thresholds.json
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: percentile-reports
          path: .lighthouseci/

Require the percentile-gate check in branch protection so a tail regression cannot merge. Wire the same evaluator into Automated Regression Detection to alert when a percentile trends toward its ceiling before it crosses, and stabilise the underlying distribution first via Statistical Noise & Flakiness Reduction so the percentile you assert on is signal, not jitter.

Lab-to-Field Drift and Sizing the Buffer

A lab percentile and a field percentile of the same name measure different populations. Lab runs come from a pinned emulated device on a throttled connection in a clean profile; field percentiles come from real hardware, real networks, warmed caches, and background tabs. The two rarely agree, and the gap is usually one-directional — field is slower in the aggregate because the real world is messier than a CI runner. That is why the calibration rule sets the lab ceiling 10–15% tighter than the field target you actually care about: if you want a field P75 LCP of 2,500 ms on mid-range mobile / Fast 3G, gate the lab P75 at roughly 2,150–2,250 ms so that the lab pass gives you headroom against the field number that reaches users.

The failBufferPercent value plays the opposite role and must not be confused with the drift margin. The buffer exists to stop flapping — a percentile that sits within measurement noise of the ceiling toggling PASS and FAIL between otherwise identical runs. A 5% buffer on a 2,500 ms ceiling tolerates an estimate up to 2,625 ms before failing, which absorbs ordinary run-to-run jitter without hiding a real regression. Size it from the observed variance of the metric on your runners, not from a round number: a metric with a coefficient of variation near 3% wants a buffer around 5%, while a noisier metric on a shared runner might need 8–10% until you reduce that variance at the source. Keep the drift margin in the ceiling and the flap tolerance in the buffer as two separate dials; collapsing them into one number leaves you unable to tell a genuinely regressed build from a merely noisy one, and a noisy metric that a Welch's-t-test-style comparison would flag is better handled at the source than papered over with a fat buffer.

Troubleshooting & Edge Cases

• Percentile flaps run-to-run → the sample is too small. A P90 over 5 runs is dominated by one observation; raise numberOfRuns to 9+, or aggregate several PR runs into a rolling window before evaluating.
• P99 is wildly unstable in lab → you cannot estimate a P99 from tens of samples. Reserve P95/P99 for field datasets with thousands of sessions; gate lab runs at P75/P90 and watch the high tail in RUM.
• Lab percentile passes but field P75 fails → expected lab-to-field gap. Set lab ceilings 10–15% tighter than the field target you actually care about, as described above.
• Mean looks fine, users complain → the average is absorbing the tail. Switch the assertion from mean/median to the percentile; that is the entire point of this layer.
• Mixed device classes in one percentile → segment first. Compute and gate P75 per device class and connection profile, never on a pooled distribution.
• A single slow third-party run poisons P90 → apply IQR or Z-score outlier filtering before the percentile step, and pin vendor versions per Third-Party Script Constraints.
• Two different percentile scripts disagree on the same data → they are using different interpolation conventions. Standardise on the linear interpolation in the shared percentile() function above and reuse it everywhere, so the CI gate, the dashboard, and the RUM pipeline all report the same number.
• A new low-traffic route has no gateable percentile → seed it with a warn-level P75 and a minimumSamples guard, then promote to error once the sample accumulates, rather than either blocking merges on noise or leaving the route ungated.

Frequently Asked Questions