Bootstrap Confidence Intervals for LCP

Largest Contentful Paint is a noisy metric, and a CI gate that reads a single Lighthouse run treats that noise as signal. This how-to, part of the Statistical Noise & Flakiness Reduction reference, shows you how to turn a handful of LCP samples into a bootstrap confidence interval for the P75, then gate on that interval instead of a point estimate. The payoff is a gate that fails your pull request only when the evidence for a budget breach is real, and stays green when a runner simply had a bad run.

The technique is deliberately assumption-light. Unlike a parametric test that assumes LCP is normally distributed, the bootstrap resamples the numbers you actually collected, so it works even though LCP distributions are right-skewed and heavy-tailed. If you also want a two-sample comparison against a baseline, pair this page with Welch's t-Test for Performance Regressions; and if your interval is wide because the environment is unstable, fix the source of variance first with Reducing Lighthouse CI Variance in Staging.

Why a Point Estimate Lies About LCP

Suppose your CI job runs Lighthouse 12 times against a preview deploy on a mid-range mobile profile throttled to Fast 3G, and your budget is an LCP P75 of 3500 ms for that exact device and connection class. You compute the P75 of the 12 samples, get 3380 ms, and the naive gate reports a pass. But those 12 numbers are one sample of a random process. Run the job again ten minutes later and the P75 might land at 3560 ms and fail. Nothing changed in the code — the runner was warmer, a neighbour VM was quieter, or a single tail sample shifted the percentile.

A confidence interval quantifies exactly this wobble. Instead of "the P75 is 3380 ms", it says "we are 95% confident the true P75 sits between 3180 ms and 3620 ms". That interval crosses the 3500 ms budget line, which is the honest answer: with only 12 samples we cannot yet tell whether we are inside or outside budget. A point estimate hides that uncertainty and forces a coin-flip decision; the interval surfaces it so the gate can act correctly.

Bootstrap Resampling of Your LCP Run Samples

The bootstrap builds that interval without any distributional assumption. Start with your observed run samples, say the 12 LCP values in milliseconds [2900, 3100, 3050, 3600, 3250, 2980, 3400, 3150, 3700, 3020, 3300, 3120]. A single bootstrap replicate is a new sample of the same size, drawn with replacement from these values. Because draws repeat, one replicate might contain 3700 twice and drop 2900 entirely; the next might over-represent the fast tail. Each replicate is a plausible alternate universe of "what those 12 runs could have looked like".

For every replicate you compute the statistic you actually care about — here the P75 LCP. Repeat the draw-and-compute loop a few thousand times and you accumulate a distribution of P75 values. That distribution is the bootstrap approximation of how much your P75 estimate would jump around if you re-ran the whole CI job over and over. The spread of that distribution is precisely the uncertainty a point estimate throws away.

Computing the P75 Interval

Once you have the sorted list of bootstrap P75 values, a 95% percentile-method confidence interval is simply the 2.5th and 97.5th percentiles of that list. If you ran 2000 replicates, the lower bound is roughly the 50th smallest value and the upper bound is roughly the 1950th. This "percentile bootstrap" is the simplest interval to reason about and the one to reach for in CI, where clarity beats the marginal accuracy of bias-corrected variants.

Two details keep the result honest. First, define your percentile function once and reuse it for both the statistic and the interval bounds, so a P75 always means the same interpolation rule; sparse-sample interpolation is covered in depth in Interpolating Percentiles From Sparse Samples. Second, remember the interval describes the P75 of this environment on a mid-range mobile at Fast 3G — it is not a statement about your users' field P75, which you should read from real-user monitoring. Choosing which percentile to gate on in the first place is discussed in Choosing Between P75 and P90 Budget Targets.

Gating on the Interval, Not the Point Estimate

Here is the decision rule that makes the whole exercise worthwhile. Compare the interval to the budget line and choose one of three outcomes rather than a binary pass/fail on the raw P75:

• Interval entirely below budget — the upper bound is under the ceiling, so you are confidently within budget. Pass the gate.
• Interval straddles the budget — you lack the evidence to call it either way. Do not fail the build on noise; emit a warning and let it through, or collect more samples.
• Interval entirely above budget — even the lower bound exceeds the ceiling, so a real breach is very likely. Fail the gate.

This is why the site's budget gate blocks only when the interval clears the budget, never on a single noisy sample. It converts a jittery metric into a stable signal: the build fails when there is statistical evidence of a regression and stays green otherwise, which is exactly the behaviour you want from a gate people are supposed to trust. The same interval-versus-threshold logic underpins Configuring Statistical Regression Alerts for the alerting side of the house.

A JavaScript Implementation

The whole procedure is a few dozen lines of dependency-free JavaScript that you can drop into a CI script. It takes the array of LCP samples, a percentile, a confidence level, and a replicate count, and returns the point estimate plus interval bounds.

// bootstrap-ci.js — percentile bootstrap CI for an LCP percentile
function percentile(sorted, p) {
  // sorted: ascending array; p in [0,1]; linear interpolation
  if (sorted.length === 1) return sorted[0];
  const rank = p * (sorted.length - 1);
  const low = Math.floor(rank);
  const high = Math.ceil(rank);
  const frac = rank - low;
  return sorted[low] + (sorted[high] - sorted[low]) * frac;
}

function statistic(samples, p) {
  const sorted = [...samples].sort((a, b) => a - b);
  return percentile(sorted, p);
}

function bootstrapCI(samples, { p = 0.75, level = 0.95, B = 2000 } = {}) {
  const n = samples.length;
  const stats = new Array(B);
  for (let b = 0; b < B; b++) {
    const resample = new Array(n);
    for (let i = 0; i < n; i++) {
      resample[i] = samples[Math.floor(Math.random() * n)];
    }
    stats[b] = statistic(resample, p);
  }
  stats.sort((a, b) => a - b);
  const alpha = 1 - level;
  return {
    point: statistic(samples, p),
    lower: percentile(stats, alpha / 2),
    upper: percentile(stats, 1 - alpha / 2),
    B,
  };
}

const lcp = [2900, 3100, 3050, 3600, 3250, 2980, 3400, 3150, 3700, 3020, 3300, 3120];
const ci = bootstrapCI(lcp, { p: 0.75, level: 0.95, B: 2000 });
const budget = 3500; // P75 LCP ms, mid-range mobile, Fast 3G

let verdict;
if (ci.upper <= budget) verdict = "pass";
else if (ci.lower > budget) verdict = "fail";
else verdict = "warn";

console.log(`P75=${ci.point.toFixed(0)}ms CI=[${ci.lower.toFixed(0)}, ${ci.upper.toFixed(0)}] -> ${verdict}`);
process.exit(verdict === "fail" ? 1 : 0);

The exit code is what your CI step keys on: process.exit(1) fails the job only in the confident-breach case, while warn and pass return zero so a noisy run never red-flags a clean pull request. Seed the generator (swap Math.random for a seeded PRNG) if you need reproducible bounds across re-runs.

How Many Resamples Do You Need?

Two counts matter and they are easy to confuse. The number of original samples n sets how wide the true interval is — twelve runs give a wide interval no matter what, and the only cure is collecting more runs. The number of bootstrap replicates B only controls how precisely you estimate that interval's edges; past a point, more replicates just stop the bounds jittering between runs. For a 95% interval, B = 2000 is a reasonable floor and B = 5000 to 10000 makes the bounds essentially stable at millisecond resolution. The loop is cheap: 2000 replicates over 12 samples runs in a few milliseconds, so there is rarely a reason to go below it in CI.

The table below shows how the interval tightens as you add real runs while holding B = 2000, all on a mid-range mobile profile at Fast 3G with a 3500 ms P75 budget. Notice the width, not the point estimate, is what changes your gate decision.

Verifying the Interval

Trust the interval only after you have checked it behaves. Run a small validation harness: draw samples from a known distribution whose true P75 you can compute analytically, build a 95% interval hundreds of times, and confirm it contains the true P75 in about 95% of trials. Coverage far below 95% means your interval is too narrow (often from too few original runs or a buggy percentile function); coverage far above means it is needlessly wide.

# Sanity-check coverage: expect ~95% of intervals to contain the true P75
node --input-type=module <<'EOF'
import { bootstrapCI } from "./bootstrap-ci.mjs";
const trueP75 = 3400;               // known target for the synthetic generator
const draw = () => 2800 + Math.round(Math.random() * 1200); // uniform-ish LCP ms
let covered = 0, trials = 500;
for (let t = 0; t < trials; t++) {
  const sample = Array.from({ length: 24 }, draw);
  const { lower, upper } = bootstrapCI(sample, { p: 0.75, level: 0.95, B: 1000 });
  if (lower <= trueP75 && trueP75 <= upper) covered++;
}
console.log(`coverage = ${(100 * covered / trials).toFixed(1)}% (target ~95%)`);
EOF

Two operational checks close the loop. Confirm that re-running the same input twice yields bounds within a few milliseconds of each other — large swings mean B is too low. And confirm the gate actually flips: feed it a synthetic sample shifted 300 ms slower and verify the verdict moves from pass to warn to fail as the shift grows. A gate you have watched change state under a known regression is one you can defend when it blocks a colleague's pull request. For the deeper theory behind significance in noisy pipelines, keep Statistical Significance Testing for Noisy CI close by.

Frequently Asked Questions