Lighthouse CI vs Custom RUM Beacons

Two engineers look at the same page and quote different numbers. One runs Lighthouse CI in a pull request and reports an LCP of 2.1s; the other pulls the RUM dashboard and reports an LCP P75 of 3.4s for mid-range Android users on Fast 3G. Both are correct, because they are answering different questions. This page, part of the Comparing Performance Testing Tools reference, is a head-to-head decision guide for the two most common signals a budget-gating team runs: a Lighthouse CI lab run that executes before merge, and Custom Performance Beacons & RUM collected from live sessions after deploy. The goal is not to crown a winner. It is to tell you exactly which signal blocks the merge, which sets the ceiling, and how to wire them together so each does the job it is actually good at.

What Each Tool Actually Measures

Lighthouse CI is a lab instrument. It launches one headless Chrome under a fixed CPU throttle and a fixed network profile, loads the URL cold, and reports a synthetic score. That determinism is its entire value proposition: run it ten times on unchanged code and the LCP moves only a few percent, so a failing build is a real signal rather than noise. What it cannot do is represent your users. It samples exactly one device and one network, on the first paint of a cold cache, with no real interaction — so its INP and CLS numbers are directional guesses, not lived experience.

Custom RUM beacons are the opposite. A small script fires a PerformanceObserver, reads the Web Vitals your visitors actually experienced, and posts them to your collector. That population is the ground truth — every phone, every flaky connection, every locale, warm cache and cold, mid-scroll interactions and all. The price of that truth is noise and delay: any single session is meaningless, and you need thousands before a P75 stabilizes. RUM is the only source that measures INP and CLS honestly, because both require real input and a full session to exist at all.

The one-line takeaway: Lighthouse CI is a controlled experiment, RUM is a census. You gate a merge on the experiment because it is repeatable and it exists before code ships. You calibrate the experiment against the census because only the census knows what your users actually endure.

Signal Latency: PR-Time vs Days of Field Data

The single biggest practical difference is when the number arrives. A Lighthouse CI run returns a verdict in roughly two to five minutes, inside the pull request, before anyone merges. That is the only window in which you can prevent a regression rather than discover one. Custom RUM cannot do this: a beacon only exists after the code is live and a real person loads the page, and a trustworthy percentile needs a traffic window on top of that. For a moderate-traffic route it can take one to three days before the LCP P75 for mid-range Android on Fast 3G stops drifting sample-to-sample.

This latency gap is why "just gate on real-user data" fails in practice. You cannot fail a build on a number that will not exist until three days after the build ships. RUM is a lagging indicator by construction. It is superb for catching slow drift, for confirming that a fix actually helped real users, and for setting honest targets — but it is structurally incapable of being a merge gate. If you try to force it into that role you either block on stale data from the previous release or you block on a tiny, noisy sample that fails builds at random.

Cost and Maintenance

The two signals also cost money and attention in very different shapes. Lighthouse CI is cheap to run and cheap to own: it is CI minutes plus a small results store, and the config in your Lighthouse CI Configuration & Storage setup rarely changes once tuned. Custom RUM is cheap per beacon but grows with traffic, and it carries real engineering ownership — a collector endpoint, a schema, sampling, storage, and an aggregation pipeline that computes percentiles you can trust.

Read the maintenance column carefully before you commit to RUM. A lab-only setup can be owned by one engineer part-time. A production RUM pipeline is a small service: you will design efficient beacon payloads, choose a sampling rate for high-traffic routes, and build the P75/P99 aggregation that turns raw events into a defensible ceiling. That work pays for itself — it is the only way to know your real numbers — but it is not free, and pretending otherwise is how RUM pipelines rot into dashboards nobody trusts.

A Decision Matrix

The choice is rarely "one or the other" forever. It is "which signal for this specific job." Map the job to the tool with the property that job requires — determinism and pre-deploy timing point to the lab; population truth and interaction fidelity point to the field.

Notice that LCP and total byte weight can be gated well in the lab, because a cold single-navigation run reproduces them faithfully. INP and CLS cannot — they land on the RUM side of every branch, because a synthetic run barely interacts with the page and never scrolls a real session. If your budget policy leans heavily on INP, that is a strong reason to stand up RUM early and to calibrate targets with Percentile-Based Threshold Tuning rather than trusting a lab INP.

The Recommended Combined Setup

The setup that works in production is neither tool alone. Lab gates the pull request; RUM calibrates the ceiling the lab enforces. The lab is your fast, deterministic tripwire that fails a merge when LCP or byte weight regress. RUM is the slow, honest oracle that periodically re-derives what that lab threshold should be, so the gate stays anchored to reality instead of drifting into a number that passes CI while users suffer.

The mechanical link between the two is a small script that turns a measured field P75 into the lab ceiling the gate should enforce. Because a cold single-navigation lab run is typically faster than the field P75, you set the lab budget a calibrated margin below the field number, so a lab pass genuinely predicts a healthy field outcome. Here is the derivation, driven by the P75 your aggregation pipeline computes for a mid-range Android device on Fast 3G:

// derive-lab-ceiling.js — set the Lighthouse CI budget from field reality
// FIELD_P75 comes from your RUM aggregation for mid-range Android on Fast 3G.
const FIELD_P75 = { lcp: 3400, cls: 0.09 }; // ms and unitless, P75
const LAB_MARGIN = 0.85; // cold lab run lands ~15% under the field P75

const labCeiling = Object.fromEntries(
  Object.entries(FIELD_P75).map(([metric, p75]) => {
    const value = metric === "cls" ? p75 : Math.round(p75 * LAB_MARGIN);
    return [metric, value];
  }),
);

console.log(JSON.stringify(labCeiling));
// → {"lcp":2890,"cls":0.09}
// LCP gates as an error (lab-faithful); CLS stays a warning until RUM confirms.

You feed that ceiling straight into the assertions block your CI reads on every pull request. The assertions map below fails the build when the lab LCP for the emulated mid-range mobile profile exceeds the derived ceiling, while keeping CLS as a non-blocking warning because its field P75 for mid-range Android on Fast 3G is the number that actually counts:

{
  "ci": {
    "assert": {
      "assertions": {
        "largest-contentful-paint": ["error", { "maxNumericValue": 2890 }],
        "cumulative-layout-shift": ["warn", { "maxNumericValue": 0.1 }],
        "total-byte-weight": ["error", { "maxNumericValue": 1600000 }]
      }
    }
  }
}

To close the loop, recompute FIELD_P75 from your beacon store on a schedule and regenerate the ceiling. This jq pipeline reads a day of collected beacons and prints the P75 LCP for the mobile segment, which becomes the next input to the derivation script:

# field-p75.sh — P75 LCP for mid-range mobile from a day of beacons
jq -s '
  map(select(.device == "mobile" and .metric == "lcp") | .value)
  | sort
  | .[ (length * 0.75) | floor ]
' beacons-2026-07-24.jsonl
# → 3400

Run the lab gate on every commit and the recalibration weekly, and the two signals stop fighting. The lab keeps merges honest in minutes; the field keeps the lab honest over days. That division of labor — deterministic tripwire in front, population oracle behind — is the setup you should reach for, and the same principle scales to the broader Synthetic Monitoring vs RUM: Trade-offs for Budget Gating discussion of which class of signal owns which decision.

Frequently Asked Questions