Comparing Performance Testing Tools

Teams that adopt three performance tools and run them everywhere end up trusting none of them, because each measures something different and they disagree by design. The fix is not to pick one tool forever — it is to know which question each tool answers and route the right tool to the right job. This guide is part of the Lighthouse CI & WebPageTest Integration reference and lays out a head-to-head decision between the three engines that matter: Lighthouse CI for fast deterministic PR gates, WebPageTest for deep connection-controlled diagnostics, and real-user beacons for the ground truth no lab can produce.

The core tension is lab versus field. Lab tools (Lighthouse CI, WebPageTest) give you a reproducible number on demand, perfect for blocking a merge, but they synthesize a single device and connection. Field tools (real-user monitoring) give you the real distribution across every device and network your users actually have, but only after code ships and only as a percentile, never on demand. A mature setup uses lab tools to gate and field tools to keep the lab honest. If you are staffing this decision by headcount rather than by tool, the companion page on choosing a performance testing stack by team size sequences which engine to adopt first.

Lab Versus Field: What Each One Can and Cannot See

Before comparing feature checklists, it helps to draw the boundary that every other difference flows from. A lab tool runs a single scripted load on hardware and a connection you control, so it produces one number you can reproduce and block a merge on. A field tool records what happened on the devices of real visitors, so it produces a distribution you cannot summon on demand and cannot get before code ships. Lighthouse CI and WebPageTest are both lab engines that differ mainly in how faithfully they model the network; RUM beacons are the only field source, and they are the only place a real Interaction to Next Paint (INP) value ever exists.

That boundary is why the three tools disagree and why the disagreement is useful. A lab tool answers "is this build fast on the device we chose?" A field tool answers "were real users fast, at the percentile we care about?" When you set an LCP P75 ceiling of 2500 ms for high-end mobile on a shaped 4G profile, the lab reproduces that one condition deterministically while the field spreads across every condition your audience actually has. Reading them as the same measurement is the mistake; reading the gap between them as a calibration signal is the discipline.

Decision Matrix

The three engines sit on different points of a depth-versus-speed and lab-versus-field plane. The matrix below maps each tool against the dimensions that decide which one to reach for.

The matrix reads as a routing table: gate with the column that is green on speed and gating fit, diagnose with the column that is green on depth and network control, and validate with the column that is green on fidelity. No single column is green everywhere, which is exactly why a complete setup uses all three for different jobs. Notice that both lab engines are red on real-user fidelity — that row is the reason a lab-only program eventually ships a regression its gates never saw, because the lab device is faster than the P75 phone in your field data.

Prerequisites and Environment

Comparing tools fairly means running them against the same target under the same conditions, or the disagreement you see is an artifact of setup, not a real difference.

• A stable staging or preview URL all three can hit. Lighthouse CI and WebPageTest run against it directly; RUM compares against production traffic for the same routes.
• @lhci/cli version 0.13 or newer and Node.js 18 or newer for the Lighthouse path, detailed in Lighthouse CI Configuration & Storage.
• A reachable WebPageTest agent and API key. A shared public instance works for a one-off comparison; a dedicated agent removes location and queue variance, covered in WebPageTest Private Instance Setup.
• A field beacon already deployed so you have a P75 baseline for mid-range mobile to compare lab numbers against, set up through Custom Performance Beacons & RUM.
• One shared budget file — the same thresholds expressed once and imported by every engine, so a disagreement is a real signal rather than a typo.

Fix the emulation before you compare anything. A Lighthouse run defaults to a Moto G-class CPU throttle and a simulated Slow 4G connection; a WebPageTest run uses whatever profile the test script names. If those two do not match, the tools will report different LCP values for reasons that have nothing to do with your code. Pin both to the same device class and connection — the same one your field P75 represents — so the only variable left is the network model each engine uses internally.

Configuration Reference: One Budget, Three Consumers

The trap is encoding the budget three times and drifting. Define it once and let each engine read it. This annotated module is the single source of truth, and every threshold in it is a P75 ceiling for high-end mobile on a shaped 4G profile.

// budget.js — one budget, consumed by every tool
// All ceilings are P75 targets for high-end mobile on a shaped 4G profile.
module.exports = {
  lcp: 2500,           // ms — Largest Contentful Paint, lab + field
  cls: 0.1,            // unitless — Cumulative Layout Shift, lab + field
  tbt: 200,            // ms — lab proxy for INP; INP itself is field-only
  inp: 200,            // ms — field gate, enforced via RUM percentiles
  bytesJs: 200000,     // bytes — script transfer budget on the initial route
  connectivity: "4G",  // WebPageTest connection profile to match the device class
  device: "moto-g",    // CPU throttle class shared by every engine
};

Lighthouse CI maps these into assertions, the WebPageTest script maps them into its comparison thresholds, and the RUM pipeline aggregates field values to P75 and compares against the same lcp, cls, and inp. When all three read this file, "Lighthouse passed but WebPageTest failed" means the engines genuinely saw different conditions — which is the signal you want. The tbt and inp split is deliberate: Total Blocking Time is the best the lab can do, and it is only a proxy, so the real 200 ms INP P75 ceiling for mid-range mobile is enforced downstream in RUM, never in the lab gate.

// lighthouserc.js — the fast gate reads the shared budget
const b = require("./budget.js");
module.exports = {
  ci: {
    collect: { numberOfRuns: 3, settings: { throttlingMethod: "simulate" } },
    assert: {
      assertions: {
        "largest-contentful-paint": ["error", { maxNumericValue: b.lcp }],
        "cumulative-layout-shift": ["error", { maxNumericValue: b.cls }],
        "total-blocking-time": ["error", { maxNumericValue: b.tbt }],
        "resource-summary:script:size": ["error", { maxNumericValue: b.bytesJs }],
      },
    },
    upload: { target: "temporary-public-storage" },
  },
};

Step-by-Step Selection Process

1. State the question. "Will this PR regress performance?" routes to Lighthouse CI. "Why did this route get slower?" routes to WebPageTest. "Are real users actually affected?" routes to RUM. Write the question down before picking a tool. Expected outcome: each task maps to exactly one primary engine.

2. Check the speed budget of the answer. A PR gate must return in under a couple of minutes, which rules out WebPageTest as a blocking check on every commit. Run it:

   time npx lhci autorun --collect.url=http://localhost:3000/

   Expected: under roughly 90 seconds for three runs on a single URL. If you need an answer faster than WebPageTest's multi-minute turnaround, the gate is Lighthouse CI.

3. Check whether network realism matters. If the regression is connection-sensitive — time to first byte, request chains, third-party blocking — only WebPageTest's real shaped connection reproduces it. Confirm the agent responds:

   curl -s "$WPT_SERVER/getLocations.php?f=json" | head -c 200

   Expected: a JSON payload listing your agent location. Empty or error means the agent is unreachable and the comparison would be invalid.

4. Anchor to field truth. Pull the live P75 for the route and compare it to the lab number. If lab and field disagree by more than roughly 15% at the P75 for mid-range mobile, trust the field and recalibrate the lab, not the other way around. The deeper split between what synthetic and field data can each prove is worked through in synthetic monitoring vs RUM tradeoffs.

Capability Comparison

The row that trips teams up most is INP measurement. Both lab engines show a proxy — Total Blocking Time — and TBT correlates with INP only loosely, so a route can pass a 200 ms TBT lab gate for high-end mobile and still post an INP P75 above 200 ms in the field on mid-range Android. That is not a tooling bug; it is the definition of a field-only metric. The contrast between a lab proxy and a real field beacon is drawn out further in Lighthouse CI vs custom RUM beacons, and the field-versus-shaped-lab split specifically for WebPageTest is covered in WebPageTest vs RUM for field data.

Where Each Tool Fits in the Pipeline

Choosing a tool per question is only half the picture; the other half is when in the delivery timeline each one runs. Lighthouse CI belongs on every commit because it is cheap and fast. WebPageTest belongs on a subset — routes labeled network-sensitive, or a nightly sweep — because its multi-minute turnaround cannot sit in the critical path of a merge. RUM runs forever, sampling real sessions and rolling them up to percentiles that arrive hours after ship. Laying them on a single timeline makes the non-overlap obvious: they are not competitors racing for the same slot, they are relay legs.

Extending the right end of that timeline is its own discipline: scheduled synthetic runs and drift alerting that catch regressions no single PR introduced, covered in continuous performance monitoring. A nightly Lighthouse sweep on the top ten routes catches slow creep — a dependency that adds 3% to the JavaScript payload each week until the 200000 byte initial-route budget breaks — that no per-PR gate would ever flag, because each individual PR passed.

Cost and Maintenance Tradeoffs

The sticker price of each tool hides the real cost, which is maintenance. Lighthouse CI is free to run but consumes CI minutes on every commit and demands a low-variance runner, or its numbers wobble and teams start ignoring red. WebPageTest is cheap per test but expensive to keep honest: a shared public agent introduces queue and location variance that makes comparisons meaningless, so a serious program self-hosts an agent and pays for the compute and the babysitting. RUM has no per-test cost at all but the highest fixed cost — an ingest endpoint, a storage layer, and an aggregation pipeline that rolls raw events to P75 and P90 without falling over at traffic peaks.

Weigh those costs against how often each answer changes something. The fast gate earns its keep because it runs constantly and blocks bad merges before they ship — the highest-leverage slot. WebPageTest earns its keep on the rare, expensive regression where a waterfall saves an afternoon of guessing. RUM earns its keep by being the only thing that can tell you the lab budget drifted out of touch with real users. A team that cannot yet afford all three should adopt them in that order — gate, then diagnose, then validate — which is exactly the sequencing the team-size stack guide lays out.

CI Enforcement Snippet

In a complete pipeline the three engines occupy different stages: Lighthouse CI gates every PR, WebPageTest runs on connection-sensitive routes or nightly, and RUM aggregation runs continuously and feeds back into the budget. This workflow wires the two synthetic gates against the shared budget.

name: Performance Gating
on:
  pull_request:
    branches: [main]

jobs:
  fast-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: "20", cache: "npm" }
      - run: npm ci && npm run build
      - name: Lighthouse CI (every PR)
        run: npx lhci autorun
        env:
          LHCI_GITHUB_APP_TOKEN: ${{ secrets.LHCI_GITHUB_APP_TOKEN }}

  deep-gate:
    runs-on: ubuntu-latest
    # only on routes where network realism matters
    if: contains(github.event.pull_request.labels.*.name, 'network-sensitive')
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: "20", cache: "npm" }
      - run: npm ci
      - name: WebPageTest (deep diagnostics)
        run: node wpt-gate.js
        env:
          WPT_SERVER: ${{ secrets.WPT_SERVER }}
          WPT_API_KEY: ${{ secrets.WPT_API_KEY }}

The fast-gate is always required; the deep-gate runs only when a PR is labeled network-sensitive, so WebPageTest's slower turnaround never blocks routine merges. The small Node script the deep gate calls reads the same budget module and fails the job when a shaped-connection LCP crosses the shared ceiling:

// wpt-gate.js — deep gate reads the same budget as the fast gate
const b = require("./budget.js");

async function main() {
  const base = process.env.WPT_SERVER;
  const key = process.env.WPT_API_KEY;
  const url = "https://staging.example.com/checkout";
  const start = await fetch(
    `${base}/runtest.php?f=json&k=${key}&url=${encodeURIComponent(url)}&connectivity=${b.connectivity}&runs=3&fvonly=1`
  ).then((r) => r.json());
  const testId = start.data.testId;
  let result;
  for (let i = 0; i < 30; i++) {
    result = await fetch(`${base}/jsonResult.php?test=${testId}`).then((r) => r.json());
    if (result.statusCode === 200) break;
    await new Promise((r) => setTimeout(r, 5000));
  }
  const lcp = result.data.median.firstView["chromeUserTiming.LargestContentfulPaint"];
  if (lcp > b.lcp) {
    console.error(`WPT LCP ${lcp}ms exceeds P75 ceiling ${b.lcp}ms on ${b.connectivity}`);
    process.exit(1);
  }
  console.log(`WPT LCP ${lcp}ms within budget`);
}

main();

Troubleshooting and Edge Cases

• Lighthouse and WebPageTest disagree on LCP → Lighthouse simulated the network while WebPageTest shaped a real one. For connection-sensitive metrics WebPageTest is authoritative; align the connectivity profile to the device class before concluding either is wrong.
• Lab passes but RUM shows users are slow → the lab device and connection are faster than your real P75 user on mid-range mobile. The lab budget is stale; retighten it against field data through Custom Performance Beacons & RUM.
• RUM looks fine but the lab keeps failing → the runner is noisier than production. Fix the environment — raise numberOfRuns, switch to simulate, isolate the runner — rather than loosening the budget.
• WebPageTest results swing between runs → public-instance queue contention or location variance. Move to a dedicated agent so the connection and hardware are fixed.
• INP regressions slip through lab gates → INP cannot be measured in a lab; both lab tools only proxy it via TBT. Enforce the real INP P75 ceiling of 200 ms for mid-range mobile through RUM percentiles, not the lab.
• All three drift after a dependency bump → a third-party tag got heavier. This is a real regression, not a tooling artifact; track it against Third-Party Script Constraints.
• The deep gate times out in CI → the agent queue is backed up or the polling loop is too short. Raise the retry ceiling, or move WebPageTest to a nightly schedule so it never sits in the merge path at all.

Frequently Asked Questions