Lighthouse CI vs WebPageTest Decision Guide

Both Lighthouse CI and WebPageTest run synthetic audits in a controlled environment, which is exactly why teams waste time arguing over which one to standardize on — they are not substitutes, they answer different questions. This guide, part of the Comparing Performance Testing Tools reference, makes the choice concrete: reach for Lighthouse CI when you need a fast deterministic verdict to gate a merge, and reach for WebPageTest when you need to explain a regression through its filmstrip, request waterfall, and connection view. The decision is not "which tool is better" — it is "what does this task need."

The split comes down to turnaround and depth. Lighthouse CI returns a median in seconds and fits inside a required status check on every commit. WebPageTest takes minutes per run but shapes a real connection and captures frame-by-frame rendering and per-request timing that Lighthouse never exposes. Pick by which of those you need right now, and wire both so the fast tool guards the merge while the deep tool waits on standby for the runs that fail. If your question is instead lab-versus-field rather than lab-versus-lab, pair this with Synthetic Monitoring vs RUM: Trade-offs for Budget Gating, which covers when neither synthetic tool is the right answer.

What Each Tool Measures

The table is the whole decision. If you scan the "fits a per-PR gate" row, Lighthouse CI is the gate; if you scan the filmstrip, waterfall, and connection-view rows, WebPageTest is the diagnostic. The decision tree below collapses those rows into two questions you can ask about any task in front of you.

Diagnostic: Mapping What Each Sees

Run both against the same URL and the difference becomes obvious. Lighthouse CI gives a category score and metric values:

npx lhci collect --url=https://staging.example.com/ --numberOfRuns=3
cat .lighthouseci/lhr-*.json | npx json 'audits["largest-contentful-paint"].numericValue'

Expected output — a single number per run, e.g. 2410. That is the verdict: a value to compare against the budget. It does not tell you which request delayed LCP. On the default Lighthouse mobile preset (mid-range mobile, simulated Slow 4G, 4x CPU throttle), an LCP of 2410 clears a P75 ceiling of 2500 ms for that device class, so the gate would pass.

WebPageTest answers the "which request" question. Submit the same URL and pull the waterfall and connection breakdown:

curl -s "$WPT_SERVER/runtest.php?url=https://staging.example.com/&k=$WPT_API_KEY&f=json&runs=3&connectivity=4G&fvonly=1"
# poll the returned testId, then:
curl -s "$WPT_SERVER/jsonResult.php?test=$TEST_ID" | npx json 'data.median.firstView.LargestContentfulPaint'

Expected: the same metric (e.g. 2780) plus a requests array, breakdown by content type, and a filmstrip URL. The 10–20% gap between 2410 and 2780 is the simulated-versus-shaped-network difference, not a bug — and only WebPageTest shows you which host's TLS handshake cost the difference. That 2780 measured on a shaped 4G profile is a different device-and-connection context than the simulated 2410, which is exactly why you never compare the two numbers without first stating the profile behind each.

Implementation: Running Both

Run Lighthouse CI as the always-on gate and WebPageTest as an on-demand diagnostic that maps its metrics back to the same budget. The topology is a fast guard on the merge path with a slow, deep branch that only fires when the guard fails or when a nightly job wants root cause on a suspect route.

This script submits a URL to WebPageTest and prints the same fields Lighthouse asserts, so the two are directly comparable.

// wpt-compare.js — fetch the metrics Lighthouse gates, from WebPageTest
const WebPageTest = require("webpagetest");
const wpt = new WebPageTest(process.env.WPT_SERVER, process.env.WPT_API_KEY);

const URL = process.argv[2];

wpt.runTest(
  URL,
  { connectivity: "4G", runs: 3, fvonly: true, pollResults: 5, timeout: 300 },
  (err, result) => {
    if (err) { console.error(err); process.exit(2); }
    const m = result.data.median.firstView;
    console.log(JSON.stringify({
      lcp: m["chromeUserTiming.LargestContentfulPaint"],
      cls: m["chromeUserTiming.CumulativeLayoutShift"],
      tbt: m.TotalBlockingTime,
      bytesJs: m.breakdown.js.bytes,
      filmstrip: result.data.median.firstView.videoFrames ? "captured" : "none",
    }, null, 2));
  }
);

chromeUserTiming.LargestContentfulPaint is WebPageTest's equivalent of Lighthouse's largest-contentful-paint.numericValue, and TotalBlockingTime maps directly to Lighthouse's total-blocking-time. Mapping the field names is what makes a cross-tool comparison meaningful rather than apples-to-oranges. Keep the mapping in one place so a rename never silently compares the wrong pair of numbers.

CI Assertion Comparison

The same budget is enforced differently by each. Lighthouse CI asserts declaratively in lighthouserc.json. The ceilings below are P75 targets for the default mid-range mobile, simulated Slow 4G preset — if you gate desktop separately, keep a second config with its own numbers rather than reusing these mobile ceilings:

{
  "ci": {
    "assert": {
      "assertions": {
        "largest-contentful-paint": ["error", { "maxNumericValue": 2500 }],
        "total-blocking-time": ["error", { "maxNumericValue": 200 }],
        "cumulative-layout-shift": ["error", { "maxNumericValue": 0.1 }]
      }
    }
  }
}

WebPageTest has no built-in assertion engine, so the gate is the exit code of the comparison script:

// append to wpt-compare.js to gate
const BUDGET = { lcp: 2500, tbt: 200, cls: 0.1 };
const breach = Object.entries(BUDGET).some(([k, max]) => current[k] > max);
process.exit(breach ? 1 : 0);

Lighthouse CI gives you assertions for free; WebPageTest gives you depth but you write the assertion yourself. On a per-commit gate that difference alone favors Lighthouse CI. Team size sharpens the choice further — a two-person team that cannot staff a private WebPageTest agent should lean almost entirely on the hosted Lighthouse gate, a trade-off spelled out in Choosing a Performance Testing Stack by Team Size.

Matching Emulation So the Numbers Agree

The single biggest source of "the two tools disagree" tickets is mismatched emulation. Lighthouse's default mobile run simulates Slow 4G with a 4x CPU slowdown; WebPageTest's connectivity=4G shapes a faster line and applies no CPU throttle unless you set one. Comparing those two is comparing different device-and-connection contexts, and the LCP gap that results is arithmetic, not a regression. Before you trust any cross-tool delta, pin both sides to the same profile: match WebPageTest's connectivity, latency, and throughput to the Lighthouse throttling block, and add a CPU multiplier on the WebPageTest agent so both apply the same processor penalty.

The chart makes the untuned gap visible. Two runs of the same build, same URL, report LCP that straddles the P75 ceiling only because the network models differ — Lighthouse's simulated Slow 4G reads under budget while WebPageTest's shaped 4G reads over it.

Once the profiles agree and the gap persists, the breach is genuine and belongs to the shaped connection. For calibrating the CPU multiplier and network profile so a CI runner reproduces field conditions, see Device and Network Emulation Weighting.

Verification: When to Trust Which

After running both, decide which number to believe by which question you asked. For a merge gate, trust Lighthouse CI — it is deterministic and fast, and its simulated network is consistent across every PR. For a root-cause investigation or a connection-sensitive metric (TTFB, request chains, third-party blocking), trust WebPageTest, because its shaped real connection reproduces conditions Lighthouse only approximates.

A passing Lighthouse gate looks like:

✅  assertions passed for https://staging.example.com/
Done running Lighthouse!

A WebPageTest comparison that confirms or contradicts it prints the mapped metrics and an exit code. If Lighthouse passes but WebPageTest's shaped-4G LCP breaches at P75, and you have already matched the emulation profiles, the regression is connection-sensitive and real — Lighthouse's simulation masked it, and the budget should be enforced against WebPageTest for that route. To remove location and queue variance from WebPageTest before trusting it as authoritative, run it on a dedicated agent per WebPageTest Private Instance Setup, and keep the Lighthouse side configured per Lighthouse CI Configuration and Storage.

Frequently Asked Questions