Lighthouse CI vs WebPageTest Decision Guide #

Both Lighthouse CI and WebPageTest run synthetic audits in a controlled environment, which is exactly why teams waste time arguing over which one to standardize on — they are not substitutes, they answer different questions. This guide, part of the Comparing Performance Testing Tools reference, makes the choice concrete: reach for Lighthouse CI when you need a fast deterministic verdict to gate a merge, and reach for WebPageTest when you need to explain a regression through its filmstrip, request waterfall, and connection view. The decision is not "which tool is better" — it is "what does this task need."

The split comes down to turnaround and depth. Lighthouse CI returns a median in seconds and fits inside a required status check on every commit. WebPageTest takes minutes per run but shapes a real connection and captures frame-by-frame rendering and per-request timing that Lighthouse never exposes. Pick by which of those you need right now.

What Each Tool Measures #

The table is the whole decision. If you scan the "fits a per-PR gate" row, Lighthouse CI is the gate; if you scan the filmstrip, waterfall, and connection-view rows, WebPageTest is the diagnostic.

Diagnostic: Mapping What Each Sees #

Run both against the same URL and the difference becomes obvious. Lighthouse CI gives a category score and metric values:

npx lhci collect --url=https://staging.example.com/ --numberOfRuns=3
cat .lighthouseci/lhr-*.json | npx json 'audits["largest-contentful-paint"].numericValue'

Expected output — a single number per run, e.g. 2410. That is the verdict: a value to compare against the budget. It does not tell you which request delayed LCP.

WebPageTest answers the "which request" question. Submit the same URL and pull the waterfall and connection breakdown:

curl -s "$WPT_SERVER/runtest.php?url=https://staging.example.com/&k=$WPT_API_KEY&f=json&runs=3&connectivity=4G&fvonly=1"
# poll the returned testId, then:
curl -s "$WPT_SERVER/jsonResult.php?test=$TEST_ID" | npx json 'data.median.firstView.LargestContentfulPaint'

Expected: the same metric (e.g. 2780) plus a requests array, breakdown by content type, and a filmstrip URL. The 10–20% gap between 2410 and 2780 is the simulated-versus-shaped-network difference, not a bug — and only WebPageTest shows you which host's TLS handshake cost the difference.

Implementation: Running Both #

Run Lighthouse CI as the always-on gate and WebPageTest as an on-demand diagnostic that maps its metrics back to the same budget. This script submits a URL to WebPageTest and prints the same fields Lighthouse asserts, so the two are directly comparable.

// wpt-compare.js — fetch the metrics Lighthouse gates, from WebPageTest
const WebPageTest = require("webpagetest");
const wpt = new WebPageTest(process.env.WPT_SERVER, process.env.WPT_API_KEY);

const URL = process.argv[2];

wpt.runTest(
  URL,
  { connectivity: "4G", runs: 3, fvonly: true, pollResults: 5, timeout: 300 },
  (err, result) => {
    if (err) { console.error(err); process.exit(2); }
    const m = result.data.median.firstView;
    console.log(JSON.stringify({
      lcp: m["chromeUserTiming.LargestContentfulPaint"],
      cls: m["chromeUserTiming.CumulativeLayoutShift"],
      tbt: m.TotalBlockingTime,
      bytesJs: m.breakdown.js.bytes,
      filmstrip: result.data.median.firstView.videoFrames ? "captured" : "none",
    }, null, 2));
  }
);

chromeUserTiming.LargestContentfulPaint is WebPageTest's equivalent of Lighthouse's largest-contentful-paint.numericValue, and TotalBlockingTime maps directly to Lighthouse's total-blocking-time. Mapping the field names is what makes a cross-tool comparison meaningful rather than apples-to-oranges.

CI Assertion Comparison #

The same budget is enforced differently by each. Lighthouse CI asserts declaratively in lighthouserc.json:

{
  "ci": {
    "assert": {
      "assertions": {
        "largest-contentful-paint": ["error", { "maxNumericValue": 2500 }],
        "total-blocking-time": ["error", { "maxNumericValue": 200 }],
        "cumulative-layout-shift": ["error", { "maxNumericValue": 0.1 }]
      }
    }
  }
}

WebPageTest has no built-in assertion engine, so the gate is the exit code of the comparison script:

// append to wpt-compare.js to gate
const BUDGET = { lcp: 2500, tbt: 200, cls: 0.1 };
const breach = Object.entries(BUDGET).some(([k, max]) => current[k] > max);
process.exit(breach ? 1 : 0);

Lighthouse CI gives you assertions for free; WebPageTest gives you depth but you write the assertion yourself. On a per-commit gate that difference alone favors Lighthouse CI.

Verification: When to Trust Which #

After running both, decide which number to believe by which question you asked. For a merge gate, trust Lighthouse CI — it is deterministic and fast, and its simulated network is consistent across every PR. For a root-cause investigation or a connection-sensitive metric (TTFB, request chains, third-party blocking), trust WebPageTest, because its shaped real connection reproduces conditions Lighthouse only approximates.

A passing Lighthouse gate looks like:

✅  assertions passed for https://staging.example.com/
Done running Lighthouse!

A WebPageTest comparison that confirms or contradicts it prints the mapped metrics and an exit code. If Lighthouse passes but WebPageTest's shaped-4G LCP breaches, the regression is connection-sensitive and real — Lighthouse's simulation masked it, and the budget should be enforced against WebPageTest for that route. To remove location and queue variance from WebPageTest before trusting it as authoritative, run it on a dedicated agent per WebPageTest Private Instance Setup, and keep the Lighthouse side configured per Lighthouse CI Configuration & Storage.

Frequently Asked Questions #