Benchmarking Competitors With WebPageTest

A performance budget calibrated only against your own history answers one question — "are we getting slower?" — but never the question your product leadership actually asks: "are we losing the race?" This how-to is part of the Continuous Performance Monitoring reference, and it shows you how to run scheduled WebPageTest API tests against a fixed set of competitor URLs, extract their Largest Contentful Paint, Cumulative Layout Shift, and Total Blocking Time, store the series, chart it, and then use your relative standing to set targets that move with the field instead of drifting on their own.

Competitor benchmarking sits alongside your internal synthetic runs, not instead of them. Your PR gates and nightly runs protect against self-inflicted regressions; the competitor series tells you whether a ceiling you set eighteen months ago is still competitive. When three rivals ship an edge-rendered rewrite and their mobile LCP P75 drops from 3800 ms to 2400 ms on an emulated mid-range Android over a throttled 4G profile, a budget frozen at 4000 ms is now the slowest ceiling in the market — and no internal alert will ever tell you that.

Why Anchor Budgets to the Competitive Field

Field data from your own users, discussed under Core Web Vitals budget allocation, tells you where you stand with the people who already chose you. It cannot tell you what the visitor comparing three checkout flows in three browser tabs experiences on the other two. WebPageTest gives you a lab you can point at any public URL, so it is the natural instrument for measuring sites you do not own.

The output of this workflow is a single comparative fact you can defend in a planning meeting: "our mobile LCP P75 is 3600 ms; the median of our five tracked competitors is 2900 ms; two of them are under 2500 ms." That sentence turns an abstract budget number into a ranking, and rankings drive prioritisation far better than absolute thresholds do.

Scheduling WebPageTest API Runs Against Competitor URLs

Start from a working API client. If you have not already provisioned a key and confirmed you can submit and poll tests, follow Configuring WebPageTest API for Automated Testing first — everything below assumes a WPT_API_KEY and, ideally, a private instance so you are not throttled by the public queue.

Keep the competitor list in version control as data, not code, so a product manager can add a rival without a deploy. Pin the test location, device profile, connection profile, and run count for every URL, because a benchmark is only comparable if every site is measured under identical emulation. Nine runs per URL gives you a stable median without exhausting the queue.

# competitors.yml — the tracked field, edited by anyone
location: "ec2-eu-west-1:Chrome"
connectivity: "4G"          # 9 Mbps down, 170 ms RTT
device: "Motorola G (gen 4)"
runs: 9
firstViewOnly: true
targets:
  - key: "our-site"
    label: "Our checkout"
    url: "https://www.example.com/checkout"
  - key: "rival-a"
    label: "Rival A"
    url: "https://www.rival-a.com/cart"
  - key: "rival-b"
    label: "Rival B"
    url: "https://shop.rival-b.com/basket"
  - key: "rival-c"
    label: "Rival C"
    url: "https://www.rival-c.com/checkout"

Submitting a test returns a JSON test ID; you then poll the result endpoint until the run completes. WebPageTest computes percentile-style summary values across your run count, so requesting the median view gives you the representative number per metric rather than a lucky-fast first paint. The scheduler below submits every target, waits, and collects the median first-view metrics.

// benchmark.js — submit competitor URLs and poll for medians
import fs from "node:fs";
import yaml from "js-yaml";

const KEY = process.env.WPT_API_KEY;
const HOST = process.env.WPT_HOST || "https://www.webpagetest.org";
const cfg = yaml.load(fs.readFileSync("competitors.yml", "utf8"));

async function submit(url) {
  const params = new URLSearchParams({
    url, k: KEY, f: "json", runs: String(cfg.runs),
    location: `${cfg.location}.${cfg.connectivity}`,
    mobile: "1", mobileDevice: cfg.device, fvonly: "1"
  });
  const res = await fetch(`${HOST}/runtest.php?${params}`);
  const body = await res.json();
  if (body.statusCode !== 200) throw new Error(body.statusText);
  return body.data.testId;
}

async function poll(testId) {
  for (let attempt = 0; attempt < 60; attempt++) {
    const res = await fetch(`${HOST}/jsonResult.php?test=${testId}&k=${KEY}`);
    const body = await res.json();
    if (body.statusCode === 200) return body.data;
    if (body.statusCode >= 400) throw new Error(body.statusText);
    await new Promise(r => setTimeout(r, 15000));
  }
  throw new Error(`timed out waiting for ${testId}`);
}

const results = [];
for (const t of cfg.targets) {
  const testId = await submit(t.url);
  const data = await poll(testId);
  results.push({ key: t.key, label: t.label, data });
}
fs.writeFileSync("raw-results.json", JSON.stringify(results));

Wrap this in a scheduled job — a GitHub Actions schedule trigger or any cron runner — that fires once a day at a fixed hour. A fixed hour matters: measuring your rivals at 06:00 UTC every day removes time-of-day CDN and origin-load variance from the comparison. The same discipline underpins scheduling nightly Lighthouse runs for your own pages.

Extracting LCP, CLS, and TBT From the Result JSON

The WebPageTest median result nests the numbers you want under data.median.firstView. Largest Contentful Paint and Total Blocking Time arrive in milliseconds; Cumulative Layout Shift is a unitless score. Pull exactly those three, tag them with the run timestamp and the target key, and discard the rest of the multi-megabyte payload so your storage stays lean.

// extract.js — reduce each raw result to three metrics + metadata
import fs from "node:fs";
const raw = JSON.parse(fs.readFileSync("raw-results.json", "utf8"));
const capturedAt = new Date().toISOString();

const rows = raw.map(({ key, label, data }) => {
  const fv = data.median.firstView;
  return {
    captured_at: capturedAt,
    target_key: key,
    label,
    lcp_ms: Math.round(fv["chromeUserTiming.LargestContentfulPaint"] ?? fv.LargestContentfulPaint),
    cls: Number((fv["chromeUserTiming.CumulativeLayoutShift"] ?? fv.CumulativeLayoutShift).toFixed(3)),
    tbt_ms: Math.round(fv.TotalBlockingTime),
    run_count: data.runs ?? null
  };
});
fs.writeFileSync("rows.json", JSON.stringify(rows, null, 2));
console.table(rows.map(r => ({ site: r.label, lcp: r.lcp_ms, cls: r.cls, tbt: r.tbt_ms })));

Storing and Charting the Series

A single row per target per run day is all you need. A narrow table keeps queries simple and lets you compute the field median on the fly. Any SQL store works; the schema below is deliberately minimal.

CREATE TABLE competitor_metrics (
  captured_at TIMESTAMPTZ NOT NULL,
  target_key  TEXT        NOT NULL,
  label       TEXT        NOT NULL,
  lcp_ms      INTEGER     NOT NULL,
  cls         NUMERIC(5,3) NOT NULL,
  tbt_ms      INTEGER     NOT NULL,
  PRIMARY KEY (captured_at, target_key)
);

-- Latest standing: you against the field median for LCP
SELECT label, lcp_ms,
       lcp_ms - (SELECT percentile_cont(0.5) WITHIN GROUP (ORDER BY lcp_ms)
                 FROM competitor_metrics
                 WHERE captured_at = (SELECT MAX(captured_at) FROM competitor_metrics)
                   AND target_key <> 'our-site') AS delta_vs_field_ms
FROM competitor_metrics
WHERE captured_at = (SELECT MAX(captured_at) FROM competitor_metrics)
ORDER BY lcp_ms;

Once the rows accumulate, chart your line against the field. The most useful single view plots your metric and the competitor median over time so a converging or diverging gap is obvious at a glance. The same Grafana patterns used for visualizing budget trends with Grafana apply here — point one panel at this table instead of your internal metrics store. The snapshot below shows a representative field on an emulated mid-range Android over the 4G profile.

Caveats That Will Corrupt the Comparison

A competitor benchmark is only as trustworthy as the conditions you control for, and public sites are hostile measurement targets. Treat the following as first-class threats to validity, not footnotes.

Their A/B tests. A rival running a homepage experiment may serve your synthetic agent a variant that half their real users never see. A single median across nine runs blurs this, but a bimodal LCP distribution — clustered at 2400 ms and 3600 ms on the same device profile — is a tell. Log the raw per-run values, not just the median, so you can spot bimodality and flag that day's number as unreliable.

Geo and location skew. Measure every competitor from the same WebPageTest location you measure yourself from. A rival whose origin sits in the same region as your test node will look artificially fast; testing from a location central to your actual customer base keeps the ranking honest.

Cookie walls and consent interstitials. In regions with mandatory consent, the first view a synthetic agent sees is often a full-screen banner, so the "LCP element" is the consent modal rather than the product content. Detect this by checking whether the LCP element selector in the WebPageTest result points at a consent container, and exclude those runs.

Turning Relative Standing Into Targets

The point of the series is to set a target, not to admire a chart. The durable rule is to budget to the field, not to a fixed number. A good competitive LCP target is "beat the field median by 10 percent" rather than a static ceiling: if the median of your tracked rivals on a mid-range Android over the 4G profile is 2900 ms at P75, your target becomes 2610 ms, and it tightens automatically as the field improves.

Convert the standing into an explicit budget the same way you would any other ceiling, then feed it into your percentile-based tuning. The practices in percentile-based threshold tuning turn "10 percent under the field median" into a concrete P75 number your gates can enforce, and pairing this with alerting on synthetic performance drift means you are notified both when you regress and when the field pulls ahead of you.

Keep the cadence honest. Recompute the target monthly, not on every run, so a single noisy competitor day never yanks your budget. A month of daily medians gives roughly 30 samples per rival, which is enough to compute a stable field-median P75 without over-reacting to one bad afternoon on someone else's origin.

Verification

Before you trust a competitor number in a planning deck, confirm the pipeline end to end. Run the checks below and treat any failure as a reason to quarantine that day's data.

A quick smoke test confirms the freshness and completeness checks in one query, which you can wire into the same job that publishes the chart.

SELECT captured_at::date AS day,
       COUNT(*)                                    AS targets,
       COUNT(*) FILTER (WHERE target_key = 'our-site') AS has_us,
       MIN(lcp_ms) AS best_lcp, MAX(lcp_ms) AS worst_lcp
FROM competitor_metrics
WHERE captured_at >= now() - interval '2 days'
GROUP BY 1
ORDER BY 1 DESC;

If targets is short a row or has_us is zero, a submission failed or your own URL fell out of the config, and the day's ranking is not safe to publish. Re-run the job before you circulate the number.

Frequently Asked Questions