Configuring WebPageTest API for Automated Testing

Automated gating needs the WebPageTest API, not the UI: a deterministic submit-poll-parse loop that returns the median metric and a clean exit code CI can act on. This guide drives that loop end to end against the instance you provisioned in WebPageTest Private Instance Setup, so a regression on a controlled connection profile blocks the merge. The work splits into four exact steps — construct a deterministic payload, submit it, poll jsonResult.php until complete, and compare the parsed median against a budget — and each step has a failure mode that silently hangs a pipeline if you skip it.

The whole point of using a private instance over the public grid is repeatability. A shared agent introduces queue jitter and a floating connection profile, which turns a budget gate into a coin flip. When you pin the agent, the connectivity profile, and the run count, the median metric on staging becomes stable enough to gate on — typically within ±5% run-to-run for LCP on a Cable profile against a mid-range desktop agent. Everything below assumes that determinism, and every threshold is stated at a percentile with its device and connection context so the number means the same thing tomorrow.

How the Gate Flows End to End

Before wiring anything, hold the whole loop in your head as one data flow. CI submits a payload to runtest.php and gets back a test ID. It hands that ID to jsonResult.php and polls until the status flips to complete. It reads exactly one number per metric — the median first view — and compares that against two things: a static budget ceiling and a versioned baseline. The exit code is the only thing the pipeline cares about.

API Parameter Reference

The runtest.php endpoint is strict about field types. These are the parameters that matter for a deterministic CI run; omitting location defaults to a shared agent and ruins reproducibility, and firstViewOnly roughly halves execution time while preserving Core Web Vitals. Treat every field as load-bearing — a single wrong value silently changes what the median means, and the gate downstream cannot tell the difference between a real regression and a mislabeled connection profile.

The connectivity profile is the single biggest lever on the numbers. A checkout page that lands LCP at 1820 ms P75 on Cable against a desktop agent can land at 4100 ms P75 on 3G against a mid-range mobile agent — the same code, a different budget entirely. That is why device and network belong in the budget definition itself, a split covered in Device and Network Emulation Weighting. Pick one profile per gate and never mix them in the same threshold map.

Diagnostic Steps

Validate the API key with a cheap call before submitting any test — distinguish 401 (bad key) from 403 (rate-limited or IP-blocked).

curl -s -o /dev/null -w "%{http_code}\n" \
  -H "X-API-Key: $WPT_API_KEY" \
  "$WPT_BASE_URL/getLocations.php?f=json"
# Expected: 200

Confirm the location you intend to submit to actually has idle agents, so the test does not silently queue.

curl -s "$WPT_BASE_URL/getLocations.php?f=json" \
  | jq '.data[] | {location: .id, pending: .PendingTests.Total}'
# Expected: your location present with a low pending count

If pending climbs into double digits, you are about to submit into a backlog. On a private pool sized for CI, keep concurrent submissions per key at or below the agent count so no test waits behind more than one other. A queued test still counts against your poll timeout, so a saturated location is indistinguishable from a broken one once the loop starts.

Implementation

This script submits one test, polls jsonResult.php every 5 seconds with a hard 180-second timeout (so a stuck agent cannot hang CI), then extracts the median first-view metrics and compares each against a threshold map plus a regression check against a versioned baseline.

#!/usr/bin/env bash
set -euo pipefail

# 1. Submit
ID=$(curl -s -X POST "$WPT_BASE_URL/runtest.php" \
  -H "X-API-Key: $WPT_API_KEY" -H "Content-Type: application/json" \
  -d '{"url":"https://staging.example.com/checkout?wpt_bypass=1","location":"us-east-1:Chrome.Cable","runs":3,"firstViewOnly":true,"connectivity":"Cable","f":"json"}' \
  | jq -r '.data.id')
[ -n "$ID" ] && [ "$ID" != "null" ] || { echo "Submit failed: no test id"; exit 1; }
echo "Submitted test $ID"

# 2. Poll (5s interval, 180s hard timeout)
for i in $(seq 1 36); do
  RESPONSE=$(curl -s "$WPT_BASE_URL/jsonResult.php?test=$ID&f=json")
  STATUS=$(echo "$RESPONSE" | jq -r '.statusCode')
  [ "$STATUS" = "200" ] && break
  [ "$STATUS" = "100" ] && { echo "in progress ($((i*5))s)"; sleep 5; continue; }
  [ "$STATUS" = "101" ] && { echo "queued ($((i*5))s)"; sleep 5; continue; }
  echo "Unexpected status $STATUS"; exit 1
done
[ "$STATUS" = "200" ] || { echo "Timeout after 180s"; exit 1; }

# 3. Parse the median first view
LCP=$(echo "$RESPONSE" | jq -r '.data.median.firstView.LargestContentfulPaint')
CLS=$(echo "$RESPONSE" | jq -r '.data.median.firstView.CumulativeLayoutShift')
SI=$(echo "$RESPONSE"  | jq -r '.data.median.firstView.SpeedIndex')

# 4. Budget + regression gate
declare -A T=( ["LCP"]=2500 ["CLS"]=0.1 ["SpeedIndex"]=2500 )
(( $(echo "$LCP > ${T[LCP]}" | bc -l) )) && { echo "FAIL LCP $LCP > ${T[LCP]}"; exit 2; }
(( $(echo "$CLS > ${T[CLS]}" | bc -l) )) && { echo "FAIL CLS $CLS > ${T[CLS]}"; exit 2; }
(( $(echo "$SI  > ${T[SpeedIndex]}" | bc -l) )) && { echo "FAIL SpeedIndex $SI > ${T[SpeedIndex]}"; exit 2; }

BASELINE_LCP=$(jq -r '.LCP' baseline.json)
(( $(echo "$LCP > ($BASELINE_LCP * 1.1)" | bc -l) )) && { echo "REGRESSION: LCP +10% vs baseline"; exit 2; }

echo "All budgets passed (LCP ${LCP}ms, CLS ${CLS}, SI ${SI}ms)."
exit 0

Reserve exit 0 for pass, exit 1 for non-blocking warnings, and exit 2 for a hard fail that blocks the PR. Keeping infrastructure failures (exit 1) distinct from budget breaches (exit 2) is what lets you page the platform team for the former and the authoring team for the latter without reading logs.

The Poll Loop as a State Machine

The poll loop looks trivial but it has three exits, and every one of them must be reachable or CI hangs. The status field is authoritative: 100/101 mean keep waiting, 200 means parse, anything else means abort immediately rather than burning the full timeout. The iteration cap is the fourth exit — after 36 polls the loop leaves without a 200 and the guard turns that into a timeout failure.

CI Gating Assertion

Wire the script into a job so its exit code becomes a required status check. The thresholds enforced here — LCP at or below 2500 ms, CLS at or below 0.1, Speed Index at or below 2500 ms at the P75 of the Cable profile on a desktop agent — must match the budget your team agreed. If you gate mobile separately, run a second job with its own location and a looser ceiling; a single map cannot serve two device classes honestly.

- name: WebPageTest budget gate
  run: ./scripts/wpt-gate.sh
  env:
    WPT_BASE_URL: ${{ secrets.WPT_SERVER }}
    WPT_API_KEY: ${{ secrets.WPT_API_KEY }}
# Exit 2 from the script fails this step and blocks the merge when the check is
# required in branch protection. Stagger across a matrix at max 5 concurrent
# submissions per key to avoid queue saturation.

The +10% regression clause is the other half of the gate, and it needs a trustworthy baseline.json. Never promote a baseline from a red build or a one-off run — derive it from the rolling median of green builds on main, the practice detailed in Historical Baseline Calibration. A static ceiling catches absolute breaches; the baseline clause catches slow drift that stays under the ceiling for months.

Reading the Median Against the Budget

The gate compares one number, but you should understand the shape behind it. With runs: 3 WebPageTest returns the run whose Speed Index is the median, and its LCP and CLS come from that same run — not a per-metric median. That coupling matters: a single slow run can drag the reported LCP even when two of three runs were healthy. The chart below shows a passing checkout page, both timing metrics comfortably under the 2500 ms Cable-profile ceiling.

Because the median rides on run-to-run variance, choosing the right percentile and run count is its own discipline — one runs: 1 sample is noisy enough to false-fail a gate. Widening to 3 or 5 runs and gating on the P75 across them is the safer default, and the trade-off between P75 and P90 targets is worked through in Percentile-Based Threshold Tuning.

Verification

Confirm the loop works before trusting the gate. A passing run prints the parsed median line and exits 0; a budget breach prints the failing metric and exits 2.

./scripts/wpt-gate.sh; echo "exit=$?"
# Healthy pass:
#   Submitted test 240620_AB_1234
#   All budgets passed (LCP 1820ms, CLS 0.04, SI 1900ms).
#   exit=0
# Hard fail:
#   FAIL LCP 2740 > 2500
#   exit=2

Check three things: the submit step returns a non-empty test ID, the poll exits within the 180-second timeout rather than hanging, and the exit code matches the budget verdict. To cut redundant API calls on unchanged commits, hash the URL plus payload (sha256sum) and skip submission when a cached result for that hash already exists. Run the gate against a deliberately slowed page once — throttle a hero image or inject a blocking script — to confirm exit 2 actually fires; a gate that has never failed in testing is a gate you cannot trust to fail in production.

Frequently Asked Questions