Calibrating CPU Throttling for CI Runners

A cpuSlowdownMultiplier of 4 is correct on exactly one machine: the one Lighthouse's default was tuned against. On a faster runner, a 4× multiplier under-throttles and your Total Blocking Time looks artificially good; on a slower or noisier runner, the same 4× over-throttles and the gate flaps red on builds that are actually fine. The multiplier is a ratio, and a fixed ratio applied to variable hardware produces a moving emulation target. This guide — part of the Device & Network Emulation Weighting reference — replaces the constant with a multiplier computed at runtime from the runner's measured CPU speed, so every run emulates the same mid-range device regardless of which machine GitHub hands you.

The fix hinges on one number Lighthouse already reports: BenchmarkIndex, an unitless score of how fast the host CPU executed a fixed workload during the run. Higher means faster. Calibrate once against a target device's BenchmarkIndex, then scale the multiplier on every run to hit that target. Because the same emulated device underpins every threshold you gate on, this is upstream of choosing between P75 and P90 budget targets — a wandering CPU emulation makes any percentile you pick meaningless.

Why the Ratio Drifts

The default 4× comes from an assumption baked into Lighthouse years ago: the reference desktop was roughly 4× faster than the mid-range phone it wanted to emulate. That relationship only holds if the reference desktop is the machine you run on. A modern hosted runner is faster than that reference, so 4× no longer lands the emulated CPU on a mid-range phone — it lands it somewhere faster, and TBT at P75 on that emulated device flatters your build. The core problem is that the multiplier encodes a difference between two machines, but you only control one of them.

Two independent sources of drift stack on top of each other. Hardware drift comes from GitHub silently rotating the underlying instance type, or from a self-hosted fleet mixing 4-vCPU and 8-vCPU boxes. Load drift comes from noisy neighbours: a shared runner executing a heavy webpack build in a parallel job runs the Lighthouse CPU workload slower than the same runner idle. A fixed multiplier corrects for neither. A multiplier derived from the live BenchmarkIndex corrects for both, because the index is measured during the run and already contains whatever slowdown the current hardware and current load impose.

Runner-vs-Device CPU Benchmark Reference

The table maps representative hardware to its typical BenchmarkIndex and shows the multiplier needed to emulate a mid-range phone (target BenchmarkIndex ≈ 700, roughly a Moto G4 on Fast 3G). The multiplier is simply the runner's index divided by the target.

The rightmost column is the point: a fixed 4× is wrong almost everywhere, and how wrong depends on load, so it drifts build to build. A multiplier derived from the live BenchmarkIndex holds the emulated device constant even as the runner's real speed varies. The chart below plots the required multiplier per host against the naive fixed line so the gap is visible at a glance.

Diagnostic Steps

1. Read the BenchmarkIndex from any existing report. It lives in the run's environment block.

   npx lighthouse https://staging.example.com --only-categories=performance \
     --output=json --output-path=./run.json --quiet
   node -e "console.log('BenchmarkIndex:', require('./run.json').environment.benchmarkIndex)"

   Example output: BenchmarkIndex: 1287 — a fast 4-vCPU hosted runner, far above the 700 target.

2. Sample it across several runs to see how much the runner's speed varies under CI load. A spread wider than ±15% means the runner is contended and needs pinning before calibration is meaningful.

   for i in 1 2 3; do
     npx lighthouse https://staging.example.com --only-categories=performance \
       --output=json --output-path=./b$i.json --quiet
     node -e "console.log(require('./b$i.json').environment.benchmarkIndex)"
   done

   Example output: 1290, 1305, 1180 — a 10% spread, acceptable; cap outliers per the variance protocol below.

Implementation

Compute the multiplier dynamically from a calibration run's BenchmarkIndex, then feed it into the throttling config. The target index defines the device you are emulating; clamp the result so a wildly off runner cannot produce an absurd multiplier.

// dynamic-cpu-throttle.js — derive cpuSlowdownMultiplier from runner speed.
const { execSync } = require("child_process");
const fs = require("fs");

const TARGET_BENCHMARK_INDEX = 700; // mid-range phone (≈ Moto G4)
const URL = process.env.CALIBRATION_URL || "https://staging.example.com";

// One quick calibration run to read this runner's CPU speed.
execSync(
  `npx lighthouse ${URL} --only-categories=performance ` +
  `--output=json --output-path=./calib.json --quiet ` +
  `--chrome-flags="--headless --no-sandbox --disable-dev-shm-usage"`,
  { stdio: "inherit" }
);

const { benchmarkIndex } = JSON.parse(
  fs.readFileSync("./calib.json", "utf8")).environment;

// Faster runner (higher index) needs a larger multiplier to feel as slow
// as the target device. Clamp to a sane range.
const raw = benchmarkIndex / TARGET_BENCHMARK_INDEX;
const multiplier = Math.min(Math.max(raw, 1), 8);

console.log(`BenchmarkIndex ${benchmarkIndex} → cpuSlowdownMultiplier ${multiplier.toFixed(2)}`);
fs.writeFileSync("./cpu-throttle.json",
  JSON.stringify({ cpuSlowdownMultiplier: Number(multiplier.toFixed(2)) }, null, 2));

Run this as the first step of the performance job; the emitted cpu-throttle.json is then merged into the Lighthouse throttling settings so the gated run uses the calibrated multiplier instead of the static 4×. The calibration run only needs to touch the CPU, so --only-categories=performance keeps it fast; you are reading the environment block, not the scores. The data flow from raw index to gated run is the pipeline below.

Choosing the Target Index

The target BenchmarkIndex is a policy decision, not a constant. A value of ~700 emulates a mid-range Android phone (Moto G4 class) on Fast 3G, which is a reasonable worst-case for a global consumer audience whose P75 device is a budget handset. If your field data shows your P75 visitor is on a faster phone — for instance a mid-tier device on a 4G connection — a target of ~1000 to ~1100 emulates that instead, and your TBT gate will fire on genuinely slow regressions rather than on emulation that is harsher than any real user. Pick the target from your Real User Monitoring device distribution, not from the default, and keep desktop and mobile targets separate the same way you keep mobile and desktop budgets divergent.

Note that CPU throttling is only half the emulation. Network throttling (throttling.rttMs, throttling.throughputKbps) shapes when bytes arrive and therefore Largest Contentful Paint, while CPU throttling shapes how long the main thread stays busy and therefore Total Blocking Time and Interaction to Next Paint. Calibrating CPU without pinning network leaves LCP at P75 on a mid-range phone still drifting; treat the two as a pair. The multiplier work here fixes the CPU axis; the network axis is a fixed profile you set once and do not scale per runner.

CI Gating Assertion

Wire the calibration step ahead of collection, and assert on the BenchmarkIndex itself so a runner that is too slow or too contended to emulate reliably fails loudly rather than silently skewing the gate. The lighthouserc assertion below treats an out-of-band runner as an environment failure, not a code failure. Slotting the calibration step into a broader GitHub Actions performance matrix lets you calibrate each matrix leg independently.

- name: Calibrate CPU throttle
  run: node ./scripts/dynamic-cpu-throttle.js
  env:
    CALIBRATION_URL: https://staging.example.com

- name: Run Lighthouse CI with calibrated throttle
  run: npx lhci autorun --collect.settings.throttling.cpuSlowdownMultiplier=$(node -e "console.log(require('./cpu-throttle.json').cpuSlowdownMultiplier)")

{
  "ci": {
    "assert": {
      "assertions": {
        "uses-text-compression": "off",
        "metric-tbt": ["error", { "maxNumericValue": 350 }],
        "diagnostics": ["warn", { "maxNumericValue": 0 }]
      },
      "assertMatrix": [
        {
          "matchingUrlPattern": ".*",
          "preset": "lighthouse:no-pwa"
        }
      ]
    }
  }
}

The metric-tbt ceiling of 350 ms here is expressed for a mid-range phone emulated on Fast 3G at the P75 target device; on a desktop profile emulating a mid-tier laptop you would set a far lower ceiling because the emulated CPU is faster. If the runner's BenchmarkIndex falls below roughly 500 or above 3000, the derived multiplier hits a clamp boundary — treat that as a signal to pin the runner rather than trust the result.

Verification

After calibration, the emulated device should be stable even though the raw runner speed is not. Confirm two things:

• The multiplier tracks the runner. Print it from cpu-throttle.json on each run; on a hosted 4-vCPU runner it should land near 1.2–1.9×, not 4×. If it reads exactly 4×, the static default is still in effect and calibration did not run.
• TBT variance shrinks. Compare the standard deviation of Total Blocking Time across five builds before and after calibration. Passing output is a tighter spread — for example σ dropping from ~80 ms to under ~30 ms at P75 on a mid-range phone emulated on Fast 3G — because the emulated CPU is now constant. Persistent wide variance after calibration means the runner is contended; pin it and apply the denoising in Reducing Lighthouse CI Variance in Staging.

The chart below shows the expected shape of that improvement: five TBT samples before calibration scatter widely around the target, and five after calibration cluster tightly on it.

A tighter spread also makes downstream statistics honest: a regression detector comparing two builds only works when within-build noise is small relative to the change it hunts for, which is why calibration pairs naturally with statistical significance testing for noisy CI.

Frequently Asked Questions