Choosing a Performance Testing Stack by Team Size

The most common way teams waste money on performance tooling is buying the stack that fits the org they wish they were. A solo maintainer does not need a private WebPageTest fleet, and a fifty-engineer platform group cannot govern quality with one developer's laptop scores. This how-to, part of the Comparing Performance Testing Tools reference, sizes the stack to the team: three concrete tiers, a table that maps headcount and traffic to tooling, a migration path with explicit trigger conditions, and the real cost and maintenance load each tier carries.

The organizing principle is simple. Your tooling investment should track two variables — how many people can break performance, and how many users feel it when they do. A single developer breaking a budget affects one review; a shared self-hosted history changes how twenty developers argue about regressions; and once field traffic crosses into the millions of monthly views, synthetic checks alone stop being enough and you need real-user data feeding the same budgets. Pick the smallest tier that covers your current reality, then migrate when a documented trigger fires — never before.

The Three Tiers at a Glance

Each tier is additive: the mid tier keeps the solo tier's Lighthouse CI gate and adds server-side history plus sampled field data; the large tier keeps both and adds a shaped-network diagnostic and a full observability pipeline. Nothing is thrown away when you move up, which is what makes the migration path cheap.

Read the table by finding the row where your reality no longer fits the current column. If two developers now trip over each other's local Lighthouse numbers, you have outgrown Solo. If your P75 LCP measured on mid-range mobile over Fast 3G looks healthy in CI but users complain anyway, you have outgrown Mid and need field data. The diagram below shows the three stacks side by side.

Tier 1: Solo — Lighthouse CI Only

For a solo maintainer or a two-person side project, Lighthouse CI running on every pull request is the entire stack. It needs no server, no database, and no field-data pipeline. The lhci autorun command collects three runs, compares medians against assertions, and uploads a report to temporary public storage that expires on its own — zero infrastructure to maintain. Set a single realistic ceiling and gate on it. A workable starting budget is an LCP P75 of 3000 ms and a Total Blocking Time P75 of 300 ms measured on the emulated mid-range mobile profile Lighthouse ships (Moto G-class CPU, simulated Slow 4G), which is the default mobile preset.

# lighthouserc.yml — the complete solo-tier config
ci:
  collect:
    url:
      - https://staging.example.com/
    numberOfRuns: 3
    settings:
      preset: mobile
  assert:
    assertions:
      largest-contentful-paint:
        - error
        - maxNumericValue: 3000
      total-blocking-time:
        - error
        - maxNumericValue: 300
      cumulative-layout-shift:
        - error
        - maxNumericValue: 0.1
  upload:
    target: temporary-public-storage

Wire it into CI as a required check and you are done:

# .github/workflows/lhci.yml
name: Lighthouse CI
on: pull_request
jobs:
  lhci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci && npm run build
      - run: npm install -g @lhci/[email protected]
      - run: lhci autorun

The honest limitation: temporary public storage keeps no history you own, so you cannot chart a slow drift over months, and with no field data you are trusting that Lighthouse's emulated mid-range mobile on simulated Slow 4G resembles your actual users. For one developer shipping to low traffic, that trade is correct — the whole tier costs about an hour to set up and an hour a month to maintain. You graduate when a second and third developer arrive and you need a shared, durable record of who regressed what.

Tier 2: Mid — Self-Hosted LHCI Server and RUM Sampling

Once five to twenty developers share a codebase, the temporary report that vanishes overnight becomes the bottleneck. The mid tier fixes that by running your own self-hosted Lighthouse CI server, which stores every run in a database you control, renders trend charts, and computes a statistical baseline from recent commits instead of a fixed number you guessed. Point the same lhci autorun at it by changing only the upload target:

  upload:
    target: lhci
    serverBaseUrl: https://lhci.internal.example.com
    token: ${{ secrets.LHCI_BUILD_TOKEN }}

The second half of this tier is field data. Lighthouse's lab numbers cannot see your real device and network mix, so you add a lightweight Real User Monitoring beacon. At mid-tier traffic you do not need every session — sampling one in ten keeps payload and storage costs low while still yielding a stable P75. The Custom Performance Beacons and RUM reference covers the beacon design; a minimal sampled collector looks like this:

import { onLCP, onINP, onCLS } from 'web-vitals';

const SAMPLE_RATE = 0.1; // 10% of sessions
const sampled = Math.random() < SAMPLE_RATE;

function report(metric) {
  if (!sampled) return;
  const body = JSON.stringify({
    name: metric.name,
    value: metric.value,
    id: metric.id,
    path: location.pathname,
    conn: navigator.connection?.effectiveType ?? 'unknown',
  });
  navigator.sendBeacon('/rum', body);
}

onLCP(report);
onINP(report);
onCLS(report);

With this in place the team can compare the lab gate against the field. A useful mid-tier target is a field LCP P75 at or under 2500 ms and an INP P75 at or under 200 ms, both measured across sampled real mobile sessions rather than an emulated device — those are the thresholds Grafana alerts on. Setup runs one to two days including the server deploy; maintenance is roughly six hours a month covering server upgrades, database backups, and the occasional sampling-rate tune. The diagram maps how the two data sources flow into one place.

Tier 3: Large — WebPageTest Private Instance, Full RUM, and Datadog

At twenty-plus engineers and traffic above two million monthly views, two new requirements appear that the mid tier cannot serve. First, when a regression slips through you need to explain it, not just detect it — that means a shaped real-network diagnostic with a filmstrip and per-request waterfall, which is a WebPageTest private instance running on dedicated agents so results are reproducible. Second, a 10% sample is now both unnecessarily large in volume and too coarse for tail analysis, so you move to a full RUM pipeline that computes proper percentiles and pushes them into an observability platform alongside your other production signals.

The percentile work is real engineering: computing a trustworthy P75 and P99 from a firehose of beacons requires a P75/P99 aggregation pipeline rather than a naive average, and at this scale you often switch from a fixed sample rate to smarter strategies. Route the aggregated metrics into Datadog so budget breaches page the same on-call rotation as everything else — the Datadog integration reference walks through the monitors. A large-tier field target tightens to an LCP P75 at or under 2500 ms with a P90 at or under 4000 ms, both on real mobile sessions, and a Datadog monitor fires when a deploy pushes the P75 above ceiling for fifteen minutes.

# .github/workflows/perf-large.yml — nightly WebPageTest on key routes
name: WebPageTest nightly
on:
  schedule:
    - cron: '0 3 * * *'
jobs:
  wpt:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        route: ['/', '/product', '/checkout']
    steps:
      - name: Submit test
        run: |
          curl -s "$WPT_SERVER/runtest.php?url=https://www.example.com${{ matrix.route }}&k=$WPT_API_KEY&f=json&runs=5&connectivity=4G&location=agent-01" > run.json
          echo "TEST_ID=$(cat run.json | npx json data.testId)" >> "$GITHUB_ENV"
    env:
      WPT_SERVER: ${{ secrets.WPT_SERVER }}
      WPT_API_KEY: ${{ secrets.WPT_API_KEY }}

This tier costs one to two weeks to stand up and roughly twenty hours a month to run: agent maintenance, pipeline reliability, monitor tuning, and reviewing the field-versus-lab gap. That is a real headcount cost, which is exactly why you do not adopt it until traffic and team size justify it.

The Migration Path Between Tiers

Tiers are additive, so migration is layering, not rebuilding. Moving from Solo to Mid means standing up the server and repointing the existing lhci autorun upload target — the assertions you already wrote stay identical. Moving from Mid to Large means adding WebPageTest and the Datadog sink beside the LHCI server you already run; the mid-tier Grafana dashboard can stay or be folded into Datadog. Never migrate on a calendar. Migrate when a specific trigger fires.

The Solo-to-Mid trigger is organizational: a third developer joins, local scores start disagreeing, and you need a durable shared record to settle "did my PR regress LCP?" The Mid-to-Large trigger is a mix of scale and evidence — traffic crosses roughly two million monthly views, or your field P75 diverges from your lab gate often enough that a 10% sample and a single lab profile no longer explain what users feel. If neither trigger has fired, adding tooling only adds maintenance you will resent.

Cost and Maintenance per Tier

The table below turns the tiers into a budget line. Costs are directional monthly figures for a typical setup on commodity cloud infrastructure; the maintenance column is the hours a real engineer spends keeping it healthy, which usually dwarfs the hosting bill.

The chart below plots the maintenance load, because that is the number teams underestimate. Hosting is cheap; the standing engineering time to keep agents, pipelines, and monitors trustworthy is what actually gates how large a stack you can afford to run.

Frequently Asked Questions