GitHub Actions Performance Matrices

A single Lighthouse run on ubuntu-latest tells you almost nothing about how a page behaves on a mid-range phone over Fast 3G — yet that is where most of your users live. Architecting a deterministic gate means running the same audit across a deliberate grid of URLs, device classes, and network profiles, then enforcing a per-axis budget so a regression on mid-range mobile 4G blocks the merge even when desktop fiber stays green. This guide is part of the Lighthouse CI & WebPageTest Integration reference, and it replaces aggregate scoring with strict per-variant threshold enforcement.

The mechanics rest on three coupled decisions: which axes to vary (coverage), how to fan them out without exploding runner cost (the matrix), and how to merge the parallel outcomes back into one required status check (the gate). Get the axes wrong and you gate on conditions no user experiences; get the fan-out wrong and CI becomes the bottleneck it was meant to protect. Each PR-time matrix run is the enforcement counterpart to the scheduled synthetic checks described in Continuous Performance Monitoring — the matrix blocks bad merges, the nightly runs catch drift the matrix never sees.

Architecture Overview

One workflow trigger expands — via strategy.matrix — into N parallel jobs, each pinned to a distinct device/network profile. Every job runs an isolated audit, uploads its report as a namespaced artifact, and reports a pass/fail. A final merge job collects the artifacts and produces a single status check that branch protection gates on. The merge job is not decoration: branch protection can only require a fixed, named check, and matrix jobs carry per-variant names that change as you edit the grid. Funneling every variant through one stable check name is what makes the gate durable across matrix edits.

Everything downstream of the fan-out is deterministic by construction: each job is hermetic (checkout, build, audit, upload), it never reads another variant's state, and the merge job only inspects the boolean result of the whole grid. That isolation is what lets you reason about failures — a red gate points at exactly one variant, and its artifact holds the numbers that explain it.

Choosing Which Axes to Vary

The temptation is to test everything: every device, every network, every viewport, every locale. That produces a grid nobody reads and a bill nobody approves. The discipline is the opposite — vary only axes that move a metric a real user would feel, and pin everything else to a fixed value so cross-variant comparisons stay honest. An axis earns a slot in the matrix only when it changes the user experience and you have field data to anchor its budget. Without field data you cannot set a defensible ceiling, so the variant enters the grid as warn until enough real-user samples accumulate to justify an error gate.

In practice this yields a small, opinionated grid. Two or three device/network profiles, two or three high-value URLs (usually the landing page and the highest-intent conversion route such as checkout), and every other knob — viewport within a device class, locale, cookie state — pinned. If you suspect an axis matters but cannot prove it from field data, run it warn-only for a fortnight and read the variance before you let it block anyone. The percentile you anchor to is itself a decision worth making deliberately, covered in Percentile-Based Threshold Tuning.

Prerequisites and Environment

• @lhci/cli ≥ 0.13 installed as a dev dependency so the version is pinned in package-lock.json and every runner resolves the identical binary.
• Node.js ≥ 18, Chrome ≥ 120 — ubuntu-latest ships a compatible Chrome, but pin the runner image tag rather than the floating latest alias if you need byte-for-byte reproducibility.
• Branch protection configured to require the merged gate status check; without it the matrix runs but never blocks anything, which is the single most common reason a "gated" repo still ships regressions.
• A consistent runner CPU baseline. Hosted runners drift in core count and clock speed; if your throttling depends on it, calibrate with Device & Network Emulation Weighting and validate the multiplier following Calibrating CPU Throttling for CI Runners before trusting the numbers.

Inject anything environment-specific through workflow env rather than hardcoding it, and keep the budget definition in version control alongside lighthouserc.json as described in Lighthouse CI Configuration & Storage. Treat the config as data: the workflow supplies which profile to run, and the config file is a pure function of those inputs.

Configuration Reference

Define discrete axes with an include array, not a Cartesian device × network × viewport product — the latter explodes into dozens of jobs and unpredictable wall-clock time. Each include entry is one runnable profile, and its fields are injected into the audit step as environment variables.

# .github/workflows/perf-matrix.yml
name: Performance Matrix Gating
on:
  pull_request:
    branches: [main]

concurrency:
  group: perf-matrix-${{ github.ref }}
  cancel-in-progress: true

jobs:
  lighthouse-matrix:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false        # one failing variant must not cancel the rest
      max-parallel: 4         # cap runner fan-out to control cost and CPU contention
      matrix:
        include:
          - profile: desktop-fiber
            preset: desktop
            cpu_throttle: 1
            network: "fiber"
            url: "http://localhost:8080/"
          - profile: mobile-4g
            preset: mobile
            cpu_throttle: 4
            network: "4G"
            url: "http://localhost:8080/"
          - profile: mobile-3g
            preset: mobile
            cpu_throttle: 4
            network: "Fast3G"
            url: "http://localhost:8080/checkout"
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
      - run: npm ci
      - run: npm run build
      - name: Run Lighthouse CI (${{ matrix.profile }})
        run: npx lhci autorun
        env:
          LHCI_PRESET: ${{ matrix.preset }}
          LHCI_CPU_SLOWDOWN: ${{ matrix.cpu_throttle }}
          LHCI_URL: ${{ matrix.url }}
      - name: Upload variant report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: lhci-${{ matrix.profile }}
          path: .lighthouseci/

The matching lighthouserc.js reads those variables so one config serves every variant. fail-fast: false keeps the full picture visible on a breach; max-parallel: 4 is the single most important cost lever.

// lighthouserc.js
module.exports = {
  ci: {
    collect: {
      url: [process.env.LHCI_URL],
      numberOfRuns: 3,
      settings: {
        preset: process.env.LHCI_PRESET || "desktop",
        throttlingMethod: "simulate",
        throttling: { cpuSlowdownMultiplier: Number(process.env.LHCI_CPU_SLOWDOWN || 1) },
        chromeFlags: "--no-sandbox --disable-dev-shm-usage",
      },
    },
    assert: {
      assertions: {
        "categories:performance": ["error", { minScore: 0.9 }],
        "largest-contentful-paint": ["error", { maxNumericValue: 2500 }],
        "cumulative-layout-shift": ["error", { maxNumericValue: 0.1 }],
        "total-blocking-time": ["error", { maxNumericValue: 200 }],
      },
    },
  },
};

Notice that numberOfRuns: 3 collects three passes and Lighthouse CI asserts against the median, which is the cheapest available defence against single-run noise. On the noisiest mobile variants you will raise this to five, at a proportional cost in wall-clock time — the trade-off the next section quantifies.

Modeling Cost and Wall-Clock Time

The reason the include list matters is arithmetic. A bare Cartesian matrix multiplies its axes: three device classes crossed with three networks crossed with three viewports is 27 jobs, and at three Lighthouse runs each that is 81 audits per pull request. The same coverage a team actually cares about — three named profiles — is three jobs and nine audits. The include list is not merely tidier; it is an order of magnitude cheaper for coverage a human can actually reason about.

Wall-clock time is governed by max-parallel, not job count. If a single variant takes about four and a half minutes to build and run three audits, three variants at max-parallel: 4 all run concurrently and the matrix finishes in roughly five minutes plus a short merge job — under six minutes end to end. Drop to max-parallel: 2 and the same three variants run in two waves, pushing wall-clock time past ten minutes. The lever cuts both ways: high parallelism is fast but increases CPU contention on shared runners, which inflates CPU-bound metrics. When a mid-range mobile variant throttled at cpu×4 starts flaking near its P75 TBT ceiling of 350 ms on Fast 3G, lowering max-parallel trades a minute of wall-clock time for a stabler median. Budget the grid so the common case — a green PR to main — clears in well under ten minutes, because a gate developers wait on is a gate developers route around.

Step-by-Step Implementation

1. Define the axes. List each device/network profile as an include entry. Start with two — desktop fiber and mid-range mobile 4G — and add a Fast 3G or low-end variant only once those are green and stable for a couple of weeks.

2. Bind matrix variables into the audit. Pass matrix.preset, matrix.cpu_throttle, and matrix.url as env to the lhci autorun step and read them in lighthouserc.js. Validate locally before you ever push:

   LHCI_PRESET=mobile LHCI_CPU_SLOWDOWN=4 LHCI_URL=http://localhost:8080/ npx lhci autorun

   Expected tail: Done running Lighthouse!, then All results processed! with an assertion summary per URL. If an assertion fails locally it will fail in CI identically, because the config is a pure function of those three variables.

3. Namespace the artifacts. Use name: lhci-${{ matrix.profile }} on the upload step so parallel jobs never clobber each other's reports — non-unique artifact names are the classic cause of a merged report containing only one variant. With actions/upload-artifact@v4 a name collision is a hard error rather than a silent overwrite, but a templated name sidesteps the problem entirely.

4. Add the merge gate. A dependent job downloads every lhci-* artifact and emits one status check. Require that check in branch protection, and confirm on a throwaway PR that a deliberately over-budget change actually turns the check red. A gate you have never seen fail is a gate you cannot trust.

Threshold Calibration

Vary axes that change the user's experience, and hold every other variable pinned. The two axes that move metrics most are CPU throttling (CPU-bound TBT and INP) and network profile (LCP and Speed Index). Derive each ceiling from the P75 of your field data for that device class, then set the lab assertion 10–15% tighter so lab noise does not let a real-world regression slip through. The matrix below is a representative starting grid, not a copy-paste budget — your field data sets the real numbers.

The ceilings widen as the device and network degrade because the same code genuinely takes longer to become interactive on a mid-range phone over Fast 3G than on a desktop over fiber — a flat cross-device budget would either be unreachable on mobile or trivially loose on desktop. Keep a newly added variant at warn until its baseline holds for two consecutive weeks, then promote it to error. For the per-device split that justifies separate ceilings, see Mobile vs Desktop Budget Divergence; for the PR-time collection loop that feeds these assertions, see Running Lighthouse CI on Every Pull Request.

CI Enforcement Snippet

This merge job aggregates the parallel variant artifacts into one required status check. Branch protection requires performance-gate, so any single failing variant makes the PR unmergeable.

  performance-gate:
    needs: lighthouse-matrix
    runs-on: ubuntu-latest
    if: always()
    steps:
      - name: Download all variant reports
        uses: actions/download-artifact@v4
        with:
          path: reports
          pattern: lhci-*
      - name: Fail if any variant failed
        run: |
          if [ "${{ needs.lighthouse-matrix.result }}" != "success" ]; then
            echo "::error::One or more matrix variants breached budget"
            exit 1
          fi
          echo "All matrix variants passed budget."

Because needs.lighthouse-matrix.result is success only when every matrix job succeeds, this one check faithfully reflects the whole grid. The if: always() is load-bearing: without it the merge job is skipped whenever any variant fails, and a skipped required check does not block a merge — the exact failure mode a gate is meant to prevent. Speed the whole pipeline up with Setting Up GitHub Actions Caching for Faster CI.

Troubleshooting and Edge Cases

• Flaky failures on parallel jobs → fan-out increases CPU contention, which inflates TBT/INP. Lower max-parallel, raise numberOfRuns to 5, and pin throttlingMethod: simulate so timings do not depend on the runner's real load.
• Merged report shows only one variant → artifact names collide. Ensure every upload uses lhci-${{ matrix.profile }} and every variant profile is unique.
• One slow variant cancels the others → set fail-fast: false so a breach on Fast 3G does not abort the desktop job and hide its result.
• Combinatorial runner explosion → never use bare matrix: { device: [...], network: [...] }; enumerate include entries so the job count is exactly what you listed.
• download-artifact finds nothing → the upload step needs if: always(), or failed variants never publish their reports for the merge job.
• Skipped gate lets a red PR merge → the merge job also needs if: always(); a required check that is skipped counts as satisfied, so the gate must always run even when variants fail.
• Runner drift between jobs → pin a fixed runner image and CPU slowdown per variant; uncalibrated CPU throttling corrupts cross-variant comparisons.
• Cost creep → gate the full grid on PRs to main only and run a reduced two-variant grid on draft PRs.

Frequently Asked Questions