GitHub Actions Performance Matrices #

A single Lighthouse run on ubuntu-latest tells you almost nothing about how a page behaves on a mid-range phone over Fast 3G — yet that is where most of your users live. Architecting a deterministic gate means running the same audit across a deliberate grid of URLs, device classes, and network profiles, then enforcing a per-axis budget so a regression on mobile 4G blocks the merge even when desktop fiber stays green. This guide is part of the Lighthouse CI & WebPageTest Integration reference, and it replaces aggregate scoring with strict per-variant threshold enforcement.

The mechanics rest on three coupled decisions: which axes to vary (coverage), how to fan them out without exploding runner cost (the matrix), and how to merge the parallel outcomes back into one required status check (the gate). Get the axes wrong and you gate on conditions no user experiences; get the fan-out wrong and CI becomes the bottleneck it was meant to protect.

Architecture Overview #

One workflow trigger expands — via strategy.matrix — into N parallel jobs, each pinned to a distinct device/network profile. Every job runs an isolated audit, uploads its report as a namespaced artifact, and reports a pass/fail. A final merge job collects the artifacts and produces a single status check that branch protection gates on.

Prerequisites & Environment #

• @lhci/cli ≥ 0.13 installed as a dev dependency so the version is pinned in package-lock.json.
• Node.js ≥ 18, Chrome ≥ 120 — ubuntu-latest ships a compatible Chrome.
• Branch protection configured to require the merged gate status check; without it the matrix runs but never blocks anything.
• A consistent runner CPU baseline. Hosted runners drift in core count and clock speed; if your throttling depends on it, calibrate with Device & Network Emulation Weighting before trusting the numbers.

Inject anything environment-specific through workflow env rather than hardcoding it, and keep the budget definition in version control alongside lighthouserc.json as described in Lighthouse CI Configuration & Storage.

Configuration Reference #

Define discrete axes with an include array, not a Cartesian device × network × viewport product — the latter explodes into dozens of jobs and unpredictable wall-clock time. Each include entry is one runnable profile, and its fields are injected into the audit step as environment variables.

# .github/workflows/perf-matrix.yml
name: Performance Matrix Gating
on:
  pull_request:
    branches: [main]

concurrency:
  group: perf-matrix-${{ github.ref }}
  cancel-in-progress: true

jobs:
  lighthouse-matrix:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    strategy:
      fail-fast: false        # one failing variant must not cancel the rest
      max-parallel: 4         # cap runner fan-out to control cost and CPU contention
      matrix:
        include:
          - profile: desktop-fiber
            preset: desktop
            cpu_throttle: 1
            network: "fiber"
            url: "http://localhost:8080/"
          - profile: mobile-4g
            preset: mobile
            cpu_throttle: 4
            network: "4G"
            url: "http://localhost:8080/"
          - profile: mobile-3g
            preset: mobile
            cpu_throttle: 4
            network: "Fast3G"
            url: "http://localhost:8080/checkout"
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
      - run: npm ci
      - run: npm run build
      - name: Run Lighthouse CI (${{ matrix.profile }})
        run: npx lhci autorun
        env:
          LHCI_PRESET: ${{ matrix.preset }}
          LHCI_CPU_SLOWDOWN: ${{ matrix.cpu_throttle }}
          LHCI_URL: ${{ matrix.url }}
      - name: Upload variant report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: lhci-${{ matrix.profile }}
          path: .lighthouseci/

The matching lighthouserc.js reads those variables so one config serves every variant. fail-fast: false keeps the full picture visible on a breach; max-parallel: 4 is the single most important cost lever.

// lighthouserc.js
module.exports = {
  ci: {
    collect: {
      url: [process.env.LHCI_URL],
      numberOfRuns: 3,
      settings: {
        preset: process.env.LHCI_PRESET || "desktop",
        throttlingMethod: "simulate",
        throttling: { cpuSlowdownMultiplier: Number(process.env.LHCI_CPU_SLOWDOWN || 1) },
        chromeFlags: "--no-sandbox --disable-dev-shm-usage",
      },
    },
    assert: {
      assertions: {
        "categories:performance": ["error", { minScore: 0.9 }],
        "metric-lcp": ["error", { maxNumericValue: 2500 }],
        "cumulative-layout-shift": ["error", { maxNumericValue: 0.1 }],
        "total-blocking-time": ["error", { maxNumericValue: 200 }],
      },
    },
  },
};

Step-by-Step Implementation #

1. Define the axes. List each device/network profile as an include entry. Start with two — desktop fiber and mid-range mobile 4G — and add a 3G or low-end variant only once those are green and stable.

2. Bind matrix variables into the audit. Pass matrix.preset, matrix.cpu_throttle, and matrix.url as env to the lhci autorun step and read them in lighthouserc.js. Validate locally:

   LHCI_PRESET=mobile LHCI_CPU_SLOWDOWN=4 LHCI_URL=http://localhost:8080/ npx lhci autorun

   Expected tail: Done running Lighthouse!, then All results processed! with an assertion summary per URL.

3. Namespace the artifacts. Use name: lhci-${{ matrix.profile }} on the upload step so parallel jobs never clobber each other's reports — non-unique artifact names are the classic cause of a merged report containing only one variant.

4. Add the merge gate. A dependent job downloads every lhci-* artifact and emits one status check. Require that check in branch protection.

Threshold Calibration #

Vary axes that change the user's experience, and hold every other variable pinned. The two axes that move metrics most are CPU throttling (CPU-bound TBT/INP) and network profile (LCP and Speed Index). Derive each ceiling from the P75 of your field data for that device class, then set the lab assertion 10–15% tighter. The matrix below is a representative starting grid, not a copy-paste budget.

Keep a newly added variant at warn until its baseline holds for two consecutive weeks, then promote it to error. For the per-device split that justifies separate ceilings, see Mobile vs Desktop Budget Divergence.

CI Enforcement Snippet #

This merge job aggregates the parallel variant artifacts into one required status check. Branch protection requires performance-gate, so any single failing variant makes the PR unmergeable.

  performance-gate:
    needs: lighthouse-matrix
    runs-on: ubuntu-latest
    if: always()
    steps:
      - name: Download all variant reports
        uses: actions/download-artifact@v4
        with:
          path: reports
          pattern: lhci-*
      - name: Fail if any variant failed
        run: |
          if [ "${{ needs.lighthouse-matrix.result }}" != "success" ]; then
            echo "::error::One or more matrix variants breached budget"
            exit 1
          fi
          echo "All matrix variants passed budget."

Because needs.lighthouse-matrix.result is success only when every matrix job succeeds, this one check faithfully reflects the whole grid. Speed the whole pipeline up with Setting Up GitHub Actions Caching for Faster CI.

Troubleshooting & Edge Cases #

• Flaky failures on parallel jobs → fan-out increases CPU contention, which inflates TBT/INP. Lower max-parallel, raise numberOfRuns to 5, and pin throttlingMethod: simulate so timings do not depend on the runner's real load.
• Merged report shows only one variant → artifact names collide. Ensure every upload uses lhci-${{ matrix.profile }} and every variant profile is unique.
• One slow variant cancels the others → set fail-fast: false so a breach on 3G does not abort the desktop job and hide its result.
• Combinatorial runner explosion → never use bare matrix: { device: [...], network: [...] }; enumerate include entries so the job count is exactly what you listed.
• download-artifact finds nothing → the upload step needs if: always(), or failed variants never publish their reports for the merge job.
• Runner drift between jobs → pin a fixed runner image and CPU slowdown per variant; uncalibrated CPU throttling corrupts cross-variant comparisons.
• Cost creep → gate the full grid on PRs to main only and run a reduced two-variant grid on draft PRs.

Frequently Asked Questions #