Device & Network Emulation Weighting

Emulation weighting transforms synthetic performance testing from isolated lab runs into deterministic CI gates. By mapping synthetic device and network profiles to real-user traffic distributions, teams enforce performance budgets that reflect actual user impact. This methodology directly feeds into the broader Threshold Calibration & Baseline Management strategy for deterministic CI gating, ensuring that pull requests only merge when weighted metric aggregates stay within approved limits.

Architecture of the Weighting Matrix

Construct a deterministic profile matrix that maps Lighthouse/WebPageTest presets to observed RUM traffic. The matrix must normalize to 1.0 and align with CI runner concurrency limits.

Implementation steps:

  • Extract RUM traffic distribution by device class (mobile, tablet, desktop) and connection type (4G, 3G, slow-4G).
  • Map distributions to standard emulation presets (moto-g4, desktop-chrome, 3G-fast, 4G-good).
  • Normalize all weights to sum exactly to 1.0.
  • Validate total concurrent runs against CI runner capacity constraints.

weighting-matrix.json

{
 "profiles": [
 { "id": "mobile-3g", "device": "moto-g4", "network": "3G-fast", "weight": 0.45 },
 { "id": "mobile-4g", "device": "moto-g4", "network": "4G-good", "weight": 0.25 },
 { "id": "desktop-4g", "device": "desktop-chrome", "network": "4G-good", "weight": 0.20 },
 { "id": "desktop-eth", "device": "desktop-chrome", "network": "desktop", "weight": 0.10 }
 ],
 "metadata": {
 "version": "1.2.0",
 "normalized_sum": 1.0,
 "source": "rum_traffic_q3_2024"
 }
}

lighthouse-emulation-config.js

const matrix = require('./weighting-matrix.json');

module.exports = matrix.profiles.map(p => ({
 id: p.id,
 lighthouseFlags: {
 chromeFlags: ['--headless', '--no-sandbox'],
 formFactor: p.device.includes('desktop') ? 'desktop' : 'mobile',
 throttlingMethod: 'simulate',
 throttling: {
 rttMs: p.network === '3G-fast' ? 150 : p.network === '4G-good' ? 40 : 0,
 throughputKbps: p.network === '3G-fast' ? 1600 : p.network === '4G-good' ? 9000 : 10000000,
 cpuSlowdownMultiplier: p.device.includes('desktop') ? 1 : 4
 }
 }
}));

Step-by-Step CI Execution & Aggregation Pipeline

Execute parallel emulation runs using a matrix strategy. Aggregate results deterministically before applying CI gates.

ci-pipeline.yml (GitHub Actions)

name: Weighted Performance Gate
on: [pull_request]
jobs:
 emulate:
 runs-on: ubuntu-latest
 strategy:
 matrix:
 profile: [mobile-3g, mobile-4g, desktop-4g, desktop-eth]
 fail-fast: false
 steps:
 - uses: actions/checkout@v4
 - name: Run Lighthouse
 run: npx lighthouse https://staging.example.com --config-path=./lighthouse-emulation-config.js --output=json --output-path=./results/${{ matrix.profile }}.json
 - uses: actions/upload-artifact@v4
 with:
 name: lh-${{ matrix.profile }}
 path: ./results/${{ matrix.profile }}.json

 aggregate:
 needs: emulate
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - uses: actions/download-artifact@v4
 with:
 path: ./results
 - name: Weighted Aggregation
 run: node ./scripts/aggregate-weights.js

aggregate-weights.js

const fs = require('fs');
const path = require('path');
const matrix = require('../weighting-matrix.json');

const resultsDir = './results';
let weightedLCP = 0, weightedINP = 0, weightedCLS = 0;

matrix.profiles.forEach(p => {
 const filePath = path.join(resultsDir, `lh-${p.id}.json`);
 const report = JSON.parse(fs.readFileSync(filePath, 'utf8'));
 const { lcp, inp, cls } = report.audits;

 weightedLCP += (lcp.numericValue * p.weight);
 weightedINP += (inp.numericValue * p.weight);
 weightedCLS += (cls.numericValue * p.weight);
});

console.log(`Weighted LCP: ${weightedLCP.toFixed(2)}ms`);
console.log(`Weighted INP: ${weightedINP.toFixed(2)}ms`);
console.log(`Weighted CLS: ${weightedCLS.toFixed(4)}`);

if (weightedLCP > 2500 || weightedINP > 200 || weightedCLS > 0.1) {
 process.exit(1);
}

Handling Synthetic Variance & Flakiness

Synthetic environments introduce metric variance that skews weighted aggregates. Outlier filtering must precede weight application per established Statistical Noise & Flakiness Reduction protocols.

Implementation steps:

  • Run three baseline iterations per profile to establish variance bounds.
  • Apply a median filter to LCP, INP, and CLS values before multiplying by profile weights.
  • Configure CI to retry any single profile run exceeding 15% standard deviation from the median.
  • Log variance deltas to a structured output file for QA review and pipeline auditing.

Calculating Weighted Thresholds for CI Gating

Convert raw synthetic metrics into a single pass/fail gate using weighted aggregation. The formula ensures high-traffic profiles dominate the CI decision.

Formula: Weighted_Metric = Σ(Profile_Weight * Metric_Value)

Implementation steps:

  • Define base metric targets per profile (e.g., Mobile-3G LCP ≤ 2.8s, Desktop-4G LCP ≤ 1.5s).
  • Apply weight multipliers to each profile’s median metric value.
  • Compute the aggregate weighted score across all active profiles.
  • Set CI exit codes based on weighted thresholds, aligning with Percentile-Based Threshold Tuning for accurate user-impact modeling.

Example threshold logic:

const THRESHOLDS = { LCP: 2500, INP: 200, CLS: 0.1 };
const weightedScores = { LCP: 2340, INP: 185, CLS: 0.085 };

Object.keys(THRESHOLDS).forEach(metric => {
 if (weightedScores[metric] > THRESHOLDS[metric]) {
 console.error(`CI Gate Failed: ${metric} exceeded threshold`);
 process.exit(1);
 }
});

Advanced Configuration: Dynamic Weight Adjustment

Static matrices degrade as traffic patterns shift. Implement environment-driven overrides to adjust weights dynamically without pipeline redeployment.

Implementation steps:

  • Inject traffic-shape JSON via CI environment variables (TRAFFIC_MATRIX_OVERRIDE).
  • Configure fallback to the static weighting-matrix.json on fetch failure or parse error.
  • Validate weight normalization in a pre-flight script before spawning parallel jobs.
  • Enable conditional gating per deployment environment (e.g., stricter weights for production, relaxed for staging).
  • Transition to regional traffic shaping, demonstrating how geo-specific routing overrides integrate with Weighting Budgets by User Geography for localized budget enforcement.

dynamic-weight-loader.js

const fs = require('fs');
const defaultMatrix = require('./weighting-matrix.json');

function loadMatrix() {
 try {
 const override = process.env.TRAFFIC_MATRIX_OVERRIDE;
 if (!override) return defaultMatrix;
 const parsed = JSON.parse(override);
 const sum = parsed.profiles.reduce((acc, p) => acc + p.weight, 0);
 if (Math.abs(sum - 1.0) > 0.001) throw new Error('Weights do not normalize to 1.0');
 return parsed;
 } catch (err) {
 console.warn('Dynamic matrix load failed, falling back to static:', err.message);
 return defaultMatrix;
 }
}

module.exports = loadMatrix();

QA Validation & Rollout Checklist

Enforce strict validation before merging weighted gating into production pipelines.

Implementation steps:

  • Execute a dry-run against the main branch with CI_DRY_RUN=true to verify artifact collection and aggregation logic.
  • Compare weighted scores against historical baselines to detect threshold drift.
  • Validate CI gating exit codes (0 for pass, 1 for fail) under simulated budget breaches.
  • Document weight matrix versioning and attach RUM data snapshots to the PR.
  • Enable production enforcement only after QA sign-off and a 7-day shadow mode period.