What is a performance regression?

A performance regression is a code change that makes the system slower or consume more resources. Individual regressions may be small (adding 10ms to a request), but they accumulate over time. After months of undetected small regressions, an application can be 40% slower than a previous release.

Why can't you just compare two benchmark numbers to detect regression?

Performance measurements have inherent variance from garbage collection, CPU cache behavior, OS scheduling, and background processes. The same code may produce 52ms on one run and 58ms on the next. Simple comparison produces unacceptable false positive rates. Statistical methods like the Mann-Whitney U test account for this variance.

What is the Mann-Whitney U test for performance?

Mann-Whitney U is a non-parametric statistical test that compares two sample sets without assuming normal distribution. For regression detection, collect multiple measurements from both baseline and candidate builds, then test whether the candidate sample is drawn from a distribution with a larger median. It is robust for right-skewed performance data.

How do you handle false positives in regression detection?

Provide easy override mechanisms (like a PR label), track the override rate (above 20% means thresholds are too sensitive), allow re-runs to verify reproducibility, and require both statistical AND practical significance (minimum effect size). The goal is keeping the system trustworthy so engineers don't learn to ignore it.

What should you benchmark for regression detection?

Focus on operations users experience: critical path flows (login, search, checkout), hot paths called millions of times where small regressions multiply, application startup time, memory allocation patterns, and frequently-executed database queries. Not every function needs a benchmark — prioritize by user impact.

Performance Regression Detection: Catching Slowdowns Before Users Do

Tom Lindgren · July 28, 2026 · 13 min read

A performance regression is a commit that makes things slower. It may add 15ms to a database query, increase memory allocation by 8%, or add one extra HTTP round trip to a user flow. Individually, each regression seems minor. But regressions accumulate. After six months of small regressions that nobody noticed, the application is 40% slower than the previous release and nobody knows which of the 2,000 merged commits is responsible.

Performance regression detection catches these slowdowns at the point they are introduced—during code review and CI/CD—before they compound into noticeable degradation.

Why Detection Is Hard

Unlike functional regressions, which produce objectively wrong output (a test either passes or fails), performance measurements are inherently noisy. The same code running on the same hardware produces different response times on each execution due to garbage collection, CPU cache behavior, OS scheduling, and background processes. A benchmark that shows 52ms on one run and 58ms on the next has not necessarily regressed—the 11% difference may be within normal variance.

This noise-to-signal problem means that naive comparison ("is the new result slower than the old result?") produces unacceptable false positive rates. Every other CI run flags a "regression" that is actually noise, and engineers learn to ignore the results. Effective detection requires statistical rigor.

Statistical Comparison Methods

Threshold-Based Detection

The simplest approach: flag a regression when the new result exceeds the baseline by more than a fixed percentage. "Alert if P99 latency increases by more than 10%." Simple to implement and understand, but the fixed threshold ignores the variance of the measurement. A 10% threshold is too loose for stable metrics with 1% variance and too tight for noisy metrics with 15% variance.

Standard Deviation Method

Compare the new measurement against the mean and standard deviation of the baseline measurements. Flag a regression when the new result exceeds mean + (k × standard deviation). With k=2, this flags results outside the 95th percentile of expected variation. With k=3, it flags outside the 99.7th percentile. This automatically adapts to metric variance but assumes a normal distribution, which performance data often is not.

Mann-Whitney U Test

A non-parametric statistical test that compares two samples without assuming normal distribution. Collect multiple measurements from both the baseline and candidate builds, then test whether the candidate sample is drawn from a distribution with a larger median. This is the most robust approach for performance data, which is typically right-skewed.

import numpy as np from scipy import stats def detect_regression(baseline_samples, candidate_samples, alpha=0.05, effect_size_threshold=0.05): """ Detect performance regression using Mann-Whitney U test. Returns (is_regression, p_value, effect_size) """ # Mann-Whitney U test (one-sided: is candidate slower?) statistic, p_value = stats.mannwhitneyu( baseline_samples, candidate_samples, alternative='less' # test if candidate > baseline ) # Effect size: relative difference in medians baseline_median = np.median(baseline_samples) candidate_median = np.median(candidate_samples) effect_size = (candidate_median - baseline_median) / baseline_median # Regression = statistically significant AND practically significant is_regression = (p_value < alpha) and (effect_size > effect_size_threshold) return is_regression, p_value, effect_size # Usage baseline = [48, 51, 49, 52, 50, 47, 53, 49, 51, 50] # ms candidate = [55, 58, 54, 57, 56, 53, 59, 55, 57, 56] # ms regressed, p, effect = detect_regression(baseline, candidate) # regressed=True, p=0.0001, effect=0.12 (12% slower)

Practical significance vs. statistical significance: A statistically significant result (low p-value) only means the difference is unlikely to be noise. It does not mean the difference matters. A 0.5% increase in latency may be statistically significant with enough samples but practically irrelevant. Always require both statistical significance and a minimum effect size before flagging a regression.

Baseline Management

The quality of regression detection depends entirely on the quality of the baseline. A baseline that is too old includes improvements that have since been made, making regressions harder to detect. A baseline that is too recent may have already been contaminated by recent regressions.

Rolling Baseline

Maintain a rolling baseline from the last N successful builds on the main branch. Each new merge updates the baseline with fresh measurements. This approach adapts naturally as the codebase evolves. The window size (N) balances stability (larger window = less noise) against sensitivity (smaller window = faster detection).

Tagged Baseline

Pin the baseline to a specific release tag. All performance comparisons are made against the last released version. This approach is simpler and more stable but may miss regressions that accumulate between releases. It works best for projects with frequent releases (weekly or biweekly).

Environment Consistency

Performance baselines are only valid if the measurement environment is consistent. A benchmark run on a shared CI runner with varying load is not comparable to one run on a dedicated machine. For reliable regression detection:

Use dedicated, consistent hardware for performance benchmarks
Pin CPU frequency to prevent turbo boost variance
Disable hyperthreading for predictable core performance
Use cgroups to isolate benchmark processes from other workloads
Run warm-up iterations before measurement to eliminate JIT and cache effects

CI/CD Integration

Performance Gates

Add performance comparison as a required CI check. The pipeline runs benchmarks on the candidate branch, compares against the baseline, and reports results as a PR comment or CI status check. A detected regression blocks the merge until the author addresses it.

# GitHub Actions performance gate name: Performance Regression Check on: [pull_request] jobs: benchmark: runs-on: [self-hosted, perf-runner] steps: - uses: actions/checkout@v4 - name: Run benchmarks run: | # Run each benchmark 10 times for statistical power for i in $(seq 1 10); do ./run-benchmarks.sh >> candidate-results.json done - name: Fetch baseline run: | git checkout main for i in $(seq 1 10); do ./run-benchmarks.sh >> baseline-results.json done - name: Compare results run: | python compare-performance.py \ --baseline baseline-results.json \ --candidate candidate-results.json \ --threshold 0.05 \ --alpha 0.05 \ --output report.md - name: Comment on PR uses: actions/github-script@v7 with: script: | const fs = require('fs'); const report = fs.readFileSync('report.md', 'utf8'); github.rest.issues.createComment({ owner: context.repo.owner, repo: context.repo.repo, issue_number: context.issue.number, body: report });

Tiered Testing Strategy

Not all performance tests should run on every commit. Structure tests in tiers based on execution time and sensitivity:

Tier	When	Duration	Scope
Microbenchmarks	Every PR	2-5 min	Hot functions, critical algorithms
Component benchmarks	Every merge to main	10-20 min	API endpoints, query performance
Full load test	Pre-release	30-60 min	Complete system under production load
Soak test	Weekly	4-8 hours	Memory leaks, resource exhaustion

What to Benchmark

Benchmark the operations that users experience. Not every function needs a benchmark. Focus on:

Critical path operations: Login, search, checkout, page load—the flows that define user experience and map to SLOs
Hot paths: Functions called millions of times where small regressions multiply into large aggregate impact
Startup time: Application boot time affects deployment velocity and recovery from failures
Memory allocation: Heap size, allocation rate, and GC pause time under representative load
Database queries: Execution plans and response times for frequently-executed queries

Handling False Positives

Even with statistical rigor, false positives will occur. The response to false positives determines whether the team trusts the regression detection system or ignores it:

Easy override: Provide a mechanism to acknowledge and dismiss false positives in the CI check (e.g., a PR label like perf-acknowledged). Make overriding easy but tracked.
Track override rate: If more than 20% of regression flags are overridden, the detection sensitivity is too high. Adjust thresholds or improve measurement consistency.
Re-run capability: Allow engineers to re-run the performance check to verify whether the regression is reproducible. Transient regressions that disappear on re-run are environment noise, not code problems.

Tracking Performance Over Time

Beyond per-commit regression detection, maintain a longitudinal view of performance across releases. Plot key metrics over time to detect gradual degradation that falls below per-commit detection thresholds. A 1% regression per week is undetectable on any single commit but adds up to 50% degradation over a year.

Build a performance trend dashboard that shows:

Key benchmark results over the last 90 days
Trend lines with linear regression to project future performance
Annotations for major releases, infrastructure changes, and dependency upgrades
Comparison against SLO thresholds to show how much headroom remains

Key Takeaways

Performance regressions accumulate silently. Small regressions that nobody notices compound into severe degradation over months. Detect them at the commit level.
Use statistical methods (Mann-Whitney U test) for comparison, not simple thresholds. Require both statistical significance and practical significance (minimum effect size) to flag regressions.
Maintain consistent baselines on dedicated hardware. Rolling baselines adapt to codebase evolution; tagged baselines provide stable reference points.
Integrate performance gates into CI/CD with tiered testing: microbenchmarks on every PR, component benchmarks on merge, full load tests pre-release.
Track performance longitudinally across releases to detect gradual degradation below per-commit detection thresholds.