Methodology
How CodeGecko makes the call.
Every verdict comes from real visits to your production site and five rules, applied the same way to every release. No lab runs, no models, no guesses. Here are the rules, with the numbers.
Rule 1 — Like with like
Compare like with like.
Visits are grouped by release, route, device class and metric. Each release is compared only with the one before it.
- Device class comes from the browser’s own hint, or the viewport: under 768px is mobile.
- Routes are templates. IDs, slugs and numbers become placeholders like :id and :slug, so every product page counts together.
- Set your own groups with data-routes, or merge pages in the dashboard.
One group: same page, device and metric. Its p75 is compared with the same group in v1.2.7.
Rule 2 — Statistic
Take the 75th percentile.
Three in four visits were this fast or faster. It follows the slow tail without being thrown by one outlier, and it’s what Google uses for Core Web Vitals.
JavaScript errors are different
Ten LCP values, sorted (s)
- 11.80
- 21.95
- 32.10
- 42.20
- 52.35
- 62.50
- 72.65
- 82.90
- 93.30
- 103.60
Rule 3 — Enough visits
30 on both sides, or we wait.
A verdict needs at least 30 visits for the group in both the new release and the one before. While the newest release is short, it’s gathering data. The release before stopped getting visits when the new one replaced it, so if it’s short, the result is inconclusive for good, not a guess.
Rule 4 — Threshold
Small wobbles aren’t changes.
A change only counts when the p75 moves by more than the threshold. Each metric has a floor, so tiny values don’t flip verdicts on noise.
threshold = max(floor, 5% × previous p75)
LCP · product page
regressed- p75
- 2.63s → 3.13s
- threshold
- max(100ms, 5% × 2.63s) = 132ms
- change
- +500ms
INP · homepage
improved- p75
- 196ms → 157ms
- threshold
- max(20ms, 5% × 196ms) = 20ms
- change
- −39ms
Rule 5 — Verdicts
From metrics to a release verdict.
Each metric gets one of five verdicts. The release label is the first rule below that matches, worked out for one page and device at a time: / first, or your busiest page when there is no /.
- improved
Dropped by at least the threshold.
- regressed
Rose by at least the threshold.
- unchanged
Enough data, but moved less than the threshold.
- Gathering data
The newest release has fewer than 30 visits so far. More can still come.
- inconclusive
A side that was replaced by a newer release ended with fewer than 30 visits.
Verdicts aren’t health badges
Release label, first match wins
- 1All metrics short, some still gatheringGathering data
- 2All metrics short, none can get more visitsinconclusive
- 3Some improved and some regressedmixed
- 4Any regressedregressed
- 5Any still gatheringGathering data
- 6Any inconclusiveinconclusive
- 7Any improvedimproved
- 8Otherwiseunchanged
LoAF and page weight count once they have enough data, but never hold a release back while they wait: LoAF is only measured in Chrome and Edge. Load steps and third-party cost get their own verdicts without affecting the release label. The Release label column in Rule 4 shows which is which.
Read it the right way
What a verdict is, and isn’t.
Not a significance test
No p-values or confidence intervals. A verdict means a practical bar was cleared.
Not a cause
Traffic, networks or third-party scripts can move numbers. Treat a regression as a signal.
Route groups can hide things
A route like /products/:slug counts every product together, so one slow product or a shift in mix can hide in it.
Full page loads only
Soft navigations inside single-page apps aren’t measured yet.
See the rules on real releases.
