I asked five times and got one answer
PageSpeed Insights caches its result and hands it back, so running it five times and taking the median can give you a single measurement wearing five hats. Here's how I found out, and the second version of the problem that walked straight past my first fix.
I pushed a fix to my website on the 30th of July, watched the deploy go green, and measured the page. The number came back almost exactly where it had been before I touched anything.
That's a particular sinking feeling and I know it well enough by now, you go straight to assuming you fixed the wrong thing. So I read the diff again. Then I checked I was measuring the right URL, which I was, twice, because that's usually what it is. Then I stopped looking at the score and looked at the raw response underneath it, and there it was. Two of my three runs carried a timestamp from before the deploy.
They weren't measurements of the page I'd just shipped. They were measurements of the page I'd replaced, handed back to me as if they were new!
The tool did nothing wrong
Turns out, Google only actually re-analyses a given URL about once a minute. Ask again inside that window and you get the stored copy of the last analysis, with its original timestamp still attached. That's the part that saves you, but only if you happen to look.
Nothing about that is unreasonable though. Running Lighthouse costs Google real money, the API costs me nothing, and caching is the obvious answer to a free service that people hammer. I'd have done the exact same thing.
The trouble is what it does to the advice everyone gives, mine very much included. One Lighthouse run is noise. Total Blocking Time can swing threefold on a page nobody has touched, so the standard, sensible, repeated everywhere guidance is, run it a few times and take the middle number.
That's just become a problem, as it only works if the runs are independent, which clearly they aren't. Hand a median the same run three times and it will faithfully report that run, with all the authority of an average and none of the substance behind it. My median wasn't wrong exactly, it counted the votes correctly, it was just unaware that two of the three voters were the same person.
The second opinion who turned out to be the same doctor
You'll know the shape of this from anywhere else in life. You get a diagnosis you don't like, so you ask somebody else, and if you're still unhappy you ask a third. The comfort comes from the independence. Three people who've never met, all saying the same thing, that's worth listening to.
Now imagine you find out afterwards that the second and third were the first one again, same diagnostician phoning you back from a different room. Nothing anybody said was untrue. You just get one opinion with the confidence of three. Think about it, that's worse than having one opinion and knowing it.
That was my score. Three calls, one measurement.
The fix, and the version of the problem that walked past it
The first fix is the obvious one, throw away any run whose timestamp you've already seen. It's cheap, it's exact, and it caught the case that started all this.
Then on the 1st of August I ran eleven measurements against a single URL and got four apparently distinct analyses out of them, and something about two of the four bothered me. They agreed on First Contentful Paint to thirteen decimal places. The same number twice, "1686.8756582616209" ms.
Two independent Lighthouse runs don't agree to the femtosecond (yeah, I had to look that up), they can't possibly. It was one analysis served to me twice under two different timestamps. The plot thickens.
Turns out, because Google runs more than one backend, and those backends don't share a clock, the copy came back wearing a timestamp new enough to sail past my check.
What makes that one worth writing down isn't the leak, it's which way round it fooled me. Two runs agreeing is normally the thing you want to see. Agreement is evidence. Here the agreement was the giveaway and, if those two numbers had been a bit less absurdly identical, I'd have read them as corroboration. That would have left me feeling better about a figure that deserved less confidence, not more.
The second rule came out of that. Drop a run when everything it measured is identical to a run you already have. You can't tell a genuine repeat from a replay, so count them as one, which understates your sample rather than overstating your certainty. If I have to be wrong, I'd rather be wrong in the direction that makes me go and look again.
The tool now says how many genuinely distinct analyses it managed to collect, every time. A median of five, that was really a median of one, has to say so.
Asking harder doesn't get you there
The instinct once you understand the cache is to poll faster, and I measured that too, because instincts are cheap.
| Gap between asks | Calls made | Distinct analyses | Five distinct after |
|---|---|---|---|
| 5s | 60 | 9 | 136s |
| 10s | 30 | 9 | 144s |
| 15s | 20 | 8 | 149s |
Sixty calls to get nine real measurements. Polling three times as hard buys you thirteen seconds and spends three times the quota to do it, because the thing you're waiting for isn't the network, it's Google deciding to look at your page again. The cost of an honest measurement here is time, and there's no way to pay it in requests instead.
I settled on 15 seconds, which is the bottom row, and the reason it's the default is that it's the one that finally stops pretending.
While we're at it, the lab isn't your users
A different, but important, point to consider (and the one I'd put money on being the most useful to most people reading this).
Look at this. These two were both true of the same page on the same afternoon.
| LCP | Verdict | |
|---|---|---|
| Lighthouse lab, mobile | 21.36 s | performance 27 out of 100 |
| Real Chrome users, 28-day p75 | 1.20 s | FAST |
That's the BBC home page. By the lab it's a catastrophe, heads should roll, cue the gnashing of teeth! Now look at what actually happened, to real people, speed.
Neither number is lying. The lab result is one simulated run on a throttled connection and a deliberately slowed processor, built to be pessimistic so that problems show up while you're still in a position to fix them. The field result is Chrome telling you what actually happened to real visitors over 28 days, at the 75th percentile, which is a different question with a different answer. The failure isn't in either measurement, it's in quoting one of them as though it settled the other.
What I actually changed
Deduplicating twice, once on the timestamp and once on the numbers, and reporting the count of distinct analyses next to the median so it can't hide. Fifteen seconds between polls. The spread quoted alongside the median everywhere, because a middle number with no range around it is the same dishonesty in a better disguise.
None of which makes the score more accurate, but it does make it more honest, which isn't the same thing and is the only one of the two I can actually deliver.
I'd totally got used to treating pagespeed results as a reading off a calibrated instrument. Truth is, it's closer to asking somebody who's already told you twice.