Can we know whether a profiler is accurate?
4 comments
·October 15, 2025comex
scottgg
Do you have a source for “with very little observer effect”? I don’t know better, it just seems like a big assumption the CPU can emit all this extra stuff without behaving differently.
satisfice
In the early nineties I was test manager of the Borland Profiler. I didn’t supervise the tester of the profiler closely enough, and discovered only when customers complained that the profiler results were off by a quarter second on every single measurement reported.
It turns out that the tester had not been looking closely at the output, other than to verify that output consisted of numbers. He didn’t have any ideas about how to test it, so he opted for mere aesthetics.
This is one of many incidents that convinced me to look closely and carefully at the work of testers I depend upon. Testing is so easy to fake.
dboreham
In my experience a very large proportion of all automated testing is like this if you go poking into what it does.
Another option is to use the "processor trace" functionality available in Intel and Apple CPUs. This can give you a history of every single instruction executed and timing information every few instructions, with very little observer effect. Probably way more accurate than the approach in the paper, though you need the right kind of CPU and you have to deal with a huge amount of data being collected.