fix summary ratio and ratioSD values for baseline #2648#2988
fix summary ratio and ratioSD values for baseline #2648#2988timcassell merged 2 commits intodotnet:masterfrom
Conversation
This comment was marked as off-topic.
This comment was marked as off-topic.
|
@AndreyAkinshin Hey Andrey! It'd be great to have your thoughts on this PR. Thanks! |
AndreyAkinshin
left a comment
There was a problem hiding this comment.
This PR addresses a real user-facing problem — the baseline ratio deviating from 1.00 due to the all-pairs division approach — and doesn't introduce any regressions for non-baseline ratios. The fix is scoped and safe, so let's merge it.
Status update on the broader problem:
I'm working on a new statistical engine called Pragmastat. The latest version already includes Ratio and RatioBounds functions designed to replace the current BenchmarkDotNet ratio implementation. That implementation is quite flawed — when I originally wrote it, I wasn't aware of better solutions and picked an approach that felt reasonable at the time. After several years of research, I've arrived at a much better way to estimate ratios between benchmarks, which you can explore in the current Pragmastat.
The underlying research work is finished. What remains is implementing a few additional estimators and reworking the summary tables before it can be merged into BenchmarkDotNet. After that, we can completely drop the current ratio and statistics implementation, since it has multiple flaws and can be misleading in certain cases.
No ETA, but I hope to finish in the upcoming months. I've given similar estimates before, but this time I'm powered by an LLM-based coding agent, so progress has gotten a huge acceleration boost.
Bottom line: this PR is good to merge as-is. Further polishing of the current ratio logic isn't worth the investment — it will all be replaced by the Pragmastat-based engine.
Ratio and RatioSD summary values are fixed for baseline (1.0 and 0.0, respectively)