Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert benchmarks to criterion. #23

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

justinlatimer
Copy link

Allows benching on stable. Statistics are better, but still quite a lot of variance in the timings from run to run.

@zslayton
Copy link
Owner

Thanks for the PR! I'm not very familiar with criterion -- I'll try to take a look at this over the weekend.

Statistics are better, but still quite a lot of variance in the timings from run to run.

If it's easy, could you post a before/after of the benchmark output so I can get a better idea of what you mean?

@justinlatimer
Copy link
Author

criterion tracks changes across runs - the benchmarks for lifeguard vary a lot across runs (on my machine anyway) that criterion detects the variance as statistically signification.

allocation standard     time:   [1.0051 ms 1.0405 ms 1.0828 ms]
                        change: [+4.4991% +7.4272% +10.486%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 14 outliers among 100 measurements (14.00%)
  5 (5.00%) high mild
  9 (9.00%) high severe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants