Benchmark visualizer: Venz

[webpro]

Hi there!

First of all, thanks a lot for building and maintaining hyperfine, it's a great and omnipresent tool.

Since I had a desire to visualize a range of benchmarks over time in various situations, I thought it might be useful to render charts in the browser for nice "dataviz". So one thing led to another, and an early "local-only" version to try is here: https://try.venz.dev. No data leaves the browser.

Use the flow/form to manage commands and provide titles and colors. Or drop JSON output directly onto https://try.venz.dev/chart (either drop the file, or paste JSON as text). Todo:

• Render "summary" in chart (to compare benchmarks and see which one is fastest etc)
• Support other output formats (eg CSV), maybe even also the CLI output itself
• Improve UX/accessbility

Feel free to provide any kind of feedback or suggest ideas. Better chart types, etc. etc. Anything.

Plenty more ideas I have. Already have a "remote storage" version working (local = IndexedDB, remote = SQLite), and I'm looking for some kind of "on-demand bare metal" cloud solutions to run benchmarks in an isolated and stable environment. End-users would then initiate such runs from the Venz UI, while also allowing more "prepare" and "build" commands, etcetera. I might also start support for other benchmark running tools.

For now, at least we have a public visualizer for hyperfine output. Here's hoping to make running benchmarks more fun and accessible to more people.

By the way, the name Venz is a portmanteau(?) of "bENchmark VisualiZer".

[webpro]

For anyone interested, I'm building this in public. Follow me on Bluesky for progress updates: https://bsky.app/profile/webpro.nl

[sharkdp]

Wow — this is really cool, thank you for sharing it here! Great idea and very nice execution.

For anyone interested, I'm building this in public

Is the project open source too?

Since I had a desire to visualize a range of benchmarks over time in various situations, I thought it might be useful to render charts in the browser for nice "dataviz".

Just to be sure: you have seen our visualization scripts at https://github.com/sharkdp/hyperfine/tree/master/scripts?

Support other output formats (eg CSV), maybe even also the CLI output itself

Personally, I don't see the need for this. JSON seems like the best fit here?

Feel free to provide any kind of feedback or suggest ideas. Better chart types, etc. etc. Anything.

I think the charts could use a bit more polish. I'd like to see axis labels (wow, I just noticed that it does have labels, but the contrast of the font is so low, I didn't see them), a legend (in the chart, not below it), etc.

One thing I would find very valuable personally would be to edit some of the chart properties interactively. This is something we can't easily do with the Python scripts. For example, I often run a benchmark, store the results, but forget to set a nice label. Being able to adjust the command names would be really neat. And it would be really cool to change colors in the chart view, not before.

Another thing that would be useful is support for parametrized benchmarks (see --parameter-scan and --parameter-list). These benchmarks often lend themselves well to interesting plotting opportunities.

[webpro]

Wow — this is really cool, thank you for sharing it here! Great idea and very nice execution.

Thanks! Much appreciated.

For anyone interested, I'm building this in public

Is the project open source too?

Not yet, but it will be.

Since I had a desire to visualize a range of benchmarks over time in various situations, I thought it might be useful to render charts in the browser for nice "dataviz".

Just to be sure: you have seen our visualization scripts at https://github.com/sharkdp/hyperfine/tree/master/scripts?

I glanced over them, but didn't actually try since I figured there wouldn't be too much interactivity and I wanted to play with Solid JS and D3. Also I'm not doing much Python so didn't fancy going through installing things. But I might still try, let me know if there's something of interest.

Support other output formats (eg CSV), maybe even also the CLI output itself

Personally, I don't see the need for this. JSON seems like the best fit here?

Yeah, no priority in supporting more than JSON. Was thinking it could be neat to pipe hyperfine results to clipboard so one can then paste into browser without even adding --export-json [file] but it would obviously miss out on individual data points.

Feel free to provide any kind of feedback or suggest ideas. Better chart types, etc. etc. Anything.

I think the charts could use a bit more polish. I'd like to see axis labels (wow, I just noticed that it does have labels, but the contrast of the font is so low, I didn't see them), a legend (in the chart, not below it), etc.

Do you mean in the browser or in the generated image? In the browser contrast should be very high. Added a screenshot below just in case. The images aren't really looked after yet, they've a transparent background, I figured for now that's flexible. There's also a simple legend you can toggle, but not too fond of it.

One thing I would find very valuable personally would be to edit some of the chart properties interactively. This is something we can't easily do with the Python scripts. For example, I often run a benchmark, store the results, but forget to set a nice label. Being able to adjust the command names would be really neat. And it would be really cool to change colors in the chart view, not before.

That has now been added. And an option to save that test as a new item. So you can approach in two ways.

Another thing that would be useful is support for parametrized benchmarks (see --parameter-scan and --parameter-list). These benchmarks often lend themselves well to interesting plotting opportunities.

Great. I already had something for that so let's just say it's on the TODO list (that's why the strategy only has a "default" option, there'll be another option for this)

[sharkdp]

Do you mean in the browser or in the generated image? In the browser contrast should be very high. Added a screenshot below just in case.

I was not using dark mode. Anyway, contrast is much better now after the update. Thanks. There is a new problem where the y axis labels seem to have another label behind them in gray?

That has now been added. And an option to save that test as a new item. So you can approach in two ways.

Very nice

Great. I already had something for that so let's just say it's on the TODO list (that's why the strategy only has a "default" option, there'll be another option for this)

👍

One additional feedback point: the toggle buttons look very nice, but I find the UX not really great. I think I'd prefer something were I could immediately select the option I want. And immediately understand what I'm changing without having to interpret the icons.

[sharkdp]

@webpro I moved this to discussions, as there is nothing really actionable in the ticket.

[webpro]

Apologies for dumping in the issues indeed, apparently I missed discussions are open on here.

Anyway, made some progress so here's an update:

• addressed the issues with "ghost" axis labels and unclear icons
• dark/light/high-contrast modes
• new hyperfine-parameter type (detects parameters in pasted JSON if /chart is the starting point); no support for step-size
• new simple chart types "line" and "bar"
• improved underlying data model, so the standard configuration type became the default, and hyperfine-default and hyperfine-parameter are basically extensions of that

[webpro]

btw, it tries to migrate the indexeddb to the new data model

so if there were customizations it should migrate those and the data is kept, but if you want to get the new examples you can delete the database and refresh the page in the browser's inspector:

[webpro]

Would you be interested in a PR to add this to https://github.com/sharkdp/hyperfine#json and/or https://github.com/sharkdp/hyperfine#integration-with-other-tools?

[sharkdp]

https://github.com/sharkdp/hyperfine#integration-with-other-tools?

Yes!