Benchmarks Issues

[wilwade]

Came across some issues with how we are running benchmarks with @enddynayn

1. The frequency-bench chain is enabled for the frequency feature, but should only be enabled for runtime-benchmarks feature
2. When we are running the benchmarks in run_benchmarks.sh it is running with features=runtime-benchmarks,frequency-lint-check. Should it instead be running with features=runtime-benchmarks,frequency?
   • Would this cause issues with non-mainnet pallets?
   • Perhaps it should run this way for release only?
3. node/service/src/chain_spec/frequency.rs benchmark_mainnet_config should be limited to the runtime-benchmarks feature

[wilwade]

@aramikm can you verify that these are correct actions from your knowledge?

[aramikm]

Looks fine to me. One other thing to consider is that we run benchmarks 2 ways

1. As release to generate correct weights
2. As tests (when writing and debugging)

The 2nd use-case should also be taken care of if not already.

[aramikm]

Aside form pallets inclusion or exclusion, using frequency feature or not would have effect if we have feature based logic in our pallets that might alter the weight of benchmarks. I don't think we have anything like that