Disable metrics buckets

[flaviostutz]

Only more advanced users of Prometheus metrics really uses buckets/histograms. By removing metrics buckets, we will reduce in over 60% the amount of processing and byte output in our metrics endpoint with no loss in quality.

Currently we have something like this for each metrics/label combination:

request_seconds_bucket 1
request_seconds_bucket 2
request_seconds_bucket 3
request_seconds_bucket 4
request_seconds_bucket +inf
request_seconds_sum
request_seconds_count
response_size_bytes

By removing buckets, we would reduce to something like

request_seconds_sum
request_seconds_count
response_size_bytes

Proposal

• Disable buckets output by default (will only generate _sum and _count metrics for each request)
• Only enable buckets if configured explicitly using existing param "buckets"

<init-param>
    <param-name>buckets</param-name>
    <param-value>0.1,0.3,2,10</param-value>
</init-param>

[gilliardmacedo]

@flaviostutz I agree with the topic but I'm concerned about solution: We have a big brother protocol implementation and the protocol defines this metrics. Default use of servlet-monitor will become "not-big-brother-compliant". Maybe a protocol discusion? Besides that, a present servlet-monitor user will lose features when simply upgrading lib(and without protocol upgrade)?

[flaviostutz]

I see your worries. Let’s discuss the bucket removal in a big brother repo issue. The “default” behavior may be discussed there too. Regarding to compatibility with users already using it, I think the community is still very young and there is space for this kind of change in spec, what do you think?

…

Sent from my iPhone

On 12 May 2021, at 17:25, Gilliard Macedo ***@***.***> wrote:  @flaviostutz I agree with the topic but I'm concerned about solution: We have a big brother protocol implementation and the protocol defines this metrics. Default use of servlet-monitor will become "not-big-brother-compliant". Maybe a protocol discusion? Besides that, a present servlet-monitor user will lose features when simply upgrading lib(and without protocol upgrade)? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[CarlosPanarello]

Maybe in this case we need to add a new metric, like simple_ request_seconds. With this we can use both metrics in cases that we need a histogram and some cases we need just a simple count with average time info.
Because if we change this the dashboards created expecting a histogram will be broken.

[flaviostutz]

One can use the same request_seconds_sum and request_seconds_count metric with or without using buckets, as buckets adds other metrics suffixed by _bucket. I don’t think we have such a great deal with current clients that would require adding this entropy to the metrics. If we disable buckets by default, clients will see their “histogram” panels stop working so that now they will only need to activate buckets on init-param. By experience very little people know really how to use buckets, so very few uses it. And if they use, they will know how to fix, I think. I think this is easy to detect and to fix.

…

Sent from my iPhone

On 13 May 2021, at 07:38, Carlos Eduardo Panarello ***@***.***> wrote:  Maybe in this case we need to add a new metric, like simple_ request_seconds. With this we can use both metrics in cases that we need a histogram and some cases we need just a simple count with average time info. Because if we change this the dashboards created expecting a histogram will be broken. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[CarlosPanarello]

My suggestion is to have both metrics in the same time in the project. So you can choose two flavors of metrics with or without buckets, default it will be using without. For this we have to define another name for metric, because of this i said to use simple_ request_seconds.

[flaviostutz]

We don't need both at the same time because buckets are only additional metrics (suffixed by "_bucket") to the same base metrics name.

For example, without buckets, it would be:
request_seconds_count
request_seconds_sum

When buckets are enabled, it will have those two metrics and, additionally:
request_seconds_bucket

So if we use two different names we won't have this kind of optimization in the number of metrics and will duplicate the base metrics (those suffixed by _count and _sum), which would be an unnecessary overhead.