Values in Grafana seem inflated after some time, possibly after Thanos compaction.

[luuktrs]

When requesting metrics in Grafana, data after some days seems quite inflated.

• When the same data is requested another day, the result of the query can be quite inflated. See screenshots where earlier the data seems normal but when requested another day it is suddenly inflated.
• The problematic results always seem to start occurring around 0200 in the morning, but the actual day changes. One day all results younger than 32 hours are correct(consistent with earlier results from the same query), another day all results younger than 6 days are correct
• There aren't any very specific error messages in Thanos' logging
• When the metric in question has few data points, the result graph seems to always be correct, but when there's a lot of data in a metric, the resulting graph shows this problem

You can see on the screenshots provided. Same timeframe, but suddenly the data starts to inflate multiple times over.

The query executed:
sum by (datname,usename) (cnpg_backends_total{namespace="$namespace",release="$release",pod=~"${cluster}-.*",usename!="cnpg_pooler_pgbouncer",datname!="postgres",usename!="postgres"})

It seems to resemble what happens in #922 but this should be fixed years ago.

I dont see any indication in the parameters of the various components. The timeframe doesnt match up to any parameters mentioned.
Store gateway:
--max-time=-4h
--min-time=-1d

Receiver:
--tsdb.retention=8h

Compactor:
--retention.resolution-raw=14d
--retention.resolution-5m=90d
--retention.resolution-1h=2y

Is there anyone who knows what the problem could be?

[dosubot[bot]]

Hey there, @luuktrs! I'm here to help you with bug fixes, questions, and becoming a contributor. Let's tackle this issue together!

I found a similar unsolved discussion titled "[Thanos Query] Query latency rises every 2 hours" which might be related to your issue, although it focuses on query latency rather than data inflation ^[1]. Additionally, there are two open issues that might be relevant:

1. "Problem with receiver and store" ^[2].
2. "Receive: compaction failure causing query returning irrelevant time series" ^[3].

Regarding your issue, it seems like the problem might be related to how Thanos handles downsampling and aggregation. Thanos creates downsampled blocks at different resolution levels, and the query resolution might not align with these downsampling intervals, leading to inflated data in Grafana. The default downsampling intervals are 5 minutes and 1 hour, with downsampling starting at 40 hours and 10 days, respectively ^[4][5].

To troubleshoot, you can try the following:

1. Test the queries directly in the Thanos query UI, selecting different levels of downsampling (raw, 5m, 1h) to see if the issue persists.
2. Disable auto-downsampling to observe if the metrics appear more accurate.
3. Ensure that the query resolution aligns with the downsampling intervals to avoid discrepancies.

These steps might help in identifying if the issue is related to downsampling and aggregation.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}