Dropping label to resolve a high cardinality metric series?

[mwalz0]

I see a lot of discussion online about this, but it seems they are all missing two points on block identification(q1) and metric collision/deduplication(q2).

Background: We have a metric "order_total" that was impacted by a high cardinality label "store_number" that added thousands of unique label values to the metric series. We have since removed the "store_number" label from the backend metric that Prometheus is scraping. However, when querying this metric in Thanos for the date ranges it was affected, we experience very high resource usage on the querier, enough to crash the querier entirely.

2 questions on this matter:

1. I have found that metric relabeling is possible using Thanos tools, however the command "thanos tools bucket rewrite" requires you to know the block ID that is affected. I assume there are multiple blocks that contain this order_total{store_number=".*"} metric series, but "thanos tools bucket inspect -l" will only show match on external labels of the block (not any underlying metric series labels). Is there any way to inspect further into these blocks and find which specific block IDs to issue the rewrites on?

2. While this metric was actively impacted, I tried removing the high cardinality aspect during Prometheus ingestion by adjusting the Prometheus scrape config to drop the "store_number" label and not ingest into Prometheus (and subsequently, Thanos). While testing this I ran into an issue with metric collision/deduplication in Prometheus, described in this Prometheus users group. Essentially, Prometheus was dropping what was now multiple identical metric series and not aggregating them all into one. After issuing the Thanos rewrite command, will I run into the same issue with having identical metric series colliding and Thanos attempting to deduplicate, rather than aggregate them?

Thanks in advance for any replies.

[dosubot[bot]]

@mwalz0 Hello there! I'm here to help you with your issue. I can assist you in solving bugs, answering questions, and becoming a contributor.

1. To identify specific block IDs that contain the order_total{store_number=".*"} metric series, you can use the thanos tools bucket inspect command with the --selector flag to filter blocks based on metric series labels. This allows you to inspect blocks more deeply than just external labels. Unfortunately, if this feature is not available, you might need to manually inspect the blocks or use a custom script to parse the block data for the specific metric series.

2. Thanos handles deduplication by merging samples from series with the same label set. After you issue the Thanos rewrite command, Thanos will merge series with identical labels by combining samples based on their timestamps. This process ensures that the resulting series contains a unified set of samples without duplicates, effectively handling the deduplication process ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[mwalz0]

@dosu I don't think this bot is correct in saying that bucket inspect with --selector flag filters on metric series labels. The only labels I find this can filter on is each block's external labels such as prometheus_replica and other custom external labels.

If the answer to question 2 is correct, I suppose the only solution to dealing with this high cardinality metric is to completely drop all metric series for the order_total for the time range it was affected? I guess this is a viable option because right now the metric is totally unusable (but we were really hoping for a way to strip the high cardinality label while still aggregating all the series we have collected, so it becomes usable again).

^ This goes back to question 1 of how do I identify the blocks that were affected if bucket inspect can only filter on external labels? I feel like there must be a way to inspect the metric series of blocks that have metrics matching order_total{store_number=~".*"} ??

[dosubot[bot]]

To address your concerns:

1. You are correct that the thanos tools bucket inspect command primarily filters based on external labels and does not directly allow filtering on metric series labels. However, Thanos provides functionality for deeper inspection of block contents through the TSDBStore methods like LabelNames and LabelValues, which can be used to query for label names and values within a specified time range. This can help identify metric series labels within blocks ^[1].

2. Regarding deduplication, Thanos does handle deduplication by merging samples from series with the same label set. If you decide to drop the high cardinality label using the thanos tools bucket rewrite command, Thanos will merge series with identical labels by combining samples based on their timestamps, effectively handling deduplication ^[2].

If you want to strip the high cardinality label while still aggregating the series, you can use the thanos tools bucket rewrite command with a relabel configuration to drop the store_number label. This approach allows you to modify the series labels without losing the data entirely. However, identifying the specific blocks affected by the high cardinality label remains a challenge without direct support for filtering on metric series labels. You might need to explore custom scripts or tools to parse block data for specific metric series.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[mwalz0]

So it sounds the only way I can fix this is by running a thanos tools bucket rewrite --rewrite.to-delete-config-file command on every single block for the time range this metric was affected. This seems cumbersome and extremely time consuming. If anyone has a better idea of how I can clean this metric of its high cardinality or delete it entirely, please let me know. I'm really surprised I haven't found anything online about anyone else running into the same problem.

[mwalz0]

Figured this out. For my setup, Thanos compactor would compact our metrics from sidecar into two week blocks, replicated 3x for each Prometheus replica we run. I believe this is the default block time range, but this is important to know for identifying your bad blocks. I wasn't too familiar with exactly how Compactor deals with block downsampling/retention so I was able to learn that.

1. Identified the blocks based on the time range I knew the metric was impacted.
2. I scaled compactor to 0 before I ran the bucket rewrites (not sure if it was necessary to stop compactor, but it would eventually halt due to overlap anyways later in instructions)
3. Ran
   thanos bucket rewrite --no-dry-run --delete-blocks --rewrite.to-delete-config-file *deleteconfig* --objstore.config-file *objstoreconfig* --id *individualBlockID*
   to delete the metric series that were in each block. I never was able to figure out how to actually "clean" and aggregate the data into one metric series with relabeling like I hoped to, I think this would be a great feature if someone added it but I can see how it would be difficult.

• Important note: thanos bucket rewrite will only work on blocks that HAVE NOT BEEN DOWNSAMPLED. This means that you should only need to run it on the blocks that have a "0s" resolution time. This took me a while to figure out as it's not really mentioned in the bucket rewrite documentation.

5. Now, even though I added the --delete-blocks flag, when I would go to start up compactor it eventually halted on "block overlap" for the source block and new block created from the command. I guess it starts processing the new block before it can process the deletes? To solve this, I just went into my object storage platform and manually deleted the source block from there.
6. Lastly, you'll also need to delete all the downsampled blocks from object storage that were created from the source block(s). These would also halt the compactor based on "block overlap" once the compactor starts back up. This is because the compactor will begin downsampling this newly written block once it starts back up, which is really convenient because that makes it so you only need to rewrite 1/3 of all blocks in object storage, and compactor does the rest.
7. Start compactor back up, and now you're clear of the problematic metric.

Hope this can help someone else, because it was a big headache for me until I got it working. 🦦