handler queue full

[eric-engberg]

What's wrong?

I'm running alloy as a daemonset. When an alloy pod starts it's fine for awhile but after about 5 minutes or so the pod will start spamming an error about the handler queue being full. This happens on all pods. Can't find any info online about what this or how to fix it.

Steps to reproduce

Install alloy v1.6.1 as a daemonset. Observe errors occuring after about 5 minutes.

System information

EKS Bottle Rocket - Linux grafana-alloy-2fwjc 6.1.124 #1 SMP PREEMPT_DYNAMIC Sat Jan 25 00:17:27 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Software version

Alloy v1.6.1 (Chart 0.11.0)

Configuration

Logs

ts=2025-03-05T19:27:15.874814205Z level=warn msg="handler queue full, dropping message (8) from=10.71.159.101:55202" service=cluster subsystem=memberlist

[thampiotr]

Hi, thanks for raising this. I have a few questions that would help investigate as we couldn't repro.

How many instances do you run?
Do you need clustering to be enabled in your DaemonSet at all?
Do you use components that have clustering_enabled set to true?
If yes, which ones?

[eric-engberg]

One cluster has 80 instances currently and the other has 196. I asked about this error a few weeks ago in slack and was told that the memberlist should be able to support way more than that.

I currently only have alloy deployed as a daemonset. Clustering is enabled on prometheus scrapes and service monitors. Those are the only 2 things with clustering enabled currently.

[thampiotr]

Do you experience any symptoms other than the log line? Perhaps it's unnecessarily on warn level. But we'd need to verify this to be sure.

[eric-engberg]

Not that I'm aware of. It seems to work fine as far as I can tell.