Metrics monitoring for endpoints and upstreams

[dipack95]

Hi @andydunstall -- is it possible to add a prometheus metric for each node to list the specific endpoints connected to it, along with a corresponding upstream count? I really appreciate the ability to query the cluster's status as a whole using the /status/cluster/nodes and /status/cluster/nodes/:nodeId endpoints, and would like to able to incorporate them into our existing metrics gathering setup without having to write a cron function to poll the cluster periodically

[andydunstall]

is it possible to add a prometheus metric for each node to list the specific endpoints connected to it

Hmm, you mean include the endpoint ID label in the prometheus metric?

I'm worried that would be too high cardinality as Piko could have 1000s of endpoints which becomes impractical (such as hosted Prometheus solutions charge per active series)

So i'm not sure prometheus is the right fit for that use case?

[dipack95]

I'm worried that would be too high cardinality as Piko could have 1000s of endpoints which becomes impractical

Okay yeah that's what I suspected was the reason here!

For context, we're currently using Piko to monitor a single endpoint per-deployment. We expect there to be a few hundred such deployments at any given time, and so we're not too worried about the cardinality of the metrics themselves, hence the suggestion to potentially include it in the Prometheus metrics themselves

[andydunstall]

You could always host you're own prometheus endpoint that queries /status? It should be pretty simple to setup?

We expect there to be a few hundred such deployments at any given time, and so we're not too worried about the cardinality of the metrics themselves

Sure, though of course adding the metric affects everyone else using Piko (including us running Piko at DragonflyDB). So I'll close this issue for now

[dipack95]

Thanks @andydunstall, that's what we're going to do here!