The metrics refresh time might be much larger than the refreshMetricsInterval

[spacewander]

In the current implementation, the pod metrics get refreshed for each refreshMetricsInterval + the time of refreshMetrics:
https://github.com/kubernetes-sigs/llm-instance-gateway/blob/918960cf7f4b2d53d912e6a08b7378f20d20bef7/pkg/ext-proc/backend/provider.go#L82-L87

As the fetchMetricsTimeout is 5s, when the refresh is aborted because of the timeout that happened in one of the requests, the time of refreshMetrics will be 5s. This time is much larger than the refreshMetricsInterval. As a result, stale info would be used for seconds.

One of the solution may be using a separate refresh loop per pod, so a timed-out pod metrics refresh request would not affect other requests. And we may drop the pod which can't provide up-to-date metrics.