[grafana] some dashboards seems to work not properly

[phac008]

some data is missing, some dashboards only have monitoring namespace selectable

[jkleinlercher]

dashboards which have "no data":

• etcd dashboard: https://grafana-127-0-0-1.nip.io/d/c2f4e12cdf69feb95caa41a5a1b423d9/etcd?orgId=1&refresh=10s
• kubernetes api-server: https://grafana-127-0-0-1.nip.io/d/09ec8aa1e996d6ffcd6817bbaff4db1b/kubernetes-api-server?orgId=1&refresh=10s

etcd metrics missing which are used in the dashboard:

etcd_server_has_leader
etcd_server_leader_changes_seen_total
grpc_server_started_total
grpc_server_handled_total
etcd_mvcc_db_total_size_in_bytes
etcd_disk_wal_fsync_duration_seconds_bucket
etcd_disk_backend_commit_duration_seconds_bucket
etcd_network_client_grpc_received_bytes_total
etcd_network_client_grpc_sent_bytes_total
etcd_network_peer_received_bytes_total
etcd_network_peer_sent_bytes_total
etcd_network_peer_round_trip_time_seconds_bucket
process_resident_memory_bytes exists, but not for {job=~".*etcd.*"}

don't know if this is something k3d or kind specific ... investigate

[jkleinlercher]

etcd in k3d: i think it is due to the architecture of k3s/k3d why api-server and etcd are not accessible and maybe also don't expose those metrics by default. However, I found a good issue which might have the solution for this

k3s-io/k3s#425 (comment) (and the comments below)

will try that. Maybe it also fixes the apiserver dashboard

We will propbably need to set it with "--k3s-arg=--cluster-init --k3s-arg=--etcd-expose-metrics=true"

Only with cluster-init a etcd server is used ?!? investigating

[jkleinlercher]

kubernetes-sigs/kind#2695 (comment) could help for kind-Cluster

[jkleinlercher]

before fixing this on k3d (which is maybe also a not representative K8s architecture), I tested also on metalstack.cloud, if this problem persists: Also on metalstack.cloud the etcd dashboard prints "No data" ... investigating on metalstack

I found out that also in Gardener etcd Cluster is not part of the master node. In Gardener the control-plane of the shoot clusters is in the shoot control-plane namespace in the seed clusters:

https://github.com/gardener/gardener/blob/master/docs/concepts/etcd.md
https://github.com/gardener/gardener/blob/master/docs/development/monitoring-stack.md

So this is also not a standard architecture.
@phac008 lets have a chat when and how to tackle this issue.

[jkleinlercher]

I will check: Are those dashboards still in grafana and still don‘t work? If not, I close this ticket

[jkleinlercher]

those dashboards got replaced by new ones