gRPC Health Check must be implemented in the distributor

[kworkbee]

Is your feature request related to a problem? Please describe.
I want to pass traces (coming in from outside K8s cluster) to the distributor through AWS ALB with gRPC as backend protocol. In this case, ALB Health Check related annotations should be specified in Ingress but if the backend protocol is gRPC, health check method should also return the status code based on gRPC.
However, the status code currently returned by the /ready HTTP endpoint is an HTTP status code, so there is a problem that the gRPC-based ALB cannot be configured at the front-end of the distributor.

Describe the solution you'd like
A standard gRPC Health Endpoint (which conforms to standard proto) should be added into distributor.

[luvpreetsingh]

I was looking for the same and landed at this issue.

[luvpreetsingh]

So, is there no way of doing it as of now?

I am facing the same issue but on GCP.

[joe-elliott]

The gRPC service running on port 4317 is here: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver

I'm not sure we can add a healthcheck endpoint to it. I have been wanting to write our own otel receiver native into tempo for awhile now which would allow us to do this, but that's on the backburner.

[joe-elliott]

We do register gRPC health checks to the standard gRPC server in the distributor. Unsure if you can point the ALB to that:

tempo/cmd/tempo/app/app.go

Lines 197 to 201 in 3d50193

	grpc_health_v1.RegisterHealthServer(t.Server.GRPC(),
	grpcutil.NewHealthCheckFrom(
	grpcutil.WithShutdownRequested(shutdownRequested),
	grpcutil.WithManager(sm),
	))

[joe-elliott]

This issue appears relevant: open-telemetry/opentelemetry-collector#3040

[luvpreetsingh]

We do register gRPC health checks to the standard gRPC server in the distributor

@joe-elliott Would you be kind enough to share grpcurl for the distributor health check call?

[mapno]

We do register gRPC health checks to the standard gRPC server in the distributor

@joe-elliott Would you be kind enough to share grpcurl for the distributor health check call?

The server does not support reflection, so you'll need to have the healthcheck proto somewhere locally if you want to query it with grpcurl

grpcurl -v -plaintext -import-path <path> -proto google/grpc/health/v1/health.proto localhost:9095 grpc.health.v1.Health/Check

[luvpreetsingh]

Looks like GCP doesn't support GRPC health checks

https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-configuration#direct_health

[luvpreetsingh]

This is confusing to me. We can't use GRPC ingestion on Grafana Tempo with the current implementation. Am I missing something really basic or is tempo missing something?

[luvpreetsingh]

For the OTLP HTTP endpoint, is there any health check endpoint available in the distributor?

I can give the HTTP health check endpoint and trick the LB.

I tried using the readinessProbe as health check, which is https://3100/ready, but the issue is that doing HTTPS health check with GCP LB is tricky

[kworkbee]

We do register gRPC health checks to the standard gRPC server in the distributor. Unsure if you can point the ALB to that:

tempo/cmd/tempo/app/app.go

Lines 197 to 201 in 3d50193

grpc_health_v1.RegisterHealthServer(t.Server.GRPC(),
grpcutil.NewHealthCheckFrom(
grpcutil.WithShutdownRequested(shutdownRequested),
grpcutil.WithManager(sm),
))

Hi, @joe-elliott Thanks for your response. I missed the implementation of app.go, so I tried to port-forward traffic port (4317) and call grpcurl as below.

grpcurl -v -plaintext -import-path ./ -d '{"service":""}' -proto health.proto localhost:4317 grpc.health.v1.Health/Check                                                         

I saw that it's coming out as 'UNIMPLEMENTED', but I don't know if I called it correctly. (Sorry but I'm not used to gRPC and golang enough yet)

Resolved method descriptor:
// If the requested service is unknown, the call will fail with status
// NOT_FOUND.
rpc Check ( .grpc.health.v1.HealthCheckRequest ) returns ( .grpc.health.v1.HealthCheckResponse );

Request metadata to send:
(empty)

Response headers received:
(empty)

Response trailers received:
content-type: application/grpc
Sent 1 request and received 0 responses
ERROR:
  Code: Unimplemented
  Message: unknown service grpc.health.v1.Health

[joe-elliott]

Port 4317 is the OTel gRPC receiver. This service is not managed by us and I don't believe we can add a health check here.

By default port 9950 is the native Tempo gRPC receiver. That should have a health check server registered.

The linked OTEL issue above contains work arounds folks have used to get the OTel collector working with an ALB which may help you as well.