How to export trace data to prometheus ?

[jbarotin]

Hi,

We have set the span metrics-generator unfortunately it only exports metrics for span.

Is there a solution to get Traces metrics exported to Prometheus ? We are mostly interested in trace duration metrics.

[joe-elliott]

Metrics generator does not produce any trace level metrics. Some trace metrics would be very difficult to calculate as you would need to accumulate the trace in one place before creating the metric and make a guess about whether or not the trace was finished.

e.g. span count per trace, service count per trace, trace depth, total duration calculated as max end - min start ...

Other metrics would be simpler if you assume that an empty parent id indicates trace root.

e.g. root span name, root service name, total duration calculated as duration of root span ...

What kinds of metrics are you interested in? This is not an initiative we are interested in driving, but depending on the complexity of the proposal we may accept PRs here.

[jbarotin]

Thanks for the answer @joe-elliott, we want to create an indicator corresponding to the whole message processing time in our CEP architecture from producer to consumer. We already have a Prometheus with an hsitory of 6 months, and we don't want enable a such history on Tempo, so we are looking for a solution to feed our Prometheus with the trace duration.

[joe-elliott]

the fundamental issue here is when does Tempo give up on waiting for spans, consider the trace complete and record the full trace length? you'd have to store a map of trace ids to min start time and max end time which could get costly depending on how long you're willing to wait for spans and how many traces per second your system consumes.

I'm not opposed to work in this direction, but I'm not sure if it work past a certain scale due to memory cost of holding all that information.

[jbarotin]

I understand, in our case we have to make sure the spans are done (message processed & consumed), we'll trying building our own solution throught a custom script.