Adding global tags for metrics

[EugeneKrapivin]

I'm currently at the stage of finding all the missing pieces to have a comparable reporting "feeling" we are currently having.

Currently all metrics reported from our services have some default tags/dimensions, like: dc, environment, service name. We aren't recording them manually every time we record a metric, but rather set those values as defaults at setup time. this really help since when querying we sometimes have metrics from multiple services, multiple service versions, multiple environments and even at time multiple datacenters in the same TSDB.

how would we go around adding those tags to all metrics exported from the service? I thought that adding those as attributes to the resource would do the trick

metricsBuilder.ConfigureResource(resourceBuilder =>
{
    resourceBuilder
        .AddService("ClickService")
        .AddAttributes(new[]
        {
            new KeyValuePair<string, object>("dc", Environment.GetEnvironmentVariable("dc") ?? "us1"),
            new KeyValuePair<string, object>("env", Environment.GetEnvironmentVariable("env") ?? "local"),
        })
        .AddTelemetrySdk();
});

however, I was wrong.
I would assume that there would be a way to set default tags on a Meter so that the process of reporting default tags is not repetitive on each Add/Record call.

[cijothomas]

Resource is what you need to model environments, service name etc. Is that not working for you? Could you elaborate more on why is it not working?

[EugeneKrapivin]

After researching a bit further and having some time to pounder, I still feel that there is a gap between the spec and real world use.
For example: I have a system with a couple of hundreds of services, each having multiple instances, some deployed in k8s and some are still deployed as processes in nomad. It is a common use-case to have my services report as part of the metric labels information like the dc, env, region, service name, service version so that we could easily filter and query the data, especially when debugging an issue in real-time. This is even more prominent when using contrib instrumentations, for which I have no control over labels.

What would be the suggested methodology to achieve that at this point in time?

From reading the spec a bit "Producers SHOULD prevent the presence of multiple Metric identities for a given name with the same Resource and Scope attributes." (source: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/data-model.md#opentelemetry-protocol-data-model-producer-recommendations). Doesn't that imply that metrics aggregation (and reporting I would assume) should take in accounts the resource attributes?

Thanks,
Eugene.

[cijothomas]

After researching a bit further and having some time to pounder, I still feel that there is a gap between the spec and real world use.
For example: I have a system with a couple of hundreds of services, each having multiple instances, some deployed in k8s and some are still deployed as processes in nomad. It is a common use-case to have my services report as part of the metric labels information like the dc, env, region, service name, service version so that we could easily filter and query the data, especially when debugging an issue in real-time. This is even more prominent when using contrib instrumentations, for which I have no control over labels.

What would be the suggested methodology to achieve that at this point in time?

These should be modelled as Resource by the application owner. It should not be reported as attributes/tags by any instrumentation library.

[cijothomas]

Doesn't that imply that metrics aggregation (and reporting I would assume) should take in accounts the resource attributes?

No.

[EugeneKrapivin]

These should be modelled as Resource by the application owner. It should not be reported as attributes/tags by any instrumentation library.

However, resource Attributes and Service information is not passed along (at least to Prometheus) making them a bit redundant, since they cannot be really used to filter data points. What are the use of those Resource, Service and attribute definitions if they aren't externalized by the Exporter?

[cijothomas]

What are the use of those Resource, Service and attribute definitions if they aren't externalized by the Exporter?

Its an exporter issue only and must be fixed by the exporter itself.
The official OTLP Exporter supports it.
The official Prometheus exporter would also support it. (The spec is experimental for that).
For any other exporter, please file an issue with the exporter authors.

[jaaywags]

I found this solution to be what I needed.

I had dotnet sending attributes the same exact way, but

The resource attributes to metrics label conversion is disabled by default

In your open telemetry collector config file, you should have an exporter for prometheus. Update it to look like this.

exporters:
    prometheus:
      endpoint: "0.0.0.0:8889"
      resource_to_telemetry_conversion:
        enabled: true

[cijothomas]

https://prometheus.io/docs/guides/opentelemetry/#promoting-resource-attributes
Prometheus natively supports Resource now as well.