CPU Performance issue post upgrade from 1.9.0

[mrblonde91]

So recently upgraded Otel in .net 9 from 1.9.0 to the 1.11.1 and have started seeing a performance degradation that seems to center around traces. We're running our application on ECS in EC2, we send traces, metrics etc to the collector on the ec2 instance.

So the basic issue we encounter post upgrade is with no sampling, our cpu often tends to max out at 100% on the ECS container.(used hit 70% max) I narrowed the cause down to traces after sampling the data down to 10% and it became healthy again. Anyone else encountered anything similar post upgrade? I'm guessing there's some feature enabled rather than an actual bug. It generally takes about an hour for the issue to fully manifest.

Sample of our tracing config is below.

        options.AddSource(CustomTraces)
            .SetResourceBuilder(rb) 
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddConnectorNet()/mysql
            .AddRedisInstrumentation()
            .ConfigureRedisInstrumentation(instrumentation =>
            {
                instrumentation.AddConnection(readerConnectionMultiplexer);
                instrumentation.AddConnection(writerConnectionMultiplexer);
            })
            .AddAWSInstrumentation()
            .AddOtlpExporter(otlpOptions => { otlpOptions.Endpoint = new Uri(endpoint + "/api/v1/traces"); });

[TimothyMothra]

Hi @mrblonde91, there's a lot of components in your config shown.
If you can narrow this down to which components and which version, we would be able to take a look.

[mrblonde91]

Apologies on the delay on this. I omitted our filtering on traces and this appears to be the cause of the cpu spike. Basically we've got 3 endpoints that we're omitting on aspnetcoreinstrumentation. The filtering seems to be cause of the cpu implosion. The most likely thing I'm thinking is we're hitting some exceptions with this logic that's blowing up the cpu.

            .AddAspNetCoreInstrumentation(opt =>
            {
                opt.Filter = (context) =>
                {
                    if (context.Request.Path.StartsWithSegments(new PathString("/health")) 
                        || context.Request.Path.StartsWithSegments(new PathString("/secondaryendpoint"))
                        || context.Request.Path.StartsWithSegments(new PathString("/thirdendpoint")))
                    {
                        return false;
                    }
            
                    return true;
                };
            })