High memory usage with OpenTelemetry integration

[danpaul88]

I'm unsure if this is a bug or an artifact of how I'm using Sentry, so rather than creating an issue I though I'd check to see if anyone has come across this before...

When using Sentry.OpenTelemetry (v3.41.3) with OpenTelemetry.Instrumentation.AspNetCore (v1.6.00-beta.3) and OpenTelemetry.Instrumentation.Http (v1.6.00-beta.3) I'm seeing what appears to be a memory leak in the SentrySpanProcessor

SentrySpanProcessor is holding ~700mb of the ~900mb of memory in this service, which was idle at the time of the memory dump and had been running for ~35 days, which suggests it's slowly accumulating more and more spans in SentrySpanProcessor._map.

Looking at a random selection of the spans, they all seem to be System.Net.Http.HttpRequestOut spans

Does anyone have any ideas why we're accumulating so much stuff in the SentrySpanProcessor and whether there's anything we can do to clear it out, short of periodically restarting the services to reclaim memory (not ideal! )

[jamescrosswell]

Hm, normally those _map entries would be removed here:

sentry-dotnet/src/Sentry.OpenTelemetry/SentrySpanProcessor.cs

Line 198 in 818e404

_map.TryRemove(data.SpanId, out _);

Are you able to simulate the load etc. required to observe this with Diagnostic logging turned on? So initialize Sentry with:

   options.Debug = false;
   options.DiagnosticLevel = SentryLevel.Debug;

I'm wondering if there are any errors in the SpanProcessor.OnEnd method (that would be preventing items from being removed from the _map)... The only other thing I could think of is Activities that get started but not finished, but then you'd see loads of those in your memory dump as well (not the case, I'm assuming).

[danpaul88]

Thanks for getting back to me so quickly. I'm actually on annual leave this week so I don't have the code in front of me, but from memory there weren't any activities in the memory dump - I'll double check on Monday though to be sure.

I spent a few hours on Friday trying to replicate the issue in a smaller test case without success so far, so it's clearly some kind of edge case the service in question is triggering - I'll continue trying to replicate it in a simpler test case on Monday and see if I get anywhere.

I'll also set up the original service that has the issue to just the debug options you've suggested and get back to you with the results 👍

[danpaul88]

I can confirm there are not large numbers of Activity objects in the memory snapshot;

As far as I can see, the only types with large numbers of instances are related to the items stuck in the span processors _map - not sure if the following list of types with lots of instances will be useful, but perhaps it will highlight any unexpected allocations that could be causing the problem.

I'm in the process of setting up the debug-enabled service instance and will get back to you with the results of that, just wanted to drop this initial feedback into the thread whilst I remember it :)

[danpaul88]

After much experimenting, I've figured out what appears to be the cause of the issue. It looks like spans that are filtered using the OpenTelemetry FilterHttpRequestMessage - e.g.;

[TracingProviderBuilder].AddHttpClientInstrumentation(options => options.FilterHttpRequestMessage = ShouldInstrumentHttpRequest)

Are causing TransactionTracer instances to be created inside SentrySpanProcessor._map that never get cleaned up.

I've uploaded a service that reproduces the issue to a repo - I've been dipping in and out of this trying different approaches to reproduce the issue so please excuse any messy code :)

[jamescrosswell]

After much experimenting, I've figured out what appears to be the cause of the issue. It looks like spans that are filtered using the OpenTelemetry FilterHttpRequestMessage - e.g.;

[TracingProviderBuilder].AddHttpClientInstrumentation(options => options.FilterHttpRequestMessage = ShouldInstrumentHttpRequest)

Are causing TransactionTracer instances to be created inside SentrySpanProcessor._map that never get cleaned up.

Thanks heaps @danpaul88 ! I was able to reproduce this locally and I've created an issue to track this:

• Http.HttpRequestOut OpenTelemetry spans not finished if filtered #3197

Now to fix it (hopefully)...

[danpaul88]

No problem at all! Good to hear the repo case worked for you too, I'll look forward to your fix being merged and released 👍