Spam of StreamResetExceptions

[david-ribeiro-sumup]

Describe the bug

We just migrated away from honeycomb open telemetry library to the opentelemetry java implementation.
We started getting spammed with multiple connection related exceptions:

• okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR
• java.net.SocketException: Broken pipe

Steps to reproduce

We started the service and the error logs are very frequent. We are using the library to export traces to Honeycomb using gRPC.

Expected behavior

No traces seem to be getting lost, however we get spammed with errors in the logs.

Actual behavior

There should be no errors in the logs.

Javaagent or library instrumentation version

v2.10.0

Environment

Docker image: eclipse-temurin:21-jre-alpine

Additional context

Some stack traces:

IllegalStateException
java.net.SocketException: Broken pipe
io.opentelemetry.exporter.internal.marshal.Serializer$RepeatedElementPairWriter in accept at line 609
io.opentelemetry.javaagent.shaded.io.opentelemetry.api.internal.ImmutableKeyValuePairs in forEach at line 94
io.opentelemetry.exporter.internal.marshal.Serializer in serializeRepeatedMessageWithContext at line 536
io.opentelemetry.exporter.internal.otlp.metrics.NumberDataPointStatelessMarshaler in writeTo at line 45
io.opentelemetry.exporter.internal.otlp.metrics.NumberDataPointStatelessMarshaler in writeTo at line 24
io.opentelemetry.exporter.internal.marshal.ProtoSerializer in serializeRepeatedMessageWithContext at line 239
io.opentelemetry.exporter.internal.marshal.Serializer in serializeRepeatedMessageWithContext at line 480
...

SocketException
Broken pipe
sun.nio.ch.SocketDispatcher in write0
sun.nio.ch.SocketDispatcher in write
sun.nio.ch.NioSocketImpl in tryWrite
sun.nio.ch.NioSocketImpl in implWrite
sun.nio.ch.NioSocketImpl in write
sun.nio.ch.NioSocketImpl$2 in write
java.net.Socket$SocketOutputStream in write

IllegalStateException
io.opentelemetry.exporter.internal.marshal.Serializer$RepeatedElementPairWriter in accept at line 609
java.util.IdentityHashMap in forEach
io.opentelemetry.exporter.internal.marshal.Serializer in serializeRepeatedMessageWithContext at line 513
io.opentelemetry.exporter.internal.otlp.logs.ResourceLogsStatelessMarshaler in writeTo at line 44
io.opentelemetry.exporter.internal.otlp.logs.ResourceLogsStatelessMarshaler in writeTo at line 28
io.opentelemetry.exporter.internal.marshal.Serializer$RepeatedElementPairWriter in accept at line 606
java.util.IdentityHashMap in forEach
io.opentelemetry.exporter.internal.marshal.Serializer in serializeRepeatedMessageWithContext at line 513
io.opentelemetry.exporter.internal.otlp.logs.LowAllocationLogsRequestMarshaler in writeTo at line 73
io.opentelemetry.exporter.internal.marshal.Marshaler in writeBinaryTo at line 23
io.opentelemetry.exporter.sender.okhttp.internal.GrpcRequestBody in writeTo at line 68
okhttp3.internal.http.CallServerInterceptor in intercept at line 62
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
okhttp3.internal.connection.ConnectInterceptor in intercept at line 34
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
okhttp3.internal.cache.CacheInterceptor in intercept at line 95
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
okhttp3.internal.http.BridgeInterceptor in intercept at line 83
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
okhttp3.internal.http.RetryAndFollowUpInterceptor in intercept at line 76
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
io.opentelemetry.exporter.sender.okhttp.internal.RetryInterceptor in intercept at line 91
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
okhttp3.internal.connection.RealCall in getResponseWithInterceptorChain$okhttp at line 201
okhttp3.internal.connection.RealCall$AsyncCall in run at line 517
java.util.concurrent.ThreadPoolExecutor in runWorker
java.util.concurrent.ThreadPoolExecutor$Worker in run
java.lang.Thread in run

Cause:

StreamResetException
stream was reset: NO_ERROR

okhttp3.internal.http2.Http2Stream in checkOutNotClosed$okhttp at line 646
okhttp3.internal.http2.Http2Stream$FramingSink in emitFrame at line 557
okhttp3.internal.http2.Http2Stream$FramingSink in write at line 532
okio.ForwardingSink in write at line 29
okhttp3.internal.connection.Exchange$RequestBodySink in write at line 223
okio.RealBufferedSink in emitCompleteSegments at line 256
okio.RealBufferedSink$outputStream$1 in write at line 118
io.opentelemetry.exporter.internal.marshal.CodedOutputStream$OutputStreamEncoder in doFlush at line 632
io.opentelemetry.exporter.internal.marshal.CodedOutputStream$OutputStreamEncoder in write at line 608
io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil in writeUtf8 at line 440
io.opentelemetry.exporter.internal.marshal.StatelessMarshalerUtil in writeUtf8 at line 426
io.opentelemetry.exporter.internal.marshal.ProtoSerializer in writeString at line 163
io.opentelemetry.exporter.internal.marshal.Serializer in serializeStringWithContext at line 235
io.opentelemetry.exporter.internal.otlp.StringAnyValueStatelessMarshaler in writeTo at line 29
io.opentelemetry.exporter.internal.otlp.AnyValueStatelessMarshaler in writeTo at line 36
io.opentelemetry.exporter.internal.otlp.AnyValueStatelessMarshaler in writeTo at line 24
io.opentelemetry.exporter.internal.marshal.Serializer in serializeMessageWithContext at line 294
io.opentelemetry.exporter.internal.otlp.logs.LogStatelessMarshaler in writeTo at line 39
io.opentelemetry.exporter.internal.otlp.logs.LogStatelessMarshaler in writeTo at line 25
io.opentelemetry.exporter.internal.marshal.ProtoSerializer in serializeRepeatedMessageWithContext at line 239
io.opentelemetry.exporter.internal.otlp.logs.InstrumentationScopeLogsStatelessMarshaler in writeTo at line 37
io.opentelemetry.exporter.internal.otlp.logs.InstrumentationScopeLogsStatelessMarshaler in writeTo at line 21
io.opentelemetry.exporter.internal.marshal.Serializer$RepeatedElementPairWriter in accept at line 606
java.util.IdentityHashMap in forEach
io.opentelemetry.exporter.internal.marshal.Serializer in serializeRepeatedMessageWithContext at line 513
io.opentelemetry.exporter.internal.otlp.logs.ResourceLogsStatelessMarshaler in writeTo at line 44
io.opentelemetry.exporter.internal.otlp.logs.ResourceLogsStatelessMarshaler in writeTo at line 28
io.opentelemetry.exporter.internal.marshal.Serializer$RepeatedElementPairWriter in accept at line 606
java.util.IdentityHashMap in forEach
io.opentelemetry.exporter.internal.marshal.Serializer in serializeRepeatedMessageWithContext at line 513
io.opentelemetry.exporter.internal.otlp.logs.LowAllocationLogsRequestMarshaler in writeTo at line 73
io.opentelemetry.exporter.internal.marshal.Marshaler in writeBinaryTo at line 23
io.opentelemetry.exporter.sender.okhttp.internal.GrpcRequestBody in writeTo at line 68
okhttp3.internal.http.CallServerInterceptor in intercept at line 62
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
okhttp3.internal.connection.ConnectInterceptor in intercept at line 34
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
okhttp3.internal.cache.CacheInterceptor in intercept at line 95
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
okhttp3.internal.http.BridgeInterceptor in intercept at line 83
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
okhttp3.internal.http.RetryAndFollowUpInterceptor in intercept at line 76
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
io.opentelemetry.exporter.sender.okhttp.internal.RetryInterceptor in intercept at line 91
okhttp3.internal.http.RealInterceptorChain in proceed at line 109
okhttp3.internal.connection.RealCall in getResponseWithInterceptorChain$okhttp at line 201
okhttp3.internal.connection.RealCall$AsyncCall in run at line 517
java.util.concurrent.ThreadPoolExecutor in runWorker
java.util.concurrent.ThreadPoolExecutor$Worker in run
java.lang.Thread in run

[trask]

likely a duplicate of open-telemetry/opentelemetry-java#6946

[david-ribeiro-sumup]

We just use Open Telemetry to export traces, it seems the server was rejecting logs and metrics. Disabling the export of those fixed our problem. Thanks!