Skip to content
This repository has been archived by the owner on Mar 21, 2022. It is now read-only.

Waiting threads when using a client concurrently #727

Closed
stephan-huttenhuis opened this issue Apr 25, 2017 · 12 comments
Closed

Waiting threads when using a client concurrently #727

stephan-huttenhuis opened this issue Apr 25, 2017 · 12 comments
Labels
stale

Comments

@stephan-huttenhuis
Copy link
Contributor

Description

Using a client from multiple threads concurrently results in threads that wait forever.

How to reproduce

Run the following code:


import java.util.concurrent.CountDownLatch;

import com.spotify.docker.client.DefaultDockerClient;
import com.spotify.docker.client.DockerClient.ListContainersParam;

public final class Docker {

    public static void main(final String[] args) throws Exception {
        final DefaultDockerClient client = DefaultDockerClient.builder().uri("unix:///var/run/docker.sock").build();
        final int threadCount = 10;
        final CountDownLatch startLatch = new CountDownLatch(threadCount);
        final CountDownLatch endLatch = new CountDownLatch(threadCount);
        for (int i = 0; i < threadCount; ++i) {
            new Thread(new Worker(i, client, startLatch, endLatch)).start();
        }
        endLatch.await();
        System.out.println("All done");
    }

    static class Worker implements Runnable {
        DefaultDockerClient myClient;
        final CountDownLatch myEndLatch, myStartLatch;
        final int myI;

        public Worker(final int i, final DefaultDockerClient client, final CountDownLatch startLatch,
                final CountDownLatch endLatch) {
            myI = i;
            myClient = client;
            myStartLatch = startLatch;
            myEndLatch = endLatch;
        }

        public void run() {
            try {
                myStartLatch.countDown();
                myStartLatch.await();
                for (int i = 0; i < 10; i++) {
                    final ListContainersParam query = ListContainersParam.allContainers(true);
                    myClient.listContainers(query);
                    myClient.ping();
                }
            } catch (final Throwable e) {
                System.out.println(
                        "Could not retrieve containers for: " + myI + " - " + e.getMessage() + " " + e.getCause());
                e.printStackTrace();
                if(e.getCause() != null) {
                    System.out.println("cause: ");
                    e.getCause().printStackTrace();
                }
            } finally {
                System.out.println("done " + myI);
                myEndLatch.countDown();
            }
        }
    }
}

What do you expect

The process ends normally

What happened instead

The process keeps running because of waiting threads

"Thread-3" #13 prio=5 os_prio=0 tid=0x00007fed944ad000 nid=0x793b waiting on condition [0x00007fed64a89000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000745ea40a0> (a jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
	at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
	at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2194)
	at com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:477)
	at Docker$Worker.run(Docker.java:54)
	at java.lang.Thread.run(Thread.java:745)

Software:

  • docker version:
Client:
 Version:      17.04.0-ce
 API version:  1.28
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Mon Apr  3 18:01:50 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.04.0-ce
 API version:  1.28 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Mon Apr  3 18:01:50 2017
 OS/Arch:      linux/amd64
 Experimental: false
  • Spotify's docker-client version: 8.3.2
@mattnworb
Copy link
Member

Thanks for the report. I think the actual problem is within the jersey client and it's async-executor threads; DefaultDockerClient.request(DefaultDockerClient.java:2194) is just waiting for the futures returned by jersey to complete.

here is the full dump:

/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=64197:/Applications/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/lib/tools.jar:/Users/mattbrown/code/docker-client/target/test-classes:/Users/mattbrown/code/docker-client/target/classes:/Users/mattbrown/.m2/repository/org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12.jar:/Users/mattbrown/.m2/repository/com/google/guava/guava/18.0/guava-18.0.jar:/Users/mattbrown/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-json-provider/2.6.0/jackson-jaxrs-json-provider-2.6.0.jar:/Users/mattbrown/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-base/2.6.0/jackson-jaxrs-base-2.6.0.jar:/Users/mattbrown/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.0/jackson-core-2.6.0.jar:/Users/mattbrown/.m2/repository/com/fasterxml/jackson/module/jackson-module-jaxb-annotations/2.6.0/jackson-module-jaxb-annotations-2.6.0.jar:/Users/mattbrown/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-guava/2.6.0/jackson-datatype-guava-2.6.0.jar:/Users/mattbrown/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.0/jackson-databind-2.6.0.jar:/Users/mattbrown/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar:/Users/mattbrown/.m2/repository/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar:/Users/mattbrown/.m2/repository/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar:/Users/mattbrown/.m2/repository/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar:/Users/mattbrown/.m2/repository/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/Users/mattbrown/.m2/repository/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar:/Users/mattbrown/.m2/repository/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar:/Users/mattbrown/.m2/repository/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar:/Users/mattbrown/.m2/repository/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar:/Users/mattbrown/.m2/repository/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar:/Users/mattbrown/.m2/repository/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar:/Users/mattbrown/.m2/repository/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar:/Users/mattbrown/.m2/repository/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/Users/mattbrown/.m2/repository/org/glassfish/jersey/connectors/jersey-apache-connector/2.22.2/jersey-apache-connector-2.22.2.jar:/Users/mattbrown/.m2/repository/org/glassfish/jersey/media/jersey-media-json-jackson/2.22.2/jersey-media-json-jackson-2.22.2.jar:/Users/mattbrown/.m2/repository/org/glassfish/jersey/ext/jersey-entity-filtering/2.22.2/jersey-entity-filtering-2.22.2.jar:/Users/mattbrown/.m2/repository/org/apache/commons/commons-compress/1.9/commons-compress-1.9.jar:/Users/mattbrown/.m2/repository/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.jar:/Users/mattbrown/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/mattbrown/.m2/repository/commons-codec/commons-codec/1.9/commons-codec-1.9.jar:/Users/mattbrown/.m2/repository/org/apache/httpcomponents/httpcore/4.4.5/httpcore-4.4.5.jar:/Users/mattbrown/.m2/repository/com/github/jnr/jnr-unixsocket/0.8/jnr-unixsocket-0.8.jar:/Users/mattbrown/.m2/repository/com/github/jnr/jnr-ffi/2.0.3/jnr-ffi-2.0.3.jar:/Users/mattbrown/.m2/repository/com/github/jnr/jffi/1.2.9/jffi-1.2.9.jar:/Users/mattbrown/.m2/repository/com/github/jnr/jffi/1.2.9/jffi-1.2.9-native.jar:/Users/mattbrown/.m2/repository/org/ow2/asm/asm/5.0.3/asm-5.0.3.jar:/Users/mattbrown/.m2/repository/org/ow2/asm/asm-commons/5.0.3/asm-commons-5.0.3.jar:/Users/mattbrown/.m2/repository/org/ow2/asm/asm-analysis/5.0.3/asm-analysis-5.0.3.jar:/Users/mattbrown/.m2/repository/org/ow2/asm/asm-tree/5.0.3/asm-tree-5.0.3.jar:/Users/mattbrown/.m2/repository/org/ow2/asm/asm-util/5.0.3/asm-util-5.0.3.jar:/Users/mattbrown/.m2/repository/com/github/jnr/jnr-x86asm/1.0.2/jnr-x86asm-1.0.2.jar:/Users/mattbrown/.m2/repository/com/github/jnr/jnr-constants/0.8.7/jnr-constants-0.8.7.jar:/Users/mattbrown/.m2/repository/com/github/jnr/jnr-enxio/0.9/jnr-enxio-0.9.jar:/Users/mattbrown/.m2/repository/com/github/jnr/jnr-posix/3.0.12/jnr-posix-3.0.12.jar:/Users/mattbrown/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/mattbrown/.m2/repository/org/bouncycastle/bcpkix-jdk15on/1.52/bcpkix-jdk15on-1.52.jar:/Users/mattbrown/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.52/bcprov-jdk15on-1.52.jar:/Users/mattbrown/.m2/repository/com/google/auto/value/auto-value/1.3/auto-value-1.3.jar:/Users/mattbrown/.m2/repository/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar:/Users/mattbrown/.m2/repository/com/google/code/findbugs/annotations/3.0.1/annotations-3.0.1.jar:/Users/mattbrown/.m2/repository/net/jcip/jcip-annotations/1.0/jcip-annotations-1.0.jar:/Users/mattbrown/.m2/repository/junit/junit/4.12/junit-4.12.jar:/Users/mattbrown/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/Users/mattbrown/.m2/repository/org/mockito/mockito-core/1.10.19/mockito-core-1.10.19.jar:/Users/mattbrown/.m2/repository/org/objenesis/objenesis/2.1/objenesis-2.1.jar:/Users/mattbrown/.m2/repository/org/hamcrest/hamcrest-library/1.3/hamcrest-library-1.3.jar:/Users/mattbrown/.m2/repository/com/spotify/logging/2.1.1/logging-2.1.1.jar:/Users/mattbrown/.m2/repository/ch/qos/logback/logback-classic/1.0.13/logback-classic-1.0.13.jar:/Users/mattbrown/.m2/repository/ch/qos/logback/logback-core/1.0.13/logback-core-1.0.13.jar:/Users/mattbrown/.m2/repository/uk/co/flamingpenguin/jewelcli/jewelcli/0.6/jewelcli-0.6.jar:/Users/mattbrown/.m2/repository/net/kencochrane/raven/raven-logback/4.1.2/raven-logback-4.1.2.jar:/Users/mattbrown/.m2/repository/net/kencochrane/raven/raven/4.1.2/raven-4.1.2.jar:/Users/mattbrown/.m2/repository/com/google/jimfs/jimfs/1.0/jimfs-1.0.jar:/Users/mattbrown/.m2/repository/joda-time/joda-time/2.8.2/joda-time-2.8.2.jar:/Users/mattbrown/.m2/repository/org/awaitility/awaitility/2.0.0/awaitility-2.0.0.jar:/Users/mattbrown/.m2/repository/cglib/cglib-nodep/3.1/cglib-nodep-3.1.jar com.spotify.docker.ThreadTest
objc[41557]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java (0x104e814c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x1067b64e0). One of the two will be used. Which one is undefined.
done 9
done 7
done 1
done 5
done 0
done 6
done 3
2017-04-25 19:33:18
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode):

"jersey-client-async-executor-9" #30 prio=5 os_prio=31 tid=0x00007fd598a28000 nid=0x7203 runnable [0x000070000bd7e000]
   java.lang.Thread.State: RUNNABLE
	at com.kenai.jffi.Foreign.invokeN3O1(Native Method)
	at com.kenai.jffi.Invoker.invokeN3(Invoker.java:1061)
	at jnr.enxio.channels.Native$LibC$jnr$ffi$1.read(Unknown Source)
	at jnr.enxio.channels.Native.read(Native.java:95)
	at jnr.enxio.channels.NativeSocketChannel.read(NativeSocketChannel.java:70)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
	- locked <0x0000000797eaa5e8> (a java.lang.Object)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
	- locked <0x0000000797eaab40> (a sun.nio.ch.ChannelInputStream)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:251)
	at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)
	at org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)
	at org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed(ResponseEntityProxy.java:128)
	at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
	at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:174)
	at java.io.BufferedInputStream.close(BufferedInputStream.java:483)
	at java.io.FilterInputStream.close(FilterInputStream.java:181)
	at org.glassfish.jersey.apache.connector.ApacheConnector$3.close(ApacheConnector.java:644)
	at java.io.FilterInputStream.close(FilterInputStream.java:181)
	at org.glassfish.jersey.apache.connector.ApacheConnector$HttpClientResponseInputStream.close(ApacheConnector.java:621)
	at org.glassfish.jersey.message.internal.EntityInputStream.close(EntityInputStream.java:158)
	at org.glassfish.jersey.message.internal.InboundMessageContext$EntityContent.close(InboundMessageContext.java:156)
	at org.glassfish.jersey.message.internal.InboundMessageContext$EntityContent.close(InboundMessageContext.java:148)
	at org.glassfish.jersey.message.internal.ReaderWriter.safelyClose(ReaderWriter.java:203)
	at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:896)
	at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:834)
	at org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:368)
	at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:867)
	at org.glassfish.jersey.client.JerseyInvocation.access$800(JerseyInvocation.java:92)
	at org.glassfish.jersey.client.JerseyInvocation$6.completed(JerseyInvocation.java:836)
	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:198)
	at org.glassfish.jersey.client.ClientRuntime.access$300(ClientRuntime.java:79)
	at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:180)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
	at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"jersey-client-async-executor-8" #29 prio=5 os_prio=31 tid=0x00007fd597a90000 nid=0x6a03 runnable [0x000070000b972000]
   java.lang.Thread.State: RUNNABLE
	at com.kenai.jffi.Foreign.invokeN3O1(Native Method)
	at com.kenai.jffi.Invoker.invokeN3(Invoker.java:1061)
	at jnr.enxio.channels.Native$LibC$jnr$ffi$1.read(Unknown Source)
	at jnr.enxio.channels.Native.read(Native.java:95)
	at jnr.enxio.channels.NativeSocketChannel.read(NativeSocketChannel.java:70)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
	- locked <0x0000000796839f08> (a java.lang.Object)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
	- locked <0x000000079683a460> (a sun.nio.ch.ChannelInputStream)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
	at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
	at org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
	at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
	at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
	at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487)
	at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:178)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
	at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"jersey-client-async-executor-1" #23 prio=5 os_prio=31 tid=0x00007fd598b32800 nid=0x6603 runnable [0x000070000b76b000]
   java.lang.Thread.State: RUNNABLE
	at com.kenai.jffi.Foreign.invokeN3O1(Native Method)
	at com.kenai.jffi.Invoker.invokeN3(Invoker.java:1061)
	at jnr.enxio.channels.Native$LibC$jnr$ffi$1.read(Unknown Source)
	at jnr.enxio.channels.Native.read(Native.java:95)
	at jnr.enxio.channels.NativeSocketChannel.read(NativeSocketChannel.java:70)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
	- locked <0x00000007982beb98> (a java.lang.Object)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
	- locked <0x00000007982bf0f0> (a sun.nio.ch.ChannelInputStream)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:206)
	at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	- locked <0x00000007982c0b78> (a java.io.BufferedInputStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:102)
	at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream.read(ReaderInterceptorExecutor.java:296)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	- locked <0x00000007982c3208> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.Reader.read(Reader.java:140)
	at org.glassfish.jersey.message.internal.ReaderWriter.readFromAsString(ReaderWriter.java:175)
	at org.glassfish.jersey.message.internal.ReaderWriter.readFromAsString(ReaderWriter.java:160)
	at org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.readFromAsString(AbstractMessageReaderWriterProvider.java:117)
	at org.glassfish.jersey.message.internal.StringMessageProvider.readFrom(StringMessageProvider.java:77)
	at org.glassfish.jersey.message.internal.StringMessageProvider.readFrom(StringMessageProvider.java:59)
	at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(ReaderInterceptorExecutor.java:256)
	at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(ReaderInterceptorExecutor.java:235)
	at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:155)
	at org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1085)
	at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:874)
	at org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:808)
	at org.glassfish.jersey.client.ClientResponse.readEntity(ClientResponse.java:326)
	at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:803)
	at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92)
	at org.glassfish.jersey.client.JerseyInvocation$5.completed(JerseyInvocation.java:773)
	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:198)
	at org.glassfish.jersey.client.ClientRuntime.access$300(ClientRuntime.java:79)
	at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:180)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
	at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"Thread-9" #19 prio=5 os_prio=31 tid=0x00007fd597a23800 nid=0x5e03 waiting on condition [0x000070000b361000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000798063af0> (a jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
	at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
	at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2188)
	at com.spotify.docker.client.DefaultDockerClient.listContainers(DefaultDockerClient.java:509)
	at com.spotify.docker.ThreadTest$Worker.run(ThreadTest.java:41)
	at java.lang.Thread.run(Thread.java:745)

"Thread-5" #15 prio=5 os_prio=31 tid=0x00007fd598a1d000 nid=0x5603 waiting on condition [0x000070000af55000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000079667fbe8> (a jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
	at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
	at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2198)
	at com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:477)
	at com.spotify.docker.ThreadTest$Worker.run(ThreadTest.java:42)
	at java.lang.Thread.run(Thread.java:745)

"Thread-3" #13 prio=5 os_prio=31 tid=0x00007fd597a18800 nid=0x5203 waiting on condition [0x000070000ad4f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000797fa2868> (a jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
	at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
	at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2198)
	at com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:477)
	at com.spotify.docker.ThreadTest$Worker.run(ThreadTest.java:42)
	at java.lang.Thread.run(Thread.java:745)

"Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007fd59792c000 nid=0x4a03 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=31 tid=0x00007fd598154000 nid=0x4803 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=31 tid=0x00007fd597857800 nid=0x4603 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=31 tid=0x00007fd598153000 nid=0x4403 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Ctrl-Break" #5 daemon prio=5 os_prio=31 tid=0x00007fd597928800 nid=0x4203 runnable [0x000070000a537000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	- locked <0x000000074058bf80> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	- locked <0x000000074058bf80> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fd59880d000 nid=0x4003 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fd598035800 nid=0x3103 in Object.wait() [0x000070000a331000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x000000074058c130> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	- locked <0x000000074058c130> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fd59702e000 nid=0x2f03 in Object.wait() [0x000070000a22e000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000740227948> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x0000000740227948> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=31 tid=0x00007fd597808000 nid=0x1c03 waiting on condition [0x0000700009c1c000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074047f120> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
	at com.spotify.docker.ThreadTest.main(ThreadTest.java:18)

"VM Thread" os_prio=31 tid=0x00007fd597029800 nid=0x2d03 runnable 

"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fd597814000 nid=0x2503 runnable 

"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007fd598803800 nid=0x2703 runnable 

"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007fd598804000 nid=0x2903 runnable 

"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007fd598804800 nid=0x2b03 runnable 

"VM Periodic Task Thread" os_prio=31 tid=0x00007fd598871800 nid=0x4c03 waiting on condition 

JNI global references: 370

Heap
 PSYoungGen      total 71680K, used 48683K [0x0000000795580000, 0x000000079ed80000, 0x00000007c0000000)
  eden space 66560K, 73% used [0x0000000795580000,0x000000079850af38,0x0000000799680000)
  from space 5120K, 0% used [0x0000000799680000,0x0000000799680000,0x0000000799b80000)
  to   space 11264K, 0% used [0x000000079e280000,0x000000079e280000,0x000000079ed80000)
 ParOldGen       total 46592K, used 6305K [0x0000000740000000, 0x0000000742d80000, 0x0000000795580000)
  object space 46592K, 13% used [0x0000000740000000,0x0000000740628598,0x0000000742d80000)
 Metaspace       used 22928K, capacity 23208K, committed 23296K, reserved 1069056K
  class space    used 2736K, capacity 2811K, committed 2816K, reserved 1048576K


Process finished with exit code 130 (interrupted by signal 2: SIGINT)

@mattnworb
Copy link
Member

unless my stacktraces are specific to running on a mac.

there are three threads with a stacktrace like:

"jersey-client-async-executor-9" #30 prio=5 os_prio=31 tid=0x00007fd598a28000 nid=0x7203 runnable [0x000070000bd7e000]
   java.lang.Thread.State: RUNNABLE
	at com.kenai.jffi.Foreign.invokeN3O1(Native Method)
	at com.kenai.jffi.Invoker.invokeN3(Invoker.java:1061)
	at jnr.enxio.channels.Native$LibC$jnr$ffi$1.read(Unknown Source)
	at jnr.enxio.channels.Native.read(Native.java:95)
	at jnr.enxio.channels.NativeSocketChannel.read(NativeSocketChannel.java:70)

@mattnworb mattnworb mentioned this issue May 1, 2017
Closed
@unicolet unicolet mentioned this issue May 16, 2017
Merged
@GameScripting
Copy link

GameScripting commented Sep 23, 2017 •
edited
Loading

Is a DockerClient instance supposed to be reused? Will this issue be fixed or "just use a different DockerClient per thread"?

@GameScripting
Copy link

GameScripting commented Sep 23, 2017 •
edited
Loading

For me this also happens when using a unique DockerClient per Thread. Now checking if this also happens when not using the unix socket but the daemons http interface.

@mattnworb
Copy link
Member

The intention is for you to be able to use a single instance safely.

@GameScripting
Copy link

GameScripting commented Sep 23, 2017 •
edited
Loading

Unfortunately for me it was not possible to switch from unix-socket to http-socket (so i didnt test anything). What works now for me as a workarround is to put all calls to a single DockerClient into a synchronized-block so no parallel requests are send.

@daniloradenovic
Copy link

daniloradenovic commented Nov 8, 2017 •
edited
Loading

Just to confirm that this also happens when using single instance of DefaultDockerClient per thread.

@daniloradenovic
Copy link

daniloradenovic commented Feb 15, 2018 •
edited
Loading

@stephan-huttenhuis I've taken your code, slightly modified it and with a little help of git bisect I I've narrowed down the cause.

It indeed seems to be the problem in jersey-client, as @mattnworb suggested, since this issue was introduced with dcbe3b6 when jersey-client was upgraded from 2.19 to 2.22.2. Using the latest currently available version of jersey-client, 2.26, does not solve this.

I've opened an issue on jersey project jersey/jersey/issues/3772 . Until then, for anyone interested, version 6.1.1 of the docker-client is the most recent one that does not contain this issue.

@kqiang
Copy link

kqiang commented Mar 19, 2018

Downgraded from 8.8.2 to 6.1.1. Now I am having this issue: #446

@stale
Copy link

stale bot commented Oct 9, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Oct 9, 2018
@stale stale bot closed this as completed Oct 16, 2018
@aviillouz
Copy link

aviillouz commented Nov 6, 2018 •
edited
Loading

Can someone please explain how workaround this issue? Like @daniloradenovic I was trying to create new client for concurrent requests but it doesn't solve this.

@daniloradenovic
Copy link

@aviillouz If you are not tied to later versions of the docker client, you can try with 6.1.1.

@vreuland vreuland mentioned this issue Dec 7, 2018
Closed
@mad-p mad-p mentioned this issue Dec 26, 2018
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@mattnworb @aviillouz @GameScripting @daniloradenovic @stephan-huttenhuis @kqiang