Connection reset by peer in Linux

[gfl]

Description

I am trying to run commands in the docker containers by using the docker-client library in my integration tests.
This approach is working fine in my local machine (MAC OS X with Docker for Mac). When I try to do the same in the CI machine (Debian Wheezy with kernel v3.16.7) most of the commands fail saying that the socket connection has been reset by peer.

Do you have any idea why this is happening?

How to reproduce

My code looks something like this (Scala):

val docker = new DefaultDockerClient(URI.create("unix:///var/run/docker.sock"))
val container = docker.listContainers().find(_.image().contains("kafka")).get
val command = Array(
      "kafka-topics",
      "--zookeeper", "zookeeper",
      "--list"
    )
val execCreate = docker.execCreate(container.id(), command,
    DockerClient.ExecCreateParam.attachStdout(), DockerClient.ExecCreateParam.attachStderr())
docker.execStart(execCreate.id).readFully().split("\n")

What do you expect

I would expect to be able to run commands in the docker containers from my Scala code in a similar way as I do with the docker exec command line.

What happened instead

A few commands are run successfully and then I start getting the error of connection reset by peer.

Software:

• docker version: 1.12.1
• Spotify's docker-client version: 6.0.0

Full backtrace

[info]   java.lang.RuntimeException: java.io.IOException: Connection reset by peer
[info]   at com.google.common.base.Throwables.propagate(Throwables.java:160)
[info]   at com.spotify.docker.client.LogStream.computeNext(LogStream.java:61)
[info]   at com.spotify.docker.client.LogStream.computeNext(LogStream.java:35)
[info]   at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
[info]   at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
[info]   at com.spotify.docker.client.LogStream.readFully(LogStream.java:81)
[info]   at my.code.KafkaDockerManager$class.listKafkaTopics(KafkaDockerManager.scala:37)
[info]   ...
[info]   Cause: java.io.IOException: Connection reset by peer
[info]   at jnr.enxio.channels.NativeSocketChannel.read(NativeSocketChannel.java:82)
[info]   at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
[info]   at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
[info]   at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
[info]   at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
[info]   at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198)
[info]   at org.apache.http.impl.io.IdentityInputStream.read(IdentityInputStream.java:93)
[info]   at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
[info]   at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
[info]   at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
[info]   ...

[gfl]

I've just forgot to say that the following command works fine in the same Linux machine:

docker -H unix:///var/run/docker.sock ps

[johnflavin]

I can see where the problem is happening, if not what. When you create the exec you get back a stream, and when you call readFully() you’re trying to exhaust the stream and parse log messages out of it.

In my experience doing a similar task, reading docker events off the stream, I found that A. the docker API will leave streams open and ready to send more even when there is no more to send, and B. the apache http library that docker-client uses to read the stream does not handle that situation gracefully.

I am not certain that's the same problem you’re having. I don't know why the connection would be reset during that process. I can't give you any real advice or direction.

Can you reproduce the problem reliably? Is it just with your own images, or can you run some command in busybox or whatever that you can get to trigger the same error? Which CI service are you using?

[gfl]

Hi @johnflavin,

Thanks for the quick response. My CI service is Jenkins. But even running the tests manually in that machine (Debian Wheezy) gives the same results.

I tried to get rid of all the readFully() and make sure that I close all the LogStreams in my code. But still the same issue. Moreover, the error happens almost immediately.

I will try to find a way to reproduce in problem in a way I can share with you.

Nevertheless, I am starting to think that the main issue is I am not using your library for the right purpose. Do you have any suggestion of how to run integration tests with Docker in an isolated and multi platform way? We are considering running the tests inside one of the containers as an alternative.

Thanks

[johnflavin]

Are you able to run your docker exec ... commands by hand on the Wheezy machine? As in, are the errors present when you’re doing any docker exec on that machine, or only when you do it through docker-client? If the former, then it seems like docker has a problem on Wheezy or you have a problem with your installation; if the latter, then there is some actual problem in docker-client that we need to fix.

I can't say whether or not you’re using this library for the "right" purpose. I don't know specifically what you’re trying to accomplish. But I can tell you that the purpose of this library is very simple: to let you communicate with your docker server from within java code (or scala, or groovy, etc.). If you wanted to, you could eschew the use of docker-client and accomplish all the exact same tasks by directly calling the docker REST API; you'd lose a ton of convenience functions, but you could also tailor things to just what you need.

[gfl]

I've got the error with docker-client. When trying to use docker exec it works fine:

$ docker exec -i my_kafka_1 kafka-console-producer --broker-list localhost:9092 --topic test << EOF
> > message 1
> > message 2
> > message 3
> > EOF
>
^C
$ docker -H unix:///var/run/docker.sock exec -i my_kafka_1 kafka-console-consumer --zookeeper zookeeper --from-beginning --topic test --max-messages 10
message 1
message 2
message 3
^C
$ docker -H unix:///var/run/docker.sock exec -i my_kafka_1 kafka-topics --zookeeper zookeeper --list
test

The only potential issue is that I have to terminate the commands, since both kafka-console-producer and kafka-console-consumer will never finish.

I've just noticed that another test that I have with Cassandra is working fine (I use docker-client to run some CQL statements). So definitely it has to be an issue with Kafka. I will try to investigate further and let you know if I manage to solve the issues.

[jalbr74]

For what it's worth, I'm seeing the same "java.io.IOException: Connection reset by peer" error. It seems intermittent, though. Probably happening every one out of three times.

I can reproduce it with the following client application:

import com.spotify.docker.client.DefaultDockerClient;
import com.spotify.docker.client.DockerClient;
import com.spotify.docker.client.LogStream;
import com.spotify.docker.client.exceptions.DockerException;
import com.spotify.docker.client.messages.ExecCreation;

public class DockerTest {
    public static void main(String arg[]) {
        try (DockerClient docker = new DefaultDockerClient("unix:///var/run/docker.sock")) {
            String[] cmd = {"echo", "'All is well'"};

            ExecCreation execCreation = docker.execCreate("ssprservice", cmd, DockerClient.ExecCreateParam.attachStdout(), DockerClient.ExecCreateParam.attachStderr());
            final LogStream output = docker.execStart(execCreation.id());

            String execOutput = output.readFully();
            System.out.println("Docker command output: " + execOutput);
        } catch (DockerException | InterruptedException e) {
            e.printStackTrace();
        }
    }
}

Here's the stack trace I see after running it 3-4 times:

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Connection reset by peer
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at com.spotify.docker.client.DefaultLogStream.computeNext(DefaultLogStream.java:58)
        at com.spotify.docker.client.DefaultLogStream.computeNext(DefaultLogStream.java:35)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at com.spotify.docker.client.DefaultLogStream.readFully(DefaultLogStream.java:77)
        at DockerTest.main(DockerTest.java:15)
Caused by: java.io.IOException: Connection reset by peer
        at jnr.enxio.channels.NativeSocketChannel.read(NativeSocketChannel.java:82)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:71)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:121)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:115)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198)
        at org.apache.http.impl.io.IdentityInputStream.read(IdentityInputStream.java:93)
        at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:257)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:297)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:356)
        at java.io.FilterInputStream.read(FilterInputStream.java:144)
        at java.io.FilterInputStream.read(FilterInputStream.java:144)
        at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:102)
        at org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream.read(ReaderInterceptorExecutor.java:296)
        at com.google.common.io.ByteStreams.read(ByteStreams.java:733)
        at com.spotify.docker.client.LogReader.nextMessage(LogReader.java:49)
        at com.spotify.docker.client.DefaultLogStream.computeNext(DefaultLogStream.java:56)

Docker Version: 1.12.5, build 8eab29e
Docker Client Jar Version: 7.0.0

Is there anything else you think I should try?

[rabelenda]

I'm facing a similar issue, sounds like a race condition or something. Was having the issue in an ubuntu server, and once I enabled docker daemon debug mode it stopped for the time being, but at least whenever it appears I hope to have more data to be able to trace further the issue. In my local machine with docker for mac, I was not able to reproduce the issue.

I am running Docker Version: 1.13.0, with docker client library 7.0.2.

[gbalats]

I'm also experiencing the same issue, but it happens more than half of the times I run it.

Docker Version: 1.12.6
Docker Client Version: 7.0.0

[aatxe]

I'm experiencing this issue very frequently as well (more often than not).

Docker Version: 1.13.0
Docker Client Version: 7.0.0

[saidbouras]

Same problem for me (version 1.12.3)
I found that it work on docker prior to version 1.11.2 (included).

Like johnflavin said in his first post, it's due to the stream from docker.execStart(..) function.
Any operation on that stream triggers the error because the stream is still open waiting more datas.

Any solution to parse the logs ?

[diemol]

I also have the same problem but I didn't know there was an issue created about it (just found out since now I added myself as a watcher).

What I did as a workaround was to wrap the output.readFully() in a try - catch clause, as seen here.

It does not solve the issue, but maybe it helps you to move on while a solution is found.

[johnflavin]

I've taken another look at this thread, and I think I see two different but related issues:

1. The issue of execCreate blocking the thread. This one is probably not solvable right now. Calling LogStream.readFully()—or really any method on LogStream—when the stream from docker is still open but without any data will cause the thread to block. There really isn't anything we can do to change that*, because all the libraries that we use to make the connections are built on blocking IO.
2. The intermittent RuntimeExceptions are confusing to me. It appears to me that the "connection reset by peer" error is from the connection being disrupted or aborted from docker's end. Until and unless someone can find a way to reliably reproduce this, I think that @diemol's solution is probably the way to go: just catch the RuntimeException.

*I did want to try one possible solution to the problem that reading from a still-open-but-no-data connection will block: check the input stream's available() method before reading. However, that didn't work like I expected. I got back a 0 which means "no data available" even when reading from the stream would not only not block but would actually return data. I filed a bug with Apache HTTP Core about it: HTTPCORE-449.

[saidbouras]

@johnflavin It seems that the LogStream class don't have access to the input stream's available()method or i am probably missing something here.

I use a cassandra container for my tests and i execute some statement into, so i want to check out the log for an error and crashed when it fails, so i don't mind having a 0 because it will mean that all cassandra statements are corrects.

[johnflavin]

I'm not sure if it would work for everyone in this thread, but I suspect that some of the reasons that people are using exec commands would be better served by having images with HEALTHCHECK commands.

[GameScripting]

As of Docker

Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:02:43 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:02:43 2017
 OS/Arch:      linux/amd64
 Experimental: false

with version com.spotify:docker-client:8.1.1 I cannot get this to work any single time. For me its finally broken.

[johnflavin]

@GameScripting What command are you trying to exec? You say it never works in docker-client 8.1.1; can you confirm that it did work at least sometimes in older versions of docker-client?

I'm digging into your comment to try to answer some questions:

1. Is this actually a regression? Was there a thing that reproducibly worked before that reproducibly does not work now?
2. If so, is it in...
   2a. docker-client? Or
   2b. docker?

[johnflavin]

@rinscy Sorry, meant to respond to this a while ago. The call to available() would be in LogReader, not LogStream. Unfortunately I don't think that approach will work. The issue I reported in HTTPCORE-449, which renders available() useless, is probably not going to be fixed soon if ever. Then if a fix did come out all the docker-client dependencies that use it would need to be updated for that, etc.

[saidbouras]

@johnflavin Yes i just notice that.
I have to tell that when i use docker 1.11.2 or prior it works very well, but when i use a version above 1.11.2 (like 1.12) it fail.
It has to be link to change of docker API.

[ArcticPheenix]

I'm not sure if anyone else has tried this, but I noticed that this issue seems be revolve around using the UNIX socket that the Docker daemon sets up by default. I was wondering why I wasn't running into the same issue, so I tried to replicate it on my end. In my environment and use case, I need to communicate with local and remote Docker hosts. As such, I've configured my daemons to listen on TCP port 2375 (no TLS). Using the code snippet provided by @jalbr74, I replaced the string literal for the URL so that it uses the TCP port on the box. The result is that the provided snippet works. To double check, I used the original snippet with the UNIX socket, and received the same failures as others here.

Including my snippet:

package test;

import com.spotify.docker.client.DefaultDockerClient;
import com.spotify.docker.client.DockerClient;
import com.spotify.docker.client.LogStream;
import com.spotify.docker.client.exceptions.DockerException;
import com.spotify.docker.client.messages.ExecCreation;
import org.testng.annotations.Test;

public class TestDockerDriver {
    private static final String URL = "http://localhost:2375";
    private static final String[] CMD = {"/bin/bash"};
    private static final String IMAGE = "ubuntu:latest";

    @Test
    public void testDockerClient() {
        try (DockerClient docker = new DefaultDockerClient("http://localhost:2375")) {
            String[] cmd = {"echo", "'All is well'"};

            ExecCreation execCreation = docker.execCreate("test", cmd, DockerClient.ExecCreateParam.attachStdout(), DockerClient.ExecCreateParam.attachStderr());
            final LogStream output = docker.execStart(execCreation.id());

            String execOutput = output.readFully();
            System.out.println("Docker command output: " + execOutput);
        } catch (DockerException | InterruptedException e) {
            e.printStackTrace();
        }
    }
}

EDIT: For purposes of clarity, I'm using the 8.1.1 JAR with Docker 1.12.6 and 1.13.1, both locally and remotely, on Fedora 25 Server, Ubuntu 16.04 Server, and openSUSE LEAP 42.2.

[johnflavin]

@ArcticPheenix Ok, it sounds like you’re able to reliably reproduce the error, and when you switch out the default /var/run/docker.sock connection for a http://localhost:2375 the error goes away. I want you to help me figure out how I can reproduce this behavior, if you are willing.

The test you have provided is a good start, but that by itself doesn't let me reproduce the behavior. We already have a test that creates a simple exec like this: DefaultDockerClientTest#testExec. Plus, your test code doesn't appear to be complete. If that test is supposed to work, it looks like you have to start a container named "test" before running the test. (Because the first argument to DockerClient#execCreate is supposed to be the ID (or name, I suppose) of a running container.)

I think what I want to do is figure out if we can reproduce this error by testing on different operating systems. You mentioned several: Fedora 25 Server, Ubuntu 16.04 Server, and openSUSE LEAP 42.2. But I can't tell exactly which were working, and whether they worked all the time or whether they failed using the socket and worked using tcp. So could you give more detailed information of all the configurations you have tried, socket vs. tcp, server OS, what works, what doesn't, etc.?

What other variables have been mentioned in the thread?

• In the original post, @gfl said they were seeing the error when using "Debian Wheezy with kernel v3.16.7".
• @gfl also said the error does not show up on Docker for Mac. I can confirm that, because that's what I'm using.
• Our Travis CI builds use Ubuntu 14.04, and they do not see this error.
• @rabelenda said they do see the error on "an Ubuntu server", but that if docker is in debug mode it goes away. That's curious. It makes me wonder if Docker for Mac defaults to being in debug mode, and maybe that's why it doesn't show this error? ...<pause to look that up>... Yes, according to the Docker for Mac release notes, as of Beta 3 Release (2016-03-15 1.10.3-beta3), "Docker runs in debug mode by default for new installs". So that might explain why I've never seen this on my machines. However, our Travis CI builds do not run docker in debug mode. See the docker opts here: .travis.sh#L35. So something else must be going on there.
• @rinscy has said that the error only happens for docker versions greater than 1.11.2.

If anyone else wants to report in, please provide the following:

• Docker server version
• Is docker running in debug mode?
• docker-client version
• Server OS, optionally with kernel version
• Are you connecting to docker using a socket or tcp? Does the error go away if you use the other one?

Now that I've typed this all out, it makes me strongly suspect that this is an issue in docker's API and not in docker-client. Still, if someone can help track it down and reproduce it, that will make it all the easier to report to docker.

[ArcticPheenix]

@johnflavin I'm about to head out to lunch, but I'll get you the info you need. The root of the issue may or may not be twofold. Since I can reliably use these calls on versions of Docker that others are having issue with, that would seem to suggest that the issue(s) may be centered around UNIX sockets, and possibly the updated Docker API.

Give me a bit to get some more data about versions, platforms, etc. I can also test on Mac, if you want another data point. I work with @jalbr74, and only started looking at this project as of yesterday.

[ArcticPheenix]

@johnflavin I wanted to apologize for the delay. Things at work have become quite busy. I'll paste my code snippet for a test class that allows me to reliably reproduce the issue on Fedora 25 Server with Docker 1.12.6, 1.13.1, and the new 17.03.0-ce. API versions range from 1.24 (Docker 1.12.6) through 1.26 (Docker 17.03.0-ce). In each instance, if I use the UNIX socket, I will get the issue. If I connect via TCP, I won't. Doing a cursory glance at the code, I don't think the issue lies with the docker-client library, but rather with Apache or Docker.

[saidbouras]

@johnflavin I don't know if it will be useful but for me, switch from the default connection /var/run/docker.sock connection to a http://localhost:2375 worked as well
Now i'm not sure if it's safe to let that without https but for my usecase (for testing purposes) it's painful to create a certificate and enable tls just for this.

[johnflavin]

I agree with @ArcticPheenix; it does not look to me like the source of this problem is inside any docker-client code. Here are my thoughts:

• I assume that the error is still occasionally happening with the tcp connection, but that it is non-fatal.
• I think it makes sense given what we have seen that there is a difference between connecting over the socket vs. tcp. The couple stacktraces that have been posted have all started with an error in jnr.enxio.channels.NativeSocketChannel.read(NativeSocketChannel.java:82). Note that the class is NativeSocketChannel; this code would only be executing for a socket connection and not for a tcp connection.
• I can construct a narrative in my head about why the error is fatal with the socket but not fatal with tcp: perhaps whatever deep layer of code is handling the tcp connection is better about recovering from "connection reset by peer" errors? Maybe it can even reconnect and continue? I really don't know what code is managing this connection, so I can only speculate.
• The ultimate source of this error is a "connection reset by peer" from docker. So regardless of how that error gets handled on the client side and whether it is fatal or not, there will still be a problem until docker fixes whatever is happening. I think that everyone experiencing this should check out Executing commnd in container hangs randomly (exec attach failed with error - connection reset by peer) moby/moby#31421; it looks very similar to what you have described here.

[saidbouras]

Exactly i know for sure now that the probleme is not docker-client code (I forgot to tell you that in my previous message, but i forgot).
Thx

[ajmalrehman]

I am getting similar issue in my case following exception occurs:-

com.spotify.docker.client.DockerRequestException: Request error: POST unix://localhost:80/containers/e66871b0fcc0c22ccdf710a2f0efb213c21d9c360759cb20d8bce50dfc12824e/start: 500
	at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1306)
	at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1277)
	at com.spotify.docker.client.DefaultDockerClient.containerAction(DefaultDockerClient.java:456)
	at com.spotify.docker.client.DefaultDockerClient.startContainer(DefaultDockerClient.java:448)
	at com.webexecutor.docker.container.RunRubyDocker.buildRubyDocker(RunRubyDocker.java:50)
	at com.webexecutor.code.compiler.service.ExecutorModeService.startDocker(ExecutorModeService.java:64)
	at com.webecutor.java.controller.ExecutorController.doPost(ExecutorController.java:55)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 Internal Server Error
	at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1002)
	at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:799)
	at org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:91)
	at org.glassfish.jersey.client.JerseyInvocation$5.completed(JerseyInvocation.java:760)
	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:197)
	at org.glassfish.jersey.client.ClientRuntime.access$300(ClientRuntime.java:78)
	at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:179)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
	at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:209)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more

My java class is 👎

public class RunRubyDocker {

public String buildRubyDocker(String filename){
	String execOutput = null;
	
	try {
		final DockerClient docker = DefaultDockerClient.fromEnv().build();
		docker.pull("ruby:2.1-onbuild");
		final String[] ports = {"33", "9393"};
		final Map<String, List<PortBinding>> portBindings = new HashMap<>();
		for (String port : ports) {
		    List<PortBinding> hostPorts = new ArrayList<>();
		    hostPorts.add(PortBinding.of("0.0.0.0", port));
		    portBindings.put(port, hostPorts);
		}
		List<PortBinding> randomPort = new ArrayList<>();
		randomPort.add(PortBinding.randomPort("0.0.0.0"));
		portBindings.put("443", randomPort);
		final HostConfig hostConfig = HostConfig.builder().binds("/home/itcostcut/mydockerbuild:/usr/src/app/mydockerbuild").
				portBindings(portBindings).build();
		

		
		final ContainerConfig containerConfig = ContainerConfig.builder()
			    .hostConfig(hostConfig)
			    .image("ruby:2.1-onbuild").exposedPorts(ports)
			    .cmd("sh", "-c", "while :; do sleep 1; done")
			    .build();
		final ContainerCreation creation = docker.createContainer(containerConfig);
		final String id = creation.id();

		// Start container
		docker.startContainer(id);
		// Exec command inside running container with attached STDOUT and STDERR
		
		final String[] command = {"bash", "-c", "cd mydockerbuild/ && ruby "+filename};
		System.out.println(command);
		final String execCreation = docker.execCreate(
		    id, command, DockerClient.ExecCreateParam.attachStdout(),
		    DockerClient.ExecCreateParam.attachStderr());
		
		final LogStream output = docker.execStart(execCreation);
		execOutput = output.readFully();
		System.out.println(execOutput);
		// Kill container
		docker.killContainer(id);

		// Remove container
		docker.removeContainer(id);

		// Close the docker client
		docker.close();
	} catch (DockerException | InterruptedException e) {
		// TODO Auto-generated catch block
		e.printStackTrace();
	} catch (DockerCertificateException e) {
		// TODO Auto-generated catch block
		e.printStackTrace();
	}
	return execOutput;
	
}

}

Docker version:-

Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Fri Mar 24 00:45:26 2017
OS/Arch: linux/amd64

Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Fri Mar 24 00:45:26 2017
OS/Arch: linux/amd64
Experimental: false

Please help. I am getting this issue error every 3 time when i am executing my code.
Thanks in advance

[ajmalrehman]

@johnflavin I am not able to fix this issue please suggest.

[johnflavin]

@ajmalrehman Can you give some more information by answering the questions at the bottom of this comment above?

[dan-v]

• Docker server version: 17.04.0-ce
• Docker client version: 17.04.0-ce
• Is docker running in debug mode? No. If I enabled debug mode then I am unable to reproduce the issue.
• Server OS, optionally with kernel version: CentOS 7.2 Kernel Version: 3.10.0-514.16.1.el7.x86_64. Also have reproduced this on CentOS 7.3 and Ubuntu 16.04.
• Are you connecting to docker using a socket or tcp? Does the error go away if you use the other one? I'm using docker socket. I've verified that I'm only able to reproduce the issue using unix sockets and not with TCP.

I've created a repro program and put it here: https://github.com/dan-v/spotify-docker-client-issue-513. In order to reproduce I have found that I must be run on the following

• A system with multiple CPUs/cores. On the same VM, if I drop down to a single CPU I no longer see this issue.
• Docker daemon must not be in debug mode, as soon as debug mode is flipped on the error goes away
• Must be running against docker unix socket. With TCP communication I do not see this error.

Here is example of the issue

java -cp ~/repro.jar spotify.Testing
Running spotify test..
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Started busybox container: e3cebcfa3c38a30e77df336fb197aa5833d48b57d9feeb2e2d781d5f510abd43
Now running exec loop in this container..
Exit code: 0
java.lang.RuntimeException: java.io.IOException: Connection reset by peer
Exit code: 0
java.lang.RuntimeException: java.io.IOException: Connection reset by peer
Exit code: 0

The same program but run against TCP instead of unix socket

java -cp ~/repro.jar spotifytcp.Testing
Running spotify test..
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Started busybox container: b255f293c8c4e86f1d2404392014e7733591169ff28a8349bb84cf3645175a76
Now running exec loop in this container..
Exit code: 0
Exit code: 0
Exit code: 0
Exit code: 0
Exit code: 0

I also tested out the docker-java library which doesn't have this issue.

java -cp ~/repro.jar nonspotify.Testing
Running non spotify test..
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Started busybox container: 437b21c00b6d6e3036a90a2da098ca62dd7bf8994b9c8f3912a581f211008952
Now running exec loop in this container..
Exit code: 0
Exit code: 0
Exit code: 0
Exit code: 0
Exit code: 0

I then enable debug mode for docker daemon and looked at what each library was doing in terms of API calls. The spotify library was making calls a little differently.

Spotify library

Apr 28 14:38:46 localhost dockerd: time="2017-04-28T14:38:46.701798243-07:00" level=debug msg="Calling GET /exec/45b365a62ef66e0b991dcdcef911085e34d60ab7dabc75282c1b88b754f17b7f/json"
Apr 28 14:38:46 localhost dockerd: time="2017-04-28T14:38:46.704656346-07:00" level=debug msg="Calling GET /containers/83a4642cdaf7094377e64787a44afd1b1d3aba6cfb8e795796a52ec62c7f8273/json"
Apr 28 14:38:46 localhost dockerd: time="2017-04-28T14:38:46.710976473-07:00" level=debug msg="Calling POST /containers/83a4642cdaf7094377e64787a44afd1b1d3aba6cfb8e795796a52ec62c7f8273/exec"
Apr 28 14:38:46 localhost dockerd: time="2017-04-28T14:38:46.711154220-07:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdout\":true,\"Cmd\":[\"sleep\",\"5\"]}"
Apr 28 14:38:46 localhost dockerd: time="2017-04-28T14:38:46.713413598-07:00" level=debug msg="Calling POST /exec/8cc2707b8435576aacac0efe7cc618213a53068ce1bf3661813105e174818d21/start"
Apr 28 14:38:46 localhost dockerd: time="2017-04-28T14:38:46.713487037-07:00" level=debug msg="form data: {}"
Apr 28 14:38:46 localhost dockerd: time="2017-04-28T14:38:46.713549853-07:00" level=debug msg="starting exec command 8cc2707b8435576aacac0efe7cc618213a53068ce1bf3661813105e174818d21 in container 83a4642cdaf7094377e64787a44afd1b1d3aba6cfb8e795796a52ec62c7f8273"
Apr 28 14:38:46 localhost dockerd: time="2017-04-28T14:38:46.713617648-07:00" level=debug msg="attach: stdout: begin"
Apr 28 14:38:46 localhost dockerd: time="2017-04-28T14:38:46.713693591-07:00" level=debug msg="attach: stderr: begin"
Apr 28 14:38:46 localhost dockerd: time="2017-04-28T14:38:46.777806000-07:00" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"start-process\", Id:\"83a4642cdaf7094377e64787a44afd1b1d3aba6cfb8e795796a52ec62c7f8273\", Status:0x0, Pid:\"8cc2707b8435576aacac0efe7cc618213a53068ce1bf3661813105e174818d21\", Timestamp:(*timestamp.Timestamp)(0xc420f49b30)}"

Docker-java library

Apr 28 14:40:21 localhost dockerd: time="2017-04-28T14:40:21.895494978-07:00" level=debug msg="Calling GET /exec/520176ee47e0677e43a472b3d0aa06558cde6211396251197da4e0800bab8531/json"
Apr 28 14:40:21 localhost dockerd: time="2017-04-28T14:40:21.897834671-07:00" level=debug msg="Calling POST /containers/a2c5ff9dd251cb0adfdf265f0b80b2485a3a8ac86ebf8074670215a4231baf36/exec"
Apr 28 14:40:21 localhost dockerd: time="2017-04-28T14:40:21.897923688-07:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdout\":true,\"Cmd\":[\"sleep\",\"5\"],\"containerId\":\"a2c5ff9dd251cb0adfdf265f0b80b2485a3a8ac86ebf8074670215a4231baf36\"}"
Apr 28 14:40:21 localhost dockerd: time="2017-04-28T14:40:21.901574097-07:00" level=debug msg="Calling POST /exec/0a684aece85903590602ec25b5e7111f51054679c959e752c2da33380053b20a/start"
Apr 28 14:40:21 localhost dockerd: time="2017-04-28T14:40:21.901634284-07:00" level=debug msg="form data: {\"Detach\":false}"
Apr 28 14:40:21 localhost dockerd: time="2017-04-28T14:40:21.901710124-07:00" level=debug msg="starting exec command 0a684aece85903590602ec25b5e7111f51054679c959e752c2da33380053b20a in container a2c5ff9dd251cb0adfdf265f0b80b2485a3a8ac86ebf8074670215a4231baf36"
Apr 28 14:40:21 localhost dockerd: time="2017-04-28T14:40:21.901759532-07:00" level=debug msg="attach: stdout: begin"
Apr 28 14:40:21 localhost dockerd: time="2017-04-28T14:40:21.901771868-07:00" level=debug msg="attach: stderr: begin"
Apr 28 14:40:21 localhost dockerd: time="2017-04-28T14:40:21.965883218-07:00" level=debug msg="libcontainerd: received containerd event: &types.Event{Type:\"start-process\", Id:\"a2c5ff9dd251cb0adfdf265f0b80b2485a3a8ac86ebf8074670215a4231baf36\", Status:0x0, Pid:\"0a684aece85903590602ec25b5e7111f51054679c959e752c2da33380053b20a\", Timestamp:(*timestamp.Timestamp)(0xc42155ee90)}"
Apr 28 14:40:21 localhost dockerd: time="2017-04-28T14:40:21.965957457-07:00" level=debug msg="libcontainerd: event unhandled: type:\"start-process\" id:\"a2c5ff9dd251cb0adfdf265f0b80b2485a3a8ac86ebf8074670215a4231baf36\" pid:\"0a684aece85903590602ec25b5e7111f51054679c959e752c2da33380053b20a\" timestamp:<seconds:1493415621 nanos:965592193 > "

I attempted to break it down into curl commands to see if I could reproduce it at this level - but it doesn't appear to work.

echo "spotify library in curl version.."
containerId=$(docker run -d busybox:latest sleep 99999999)
while [ 1 ]; do
  echo "Calling GET for ${containerId}"
  curl -s -o /dev/null -w "Response: %{http_code}\n" --unix-socket /var/run/docker.sock "http:/containers/${containerId}/json"
  echo "Creating container exec for ${containerId}"
  uuid=$(curl -s -H "Content-Type: application/json" --data '{"AttachStderr":true,"AttachStdout":true,"Cmd":["/bin/sh","-c","sleep 5 && echo done"]}' --request POST --unix-socket /var/run/docker.sock "http:/containers/${containerId}/exec" | jq -r .Id)
  echo "Starting container exec ${uuid}"
  curl -H "Content-Type: application/json" --data '{}' --request POST --unix-socket /var/run/docker.sock "http:/exec/${uuid}/start"
done

I found a way for me to work around the issue for now by adding DockerClient.ExecStartParameter.DETACH to the execStart call, waiting for exec to finish, and then looking at exit code. I don't need the output anyways for my use case.

java -cp ~/repro.jar spotifyworkaround.Testing
Started busybox container: c296ae580db7e850045d06ba6267ef1c662be1385f1f801d367da186cda00855
Now running exec loop in this container..
Still running: 59aa11eee358bd7acc48afbb7eefba6fbc334da8806cd9208e26ddc395d1d016
Still running: 59aa11eee358bd7acc48afbb7eefba6fbc334da8806cd9208e26ddc395d1d016
Still running: 59aa11eee358bd7acc48afbb7eefba6fbc334da8806cd9208e26ddc395d1d016
Still running: 59aa11eee358bd7acc48afbb7eefba6fbc334da8806cd9208e26ddc395d1d016
Still running: 59aa11eee358bd7acc48afbb7eefba6fbc334da8806cd9208e26ddc395d1d016
Exit code: 0
Still running: 51e5d81a4c8bd920f595b8868c1dcf55c5a28872eb77964bd4d3ac046e03a243
Still running: 51e5d81a4c8bd920f595b8868c1dcf55c5a28872eb77964bd4d3ac046e03a243
Still running: 51e5d81a4c8bd920f595b8868c1dcf55c5a28872eb77964bd4d3ac046e03a243
Still running: 51e5d81a4c8bd920f595b8868c1dcf55c5a28872eb77964bd4d3ac046e03a243
Still running: 51e5d81a4c8bd920f595b8868c1dcf55c5a28872eb77964bd4d3ac046e03a243
Exit code: 0

[dan-v]

@johnflavin - let me know if there is any additional debugging I can help with on this issue.

[johnflavin]

This is great! Thanks. One additional piece of information I would like to see is logs of the timing. Could you make a tweak to your reproduction that logs the times of each operation?

I have a suspicion (I won't call it a hypothesis, because it is based on very little) that the cause of the problem is making too many exec calls too fast. It seems that everything that can cause the issue to disappear is something that slows down some part of the process:

• docker debug logging on,
• communicating through tcp instead of socket (though I'm actually not 100% sure tcp is slower, it would make sense to me),
• keeping the exec connection attached,
• running the VM on a single core.

It is such a shame that this issue doesn't occur when docker debug logging is on. I reeeeeally want to see what is going on inside docker when this error happens.

I will try this out myself when I get some time. Thanks again, this looks really helpful.

[dan-v]

I added some timestamps. It doesn't appear that the speed in which the exec calls are executed matters. I increased the sleep command to 20 seconds and it still reproduces.

java -cp ~/repro.jar spotify.Testing
23:13:42.645 [main] INFO                 spotify.Testing - Running spotify test..
23:13:42.922 [main] INFO                 spotify.Testing - Creating busybox container..
23:13:43.817 [main] INFO                 spotify.Testing - Started busybox container: 1dfdfe1777099b72fd6220917b92884c2cba462896b54c6dbf08a539c5541417
23:13:43.818 [main] INFO                 spotify.Testing - Now running exec loop in this container..
23:13:43.818 [main] INFO                 spotify.Testing - Starting execCreate.. 1
23:13:43.964 [main] INFO                 spotify.Testing - Starting execStart.. 1
23:13:43.968 [main] INFO                 spotify.Testing - Starting readFully.. 1
23:14:04.023 [main] INFO                 spotify.Testing - Starting execInspect.. 1
23:14:04.031 [main] INFO                 spotify.Testing - Exit code: 0
23:14:04.031 [main] INFO                 spotify.Testing - Starting execCreate.. 2
23:14:04.038 [main] INFO                 spotify.Testing - Starting execStart.. 2
23:14:04.040 [main] INFO                 spotify.Testing - Starting readFully.. 2
23:14:24.096 [main] ERROR                spotify.Testing - java.lang.RuntimeException: java.io.IOException: Connection reset by peer 2
23:14:24.096 [main] INFO                 spotify.Testing - Starting execCreate.. 3
23:14:24.105 [main] INFO                 spotify.Testing - Starting execStart.. 3
23:14:24.108 [main] INFO                 spotify.Testing - Starting readFully.. 3
23:14:44.159 [main] ERROR                spotify.Testing - java.lang.RuntimeException: java.io.IOException: Connection reset by peer 3

[dan-v]

I also tried to look at the traffic being sent on the unix socket using socat. Amazingly, as soon as I run with this setup I'm again unable to reproduce the issue.

Setup a "mitm" socket. Here you will see all the traffic happening.

socat -v UNIX-LISTEN:/tmp/fake,fork UNIX-CONNECT:/var/run/docker.sock

Tell our program to use /tmp/fake socket for docker

DOCKER_HOST=unix:///tmp/fake java -cp ~/repro.jar spotify.Testing
23:50:36.826 [main] INFO                 spotify.Testing - Running spotify test..
23:50:37.090 [main] INFO                 spotify.Testing - Creating busybox container..
23:50:38.121 [main] INFO                 spotify.Testing - Started busybox container: bf4e3dbb0cf15645ede9eb08d8f953146a394b4e7acee13834313d7b49237b3d
23:50:38.122 [main] INFO                 spotify.Testing - Now running exec loop in this container..
23:50:38.122 [main] INFO                 spotify.Testing - Starting execCreate.. 1
23:50:38.262 [main] INFO                 spotify.Testing - Starting execStart.. 1
23:50:38.267 [main] INFO                 spotify.Testing - Starting readFully.. 1
23:50:43.318 [main] INFO                 spotify.Testing - Starting execInspect.. 1
23:50:43.329 [main] INFO                 spotify.Testing - Exit code: 0
23:50:43.329 [main] INFO                 spotify.Testing - Starting execCreate.. 2
23:50:43.341 [main] INFO                 spotify.Testing - Starting execStart.. 2
23:50:43.344 [main] INFO                 spotify.Testing - Starting readFully.. 2
23:50:48.395 [main] INFO                 spotify.Testing - Starting execInspect.. 2
23:50:48.400 [main] INFO                 spotify.Testing - Exit code: 0
23:50:48.400 [main] INFO                 spotify.Testing - Starting execCreate.. 3
23:50:48.416 [main] INFO                 spotify.Testing - Starting execStart.. 3
23:50:48.420 [main] INFO                 spotify.Testing - Starting readFully.. 3
23:50:53.473 [main] INFO                 spotify.Testing - Starting execInspect.. 3
23:50:53.479 [main] INFO                 spotify.Testing - Exit code: 0

[dan-v]

After playing around with this a fair bit - my suspicion (also based on very little) is that this is an issue with this client library (or one of the libraries it depends on (e.g. jersey)). Using socat I was able to send the exact byte for byte repro data to the docker unix socket, and I did not see any errors or issues in the response that would indicate an issue with the docker daemon. Anyways, I have a workaround - but let me know if I can help in any way - at this point I'm just very curious to know the root cause.

[mattnworb]

@dan-v by any chance is your test program using more than one thread? There is a weird deadlock issue going on in #727 and just want to rule that out from being related here.

[dan-v]

oops posting to right place. @mattnworb - I'm not using multiple threads in my test program.

[MrMonotone]

I am able to produce the issue with the example usage that is provided in the readme. What is the status on this?

[johnflavin]

I have tried and failed to reproduce the issue, so I haven't been able to dig any deeper into the cause. The only thing I can do right now is to point you to the workarounds that others have mentioned above. I hope one or more can work for your use case.

• Turn docker debug logging on
• Connect to docker over TCP, not the socket
• Detatch from the exec instance when starting it; i.e. do not stream the logs, just read them when the exec is finished

[rgrunber]

I've reproduced this bug as well with the snippet from @ArcticPheenix . I've only been able to confirm a lot of the same things mentioned by others, and possibly a simpler way to work around the issue entirely on the client-side (without modifying the daemon itself, or changing to a tcp endpoint).

• It only happens with UNIX sockets
• It doesn't happen when debug-level logging is enabled for the daemon (eg. -D for daemon options)
• It happens very consistently when debugging (eg. create a test that reproduces the issue in docker-client and run with -Dmaven.surefire.debug -Dtest=DefaultDockerClientTest#myTest)
• It doesn't happen when attempting to intercept the socket in any way with socat (whether going from unix to tcp, or tcp to unix, or even unix to unix).
• Occurs much more frequently on "faster" machines

The only useful thing is that the issue seems to go away for me when passing the ExecCreateParam.attachStdin() parameter to the execCreate() call in addition to the stdout/stderr options. My guess is that when STDIN is specified, the daemon drops any responsibility it may have with respect to that particular connection, and so it never sends the RST packet.

[framebassman]

I faced with issue and unfortunately switched to another library - docker-java. Here is sample of code which I use:

public String executeCommand(String containerName, String[] command) throws Throwable {
        DockerClientConfig config = DefaultDockerClientConfig.createDefaultConfigBuilder()
                .withDockerHost("unix:///var/run/docker.sock")
                .build();

        DockerClient docker = DockerClientBuilder.getInstance(config)
                .build();

        final List<Container> containers = docker.listContainersCmd().exec();
        final List<Container> filteredContainer = containers.stream().filter(c -> c.getNames()[0].contains(containerName)).collect(Collectors.toList());
        String id = filteredContainer.get(0).getId();

        ExecCreateCmdResponse execCreateCmdResponse = docker.execCreateCmd(id)
                .withCmd(command).withAttachStdout(true)
                .exec();

        ByteArrayOutputStream outputStream = new ByteArrayOutputStream();

        docker.execStartCmd(execCreateCmdResponse.getId())
                .withTty(true)
                .withStdIn(System.in)
                .exec(new ExecStartResultCallback(outputStream, System.err))
                .awaitCompletion();

        return new String(outputStream.toByteArray(),"UTF-8");
    }

[saidbouras]

@framebassman You docker cliient is listening on unix socket or local tcp ? Because with docker-driver it's only on unix socket that occurred this issue. Regards

[framebassman]

@saidbouras This issue is reproduced with docker client on my ubuntu machine only which listening unix socket

[saidbouras]

@framebassman
And with docker-java still listening at unix socket, you don't have the same problem ?

[framebassman]

@saidbouras Yep, all works fine with code above

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.