TCP connection leakage issue when using the remote port forwarding feature

[Moonergfp]

Version

2.10.0

Bug description

When utilizing the remote port forwarding functionality, I've noticed that TCP connections are not being properly closed between the SSH client process and the local port on the SSH client side.The issue occurs intermittently when the machine's resources are under significant strain, and currently, I can only reproduce it using the IntelliJ IDEA debug functionality.

As depicted in the attached image, these connections fail to close correctly between the SSH client and "near" component.

Steps to Reproduce

1. start client and server process in debug mode
   client code

 public static void main(String[] args) {
    SshClient sshClient = createSshClient();
    while (true) {
      startRemoteForwarding(sshClient);
      log.info("session is closed. starting next loop");
    }
  }

  private static void startRemoteForwarding(SshClient sshClient) {
    try (ClientSession session = sshClient.connect("abc", REMOTE_IP, REMOTE_PORT)
            .verify(10, TimeUnit.SECONDS)
            .getSession()) {
      session.auth().verify(10, TimeUnit.SECONDS);
      log.info("start remote port forwarding");
      try (PortForwardingTracker tracker = session.createRemotePortForwardingTracker(
              new SshdSocketAddress(0), new SshdSocketAddress("localhost", 8001))) {
        while (true) {
          try {
            log.info(" remote port forwarding loop... host:{},port:{}", "localhost", 8001);
            TimeUnit.SECONDS.sleep(15);
            if (session.isClosed()) {
              log.error("session is closed,recreate remote forwarding");
              break;
            }
          } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            break;
          }
        }
      }
    } catch (Exception e) {
      e.printStackTrace();
    }
  }

  private static SshClient createSshClient() {
    SshClient sshClient = SshClient.setUpDefaultClient();
    sshClient.setIoServiceFactoryFactory(new NettyIoServiceFactoryFactory());
    sshClient.setFilePasswordProvider(FilePasswordProvider.EMPTY);

    sshClient.setServerKeyVerifier(new StationKeyVerifier());
    sshClient.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);

    CoreModuleProperties.HEARTBEAT_REQUEST.set(sshClient, "keepalive@yeap.com");
    CoreModuleProperties.HEARTBEAT_INTERVAL.set(sshClient, Duration.ofSeconds(30));
    CoreModuleProperties.HEARTBEAT_REPLY_WAIT.set(sshClient, Duration.ofSeconds(20));

    sshClient.start();
    System.out.println("Client started");
    return sshClient;
  }

server code

  public static void main(String[] args) throws IOException {
    SshServer sshServer = SshServer.setUpDefaultServer();
    sshServer.setPort(SSH_PORT);
    sshServer.setIoServiceFactoryFactory(new NettyIoServiceFactoryFactory());
    sshServer.setUserAuthFactories(
            Collections.singletonList(new StationUserAuthPublicKeyFactory()));

    StationAuthenticator authenticator = new StationAuthenticator();
    sshServer.setPublickeyAuthenticator(authenticator);

    String serverHostPrivateKey = "-----BEGIN OPENSSH PRIVATE KEY-----\n" +
            "b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW\n" +
            "QyNTUxOQAAACBjTjHWMtYW7jP7cTt4FqBJFAqaj3v5vakOcTHh6HZWMAAAAKAiwJe4IsCX\n" +
            "uAAAAAtzc2gtZWQyNTUxOQAAACBjTjHWMtYW7jP7cTt4FqBJFAqaj3v5vakOcTHh6HZWMA\n" +
            "AAAECTa0CWNOJuBGfrwO1GeyGeAkFCt0PS2A5r0DQy2rwiFGNOMdYy1hbuM/txO3gWoEkU\n" +
            "CpqPe/m9qQ5xMeHodlYwAAAAHHJvb3RAaVoyemU0eGh4eW5hODl4NjA4YmQyZ1oB\n" +
            "-----END OPENSSH PRIVATE KEY-----";
    sshServer.setKeyPairProvider(
            session ->
                    SecurityUtils.getKeyPairResourceParser()
                            .loadKeyPairs(
                                    session,
                                    NamedResource.ofName("station-config"),
                                    null,
                                    serverHostPrivateKey));
    sshServer.setCommandFactory(new ProcessShellCommandFactory());

    sshServer.setPasswordAuthenticator(null);
    sshServer.setForwardingFilter(AcceptAllForwardingFilter.INSTANCE);
    sshServer.addPortForwardingEventListener(new CustomPortForwardingEventListener());

    sshServer.start();
    log.info("Server started");

    new ChannelProbe(sshServer).start();
    log.info("Probe started");

    try {
      Thread.currentThread().join();
    } catch (InterruptedException e) {
      log.info("tunnel station interrupted, stopping", e);
      sshServer.stop();
      Thread.currentThread().interrupt();
    }
  }

public class ChannelProbe {

  private SshServer sshServer;
  private ScheduledExecutorService scheduledExecutorService =
          Executors.newSingleThreadScheduledExecutor();

  private static final int STARTING_DELAY_SEC = 20;
  private static final int STARTING_DELAY_JIFFER_SEC = 10;
  private static final int PROBE_INTERVAL_SEC = 20;

  public ChannelProbe(SshServer sshServer) {
    this.sshServer = sshServer;
  }

  public static int getStartingJiffer() {
    Random random = new Random();
    return random.nextInt(STARTING_DELAY_JIFFER_SEC);
  }

  public void start() {
    int startInSec = STARTING_DELAY_SEC + getStartingJiffer();
    scheduledExecutorService = Executors.newSingleThreadScheduledExecutor();
    scheduledExecutorService.scheduleWithFixedDelay(
            () -> {
              probeTunnels();
            },
            startInSec,
            PROBE_INTERVAL_SEC,
            TimeUnit.SECONDS);
  }

  private void probeTunnels() {
    List<AbstractSession> activeSessions = sshServer.getActiveSessions();
    if (activeSessions.isEmpty()) {
      log.info("there is no active session when probe tunnel");
      return;
    }
    for (AbstractSession activeSession : activeSessions) {
      SshSessionContext sshSessionContext = activeSession.getAttribute(CustomPortForwardingEventListener.SESSION_ATTR_KEY);
      if (sshSessionContext == null) {
        log.warn("sessoin context is null. session={}", activeSession);
        continue;
      }
      try (Socket testSocket = new Socket(sshSessionContext.getPortForwardingRemoteHost(), sshSessionContext.getPortForwardingBoundPort())) {
        log.info("probe tunnel.  session={},sshSessionContext={}", activeSession, sshSessionContext);
      } catch (Exception e) {
        log.error("probe tunnel failed", e);
      }
    }
  }
}

2. Set Breakpoints
   Navigate to the org.apache.sshd.server.forward.TcpipServerChannel#handleChannelConnectResult method and set a breakpoint on the line ”if (future.isConnected())“.

The server initiates asynchronous and periodic connections to port A. Subsequently, the client will automatically connect to port 8001 to establish a connection. During this process, the code execution will pause at the breakpoint set in the TcpipServerChannel#handleChannelConnectResult method.
It is recommended that you wait for a while, perhaps a few minutes.After that, resume the code execution from the breakpoint.At this point, the session will actually be closed. Logically, all connections to port 8001 should also be terminated.
However, when you execute the ss command on the client machine with the following filter: ss -lanp |grep 8001 |grep ES |grep java, you will notice that some connections remain open and have not been properly closed.

Actual behavior

Under resource - constrained conditions and when using IntelliJ IDEA's debug mode, connections between the SSH client and the "near" component do not close as expected, resulting in resource leakage and potential system instability.

Expected behavior

Regardless of the system's resource state, once the SSH client connections are closed or the relevant operations are completed, all connections between the SSH client and the "near" component should be properly terminated, and system resources should be released.

Relevant log output

Other information

No response

[tomaswolf]

Thanks for reporting this. This race condition is ugly and, other than I had expected, not specific to this TCP/IP forwarding. The TcpipServerChannel does everything correctly. It connects to port B, and if the channel or session is closed, it also closes the IoConnector it uses to make that connection to B.

The problem is in the IoConnector and the way low-level IoSessions are created there. The race condition exists in all three implementations (NIO2, Mina, and Netty).

IoSessions are created and registered in the IoConnector, and when the IoConnector is closed, it also closes all registered sessions. But IoSessions are created asynchronously, and so it is possible that an IoSession is registered after the IoConnector has already been closed. When that happens, we end up with an IoSession that will probably remain around forever.

[Moonergfp]

Thanks for reporting this. This race condition is ugly and, other than I had expected, not specific to this TCP/IP forwarding. The TcpipServerChannel does everything correctly. It connects to port B, and if the channel or session is closed, it also closes the IoConnector it uses to make that connection to B.

The problem is in the IoConnector and the way low-level IoSessions are created there. The race condition exists in all three implementations (NIO2, Mina, and Netty).

IoSessions are created and registered in the IoConnector, and when the IoConnector is closed, it also closes all registered sessions. But IoSessions are created asynchronously, and so it is possible that an IoSession is registered after the IoConnector has already been closed. When that happens, we end up with an IoSession that will probably remain around forever.

So, in what way should I fix that issue?

[Moonergfp]

Here is my additional information regarding this issue.

There are two types of NettyIoSession objects here. The first type is for the connection established when the client connects to the server, and the second type is for the connection when the client connects to the local port (8001). When the connection between the client and the server times out, the ClientSessionImpl will be closed. During this period, if the "SSH_MSG_CHANNEL_OPEN" event is received and even after the ClientSessionImpl has been closed, a TCP connection from the client to the local port (8001) will still be established.

[tomaswolf]

So, in what way should I fix that issue?

I'm preparing a fix, for all three transports.