TCP Reset after 5mins from Azure Load Balancer

[dtcos]

I have configured a Query Stream to consume from a KSQLDB app on a Kafka cluster, the Kafka cluster is managed by Confluent Cloud and hosted on Microsoft Azure in the North Europe region. During off-peak times, my query stream stops receiving messages if the stream has been idle for more than 5 minutes (i.e. no messages in a 5 minute window). Taking a pcap trace using Wireshark I can see that the cluster endpoint I connect to sends a TCP reset after 5 minutes of idle time. This TCP Reset is not interpreted or understood by the "ksqlDB.RestApi.Client" and therefore my app continues listening using the "Subscribe" API even though the underlying TCP connection is in fact closed. I raised this issue with Confluent Cloud support and they could not replicate the issue however they did say that Azure Load Balancers terminate idle TCP connections after about 5 minutes which is very similar if not the exact issue I have.

In an attempt to find a solution, I enabled a TCP keep alive over the stream. The solution includes upgrading to .NET6 and replacing http handler "HttpClientHandler" in the "ksqlDB.RestApi.Client" lib with the more feature rich "WinHttpHandler" like so

        var winHttpHandler = new WinHttpHandler();
        winHttpHandler.TcpKeepAliveEnabled = true;
        winHttpHandler.TcpKeepAliveTime = TimeSpan.FromMinutes(2);
        winHttpHandler.TcpKeepAliveInterval = TimeSpan.FromMinutes(2);
        winHttpHandler.ReceiveHeadersTimeout = Timeout.InfiniteTimeSpan;
        winHttpHandler.ReceiveDataTimeout = Timeout.InfiniteTimeSpan;
        winHttpHandler.SendTimeout = Timeout.InfiniteTimeSpan;

        InnerHandler = winHttpHandler;// new HttpClientHandler();

the "WinHttpHandler" in .NET6 allows me to set the TCP keep alive on the socket along with setting infinite timeouts. This fixed my issue however I would like to know if setting the TCP keep alive can be exposed someway via the "ksqlDB.RestApi.Client" (and built for .NET6). Is something like this possible as I don't want to develop my own lite version of the "ksqlDB.RestApi.Client" for .NET6 with WinHttpHandler in the http pipeline?

[tomasfabian]

Hi @dtcos,
could you please try to replace the InnerHandler as in the following example?

Http interception with ksqldb .NET client.

Thank you.

[dtcos]

That worked, Thank you!

[tomasfabian]

These are great news @dtcos indeed!
Is my assumption correct that WinHttpHandler is only a MS Windows solution? Would this help also in Linux containers? I guess that no.
Sorry that I haven't had time yet to try it out. Because in that case I cannot add it as a general solution inside "ksqlDB.RestApi.Client" package.

Thank you very much @dtcos for the cooperation!
Regards Tomas

[tomasfabian]

I found this in the documentation:
WinHttpHandler is implemented as a thin wrapper on the WinHTTP interface of Windows and is only supported on Windows systems.

It would be nice to find a similar solution for Linux.

[dtcos]

Yes its specific to Windows so I can understand why it can't be added inside the "ksqlDB.RestApi.Client" package. Just an fyi if you are interested, there were 2 other (probably unrealistic) options I considered, option 1 using "HttpWebRequest" instead of "HttpClient" to avail of "ServicePointManager" which may work for other OS's however I couldn't get http2.0 working with "HttpWebRequest" therefore I would have to use the older restapi "query" instead of "query-stream". Second option was to produce a dummy message and dispatch to my topic almost like a keep alive message. I will be sticking with WinHttpHandler for now - thanks for the help