An TCP driven architecture in Kubernetes

[yquansah]

Hello 👋 ,

First of all really awesome project. Love to see these different tunneling solutions, as it solves a lot of issues with connecting to private servers.

I have a use case where I would love your input.

The overall premise is, there are multiple ssh servers running on several machines that are not publicly accessible and we would like to leverage Pico to allow proxy clients to connect to these ssh servers.

There exist in our domain a public server which runs Kubernetes that would probably run piko server and then the ssh servers running on private machines can register themselves to. I would imagine we would run a piko agent on the machines with the ssh servers, and somehow run piko forward in our public server that knows the endpoint of the upstream TCP connection.

This would need to happen in "real time" however, and by "real time" what I mean is:

You are a client requesting an ssh server from a UI, that one request should:

1. Spin up the ssh server with piko agent on the private machine (we have connectivity with the private server for admin level capabilities through a VPN (well tailscale))
2. Run piko forward on our public server for each upstream TCP connection we receive?

The last point is what I am unclear of how to tackle, and would love input from the maintainers about if I am thinking of the architecture correctly.

Thanks, and look forward to hearing from you all! 😄

[andydunstall]

Hi @yquansah 👋

Great to hear about your use case! I'm in London timezone so won't be able to come up with a good response today, but I'll have a think and get back to you tomorrow morning

[andydunstall]

Morning 👋

Spin up the ssh server with piko agent on the private machine

I don't see any issues there. Piko also has a Go SDK for opening upstream listeners (to replace Piko agent) and dialling Piko upstreams (to replace Piko forward) which might be more convenient for programatically setting up SSH servers and upstreams?

Run piko forward on our public server for each upstream TCP connection we receive?

Piko forward is built to run locally on your machine, it will listen on a local TCP port and forward to the Piko server.

As Piko forward is listening on a raw TCP port, it can't authenticate incoming traffic, so if you hosted it publicly it means anyone could send requests via Piko to your upstream listeners? Traffic between Piko forward and the Piko server can be sent via TLS and authenticated as it wraps the underlying TCP connection.

So if you were using Piko forward to connect with SSH, you'd typically run both the SSH client and Piko forward locally. (As mentioned above theres also a Go SDK incase that makes it easier).

Note Piko was originally build for HTTP traffic, and TCP support is a fairly recent addition. I haven't really thought about using Piko for SSH, so I'll see if theres a better way for Piko to support SSH this weekend, and check what other tunneling solutions do.

Let me know if you have any suggestions or if you've seen other tunneling solutions that handle SSH better :)

[yquansah]

@andydunstall Thanks for the reply, some great information in here.

As Piko forward is listening on a raw TCP port, it can't authenticate incoming traffic, so if you hosted it publicly it means anyone could send requests via Piko to your upstream listeners? Traffic between Piko forward and the Piko server can be sent via TLS and authenticated as it wraps the underlying TCP connection.

I see the concern here, and it is true that users will be able to connect a random TCP client and send data over that connection. We do want ssh only to be recognized at the private machine layer, everything downstream of the private machine should just speak tcp. So we probably should consider writing a basic software level firewall to only allow certain machines to connect.

I was actually able to get a POC quickly running in Kubernetes with TCP using piko.

The potential and promise of this project is great, if I can help contribute anywhere to help with TCP support, i'd be happy to 😄

[yquansah]

@andydunstall Ah reading more into this I see more of the concern. Basically, if the piko forward was hosted publicly, traffic between the ssh client and piko forward can be easily sniffed out by a man-in-the-middle since it is unencrypted.

[condaatje]

what if we pre-load the central ssh host key from the piko public server into authorized_hosts on the private hosts? that would mitigate MITM when establishing the tunnel, no?

and for reference @andydunstall, we do indeed want our public server to be accessible from the public internet, as we are building a p2p cloud hosting platform. Piko may be how we allow customers to rent servers that don't have a public IP - they ssh via piko, because they cannot ssh directly

[condaatje]