Listening for two different ALPNs

[joschisan]

In Fedimint we want to use Iroh for server to server connections and for server to client connections. Hence any server needs to listen for two different types of incoming connections, in two different places. Could we call accept on the endpoint in two different places simultaneously and then filter out connections by ALPN like FEDIMINT_API and FEDIMINT_P2P without losing any incoming connections? Or do we need to have two separate endpoints with different NodeIds to do this?

[matheus23]

The Endpoint stores a queue of Incoming packets that are detected to be either incoming connection attempts or just some kind of IP packet flying around:

/// State directly involved in handling incoming packets
struct RecvState {
    incoming: VecDeque<proto::Incoming>,
    connections: ConnectionSet,
    recv_buf: Box<[u8]>,
    recv_limiter: WorkLimiter,
}

When you endpoint.accept().await, you're locking the endpoint and popping off of this queue:

impl Future for Accept<'_> {
    type Output = Option<Incoming>;
    fn poll(self: Pin<&mut Self>, ctx: &mut Context<'_>) -> Poll<Self::Output> {
        let mut this = self.project();
        let mut endpoint = this.endpoint.inner.state.lock().unwrap();
        // ...
        if let Some(incoming) = endpoint.recv_state.incoming.pop_front() {
            // Release the mutex lock on endpoint so cloning it doesn't deadlock
            drop(endpoint);
            let incoming = Incoming::new(incoming, this.endpoint.inner.clone());
            return Poll::Ready(Some(incoming));
        }
        // ...
    }
}

So if you share the Endpoint between two places, then, yes, they can both accept incoming connections. However, each incoming packet will only make it to one endpoint at a time, not to both.
So if you filter for ALPN on each endpoint, you'll miss roughly half the incoming connection attempts.

EDIT: The right way to do this, is as @flub says: Use a single accept loop and route messages depending on the ALPN. The iroh::protocol::Router can do this for you.

[flub]

Another way of saying this is that for each endpoint you should have a single "accept loop".

Multiplexing 2 protocols on a single Endpoint using different ALPNs is totally fine though and we even recommend it! It will share more resources than using two independent endpoints.

The iroh::protocol::Router is the tool we usually use to multiplex several protocols: it takes care of having that single accept loop and allowing several protocols to be implemented, without having to worry much about each other. I'd recommend you look at adopting this for your two protocols.

[joschisan]

Thanks for the quick answer!