Low peer count on Gnosis #4490
Assignees
Labels
Comments
|
When you say low peer count. Do you mean 0 peers, what number does it fluctuate around? The configuration you have looks correct provided the It seems on the gnosis chain that there is no external nodes connecting to you, this can be really only one of a few problems:
It would be instructive to get maybe 30 seconds of discovery logs, to figure out exactly why you can't find peers. To do this, run lighthouse with the i.e from the CLI you could run: |
|
@AgeManning thanks for your inputs. |
|
@AgeManning other 2 nodes I managed to sync,
|
|
Ok thank you for these logs. The logs are saying that none of the bootnodes are responding to you any of your requests. The main reason I can think of this is that you are sending outbound UDP packets not on For example, you might be sending packets that have a src field of X.X.X.X:1040 and your EXTERNAL_PORT could be 9000. So even though port 9000 is open, a discovery node won't respond to you because the packets look like they are coming from another source. Do you have some outbound route that is different from the inbound route? One way to fix this, is just to remove the |
|
I think it's possible to modify discovery to allow these kinds of things. We could respond with more nodes but never add your ENR to the DHT. It is kind of sweeping the core issue under the rug however. I think it might be best to have the node fail (as it is now) when there is a misconfiguration happening in the network somewhere. I made this issue for discussion for reference: sigp/discv5#202 |
|
Will try removing enr related options and report back. |
|
Removed enr related options and now there are much more peers but all are |
|
Ok yep, thanks for the logs. If they all agree that we are sending messages on X:Y, then we set our ENR to say we are contactable at X:Y. Your node is sending messages to peers with different port mappings. I.e peers are reporting your packets coming from X:A, X:B, X:C etc. This is typical for a node behind a symmetric NAT. I think your problem lies in your SNAT configuration. If you can get your SNAT to specify a single source port, this will solve your problem. This way all packets coming from your node will look to originate from the same port (which is forwarded) and your ENR will be changed to this ip/port. |
|
You can see the ports change for yourself, by looking for the PONG response in the logs. The PONG details the IP and PORT of what your peers see as the source of the messages you are sending to them. |
|
Thanks for the help. I will close this issue now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I have one pair of nodes (Nethermind + Lighthouse) for Sepolia and it's running flawlessly. High number of connected peers and
/lighthouse/natreturns{"data":true}On the other hand I have three pair of nodes (Nethermind + Lighthouse) for Gnosis and there is an issue with low peer count.
/lighthouse/natreturns{"data":false}Configs and versions are the same for all nodes, of course except
--network=sepoliaand--network=gnosisand all nodes are synced.Version
Lighthouse ->
v4.3.0Nethermind ->
1.19.3Present Behaviour
All nodes are running on the same EKS (AWS k8s) cluster with enabled wide range of TCP and UDP ports. Using telnet and nmap shows that all ports are open, but only sepolia node returns
{"data":true}on/lighthouse/natendpoint.Nodes are deployed using latest (1.0.5) lighthouse Helm chart
Notable part of
values.yamlCommand used to start gnosis node
Command used to start sepolia node
EXTERNAL_IPandEXTERNAL_PORTare properly set and used.Expected Behaviour
Both sepolia and gnosis Lighthouse should return
{"data":true}on/lighthouse/natendpoint and have high number of connected (inbound and outbound) peers.The text was updated successfully, but these errors were encountered: