Peer count slowly decrease to 0

[mask-pp]

Description

Please provide a brief description of the issue.

The peer count of beacon_node will slowly decrease to 0 in Holesky network.

Please provide your Lighthouse and Rust version. Are you building from
stable or unstable, which commit?

sigp/lighthouse:v5.3.0

Describe the present behaviour of the application, with regards to this
issue.

issue behavior:
Once the peer count is lower than a certain threshold(about 96), the count value begins to slowly decrease but no possibility of any increase.

geth(v1.13.15) cmd:
`

• geth
  --holesky
  --datadir /data/holesky-node-full
  --metrics
  --metrics.addr "0.0.0.0"
  --http
  --http.addr "0.0.0.0"
  --http.vhosts ""
  --http.corsdomain ""
  --http.api eth,net,web3,txpool
  --ws
  --ws.addr "0.0.0.0"
  --ws.origins ""
  --ws.api eth,net,web3,txpool
  --authrpc.addr "0.0.0.0"
  --authrpc.vhosts ""
  --authrpc.jwtsecret /etc/jwt/secret.hex
  --nat extip:$EXT_IP
  --allow-insecure-unlock
  --v5disc
  `

beacon(v5.3.0) cmd:
`

• lighthouse
  - beacon
  - --network
  - holesky
  - --datadir
  - /data/lighthouse
  - --execution-endpoint
  - http://localhost:8551
  - --execution-jwt
  - /etc/jwt/secret.hex
  - --checkpoint-sync-url
  - https://checkpoint-sync.holesky.ethpandaops.io
  - --disable-deposit-contract-sync
  - --http-address
  - 0.0.0.0
  - --http
  `

How should the application behave?

The normal peer count is about 100 in holesky network, and should be restored automatically when the peer count is too few.

Please describe the steps required to resolve this issue, if known.

[AgeManning]

I think we are going to need some logs to diagnose this.

The described behaviour is similar to a node that loses an internet connection.

Debug logs can be found in the /data/lighthouse/holesky/beacon_node/logs directory. Pasting those here, or DM'ing me on discord (@AgeManning) will help us figure out the issue.

[mask-pp]

I think we are going to need some logs to diagnose this.

The described behaviour is similar to a node that loses an internet connection.

Debug logs can be found in the /data/lighthouse/holesky/beacon_node/logs directory. Pasting those here, or DM'ing me on discord (@AgeManning) will help us figure out the issue.

Thx man, since the debug log is too large, I need to crop out the useful information and then send it to you.

[michaelsproul]

@mask-pp The logs compress well. It's best if you can compress them and send the whole file as it is all potentially relevant

Something else you could check would be your time sync. Make sure you've got NTP running and that sudo timedatectl status shows you're synced. You could also try Chrony.

[mask-pp]

I think we are going to need some logs to diagnose this.
The described behaviour is similar to a node that loses an internet connection.
Debug logs can be found in the /data/lighthouse/holesky/beacon_node/logs directory. Pasting those here, or DM'ing me on discord (@AgeManning) will help us figure out the issue.

Thx man, since the debug log is too large, I need to crop out the useful information and then send it to you.

@AgeManning Hei friend, I have sent u the get log cmd privately. Hope these logs are helpful in solving the issue.

[chong-he]

Are you on VPS or running Lighthouse locally?

[mask-pp]

Are you on VPS or running Lighthouse locally?

Running in k8s

[chong-he]

Linking a similar issue here: #5271

[AgeManning]

I have been through these logs.

The logs show "Socket Updated" (you can grep through the logs for this).

This log indicates that discovery is changing contactable IP/PORT based on what others see as the src in the packets they receive.
It starts out with a port of 9000 (which is usually correct) then when it changes to a random other port, lighthouse can no longer discover peers. This is because other nodes will not respond if the ENR has invalid settings.

It typically means that the router or gateway is sending traffic to other peers on different ports other than 9000. This could be because of a symmetric nat for example. Usually on home router's this means the ports are not forward'ed correctly. Setting up a UDP port forward should make the router move traffic in and out through the same external port. If it is using other random ports, the ENR can be updated incorrectly.

I've seen this happen a few times and there is some changes to discovery we can make that might improve this situation. I'll make some PRs.

The immediate solution is to verify why traffic is being sent out on different random ports and to double check the NAT configuration for the UDP discovery traffic.

[AgeManning]

For reference. I'm suggesting the following discv5 change: sigp/discv5#265

This will resolve this issue once we make a discv5 release and put it into lighthouse. This should resolve a bunch of other related issues.

There is a downside however. This change will allow lighthouse nodes to find and maintain peers, but a misconfigured NAT will now be harder to identify, because Lighthouse will work (partially). It will be harder to notice this misconfiguration. The result will be that inbound peers will not join, because the ENR will not be advertised.

We have a metric (in grafana, in the network dash) that tells if the NAT is correctly open. Also the HTTP API lighthouse/nat should be an indicator if the nat is configured correctly.