bug: networkReachability=NotReachable log?

[dao]

Problem

Running status-im/nwaku:0.14.0 docker image, I observe previously unseen INFO log messages:
Peer reachability status topics="wakunode" tid=1 file=waku_node.nim:225 networkReachability=NotReachable confidence=1.0

Impact

I am uncertain of the impact; the node still appears to have ~2 dozen connected peers but the log suggests that the node is not reachable

To reproduce

I'm unaware of any special steps or configurations required to reproduce. The node is running relay protocol only (no store, filter, etc) with rln disabled.

Expected behavior

If the message does not actually mean that my node is unreachable, it should not cause alarm by being logged as INFO

Screenshots/logs

see attached

nwaku version/commit hash

version / git commit hash: v0.14.0
log.txt

[alrevuelta]

I am uncertain of the impact; the node still appears to have ~2 dozen connected peers but the log suggests that the node is not reachable

Are any of these connections inbound? A node can be NotReachable and still have connected peers (outbound peers).

Note though that this is an experimental feature that was just released both in our side and nim-libp2p, and works by asking other peers if they can connect to you.

Does your node/computer have the waku port open? Or even if its closed do you have a router supporting upnp?

If you do expect your node to be reachable, and you want to give it a shot, feel free to experiment with the autonatservice. Would suggest to enable askNewConnectedPeers and changing numPeersToAsk or maxQueueSize.

[alrevuelta]

@dao After some discussions with nim-libp2p team, this is a known issue on nim-libp2p. This feature works by requesting other nodes to dial you, and if they succeed, they flag you as Reachable otherwise NotReachable.

Problem is that we currently limit the maximum number of connection a node can have to 1. So if you are already connected the the peer that you are requesting to dial you to test your reachability, it will fail because you will reject that connection. This should explain why you see NotReachable.

We can also see that this metric is really unstable in our fleets, jumping quite often from NotReachable to Reachable.

By now you can ignore the logs. Thanks for reporting!

[alrevuelta]

Work is being done in vacp2p/nim-libp2p#845 and vacp2p/nim-libp2p#846. Once fixed in nim-libp2p we will bump the version and should be fixed.

[alrevuelta]

Just merged the nim-libp2p version bump containing the fixes. Will monitor our fleets (where the issue is also present) over the next 24 hours and check if the issue is resolved.

[alrevuelta]

Can verify in our wakuv2.test fleets that the nim-libp2p version bump to 67939b was deployed, containing 4ace70d53b0b0e3b58cd3bead70b967d34bd03f3 nim-libp2p version. However, I still see the issue: Reachable and NotReachable statuses are i) wrongly reported and ii) unstable.

This needs further investigation.

[alrevuelta]

An update on this. Latest nim-libp2p version included a fix but still a bit unstable. NotReachable state is still reported on nodes that are Reachable from time to time. Issue seems to happen more often when multiple nodes try to dial each other testing reachability at the same time.

This new issue was detected by nim-libp2p team and a fix should be ready soon. Once fixed, this should be closed once we bump to the latest version.

See: vacp2p/nim-libp2p#865

[alrevuelta]

v0.16.0 should fix this.

I can verify it works in a private network with v0.16.0. Will wait until we deploy this release to our fleets before closing.

Note thought that in a network containing a mix of old (v0.15.0) and new (>v0.16.0) nodes, new nodes will be biased by old nodes (they will report dialMe failed while that may not be the case)

[alrevuelta]

v0.16.0 indeed fixed this, but if other nodes did not update, they may flag us as notreachable while being reachable. Since multiple nodes are queried for our state, its just matter of consensus (if more nodes report us as reachable then we will flag ourselves are reachable).

closing since this is fixed from v0.16.0