What is ResourceLimitExceeded?

[SWvheerden]

We are seeing this in our deployment of libp2p nodes.

WARN  Outgoing connection error, ignoring: PeerId("12D3...g4CH") -> [(/ip4/x.x.x.x/udp/x/quic-v1/p2p/12D3...aYAf/p2p-circuit/p2p/12D3...4CH, Other(Custom { kind: Other, error: Other(Left(Left(Left(Connect(ResourceLimitExceeded))))) }))] // src\server\p2p\network.rs:1377

The error does not seem very descriptive. We are not running the nodes with any connection limits, while the relays are running with:

  --relay-server-max-circuits=1000
  --relay-server-max-circuits-per-peer=40

And currently, we have about just over a 1000 nodes in the network.
Any idea what resource we are exceeding here?

[dariusc93]

Hey! It may be likely that the relay server exceeded its resources which would report that error. Increasing the circuits limit would likely help though.

[SWvheerden]

but is it that?
We are running a few relays, and have about 1,200 nodes. So just above the relays. We should not have all the nodes connecting to all relays. But if ResourceLimitExceeded == server-max-circuits then it is the case, which from looking at the logs, dont seem to be the case.

[elenaf9]

Are the nodes distributed evenly between the relays or all using the same relay for listening? And do they only listen on a single relayed address?

[SWvheerden]

They are supposed to spread out among the relays. But it could be that they are clumping among a few

[elenaf9]

Yes so the possible ways to exceed the resource limits are:

• more than relay-server-max-circuits overall reservations are active on the same relay node, e.g. because a larger number of different nodes are clumping on the same relay
• one peer tries to make more than relay-server-max-circuits-per-peer active reservations at a relay node at the same time
• the rate-limit is exceeded:
  • reservation_rate_per_peer: default one reservation every 2 mins, up to 30 reservations per hour
  • reservation_rate_per_ip: default one reservation every 1 min, up to 60 reservations per hour

The last point may be relevant for you if your nodes frequently stop and start listening on the relay again.

[dariusc93]

Yes so the possible ways to exceed the resource limits are:

• more than relay-server-max-circuits overall reservations are active on the same relay node, e.g. because a larger number of different nodes are clumping on the same relay

• one peer tries to make more than relay-server-max-circuits-per-peer active reservations at a relay node at the same time

• the rate-limit is exceeded:

  • reservation_rate_per_peer: default one reservation every 2 mins, up to 30 reservations per hour
  • reservation_rate_per_ip: default one reservation every 1 min, up to 60 reservations per hour

The last point may be relevant for you if your nodes frequently stop and start listening on the relay again.

I think it might be worth adding documentation to libp2p-relay Config to explain that to reduce the confusion. I think its commented internally in the Config::default (would have to double check) but probably isnt clear about what to supply to the other functions in Config or the public fields of Config.

[retrohacker]

Tangential but related question, is there a way to determine which resource's limit was exceeded? We've been running into this a lot on our relays and had a heck of a time narrowing down which resource was exhausted. It seems like all code paths for the relay emit the same message when checking resource limits.

[dariusc93]

Tangential but related question, is there a way to determine which resource's limit was exceeded? We've been running into this a lot on our relays and had a heck of a time narrowing down which resource was exhausted. It seems like all code paths for the relay emit the same message when checking resource limits.

We probably could make it more verbose since there are different knobs in the configs.

If one did not care about the amount of resources the relay may use, one could arguably max all the values out, which would mimic how circuit v1 was, but that does pose a risk as it could be virtually unbounded in terms of how many reservations are made, the amount of data being relayed, etc.

[retrohacker]

We may have done that (temporarily) while we try to derive appropriate values for those knobs from prod metrics 😉