feat: opt-in Swarm.ResourceMgr (go-libp2p v0.18) (#8680)

* update go-libp2p to v0.18.0

* initialize the resource manager

* add resource manager stats/limit commands

* load limit file when building resource manager

* log absent limit file

* write rcmgr to file when IPFS_DEBUG_RCMGR is set

* fix: mark swarm limit|stats as experimental

* feat(cfg): opt-in Swarm.ResourceMgr

This ensures we can safely test the resource manager without impacting
default behavior.

- Resource manager is disabled by default
    - Default for Swarm.ResourceMgr.Enabled is false for now
- Swarm.ResourceMgr.Limits allows user to tweak limits per specific
  scope in a way that is persisted across restarts
- 'ipfs swarm limit system' outputs human-readable json
- 'ipfs swarm limit system new-limits.json' sets new runtime limits
  (but does not change Swarm.ResourceMgr.Limits in the config)

Conventions to make libp2p devs life easier:
- 'IPFS_RCMGR=1 ipfs daemon' overrides the config and enables resource manager
- 'limit.json' overrides implicit defaults from libp2p (if present)

* docs(config): small tweaks

* fix: skip libp2p.ResourceManager if disabled

This ensures 'ipfs swarm limit|stats' work only when enabled.

* fix: use NullResourceManager when disabled

This reverts commit b19f7c9eca4cee4187f8cba3389dc2c930258512.
after clarification feedback from
ipfs/kubo#8680 (comment)

* style: rename IPFS_RCMGR to LIBP2P_RCMGR

preexisting libp2p toggles use LIBP2P_ prefix

* test: Swarm.ResourceMgr

* fix: location of opt-in limit.json and rcmgr.json.gz

Places these files inside of IPFS_PATH

* Update docs/config.md

* feat: expose rcmgr metrics when enabled (#8785)

* add metrics for the resource manager
* export protocol and service name in Prometheus metrics
* fix: expose rcmgr metrics only when enabled

Co-authored-by: Marcin Rataj <[email protected]>

* refactor: rcmgr_metrics.go

* refactor: rcmgr_defaults.go

This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled

We keep vendored copy to ensure go-ipfs is not impacted when go-libp2p
decides to change defaults in any of the future releases.

* refactor: adjustedDefaultLimits

Cleans up the way we initialize defaults and adds a fix for case
when connection manager runs with high limits.

It also hides `Swarm.ResourceMgr.Limits` until we have a better
understanding what syntax makes sense.

* chore: cleanup after a review

* fix: restore go-ipld-prime v0.14.2

* fix: restore go-ds-flatfs v0.5.1

Co-authored-by: Lucas Molas <[email protected]>
Co-authored-by: Marcin Rataj <[email protected]>

This commit was moved from ipfs/kubo@514411b

Author: marten-seemann

peering/peering_test.go

@@ -6,20 +6,22 @@ import (
 	"time"
 
 	"github.com/libp2p/go-libp2p"
-	connmgr "github.com/libp2p/go-libp2p-connmgr"
 	"github.com/libp2p/go-libp2p-core/host"
 	"github.com/libp2p/go-libp2p-core/network"
 	"github.com/libp2p/go-libp2p-core/peer"
+	"github.com/libp2p/go-libp2p/p2p/net/connmgr"
 
 	"github.com/stretchr/testify/require"
 )
 
-func newNode(ctx context.Context, t *testing.T) host.Host {
+func newNode(t *testing.T) host.Host {
+	cm, err := connmgr.NewConnManager(1, 100, connmgr.WithGracePeriod(0))
+	require.NoError(t, err)
 	h, err := libp2p.New(
 		libp2p.ListenAddrStrings("/ip4/127.0.0.1/tcp/0"),
 		// We'd like to set the connection manager low water to 0, but
 		// that would disable the connection manager.
-		libp2p.ConnectionManager(connmgr.NewConnManager(1, 100, 0)),
+		libp2p.ConnectionManager(cm),
 	)
 	require.NoError(t, err)
 	return h
@@ -29,12 +31,12 @@ func TestPeeringService(t *testing.T) {
 	ctx, cancel := context.WithCancel(context.Background())
 	defer cancel()
 
-	h1 := newNode(ctx, t)
+	h1 := newNode(t)
 	ps1 := NewPeeringService(h1)
 
-	h2 := newNode(ctx, t)
-	h3 := newNode(ctx, t)
-	h4 := newNode(ctx, t)
+	h2 := newNode(t)
+	h3 := newNode(t)
+	h4 := newNode(t)
 
 	// peer 1 -> 2
 	ps1.AddPeer(peer.AddrInfo{ID: h2.ID(), Addrs: h2.Addrs()})