feat: go-libp2p v0.21 (rcmgr auto scaling) by marten-seemann · Pull Request #9074 · ipfs/kubo

marten-seemann · 2022-07-02T13:04:25Z

Updates to go-libp2p v0.21.
Updates the rcmgr.
Enables better metrics around the rcmgr.

Before merge:

Chore: update to go-libp2p 0.21 libp2p/go-libp2p-kad-dht#784
make a new release for kad-dht

MarcoPolo · 2022-07-03T00:01:23Z

core/node/libp2p/rcmgr.go

-		return result, err
-
+		return result, mgr.ViewTransient(func(s network.ResourceScope) error { return getLimit(s) })
 	case strings.HasPrefix(scope, config.ResourceMgrServiceScopePrefix):


Do we have to worry about spans here?

core/node/libp2p/rcmgr.go

core/node/libp2p/rcmgr_defaults.go

MarcoPolo · 2022-07-03T00:24:05Z

core/node/libp2p/rcmgr_defaults.go

-	defaultLimits.SystemBaseLimit.Conns = defaultLimits.SystemBaseLimit.ConnsOutbound + defaultLimits.SystemBaseLimit.ConnsInbound
-
-	return defaultLimits
+	// if cfg.ConnMgr.Type == "basic" {


This was weird. I would expect the resource manager be the ultimate arbiter, not connmgr.

We need something analogous though. The connection manager limit needs to be in sync with resource manager limit.

If these two are out of sync which should be the source of truth?

Probably the rcmgr, just because it's configuration is so much more complex. If I remember correctly, there were plans to deprecate the connmgr watermark configuration options in go-ipfs.

marten-seemann · 2022-07-03T10:46:10Z

core/node/libp2p/rcmgr_defaults.go

-	libp2p.SetDefaultServiceLimits(testLimiter)
+	l := rcmgr.DefaultLimits
+	libp2p.SetDefaultServiceLimits(&l)
+	limits := l.AutoScale()


Note: AutoScale dedicates 1/8 of the system memory to libp2p (which effects how many conns / streams / memory etc.) will be allowed. This might be too little for a standalone go-ipfs node, and too much for IPFS Desktop / Brave.

@lidel Do you have any "profile" setting that we can use here?

We do have a server profile which is used for adjusting specific config values as one-time change against Kubo config.

I think it is ok to go with AutoScale and revisit profile-specific defaults when in separate PR, as part of work towards enabling ResourceMgr by default (tracked in #8761)

MarcoPolo · 2022-07-04T16:01:09Z

Running this here: https://ipfs.marcopolo.io/d/MgmGIjjnk/resource-manager?orgId=1

MarcoPolo · 2022-07-06T19:12:33Z

There are two failures related to the new base256 emoji multibase: https://app.circleci.com/pipelines/github/ipfs/go-ipfs/7123/workflows/c0a0abf6-7f82-4965-8004-1413388e36fa/jobs/77371?invite=true#step-111-13053

I'm going to assign these bugs to @Jorropo since he added that multibase and the fix isn't super trivial since it involves updating some assumptions we've made in the sharness tests.

MarcoPolo · 2022-07-25T17:52:30Z

rebased on v0.14.0 After tests pass I'll ask bifrost to deploy this on bank4 for smoke testing

MarcoPolo · 2022-08-11T20:23:51Z

We should merge libp2p/go-libp2p-kad-dht#784 and make a new release first

MarcoPolo · 2022-08-11T20:24:44Z

I randomly picked @aschmahmann for a reviewer here, but feel free to reassign.

MarcoPolo · 2022-08-15T21:14:39Z

@lidel This is ready for a review, thanks!

https://github.com/ipfs/kubo/pull/9074/files?diff=split&w=1#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R97

lidel

Thank you @MarcoPolo, I believe this is ok to be included in 0.15-rc1

ResourceMgr.Enabled defaults to false, which makes rcmgr-related changes in this PR low risk.
Work towards enabling it by default we will continue #8761

fysa small changes I've made:

docs for Swarm.ResourceMgr.Allowlist
regression test so we can detect when rcmgr metrics disappear
backed out goleveldb bump (rationale below)

lidel · 2022-08-16T22:20:35Z

go.mod

-	github.com/syndtr/goleveldb v1.0.0
+	github.com/prometheus/common v0.35.0 // indirect
+	github.com/stretchr/testify v1.8.0
+	github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7


⚠️
I am unsure why this was bumped:

go-libp2p itself uses v.1.0.0?

this impacts plugin/plugins/levelds, which still uses v1.0.0 – safer to use the same version

Ideally, we would remove this dependency, and use aliases similar to this one

Since we have no time to deal with this,
I've reverted to v1.0.0 in a42848a just to play this safe.

Thanks for the flag, I’m also unsure why this was bumped

marten-seemann requested a review from MarcoPolo July 2, 2022 13:04

MarcoPolo reviewed Jul 3, 2022

View reviewed changes

marten-seemann force-pushed the rcgmr-auto-scale branch from 31c3eaa to 462224d Compare July 3, 2022 10:43

marten-seemann commented Jul 3, 2022

View reviewed changes

MarcoPolo force-pushed the rcgmr-auto-scale branch from c9887be to 2484abb Compare July 25, 2022 17:47

MarcoPolo mentioned this pull request Jul 27, 2022

go-libp2p v0.21.0 libp2p/go-libp2p#1514

Closed

41 tasks

MarcoPolo force-pushed the rcgmr-auto-scale branch from ebcda79 to c94c60a Compare August 11, 2022 17:15

MarcoPolo marked this pull request as ready for review August 11, 2022 20:23

MarcoPolo requested a review from aschmahmann August 11, 2022 20:24

BigLep mentioned this pull request Aug 12, 2022

Release 0.15 #9152

Closed

72 tasks

marten-seemann requested a review from lidel as a code owner August 12, 2022 16:46

MarcoPolo changed the title ~~WIP update go-libp2p to v0.21, use rcmgr auto scaling~~ Update go-libp2p to v0.21, use rcmgr auto scaling Aug 12, 2022

MarcoPolo self-assigned this Aug 12, 2022

marten-seemann and others added 12 commits August 15, 2022 14:12

WIP rcmgr auto limit scaling

c3589a1

Add obs to resource manager

679d3d2

Add allowlist config option

2b2efc6

Use RC version

68d9073

Reintroduce connmgr hi watermark logic

62a9829

Add sharness test for allowlist

66b974d

Fix lint

550663f

Fix err

1ff4e48

Update rcmgr dep

f4f4128

Update go-libp2p to latest master

dffdd35

mod tidy

0b3c7bc

Update pinned defaults

30fb10a

MarcoPolo added 10 commits August 15, 2022 14:12

Update checkImplicitDefaults

7245e8d

Use released version of go-libp2p v0.21;

0c58d6e

Use released version of rcmgr, use specific commit for kad-dht

629245a

Use newer rcmgr

3f42d96

Update go.mod/sum in examples/kubo-as-a-library

b0adea3

Use replace in kubo example as a library

4cd437d

Move register exporter to metrics file

27b046f

Update go-libp2p-resource-manager to v0.5.3

3ed83e4

Check err for view.Register

da54d5c

Go mod tidy

194eb2f

MarcoPolo force-pushed the rcgmr-auto-scale branch from dd0035f to babea24 Compare August 15, 2022 21:12

Update go-libp2p-kad-dht to v0.17.0

831135c

MarcoPolo force-pushed the rcgmr-auto-scale branch from babea24 to 831135c Compare August 15, 2022 21:14

Go mod tidy kubo-as-a-library

5c858d5

lidel self-assigned this Aug 16, 2022

lidel added 2 commits August 17, 2022 00:38

docs(config): Swarm.ResourceMgr.Allowlist

5ca74c7

chore: switch back to goleveldb v1.0.0

a42848a

https://github.com/ipfs/kubo/pull/9074/files?diff=split&w=1#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6R97

lidel changed the title ~~Update go-libp2p to v0.21, use rcmgr auto scaling~~ feat: go-libp2p v0.21 (rcmgr auto scaling) Aug 16, 2022

test(rcmgr): regression test prometherus metrics

77251b6

lidel approved these changes Aug 16, 2022

View reviewed changes

lidel enabled auto-merge August 16, 2022 23:34

lidel merged commit 00f2a64 into master Aug 16, 2022

lidel deleted the rcgmr-auto-scale branch August 16, 2022 23:42

lidel mentioned this pull request Sep 1, 2022

fix(test): flaky t0182-circuit-relay.sh #8967

Draft

Uh oh!

Conversation

marten-seemann commented Jul 2, 2022 • edited by MarcoPolo Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidel Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcoPolo commented Jul 4, 2022

Uh oh!

MarcoPolo commented Jul 6, 2022

Uh oh!

MarcoPolo commented Jul 25, 2022

Uh oh!

MarcoPolo commented Aug 11, 2022

Uh oh!

MarcoPolo commented Aug 11, 2022

Uh oh!

MarcoPolo commented Aug 15, 2022

Uh oh!

lidel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

marten-seemann commented Jul 2, 2022 •

edited by MarcoPolo

Loading

lidel Aug 16, 2022 •

edited

Loading