feat: warn users who are falling behind reprovides by Jorropo · Pull Request #9886 · ipfs/kubo

Jorropo · 2023-05-19T18:25:46Z

Fixes: #9704
Fixes: #9702
Fixes: #9703
Fixes: #9419

BigLep · 2023-05-23T16:46:15Z

2023-05-23 conversation: @Jorropo will pull out the go-ipfs-http-client into a separate PR and link it to #9124

BigLep

Thanks for putting this together @Jorropo . I just focused on the user documentation side of this.

I see this issue as also intending to close #9703 (I'll update the PR description), but that means addressing #9419 as well. Can you include the relevant doc suggestions from there as well?

docs/changelogs/v0.21.md

BigLep · 2023-05-30T23:30:47Z

docs/config.md

+Utilizes an alternative DHT client that searches for and maintains more information
+about the network in exchange for being more performant.


Suggested change

Utilizes an alternative DHT client that searches for and maintains more information

about the network in exchange for being more performant.

Utilizes an alternative DHT client that searches for and maintains more information

about the network in exchange for improved provide latency performance (upwards of 100x).

(I made up the number above - I can't remember the actual numbers. Maybe they are in the changelog for when we first released this?)

Can we maybe expand this by a sentence or two? Is there a talk or doc we can link to?

Maybe something along the lines of...

This accelerated DHT client builds up a bamp of various nodes that can satisfy the keyspace. This helps both with reads and writes: * Reads - Traversed DHT results have been cached and know the closest nodes to query. * Writes - In addition to having the cached traversed DHT results to know the direct n=20 nodes to store records with, records are batched so that do a single write with a large batch of records for a given provider.

(I know that's not a great writeup - but would like to have some more text that helps someone conceptually understand what is going on, or provide a link where someone can learn more so it's more self-service to reason about the system.)

I guess some of these details are covered below in the "when it is enabled" section. I mostly just want to make sure we help explain the concept of what's happening.

~~I'll rework that section in the docs, I don't think changelogs should go in that much details, just link to docs that are that detailed.~~

That is not the changelog.

Btw the number is roughly 6 millions. 6 millions times faster.
(based on the assumption of 30s per bucket rt puts, vs 5µs for an accelerated swept put)

At what percentile are we getting 30s per bucket round trip? That seems really high. I assume the p50 value is way lower, and we should compare against that.

I belive that for all 20 provides, because almost all the time one of them will timeout and the timeout is 30s ?
Anyway 5µs is so small even if puts took 250ms this would still be way faster.

core/node/provider.go

BigLep · 2023-05-30T23:46:24Z

core/node/provider.go

+				}
+
+				// How long per block that lasts us.
+				expectedProvideSpeed := reprovideInterval / time.Duration(count)


What unit of time are we dealing with here? Seconds, milliseconds? I think for clarity, it would be good to embed that into the variable names.

The computations are done in nanoseconds, time.Duration.String takes care of formating it in something relevent.

Go use a typed time approach, programmers do not have to care about the unit of time, they do operations on the time.Duration time (which internally use nanoseconds) and then various formating call let extract the precision wanted.

core/node/provider.go

BigLep · 2023-05-30T23:51:32Z

I added:

Fixes: #9814
Fixes: #9703

Jorropo · 2023-05-31T12:22:35Z

@BigLep you don't want to use

Fixes: #9814

Fixes precisely describe that the issue following will be closed when this is merged. Updates: #9814 or For: #9814 would be appropriate. 🙂

BigLep · 2023-05-31T17:52:01Z

@Jorropo : thanks. I screwed up my copy paste. I meant to do:

Fixes: #9419
Fixes: #9703

I didn't mean to mention #9814! Arg!

Anyways, fixed now!

hacdias

LGTM. Please rebase, check @BigLep's review. Sharness seems to have timed out though.

docs/config.md

BigLep

I left comments on config.md. I am happy to take another look if it helps, but I don't have any blockers here.

Thanks for handling!

BigLep · 2023-06-02T02:48:13Z

docs/changelogs/v0.21.md

+The [accelerated DHT client](docs/config.md#routingaccelerateddhtclient) is now
+the main recommended solution for users who are hosting lots of data.
+By trading some upfront DHT caching and increased memory usage,
+one gets provider throughput improvements up to 6 millions times bigger dataset.


Suggested change

one gets provider throughput improvements up to 6 millions times bigger dataset.

one gets provider throughput improvements of up to 6 million times on larger datasets!

BigLep · 2023-06-02T20:02:34Z

docs/config.md

+Utilizes an alternative DHT client that searches for and maintains more information
+about the network in exchange for being more performant.


At what percentile are we getting 30s per bucket round trip? That seems really high. I assume the p50 value is way lower, and we should compare against that.

BigLep · 2023-06-02T20:29:53Z

docs/config.md

+- DHT operations should complete much faster than with it disabled
+- A batching reprovider system will be enabled which takes advantage of some
+  properties of the experimental client to very efficiently put provider records into the network
+- The standard DHT client (and server if enabled) are run alongside the accelerated DHT client


I'm not following everything here @Jorropo .

When the AcceleratedDHTClient is enabled:

(Client) For both writing records and reading records, we use the full routing table right?

(Server) Storing records on behalf of others and responding to queries, we use the bucket routing table right?

BigLep · 2023-06-02T20:36:58Z

docs/config.md

+- A batching reprovider system will be enabled which takes advantage of some
+  properties of the experimental client to very efficiently put provider records into the network
+- The standard DHT client (and server if enabled) are run alongside the accelerated DHT client
+- The operations `ipfs stats dht` will default to showing information about the accelerated DHT client


Got it. Thinking about this more, I'm wondering if we even need this comment.

A user has these options:

ipfs stats dht wanserver ipfs stats dht lanserver ipfs stats dht wan ipfs stats dht lan

When the accelerated DHT client is enabled, that means its stats will be reported for ipfs stats dht wan right?

And ipfs stats dht is just shorthand for ipfs stats dht wan right?

I'm trying to understand the purpose of this comment. I don't think we should be capturing here what ipfs stats dht defaults to showing. I do think we should say that accelerated DHT client affects dht wan, but the other 3 are untouched.

core/node/provider.go

docs/config.md

We were quite inconsistent about this previously, some files used 1.19.1 some 1.19.x, this makes it more consistent.

hacdias

LGTM. Assuming everything passes, please merge.

Fixes: ipfs#9704 Fixes: ipfs#9702 Fixes: ipfs#9703 Fixes: ipfs#9419

Jorropo requested a review from a team as a code owner May 19, 2023 18:25

Jorropo force-pushed the accelerated-dht-client branch from 28cd95b to a10e9d7 Compare May 22, 2023 12:00

BigLep assigned Jorropo May 23, 2023

BigLep reviewed May 30, 2023

View reviewed changes

Jorropo force-pushed the accelerated-dht-client branch from a10e9d7 to bce9e58 Compare May 31, 2023 12:13

BigLep requested a review from hacdias June 1, 2023 01:16

hacdias reviewed Jun 1, 2023

View reviewed changes

docs/config.md Outdated Show resolved Hide resolved

lidel mentioned this pull request Jun 1, 2023

fix(gateway)!: no duplicate payload during subdomain redirects #9913

Merged

3 tasks

Jorropo force-pushed the accelerated-dht-client branch from bce9e58 to d0a7e03 Compare June 1, 2023 22:42

lidel mentioned this pull request Jun 2, 2023

fix(gateway): question marks in url.Path when redirecting #9894

Closed

2 tasks

Jorropo force-pushed the accelerated-dht-client branch from 79a0930 to b52d94f Compare June 2, 2023 01:27

Jorropo requested a review from BigLep June 2, 2023 01:27

Jorropo mentioned this pull request Jun 2, 2023

feat: warn users who are falling behind reprovides #9916

Closed

Jorropo force-pushed the accelerated-dht-client branch 3 times, most recently from 4dec819 to 78d5009 Compare June 2, 2023 02:37

BigLep approved these changes Jun 2, 2023

View reviewed changes

Jorropo force-pushed the accelerated-dht-client branch 7 times, most recently from bcbd21e to 93c57dc Compare June 8, 2023 07:46

ci: bump go version to 1.19.x across the whole board

0bdc57b

We were quite inconsistent about this previously, some files used 1.19.1 some 1.19.x, this makes it more consistent.

Jorropo force-pushed the accelerated-dht-client branch from 93c57dc to 81a277c Compare June 8, 2023 07:46

chore: migrate test/dependencies to boxo

cc63e25

Jorropo force-pushed the accelerated-dht-client branch from 81a277c to 7f263c2 Compare June 8, 2023 07:54

hacdias approved these changes Jun 8, 2023

View reviewed changes

feat: warn users who are falling behind reprovides

0bd3ba7

Fixes: ipfs#9704 Fixes: ipfs#9702 Fixes: ipfs#9703 Fixes: ipfs#9419

Jorropo force-pushed the accelerated-dht-client branch from 7f263c2 to 0bd3ba7 Compare June 8, 2023 08:05

Jorropo merged commit e7294cb into ipfs:master Jun 8, 2023

Jorropo deleted the accelerated-dht-client branch June 8, 2023 08:05

lidel mentioned this pull request Jul 4, 2023

feat(cmds): add version check command #8839

Closed

lidel mentioned this pull request Sep 15, 2023

Outdated Version Notice #6487

Closed

lidel mentioned this pull request Nov 24, 2023

Changed option is ignored #10231

Open

3 tasks

		Utilizes an alternative DHT client that searches for and maintains more information
		about the network in exchange for being more performant.

	one gets provider throughput improvements up to 6 millions times bigger dataset.
	one gets provider throughput improvements of up to 6 million times on larger datasets!

Uh oh!

Conversation

Jorropo commented May 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BigLep commented May 23, 2023

Uh oh!

BigLep left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jorropo Jun 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jorropo Jun 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BigLep commented May 30, 2023

Uh oh!

Jorropo commented May 31, 2023

Uh oh!

BigLep commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hacdias left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BigLep left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hacdias left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jorropo commented May 19, 2023 •

edited

Loading

Jorropo Jun 1, 2023 •

edited

Loading

Jorropo Jun 2, 2023 •

edited

Loading

BigLep commented May 31, 2023 •

edited

Loading