Skip to content

feat: first tests of Chew Large Datasets#58

Merged
daviddias merged 41 commits intomasterfrom
test-plan-1
Dec 9, 2019
Merged

feat: first tests of Chew Large Datasets#58
daviddias merged 41 commits intomasterfrom
test-plan-1

Conversation

@daviddias
Copy link
Copy Markdown
Contributor

@daviddias daviddias commented Oct 3, 2019

@daviddias when this issue was created:

I'm slowly making progress as I get myself used to how the whole testground machine is set up. Creating this Draft PR so that I can ask questions as I go.

My apologies in advance for any obvious/beginner questions. I know that soon enough it will all be clear, it is just a matter of pouring hours into it :)

This adds a new plan called Chew Large Datasets. Please read the README for further information about each test plan and how it works.

Tracking comment: #93 (comment)

@daviddias
Copy link
Copy Markdown
Contributor Author

So far, it has been useful already as I was able to adjust the language and structure of ipfs/interface-go-ipfs-core#33 with 247bf38

@daviddias daviddias mentioned this pull request Oct 3, 2019
@jimpick
Copy link
Copy Markdown
Contributor

jimpick commented Oct 7, 2019

Is this working? Should I try to run it?

@daviddias
Copy link
Copy Markdown
Contributor Author

@jimpick It is working but it does nothing special yet. It is mostly bunch of scaffolding/setup code. I'm working my way through the testground codebase and learning how to yield IPFS daemons using the IPTB setup (#60) and/or direct in-proc nodes ipfs/kubo#6695.

I'll change the state of the PR from draft to ready to review when ready.

@daviddias
Copy link
Copy Markdown
Contributor Author

daviddias commented Oct 10, 2019

Quick update: Still slaying dragons at -- ipfs/kubo#6695 --, I am super close though and I should be able to go back to writing the tests here today using that as a base.

For reference, also learned the way that Netflix spawns their IPFS daemons, super clean https://github.com/Netflix/p2plab/blob/master/peer/peer.go

@daviddias daviddias mentioned this pull request Oct 20, 2019
@daviddias daviddias force-pushed the test-plan-1 branch 2 times, most recently from 77da95f to 396bd9f Compare October 28, 2019 08:53
@daviddias
Copy link
Copy Markdown
Contributor Author

Getting the following panic when I run:

» go build . && TESTGROUND_BASEDIR=`pwd` ./testground run chew-large-datasets/ipfs-add-defaults --builder=docker:go --runner="local:exec"
resolved testground base dir from env variable: /Users/imp/code/go-projects/src/github.com/ipfs/testground
found cached docker image for: testground-chew-large-datasets:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
panic: reflect: New(nil)

goroutine 1 [running]:
reflect.New(0x0, 0x0, 0xc0004d5c80, 0x0, 0x0)
        /usr/local/go/src/reflect/value.go:2351 +0xc1
github.com/ipfs/testground/pkg/engine.coalesceConfigsIntoType(0x0, 0x0, 0xc0003164b0, 0x2, 0x2, 0xc0003164b0, 0x1, 0x2, 0x0)
        /Users/imp/code/go-projects/src/github.com/ipfs/testground/pkg/engine/engine.go:244 +0x360
github.com/ipfs/testground/pkg/engine.(*Engine).DoRun(0xc0000b3b00, 0x7ffeefbff8c9, 0x13, 0x7ffeefbff8dd, 0x11, 0x7ffeefbff90c, 0xa, 0xc0000d0b90, 0x0, 0x0, ...)
        /Users/imp/code/go-projects/src/github.com/ipfs/testground/pkg/engine/engine.go:208 +0x295
github.com/ipfs/testground/cmd.runCommand(0xc000196160, 0x0, 0xc0004fe380)
        /Users/imp/code/go-projects/src/github.com/ipfs/testground/cmd/run.go:128 +0x64b
github.com/urfave/cli.HandleAction(0x191ff20, 0x1adde90, 0xc000196160, 0xc000196160, 0x0)
        /Users/imp/code/go-projects/pkg/mod/github.com/urfave/cli@v1.22.0/app.go:523 +0xbe
github.com/urfave/cli.Command.Run(0x1aa32d8, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1ad1ebf, 0x62, 0x0, ...)
        /Users/imp/code/go-projects/pkg/mod/github.com/urfave/cli@v1.22.0/command.go:174 +0x51c
github.com/urfave/cli.(*App).Run(0xc00018a000, 0xc0000d0000, 0x5, 0x5, 0x0, 0x0)
        /Users/imp/code/go-projects/pkg/mod/github.com/urfave/cli@v1.22.0/app.go:276 +0x718
main.main()
        /Users/imp/code/go-projects/src/github.com/ipfs/testground/main.go:37 +0x244

Which comes from parsing the TOML manifest file -- https://github.com/ipfs/testground/blob/master/pkg/engine/engine.go#L228-L247

Scratching my head trying to figure out how is the manifest wrong. Tried using TOML validators including https://github.com/BurntSushi/toml and it parsed the manifest just fine.

Any tips?

@Stebalien
Copy link
Copy Markdown
Contributor

@daviddias The LocalExecutableRunner runner ("local:exec") doesn't specify a config type.

https://github.com/ipfs/testground/blob/a428f18df934d0973417648f427ff4e5f14fa5ba/pkg/runner/local_exec.go#L97

Either coalesceConfigsIntoType should handle typ == nil (and likely simply return early) or we should actually return valid types from these methods.

https://github.com/ipfs/testground/blob/a428f18df934d0973417648f427ff4e5f14fa5ba/pkg/engine/engine.go#L228-L247

@daviddias
Copy link
Copy Markdown
Contributor Author

aaaah! Epic facepalm! you are right. There is no such runner as local:exec yet. I completely missed that it was my param that was wrong and was looking for the bug in the wrong place. Thank you @Stebalien !!

@daviddias daviddias self-assigned this Oct 30, 2019
@daviddias daviddias force-pushed the test-plan-1 branch 2 times, most recently from 1c79f1d to ffe666c Compare November 9, 2019 22:00
addOptions := func(settings *coreopts.UnixfsAddSettings) error {
settings.Layout = coreopts.TrickleLayout
return nil
}
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @hacdias! I wasn't figuring this goism (if it is a goism) out

Copy link
Copy Markdown
Contributor Author

@daviddias daviddias Nov 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading this bit of code I see: "A func that takes the setting as argument and sets the option and then returns nil."

Can you explain to me if this is in fact idiomatic go and if it is, why sometimes use functional arguments such as https://github.com/ipfs/testground/blob/master/plans/dht/test/create-dht-node.go#L22-L25 and other times do like this?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@daviddias I've used Go in the past but I've never actually seen something like this. As far as I understand, what we have here is a function that takes a setting as argument and sets the option and returns an error if it occurred.

Whoever wrong the code in first place is certainly more capable of explaining why they choose to do it this way than I am. And I'm also curious

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hacdias thank you! @Stebalien could you shed some of wisdom here?

Copy link
Copy Markdown
Contributor

@aschmahmann aschmahmann Dec 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hacdias this is a pattern we frequently use in our code base https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis. It essentially hacks around Go not supporting optional parameters, by leveraging its support of variadic arguments.

The idea is that you call some function (e.g. ipfs.Unixfs().Add) with the parameters it requires (e.g. ctx), and optionally pass in any additional parameters via functions (e.g. ipfs.Unixfs().Add(ctx, option1, option2, ....). The error that you noticed is part of the function signature and as you can see no error is actually returned (we return nil). However, it's possible that sometimes we might want to return an error. For example, if we want to only set settings.A or settings.B, then if B is set after A we can throw an error. Another example is if inside the option we do some IO operation that might fail, then we can still handle the IO error. Check out that link for more info.

Copy link
Copy Markdown
Contributor Author

@daviddias daviddias Dec 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aschmahmann I get that blog post and I believed I understood it well when I wrote https://github.com/ipfs/testground/blob/master/plans/dht/test/create-dht-node.go#L22-L25. (Let's call this "Way A")

However, the way to get the trickle-dag option to Unixfs is way different in nature (Let's call it "Way B"), is is literally instead of passing the result of a func call (just like https://github.com/ipfs/testground/blob/master/plans/dht/test/create-dht-node.go#L22-L25 and what dave cheney describes) it takes in a function that received the config object to then change it.

Are both ways of doing this part of the same solution? If yes, why use Way B over Way A?

Copy link
Copy Markdown
Contributor

@aschmahmann aschmahmann Dec 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@daviddias I'm not sure if I fully understood your point about the options being "different in nature", but yes they are both part of the same solution.

For "Way A" you used dhtopts.BucketSize(bucketSize). If you look at dhtopts.BucketSize https://github.com/libp2p/go-libp2p-kad-dht/blob/4b7d6fb886de322dd54d7c597e8ab163e087ad74/opts/options.go#L151 you'll see that it's a function that returns a function with the same signature as "Way B". In short this means "Way A" is "Way B" wrapped in another function for convenience and readability.

To make this even more explicit the code here should be replaced with coreopts.Unixfs.Layout(coreopts.TrickleLayout). We actually already have a "Way A" wrapper for the function created here 😄.

I'm not sure if we have an explicitly defined pattern for where/how we put these wrapper "Way A" functions. However, they're generally defined and searchable via code-completion or full-text searching the repo that the options are defined in.

@hacdias
Copy link
Copy Markdown
Member

hacdias commented Dec 3, 2019

@daviddias here, when you wrote:

  • Directory Depth - An Array containing objects that describe how deep/nested a directory goes and the size of files that can be found throughout (default to [{depth: 10, size: 1MB}, {depth: 100, size: 1MB}]
  • How many files should there be per depth level?
  • Each file is of size, right?

@daviddias
Copy link
Copy Markdown
Contributor Author

How many files should there be per depth level?

I was just considering having the files at the directory leaves

Each file is of size, right?

Yes

@hacdias
Copy link
Copy Markdown
Member

hacdias commented Dec 3, 2019

@daviddias agreed, having files scattered around every directory may not help with any debugging because we're just trying to make sure it can add a directory correctly!

@hacdias
Copy link
Copy Markdown
Member

hacdias commented Dec 4, 2019

Is anyone aware how to add files to MFS using CoreAPI? Couldn't find any of the MFS commands. What about the CG too?

/cc @daviddias @Stebalien

License: MIT
Signed-off-by: Henrique Dias <hacdias@gmail.com>
License: MIT
Signed-off-by: Henrique Dias <hacdias@gmail.com>
License: MIT
Signed-off-by: Henrique Dias <hacdias@gmail.com>
License: MIT
Signed-off-by: Henrique Dias <hacdias@gmail.com>
License: MIT
Signed-off-by: Henrique Dias <hacdias@gmail.com>
License: MIT
Signed-off-by: Henrique Dias <hacdias@gmail.com>
@daviddias daviddias marked this pull request as ready for review December 8, 2019 14:20
@daviddias
Copy link
Copy Markdown
Contributor Author

@hacdias move to Review/QA when you want our review
image

@hacdias
Copy link
Copy Markdown
Member

hacdias commented Dec 8, 2019

Thanks for the heads up @daviddias and yes, it's ready for review. Please note only the first two tests are fully implemented. All the others are missing implementation and tracking is being done on ipfs/interface-go-ipfs-core#93!

Copy link
Copy Markdown
Contributor Author

@daviddias daviddias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nits but overall LGTM. Once those nits are handled, feel free to click the "rebase and merge"

hacdias and others added 5 commits December 8, 2019 19:34
Co-Authored-By: David Dias <daviddias.p@gmail.com>
Co-Authored-By: David Dias <daviddias.p@gmail.com>
Co-Authored-By: David Dias <daviddias.p@gmail.com>
Co-Authored-By: David Dias <daviddias.p@gmail.com>
Co-Authored-By: David Dias <daviddias.p@gmail.com>
@hacdias
Copy link
Copy Markdown
Member

hacdias commented Dec 8, 2019

Are you sure about rebasing? This PR has 41 commits and its messages are not the best. I think it would be better to squash this ones

@daviddias daviddias merged commit 9c26fb9 into master Dec 9, 2019
@daviddias daviddias deleted the test-plan-1 branch December 9, 2019 00:36
@daviddias
Copy link
Copy Markdown
Contributor Author

squashed :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants