testground: cross-version interoperability by laurentsenta · Pull Request #23 · libp2p/test-plans

laurentsenta · 2022-04-25T10:27:03Z

Add a ping test, and the composition file to build and run it with different versions of go-libp2p.

General idea:

We agree on a ping test, and implement it using different libp2p implementations,
Testground runs multiple instances of the same test, with different versions of the library.

Content

go test:

We add a single ping-interop/go test. This test can use multiple versions of libp2p with a single codebase:

It defines a go.vxxx.mod file (and it's sum) for each version,
It defines a compat module that provides a compatibility layer when there are API changes (shims).

Testground composition

We add a testground composition file go-cross-versions.toml
that runs v0.11, v.0.17, v0.19, v0.20, the current master, and an optional branch (if GitReference is in the environment) together.

Locally you can run this with:

GitReference=optionnal-custom-libp2p-branch testground run composition \
    -f ./_compositions/go-cross-versions.toml \
    --collect --wait

Then the test output is shown on the testground dashboard (localhost:8042 for a default daemon that runs locally).
Beware of testground/testground#1329 when you check results.

Note it'll use a go mod replace to test the latest master and latest version of an optional branch.

CI

We define a reusable workflow we can from another repository, it's called with:

jobs:
  run-libp2p-test-plans:
    uses: "singulargarden/test-plans/.github/workflows/test-compatibility.yml@master"
    with:
      composition_file: "ping-interop/_compositions/go-cross-versions.toml"
      test_case: "ping"
      custom_git_reference: "current-pr" # optional
      custom_git_target: "my-fork/go-libp2p" # optional
      testground_endpoint: "https://ci.testground.ipfs.team/" # optional

This action will build & run the test.

Todo:

ping/main.go

galargh · 2022-04-28T17:11:10Z

Unfortunately, I won't be able to meaningfully review this before my short break (until May 4). I'm OK with you moving forward with this though and I can review it post-merge.

marten-seemann

Not an in-depth review yet, but a few high-level thoughts:

iiuc, the current setup is limited to testing interop between exactly 2 version. It would be nice to have a setup where we could spin up nodes using multiple different versions, especially as we're going to cut more (and smaller) releases in the future. Is that possible?
code reuse: it would be nice if we could write the code once, and then run it using multiple version. In my mind, this raises two questions:
1. What if a breaking API change happened between two versions?
2. How to deal with go.mod

I don't have a good answer for 2., but for 1., it might be nice to use a build flag. The implementation could then work around the breaking interface change.

BigLep · 2022-05-27T16:38:57Z

2022-05-27 conversation: there was a verbal call on how to move forward. I'm going to move this to draft. Please publish @laurentsenta again when it's ready for review.

laurentsenta · 2022-06-24T15:34:14Z

2 updates:

I moved the proxy to proxy.golang.org, and increased every timeout, and now the test is passing consistently on regular GitHub runners! I still can't believe that was that easy, I've seen a lot of random freezes and fails when developing this on local, but it looks like I was wrong @mxinden, regular runners are enough for this test! 🙇
I updated the composition to test forks too, so we are able to test pull request from forks if we want too,
- there is a bug with branch names, but if we use commits sha we'll be fine

A few runs as example:
(check the "Run composition file" section for parameters & logs)

test without any parameter https://github.com/singulargarden/pl-github/runs/7043100620?check_suite_focus=true (test master
test with github.com/fulviodenza/go-libp2p:master https://github.com/singulargarden/pl-github/runs/7043021338?check_suite_focus=true
test with marco/canonical-logging branch https://github.com/singulargarden/pl-github/runs/7043448820?check_suite_focus=true

mxinden · 2022-06-27T01:46:51Z

@laurentsenta once this is merged, I can provide the ping Rust implementation based on https://github.com/mxinden/test-plans/tree/compatibility-libp2p. Would appreciate some help hooking it up to the composition test though.

galargh · 2022-06-27T07:15:10Z

I moved the proxy to proxy.golang.org, and increased every timeout, and now the test is passing consistently on regular GitHub runners! I still can't believe that was that easy, I've seen a lot of random freezes and fails when developing this on local, but it looks like I was wrong @mxinden, regular runners are enough for this test! 🙇

Great that we can move back to the standard runners now but let's track the setup times and have it on a radar because extending timeouts is not a great long-term solution.

Awesome work Laurent ❤️ Let's get it out 🚀

marten-seemann

LGTM, modulo a few comments.

Slight improvement would be to use the latest patch release of every minor release (e.g. v0.19.4 instead of v0.19.0).

marten-seemann · 2022-06-30T08:26:12Z

ping/go/main.go

 	"github.com/libp2p/go-libp2p-core/peer"

-	"github.com/libp2p/go-libp2p-noise"
+	noise "github.com/libp2p/go-libp2p-noise"


This repo is deprecated as of v0.20.0. Same for -tls.

To do things the correct way, we should define a NoiseProtocolID in compat.

👍

It looks like we also need to pass the Protocol constructor (security(noise.ID, noise.New)), I've implemented a getByName, then moved it into the compat package (less confident about this one).

9075466

0f506ec

marten-seemann · 2022-06-30T08:27:14Z

ping/go/main.go

 		security = libp2p.Security(noise.ID, noise.New)
 	case "tls":
 		security = libp2p.Security(tls.ID, tls.New)
+		// TODO: check w/Marten this is fine: We fall into the lowest common denominator for parameters (here, no secio anymore even for legacy versions).


Yes, no need to test secio any more.

marten-seemann · 2022-06-30T08:35:06Z

ping/go/manifest.toml


+[builders."docker:go"]
+enabled                = true
+build_base_image       = "golang:1.17-buster"


This might be a dumb question, but what does it mean to specify golang:1.17 here?

Not dumb at all, it becomes the new default for this test plan, (testground uses 1.16 by default).
We overwrite it in the composition files when we run "real" tests.

I assume if you're trying to run a quick test, you're probably working on the latest version,
so you have one less testground parameter to remember:

testground run single --plan=libp2p/ping/go --testcase=ping --builder=docker:go --runner=local:docker --instances=2 --build-cfg modfile=my.new.go.mod

That's why I was wondering, the current version is Go 1.18. Not a blocker for this PR though.

* plan/ping: fix test & use go 1.17 * plan/ping: fix dedup path * ping-interop: prepare base module and build script * ping-interop/go-v0.11: introduce legacy ping test (fix original) * ci: add interop reusable workflow * ci: add testground endpoint config * ping-interop: rewrite with new testground build config feature * ping-interop: add v0.20 version * ping-interop: fix build parameters * ci: add ping-interop-go runner * ci: use self hosted runner * ping-interop: name with the current git branch * ping-interop: use external proxy & drop build cache * ping-interop: support dependencies changes in go master * ping-interop: using longer timeout with regulars runners * ping-interop: move to ping * README: add ping/go update intructions * ci: rename compatibility test * ci: add support for forks * ping: add doc * REMOVEME: use a test setup * ping: update libp2p versions * ping: add security compat * ping: smaller compat surface

laurentsenta marked this pull request as draft April 25, 2022 10:27

laurentsenta mentioned this pull request Apr 25, 2022

EPIC1.1: As a go-libp2p maintainer I have all the tooling I need to create new interoperability tests on my machine and make sure they are used in the CI to test every release laurentsenta/pl-github#1

Open

7 tasks

laurentsenta force-pushed the feat/ping-interop branch 3 times, most recently from 27447ba to 30f20cf Compare April 25, 2022 13:37

laurentsenta commented Apr 25, 2022

View reviewed changes

ping/main.go Outdated Show resolved Hide resolved

laurentsenta mentioned this pull request Apr 26, 2022

the ping test passes consistently on local machine laurentsenta/pl-github#13

Open

1 task

laurentsenta force-pushed the feat/ping-interop branch 6 times, most recently from 352e6df to 82b1dee Compare April 28, 2022 13:29

laurentsenta added 2 commits April 28, 2022 15:30

plan/ping: fix test & use go 1.17

ef0182d

plan/ping: fix dedup path

71b6a0a

laurentsenta force-pushed the feat/ping-interop branch 2 times, most recently from ff2b51d to b58be4b Compare April 28, 2022 13:36

laurentsenta changed the title ~~WIP testground: cross-version interoperability~~ testground: cross-version interoperability Apr 28, 2022

laurentsenta marked this pull request as ready for review April 28, 2022 16:51

laurentsenta added 3 commits April 29, 2022 09:19

ping-interop: prepare base module and build script

a096bec

ping-interop/go-v0.11: introduce legacy ping test (fix original)

d42f451

ci: add interop reusable workflow

7e0c5f8

laurentsenta force-pushed the feat/ping-interop branch from b58be4b to a55cb01 Compare April 29, 2022 07:42

laurentsenta mentioned this pull request Apr 29, 2022

Run plan with two instances based on two different builds testground/testground#1305

Closed

marten-seemann reviewed May 3, 2022

View reviewed changes

laurentsenta mentioned this pull request May 17, 2022

Add additional parameters to groups in composition files testground/testground#1337

Closed

5 tasks

BigLep marked this pull request as draft May 27, 2022 16:39

laurentsenta force-pushed the feat/ping-interop branch from 5639ccc to 9fb4d5d Compare June 1, 2022 13:19

laurentsenta force-pushed the feat/ping-interop branch from e727fdc to 7259f78 Compare June 24, 2022 11:02

laurentsenta added 5 commits June 24, 2022 13:12

ping-interop: support dependencies changes in go master

38a9d70

ping-interop: using longer timeout with regulars runners

958a8e0

ping-interop: move to ping

d5a4116

README: add ping/go update intructions

523f95e

ci: rename compatibility test

f47dbfd

laurentsenta force-pushed the feat/ping-interop branch 2 times, most recently from 936c9ba to 27a7faa Compare June 24, 2022 13:20

ci: add support for forks

f4ff2c5

laurentsenta force-pushed the feat/ping-interop branch from 27a7faa to 97cf122 Compare June 24, 2022 13:51

laurentsenta added 2 commits June 24, 2022 16:31

ping: add doc

ac526ac

REMOVEME: use a test setup

785c9fa

laurentsenta force-pushed the feat/ping-interop branch from 97cf122 to 785c9fa Compare June 24, 2022 14:34

mxinden approved these changes Jun 27, 2022

View reviewed changes

marten-seemann approved these changes Jun 30, 2022

View reviewed changes

laurentsenta added 3 commits June 30, 2022 11:32

ping: update libp2p versions

ea74750

ping: add security compat

6968b42

ping: smaller compat surface

7986861

laurentsenta force-pushed the feat/ping-interop branch from 0f506ec to 7986861 Compare June 30, 2022 13:36

marten-seemann approved these changes Jul 1, 2022

View reviewed changes

laurentsenta merged commit fb5c863 into libp2p:master Jul 1, 2022

This was referenced Jul 1, 2022

ci: remove test commit #24

Merged

fix: tidy compat and update versions #25

Merged

laurentsenta mentioned this pull request Aug 30, 2022

EPIC: interop testing for go and rust libp2p - Ping Test #35

Closed

6 tasks

laurentsenta mentioned this pull request Dec 1, 2022

Milestone 2023.1: Bootstrap libp2p's interoperability testing story testground/testground#1533

Open

laurentsenta mentioned this pull request Dec 9, 2022

feat: libp2p implementation interoperability dashboard on various dimensions. Uses testplans-dsl. #85

Closed

Conversation

laurentsenta commented Apr 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Content

go test:

Testground composition

CI

Todo:

Uh oh!

Uh oh!

galargh commented Apr 28, 2022

Uh oh!

marten-seemann left a comment

Choose a reason for hiding this comment

Uh oh!

BigLep commented May 27, 2022

Uh oh!

laurentsenta commented Jun 24, 2022

Uh oh!

mxinden commented Jun 27, 2022

Uh oh!

galargh commented Jun 27, 2022

Uh oh!

marten-seemann left a comment

Choose a reason for hiding this comment

Uh oh!

marten-seemann Jun 30, 2022

Choose a reason for hiding this comment

Uh oh!

laurentsenta Jun 30, 2022

Choose a reason for hiding this comment

Uh oh!

marten-seemann Jun 30, 2022

Choose a reason for hiding this comment

Uh oh!

marten-seemann Jun 30, 2022

Choose a reason for hiding this comment

Uh oh!

laurentsenta Jun 30, 2022

Choose a reason for hiding this comment

Uh oh!

marten-seemann Jul 1, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

laurentsenta commented Apr 25, 2022 •

edited

Loading