Skip to content
This repository was archived by the owner on Aug 2, 2021. It is now read-only.

PoC: Network simulation framework#1555

Merged
skylenet merged 33 commits intoethersphere:masterfrom
skylenet:network-sim-v2
Jul 24, 2019
Merged

PoC: Network simulation framework#1555
skylenet merged 33 commits intoethersphere:masterfrom
skylenet:network-sim-v2

Conversation

@skylenet
Copy link
Copy Markdown
Contributor

@skylenet skylenet commented Jul 11, 2019 •
edited
Loading

This PR brings in a new simulation framework which has the following available adapters:

  • Exec
  • Docker
  • Kubernetes

Additional features:

  • Simulation snapshot that include information about the adapter, nodes and peer connections.

Using the different adapters:

Example can be found under: https://github.com/skylenet/swarm/blob/network-sim-v2/simulation/examples/cluster/cluster_test.go

Importing a snapshot:

Example: https://github.com/skylenet/swarm/tree/network-sim-v2/simulation/examples/snapshot

Code review

Since a lot of vendored libs have been included due to the docker/kubernetes clients, I would suggest to have a look directly @ https://github.com/skylenet/swarm/tree/network-sim-v2/simulation

skylenet added 16 commits July 2, 2019 13:34
@skylenet
simv2: wip
087b238
@skylenet
Merge branch 'master' of github.com:ethersphere/swarm into network-si…
cac6243 
…m-v2
@skylenet
simulation: exec adapter start/stop
04dd442
@skylenet
simulation: add node status to exec adapter
0dbc0ef
@skylenet
simulation: initial simulation code
93cd38b
@skylenet
simulation: exec adapter, configure path to executable
2223822
@skylenet
simulation: initial docker adapter
f17339f
@skylenet
simulation: wip kubernetes adapter
391596d
@skylenet
simulation: kubernetes adapter proxy
36fd9f5
@skylenet
simulation: implement GetAll/StartAll/StopAll
68614a4
@skylenet
simulation: kuberentes adapter - set env vars and resource limits
870b2ae
@skylenet
simulation: discovery test
ea80683
@skylenet
simulation: remove port definitions within docker adapter
27afe4d
@skylenet
simulation: simplify wait for healthy loop
0e1ea4c
@skylenet
simulation: get nat ip addr from interface
c2cfe8f
@skylenet
simulation: pull docker images automatically
fd8a6a3
@zelig zelig added the network label Jul 16, 2019
@skylenet skylenet changed the title Network sim v2 PoC: Network simulation framework Jul 22, 2019
@skylenet skylenet requested review from acud, janos and nonsense July 23, 2019 13:26
@skylenet skylenet marked this pull request as ready for review July 23, 2019 13:26
@janos
Copy link
Copy Markdown
Member

janos commented Jul 23, 2019

Total of 1570 files changed is quite heavy for the review on github webpage diff. I assume that most of them are vendored packages. Would it be better to vendor in a different PR? But then travis will fail. Any suggestions how to review?

@skylenet
Copy link
Copy Markdown
Contributor Author

skylenet commented Jul 23, 2019

Total of 1570 files changed is quite heavy for the review on github webpage diff. I assume that most of them are vendored packages. Would it be better to vendor in a different PR? But then travis will fail. Any suggestions how to review?

@janos check the PR description :P . Have a look directly at https://github.com/skylenet/swarm/tree/network-sim-v2/simulation

@janos
Copy link
Copy Markdown
Member

janos commented Jul 23, 2019

Ah, thanks @skylenet, I missed it.

@skylenet
Copy link
Copy Markdown
Contributor Author

skylenet commented Jul 24, 2019

Relates to ethersphere/user-stories#33

@nonsense
Copy link
Copy Markdown
Contributor

nonsense commented Jul 24, 2019

Overall looks good to me. Actually writing tests with this framework will be the ultimate test on how useful such a framework is, but I think so far we are going in the right direction.

ok  	github.com/ethersphere/swarm/simulation	104.191s	coverage: 14.6% of statements
ok  	github.com/ethersphere/swarm/simulation/examples/cluster	17.737s	coverage: 0.0% of statements
ok  	github.com/ethersphere/swarm/simulation/examples/snapshot	8.694s	coverage: 0.0% of statements

It'd be nice if we can speed up a bit the tests, or reduce the constants, 104s. seems a lot to me.

@skylenet
Copy link
Copy Markdown
Contributor Author

skylenet commented Jul 24, 2019

Overall looks good to me. Actually writing tests with this framework will be the ultimate test on how useful such a framework is, but I think so far we are going in the right direction.

ok  	github.com/ethersphere/swarm/simulation	104.191s	coverage: 14.6% of statements
ok  	github.com/ethersphere/swarm/simulation/examples/cluster	17.737s	coverage: 0.0% of statements
ok  	github.com/ethersphere/swarm/simulation/examples/snapshot	8.694s	coverage: 0.0% of statements

It'd be nice if we can speed up a bit the tests, or reduce the constants, 104s. seems a lot to me.

The biggest time (~50s) is spent within TestDockerAdapterBuild that actually builds our docker image. We could probably make this test optional in the future.

@nonsense
Copy link
Copy Markdown
Contributor

nonsense commented Jul 24, 2019

Right, yeah, making it optional at some point would be a good idea I think.

@skylenet skylenet merged commit 388d8cc into ethersphere:master Jul 24, 2019
@skylenet skylenet added this to the 0.4.3 milestone Jul 25, 2019
@zelig
Copy link
Copy Markdown
Member

zelig commented Jul 26, 2019

@skylenet it was naughty to merge a PR with one review. Now we protected merges needing 2 reviews.
I mean especially such a PR, it is huge and it came out of draft status like 2 days ago.

chadsr added a commit to chadsr/swarm that referenced this pull request Sep 23, 2019
@chadsr
Merge branch 'master' of github.com:ethersphere/swarm
f9ca5af 
* 'master' of github.com:ethersphere/swarm: (54 commits)
  api, chunk, cmd, shed, storage: add support for pinning content (ethersphere#1509)
  docs/swarm-guide: cleanup (ethersphere#1620)
  travis: split jobs into different stages (ethersphere#1615)
  simulation: retry if we hit a collision on tcp/udp ports (ethersphere#1616)
  api, chunk: rename Tag.New to Tag.Create (ethersphere#1614)
  pss: instrumentation and refactor (ethersphere#1580)
  api, cmd, network: add --disable-auto-connect flag (ethersphere#1576)
  changelog: fix typo (ethersphere#1605)
  version: update to v0.4.4 unstable (ethersphere#1603)
  swarm: release v0.4.3 (ethersphere#1602)
  network/retrieve: add bzz-retrieve protocol (ethersphere#1589)
  PoC: Network simulation framework (ethersphere#1555)
  network: structured output for kademlia table (ethersphere#1586)
  client: add bzz client, update smoke tests (ethersphere#1582)
  swarm-smoke: fix check max prox hosts for pull/push sync modes (ethersphere#1578)
  cmd/swarm: allow using a network interface by name for nat purposes (ethersphere#1557)
  pss: disable TestForwardBasic (ethersphere#1544)
  api, network: count chunk deliveries per peer (ethersphere#1534)
  network/newstream: new stream! protocol base implementation (ethersphere#1500)
  swarm: fix bzz_info.port when using dynamic port allocation (ethersphere#1537)
  ...
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@acud acud Awaiting requested review from acud

@janos janos Awaiting requested review from janos

1 more reviewer

@nonsense nonsense nonsense approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@skylenet skylenet

Labels

network ready for review

Projects

None yet

Milestone

0.4.3

Development

Successfully merging this pull request may close these issues.

5 participants

@skylenet @janos @nonsense @zelig @acud