feat(network): upgrade p2p layer by onur-ozkan · Pull Request #1878 · GLEECBTC/komodo-defi-framework

onur-ozkan · 2023-06-22T08:45:57Z

This PR brings new p2p layer that uses the latest stable libp2p version and deprecates the old p2p layer.

This PR is blocker for the following future works:

Gradually reduce the patches on komodo p2p fork (start with activating https://github.com/KomodoPlatform/komodo-defi-framework/blob/092d562f476ece1bef3cf3e8e50694e23688dade/mm2src/mm2_p2p/src/behaviours/peer_store.rs or relay,keep-alive functionalities)
Seperate network layer from mm2 core/swap mechanism so mm2 can be used with any network implementation
Message compression(as a new behaviour/protocol so we don't face with breaking changes) in p2p layer
Implement rate limiter behaviour/protocol
feat(network): upgrade p2p layer #1878 (comment)
feat(network): upgrade p2p layer #1878 (comment)

Breaking changes:

mesh protocols are upgraded to latest versions (like gossipsub@1.1)
latest libp2p requires using >=23 android ndk for aarch64
https://github.com/KomodoPlatform/komodo-defi-framework/blob/092d562f476ece1bef3cf3e8e50694e23688dade/scripts/ci/android-ndk.sh#L5-L6
Old mesh configuration is no more valid for latest version
https://github.com/KomodoPlatform/komodo-defi-framework/blob/0c00797b88b8f214cdac74d415f187d275be2278/mm2src/mm2_p2p/src/behaviours/atomicdex.rs#L595
Migrated to modern straem multiplexer yamux
https://github.com/KomodoPlatform/komodo-defi-framework/blob/0c00797b88b8f214cdac74d415f187d275be2278/mm2src/mm2_p2p/src/behaviours/atomicdex.rs#L917-L920
For gossipsub, now we broadcast one topic at a time
https://github.com/KomodoPlatform/komodo-defi-framework/blob/0c00797b88b8f214cdac74d415f187d275be2278/mm2src/mm2_main/src/lp_network.rs#L276

Check Cargo.lock with display rich diff button for detailed dependency report.

Signed-off-by: ozkanonur <work@onurozkan.dev>

onur-ozkan · 2023-08-23T16:18:12Z

AFAIK you are planning a specific release for this work. Do you want me to bump mm2 to 1.1.0-beta? @shamardy

shamardy · 2023-08-24T08:32:58Z

Do you want me to bump mm2 to 1.1.0-beta?

Yes, please. thanks for remembering.

Following semantic versioning a release including this PR should be 2.0.0-beta. But I will use the second digit since we only use the last digit (PATCH version) for backward compatible functionalities too. After the 1.1.0-beta release we will start using the standard semantic versioning.

Signed-off-by: ozkanonur <work@onurozkan.dev>

…f-network-stack

laruh · 2023-09-06T04:56:33Z

@onur-ozkan excellent work! could you please add doc comments for pub items (at least for new rs files in mm2_p2p crate, if you have time), so we can start code preparation for missing_docs lint

onur-ozkan · 2023-09-06T05:07:09Z

@onur-ozkan excellent work! could you please add doc comments for pub items (at least for new rs files in mm2_p2p crate, if you have time), so we can start code preparation for missing_docs lint

PR is already under sec-review and about to be merged. Would be much nicer if we ignore those crates with #![allow(missing_docs)], which can be fixed later.

ca333

secure code reviewed (based on cargo.lock / ensured pinned V >= cargo deps V / re-reviewed prior utilized deps from this list) - no malicious bits detected

general deps:

async-trait @ 0.1.52 (https://github.com/dtolnay/async-trait/tree/0.1.52 / 20bd296e0d646e2b14626b7078e045254bed26ee) outdated

derive_more @ 0.99.11 (https://github.com/JelteF/derive_more/tree/v0.99.11 / 2c95900e13ed5401aaba22176cca56f8c4851236) outdated

futures @ 0.1.29 / 0.3.28 (https://github.com/rust-lang/futures-rs/tree/0.3.28 / 1685f8bdabb8e7f007610f3494425aeabf28117d) multiple versions utilized (including outdated)

futures-rustls @ 0.21.1 / 0.22.1 (42afb9bdf6bf0a689ff51a65d93925880c085792) multiple versions utilized (including outdated)

futures-ticker @ 0.0.3 (https://github.com/antifuchs/futures-ticker/tree/v0.0.3 / f9c76c893cb17f7ec061d961d669f61325504a96)

hex @ 0.3.2 / 0.4.3 (https://github.com/KokaKiwi/rust-hex/tree/v0.3.2 / 19fd37137686c30058bd9d11d21590e726ffdf31) multiple versions utilized (including outdated)

lazy_static @ 1.4.0 (https://github.com/rust-lang-nursery/lazy-static.rs/tree/1.4.0 / 421669662b35fcb455f2902daed2e20bbbba79b6)

log @ 0.4.19 (https://github.com/rust-lang/log/tree/0.4.19 / 84ddc30ecc54e464b791dd591a5becc02f4fd06c) outdated

rand @ 0.6.5 / 0.7.3 / 0.8.4 (https://github.com/rust-random/rand/tree/0.8.4 / 8792268dfe57e49bb4518190bf4fe66176759a44) multiple versions utilized (including outdated)

regex @ 1.8.4 (https://github.com/rust-lang/regex/tree/1.8.4 / 5a34a39b72d85730065d3ffe4ce3715f2731e49a) outdated

rmp-serde @ 0.14.4 (https://github.com/3Hren/msgpack-rust/tree/rmp-serde/v0.14.4 / 00eeadfd60ae78cb0cbb0207c40efe9a2836697a) outdated

secp256k1 @ 0.20.3 / 0.24.3 (https://github.com/rust-bitcoin/rust-secp256k1/tree/secp256k1-0.24.3 / a1fb0386dcdf1fadd58c5384b2203a14fce56492) multiple versions utilized (including outdated)

serde @ 1.0.164 (https://github.com/serde-rs/serde/tree/v1.0.164 / 107018c6285e87f4cb265e1ef6c3d7931c1a15eb) outdated

serde_bytes @ 0.11.5 (https://github.com/serde-rs/bytes/tree/0.11.5 / 7f2e50869d87a50f45229d36740d9ad874ec9f4b) outdated

sha2 @ 0.8.2 / 0.9.9 / 0.10.7 (https://github.com/RustCrypto/hashes/tree/sha2-v0.10.7/sha2 / d8b088aece5659cecdce397ce48d6129b6c4c440) multiple versions utilized (including outdated)

smallvec @ 0.6.14 / 1.6.1 (https://github.com/servo/rust-smallvec/tree/v1.6.1 / 4e53e072808815ed6b847c77193c568ee076c29d) multiple versions utilized (including outdated)

syn @ 0.11.11 / 0.15.44 / 1.0.95 / 2.0.23 (https://github.com/dtolnay/syn/tree/2.0.23 / f3cd37a3fcd60f79fbf20581f91bc447e46d295a) multiple versions utilized (including outdated)

void @ 1.0.2 (https://github.com/reem/rust-void/tree/1.0.2 / a6e061227f47ba8798b7e828ed0ac4e25382eb15)

non-wasm specific deps:

instant @ 0.1.12 (https://github.com/sebcrozet/instant/tree/v0.1.12 / 9a486adeee119bdf7ecb85ba627b8112fb7eff86)
libp2p (https://github.com/KomodoPlatform/rust-libp2p.git) NOT REVIEWED
tokio @ 1.20 / 1.28.2 (https://github.com/tokio-rs/tokio/tree/tokio-1.28.2 / e87ff8a83a2edea1b23cf62269b8b31ef7f1df2d) multiple versions utilized (including outdated)

wasm specific deps:

instant @ 0.1.12 (https://github.com/sebcrozet/instant/tree/v0.1.12 / 9a486adeee119bdf7ecb85ba627b8112fb7eff86)
libp2p (https://github.com/KomodoPlatform/rust-libp2p.git) NOT REVIEWED

dev deps:

async-std @ 1.6.2 (https://github.com/async-rs/async-std/tree/caa76af745f986c2a12aac1b33b163b5a9633f05 / caa76af745f986c2a12aac1b33b163b5a9633f05 / NB: v1.6.2 tag was deleted from git) outdated
env_logger @ 0.9.3 (https://github.com/rust-cli/env_logger/tree/v0.9.3 / 3da11042a1f6a3b0808e52e534d1a39765bdb4ce) outdated

shamardy · 2023-09-27T13:06:36Z

@onur-ozkan can you please resolve conflicts so that we can merge this PR?

…f-network-stack Signed-off-by: onur-ozkan <work@onurozkan.dev>

Signed-off-by: onur-ozkan <work@onurozkan.dev>

…f-network-stack

Signed-off-by: onur-ozkan <work@onurozkan.dev>

CharlVS · 2023-12-17T16:26:59Z

For anyone getting failed Android builds when trying to build the Android app bundle, it's a simple fix to upgrade minSdkVersion in your gradle file to 28 or newer. This will then use the compatible NDK version, so there's no need to manually override it in your gradle file.

onur-ozkan added 6 commits June 22, 2023 11:41

WIP: start implementing events and request response

ea4c538

Signed-off-by: ozkanonur <work@onurozkan.dev>

WIP: dummpy implementation for p2p request-response

df3910c

Signed-off-by: ozkanonur <work@onurozkan.dev>

implement peers and ping behaviours

ca352ea

Signed-off-by: ozkanonur <work@onurozkan.dev>

implement in memory peer store behaviour

16ab3e8

Signed-off-by: ozkanonur <work@onurozkan.dev>

implement swarm runtime and partial root behaviour

7fe3672

Signed-off-by: ozkanonur <work@onurozkan.dev>

save dev state

e9c7a23

Signed-off-by: ozkanonur <work@onurozkan.dev>

onur-ozkan added the in progress label Jun 27, 2023

onur-ozkan added 12 commits July 5, 2023 19:34

[WIP] define AtomicDexGossipsub with relays in it

b459a7f

Signed-off-by: ozkanonur <work@onurozkan.dev>

partially implement relay behaviour

5174fd2

Signed-off-by: ozkanonur <work@onurozkan.dev>

extend relay::ControlAction

40e564c

Signed-off-by: ozkanonur <work@onurozkan.dev>

move to KomodoPlatform/rust-libp2p

c37d6a1

Signed-off-by: ozkanonur <work@onurozkan.dev>

save dev state

a13b817

Signed-off-by: ozkanonur <work@onurozkan.dev>

save dev state

53b7418

Signed-off-by: ozkanonur <work@onurozkan.dev>

save dev state

82fbe06

Signed-off-by: ozkanonur <work@onurozkan.dev>

WIP: implement NetworkBehaviour for AtomicDexBehaviour

d598716

Signed-off-by: ozkanonur <work@onurozkan.dev>

complete ping implementation

b2427cc

Signed-off-by: ozkanonur <work@onurozkan.dev>

implement most of the peers_exchange

3c6134f

Signed-off-by: ozkanonur <work@onurozkan.dev>

implement most of request_response

96335e0

Signed-off-by: ozkanonur <work@onurozkan.dev>

test new layer with --no-fail-fast flag

980ac88

Signed-off-by: ozkanonur <work@onurozkan.dev>

onur-ozkan force-pushed the ref-network-stack branch from 8be0055 to 980ac88 Compare July 10, 2023 21:03

onur-ozkan added 10 commits July 11, 2023 00:23

update mesh configuration and message signing

4f201ac

Signed-off-by: ozkanonur <work@onurozkan.dev>

run gossipsub

4bad749

bump libp2p

3ecf66c

add todo

7544912

Signed-off-by: ozkanonur <work@onurozkan.dev>

activate sub behaviours abstracted in CoreBehaviour

4ad9dd4

Signed-off-by: ozkanonur <work@onurozkan.dev>

bump libp2p

ee0864d

Signed-off-by: ozkanonur <work@onurozkan.dev>

print error log on GossipsubEvent::GossipsubNotSupported

6bfb404

Signed-off-by: ozkanonur <work@onurozkan.dev>

bump libp2p

252b9ad

Signed-off-by: ozkanonur <work@onurozkan.dev>

handle request-response polling

d97294d

Signed-off-by: ozkanonur <work@onurozkan.dev>

improve request-response protocol polling

1839c74

Signed-off-by: ozkanonur <work@onurozkan.dev>

shamardy added the deps-review label Aug 23, 2023

shamardy requested review from Alrighttt, DeckerSU and ca333 August 23, 2023 11:17

bump mm2

a4f28ba

Signed-off-by: ozkanonur <work@onurozkan.dev>

onur-ozkan dismissed stale reviews from shamardy and artemii235 via a4f28ba August 24, 2023 12:03

Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into re…

fea353e

…f-network-stack

This was referenced Aug 25, 2023

[bug][wasm] defi framework support beyond web_sys::window context #1948

Closed

fix(wasm worker env): refactor direct usage of window #1953

Merged

Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into re…

ec60bf9

…f-network-stack

shamardy mentioned this pull request Sep 18, 2023

chore(release): v2.0.0-beta #1968

Merged

ca333 previously approved these changes Sep 27, 2023

View reviewed changes

Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into re…

c504939

…f-network-stack Signed-off-by: onur-ozkan <work@onurozkan.dev>

onur-ozkan dismissed ca333’s stale review via c504939 September 27, 2023 13:24

smk762 mentioned this pull request Sep 27, 2023

[FR] Seednodes and NetID Update #1977

Closed

onur-ozkan added 3 commits September 29, 2023 00:20

bump default netid

73b8aac

Signed-off-by: onur-ozkan <work@onurozkan.dev>

Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into re…

41a4a85

…f-network-stack

fix wasm lint

f9dcac7

Signed-off-by: onur-ozkan <work@onurozkan.dev>

shamardy merged commit a2cb99b into dev Oct 2, 2023

shamardy deleted the ref-network-stack branch October 2, 2023 10:54

shamardy added breaking-change: P2P and removed breaking-change labels Nov 4, 2024

shamardy mentioned this pull request Sep 2, 2025

fix(orderbook): validate roots before commit #2605

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(network): upgrade p2p layer#1878

feat(network): upgrade p2p layer#1878
shamardy merged 64 commits intodevfrom
ref-network-stack

onur-ozkan commented Jun 22, 2023 •

edited

Loading

Uh oh!

onur-ozkan commented Aug 23, 2023

Uh oh!

shamardy commented Aug 24, 2023

Uh oh!

laruh commented Sep 6, 2023

Uh oh!

onur-ozkan commented Sep 6, 2023

Uh oh!

ca333 left a comment

Uh oh!

shamardy commented Sep 27, 2023

Uh oh!

CharlVS commented Dec 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

onur-ozkan commented Jun 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

onur-ozkan commented Aug 23, 2023

Uh oh!

shamardy commented Aug 24, 2023

Uh oh!

laruh commented Sep 6, 2023

Uh oh!

onur-ozkan commented Sep 6, 2023

Uh oh!

ca333 left a comment

Choose a reason for hiding this comment

Uh oh!

shamardy commented Sep 27, 2023

Uh oh!

CharlVS commented Dec 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

onur-ozkan commented Jun 22, 2023 •

edited

Loading