Skip to content

feat(event streaming): implement push-only streaming channels and SSE#1945

Merged
shamardy merged 21 commits intodevfrom
push-only-streaming-channels
Sep 28, 2023
Merged

feat(event streaming): implement push-only streaming channels and SSE#1945
shamardy merged 21 commits intodevfrom
push-only-streaming-channels

Conversation

@onur-ozkan
Copy link
Copy Markdown

@onur-ozkan onur-ozkan commented Aug 23, 2023

This implements streaming channels using mpsc(underlying part of SSE) and SSE for sending data to clients continiously. Currently, only NETWORK event is implemented, further events will be added after the collaboration with the GUI team.

Supported platforms: all other than WASM(Simply because we can't serve a server on browser. I have a solution idea to this, but it's too early to talk about).

Enabling Event Streaming:

  1. To enable event streaming, add the event_stream_configuration property to the mm2 configuration. Use the following sample values:
{
	"gui": "core_readme",
	"netid": 7777,
	// ... other parts
	"event_stream_configuration": {
		"access_control_allow_origin": "*",
		"active_events": {
			"NETWORK": { "stream_interval_seconds": 0.5 }
		}
	}
}

2- Start mm2 and wait for the initialization log message similar to: mm2_net::network_event:54] INFO NETWORK event is activated with 0.5 seconds interval.

3- Open examples/sse/index.html in your browser; events should start appearing on the screen.

Client-Side Filtering:

By default, clients listen to all events activated on the mm2 side. If you're interested in specific events, you can filter incoming events (using the filter query string parameter) to reduce network overhead.

For instance, open examples/sse/index.html with a text editor and append ?filter=TEST,DEMO,BALANCES to the EventSource url. After that, you will no longer receive NETWORK events. (Note that currently we only stream NETWORK events on mm2 side. Filtering out the NETWORK event will result in no events being received.)

Helps to #1901

Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
@onur-ozkan onur-ozkan force-pushed the push-only-streaming-channels branch from 77b0058 to 3e679b4 Compare August 23, 2023 19:46
Signed-off-by: ozkanonur <work@onurozkan.dev>
@onur-ozkan onur-ozkan force-pushed the push-only-streaming-channels branch from 3e679b4 to 04df80c Compare August 23, 2023 19:47
Signed-off-by: ozkanonur <work@onurozkan.dev>
Signed-off-by: ozkanonur <work@onurozkan.dev>
@onur-ozkan onur-ozkan force-pushed the push-only-streaming-channels branch from 6cdd7b1 to b8eca5e Compare August 24, 2023 12:01
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Signed-off-by: onur-ozkan <work@onurozkan.dev>
Copy link
Copy Markdown
Author

@onur-ozkan onur-ozkan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These notes are not related with this work. They don't seem right in general. Having TODOs will help us tracking them later.

@@ -1,3 +1,5 @@
// TODO: a lof of these implementations should be handled in `mm2_net`
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

☝️

Comment on lines +307 to +308
// TODO: This should exclude TCP internals, as including them results in having to
// handle various protocols within this function.
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

☝️

@onur-ozkan onur-ozkan marked this pull request as ready for review August 29, 2023 10:51
Signed-off-by: onur-ozkan <work@onurozkan.dev>
@shamardy shamardy self-requested a review September 4, 2023 06:00
Copy link
Copy Markdown
Collaborator

@shamardy shamardy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great Work! First review iteration!

Signed-off-by: onur-ozkan <work@onurozkan.dev>
@shamardy shamardy requested a review from laruh September 11, 2023 10:05
shamardy
shamardy previously approved these changes Sep 20, 2023
Copy link
Copy Markdown
Collaborator

@shamardy shamardy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approve

@shamardy
Copy link
Copy Markdown
Collaborator

@KomodoPlatform/qa: Refer to this comment #1945 (comment) to see how you can test this new event streaming feature. The above comment is also a starting point for documenting this feature, if you wish to begin documentation at this early stage. If you have any questions, please direct them to @onur-ozkan

@smk762
Copy link
Copy Markdown

smk762 commented Sep 22, 2023

Open examples/sse/index.html

I assume this means http://127.0.0.1:7783/examples/sse/index.html?
Response is {"error":"EOF while parsing a value at line 1 column 0"}

and in mm2.log 22 07:03:25, mm2_main::mm2::rpc:286] ERROR RPC error response: EOF while parsing a value at line 1 column 0

@onur-ozkan
Copy link
Copy Markdown
Author

onur-ozkan commented Sep 22, 2023

I assume this means http://127.0.0.1:7783/examples/sse/index.html?

No, it's independent HTML page(for testing SSE connection), you just need to open it in browser. You can serve using any web server like using python SimpleHTTPServer(see https://github.com/KomodoPlatform/komodo-defi-framework/blob/push-only-streaming-channels/examples/sse/README.md#listening-event-stream-from-komodo-defi-framework)
), nodejs http-server, nginx. Or you can open it using the filesystem directly like:

image

Copy link
Copy Markdown

@smk762 smk762 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirm that event steam appears in browser from examples/sse/index.html, and modifying index.html to add a filter works.

Is there a plan to allow the filter to be applied via browser url in future? E.g. examples/sse/index.html?filter=BALANCES instead of editing index.html

@onur-ozkan
Copy link
Copy Markdown
Author

Confirm that event steam appears in browser from examples/sse/index.html, and modifying index.html to add a filter works.

Is there a plan to allow the filter to be applied via browser url in future? E.g. examples/sse/index.html?filter=BALANCES instead of editing index.html

GUI can already achieve that by using URL parameters(e.g., https://developer.mozilla.org/en-US/docs/Web/API/Window/location) in the SSE connection string. mm2 can't know what page it's being used for.

Copy link
Copy Markdown

@kivqa kivqa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.
Event steam displayed for following places :
http://localhost:7783/event-stream
komodo-defi-framework/examples/sse/index.html
http://127.0.0.1:8000/
image
Filter works as expected: for ?filter=NETWORK events are displayed, for other values of filter event stream is empty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants