Skip to content
This repository was archived by the owner on Dec 16, 2020. It is now read-only.

datasource: use async data source for wasm#216

Merged
jplevyak merged 16 commits intoenvoyproxy:masterfrom
yxue:datasource
Nov 6, 2019
Merged

datasource: use async data source for wasm#216
jplevyak merged 16 commits intoenvoyproxy:masterfrom
yxue:datasource

Conversation

@yxue
Copy link
Copy Markdown
Member

@yxue yxue commented Sep 27, 2019

Use async data source

Description: use async data source
Risk Level: Med
Testing: Unit test
Docs Changes:
Release Notes:
Fixes #Issue: Fix #187
[Optional Deprecated:]

@yxue yxue force-pushed the datasource branch 2 times, most recently from 5de0363 to a5cc183 Compare September 27, 2019 20:58
@yxue yxue changed the title wasm: make createWasm async datasource: make createWasm async Sep 27, 2019
@yxue yxue closed this Sep 27, 2019
@yxue yxue reopened this Sep 27, 2019
@yxue yxue changed the title datasource: make createWasm async datasource: remove return value and add a callback in createWasm Sep 27, 2019
@yxue yxue changed the title datasource: remove return value and add a callback in createWasm [WIP] datasource: use async data source Sep 27, 2019
@yxue yxue changed the title [WIP] datasource: use async data source datasource: remove return value and add a callback in createWasm Sep 27, 2019
@yxue yxue force-pushed the datasource branch 3 times, most recently from 9a8b1bc to 66b784b Compare September 30, 2019 19:03
@yxue yxue changed the title datasource: remove return value and add a callback in createWasm datasource: use async data source for wasm Sep 30, 2019
@yxue yxue force-pushed the datasource branch 10 times, most recently from 320163f to f8ab13b Compare October 3, 2019 04:18
Signed-off-by: crazyxy <yxyan@google.com>
@yxue
Copy link
Copy Markdown
Member Author

yxue commented Oct 3, 2019

@jplevyak @PiotrSikora the PR is ready for review. PTAL. Thanks!

@PiotrSikora
Copy link
Copy Markdown
Contributor

@yxue could you merge master to resolve conflicts?

Signed-off-by: crazyxy <yxyan@google.com>
Signed-off-by: Yan Xue <yxyan@google.com>
auto tls_slot = context.threadLocal().allocateSlot();
// Create a base WASM to verify that the code loads before setting/cloning the for the
// individual threads.
auto plugin = std::make_shared<Common::Wasm::Plugin>(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we pull this out of the callback ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If code is not included in the Plugin, we can. I added code in the Plugin in this PR so we cannot. But we can keep code out of the Plugin. Which one would you prefer?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you pass it into Common::Wasm::createWasm(). The plugin is supposed to have only information specific to the particular plugin point, while the code is only used to create the initial Wasm (or non-cloneable thread-local Wasm(s)) but in any case is common to all plugin points. The new PR I submitted uses the copy constructor of Wasm for all the thread-local versions of the VMs, so you can store the code in Wasm and then access it if necessary in the copy constructor, but do not store it in the copy constructed Wasm(s) because it could be big and we don't want to waste the memory. The code in the base_wasm will be deleted when the last thread-local Wasm is created.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jplevyak update the PR to pass the code to Common::Wasm::createWasm() instead of putting it in Plugin

yxue added 2 commits October 10, 2019 17:55
Signed-off-by: Yan Xue <yxyan@google.com>
Signed-off-by: Yan Xue <yxyan@google.com>
Common::Wasm::getOrCreateThreadLocalWasm(*base_wasm, *configuration, dispatcher));
});
return std::make_shared<WasmAccessLog>(root_id, std::move(tls_slot), std::move(filter));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you encapsulate or create a function (e.g. in wasm.h/cc) to do this repeated code? We have a number of plugin points and are going to have more so it will be repeated several more times.

yxue added 6 commits October 14, 2019 11:26
Signed-off-by: Yan Xue <yxyan@google.com>
Signed-off-by: Yan Xue <yxyan@google.com>
Signed-off-by: Yan Xue <yxyan@google.com>
Signed-off-by: Yan Xue <yxyan@google.com>
Signed-off-by: Yan Xue <yxyan@google.com>
@jplevyak
Copy link
Copy Markdown
Contributor

As discussed offline we need a way of propagating the (currently) exception errors that are now uncaught in the async operation callback back to xDS probably via the init manager. Prior to this change they would have occurred inline during configuration and be caught. Now they will causes Envoy to exit.

@jplevyak
Copy link
Copy Markdown
Contributor

OK, LGTM, but I would like to get Piotr's opinion. @PiotrSikora

Copy link
Copy Markdown
Contributor

@PiotrSikora PiotrSikora left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yxue could you merge with master?

yxue added 4 commits November 1, 2019 21:37
Signed-off-by: crazyxy <yxyan@google.com>
Signed-off-by: crazyxy <yxyan@google.com>
Signed-off-by: crazyxy <yxyan@google.com>
@PiotrSikora
Copy link
Copy Markdown
Contributor

@jplevyak can you resolve your "change requested" and merge this? Thanks!

@jplevyak jplevyak merged commit 7c040df into envoyproxy:master Nov 6, 2019
lizan added a commit to lizan/proxy that referenced this pull request Nov 13, 2019
Signed-off-by: Lizan Zhou <lizan@tetrate.io>
istio-testing pushed a commit to istio/proxy that referenced this pull request Nov 15, 2019
* Update envoy-wasm sha

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

* fix due to ABI change

* remove compile db

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

* fix for envoyproxy/envoy-wasm#216

Signed-off-by: Lizan Zhou <lizan@tetrate.io>

* fix ci

* roll forward

Signed-off-by: Lizan Zhou <lizan@tetrate.io>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for remote datasources (RemoteAsyncDataProvider) for loading .wasm files from a URI.

3 participants