[Code] distributed Code abstraction by spacedragon · Pull Request #41374 · elastic/kibana

spacedragon · 2019-07-17T15:54:37Z

Summary

https://github.com/elastic/code/issues/1449

Basically, this PR provides an RPC style abstraction layer to dispatch resource-aware request to

local kibana instance
Code Node in the mutli-code mode
kibana-ha node( in future)

How to

define a service definition object like the following:

export const GitServiceDefinition = {
  fileTree: {
    request: {} as {
      uri: string;
      path: string;
      revision: string;
      skip: number;
      limit: number;
      withParents: boolean;
      depth: number;
      flatten: boolean;
    },
    response: {} as FileTree,
  },
};

register it

  distributedCode.registerHandler(GitServiceDefinition, /* service implementation */);

consuming it

  // get the service
  const gitService = distributedCode.serviceFor(GitServiceDefinition);
 // locate the endpoint by request and resource uri
  const endpoint = await distributedCode.locate(req, uri);
// invoke service method
  const result = await gitService.fileTree(endpoint,  params)

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

~~- [ ] This was checked for cross-browser compatibility, including a check against IE11~~
~~- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
~~- [ ] This was checked for keyboard-only and screenreader accessibility~~

For maintainers

This was checked for breaking API changes and was labeled appropriately
This includes a feature addition or change that requires a release note and was labeled appropriately

elasticmachine · 2019-07-17T21:24:37Z

Pinging @elastic/code

elasticmachine · 2019-07-23T13:44:19Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

mw-ding

mostly good. leave some comments for discussions.

x-pack/legacy/plugins/code/server/distributed/handler_adpter.ts

mw-ding · 2019-07-23T21:42:23Z

x-pack/legacy/plugins/code/server/init.ts

these refactoring could possibly bring a bunch of conflicts with the new platform migration PR. since you have already done, let's just let it be. But please do not add more refactoring of the existing code now.

I'm actually not very like the refactoring here, I think those code can be deleted if we use Dependency Injection.

yes, that could save us from copying/pasting some instances around. let's do it in a separate one maybe after the new platform migration refactoring is done.

mw-ding · 2019-07-23T21:43:52Z

x-pack/legacy/plugins/code/server/init.ts

nit: might as well add a logging for non-code-node here.

mw-ding · 2019-07-23T21:59:29Z

x-pack/legacy/plugins/code/server/init.ts

Another comment here is that since you have already introduced an adapter layer (CodeNodeAdapter, LocalAdapter), we can just leave the non-code-node logic into the abstraction of the adapter by creating a new adapter for non-code-node (e.g. RemoteHandlerAdapter so that we don't need to have a different initNonCodeNode function here. The init logic in my mind should look like

let adapter = null; if (code node) { adapter = new CodeNodeAdapter(...); } else if (non code node) { adapter = new RemoteHandlerAdapter(codeNodeUrl, ...) } else { adapter = new LocalHandlerAdapter(...); } const codeServices = new CodeServices(adapter); await initCodeNode(..., codeServices, ...)

Hopefully, this is more readable.

I see that you already have a NonCodeNodeAdapter there, just replace RemoteHandlerAdapter to NonCodeNodeAdapter in the comment above

In non-code mode, I really don't want to initiate any local resource related code(git queue, etc). That's why initNonCodeNode exists

mw-ding · 2019-07-23T22:13:11Z

x-pack/legacy/plugins/code/server/distributed/multinode/non_code_node_adapter.ts

nit: implementation -> implement

mw-ding · 2019-07-23T22:14:29Z

x-pack/legacy/plugins/code/server/distributed/multinode/non_code_node_adapter.ts

nit: a brief comment on that serverHandler will always be null here since it will be overridden by the request forwarding.

mw-ding · 2019-07-23T22:25:05Z

x-pack/legacy/plugins/code/server/distributed/resource_locator.ts

Looks like Endpoint and RequestContext are pretty much the same thing based on this PR. The implementations of toContext() of each endpoints can be moved into their constructors.

You are right, the endpoint was designed to represent the target kibana node address we are sending to. Here in Local and Multi-node mode, the address is always the same. So, they look like the same thing.

elasticmachine · 2019-07-24T05:54:55Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-24T07:19:22Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-25T04:55:02Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-25T09:36:10Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

mw-ding

LG. We might want to hold merging this PR to master until the 7.3 has been officially released just in case backporting code to 7.3 branch. Also, might need another quick sync on resolving conflicts with the new platform migration PR: #41192 (comment)

elasticmachine · 2019-07-29T03:23:21Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-30T12:48:40Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-07-31T03:18:23Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* [Code] distributed Code abstraction

…-or-edit-existing-rollup-job * 'master' of github.com:elastic/kibana: (114 commits) [ML] Fixing empty index pattern list (elastic#42299) [Markdown] Shim new platform - cleanup plugin (elastic#41760) [Code] Enable hierarchicalDocumentSymbolSupport for java language server (elastic#42233) Add New Platform mocks for data plugin (elastic#42261) Http server route handler implementation (elastic#41894) [SR] Allow custom index pattern to be used instead of selectable list when choosing indices to restore (elastic#41534) [Code] distributed Code abstraction (elastic#41374) [SIEM] Sets page titles to the current page you are on (elastic#42157) Saved Objects export API stable type order (elastic#42310) cancellation of interpreter execution (elastic#40238) [SIEM] Fixes a crash when Machine Learning influencers is an undefined value (elastic#42198) Changed the job to work with a dedicated index (elastic#42297) FTR: fix testSubjects.missingOrFail (elastic#42290) Increase retry timeout to prevent flaky tests (elastic#42291) Spaces - make space a hidden saved object type (elastic#41688) Allow applications to register feature privileges which are excluded from the base privileges (elastic#41300) Disable flaky log column reorder test (elastic#42285) Fixing add element in element reducer (elastic#42276) Fix infinite loop (elastic#42228) [Maps][File upload] Remove geojson deep clone logic, handle on maps side (elastic#41835) ...

spacedragon requested review from fantapsody, mw-ding and zfy0701 July 17, 2019 15:54

mw-ding added the Team:Code label Jul 17, 2019

spacedragon force-pushed the code_ha branch 6 times, most recently from 0f17e2e to 42a21e2 Compare July 23, 2019 11:15

spacedragon marked this pull request as ready for review July 23, 2019 11:17

spacedragon added release_note:enhancement v7.4.0 v8.0.0 labels Jul 23, 2019

spacedragon force-pushed the code_ha branch from 42a21e2 to b04dc81 Compare July 23, 2019 11:20

elastic deleted a comment from elasticmachine Jul 23, 2019

spacedragon force-pushed the code_ha branch from b04dc81 to 42df5cd Compare July 23, 2019 12:45

elastic deleted a comment from elasticmachine Jul 23, 2019

mw-ding reviewed Jul 23, 2019

View reviewed changes

spacedragon force-pushed the code_ha branch from 42df5cd to 236a4c5 Compare July 24, 2019 02:52

spacedragon force-pushed the code_ha branch from 236a4c5 to c205a60 Compare July 24, 2019 06:17

spacedragon force-pushed the code_ha branch from c205a60 to 5870157 Compare July 25, 2019 01:53

mw-ding approved these changes Jul 25, 2019

View reviewed changes

zfy0701 approved these changes Jul 29, 2019

View reviewed changes

spacedragon added 2 commits July 30, 2019 16:39

[Code] distributed Code abstraction

c31e19f

[Code] change all service to use new framework

7f6bca5

spacedragon force-pushed the code_ha branch from de5dbf4 to 7d79027 Compare July 30, 2019 09:47

[Code] change all service to use new framework

9231df0

spacedragon force-pushed the code_ha branch from 7d79027 to 9231df0 Compare July 31, 2019 02:13

spacedragon merged commit 0c8159f into elastic:master Jul 31, 2019

spacedragon mentioned this pull request Jul 31, 2019

[7.x] [Code] distributed Code abstraction (#41374) #42318

Merged

spacedragon pushed a commit to spacedragon/kibana that referenced this pull request Jul 31, 2019

[Code] distributed Code abstraction (elastic#41374)

f080bee

* [Code] distributed Code abstraction

spacedragon pushed a commit that referenced this pull request Jul 31, 2019

[Code] distributed Code abstraction (#41374) (#42318)

28fc01e

* [Code] distributed Code abstraction

Conversation

spacedragon commented Jul 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to

Checklist

For maintainers

Uh oh!

elasticmachine commented Jul 17, 2019

Uh oh!

elasticmachine commented Jul 23, 2019

💚 Build Succeeded

Uh oh!

mw-ding left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 24, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Jul 24, 2019

💚 Build Succeeded

Uh oh!

elasticmachine commented Jul 25, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Jul 25, 2019

💚 Build Succeeded

Uh oh!

mw-ding left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 29, 2019

💚 Build Succeeded

Uh oh!

elasticmachine commented Jul 30, 2019

💔 Build Failed

Uh oh!

elasticmachine commented Jul 31, 2019

💚 Build Succeeded

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

spacedragon commented Jul 17, 2019 •

edited

Loading