Skip to content

Add an internal refresh api that makes pipeline changes visible on all ingest nodes#15203

Merged
martijnvg merged 1 commit intoelastic:feature/ingestfrom
martijnvg:ingest_refresh_pipeline_after_modifications
Dec 18, 2015
Merged

Add an internal refresh api that makes pipeline changes visible on all ingest nodes#15203
martijnvg merged 1 commit intoelastic:feature/ingestfrom
martijnvg:ingest_refresh_pipeline_after_modifications

Conversation

@martijnvg
Copy link
Copy Markdown
Member

instead of only relying on async background task that periodically loads pipelines into memory. This async loading mechanism remains for loading pipelines during startup.

  • Pipeline store can now only be used when there is no .ingest index or all primary shards of .ingest have been started
  • IngestPlugin addsnode.ingest setting to true. This is used to figure out to what nodes to send the refresh request too. This setting isn't yet configurable. This will be done in a follow up issue.
  • Async background loading mechanism only runs now every 30 minutes instead of every second.

@martijnvg martijnvg added review :Distributed/Ingest Node Execution or management of Ingest Pipelines labels Dec 2, 2015
@martijnvg martijnvg changed the title Add an internal refresh api that makes pipeline changes are visible on all ingest nodes Add an internal refresh api that makes pipeline changes visible on all ingest nodes Dec 2, 2015
@martijnvg
Copy link
Copy Markdown
Member Author

I've updated this PR to move some initialisation logic out of the store into a new class called PipelineStoreBootstrapper. The pipeline store became a bit convoluted with initialisation logic.

martijnvg referenced this pull request in s1monw/elasticsearch Dec 3, 2015
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/representaion/representation/

@talevy
Copy link
Copy Markdown
Contributor

talevy commented Dec 3, 2015

would it be beneficial to have refresh action/request tests in IngestClientIT?

@martijnvg
Copy link
Copy Markdown
Member Author

I think it makes sense to add a test for this api, even though this is an internal api that is designed to be used only via put and delete pipeline apis. The reason that I did not add an integration test for this yet, is that it is already tests by the put and delete pipeline rest and ClientIT tests.

@talevy
Copy link
Copy Markdown
Contributor

talevy commented Dec 3, 2015

but these are just one-node tests, no? since this gets info from cluster-state about multiple nodes (some of which may not be ingest nodes), maybe it makes sense to test those cases as well

edit: never-mind, that is handled from within the pipelinestore

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the api is really internal, I think we can simplify this. Do we need to use a client here? Can we instead use the transport service directly? In that case we wouldn't need the RefreshAction, and the RefreshRequestBuilder. Otherwise the api ends up being exposed anyways, no matter if we say it's internal, but it doesn't have a corresponding REST handler, which makes things inconsistent.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed and this what I initially tried (like the shards exists internal api, See IndicesStore.java), but then I ran into cycle dependency issue with transport service.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok...but client depends on the transport service anyway no? I think I don't get it

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes it does. all the ingest transport stuff gets instantiate with the the
action module and not with the node modules of the plugin. (See
IngestPlugin#onModule(ActionModule) ) So that is why the current approach
works.

On 4 December 2015 at 12:47, Luca Cavanna notifications@github.com wrote:

In
plugins/ingest/src/main/java/org/elasticsearch/plugin/ingest/PipelineStore.java
#15203 (comment)
:

     DeleteRequest deleteRequest = new DeleteRequest(request);
     deleteRequest.index(PipelineStore.INDEX);
     deleteRequest.type(PipelineStore.TYPE);
     deleteRequest.id(request.id());
     deleteRequest.refresh(true);
  •    client().delete(deleteRequest, listener);
    
  •    client().delete(deleteRequest, new ActionListener<DeleteResponse>() {
    
  •        @Override
    
  •        public void onResponse(DeleteResponse deleteResponse) {
    
  •            client().execute(RefreshAction.INSTANCE, new RefreshRequest(), new ActionListener<RefreshResponse>() {
    

ok...but client depends on the transport service anyway no? I think I
don't get it


Reply to this email directly or view it on GitHub
https://github.com/elastic/elasticsearch/pull/15203/files#r46674077.

Met vriendelijke groet,

Martijn van Groningen

@martijnvg martijnvg force-pushed the ingest_refresh_pipeline_after_modifications branch from f914155 to 9af7646 Compare December 7, 2015 09:34
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use == false

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, will change.

@martijnvg
Copy link
Copy Markdown
Member Author

@javanna @talevy I've updated the PR. Changed:

  • Renamed refresh action to reload pipelines action.
  • Made the reload pipelines action an internal api (Java clients cannot invoke it).
  • Removed the refreshed field from the response. Whether the reload was successful is ignored and only logged. We should remain to rely on the background pipeline reloading mechanism.
  • Set the reload pipeline background task interval from 30 minutes to 30 seconds.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry but we have to find other solutions for this than using these injection providers. We can't sustain this kind of software engineering here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is easy. Can make this an inner class of PipelineStore and then there isn't a cyclic dependency.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is shit guice exposed at all? Can't we reduce the number of @Inject here please it's such a pain to unwire all of this. Can we simply remove the @Inject and all the wiring and to in PipelineStoreBootstrapper

ReloadPipelinesAction action = new ReloadPipelinesAction(settings, pipelineStore, clusterService, transportService);

we can go even further and also don't wire PipelineStore and just call new in the bootstrapper as well.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the reason pipeline store instance is managed by Guice is that the transport action classes uses it as a dependency. But that shouldn't be a problem, because if the store is exposed as a getter and the transport action classes depend on PipelineStoreBootstrapper then the constructor of the transport action classes can use this getter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed/Ingest Node Execution or management of Ingest Pipelines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants