Skip to content

Batch execute template and pipeline cluster state operations#86017

Merged
dakrone merged 7 commits intoelastic:masterfrom
dakrone:bulk-execute-template-and-pipeline-CS
May 4, 2022
Merged

Batch execute template and pipeline cluster state operations#86017
dakrone merged 7 commits intoelastic:masterfrom
dakrone:bulk-execute-template-and-pipeline-CS

Conversation

@dakrone
Copy link
Copy Markdown
Member

@dakrone dakrone commented Apr 19, 2022

This commit changes the cluster state operations for templates (legacy, component, and composable) as well as ingest pipelines to be bulk executed. This means that they can be processed much faster when creating/updating many simultaneously.

Relates to #77505

@dakrone dakrone changed the title WIP, bulk execute template and pipeline cluster state operations Batch execute template and pipeline cluster state operations Apr 26, 2022
@dakrone dakrone marked this pull request as ready for review April 26, 2022 21:58
@dakrone dakrone added >enhancement :Data Management/Indices APIs DO NOT USE. Use ":Distributed/Indices APIs" or ":StorageEngine/Templates" instead. labels Apr 26, 2022
@elasticmachine elasticmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Apr 26, 2022
@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@dakrone dakrone added the :Distributed/Ingest Node Execution or management of Ingest Pipelines label Apr 26, 2022
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Hi @dakrone, I've created a changelog YAML for you.

@dakrone dakrone removed the :Distributed/Ingest Node Execution or management of Ingest Pipelines label Apr 26, 2022
@martijnvg martijnvg self-requested a review April 28, 2022 15:44
Copy link
Copy Markdown
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR looks good to me.

I do wonder about testing.
Each type of operation is tested (e.g. putting a pipeline) on the unit level and integration level. I think batching implementation is tested as well in ClusterStateService is tested as well. So I think we're good here, right?

}
}

public interface PutListener {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 to see these listeners being replaced by ActionListener

@dakrone
Copy link
Copy Markdown
Member Author

dakrone commented May 3, 2022

So I think we're good here, right?

I think so too, I think adding a test specifically for this would add a ton of complexity.

@martijnvg
Copy link
Copy Markdown
Member

I think adding a test specifically for this would add a ton of complexity.

Yes, I had a similar feeling that these tests require a lot of work without adding much benefit.

Copy link
Copy Markdown
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@dakrone dakrone merged commit 9af8cf2 into elastic:master May 4, 2022
@dakrone dakrone deleted the bulk-execute-template-and-pipeline-CS branch May 4, 2022 15:39
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this pull request May 4, 2022
@DaveCTurner
Copy link
Copy Markdown
Member

For future reference you can use ClusterStateTaskExecutorUtils to simplify the job of writing tests that execute batches of cluster state update tasks. I don't think it's that necessary here, the batching is pretty straightforward, but e.g. in #86446 I'm proposing a change to the IngestServiceTests which involves running the executor code too. The tests still only ever execute singleton batches, but hopefully it's obvious how to extend them to execute multiple tasks in a single batch too.

DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this pull request May 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Data Management/Indices APIs DO NOT USE. Use ":Distributed/Indices APIs" or ":StorageEngine/Templates" instead. >enhancement Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v8.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants