Skip to content

[6.7] Improve performance of the Logstash Pipeline Viewer (#33793)#33849

Merged
ycombinator merged 1 commit intoelastic:6.7from
ycombinator:backport/6.7/pr-33793
Mar 26, 2019
Merged

[6.7] Improve performance of the Logstash Pipeline Viewer (#33793)#33849
ycombinator merged 1 commit intoelastic:6.7from
ycombinator:backport/6.7/pr-33793

Conversation

@ycombinator
Copy link
Copy Markdown
Contributor

Backports the following commits to 6.7:

Resolves elastic#27513.

_This PR is a combination of elastic#31293 (the code changes) + elastic#33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._

Summary of changes, taken from elastic#31293:

The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline.

Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser.

This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer.

Based on a simple, 1-input, 1-filter, and 1-output pipeline.

* Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets.

* Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size.

* Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size.

* Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size.
@ycombinator ycombinator added the backport This PR is a backport of another PR label Mar 26, 2019
@ycombinator
Copy link
Copy Markdown
Contributor Author

retest

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

@ycombinator ycombinator merged commit 30dd4e1 into elastic:6.7 Mar 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport This PR is a backport of another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants