Skip to content

x-pack/plugin/apm: introduce x-pack-apm plugin#97546

Merged
axw merged 53 commits intoelastic:mainfrom
axw:apm-data-streams
Oct 31, 2023
Merged

x-pack/plugin/apm: introduce x-pack-apm plugin#97546
axw merged 53 commits intoelastic:mainfrom
axw:apm-data-streams

Conversation

@axw
Copy link
Copy Markdown
Member

@axw axw commented Jul 11, 2023

Introduce the x-pack-apm plugin, which installs index templates, component templates, and ingest pipelines for Elastic APM.

This replaces the need for installing any Fleet integration package for Elastic APM, unless central configuration/fleet management is desired, simplifying setup for users of the apm-server binary. This will also simplify APM Server by removing the possibility of index templates being missing on startup.

@elasticsearchmachine elasticsearchmachine added v8.10.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jul 11, 2023
@axw axw force-pushed the apm-data-streams branch from 3c4da9b to e58ae4a Compare July 11, 2023 07:13
@felixbarny
Copy link
Copy Markdown
Member

How do we ensure rollovers when settings or mappings change? Does this PR depend on #96521?

@axw
Copy link
Copy Markdown
Member Author

axw commented Jul 12, 2023

@felixbarny we don't. We would either need #96521, or do it in the plugin:

https://github.com/elastic/elasticsearch/pull/97546/files#diff-652b74c665f6638e6113270681056a55bca105de6a22a297f83c0312634c841dR45-R47

// TODO(axw) when the template version is updated, data streams using the old
// version should be rolled over after upgrading. Ideally this would be done
// automatically: #96521

😄

@axw
Copy link
Copy Markdown
Member Author

axw commented Jul 19, 2023

@danielmitterdorfer provided some helpful pointers on Slack:

#97380 adds support for upgrading Universal Profiling data streams and indices. Maybe we can take some inspiration from there to address #96521?

#97773 adds support for migrations. I haven't looked at all the details, but perhaps this could be useful for adding runtime fields to old backing indices for schema evolution.

@eyalkoren
Copy link
Copy Markdown
Contributor

@jbaiera I addressed all review comments. As agreed with @axw and @felixbarny, full integration tests for APM components will be added in a followup effort. I removed the rollover test from the APM data plugin and added a simple one that tests the basic functionality to core instead.
Please go over the non-resolved comments and see if you can resolve them. Thanks for your great feedback 🙏

@eyalkoren eyalkoren requested a review from jbaiera October 19, 2023 10:54
Copy link
Copy Markdown
Member

@jbaiera jbaiera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for iterating on this! Given that the index template registry refactoring is on the todo list for future consideration, I think things look good to me!

@axw axw merged commit 4223cad into elastic:main Oct 31, 2023
@axw axw deleted the apm-data-streams branch October 31, 2023 05:05
@axw
Copy link
Copy Markdown
Member Author

axw commented Oct 31, 2023

Thank you for finishing this up @eyalkoren, and for your support @jbaiera! ❤️

@patpatpat123

This comment was marked as off-topic.

@tvernum

This comment was marked as resolved.

@edzhelyov

This comment was marked as off-topic.

@DaveCTurner

This comment was marked as off-topic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contributor Pull request authored by a developer outside the Elasticsearch team >feature Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v8.12.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.