Skip to content

Modularize Elasticsearch#81066

Merged
ChrisHegarty merged 374 commits intoelastic:masterfrom
ChrisHegarty:module-path2
May 20, 2022
Merged

Modularize Elasticsearch#81066
ChrisHegarty merged 374 commits intoelastic:masterfrom
ChrisHegarty:module-path2

Conversation

@ChrisHegarty
Copy link
Copy Markdown
Contributor

@ChrisHegarty ChrisHegarty commented Nov 25, 2021

This PR represents the initial phase of Modularizing Elasticsearch (with
Java Modules).

This initial phase modularizes the core of the Elasticsearch server
with Java Modules, which is then used to load and configure extension
components atop the server. Only a subset of extension components are
modularized at this stage (other components come in a later phase).
Components are loaded dynamically at runtime with custom class loaders
(same as is currently done). Components with a module-info.class are
defined to a module layer.

This architecture is somewhat akin to the Modular JDK, where
applications run on the classpath. In the analogy, the Elasticsearch
server modules are the platform (thus are always resolved and present),
while components without a module-info.class are non-modular code
running atop the Elasticsearch server modules. The extension components
cannot access types from non-exported packages of the server modules, in
the same way that classpath applications cannot access types from
non-exported packages of modules from the JDK. Broadly, the core
Elasticseach java modules simply "wrap" the existing packages and export
them. There are opportunites to export less, which is best done in more
narrowly focused follow-up PRs.

The Elasticsearch distribution startup scripts are updated to put jars
on the module path (the class path is empty), so the distribution will
run the core of the server as java modules. A number of key components
have been retrofitted with module-info.java's too, and the remaining
components can follow later. Unit and functional tests run as
non-modular (since they commonly require package-private access), while
higher-level integration tests, that run the distribution, run as
modular.

relates #78744

@ChrisHegarty ChrisHegarty changed the title Draft - Module path2 WIP - Modularize Elasticserch Dec 3, 2021
@ChrisHegarty ChrisHegarty changed the title WIP - Modularize Elasticserch WIP - Modularize Elasticsearch Dec 3, 2021
@ChrisHegarty ChrisHegarty force-pushed the module-path2 branch 4 times, most recently from 110772d to 4d1127d Compare December 3, 2021 19:11
@ChrisHegarty
Copy link
Copy Markdown
Contributor Author

@elasticmachine retest this please

@ChrisHegarty ChrisHegarty added the modularization Java Modules related label Dec 6, 2021
@ChrisHegarty ChrisHegarty self-assigned this Dec 23, 2021
Copy link
Copy Markdown
Member

@rjernst rjernst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good.

A couple high level thoughts that can be followups:

  • It's difficult to picture the hierarchy of loaders (it was already the case, but I think more so now). It might be useful to have some high level docs on the PluginService describing all the layers and loaders that we create.
  • We have the plugin loader creation already separated out to a separate jar (and separate permission), but this change now adds those permissions (get/create) also to server. Long term I'm hoping we can move plugin loading out of server into it's own isolated module (so that we can add eg asm to do some fancier dynamic wrapping of server's interaction with each plugin). Since the PluginService is now directly manipulating the Module of server, I'm not sure that is possible. Not a huge deal, just something to think about as we move forward, how we might achieve separation of the plugin service from server.

@ChrisHegarty ChrisHegarty merged commit 3071c6a into elastic:master May 20, 2022
@ChrisHegarty
Copy link
Copy Markdown
Contributor Author

This looks good.

A couple high level thoughts that can be followups:

  • It's difficult to picture the hierarchy of loaders (it was already the case, but I think more so now). It might be useful to have some high level docs on the PluginService describing all the layers and loaders that we create.
  • We have the plugin loader creation already separated out to a separate jar (and separate permission), but this change now adds those permissions (get/create) also to server. Long term I'm hoping we can move plugin loading out of server into it's own isolated module (so that we can add eg asm to do some fancier dynamic wrapping of server's interaction with each plugin). Since the PluginService is now directly manipulating the Module of server, I'm not sure that is possible. Not a huge deal, just something to think about as we move forward, how we might achieve separation of the plugin service from server.

Excellent points. I will address them in a follow up PR.

elasticsearchmachine pushed a commit that referenced this pull request May 23, 2022
This is a follow up to #81066 which ensures that we track the module
path as a input on Java compilation tasks. Since we truncate the compile
classpath and pass the module path as separate CLI arguments we need to
track those separately. We do that by simply annotating a getter on the
`CommandLineArgumentProvider`.
ywangd added a commit to ywangd/elasticsearch that referenced this pull request May 24, 2022
org.apache.logging.log4j.core is referenced by the LoggingAuditTrail
class. This PR adds it to the requires list of the security module.

Relates: elastic#81066
elasticsearchmachine pushed a commit that referenced this pull request May 24, 2022
org.apache.logging.log4j.core is referenced in the LoggingAuditTrail
class. This PR adds it to the requires list of the security module.

Relates: #81066
@ChrisHegarty ChrisHegarty added v8.3.0 and removed WIP labels Jul 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Core/Infra/Core Core issues without another label :Delivery/Build Build or test infrastructure modularization Java Modules related Team:Core/Infra Meta label for core/infra team Team:Delivery Meta label for Delivery team test-windows Trigger CI checks on Windows v8.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants