createComponents allows plugins to return a collection of objects to OpenSearch. PluginService iterates through all plugins and passes several services and registries, ie clusterService, environment, client, NamedXContentRegistry to help these plugins to configure and generate their components. These objects are then consolidated into a list, which is then bound to a Guice module during node initialization and then further used for dependency injection.

In the case of extensibility, since these extensions are running on independent processes, we must add support to transport the necessary information from the services required to successfully create these components.

For Anomaly Detection, in order to support creating a detector, the required components are AnomalyDetectionIndices, SearchFeatureDao, and ADTaskManager.

These components require several services from OpenSearch, and currently there is added support for the following :

Client : replaced by SDKClient

Some of the required services from OpenSearch are not necessary to add support for as they are not required for creating a detector :

NamedXContentRegistry : replaced with an empty collection

These objects require transport APIs to be created in order to access :

ClusterService
Settings

ClusterService is used by Anomaly Detection to access the ClusterSettings in order to add custom settings update consumers. The purpose of these setting consumers is to perform actions defined by the extension whenever a particular setting is updated, usually by updating the field corresponding to the setting with the data passed by the consumer. These custom consumers are added within the object constructors of each component, so it is necessary to enable extensions to perform these actions as part of component creation.

For example :

To add support for extensions to add custom settings update consumers during create components :

Create a component settings consumer map within extension runner of Setting objects and their setting update consumers, which components will add to as part of their constructors
Create a Transport Request from extensions to OpenSearch to add custom settings update consumers. The request parameters will include a List of settings for each component, and the DiscoveryExtension to which the consumer callback method shall point to
Use ExtensionsOrchestrator's clusterService, which is initialized here and invoke getClusterSettings.addSettingsUpdateConsumer for each transported Setting, and set each corresponding Consumer to send a UpdateSettingsRequest back to the DiscoveryExtension node along with the Setting object, Setting type, and data.
Send an ExtensionBooleanResponse as an acknowledgement to the extension upon successfully registering the setting update consumer within ClusterService's ClusterSettings
When a setting is updated, the setting updater that is registered within ClusterSettings validates the setting change and invokes the consumer. This consumer transports an UpdateSettingsRequest to the extension, which receives the request and cross-references the request's Setting with the corresponding Setting within the component setting consumer map
Once the corresponding consumer is identified, determine the setting type, cast the request data and corresponding consumer to the setting type and invoke the consumer via consumer.accept(data) to update the setting
Upon successfully updating the setting, the extension sends an ExtensionBooleanResponse back to Opensearch as an acknowledgement

Settings come from the Environment, which is created here . Extensions use environment settings to retrieve values (or default values if the setting is not present in the provided Settings object) in order to set fields during component initialization. To add support for extensions :

Previous Design for Environment Settings

Pass in environment settings from Node.java to ExtensionsOrchestrator during node initialization
Create a Transport Request from extensions to OpenSearch for environment settings

New Design for Environment Settings

Upon further discussion, support for environment settings will change. Rather than transporting all the OpenSearch environment settings to extensions, extensions will send OpenSearch a list of Setting objects that it would want to retrieve values from. The following workflow will achieve this support :

Pass in environment settings from Node.java to ExtensionsOrchestrator during node initialization
During createComponents, for each component, the Extension will send a request to OpenSearch containing a List of Setting objects
OpenSearch will receive this list and invoke Setting<T>.get(environmentSettings) to retrieve the corresponding values (or default value if the setting object is not yet registered) and map them to the setting
OpenSearch will respond with the Map<Setting, value> of which the SDK will maintain (lets name this map ComponentSettingValues)
The extension will then invoke ComponentSettingsValue.get(Setting) to retrieve the value of the setting from the OpenSearch environment

Once these transport APIs are created, integrating the opensearch-sdk with Anomaly Detection would allow us call these transport APIs to retrieve the necessary information to generate the components needed for creating a detector.

Add create component extension point support for AD #78

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions