add support for kibana policy overrides in system tests#3094
add support for kibana policy overrides in system tests#3094leehinman merged 1 commit intoelastic:mainfrom
Conversation
💚 Build Succeeded
|
| ```shell | ||
| ELASTIC_PACKAGE_KIBANA_POLICY_OVERRIDES=/tmp/overrides.yml elastic-package test system | ||
| ``` | ||
|
|
||
| and the `/tmp/overrides.yml` file has the following contents: | ||
|
|
||
| ```yaml | ||
| agent: | ||
| monitoring: | ||
| _runtime_experimental: otel | ||
| ``` |
There was a problem hiding this comment.
Given that this intended for system tests, I think it would be better to include this in the test config files.
vars: ~
data_stream:
vars: ~
overrides: |
agent:
monitoring:
_runtime_experimental: otelThough maybe the question is how this setting is intended to be used. An environment variable would be better if we expect to have CI pipelines testing everything with different configurations, while placing it in the test config files would allow to selectively enable it per test.
There was a problem hiding this comment.
We certainly could do them in the test config file.
One reason I went for the env var was because overrides are "unusual". I don't think we want to rely on them for CI. From a customer standpoint you would only use an override if something went wrong and you needed a work around until Elastic issues a fix.
The reason we want to have access to overrides for system tests is so we can test a configuration option without editing the integration in any way. And as you mentioned it gives us the option of doing a CI run with every integration using a particular override.
A concrete example is that we want to run all the metric system tests with a different runtime to make sure we get the same fields. We wouldn't want to edit all the test config files to do that.
There was a problem hiding this comment.
And as you mentioned it gives us the option of doing a CI run with every integration using a particular override.
So being more specific about CI, the idea would be to be able to open a PR with an override file and the environment variable. And this PR would be used only for testing, without merging it. Is that right? For this I agree that an environment variable would make sense.
Adds support for a new environment variable
ELASTIC_PACKAGE_KIBANA_POLICY_OVERRIDESIf this variable is set and is a path to a yaml file, then the contents of the yaml file will be used for the kibana policy overrides in the system tests.
This allows you to test how integrations will behave with experimental settings.
Example
ELASTIC_PACKAGE_KIBANA_POLICY_OVERRIDES=/tmp/overrides.yml elastic-package test systemand the
/tmp/overrides.ymlfile has the following contents:Will result in the system test running with the agent monitoring using the
otelruntime.Closes #2975