[Fleet] Enable experimental toggles in Fleet UI from Kibana config by kpollich · Pull Request #1108 · elastic/elastic-package

kpollich · 2023-01-23T13:43:25Z

Enable the experimental data stream toggles (synthetic source, TSDB, doc-values-only) in Fleet's policy editor by toggling a feature flag in the Kibana config used for launching the local Kibana instance.

Ref elastic/kibana#148418
Ref elastic/kibana#148317

Not sure if this should also be turned on in the other Kibana config files:

kibana_config_8x.yml
kibana_config_80.yml

If they should be edited too, let me know and I'll happily make the changes.

elasticmachine · 2023-01-23T13:43:29Z

Pinging @elastic/fleet (Team:Fleet)

elasticmachine · 2023-01-23T14:17:21Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-01-23T16:01:55.977+0000
Duration: 13 min 59 sec

Test stats 🧪

Test	Results
Failed	0
Passed	464
Skipped	0
Total	464

Steps errors

Expand to view the steps failures

Show only the first 10 steps failures

`Check`

Took 3 min 21 sec . View more details here
Description: make install test-check-packages-with-kind check-git-clean

`Check`

Took 3 min 58 sec . View more details here
Description: make install test-check-packages-other check-git-clean

`Check`

Took 3 min 52 sec . View more details here
Description: make install test-check-packages-with-custom-agent check-git-clean

`Check`

Took 3 min 52 sec . View more details here
Description: make install test-check-packages-benchmarks check-git-clean

`Check`

Took 3 min 26 sec . View more details here
Description: make install test-build-zip check-git-clean

`Build elastic-package`

Took 3 min 2 sec . View more details here
Description: make PACKAGE_UNDER_TEST=apache test-check-packages-parallel

`Build elastic-package`

Took 3 min 2 sec . View more details here
Description: make PACKAGE_UNDER_TEST=aws test-check-packages-parallel

`Build elastic-package`

Took 2 min 57 sec . View more details here
Description: make PACKAGE_UNDER_TEST=gcp test-check-packages-parallel

`Build elastic-package`

Took 3 min 31 sec . View more details here
Description: make PACKAGE_UNDER_TEST=nginx test-check-packages-parallel

`Build elastic-package`

Took 2 min 55 sec . View more details here
Description: make PACKAGE_UNDER_TEST=sql_input test-check-packages-parallel

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-01-23T14:17:31Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`27/27`)	💚
Files	65.833% (`79/120`)	👎 -0.076
Classes	61.017% (`108/177`)	👎 -0.359
Methods	43.825% (`291/664`)	👎 -4.817
Lines	28.083% (`2908/10355`)	👎 -3.624
Conditionals	100.0% (`0/0`)	💚

internal/profile/_static/kibana_config_default.yml

ruflin

LGTM

internal/profile/_static/kibana_config_8x.yml

Co-authored-by: Nicolas Ruflin <spam@ruflin.com>

jsoriano · 2023-01-23T17:23:21Z

Failures in CI seem related to this change, should this setting be applied only for versions >= 8.7? (We don't have an easy way to do it).

�[36mkibana_1                     |�[0m [2023-01-23T16:12:56.735+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
�[36mkibana_1                     |�[0m [2023-01-23T16:13:10.753+00:00][FATAL][root] Error: [config validation of [xpack.fleet].enableExperimental]: [experimentalDataStreamSettings] is not allowed. Allowed values are: createPackagePolicyMultiPageLayout, packageVerification, showDevtoolsRequest, diagnosticFileUploadEnabled

kpollich · 2023-01-23T19:06:44Z

Failures in CI seem related to this change, should this setting be applied only for versions >= 8.7? (We don't have an easy way to do it).

Yes, this option is only available in 8.7+. Due to the nature of Kibana's config validator, the setting will result in an invalid config for older versions.

ruflin · 2023-01-24T07:58:44Z

@jsoriano @kpollich For our current testing, is there a manual way to overwrite it by setting some environment variables or similar?

jsoriano · 2023-01-24T11:45:34Z

For our current testing, is there a manual way to overwrite it by setting some environment variables or similar?

I don't think so.

Can these flags be changed on runtime through the API?

joshdover · 2023-01-24T13:53:14Z

Kibana supports setting configs via env vars in the Docker images: https://www.elastic.co/guide/en/kibana/current/docker.html#environment-variable-config

Probably something like XPACK_FLEET_ENABLEEXPERIMENTAL=true would work.

I think this isn't automatic though, so there might be some Docker entrypoint script that needs to be updated for the image to support this. Kibana operations team would be the one to ask.

ruflin · 2023-01-24T13:53:24Z

As a manual workaround what I did for now is add xpack.fleet.enableExperimental: ["experimentalDataStreamSettings"] as an entry into .elastic-package/profiles/default/stack/kibana.config.8x.yml and run elastic-package stack up --version=8.7.0-SNAPSHOT -v and it works. This is a manual way to make sure we don't get blocked for testing.

joshdover · 2023-01-24T13:53:57Z

Ah found it, think we'd have to add the configs here: https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker#L277

ruflin · 2023-01-24T13:54:12Z

Can these flags be changed on runtime through the API?

@jsoriano Unfortunately no.

ruflin · 2023-02-01T07:46:18Z

Can we get this moving? Most of these flags were introduced so we can test it with packages. Not having it available in elastic-package is problematic as this is where most of us tests packages.

jsoriano · 2023-02-01T10:13:01Z

Can we get this moving? Most of these flags were introduced so we can test it with packages. Not having it available in elastic-package is problematic as this is where most of us tests packages.

Let me take a quick look to adding a new config file for Kibana 8.7.

jsoriano · 2023-02-01T10:43:29Z

#1128

jsoriano · 2023-02-01T13:45:25Z

Change merged as part of #1128, closing this one.

ruflin · 2023-02-02T07:15:03Z

@jsoriano Thanks!

Enable experimental toggles in Fleet UI from Kibana config

005dc74

kpollich added Team:Fleet Label for the Fleet team Team:Ecosystem Label for the Packages Ecosystem team labels Jan 23, 2023

kpollich requested a review from jsoriano January 23, 2023 13:43

kpollich self-assigned this Jan 23, 2023

kpollich mentioned this pull request Jan 23, 2023

[Fleet] Add feature flag for experimental indexing features elastic/kibana#148317

Closed

4 tasks

jsoriano reviewed Jan 23, 2023

View reviewed changes

internal/profile/_static/kibana_config_default.yml Outdated Show resolved Hide resolved

internal/profile/_static/kibana_config_default.yml Outdated Show resolved Hide resolved

jsoriano requested a review from a team January 23, 2023 15:27

Move feature flag opt-in to kibana_config_8x.yml

d9000c7

ruflin reviewed Jan 23, 2023

View reviewed changes

internal/profile/_static/kibana_config_8x.yml Outdated Show resolved Hide resolved

Update internal/profile/_static/kibana_config_8x.yml

2370594

Co-authored-by: Nicolas Ruflin <spam@ruflin.com>

jsoriano mentioned this pull request Feb 1, 2023

Add variant for Kibana >= 8.7.0 with experimental toggles enabled #1128

Merged

jsoriano closed this Feb 1, 2023

Conversation

kpollich commented Jan 23, 2023

Uh oh!

elasticmachine commented Jan 23, 2023

Uh oh!

elasticmachine commented Jan 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Build stats

Test stats 🧪

Steps errors

Check

Check

Check

Check

Check

Build elastic-package

Build elastic-package

Build elastic-package

Build elastic-package

Build elastic-package

🤖 GitHub comments

Uh oh!

elasticmachine commented Jan 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

Uh oh!

Uh oh!

ruflin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jsoriano commented Jan 23, 2023

Uh oh!

kpollich commented Jan 23, 2023

Uh oh!

ruflin commented Jan 24, 2023

Uh oh!

jsoriano commented Jan 24, 2023

Uh oh!

joshdover commented Jan 24, 2023

Uh oh!

ruflin commented Jan 24, 2023

Uh oh!

joshdover commented Jan 24, 2023

Uh oh!

ruflin commented Jan 24, 2023

Uh oh!

ruflin commented Feb 1, 2023

Uh oh!

jsoriano commented Feb 1, 2023

Uh oh!

jsoriano commented Feb 1, 2023

Uh oh!

jsoriano commented Feb 1, 2023

Uh oh!

ruflin commented Feb 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elasticmachine commented Jan 23, 2023 •

edited

Loading

`Check`

`Check`

`Check`

`Check`

`Check`

`Build elastic-package`

`Build elastic-package`

`Build elastic-package`

`Build elastic-package`

`Build elastic-package`

elasticmachine commented Jan 23, 2023 •

edited

Loading