Skip to content

add flags to give permissions to write to any dataset and namespace#157897

Merged
felixbarny merged 3 commits intoelastic:mainfrom
gsantoro:feature/routing_permissions
May 22, 2023
Merged

add flags to give permissions to write to any dataset and namespace#157897
felixbarny merged 3 commits intoelastic:mainfrom
gsantoro:feature/routing_permissions

Conversation

@gsantoro
Copy link
Copy Markdown
Contributor

@gsantoro gsantoro commented May 16, 2023

Summary

Add flags to give permissions to reroute events from an input package to other datastreams with the same type but different dataset and namespace.

elasticsearch.dynamic_dataset: true
elasticsearch.dynamic_namespace: true,

Closes: #elastic/integrations#5989

Checklist

Delete any items that are not applicable to this PR.

@gsantoro gsantoro added backport:skip This PR does not require backporting Team:Fleet Team label for Observability Data Collection Fleet team release_note:feature Makes this part of the condensed release notes v8.9.0 labels May 16, 2023
@gsantoro gsantoro self-assigned this May 16, 2023
@gsantoro gsantoro requested a review from a team as a code owner May 16, 2023 16:01
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/fleet (Team:Fleet)

@ghost
Copy link
Copy Markdown

ghost commented May 16, 2023

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@gsantoro gsantoro requested a review from felixbarny May 16, 2023 16:02
@gsantoro
Copy link
Copy Markdown
Contributor Author

I have tested the behaviour with the custom logs integration (as an example of an input package) and with some sample nginx access logs being redirected to the logs-test-test datastream.

I created the following custom pipeline logs-generic@custom that is already referenced by the custom-logs default pipeline

curl --location --request PUT 'https://localhost:9200/_ingest/pipeline/logs-generic@custom' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' \
--data '{
    "description": "My optional pipeline description",
    "processors": [
        {
            "set": {
                "description": "My optional processor description",
                "field": "my-long-field",
                "value": 10
            }
        },
        {
            "reroute": {
                "dataset": "test",
                "namespace": "test"
            }
        }
    ]
}'

@gsantoro
Copy link
Copy Markdown
Contributor Author

Closes: #elastic/integrations#5989

@gsantoro gsantoro requested a review from kpollich May 16, 2023 16:10
@felixbarny
Copy link
Copy Markdown
Member

Needs some tests, otherwise LGTM

@felixbarny felixbarny linked an issue May 17, 2023 that may be closed by this pull request
@kibana-ci
Copy link
Copy Markdown

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
fleet 129.4KB 129.5KB +117.0B
Unknown metric groups

ESLint disabled line counts

id before after diff
enterpriseSearch 19 21 +2
securitySolution 400 404 +4
total +6

Total ESLint disabled count

id before after diff
enterpriseSearch 20 22 +2
securitySolution 480 484 +4
total +6

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @gsantoro

@juliaElastic juliaElastic removed the request for review from kpollich May 22, 2023 11:30
Copy link
Copy Markdown
Contributor

@juliaElastic juliaElastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@felixbarny felixbarny merged commit b05ad20 into elastic:main May 22, 2023
@felixbarny
Copy link
Copy Markdown
Member

@juliaElastic are the API keys expected to be automatically updated and propagated to the agent or do users need to re-install the integration?

@juliaElastic
Copy link
Copy Markdown
Contributor

juliaElastic commented May 22, 2023

@felixbarny Do you mean if we introduce these changes in a package upgrade? I'm not sure, can we test it?

@joshdover
Copy link
Copy Markdown
Contributor

I would expect that it should be applied automatically to agents when the integration policy is upgraded, after some delay as the change rolls out.

@felixbarny
Copy link
Copy Markdown
Member

I guess there are two cases:

  1. When we add the elasticsearch.dynamic_dataset/elasticsearch.dynamic_namespace flag to an integration and publish a new version of the integration
  2. This change implicitly adds the flags to all input-type integrations so there won't be a new version for these integrations. For users that have already installed an input-type integration, will the change be effective once users are upgrading Kibana to 8.9.0 or do they need to update Kibana and also re-install the integration or wait for the next update of the integration?

@gsantoro gsantoro deleted the feature/routing_permissions branch May 24, 2023 08:21
delanni pushed a commit to delanni/kibana that referenced this pull request May 25, 2023
…lastic#157897)

Add flags to give permissions to reroute events from an input package to
other datastreams with the same type but different dataset and
namespace.

```
elasticsearch.dynamic_dataset: true
elasticsearch.dynamic_namespace: true,
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:feature Makes this part of the condensed release notes Team:Fleet Team label for Observability Data Collection Fleet team v8.9.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow routing for input packages

7 participants