Skip to content

Hidden data streams#63987

Merged
probakowski merged 33 commits intoelastic:masterfrom
probakowski:hidden-data-streams
Oct 30, 2020
Merged

Hidden data streams#63987
probakowski merged 33 commits intoelastic:masterfrom
probakowski:hidden-data-streams

Conversation

@probakowski
Copy link
Copy Markdown
Contributor

@probakowski probakowski commented Oct 21, 2020

This change adds possibility to mark data stream as hidden. To use it you set usual "index.hidden":true setting in data stream template.
This change also relaxes naming convention for data streams - data stream can start with . but can't start with backing index prefix (.ds- currently) to avoid clashes.

Closes #63889

martijnvg
martijnvg reviewed Oct 21, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation looks good at first glance. Can you also add a rest test for hiding a data stream? iirc some changes are also needed in security.

danhermann
danhermann reviewed Oct 21, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@danhermann danhermann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this change the behavior of the GetDataStream API? I think we'll need to add the expand_wildcards option to that API for consistency in the handling of hidden data streams.

@probakowski probakowski marked this pull request as ready for review October 22, 2020 21:48
@probakowski probakowski added :StorageEngine/Data streams Data streams and their lifecycles v7.11.0 v8.0.0 labels Oct 22, 2020
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Oct 22, 2020

Pinging @elastic/es-core-features (:Core/Features/Data streams)

@elasticmachine elasticmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Oct 22, 2020
@probakowski
Copy link
Copy Markdown
Contributor Author

probakowski commented Oct 22, 2020 •
edited
Loading

Thanks for review, could I ask for another round?
@martijnvg do you remember what it was about security? I've tested it with security on locally and used RBAC as well, it all worked correctly. I've added REST tests.
@danhermann for now I've updated GetDataStream API to show all data streams, the same goes for DeleteDataStream API. I think adding expand_wildcards param is good idea but I think it can be done in separate issue, WDYT?

@probakowski
Copy link
Copy Markdown
Contributor Author

probakowski commented Oct 22, 2020

@elasticmachine run elasticsearch-ci/1

@probakowski
Copy link
Copy Markdown
Contributor Author

probakowski commented Oct 30, 2020

@elasticmachine run elasticsearch-ci/1 (known unrelated failure)

@probakowski probakowski requested a review from martijnvg October 30, 2020 09:54
martijnvg
martijnvg approved these changes Oct 30, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@probakowski probakowski merged commit 0e81fc6 into elastic:master Oct 30, 2020
@probakowski probakowski deleted the hidden-data-streams branch October 30, 2020 10:42
@probakowski probakowski mentioned this pull request Oct 30, 2020
Merged
probakowski added a commit to probakowski/elasticsearch that referenced this pull request Oct 30, 2020
@probakowski
Hidden data streams (elastic#63987)
eaac4da 
* Hidden data streams

* whitespace reverted

* stricter ds name

* Revert "stricter ds name"

This reverts commit 100dba5.

* String.format removed

* fix test

* fix GetDataStream action

* fix test

* fix test

* rest test

* rest test

* spotless

* tests

* Delete a.json

* added expand_wildcards for GetDataStream and DeleteDataStream

* unused imports

* add hidden setting to data stream template

* fix expand_wildcards

* spotless

* fix compilation

* unused import

* yaml test

* fix test

* fix cleanup

* review

* compilation fix

* fix javadoc

* fix javadoc
@webmat webmat mentioned this pull request Oct 30, 2020
Merged
probakowski added a commit that referenced this pull request Nov 2, 2020
@probakowski
[7.x] Hidden data streams (#63987) (#64402)
a06b65c 
Backports the following commits to 7.x:

Hidden data streams (#63987)
@imotov imotov mentioned this pull request Nov 5, 2020
Closed
@cjcenizal cjcenizal mentioned this pull request Nov 18, 2020
Closed
@cjcenizal
Copy link
Copy Markdown
Contributor

cjcenizal commented Nov 18, 2020

Opened elastic/kibana#83709 to track the need to reflect these changes in the UI.

@yuliacech yuliacech mentioned this pull request Nov 20, 2020
Closed
@jrodewig jrodewig mentioned this pull request Dec 3, 2020
Merged
This was referenced Dec 7, 2020
Merged
Merged
Merged
Merged
@yuliacech yuliacech mentioned this pull request Dec 10, 2020
Closed
@stevejgordon stevejgordon mentioned this pull request Dec 17, 2020
Closed
@alisonelizabeth alisonelizabeth mentioned this pull request Jan 12, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@martijnvg martijnvg martijnvg approved these changes

@danhermann danhermann Awaiting requested review from danhermann

Assignees

No one assigned

Labels

>enhancement :StorageEngine/Data streams Data streams and their lifecycles Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v7.11.0 v8.0.0-alpha1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support hidden data streams

6 participants

@probakowski @elasticmachine @danhermann @cjcenizal @martijnvg @jakelandis