elastic/elasticsearch#63987 added the ability to specify that data streams should be hidden, shipping with 7.11. We need to update the data streams tab of Index Management to support retrieval and display of hidden data streams similarly to how hidden indices are retrieved and displayed in the indices tab.

Note: The hidden data streams feature is targeted at consumers in the Stack (elastic/elasticsearch#63889), so we won't update the Index Templates wizard with the option for defining an index template that will generate a hidden data stream.

Changes

API changes

We need to update the data streams API to retrieve all data streams including hidden ones: GET _data_stream?expand_wildcards=hidden,all.

UI changes

We need to add a toggle to the UI to allow users to reveal hidden data streams.

Also, we need to adjust the logic that renders the empty prompt. Currently, the table will be hidden and an empty prompt is rendered if the API response contains zero data streams. After we change the API to return both hidden and non-hidden data streams, the response will always contain data streams. So we'll want to update the rendering logic to render the empty prompt if there are zero non-hidden data streams and if the "Show hidden" toggle is disabled. We'll also want to ensure this toggle is always rendered, so users can enable it.

Cross-linking changes

We added logic in #66715 and #67940 to allow the user to click into a data stream's indices. Doing so will navigate from the data streams tab to the indices tab, set a filter to show only that data stream's indices, and enable the "Show hidden" toggle if the indices are known to be hidden.

We need to implement similar functionality for when the user clicks into an index's data stream. Currently, this will navigate from the indices tab to the data streams tab and set a filter to show only that index's data stream. We need to add logic for enabling the "Show hidden" toggle if that data stream is known to be hidden.

Testing

These requests will generate a hidden data stream called test.

PUT _index_template/test
{
  "template": {},
  "index_patterns": [
    "test"
  ],
  "data_stream": {
    "hidden": true   
  }
}

POST test/_doc
{
  "@timestamp": "2020-01-27"
}

# Retrieve the hidden data streams
GET _data_stream?expand_wildcards=hidden,all