Skip to content

Add time series related information to get data stream API#86395

Merged
martijnvg merged 6 commits intoelastic:masterfrom
martijnvg:get_data_stream_api_add_time_series_section
May 9, 2022
Merged

Add time series related information to get data stream API#86395
martijnvg merged 6 commits intoelastic:masterfrom
martijnvg:get_data_stream_api_add_time_series_section

Conversation

@martijnvg
Copy link
Copy Markdown
Member

@martijnvg martijnvg commented May 3, 2022

In case if a data stream is a time series data stream then include time series information.
This includes the continuous temporal ranges a time series data stream encapsulates.
This is computed based on combing the index.time_series.start_time and index.time_series.end_time
ranges of all backing indices of a time series data stream

Closes #83518

@martijnvg
Add time series related information to get data stream API
1a6ce64 
In case if a data stream is a time series data stream then include time series information.
This includes the continuous temporal ranges a time series data stream encapsulates.
This is computed based on combing the index.time_series.start_time and index.time_series.end_time
ranges of all backing indices of a time series data stream

Closes elastic#83518
@martijnvg martijnvg added the :StorageEngine/Data streams Data streams and their lifecycles label May 3, 2022
@martijnvg martijnvg marked this pull request as ready for review May 4, 2022 12:15
@elasticmachine elasticmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label May 4, 2022
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented May 4, 2022

Pinging @elastic/es-data-management (Team:Data Management)

@martijnvg
Copy link
Copy Markdown
Member Author

martijnvg commented May 4, 2022 •
edited
Loading

This change add time_series snippet to get data streams api response if a data stream is a time series data stream.
Example GET /_data_stream run:

{
    "data_streams": [
        {
            "name": "k8s",
            "timestamp_field": {
                "name": "@timestamp"
            },
            "indices": [
                {
                    "index_name": ".ds-k8s-2022.05.04-000001",
                    "index_uuid": "1C1g7at-RW6vz0kHEGx1ZA"
                },
                {
                    "index_name": ".ds-k8s-2022.05.04-000002",
                    "index_uuid": "nuh8WMrLROmrzkRPb1fu1w"
                }
            ],
            "generation": 2,
            "status": "GREEN",
            "template": "k8s-template",
            "ilm_policy": "7-days-1gb-rollover",
            "hidden": false,
            "system": false,
            "allow_custom_routing": false,
            "replicated": false,
            "time_series": {
                "temporal_ranges": [
                    {
                        "start": "2022-05-04T12:14:53.000Z",
                        "end": "2022-05-04T12:43:26.000Z"
                    }
                ]
            }
        }
    ]
}

@sethmlarson sethmlarson added the Team:Clients Meta label for clients team label May 4, 2022
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented May 4, 2022

Pinging @elastic/clients-team (Team:Clients)

@martijnvg martijnvg requested a review from nik9000 May 4, 2022 12:43
sethmlarson
sethmlarson approved these changes May 4, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@sethmlarson sethmlarson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from an API perspective

@martijnvg
Copy link
Copy Markdown
Member Author

martijnvg commented May 4, 2022

LGTM from an API perspective

Thanks @sethmlarson for reviewing!

nik9000
nik9000 reviewed May 4, 2022
View reviewed changes
...treams/src/main/java/org/elasticsearch/datastreams/action/GetDataStreamsTransportAction.java
if (dataStream.getIndexMode() == IndexMode.TIME_SERIES) {
List<Tuple<Instant, Instant>> ranges = new ArrayList<>();
Tuple<Instant, Instant> current = null;
for (Index index : dataStream.getIndices()) {
Copy link
Copy Markdown
Member

@nik9000 nik9000 May 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you fail if the indices aren't in time series order? Or warn or something? Just out of paranoia.

Copy link
Copy Markdown
Member

@nik9000 nik9000 May 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or sort them by start time.

Copy link
Copy Markdown
Member Author

@martijnvg martijnvg May 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The indices are in order in which they are created. (new data stream, rollover, rollover again -> 3 backing indices that are in order). So the start / end times should be in order.

I will add an assertion and ensure a warning log gets printed (if this ever should occur in production). Which I don't think should be the case.

@martijnvg martijnvg requested a review from nik9000 May 4, 2022 18:19
@martijnvg
Copy link
Copy Markdown
Member Author

martijnvg commented May 9, 2022

@elasticmachine update branch

@martijnvg martijnvg merged commit f14da48 into elastic:master May 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nik9000 nik9000 nik9000 approved these changes

+1 more reviewer

@sethmlarson sethmlarson sethmlarson approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

>non-issue :StorageEngine/Data streams Data streams and their lifecycles Team:Clients Meta label for clients team Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v8.3.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Include time series temporal ranges to get data stream api for data streams in time series index mode

5 participants

@martijnvg @elasticmachine @nik9000 @sethmlarson @elasticsearchmachine