Skip to content

[7.9] Add datastream.* to mappings#60592

Merged
jpountz merged 2 commits intoelastic:7.9from
ruflin:update-mappings
Aug 4, 2020
Merged

[7.9] Add datastream.* to mappings#60592
jpountz merged 2 commits intoelastic:7.9from
ruflin:update-mappings

Conversation

@ruflin
Copy link
Copy Markdown
Contributor

@ruflin ruflin commented Aug 3, 2020

The current Elastic Agent ships data to dataset.*. We plant to rename this field to datastream.type, datastream.dataset, datastream.namespace instead. To make the template forward compatible, we add these fields to 7.9 and newer.

This will need forward porting to 7.x

@ruflin ruflin requested review from dakrone and jpountz August 3, 2020 13:06
@ruflin ruflin self-assigned this Aug 3, 2020
@ruflin ruflin added the v7.9.0 label Aug 3, 2020
"properties": {
"type": {
"type": "constant_keyword",
"value": "logs"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"value": "logs"
"value": "metrics"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:-( fixed.

@jpountz jpountz added :Data Management/Indices APIs DO NOT USE. Use ":Distributed/Indices APIs" or ":StorageEngine/Templates" instead. >enhancement labels Aug 3, 2020
@jpountz
Copy link
Copy Markdown
Contributor

jpountz commented Aug 3, 2020

I wonder if this actually requires to be merged in 7.9 (as opposed to 7.10), given that Elasticsearch must be upgraded before Agent?

@ruflin
Copy link
Copy Markdown
Contributor Author

ruflin commented Aug 3, 2020

Yes, as Elasticsearch will not update the template if a template is already there and we plan in 7.9 to already ship datastream.* values from the Agent.

@jpountz
Copy link
Copy Markdown
Contributor

jpountz commented Aug 3, 2020

Thanks, I had misunderstood the plan.

@ruflin ruflin changed the title Add datastream.* to mappings [7.9] Add datastream.* to mappings Aug 3, 2020
@ruflin ruflin marked this pull request as ready for review August 3, 2020 14:54
@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

@elasticmachine elasticmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Aug 3, 2020
@ruflin ruflin requested a review from jpountz August 3, 2020 14:54
Copy link
Copy Markdown
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but @ruflin I think this needs additional labels for it being forward-ported? Usually we start with master and backport from there

@jpountz
Copy link
Copy Markdown
Contributor

jpountz commented Aug 4, 2020

I opened #60638 and will treat this PR as a backport PR.

@jpountz jpountz added backport and removed :Data Management/Indices APIs DO NOT USE. Use ":Distributed/Indices APIs" or ":StorageEngine/Templates" instead. >enhancement Team:Data Management (obsolete) DO NOT USE. This team no longer exists. v7.9.0 labels Aug 4, 2020
@jpountz jpountz merged commit e099b10 into elastic:7.9 Aug 4, 2020
ruflin added a commit to ruflin/integrations that referenced this pull request Aug 4, 2020
It is planned to move from dataset.* to datastream.*. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream.* fields. Because of this, these should also be added to the mappings of the packages.
ruflin added a commit to elastic/integrations that referenced this pull request Aug 4, 2020
It is planned to move from dataset.* to datastream.*. To make the path easier to 7.9 Elasticsearch (elastic/elasticsearch#60592) and the Elastic Agent already ship with the datastream.* fields. Because of this, these should also be added to the mappings of the packages.

The agent will ship datastream.* fields. Because of this, the import scripts can be adjusted to only use these fields.

This PR doese not rename any Golang variables. This should be done in a follow up and in sync with potential changes to the registry.
@ruflin ruflin deleted the update-mappings branch August 4, 2020 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants