@elastic/ingest-management Can you think of any other places where we query / agg data based on the dataset.* fields?

A note on testing these changes as it requires building ES from source:

• Check out elasticsearch from 7.9 branch
• Run ./gradlew run
• Adjust config of Kibana to use the following:

elasticsearch.username: "elastic-admin"
elasticsearch.password: "elastic-password"

• Checkout out Beats repo 7.9 and build agent from source
• Run PLATFORMS=darwin mage package (assuming you use darwin)
• Go to build/platforms, extract the binary and use it.

The package registry should already contain the most recent packages with the changed format (releasing now).

I've created #74257 to discuss what needs to be done for this change to work.

Did a simple smoke test using this PR, 7.9 elasticsearch from source, 7.9 agent from source, with default config (just system integration). Datasets page, metrics dashboard, and ES data streams look good:

@jen-huang @skh I updated this PR with the most recent data_stream renaming. As soon as PR's on ES and Agent side are merged, I'll start testing this locally.

We will also need to forward port this change.

We will also need to forward port this change.

Is there a reason not to use the normal merging strategy (master first, backport to 7.x and 7.9 as needed)?

@skh Main reason is that on 7.9 we are bit under time pressure and I would like to have this change in very soon so we can start testing.

I suggest after we get this into 7.9, we open master and then backport to 7.x

under time pressure

The difference between the normal workflow and merging to 7.9 first is once CI run / about 2 hours. If we can't afford these two hours, are we positively sure we should be making these changes at all?

Yes, changes are already in Elasticsearch: elastic/elasticsearch#60715

@skh As an alternative, could you help me open a PR against master directly with the backports before it is merged?

@ruflin ok to close this in favour of #74342 ?

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: cbe125c

Build metrics

✅ unchanged

History

• 💚 Build #66582 succeeded cbe125c
• 💚 Build #66564 succeeded cbe125c
• 💛 Build #66501 was flaky fa2b5db
• 💚 Build #66490 succeeded 0a695b5
• 💔 Build #66482 failed 2c9c374

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Closing in favor of #74342 (already merged and backported).