Use policy template name as default dataset for input packages#3037
Merged
jsoriano merged 8 commits intoelastic:mainfrom Nov 18, 2025
Merged
Use policy template name as default dataset for input packages#3037jsoriano merged 8 commits intoelastic:mainfrom
jsoriano merged 8 commits intoelastic:mainfrom
Conversation
Default dataset in integration packages is formed as <package name>.<data stream name>, both in elastic-package and Fleet. In the case of input packages, elastic-package was forming it as something like <package name>.<policy template name>, but Fleet uses only the policy template name. Do the same as Fleet for input packages.
Member
Author
|
test integrations |
|
Created or updated PR in integrations repository to test this version. Check elastic/integrations#15808 |
Member
Author
|
Failures in |
Member
Author
These packages set a default value for the |
jamesagarside
added a commit
to elastic/integrations
that referenced
this pull request
Nov 12, 2025
- Add use_logs_stream configuration option to enable sending data to the unified "logs" data stream - Update tcp.yml.hbs and udp.yml.hbs templates to conditionally route to logs index - Bump TCP version to 2.1.0 and UDP version to 2.3.0 - Follows the same pattern as filestream package This feature allows users to send all ingested data to the "logs" data stream instead of custom datasets when the "Write to logs streams" option is enabled in output settings. Note: System tests for logs-stream variants will fail due to a known elastic-package bug being fixed in elastic/elastic-package#3037. Policy tests pass, confirming the configuration is correct. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
jamesagarside
added a commit
to jamesagarside/integrations
that referenced
this pull request
Nov 12, 2025
- Add use_logs_stream configuration option to enable sending data to the unified "logs" data stream - Update tcp.yml.hbs and udp.yml.hbs templates to conditionally route to logs index - Bump TCP version to 2.1.0 and UDP version to 2.3.0 - Follows the same pattern as filestream package This feature allows users to send all ingested data to the "logs" data stream instead of custom datasets when the "Write to logs streams" option is enabled in output settings. Note: System tests for logs-stream variants will fail due to a known elastic-package bug being fixed in elastic/elastic-package#3037. Policy tests pass, confirming the configuration is correct. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
jamesagarside
added a commit
to jamesagarside/integrations
that referenced
this pull request
Nov 12, 2025
- Add use_logs_stream configuration option to enable sending data to the unified "logs" data stream - Update tcp.yml.hbs and udp.yml.hbs templates to conditionally route to logs index - Bump TCP version to 2.1.0 and UDP version to 2.3.0 - Follows the same pattern as filestream package This feature allows users to send all ingested data to the "logs" data stream instead of custom datasets when the "Write to logs streams" option is enabled in output settings. Note: System tests for logs-stream variants will fail due to a known elastic-package bug being fixed in elastic/elastic-package#3037. Policy tests pass, confirming the configuration is correct. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Member
Author
|
test integrations |
|
Created or updated PR in integrations repository to test this version. Check elastic/integrations#15808 |
Collaborator
💔 Build Failed
Failed CI StepsHistory
cc @jsoriano |
| if expectedDatasets == nil { | ||
| if len(expectedDatasets) == 0 { | ||
| // get dataset directly from package policy added when preparing the scenario | ||
| expectedDataset := scenario.kibanaDataStream.Inputs[0].Streams[0].DataStream.Dataset |
Contributor
There was a problem hiding this comment.
could this piece of code be replaced so it uses getExpectedDatasetForTest ?? does it make sense to add some unit test for that function to check its behaving as expected?
Member
Author
There was a problem hiding this comment.
could this piece of code be replaced so it uses
getExpectedDatasetForTest??
Good point, yes, we can reuse it, we need to keep the otel-specific code here because in some places we want it and in others we don't.
Changed in bc4bf90, also added another case for the integration tests.
does it make sense to add some unit test for that function to check its behaving as expected?
👍 added in c6d3eb2.
mrodm
approved these changes
Nov 18, 2025
Comment on lines
+2031
to
+2035
| // This dataset is the one Fleet uses to identify the stream, | ||
| // it must be <package name>.<policy template name>. This is not | ||
| // the same as the dataset used for the index template, configured | ||
| // with vars below. | ||
| Dataset: fmt.Sprintf("%s.%s", pkg.Name, policyTemplate.Name), |
Contributor
There was a problem hiding this comment.
Thanks for adding this explanation!
teresaromero
approved these changes
Nov 18, 2025
taylor-swanson
pushed a commit
to elastic/integrations
that referenced
this pull request
Dec 2, 2025
This feature allows users to send all ingested data to the "logs" data stream instead of custom datasets when the "Write to logs streams" option is enabled in output settings. Note: System tests for logs-stream variants will fail due to a known elastic-package bug being fixed in elastic/elastic-package#3037. Policy tests pass, confirming the configuration is correct. - Add use_logs_stream configuration option to enable sending data to the unified "logs" data stream - Update tcp.yml.hbs and udp.yml.hbs templates to conditionally route to logs index - Bump TCP version to 2.1.0 and UDP version to 2.3.0 - Follows the same pattern as filestream package - Changed version constraint from "^8.13.0 || ^9.0.0" to "^9.2.0" for both TCP and UDP packages since Elastic Streams was introduced in 9.2.0. --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #3036.
Default dataset in integration packages is formed as
<package name>.<data stream name>, both in elastic-package and Fleet.In the case of input packages, elastic-package was forming it as something like
<package name>.<policy template name>, but Fleet uses only the policy template name.Do the same as Fleet for input packages.