[Data Forge] Add service.logs dataset as a data stream#188786
Merged
simianhacker merged 4 commits intoelastic:mainfrom Jul 23, 2024
Merged
[Data Forge] Add service.logs dataset as a data stream#188786simianhacker merged 4 commits intoelastic:mainfrom
service.logs dataset as a data stream#188786simianhacker merged 4 commits intoelastic:mainfrom
Conversation
This PR adds the `service.logs` dataset to Data Forge. This dataset is
different because I wanted to create a fully compliant data stream. This
change also includes changes to the name of the default ingest pipeline
to `logs@custom` to work with the `logs-*-*` component templates and ingest pipelines. If
a document has `data_stream.dataset` it will now be routed to
`logs-${doc.data_stream.dataset}-default`. If the document has
`data_stream.type`, `data_stream.dataset`, and `data_stream.namespace`
it will be index to `{type}-{dataset}-{namespace}` following the default
data stream conventions.
Because I've changed the name of the ingest pipeline, I also had to
update the index templates for the other datasets.
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
kdelemme
approved these changes
Jul 22, 2024
Member
Author
|
@elasticmachine merge upstream |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Public APIs missing comments
History
To update your PR or re-run it, just comment with: |
TinLe
added a commit
to TinLe/kibana
that referenced
this pull request
Jul 30, 2024
* master: (3487 commits) `BedrockChat` & `GeminiChat` (elastic#186809) [ResponseOps] log error when ES Query rules find docs out of time range (elastic#186332) skip flaky suite (elastic#188997) [Security solution][Alert Details] Enable preview feature flag and cypress tests (elastic#188580) [EuiProviders] Warn Developer if EuiProvider is missing (elastic#184608) [Security Solution ] Fixes Timeline infinite loading bug (elastic#188943) Improve SearchSource SearchRequest type (elastic#186862) Deprecate Search Sessions config (elastic#188037) [Synthetics] Add missing monitorType and tag info in cards !! (elastic#188824) [Console Monaco] Resolve uncaught error from tokenizer (elastic#188746) [Data Forge] Add `service.logs` dataset as a data stream (elastic#188786) [Console] Fix failing bulk requests (elastic#188552) Update dependency terser to ^5.31.2 (main) (elastic#188528) [APM][ECO] Telemetry (elastic#188627) [Fleet] Fix uninstall package validation accross space (elastic#188749) Update warning on `xpack.fleet.enableExperimental` (elastic#188917) [DOCS][Cases] Automate more screenshots for cases (elastic#188697) [Fleet] Fix get one agent when feature flag disabled (elastic#188953) chore(investigate): Add investigate-app plugin from poc (elastic#188122) [Monaco Editor] Add Search functionality (elastic#188337) ...
TinLe
added a commit
to TinLe/kibana
that referenced
this pull request
Jul 30, 2024
* master: (2400 commits) `BedrockChat` & `GeminiChat` (elastic#186809) [ResponseOps] log error when ES Query rules find docs out of time range (elastic#186332) skip flaky suite (elastic#188997) [Security solution][Alert Details] Enable preview feature flag and cypress tests (elastic#188580) [EuiProviders] Warn Developer if EuiProvider is missing (elastic#184608) [Security Solution ] Fixes Timeline infinite loading bug (elastic#188943) Improve SearchSource SearchRequest type (elastic#186862) Deprecate Search Sessions config (elastic#188037) [Synthetics] Add missing monitorType and tag info in cards !! (elastic#188824) [Console Monaco] Resolve uncaught error from tokenizer (elastic#188746) [Data Forge] Add `service.logs` dataset as a data stream (elastic#188786) [Console] Fix failing bulk requests (elastic#188552) Update dependency terser to ^5.31.2 (main) (elastic#188528) [APM][ECO] Telemetry (elastic#188627) [Fleet] Fix uninstall package validation accross space (elastic#188749) Update warning on `xpack.fleet.enableExperimental` (elastic#188917) [DOCS][Cases] Automate more screenshots for cases (elastic#188697) [Fleet] Fix get one agent when feature flag disabled (elastic#188953) chore(investigate): Add investigate-app plugin from poc (elastic#188122) [Monaco Editor] Add Search functionality (elastic#188337) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the
service.logsdataset to Data Forge . The EEM project needs this dataset to test the default service logs entity definition. This dataset is different because I wanted to create a fully compliant data stream. This change also includes changes to the name of the default ingest pipeline tologs@customto work with thelogs-*-*component templates and ingest pipelines. If a document hasdata_stream.datasetit will now be routed tologs-${doc.data_stream.dataset}-default. If the document hasdata_stream.type,data_stream.dataset, anddata_stream.namespaceit will be index to{type}-{dataset}-{namespace}following the default data stream conventions.Because I've changed the name of the ingest pipeline, I also had to update the index templates for the other datasets.
Testing
Use the following YAML:
Click on "Logs" under "Observability", you should see something like:
To check the other data source, change
datasettofake_stack,fake_logs,fake_hostsand checkevent.ingestedis set on the documents; none of these show up in the "Logs Explorer", you'll have to check them out seperately via "Dev Console".