[winlog] Convert to an input package. by marc-gr · Pull Request #8010 · elastic/integrations

marc-gr · 2023-09-28T14:33:17Z

What does this PR do?

Converts winlog package from an integration to an input type package.

Removes support for Splunk data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Related issues

Closes Move winlog package to type: input #7820

Screenshots

elasticmachine · 2023-09-28T14:37:56Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-25T09:50:25.529+0000
Duration: 14 min 11 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-09-28T14:51:05Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

packages/winlog/manifest.yml

andrewkroh

I want you to be aware of https://github.com/elastic/fleet-winlog-7x-policies. It was created to support a user that needed edge processing (until LS can run Fleet integration pipelines). That project should be updated to ensure that the package policies can continue to be installed.

packages/winlog/changelog.yml

marc-gr · 2023-10-24T14:19:29Z

I want you to be aware of https://github.com/elastic/fleet-winlog-7x-policies. It was created to support a user that needed edge processing (until LS can run Fleet integration pipelines). That project should be updated to ensure that the package policies can continue to be installed.

Added a PR to that repo that can be merged after this one 👍

andrewkroh

Have you performed an upgrade test where you have winlog 1.20.0 installed, and then upgrade to this new version and everything keeps functioning?

andrewkroh · 2023-10-24T19:14:14Z

packages/winlog/fields/base-fields.yml

Can we change the type of event.module to keyword? I have heard from users that constant_keyword is a problem for uses cases where this input is used to ingest data from custom channels, and they want to make the data look like our other integrations (e.g. event.module: security).

I added dynamic ecs mapping, that should take care of this 👍

marc-gr · 2023-10-25T09:51:05Z

Have you performed an upgrade test where you have winlog 1.20.0 installed, and then upgrade to this new version and everything keeps functioning?

Yes, tried an upgrade and it kept working normally

elasticmachine · 2023-10-30T12:29:10Z

Package winlog - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=winlog

zedtran · 2024-04-30T23:14:52Z

Full disclosure: I detail this further in the official Elastic support portal under case #01610754.

Noticing an issue on a fresh stack version 8.12.1 install where the create fleet package policy API for the winlog integration fails.

Data stream backing index template "logs-winlog.winlog", an ingest pipeline, and component templates "logs-winlog.winlog@package" and "logs-winlog.winlog@custom" are not loaded with an apparent Kibana log message error which reads:

[2024-04-30T20:34:20.445+00:00][ERROR][plugins.fleet] Error: Stream template not found, unable to find dataset winlog.winlog
    at _compilePackageStream (/usr/share/kibana/node_modules/@kbn/fleet-plugin/server/services/package_policy.js:1442:11)
    at /usr/share/kibana/node_modules/@kbn/fleet-plugin/server/services/package_policy.js:1391:55
    at Array.map (<anonymous>)
    at _compilePackageStreams (/usr/share/kibana/node_modules/@kbn/fleet-plugin/server/services/package_policy.js:1391:41)
    at /usr/share/kibana/node_modules/@kbn/fleet-plugin/server/services/package_policy.js:1356:35
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 0)

Did a manual upload of winlog version 2.1.1 where you can see the .kibana_ingest document referenced by _id: epm-packages:winlog appears to be missing objects/refs typically listed in _source['epm-packages]['installed_es'].

GET .kibana_ingest/_search?q=epm-packages.name:winlog
{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.3862942,
    "hits": [
      {
        "_index": ".kibana_ingest_8.12.1_001",
        "_id": "epm-packages:winlog",
        "_score": 1.3862942,
        "_source": {
          "epm-packages": {
            "installed_kibana": [],
            "installed_kibana_space_id": "default",
            "installed_es": [],
            "package_assets": [
              {
                "id": "293a22b2-8e4e-5c7d-8249-6b32a38a651b",
                "type": "epm-packages-assets"
              },
              {
                "id": "2367f0f5-ce06-5e20-b9dc-75f5807da180",
                "type": "epm-packages-assets"
              },
              {
                "id": "cc49d170-a6c9-56b3-a40f-fa662f700661",
                "type": "epm-packages-assets"
              },
              {
                "id": "c9576881-c05a-5c54-99aa-40f2c0fefd7d",
                "type": "epm-packages-assets"
              },
              {
                "id": "b3de468f-d2e5-5ab6-9e95-ade91d0371e4",
                "type": "epm-packages-assets"
              },
              {
                "id": "d8fb4d51-b5ff-5446-8a75-d9da1ecde649",
                "type": "epm-packages-assets"
              },
              {
                "id": "fec7a0e7-5d0a-5ad0-bad3-e6bddc286ccd",
                "type": "epm-packages-assets"
              },
              {
                "id": "db6a867f-8dcc-5bac-8289-c20ce3b8bb9f",
                "type": "epm-packages-assets"
              },
              {
                "id": "eae37891-e71b-5bad-b369-c717ba725f92",
                "type": "epm-packages-assets"
              },
              {
                "id": "ee1dd87b-e640-56d7-a149-7503868a51b3",
                "type": "epm-packages-assets"
              },
              {
                "id": "f51e6b17-6813-576d-bbc0-5aa4d35defd7",
                "type": "epm-packages-assets"
              }
            ],
            "es_index_patterns": {},
            "name": "winlog",
            "version": "2.1.1",
            "install_version": "2.1.1",
            "install_status": "installed",
            "install_started_at": "2024-04-30T19:26:54.301Z",
            "install_source": "upload",
            "install_format_schema_version": "1.1.0",
            "verification_status": "verified",
            "verification_key_id": "d27d666cd88e42b4",
            "latest_install_failed_attempts": []
          },
          "type": "epm-packages",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "typeMigrationVersion": "10.1.0",
          "updated_at": "2024-04-30T19:26:54.903Z",
          "created_at": "2024-04-29T23:46:55.955Z"
        }
      }
    ]
  }
}

Can anyone confirm whether or not they can replicate this issue regarding the PR in subject?

marc-gr added enhancement New feature or request Team:Security-External Integrations breaking change Integration:winlog Custom Windows Event Logs labels Sep 28, 2023

marc-gr force-pushed the feat/winlog-input branch from dc3f4b8 to 1a80271 Compare September 28, 2023 14:33

Convert to an input package.

fe3980b

marc-gr force-pushed the feat/winlog-input branch from 1a80271 to fe3980b Compare September 28, 2023 14:38

marc-gr marked this pull request as ready for review September 28, 2023 14:51

marc-gr requested a review from a team as a code owner September 28, 2023 14:51

taylor-swanson reviewed Sep 28, 2023

View reviewed changes

packages/winlog/manifest.yml Outdated Show resolved Hide resolved

andrewkroh reviewed Sep 28, 2023

View reviewed changes

packages/winlog/changelog.yml Outdated Show resolved Hide resolved

marc-gr added 2 commits October 24, 2023 15:27

Merge remote-tracking branch 'upstream/main' into feat/winlog-input

9371087

Change change type to breaking-change

9e669bc

marc-gr mentioned this pull request Oct 24, 2023

Change policies to work with new winlog input package v2.0.0 elastic/fleet-winlog-7x-policies#9

Merged

Bump kibana version

6166ace

marc-gr requested review from andrewkroh and taylor-swanson October 24, 2023 14:19

andrewkroh reviewed Oct 24, 2023

View reviewed changes

Edit README, and add dynamic ECS mappings

bdb1561

marc-gr requested a review from andrewkroh October 25, 2023 09:51

andrewkroh approved these changes Oct 26, 2023

View reviewed changes

marc-gr merged commit a8fb41c into elastic:main Oct 30, 2023

marc-gr deleted the feat/winlog-input branch October 30, 2023 12:04

jamiehynds mentioned this pull request Nov 8, 2023

[Custom Packages] Moving to Input packages #8435

Open

13 tasks

zedtran mentioned this pull request May 1, 2024

Fleet Package Policy API: Error Adding Winlog Integration - Stream Template Not Found #9769

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[winlog] Convert to an input package.#8010

[winlog] Convert to an input package.#8010
marc-gr merged 5 commits intoelastic:mainfrom
marc-gr:feat/winlog-input

marc-gr commented Sep 28, 2023

Uh oh!

elasticmachine commented Sep 28, 2023 •

edited

Loading

Build stats

Uh oh!

elasticmachine commented Sep 28, 2023

Uh oh!

Uh oh!

andrewkroh left a comment

Uh oh!

Uh oh!

marc-gr commented Oct 24, 2023

Uh oh!

andrewkroh left a comment

Uh oh!

andrewkroh Oct 24, 2023

Uh oh!

marc-gr Oct 25, 2023

Uh oh!

marc-gr commented Oct 25, 2023

Uh oh!

elasticmachine commented Oct 30, 2023

Uh oh!

zedtran commented Apr 30, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

marc-gr commented Sep 28, 2023

What does this PR do?

Checklist

Related issues

Screenshots

Uh oh!

elasticmachine commented Sep 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

🤖 GitHub comments

Uh oh!

elasticmachine commented Sep 28, 2023

Uh oh!

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

marc-gr commented Oct 24, 2023

Uh oh!

andrewkroh left a comment

Choose a reason for hiding this comment

Uh oh!

andrewkroh Oct 24, 2023

Choose a reason for hiding this comment

Uh oh!

marc-gr Oct 25, 2023

Choose a reason for hiding this comment

Uh oh!

marc-gr commented Oct 25, 2023

Uh oh!

elasticmachine commented Oct 30, 2023

Uh oh!

zedtran commented Apr 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elasticmachine commented Sep 28, 2023 •

edited

Loading

zedtran commented Apr 30, 2024 •

edited

Loading