Skip to content

Add support for azure-eventhub input in integrations#1894

Merged
narph merged 14 commits intoelastic:masterfrom
narph:add-azure-eventhub
Nov 11, 2021
Merged

Add support for azure-eventhub input in integrations#1894
narph merged 14 commits intoelastic:masterfrom
narph:add-azure-eventhub

Conversation

@narph
Copy link
Copy Markdown
Contributor

@narph narph commented Oct 11, 2021 •
edited
Loading

What does this PR do?

Add support for azure-eventhub input in integrations

  • we allowed some consistency to the azure-eventhub fields in the other azure logs data streams
  • the option Parse azure message will do limited json parsing of the message including the most generic fields like subscriptionId and resourceId (frequently asked by azure users). I have added below the 2 examples of events.

Limitations:

  • the reason the azure-eventhub input is in the Azure Logs package is because it shares configuration options with the rest of the azure log data streams. This could cause some confusion here as the azure-eventhub input is not limited to collecting logs only, can collect any type events.
  • the data stream for the input has the type logs , I assume the allowed types are logs, metrics, traces. What do we do with inputs as this or kafka etc which can retrieve any type of messages users are streaming to their eventhubs. Should we add an additional type messages, events or decide on one of the existing ones?

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

Related issues

Screenshots

image

image

overlapping error is fixed in future release.

Ex of event not parsed:

   {
        "_index" : ".ds-logs-azure.eventhub-default-2021.10.18-000001",
        "_type" : "_doc",
        "_id" : "WExik3wBs4hpITkzfNnM",
        "_score" : null,
        "_source" : {
          "agent" : {
            "hostname" : "docker-fleet-agent",
            "name" : "docker-fleet-agent",
            "id" : "b659ebed-5338-45f3-9762-1bcf2c1ad0e1",
            "type" : "filebeat",
            "ephemeral_id" : "42a183d5-df19-4008-8776-d9765bc01d50",
            "version" : "7.15.0"
          },
          "elastic_agent" : {
            "id" : "b659ebed-5338-45f3-9762-1bcf2c1ad0e1",
            "version" : "7.15.0",
            "snapshot" : true
          },
          "message" : """{"ReleaseVersion":"6.2021.39.11+d1f0e29.release_2021w39_az","RoleLocation":"North Europe","callerIpAddress":"77.170.179.229","category":"Administrative","correlationId":"a75a0752-ebbb-42bf-831f-43788a8c1193","durationMs":"0","identity":.....,"eventCategory":"Administrative","hierarchy":"4fa94b7d-a743-486f-abcc-6c276c44cf4b/5341238b-665c-4eb4-b259-b250371ae430/7657426d-c4c3-44ac-88a2-3b2cd59e6dba","message":"Microsoft.ServiceBus/namespaces/delete"},"resourceId":"/SUBSCRIPTIONS/7657426D-C4C3-44AC-88A2-3B2CD59E6DBA/RESOURCEGROUPS/OBS-TEST/PROVIDERS/MICROSOFT.SERVICEBUS/NAMESPACES/TESTOBS","resultSignature":"Started.","resultType":"Start","tenantId":"4fa94b7d-a743-486f-abcc-6c276c44cf4b","time":"2021-10-15T09:08:29.9268177Z"}""",
          "azure-eventhub" : {
            "sequence_number" : 1215,
            "consumer_group" : "$Default",
            "offset" : 274878093752,
            "eventhub" : "insights-activity-logs",
            "enqueued_time" : "2021-10-15T09:14:25.419Z"
          },
          "tags" : [
            "azure-eventhub"
          ],
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2021-10-18T12:31:17.027Z",
          "ecs" : {
            "version" : "1.12.0"
          },
          "data_stream" : {
            "namespace" : "default",
            "type" : "logs",
            "dataset" : "azure.eventhub"
          },
          "host" : {
            "hostname" : "docker-fleet-agent",
            "os" : {
              "kernel" : "4.19.128-microsoft-standard",
              "codename" : "Core",
              "name" : "CentOS Linux",
              "family" : "redhat",
              "type" : "linux",
              "version" : "7 (Core)",
              "platform" : "centos"
            },
            "containerized" : true,
            "ip" : [
              "172.27.0.7"
            ],
            "name" : "docker-fleet-agent",
            "id" : "6505f7ca36739e7eb909bdb52bf3ec18",
            "mac" : [
              "02:42:ac:1b:00:07"
            ],
            "architecture" : "x86_64"
          },
          "event" : {
            "agent_id_status" : "verified",
            "ingested" : "2021-10-18T12:31:17Z",
            "kind" : "event",
            "dataset" : "azure.eventhub"
          }
        },
        "sort" : [
          1634560277027
        ]
      }

if parse_message is enabled:

 {
        "_index" : ".ds-logs-azure.eventhub-default-2021.10.18-000001",
        "_type" : "_doc",
        "_id" : "l0xZk3wBs4hpITkz-syS",
        "_score" : null,
        "_source" : {
          "agent" : {
            "hostname" : "docker-fleet-agent",
            "name" : "docker-fleet-agent",
            "id" : "b659ebed-5338-45f3-9762-1bcf2c1ad0e1",
            "ephemeral_id" : "6580f640-c603-4f59-93fe-b4a546a54630",
            "type" : "filebeat",
            "version" : "7.15.0"
          },
          "elastic_agent" : {
            "id" : "b659ebed-5338-45f3-9762-1bcf2c1ad0e1",
            "version" : "7.15.0",
            "snapshot" : true
          },
          "azure-eventhub" : {
            "sequence_number" : 1189,
            "consumer_group" : "$Default",
            "offset" : 261993165400,
            "eventhub" : "insights-activity-logs",
            "enqueued_time" : "2021-10-18T03:17:49.601Z"
          },
          "tags" : [
            "parse_message",
            "azure-eventhub"
          ],
          "cloud" : {
            "provider" : "azure"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2021-10-18T03:14:41.682Z",
          "ecs" : {
            "version" : "1.12.0"
          },
          "data_stream" : {
            "namespace" : "default",
            "type" : "logs",
            "dataset" : "azure.eventhub"
          },
          "host" : {
            "hostname" : "docker-fleet-agent",
            "os" : {
              "kernel" : "4.19.128-microsoft-standard",
              "codename" : "Core",
              "name" : "CentOS Linux",
              "type" : "linux",
              "family" : "redhat",
              "version" : "7 (Core)",
              "platform" : "centos"
            },
            "containerized" : true,
            "ip" : [
              "172.27.0.7"
            ],
            "name" : "docker-fleet-agent",
            "id" : "6505f7ca36739e7eb909bdb52bf3ec18",
            "mac" : [
              "02:42:ac:1b:00:07"
            ],
            "architecture" : "x86_64"
          },
          "event" : {
            "agent_id_status" : "verified",
            "ingested" : "2021-10-18T12:22:00Z",
            "kind" : "event",
            "dataset" : "azure.eventhub"
          },
          "azure" : {
            "subscription_id" : "7657426D-C4C3-44AC-88A2-3B2CD59E6DBA",
            "resource" : {
              "provider" : "MICROSOFT.APPPLATFORM/SPRING",
              "name" : "RK-SPRINGCLOUD-TEST",
              "id" : "/SUBSCRIPTIONS/7657426D-C4C3-44AC-88A2-3B2CD59E6DBA/RESOURCEGROUPS/RK-TEST-RESOURCES/PROVIDERS/MICROSOFT.APPPLATFORM/SPRING/RK-SPRINGCLOUD-TEST",
              "group" : "RK-TEST-RESOURCES"
            },
            "eventhub" : {
              "level" : "Information",
              "correlationId" : "e0521f42-19ff-40e3-bc18-b3ec1f91cbf0",
              "operationName" : "Microsoft.Resourcehealth/healthevent/Resolved/action",
              "category" : "ResourceHealth",
              "resultType" : "Resolved",
              "properties" : {
                "eventCategory" : "ResourceHealth",
                "eventProperties" : {
                  "cause" : "Unknown",
                  "currentHealthStatus" : "Available",
                  "details" : "Unknown",
                  "previousHealthStatus" : "Degraded",
                  "title" : "Unknown",
                  "type" : "Unknown"
                }
              }
            }
          }
        },
        "sort" : [
          1634526881682
        ]
      }

@P1llus , @ruflin , @ravikesarwani , @andrewkroh, @jsoriano would love to hear your thoughts here

@narph narph self-assigned this Oct 11, 2021
@narph narph marked this pull request as draft October 11, 2021 15:00
@narph narph added Integration:azure Azure Logs Team:Integrations Label for the Integrations team labels Oct 11, 2021
@narph narph changed the title Add support for azure-eventhub input in itnegrations Add support for azure-eventhub input in integrations Oct 11, 2021
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Oct 11, 2021 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-11-11T13:30:31.078+0000

  • Duration: 13 min 8 sec

  • Commit: 06ca2f1

Test stats 🧪

Test Results
Failed 0
Passed 79
Skipped 0
Total 79

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@narph narph changed the title Add support for azure-eventhub input in integrations POC - Add support for azure-eventhub input in integrations Oct 11, 2021
@narph narph changed the title POC - Add support for azure-eventhub input in integrations Add support for azure-eventhub input in integrations Oct 28, 2021
ruflin
ruflin reviewed Oct 29, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the type, lets stick to logs for the moment. But as you pointed out, ideally we would route it to the correct data stream. For me the end goal is to have document based event routing in place (elastic/elasticsearch#63798) and then the package would only need to configure this. But it is not something we have yet. This leaves us with 2 places to route the data: On the input side or in the ingest pipeline. What values do we have available inside the event to make a decision on where it should end up?

@narph
Copy link
Copy Markdown
Contributor Author

narph commented Nov 1, 2021

For the type, lets stick to logs for the moment. But as you pointed out, ideally we would route it to the correct data stream. For me the end goal is to have document based event routing in place (elastic/elasticsearch#63798) and then the package would only need to configure this. But it is not something we have yet. This leaves us with 2 places to route the data: On the input side or in the ingest pipeline. What values do we have available inside the event to make a decision on where it should end up?

the routing would be great, unfortunately, there isn't much that Azure gives away on detecting which is a log event, trace, metric, or even key value format (for key vault resources for example).

@narph narph marked this pull request as ready for review November 3, 2021 12:44
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Nov 3, 2021

Pinging @elastic/integrations (Team:Integrations)

jsoriano
jsoriano reviewed Nov 10, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

packages/azure/data_stream/eventhub/sample_event.json
"version": "7.15.0",
"snapshot": true
},
"message": "{\\\"ReleaseVersion\\\":\\\"6.2021.39.11+d1f0e29.release_2021w39_az\\\",\\\"RoleLocation\\\":\\\"North Europe\\\",\\\"callerIpAddress\\\":\\\"77.170.179.229\\\",\\\"category\\\":\\\"Administrative\\\",\\\"correlationId\\\":\\\"a75a0752-ebbb-42bf-831f-43788a8c1193\\\",\\\"durationMs\\\":\\\"0\\\",\\\"identity\\\":{\\\"authorization\\\":{\\\"action\\\":\\\"Microsoft.ServiceBus\\/namespaces\\/delete\\\",\\\"evidence\\\":{\\\"principalId\\\":\\\"68b1adf93eb744b08eb8ce96522a08d3\\\",\\\"principalType\\\":\\\"User\\\",\\\"role\\\":\\\"Owner\\\",\\\"roleAssignmentId\\\":\\\"7f06f09dd6764b44930adbec3f10e92b\\\",\\\"roleAssignmentScope\\\":\\\"\\/providers\\/Microsoft.Management\\/managementGroups\\/5341238b-665c-4eb4-b259-b250371ae430\\\",\\\"roleDefinitionId\\\":\\\"8e3af657a8ff443ca75c2fe8c4bcb635\\\"},\\\"scope\\\":\\\"\\/subscriptions\\/7657426d-c4c3-44ac-88a2-3b2cd59e6dba\\/resourcegroups\\/obs-test\\/providers\\/Microsoft.ServiceBus\\/namespaces\\/testobs\\\"},\\\"claims\\\":{\\\"aio\\\":\\\"ATQAy\\/8TAAAAgFUjNWoJWKgHlAK2AL92UMeUsb6VD5zck\\/myDZPucX5V3Gc8SDMg5vTV28NUy5N7\\\",\\\"appid\\\":\\\"c44b4083-3bb0-49c1-b47d-974e53cbdf3c\\\",\\\"appidacr\\\":\\\"2\\\",\\\"aud\\\":\\\"https:\\/\\/management.core.windows.net\\/\\\",\\\"exp\\\":\\\"1634290798\\\",\\\"groups\\\":\\\"644c6686-9ef1-4b69-9410-107664a9e1f0,9ed1993c-ce9c-4915-a04d-58c6f5f7ee12,a953f548-26ab-47b2-be7d-65586b7bcc2a\\\",\\\"http:\\/\\/schemas.microsoft.com\\/2012\\/01\\/devicecontext\\/claims\\/identifier\\\":\\\"1060004c-63dc-465b-b868-ec6547176c58\\\",\\\"http:\\/\\/schemas.microsoft.com\\/claims\\/authnclassreference\\\":\\\"1\\\",\\\"http:\\/\\/schemas.microsoft.com\\/claims\\/authnmethodsreferences\\\":\\\"pwd,rsa\\\",\\\"http:\\/\\/schemas.microsoft.com\\/identity\\/claims\\/objectidentifier\\\":\\\"68b1adf9-3eb7-44b0-8eb8-ce96522a08d3\\\",\\\"http:\\/\\/schemas.microsoft.com\\/identity\\/claims\\/scope\\\":\\\"user_impersonation\\\",\\\"http:\\/\\/schemas.microsoft.com\\/identity\\/claims\\/tenantid\\\":\\\"4fa94b7d-a743-486f-abcc-6c276c44cf4b\\\",\\\"http:\\/\\/schemas.xmlsoap.org\\/ws\\/2005\\/05\\/identity\\/claims\\/givenname\\\":\\\"Mariana\\\",\\\"http:\\/\\/schemas.xmlsoap.org\\/ws\\/2005\\/05\\/identity\\/claims\\/name\\\":\\\"mariana@elastic.co\\\",\\\"http:\\/\\/schemas.xmlsoap.org\\/ws\\/2005\\/05\\/identity\\/claims\\/nameidentifier\\\":\\\"a9L2WR3XZN5ANzAqwLx_4aamU49JG6kqaE5JZkXdeNs\\\",\\\"http:\\/\\/schemas.xmlsoap.org\\/ws\\/2005\\/05\\/identity\\/claims\\/surname\\\":\\\"Dima\\\",\\\"http:\\/\\/schemas.xmlsoap.org\\/ws\\/2005\\/05\\/identity\\/claims\\/upn\\\":\\\"mariana@elastic.co\\\",\\\"iat\\\":\\\"1634286898\\\",\\\"ipaddr\\\":\\\"77.170.179.229\\\",\\\"iss\\\":\\\"https:\\/\\/sts.windows.net\\/4fa94b7d-a743-486f-abcc-6c276c44cf4b\\/\\\",\\\"name\\\":\\\"Mariana Dima\\\",\\\"nbf\\\":\\\"1634286898\\\",\\\"puid\\\":\\\"1003200045B17AD4\\\",\\\"rh\\\":\\\"0.AVEAfUupT0Onb0irzGwnbETPS4NAS8SwO8FJtH2XTlPL3zxRAA8.\\\",\\\"uti\\\":\\\"yUcYeZwj9EWeA-rTCtRwAA\\\",\\\"ver\\\":\\\"1.0\\\",\\\"wids\\\":\\\"5d6b6bb7-de71-4623-b4af-96380a352509\\\",\\\"xms_tcdt\\\":\\\"1469565974\\\"}},\\\"level\\\":\\\"Information\\\",\\\"operationName\\\":\\\"MICROSOFT.SERVICEBUS\\/NAMESPACES\\/DELETE\\\",\\\"properties\\\":{\\\"entity\\\":\\\"\\/subscriptions\\/7657426d-c4c3-44ac-88a2-3b2cd59e6dba\\/resourcegroups\\/obs-test\\/providers\\/Microsoft.ServiceBus\\/namespaces\\/testobs\\\",\\\"eventCategory\\\":\\\"Administrative\\\",\\\"hierarchy\\\":\\\"4fa94b7d-a743-486f-abcc-6c276c44cf4b\\/5341238b-665c-4eb4-b259-b250371ae430\\/7657426d-c4c3-44ac-88a2-3b2cd59e6dba\\\",\\\"message\\\":\\\"Microsoft.ServiceBus\\/namespaces\\/delete\\\"},\\\"resourceId\\\":\\\"\\/SUBSCRIPTIONS\\/7657426D-C4C3-44AC-88A2-3B2CD59E6DBA\\/RESOURCEGROUPS\\/OBS-TEST\\/PROVIDERS\\/MICROSOFT.SERVICEBUS\\/NAMESPACES\\/TESTOBS\\\",\\\"resultSignature\\\":\\\"Started.\\\",\\\"resultType\\\":\\\"Start\\\",\\\"tenantId\\\":\\\"4fa94b7d-a743-486f-abcc-6c276c44cf4b\\\",\\\"time\\\":\\\"2021-10-15T09:08:29.9268177Z\\\"}\\r\\n",
Copy link
Copy Markdown
Member

@jsoriano jsoriano Nov 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a rename to move message to event.original, is this outdated?

Copy link
Copy Markdown
Member

@jsoriano jsoriano Nov 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarified offline, this pipeline is conditionally executed:

  - pipeline:
      if: "ctx?.tags != null && ctx.tags.contains('parse_message')"
      name: '{{ IngestPipeline "parsed-message" }}'

@narph narph merged commit b707b5d into elastic:master Nov 11, 2021
@narph narph deleted the add-azure-eventhub branch November 11, 2021 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ruflin ruflin ruflin left review comments

@P1llus P1llus P1llus left review comments

@jsoriano jsoriano jsoriano approved these changes

Assignees

@narph narph

Labels

Integration:azure Azure Logs Team:Integrations Label for the Integrations team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@narph @elasticmachine @jsoriano @ruflin @P1llus