[AWS Billing] Add processor and leader election support by BenB196 · Pull Request #8414 · elastic/integrations

BenB196 · 2023-11-07T00:13:19Z

Proposed commit message

Add support for defining processors to AWS Billing
- Useful for doing things like adding tags/data manipulation.
Add support for enabling leaderelection.
- Useful for if you want to run the integration on Kubernetes.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

I've added both of these settings under advanced as I think both won't see too frequent of usage.

How to test this PR locally

Integration before upgrade

inputs:
  - id: aws/metrics-billing-30d07b98-4f9b-4735-9116-aa37b164a07e
    name: aws-1
    revision: 1
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.8.5
    data_stream:
      namespace: default
    package_policy_id: 30d07b98-4f9b-4735-9116-aa37b164a07e
    streams:
      - id: aws/metrics-aws.billing-30d07b98-4f9b-4735-9116-aa37b164a07e
        data_stream:
          dataset: aws.billing
          type: metrics
        metricsets:
          - billing
        period: 24h
        include_linked_accounts: true
        cost_explorer_config.group_by_dimension_keys:
          - AZ
          - INSTANCE_TYPE
          - SERVICE
          - LINKED_ACCOUNT
        cost_explorer_config.group_by_tag_keys:
          - 'aws:createdBy'

Integration after upgrade

(Observe that neither processors nor condition are added here as it should be)

inputs:
  - id: aws/metrics-billing-30d07b98-4f9b-4735-9116-aa37b164a07e
    name: aws-1
    revision: 2
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.9.0
    data_stream:
      namespace: default
    package_policy_id: 30d07b98-4f9b-4735-9116-aa37b164a07e
    streams:
      - id: aws/metrics-aws.billing-30d07b98-4f9b-4735-9116-aa37b164a07e
        data_stream:
          dataset: aws.billing
          type: metrics
        metricsets:
          - billing
        period: 24h
        include_linked_accounts: true
        cost_explorer_config.group_by_dimension_keys:
          - AZ
          - INSTANCE_TYPE
          - SERVICE
          - LINKED_ACCOUNT
        cost_explorer_config.group_by_tag_keys:
          - 'aws:createdBy'

Leader Election enabled, Processors null

inputs:
  - id: aws/metrics-billing-30d07b98-4f9b-4735-9116-aa37b164a07e
    name: aws-1
    revision: 3
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.9.0
    data_stream:
      namespace: default
    package_policy_id: 30d07b98-4f9b-4735-9116-aa37b164a07e
    streams:
      - id: aws/metrics-aws.billing-30d07b98-4f9b-4735-9116-aa37b164a07e
        data_stream:
          dataset: aws.billing
          type: metrics
        metricsets:
          - billing
        period: 24h
        include_linked_accounts: true
        cost_explorer_config.group_by_dimension_keys:
          - AZ
          - INSTANCE_TYPE
          - SERVICE
          - LINKED_ACCOUNT
        cost_explorer_config.group_by_tag_keys:
          - 'aws:createdBy'
        condition: '${kubernetes_leaderelection.leader} == true'

Leader Election Enabled, Processors Set

inputs:
  - id: aws/metrics-billing-30d07b98-4f9b-4735-9116-aa37b164a07e
    name: aws-1
    revision: 4
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.9.0
    data_stream:
      namespace: default
    package_policy_id: 30d07b98-4f9b-4735-9116-aa37b164a07e
    streams:
      - id: aws/metrics-aws.billing-30d07b98-4f9b-4735-9116-aa37b164a07e
        data_stream:
          dataset: aws.billing
          type: metrics
        metricsets:
          - billing
        period: 24h
        include_linked_accounts: true
        cost_explorer_config.group_by_dimension_keys:
          - AZ
          - INSTANCE_TYPE
          - SERVICE
          - LINKED_ACCOUNT
        cost_explorer_config.group_by_tag_keys:
          - 'aws:createdBy'
        condition: '${kubernetes_leaderelection.leader} == true'
        processors:
          - add_tags:
              tags:
                - test

Leader Elastic disabled, Processors set

inputs:
  - id: aws/metrics-billing-30d07b98-4f9b-4735-9116-aa37b164a07e
    name: aws-1
    revision: 5
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.9.0
    data_stream:
      namespace: default
    package_policy_id: 30d07b98-4f9b-4735-9116-aa37b164a07e
    streams:
      - id: aws/metrics-aws.billing-30d07b98-4f9b-4735-9116-aa37b164a07e
        data_stream:
          dataset: aws.billing
          type: metrics
        metricsets:
          - billing
        period: 24h
        include_linked_accounts: true
        cost_explorer_config.group_by_dimension_keys:
          - AZ
          - INSTANCE_TYPE
          - SERVICE
          - LINKED_ACCOUNT
        cost_explorer_config.group_by_tag_keys:
          - 'aws:createdBy'
        processors:
          - add_tags:
              tags:
                - test

Related issues

Screenshots

elasticmachine · 2023-11-07T00:17:10Z

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-11-15T23:50:39.654+0000
Duration: 3 min 9 sec

Steps errors

Expand to view the steps failures

`Load a resource file from a library`

Took 0 min 0 sec . View more details here
Description: approval-list/elastic/integrations.yml

`Error signal`

Took 0 min 0 sec . View more details here
Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/BenB196 return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/BenB196 : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/BenB196 : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/orgs/members#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

botelastic · 2023-12-16T00:03:15Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

mrodm · 2023-12-21T09:42:54Z

Hi @BenB196, please update your branch with the latest contents from main branch. There was an important PR merged updating the CI pipelines. Thanks!

botelastic · 2024-01-20T11:21:00Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

BenB196 · 2024-01-26T15:30:13Z

Hi, don't mean to bother, but would it be possible to get a review of this PR?

botelastic · 2024-02-25T15:41:56Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

packages/aws/data_stream/billing/manifest.yml

ishleenk17 · 2024-03-09T18:11:23Z

/test

ishleenk17 · 2024-03-11T09:08:39Z

/test

ishleenk17 · 2024-03-11T10:11:32Z

The changes look fine. There is a buildkite failure. I have given a retrigger. Lets wait for the CI/Cd results.
@elastic/obs-ds-hosted-services : Could you review as well ?

ishleenk17 · 2024-03-11T16:02:05Z

/test

ishleenk17

Looks good! @elastic/obs-ds-hosted-services : Please review as well

BenB196 · 2024-04-05T11:04:20Z

@ishleenk17 sorry to bother, is there anything left that needs to be addressed in this PR?

ishleenk17 · 2024-04-05T11:10:25Z

There are other code owner reviews pending.
@elastic/security-service-integrations : Could you please take a look at it as code owners as well.

ishleenk17 · 2024-04-05T11:12:31Z

/test

ishleenk17 · 2024-04-05T11:13:05Z

@BenB196 : You have the approvals now. I have triggered the build.
If it succeeds, will merge.

elasticmachine · 2024-04-05T12:30:27Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2024-04-05T12:31:08Z

💚 Build Succeeded

Buildkite Build
Commit: eebd86d

History

💚 Build #9491 succeeded 67c4317
💔 Build #9477 failed 67c4317
💔 Build #9465 failed 67c4317

elastic-sonarqube · 2024-04-05T12:31:08Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% 0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-04-05T14:10:10Z

Package aws - 2.14.0 containing this change is available at https://epr.elastic.co/search?package=aws

add processor and leader election support to aws billing

c1efac0

BenB196 requested review from a team as code owners November 7, 2023 00:13

Update changelog.yml

c7741e9

BenB196 added 2 commits November 6, 2023 19:22

fix changelog typo

12a539c

better changelog wording

a8f156f

andrewkroh added the Integration:aws AWS label Nov 7, 2023

BenB196 and others added 2 commits November 15, 2023 18:48

Merge branch 'main' into aws-billing-processors-leader-elect

20bb0bc

Update manifest.yml

bf6ea9c

botelastic bot added the Stalled label Dec 16, 2023

botelastic bot removed the Stalled label Dec 21, 2023

Merge branch 'main' into aws-billing-processors-leader-elect

a26ceb6

BenB196 requested review from a team as code owners December 21, 2023 11:20

botelastic bot added the Stalled label Jan 20, 2024

Merge branch 'main' into aws-billing-processors-leader-elect

6924fa5

BenB196 requested a review from a team as a code owner January 26, 2024 15:27

botelastic bot removed the Stalled label Jan 26, 2024

botelastic bot added the Stalled label Feb 25, 2024

Merge branch 'main' into aws-billing-processors-leader-elect

67c4317

botelastic bot removed the Stalled label Mar 6, 2024

ishleenk17 reviewed Mar 8, 2024

View reviewed changes

packages/aws/data_stream/billing/manifest.yml Show resolved Hide resolved

BenB196 requested a review from ishleenk17 March 8, 2024 11:21

ishleenk17 approved these changes Mar 12, 2024

View reviewed changes

MichaelKatsoulis approved these changes Mar 12, 2024

View reviewed changes

Merge branch 'main' into aws-billing-processors-leader-elect

eebd86d

BenB196 requested review from MichaelKatsoulis and ishleenk17 March 18, 2024 15:02

ShourieG approved these changes Apr 5, 2024

View reviewed changes

ishleenk17 enabled auto-merge (squash) April 5, 2024 11:16

ishleenk17 disabled auto-merge April 5, 2024 11:17

ishleenk17 merged commit fac9ab9 into elastic:main Apr 5, 2024

BenB196 deleted the aws-billing-processors-leader-elect branch April 5, 2024 14:30

kaiyan-sheng mentioned this pull request Jul 16, 2024

[AWS] Add leader election for all metric data streams #10500

Closed

Conversation

BenB196 commented Nov 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Nov 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❕ Build Aborted

Build stats

Steps errors

Load a resource file from a library

Error signal

🤖 GitHub comments

Uh oh!

botelastic bot commented Dec 16, 2023

Uh oh!

mrodm commented Dec 21, 2023

Uh oh!

botelastic bot commented Jan 20, 2024

Uh oh!

BenB196 commented Jan 26, 2024

Uh oh!

botelastic bot commented Feb 25, 2024

Uh oh!

Uh oh!

ishleenk17 commented Mar 9, 2024

Uh oh!

ishleenk17 commented Mar 11, 2024

Uh oh!

ishleenk17 commented Mar 11, 2024

Uh oh!

ishleenk17 commented Mar 11, 2024

Uh oh!

ishleenk17 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BenB196 commented Apr 5, 2024

Uh oh!

ishleenk17 commented Apr 5, 2024

Uh oh!

ishleenk17 commented Apr 5, 2024

Uh oh!

ishleenk17 commented Apr 5, 2024

Uh oh!

elasticmachine commented Apr 5, 2024

🚀 Benchmarks report

Uh oh!

elasticmachine commented Apr 5, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Apr 5, 2024

Quality Gate passed

Uh oh!

elasticmachine commented Apr 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

BenB196 commented Nov 7, 2023 •

edited

Loading

elasticmachine commented Nov 7, 2023 •

edited

Loading

`Load a resource file from a library`

`Error signal`

ishleenk17 left a comment •

edited

Loading