Skip to content

[Custom AWS Logs] Add latency config parameter into aws-cloudwatch input#4859

Merged
tommyers-elastic merged 2 commits intoelastic:mainfrom
kaiyan-sheng:add_latency
Dec 19, 2022
Merged

[Custom AWS Logs] Add latency config parameter into aws-cloudwatch input#4859
tommyers-elastic merged 2 commits intoelastic:mainfrom
kaiyan-sheng:add_latency

Conversation

@kaiyan-sheng
Copy link
Copy Markdown

@kaiyan-sheng kaiyan-sheng commented Dec 16, 2022 •
edited
Loading

What does this PR do?

This PR is to add latency config parameter into aws-cloudwatch input. During testing, I found we are also missing the number_of_workers parameter too.

Here is what the agent policy looks like after installing the integration with specifying latency:

inputs:
  - id: aws-cloudwatch-aws_logs-bcb0ad28-0671-41af-bc9a-9657b6d40f3e
    name: aws_logs-1
    revision: 1
    type: aws-cloudwatch
    use_output: default
    meta:
      package:
        name: aws_logs
        version: 0.3.2
    data_stream:
      namespace: default
    streams:
      - id: aws-cloudwatch-aws_logs.generic-bcb0ad28-0671-41af-bc9a-9657b6d40f3e
        data_stream:
          dataset: aws_logs.generic
        log_group_arn: >-
          arn:aws:logs:eu-west-1:123456789:log-group:/aws/lambda/apm-test-api-gateway-v1:*
        start_position: beginning
        scan_frequency: 1m
        api_sleep: 200ms
        latency: 30m
        access_key_id: a
        secret_access_key: b
        session_token: c
        tags:
          - forwarded
        publisher_pipeline.disable_host: true

and here is what the agent policy looks like with specifying number_of_workers:

inputs:
  - id: aws-cloudwatch-aws_logs-16342400-df06-4e9e-8ef6-9f163c25ef69
    name: aws_logs-1
    revision: 1
    type: aws-cloudwatch
    use_output: default
    meta:
      package:
        name: aws_logs
        version: 0.3.5
    data_stream:
      namespace: default
    streams:
      - id: aws-cloudwatch-aws_logs.generic-16342400-df06-4e9e-8ef6-9f163c25ef69
        data_stream:
          dataset: aws_logs.generic
        log_group_name_prefix: /kaiyan/test
        number_of_workers: 5
        region_name: null
        start_position: beginning
        scan_frequency: 1m
        api_sleep: 200ms
        access_key_id: a
        secret_access_key: b
        session_token: c
        tags:
          - forwarded
        publisher_pipeline.disable_host: true

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

Screenshot

Screen Shot 2022-12-16 at 2 54 06 PM

@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner December 16, 2022 20:20
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 16, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-12-16T21:59:15.895+0000

  • Duration: 13 min 58 sec

Test stats 🧪

Test Results
Failed 0
Passed 2
Skipped 0
Total 2

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 16, 2022

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (0/0) 💚
Files 100.0% (0/0) 💚 2.251
Classes 100.0% (0/0) 💚 2.251
Methods 66.667% (2/3) 👎 -24.832
Lines 100.0% (0/0) 💚 8.341
Conditionals 100.0% (0/0) 💚

@kaiyan-sheng kaiyan-sheng self-assigned this Dec 16, 2022
@kaiyan-sheng kaiyan-sheng added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Dec 16, 2022
@tommyers-elastic tommyers-elastic merged commit 5cb3b6f into elastic:main Dec 19, 2022
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Dec 19, 2022

Package aws_logs - 0.3.1 containing this change is available at https://epr.elastic.co/search?package=aws_logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@francescayeye francescayeye francescayeye approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@kaiyan-sheng kaiyan-sheng

Labels

Team:Cloud-Monitoring Label for the Cloud Monitoring team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[AWS Custom Logs] Missing latency config parameter for aws-cloudwatch input

4 participants

@kaiyan-sheng @elasticmachine @francescayeye @tommyers-elastic