Skip to content

[AWS] Add CloudWatch logs latency config option#5777

Merged
zmoog merged 5 commits intomainfrom
zmoog/add-cloudwatch-logs-latency-option
Apr 4, 2023
Merged

[AWS] Add CloudWatch logs latency config option#5777
zmoog merged 5 commits intomainfrom
zmoog/add-cloudwatch-logs-latency-option

Conversation

@zmoog
Copy link
Copy Markdown
Contributor

@zmoog zmoog commented Apr 3, 2023 •
edited
Loading

Motivation

If the logs collection starts at 10:00, then the input will gather logs in this time window:

  • start: 09:59
  • end: 10:00

Unfortunately, logs sometimes become available in CloudWatch with a delay; in these circumstances, the input of the CloudWatch logs could miss them.

For example, if a log event is:

  • created: 09:59:30
  • available 10:00:30

The input will miss that event with the default latency setting 1.

However, the CloudWatch Logs input has a setting called latency that allows users to adjust the logs collection time window to cope with the AWS services' latency.

Change description

Make the CloudWatch Logs input setting option latency available in the integration settings.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

Screenshots

Settings UI

CleanShot 2023-04-04 at 15 50 36@2x

Documentation

CleanShot 2023-04-04 at 13 43 05@2x

Footnotes

  1. The input will pick that log event up at the next restart if "start position" is set to "beginning". ↩

@zmoog zmoog added enhancement New feature or request Team:Cloud-Monitoring Label for the Cloud Monitoring team labels Apr 3, 2023
@zmoog zmoog self-assigned this Apr 3, 2023
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Apr 3, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-04-04T13:48:40.658+0000

  • Duration: 52 min 33 sec

Test stats 🧪

Test Results
Failed 0
Passed 188
Skipped 4
Total 192

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Apr 3, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (15/15) 💚
Files 93.75% (15/16) 👎 -6.25
Classes 93.75% (15/16) 👎 -6.25
Methods 86.131% (236/274) 👎 -9.107
Lines 85.925% (7387/8597) 👎 -8.88
Conditionals 100.0% (0/0) 💚

@zmoog zmoog force-pushed the zmoog/add-cloudwatch-logs-latency-option branch from a57e176 to 5e0d1b7 Compare April 4, 2023 11:45
@zmoog zmoog marked this pull request as ready for review April 4, 2023 11:45
@zmoog zmoog requested a review from a team as a code owner April 4, 2023 11:45
@endorama
Copy link
Copy Markdown
Contributor

endorama commented Apr 4, 2023

This PR closes #5712

endorama
endorama reviewed Apr 4, 2023
View reviewed changes
packages/aws/data_stream/cloudwatch_logs/manifest.yml
show_user: false
default: 200ms
description: This is used to sleep between AWS FilterLogEvents API calls inside the same collection period. `FilterLogEvents` API has a quota of 5 transactions per second (TPS)/account/Region. This value should only be adjusted when there are multiple Filebeats or multiple Filebeat inputs collecting logs from the same region and AWS account.
- name: latency
Copy link
Copy Markdown
Contributor

@endorama endorama Apr 4, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see there is a description in the screenshot and I expected to see it here. Is this missing the description field?

Copy link
Copy Markdown
Contributor Author

@zmoog zmoog Apr 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks (again) for noticing this!

I missed the line with the description field because it was attached to another change for a different PR 🤦🤦

CleanShot 2023-04-04 at 15 21 25@2x

Copy link
Copy Markdown
Contributor Author

@zmoog zmoog Apr 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added with 17afdcf

packages/aws/_dev/build/docs/cloudwatch.md

Log events on the busies log groups may require a longer time before they are available to CloudWatch Logs.

The CloudWatch integration offers the `latency` setting to cope with this scenario. Latency translates the query's time range to consider the CloudWatch Logs latency. For example, a `5m` latency means the integration will query CloudWatch for logs available 5 minutes ago.
Copy link
Copy Markdown
Contributor

@endorama endorama Apr 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would document the format expected by this setting, as it may not be familiar. (es a link to the documentation for the underlying string parser that converts strings to time durations)

Copy link
Copy Markdown
Contributor Author

@zmoog zmoog Apr 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! Adding it.

Copy link
Copy Markdown
Contributor Author

@zmoog zmoog Apr 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about this?

CleanShot 2023-04-04 at 15 46 04@2x

Details at 2091d4e

@zmoog
Copy link
Copy Markdown
Contributor Author

zmoog commented Apr 4, 2023 •
edited
Loading

This PR closes #5712

Oh boy, thank you for the heads up; I forgot to add a reference to this issue! 🤦 (added in the PR description)

@zmoog zmoog requested a review from endorama April 4, 2023 14:06
@zmoog zmoog merged commit 04e6661 into main Apr 4, 2023
@zmoog zmoog deleted the zmoog/add-cloudwatch-logs-latency-option branch April 4, 2023 16:23
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Apr 4, 2023

Package aws - 1.33.0 containing this change is available at https://epr.elastic.co/search?package=aws

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dmathieu dmathieu dmathieu approved these changes

@endorama endorama Awaiting requested review from endorama

Assignees

@zmoog zmoog

Labels

enhancement New feature or request Integration:aws AWS Team:Cloud-Monitoring Label for the Cloud Monitoring team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AWS Cloudwatch integration missing latency option

5 participants

@zmoog @elasticmachine @endorama @dmathieu @andrewkroh