Skip to content

AWS package: revert back change to dynamic_dataset/namespace#6806

Closed
gsantoro wants to merge 2 commits intoelastic:mainfrom
gsantoro:feature/aws_dynamic_dataset
Closed

AWS package: revert back change to dynamic_dataset/namespace#6806
gsantoro wants to merge 2 commits intoelastic:mainfrom
gsantoro:feature/aws_dynamic_dataset

Conversation

@gsantoro
Copy link
Copy Markdown
Contributor

@gsantoro gsantoro commented Jul 4, 2023

What does this PR do?

Revert back changes from issue elastic/kibana#157897.

Removing the following configs from the system package manifest

elasticsearch.dynamic_dataset: true
elasticsearch.dynamic_namespace: true

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@gsantoro gsantoro added enhancement New feature or request v8.9.0 labels Jul 4, 2023
@gsantoro gsantoro requested review from felixbarny and ruflin July 4, 2023 10:30
@gsantoro gsantoro self-assigned this Jul 4, 2023
@gsantoro gsantoro requested a review from a team as a code owner July 4, 2023 10:30
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Jul 4, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-07-04T10:33:15.902+0000

  • Duration: 53 min 39 sec

Test stats 🧪

Test Results
Failed 0
Passed 194
Skipped 4
Total 198

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link
Copy Markdown

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (15/15) 💚
Files 93.75% (15/16) 👎 -6.25
Classes 93.75% (15/16) 👎 -6.25
Methods 85.714% (240/280) 👍
Lines 85.925% (7387/8597) 👎 -7.846
Conditionals 100.0% (0/0) 💚

@@ -196,6 +196,3 @@ streams:
title: Dataset name
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has a dataset config. Maybe this one needs to keep routing? @felixbarny

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I guess so, but I don't really know what this data stream is doing. Is it monitoring cloud watch itself or is it an input for logs coming via cloud watch?

What's the difference between this and the aws_logs package?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe someone from @elastic/obs-cloud-monitoring can chime in.

What I can see from the aws.cloudwatch_logs/manifest.yml

title: AWS CloudWatch logs via S3 (Deprecated)
description: (Deprecated) Please use Custom AWS Logs integration instead

this datastream was ingesting cloudwatch logs from s3 and it might be deprecated in favour of aws_log.generic datastream-based integraiton. I believe this last one might need to be an input package instead, according to the definition used in our last meetin @ruflin .

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are removing this option (reading logs in cloudwatch via s3) since it does not make sense. If the user has logs in cloudwatch already, then aws-cloudwatch input should be used. There is no need to route all logs to S3 from cloudwatch and then ingest them with aws-s3 input. This option is marked as deprecated for a while and we will remove it soon.

@gsantoro
Copy link
Copy Markdown
Contributor Author

gsantoro commented Jul 7, 2023

I had a chat with @girodav about this PR since he approved the related one on aws_logs. He told me that he is not 100% of this change. Personally, I would keep the routing in place and close this PR without merging. @felixbarny ?

@felixbarny
Copy link
Copy Markdown
Member

sgtm

@gsantoro
Copy link
Copy Markdown
Contributor Author

gsantoro commented Jul 7, 2023

Given the discussion at #6806 (comment) and the fact that one of the datastream is still active (not deprecated), we decided to keep this change in place and to NOT merge this PR

@gsantoro gsantoro closed this Jul 7, 2023
@gsantoro gsantoro deleted the feature/aws_dynamic_dataset branch July 7, 2023 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request v8.9.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants