fix: Handle NotFound error when syncing Subscriptions with deleted topic (aws_sns_subscriptions)#14771
Merged
kodiakhq[bot] merged 3 commits intocloudquery:mainfrom Oct 20, 2023
Merged
Conversation
NotFound error when syncing Subscription with deleted t…NotFound error when syncing Subscription with deleted topic (aws_sns_subscriptions)
…opic (`aws_sns_subscriptions`)
5513ecb to
a5f7dcc
Compare
erezrokah
approved these changes
Oct 20, 2023
NotFound error when syncing Subscription with deleted topic (aws_sns_subscriptions)NotFound error when syncing Subscriptions with deleted topic (aws_sns_subscriptions)
hydratim
pushed a commit
to hydratim/cloudquery
that referenced
this pull request
Oct 20, 2023
…topic (`aws_sns_subscriptions`) (cloudquery#14771) #### Summary AWS allows users to delete a SNS Topic even if it has active subscribers. <del>This causes a scenario where the user is unable to delete the hanging subscribers and apparently requires AWS support to manually intervene, see https://stackoverflow.com/a/64568018/8836650</del> EDIT: actually it appears you can delete hanging subscriptions now as long as they're confirmed, but its still doesn't seem possible to delete unconfirmed ones. When CloudQuery encounters a Subscription that is in this state it throws an error: ```bash 11:40AM ERR pre resource resolver failed error="operation error SNS: GetSubscriptionAttributes, https response error StatusCode: 404, RequestID: (redacted), NotFound: Topic does not exist" client=(redacted) module=aws-src table=aws_sns_subscriptions ``` This causes CloudQuery to lose data by not syncing that subscription resource. This PR handles that error and allows users to join their `aws_sns_subscriptions` and `aws_sns_topics` table to find all hanging subscriptions in their orgs. _Sorry about the spam of fix PRs 😅_ <!-- Explain what problem this PR addresses --> <!--
kodiakhq bot
pushed a commit
that referenced
this pull request
Oct 23, 2023
🤖 I have created a release *beep* *boop* --- ## [22.16.0](plugins-source-aws-v22.15.2...plugins-source-aws-v22.16.0) (2023-10-23) ### This Release has the Following Changes to Tables - Table `aws_efs_filesystems`: column added with name `file_system_policy` and type `utf8` ### Features * Add `policy` column to `aws_efs_filesystems` table ([#14672](#14672)) ([833b9c2](833b9c2)) ### Bug Fixes * Changed the condition to check for policies in policies table ([#13935](#13935)) ([f136331](f136331)) * **deps:** Update github.com/cloudquery/arrow/go/v14 digest to f46436f ([#14803](#14803)) ([f5248d7](f5248d7)) * **deps:** Update module github.com/cloudquery/codegen to v0.3.10 ([#14773](#14773)) ([98f3e2c](98f3e2c)) * **deps:** Update module github.com/cloudquery/codegen to v0.3.11 ([#14870](#14870)) ([4fa917d](4fa917d)) * Don't call GetFunctionCodeSigningConfig or GetRuntimeManagementConfig on Lambdas packaged as Images (`aws_lambda_functions`) ([#14729](#14729)) ([6fc30d3](6fc30d3)) * Handle `NotFound` error when syncing Subscriptions with deleted topic (`aws_sns_subscriptions`) ([#14771](#14771)) ([6fcf43d](6fcf43d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AWS allows users to delete a SNS Topic even if it has active subscribers.
This causes a scenario where the user is unable to delete the hanging subscribers and apparently requires AWS support to manually intervene, see https://stackoverflow.com/a/64568018/8836650EDIT: actually it appears you can delete hanging subscriptions now as long as they're confirmed, but its still doesn't seem possible to delete unconfirmed ones.When CloudQuery encounters a Subscription that is in this state it throws an error:
11:40AM ERR pre resource resolver failed error="operation error SNS: GetSubscriptionAttributes, https response error StatusCode: 404, RequestID: (redacted), NotFound: Topic does not exist" client=(redacted) module=aws-src table=aws_sns_subscriptionsThis causes CloudQuery to lose data by not syncing that subscription resource. This PR handles that error and allows users to join their
aws_sns_subscriptionsandaws_sns_topicstable to find all hanging subscriptions in their orgs.Sorry about the spam of fix PRs 😅