feat: Added checking of finished fetches before running policy by amanenk · Pull Request #444 · cloudquery/cloudquery

amanenk · 2022-01-26T12:42:39Z

added logic that checks for fetch summaries that satisfy policy reqirements before running policy
fetch summary client is moved to separate package to prevent import cycle
closes When fetch doesn't run policy message is not clear enough. #317

pkg/client/fetch_summary/fetch_summary.go

pkg/client/client.go

pkg/client/fetch_summary/fetch_summary.go

disq

nits about naming things

pkg/client/fetch_summary/fetch_summary.go

pkg/client/fetch_summary/fetch_summary_test.go

pkg/client/fetch_summary/fetch_summary.go

disq

I still don't like fetch_summary.FetchSummary but OK otherwise

pkg/client/fetch_summary/fetch_summary.go

yevgenypats

LGTM. one nit.

pkg/policy/execute_test.go

added migrations for timescaledb

# Conflicts: # pkg/client/client.go # pkg/client/fetch.go # pkg/policy/execute.go # pkg/policy/execute_test.go

disq

You don’t need timescale migrations since the migrator always runs on the postgres dialect.

pkg/policy/execute.go

pkg/client/meta_storage/fetch_summary.go

Co-authored-by: Kemal <223029+disq@users.noreply.github.com>

# Conflicts: # pkg/client/client.go # pkg/policy/execute.go # pkg/policy/execute_test.go

roneli

LGTM

* upstream/main: chore: Update SDK (cloudquery#557) feat: Drift: List unimplemented resources in debug mode (cloudquery#558) fix: Policy describe should run without config.hcl requirement (cloudquery#546) feat: Limit scope of views (cloudquery#556) feat: Simplify fetch warning/error counts (unless -v is on) (cloudquery#554) fix: Add ProviderVersion to the 'cloudquery.fetches' meta table (cloudquery#555) chore: Sync from .github (cloudquery#553) feat: Added checking of finished fetches before running policy (cloudquery#444)

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/hashicorp/go-hclog](https://togithub.com/hashicorp/go-hclog) | require | patch | `v1.2.1` -> `v1.2.2` | --- ### Release Notes <details> <summary>hashicorp/go-hclog</summary> ### [`v1.2.2`](https://togithub.com/hashicorp/go-hclog/releases/tag/v1.2.2) [Compare Source](https://togithub.com/hashicorp/go-hclog/compare/v1.2.1...v1.2.2) #### What's Changed - fix various typos in comments by [@marco-m](https://togithub.com/marco-m) in [https://github.com/hashicorp/go-hclog/pull/115](https://togithub.com/hashicorp/go-hclog/pull/115) - Omit empty colon when message is empty. Fixes [#109](https://togithub.com/hashicorp/go-hclog/issues/109) by [@evanphx](https://togithub.com/evanphx) in [https://github.com/hashicorp/go-hclog/pull/116](https://togithub.com/hashicorp/go-hclog/pull/116) **Full Changelog**: hashicorp/go-hclog@v1.2.1...v1.2.2 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 3am on Monday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, click this checkbox. --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

Co-authored-by: Renovate Bot <renovate@whitesourcesoftware.com>

🤖 I have created a release *beep* *boop* --- ## [0.12.3](cloudquery/cq-provider-azure@v0.12.2...v0.12.3) (2022-08-07) ### Features * Add support for Tenants ([#412](cloudquery/cq-provider-azure#412)) ([940af7f](cloudquery/cq-provider-azure@940af7f)) ### Bug Fixes * **deps:** Update module github.com/Azure/go-autorest/autorest to v0.11.28 ([#443](cloudquery/cq-provider-azure#443)) ([bc87594](cloudquery/cq-provider-azure@bc87594)) * **deps:** Update module github.com/cloudquery/cq-gen to v0.0.7 ([#453](cloudquery/cq-provider-azure#453)) ([361d908](cloudquery/cq-provider-azure@361d908)) * **deps:** Update module github.com/cloudquery/cq-gen to v0.0.8 ([#457](cloudquery/cq-provider-azure#457)) ([7d61851](cloudquery/cq-provider-azure@7d61851)) * **deps:** Update module github.com/hashicorp/go-hclog to v1.2.2 ([#444](cloudquery/cq-provider-azure#444)) ([6c769c9](cloudquery/cq-provider-azure@6c769c9)) * **deps:** Update tubone24/update_release digest to 2146f15 ([#389](cloudquery/cq-provider-azure#389)) ([e5682fd](cloudquery/cq-provider-azure@e5682fd)) * **docs:** Some minor docs fixes ([#454](cloudquery/cq-provider-azure#454)) ([6feb015](cloudquery/cq-provider-azure@6feb015)) * **tests:** Remove terraform for Azure ExpressRoute port ([#440](cloudquery/cq-provider-azure#440)) ([7413740](cloudquery/cq-provider-azure@7413740)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

fetches check before running policy

4ad4c06

amanenk requested a review from roneli January 26, 2022 12:42

github-actions bot added the feat label Jan 26, 2022

amanenk added 2 commits January 26, 2022 14:52

added more tests

6e4d06a

added more tests

7bd5c5a

github-actions bot added feat and removed feat labels Jan 26, 2022

amanenk marked this pull request as ready for review January 28, 2022 10:19

github-actions bot added feat and removed feat labels Jan 28, 2022

disq reviewed Jan 31, 2022

View reviewed changes

pkg/client/fetch_summary/fetch_summary.go Outdated Show resolved Hide resolved

disq reviewed Jan 31, 2022

View reviewed changes

pkg/client/client.go Outdated Show resolved Hide resolved

fix

7a28410

disq reviewed Jan 31, 2022

View reviewed changes

pkg/client/fetch_summary/fetch_summary.go Outdated Show resolved Hide resolved

amanenk added 2 commits January 31, 2022 17:27

fix

c8420f5

fix

897382d

amanenk requested a review from disq January 31, 2022 17:08

disq requested changes Jan 31, 2022

View reviewed changes

naming fixes

a689ca6

amanenk requested a review from disq January 31, 2022 17:41

disq approved these changes Jan 31, 2022

View reviewed changes

roneli suggested changes Feb 1, 2022

View reviewed changes

pkg/client/fetch_summary/fetch_summary.go Outdated Show resolved Hide resolved

yevgenypats approved these changes Feb 1, 2022

View reviewed changes

pkg/policy/execute_test.go Show resolved Hide resolved

amanenk added 4 commits February 1, 2022 16:53

core migration moved to fetch summary client

dbc0d68

added migrations for timescaledb

linter fix

544d430

doc

a9bc4c5

fixed typo

b19dc9f

amanenk requested a review from roneli February 1, 2022 15:33

amanenk added 2 commits February 7, 2022 12:46

Merge remote-tracking branch 'upstream/main' into policy_fetch_summary

8cda800

# Conflicts: # pkg/client/client.go # pkg/client/fetch.go # pkg/policy/execute.go # pkg/policy/execute_test.go

merge

c6291c1

amanenk requested a review from disq February 7, 2022 13:59

package renamed

30a4a9a

amanenk force-pushed the policy_fetch_summary branch from 6542f99 to 30a4a9a Compare February 7, 2022 19:39

disq requested changes Feb 7, 2022

View reviewed changes

removed timescale migrations

29ecd47

disq requested changes Feb 8, 2022

View reviewed changes

pkg/policy/execute.go Outdated Show resolved Hide resolved

pkg/policy/execute.go Outdated Show resolved Hide resolved

pkg/client/meta_storage/fetch_summary.go Outdated Show resolved Hide resolved

amanenk and others added 4 commits February 8, 2022 15:46

Update pkg/policy/execute.go

f091b1e

Co-authored-by: Kemal <223029+disq@users.noreply.github.com>

Update pkg/policy/execute.go

9f870d5

Co-authored-by: Kemal <223029+disq@users.noreply.github.com>

Update pkg/client/meta_storage/fetch_summary.go

13ca3d2

Co-authored-by: Kemal <223029+disq@users.noreply.github.com>

tests fix

2cc1be7

amanenk requested a review from disq February 8, 2022 15:09

disq approved these changes Feb 8, 2022

View reviewed changes

amanenk added 2 commits March 7, 2022 18:10

Merge remote-tracking branch 'upstream/main' into policy_fetch_summary

45ec4eb

# Conflicts: # pkg/client/client.go # pkg/policy/execute.go # pkg/policy/execute_test.go

tests fix

79bb5f2

amanenk requested a review from yevgenypats March 8, 2022 06:00

roneli approved these changes Mar 17, 2022

View reviewed changes

disq merged commit 327398f into cloudquery:main Mar 22, 2022

This was referenced May 10, 2022

chore(main): Release v0.23.0 #678

Closed

chore(main): Release v0.23.0 #680

Closed

cq-bot mentioned this pull request Aug 9, 2022

chore: release monorepo #1167

Merged

erezrokah pushed a commit that referenced this pull request Aug 14, 2022

fix(deps): Update tubone24/update_release digest to b9bbf61 (#444)

34e989e

Co-authored-by: Renovate Bot <renovate@whitesourcesoftware.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Added checking of finished fetches before running policy#444

feat: Added checking of finished fetches before running policy#444
disq merged 21 commits intocloudquery:mainfrom
amanenk:policy_fetch_summary

amanenk commented Jan 26, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

disq left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

disq left a comment

Uh oh!

Uh oh!

yevgenypats left a comment

Uh oh!

Uh oh!

disq left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roneli left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

amanenk commented Jan 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

disq left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

disq left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yevgenypats left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

disq left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roneli left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

amanenk commented Jan 26, 2022 •

edited

Loading