Skip to content

feat: Add the names of tables to the periodic logger#738

Merged
kodiakhq[bot] merged 6 commits intocloudquery:mainfrom
shimonp21:feat_log_tables
May 30, 2023
Merged

feat: Add the names of tables to the periodic logger#738
kodiakhq[bot] merged 6 commits intocloudquery:mainfrom
shimonp21:feat_log_tables

Conversation

@shimonp21
Copy link
Copy Markdown
Contributor

@shimonp21 shimonp21 commented Mar 27, 2023

if the number of running tables is small (less than e.g. 30), log all their names in the periodic 'sync in progress' logs.

Example log lines:

2023-03-27T08:11:52Z INF Sync in progress module=gcp-src num_in_progress_tables=160 total_errors=0 total_panics=0 total_resources=81
2023-03-27T08:11:53Z INF Sync in progress module=gcp-src num_in_progress_tables=88 total_errors=2 total_panics=0 total_resources=1240
2023-03-27T10:48:27Z INF Sync in progress in_progress_tables=["gcp_compute_zones"] module=gcp-src num_in_progress_tables=1 total_errors=0 total_panics=0 total_resources=5837

Summary


Use the following steps to ensure your PR is ready to be reviewed

  • Read the contribution guidelines 🧑‍🎓
  • Run go fmt to format your code 🖊
  • Lint your changes via golangci-lint run 🚨 (install golangci-lint here)
  • Update or add tests 🧪
  • Ensure the status checks below are successful ✅

@shimonp21 shimonp21 requested a review from yevgenypats as a code owner March 27, 2023 11:11
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 27, 2023

⏱️ Benchmark results

Comparing with f5f7cd7

  • DefaultConcurrencyDFS-2 resources/s: 10,334 ⬆️ 0.60% increase vs. f5f7cd7
  • DefaultConcurrencyRoundRobin-2 resources/s: 12,377 ⬆️ 1.96% increase vs. f5f7cd7
  • Glob-2 ns/op: 260.3 ⬆️ 26.28% increase vs. f5f7cd7
  • TablesWithChildrenDFS-2 resources/s: 24,351 ⬇️ 17.93% decrease vs. f5f7cd7
  • TablesWithChildrenRoundRobin-2 resources/s: 23,577 ⬇️ 22.04% decrease vs. f5f7cd7
  • TablesWithRateLimitingDFS-2 resources/s: 28.37 ⬇️ 1.16% decrease vs. f5f7cd7
  • TablesWithRateLimitingRoundRobin-2 resources/s: 822.8 ⬇️ 4.68% decrease vs. f5f7cd7

Copy link
Copy Markdown
Member

@erezrokah erezrokah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @shimonp21, added a couple of comments

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 27, 2023

Codecov Report

Patch coverage: 83.01% and project coverage change: +0.12 🎉

Comparison is base (9d83546) 47.52% compared to head (e47fbcd) 47.65%.

❗ Current head e47fbcd differs from pull request most recent head d0929e2. Consider uploading reports for the commit d0929e2 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #738      +/-   ##
==========================================
+ Coverage   47.52%   47.65%   +0.12%     
==========================================
  Files          55       55              
  Lines        5176     5078      -98     
==========================================
- Hits         2460     2420      -40     
+ Misses       2453     2408      -45     
+ Partials      263      250      -13     
Impacted Files Coverage Δ
plugins/source/scheduler.go 54.38% <0.00%> (-3.56%) ⬇️
plugins/source/metrics.go 66.08% <100.00%> (+17.40%) ⬆️
plugins/source/scheduler_dfs.go 64.83% <100.00%> (+0.79%) ⬆️

... and 17 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@erezrokah erezrokah changed the title Fix: Add the names of tables to the periodic logger feat: Add the names of tables to the periodic logger Mar 27, 2023
@github-actions github-actions bot added the feat label Mar 27, 2023
Copy link
Copy Markdown
Member

@erezrokah erezrokah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow up @shimonp21, added a few more comments

@shimonp21 shimonp21 requested a review from erezrokah March 27, 2023 12:23
Copy link
Copy Markdown
Member

@erezrokah erezrokah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving pending @disq comments

if the number of running tables is small (less than e.g. 30),
log all their names in the periodic 'sync in progress' logs.
@shimonp21 shimonp21 requested a review from disq March 28, 2023 05:04
@kodiakhq kodiakhq bot merged commit 72e1d49 into cloudquery:main May 30, 2023
kodiakhq bot pushed a commit that referenced this pull request May 31, 2023
🤖 I have created a release *beep* *boop*
---


## [3.8.0](v3.7.0...v3.8.0) (2023-05-31)


### Features

* Add the names of tables to the periodic logger ([#738](#738)) ([72e1d49](72e1d49))
* Separate Queued Tables from In Progress Tables ([#920](#920)) ([dcb5d26](dcb5d26))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants