Skip to content

feat: Separate Queued Tables from In Progress Tables#920

Merged
kodiakhq[bot] merged 5 commits intocloudquery:mainfrom
bbernays:seperate-queued
May 31, 2023
Merged

feat: Separate Queued Tables from In Progress Tables#920
kodiakhq[bot] merged 5 commits intocloudquery:mainfrom
bbernays:seperate-queued

Conversation

@bbernays
Copy link
Copy Markdown
Contributor

Summary

When users limit the concurrency of their sync, tables that have not been started will appear to be "in-progress" even though they are waiting to start...

This PR separates queued tables from in-progress by defining in-progress tables as those that have no end-time, but do have a non-zero start time. queued tables are those tables that have a zero start and end time

@bbernays bbernays requested a review from yevgenypats as a code owner May 30, 2023 18:01
@bbernays bbernays changed the title feat: Separate Queued Tables from InProgress feat: Separate Queued Tables from In Progress Tables May 30, 2023
@github-actions github-actions bot added feat and removed feat labels May 30, 2023
@github-actions
Copy link
Copy Markdown

github-actions bot commented May 30, 2023

⏱️ Benchmark results

Comparing with 72e1d49

  • DefaultConcurrencyDFS-2 resources/s: 10,505 ⬆️ 8.41% increase vs. 72e1d49
  • DefaultConcurrencyRoundRobin-2 resources/s: 12,284 ⬇️ 1.74% decrease vs. 72e1d49
  • Glob-2 ns/op: 200.2 ⬇️ 19.28% decrease vs. 72e1d49
  • TablesWithChildrenDFS-2 resources/s: 29,006 ⬆️ 14.23% increase vs. 72e1d49
  • TablesWithChildrenRoundRobin-2 resources/s: 26,728 ⬆️ 7.46% increase vs. 72e1d49
  • TablesWithRateLimitingDFS-2 resources/s: 28.26 ⬆️ 0.28% increase vs. 72e1d49
  • TablesWithRateLimitingRoundRobin-2 resources/s: 851.7 ⬇️ 0.41% decrease vs. 72e1d49

@codecov
Copy link
Copy Markdown

codecov bot commented May 30, 2023

Codecov Report

Patch coverage: 72.72% and project coverage change: +0.11 🎉

Comparison is base (72e1d49) 47.89% compared to head (d95d0cc) 48.00%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #920      +/-   ##
==========================================
+ Coverage   47.89%   48.00%   +0.11%     
==========================================
  Files          55       55              
  Lines        5226     5245      +19     
==========================================
+ Hits         2503     2518      +15     
- Misses       2460     2464       +4     
  Partials      263      263              
Impacted Files Coverage Δ
plugins/source/scheduler.go 52.54% <0.00%> (-1.85%) ⬇️
plugins/source/metrics.go 70.00% <100.00%> (+3.91%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@kodiakhq kodiakhq bot merged commit dcb5d26 into cloudquery:main May 31, 2023
@bbernays bbernays deleted the seperate-queued branch May 31, 2023 15:18
kodiakhq bot pushed a commit that referenced this pull request May 31, 2023
🤖 I have created a release *beep* *boop*
---


## [3.8.0](v3.7.0...v3.8.0) (2023-05-31)


### Features

* Add the names of tables to the periodic logger ([#738](#738)) ([72e1d49](72e1d49))
* Separate Queued Tables from In Progress Tables ([#920](#920)) ([dcb5d26](dcb5d26))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants