Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: cloudquery/plugin-sdk
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: f6818cb
Choose a base ref
...
head repository: cloudquery/plugin-sdk
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 5da64aa
Choose a head ref
  • 3 commits
  • 14 files changed
  • 3 contributors

Commits on Jan 4, 2023

  1. feat: Add scheduler option and introduce Round Robin scheduler (#545)

    This adds an optional Scheduler option to source configs. It still
    defaults to DFS, our current scheduler. However, for cases like GCP,
    alternative scheduling algorithms may be beneficial for large
    deployments (as explained in
    cloudquery/cloudquery#5389). For this reason,
    this change also introduces a new (currently opt-in only) scheduler
    called `round-robin`.
    
    A new benchmark scenario has been added to showcase when round robin
    really shines. In this scenario, strict global rate limits mean that
    waiting for each table to finish all of its clients is not the most
    efficient schedule. Instead, a schedule that interleaves the clients for
    all tables (as round robin does) performs near optimally, and the
    benchmarks show this difference:
    
    ```
    BenchmarkDefaultConcurrencyDFS
    BenchmarkDefaultConcurrencyDFS-8              	       1	     10835 resources/s	73788592 B/op	 1057929 allocs/op
    BenchmarkDefaultConcurrencyRoundRobin
    BenchmarkDefaultConcurrencyRoundRobin-8       	       2	     11276 resources/s	72121956 B/op	 1054439 allocs/op
    BenchmarkTablesWithChildrenDFS
    BenchmarkTablesWithChildrenDFS-8              	       1	     31900 resources/s	465282072 B/op	 6859524 allocs/op
    BenchmarkTablesWithChildrenRoundRobin
    BenchmarkTablesWithChildrenRoundRobin-8       	       1	     33276 resources/s	463475144 B/op	 6856392 allocs/op
    BenchmarkTablesWithRateLimitingDFS
    BenchmarkTablesWithRateLimitingDFS-8          	       1	        28.30 resources/s	 9098456 B/op	  132129 allocs/op
    BenchmarkTablesWithRateLimitingRoundRobin
    BenchmarkTablesWithRateLimitingRoundRobin-8   	       1	       821.5 resources/s	 8213856 B/op	  112748 allocs/op
    PASS
    ```
    
    In the `BenchmarkTablesWithRateLimiting` case, Round Robin completes in
    just over 1s, DFS completes in 35s.
    
    ## Notes for Reviewers
    
    - No functional changes to the DFS scheduler were made in this PR, but
    some functions were moved to a different file so they can be re-used by
    Round Robin.
    - I'm also removing `targetResources/s` from the benchmarks in this PR.
    They were useful once, but with rate limiting scenarios it's becoming
    too difficult to maintain accuracy.
    hermanschaaf authored Jan 4, 2023
    1 Configuration menu
    Copy the full SHA
    d89a911 View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2023

  1. feat: Add unwrap option to transformations (#573)

    Follow-up on transformation. Adding unwrap option as needed in some cases where we create our own wrapper struct.
    
    This also moves the tests from codegen.
    yevgenypats authored Jan 5, 2023
    1 Configuration menu
    Copy the full SHA
    a17ee4b View commit details
    Browse the repository at this point in the history
  2. chore(main): Release v1.19.0 (#572)

    🤖 I have created a release *beep* *boop*
    ---
    
    
    ## [1.19.0](v1.18.0...v1.19.0) (2023-01-05)
    
    
    ### Features
    
    * Add scheduler option and introduce Round Robin scheduler ([#545](#545)) ([d89a911](d89a911))
    * Add unwrap option to transformations ([#573](#573)) ([a17ee4b](a17ee4b))
    
    ---
    This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
    cq-bot authored Jan 5, 2023
    1 Configuration menu
    Copy the full SHA
    5da64aa View commit details
    Browse the repository at this point in the history
Loading