Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: cloudquery/plugin-sdk
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.6.0
Choose a base ref
...
head repository: cloudquery/plugin-sdk
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.7.0
Choose a head ref
  • 4 commits
  • 7 files changed
  • 2 contributors

Commits on Nov 21, 2022

  1. feat: Resolve table relations in parallel (#416)

    This changes the resolver for table relations to work concurrently, like parent tables. To do so safely and without the risk of deadlock, we instantiate one semaphore per depth level. (The size of the semaphore is decreased logarithmically for each depth.)
    
    This change will only improve performance for tables with child relations.
    
    To compare the performance before and after, I used the benchmarks in #415
    
    ## Before
    ```
    goos: darwin
    goarch: arm64
    pkg: github.com/cloudquery/plugin-sdk/plugins
    BenchmarkDefaultConcurrency-8                     	       1	     11957 resources/s	     12626 targetResources/s	73597280 B/op	 1032425 allocs/op
    BenchmarkTablesWithChildrenDefaultConcurrency-8   	       1	       545.9 resources/s	     40606 targetResources/s	461622584 B/op	 6690105 allocs/op
    PASS
    ok  	github.com/cloudquery/plugin-sdk/plugins	150.596s
    ```
    
    ## After
    ```
    goos: darwin
    goarch: arm64
    pkg: github.com/cloudquery/plugin-sdk/plugins
    BenchmarkDefaultConcurrency-8                     	       1	     11373 resources/s	     12626 targetResources/s	73692608 B/op	 1033614 allocs/op
    BenchmarkTablesWithChildrenDefaultConcurrency-8   	       1	     30162 resources/s	     40606 targetResources/s	464285672 B/op	 6697508 allocs/op
    PASS
    ok  	github.com/cloudquery/plugin-sdk/plugins	6.241s
    ```
    
    ## Analysis
    
    This change focuses on the `BenchmarkTablesWithChildrenDefaultConcurrency` case. The change improves `resources/s` from 545.9 to 30162, an improvement of about 55x. Memory and allocs are mostly the same.
    
    The small downward change in `resources/s` in the `BenchmarkDefaultConcurrency` case is likely due to the few additional allocs needed. 
    
    Closes #358
    hermanschaaf authored Nov 21, 2022
    Configuration menu
    Copy the full SHA
    aadbde9 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2022

  1. fix: Skip very large gRPC messages, log when it happens (#421)

    Context: cloudquery/cloudquery#4834
    
    This change:
     - bumps the gRPC max message size from 50MiB to 100MiB
     - logs the name of the table to Sentry when we see a message over 50MiB
     - logs to the user if we drop a message because it exceeds 100MiB
    hermanschaaf authored Nov 22, 2022
    Configuration menu
    Copy the full SHA
    0874d58 View commit details
    Browse the repository at this point in the history
  2. fix: Revert "feat: Resolve table relations in parallel" (#422)

    Reverts #416
    
    Nothing wrong with it as far as we know, but we want to release it separately from #421
    hermanschaaf authored Nov 22, 2022
    Configuration menu
    Copy the full SHA
    655a04b View commit details
    Browse the repository at this point in the history
  3. chore(main): Release v1.7.0 (#420)

    🤖 I have created a release *beep* *boop*
    ---
    
    
    ## [1.7.0](v1.6.0...v1.7.0) (2022-11-22)
    
    ### Bug Fixes
    
    * Skip very large gRPC messages, log when it happens ([#421](#421)) ([0874d58](0874d58))
    
    ---
    This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
    cq-bot authored Nov 22, 2022
    Configuration menu
    Copy the full SHA
    c43b213 View commit details
    Browse the repository at this point in the history
Loading