Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: cloudquery/plugin-sdk
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.7.0
Choose a base ref
...
head repository: cloudquery/plugin-sdk
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.8.0
Choose a head ref
  • 2 commits
  • 3 files changed
  • 2 contributors

Commits on Nov 23, 2022

  1. feat: Resolve table relations in parallel (#423) 

    This changes the resolver for table relations to work concurrently, like parent tables. To do so safely and without the risk of deadlock, we instantiate one semaphore per depth level. (The size of the semaphore is decreased logarithmically for each depth.)

This change will only improve performance for tables with child relations.

To compare the performance before and after, I used the benchmarks in #415

## Before
```
goos: darwin
goarch: arm64
pkg: github.com/cloudquery/plugin-sdk/plugins
BenchmarkDefaultConcurrency-8                     	       1	     11957 resources/s	     12626 targetResources/s	73597280 B/op	 1032425 allocs/op
BenchmarkTablesWithChildrenDefaultConcurrency-8   	       1	       545.9 resources/s	     40606 targetResources/s	461622584 B/op	 6690105 allocs/op
PASS
ok  	github.com/cloudquery/plugin-sdk/plugins	150.596s
```

## After
```
goos: darwin
goarch: arm64
pkg: github.com/cloudquery/plugin-sdk/plugins
BenchmarkDefaultConcurrency-8                     	       1	     11373 resources/s	     12626 targetResources/s	73692608 B/op	 1033614 allocs/op
BenchmarkTablesWithChildrenDefaultConcurrency-8   	       1	     30162 resources/s	     40606 targetResources/s	464285672 B/op	 6697508 allocs/op
PASS
ok  	github.com/cloudquery/plugin-sdk/plugins	6.241s
```

## Analysis

This change focuses on the `BenchmarkTablesWithChildrenDefaultConcurrency` case. The change improves `resources/s` from 545.9 to 30162, an improvement of about 55x. Memory and allocs are mostly the same.

The small downward change in `resources/s` in the `BenchmarkDefaultConcurrency` case is likely due to the few additional allocs needed. 

Closes #358
Copy of #416 after it was reverted to schedule its release for another time.
    @hermanschaaf
    hermanschaaf authored Nov 23, 2022
    Configuration menu
    Copy the full SHA
    ede04b7 View commit details
    Browse the repository at this point in the history

  2. chore(main): Release v1.8.0 (#427) 

    🤖 I have created a release *beep* *boop*
---


## [1.8.0](v1.7.0...v1.8.0) (2022-11-23)


### Features

* Resolve table relations in parallel ([#423](#423)) ([ede04b7](ede04b7))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
    @cq-bot
    cq-bot authored Nov 23, 2022
    Configuration menu
    Copy the full SHA
    576d5f0 View commit details
    Browse the repository at this point in the history
Loading