Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: cloudquery/plugin-sdk
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.5.3
Choose a base ref
...
head repository: cloudquery/plugin-sdk
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.6.0
Choose a head ref
  • 6 commits
  • 24 files changed
  • 3 contributors

Commits on Nov 16, 2022

  1. feat: Add option to plugin doc command to output tables as JSON (#347)

    Fixes #309
    
    This adds a `--format json` option to the source plugin `doc` command, which allows all tables to be dumped as a single JSON file. This can then be further parsed with `jq` for various purposes (such as comparing two versions of a plugin to see what changed). 
    
    ### Example: List first 5 top-level tables of AWS plugin
    
    ```
    ./aws doc --format json .
    ```
    
    ```
    $ cat __tables.json | jq -r '.[].name' | head -n 5
    aws_accessanalyzer_analyzers
    aws_acm_certificates
    aws_apigateway_api_keys
    aws_apigateway_client_certificates
    aws_apigateway_domain_names
    ```
    hermanschaaf authored Nov 16, 2022
    Configuration menu
    Copy the full SHA
    c1b4240 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2022

  1. fix: Fix Destination testing suite (#417)

    Seems we weren't running some tests. I had to invert the variables which actually also make sense so they will be enabled by default. Let's say we add a new test this will detect if a plugin is not supporting a new mode.
    yevgenypats authored Nov 20, 2022
    Configuration menu
    Copy the full SHA
    4771efa View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2022

  1. chore: Add source plugin benchmark (#415)

    This adds a source plugin benchmark that runs in-memory. It allows for different scenarios to be set up, but currently comes with only two: one scenario without child tables, and one with child tables. 
    
    Here are the results on my machine:
    
    ```
    BenchmarkDefaultConcurrency
    BenchmarkDefaultConcurrency-8                     	       1	     11637 resources/s	     12626 targetResources/s	73510696 B/op	 1032412 allocs/op
    BenchmarkTablesWithChildrenDefaultConcurrency
    BenchmarkTablesWithChildrenDefaultConcurrency-8   	       1	       544.6 resources/s	     40606 targetResources/s	461292456 B/op	 6689992 allocs/op
    PASS
    ```
    
    The output is different from normal Go benchmarks, and I feel it better fits our use-case. It shows the actual `resources/s`, similar to what the CLI would show. It also shows a `targetResources/s`, an attempt to calculate the theoretical best value we could aim for. 
    
    As the above results show, we are doing pretty well in the case with no child relations, but we are leaving a lot of room for improvement when child relations are involved. In theory, these could be sped up by almost 100x (this might use a lot of memory though, of course). In both cases it seems like we could also be doing better in terms of number of allocs and B/op, but that is pending further research.
    
    Closes cloudquery/cloudquery#3721
    hermanschaaf authored Nov 21, 2022
    Configuration menu
    Copy the full SHA
    962f3f7 View commit details
    Browse the repository at this point in the history
  2. feat: Support ${file:./path} expansion in spec (#418)

    Per user request. we had a similar version of that in v0 but now it is generic and available to all specs and not only to a specific destination.
    yevgenypats authored Nov 21, 2022
    Configuration menu
    Copy the full SHA
    58d7c44 View commit details
    Browse the repository at this point in the history
  3. fix: Increase GRPC message size limit to 50MiB (#419)

    The default was 4MiB, and users have reported running into this limit on occasion. If 50MiB isn't enough, we'll probably need to implement some logic to drop messages at that point since they'll be too big to be useful for many databases anyway. Let's see if increasing the limit to 50MiB solves the problem first though.
    
    cloudquery/cloudquery#4834
    hermanschaaf authored Nov 21, 2022
    Configuration menu
    Copy the full SHA
    a54c6ea View commit details
    Browse the repository at this point in the history
  4. chore(main): Release v1.6.0 (#412)

    🤖 I have created a release *beep* *boop*
    ---
    
    
    ## [1.6.0](v1.5.3...v1.6.0) (2022-11-21)
    
    
    ### Features
    
    * Add option to plugin doc command to output tables as JSON ([#347](#347)) ([c1b4240](c1b4240))
    * Support ${file:./path} expansion in spec ([#418](#418)) ([58d7c44](58d7c44))
    
    
    ### Bug Fixes
    
    * Fix Destination testing suite ([#417](#417)) ([4771efa](4771efa))
    * Increase GRPC message size limit to 50MiB ([#419](#419)) ([a54c6ea](a54c6ea))
    
    ---
    This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
    cq-bot authored Nov 21, 2022
    Configuration menu
    Copy the full SHA
    c6655f9 View commit details
    Browse the repository at this point in the history
Loading