-
Notifications
You must be signed in to change notification settings - Fork 26
Comparing changes
Open a pull request
base repository: cloudquery/plugin-sdk
base: 1a2a9d9
head repository: cloudquery/plugin-sdk
compare: c6655f9
- 6 commits
- 24 files changed
- 3 contributors
Commits on Nov 16, 2022
-
feat: Add option to plugin doc command to output tables as JSON (#347)
Fixes #309 This adds a `--format json` option to the source plugin `doc` command, which allows all tables to be dumped as a single JSON file. This can then be further parsed with `jq` for various purposes (such as comparing two versions of a plugin to see what changed). ### Example: List first 5 top-level tables of AWS plugin ``` ./aws doc --format json . ``` ``` $ cat __tables.json | jq -r '.[].name' | head -n 5 aws_accessanalyzer_analyzers aws_acm_certificates aws_apigateway_api_keys aws_apigateway_client_certificates aws_apigateway_domain_names ```
Configuration menu - View commit details
-
Copy full SHA for c1b4240 - Browse repository at this point
Copy the full SHA c1b4240View commit details
Commits on Nov 20, 2022
-
fix: Fix Destination testing suite (#417)
Seems we weren't running some tests. I had to invert the variables which actually also make sense so they will be enabled by default. Let's say we add a new test this will detect if a plugin is not supporting a new mode.
Configuration menu - View commit details
-
Copy full SHA for 4771efa - Browse repository at this point
Copy the full SHA 4771efaView commit details
Commits on Nov 21, 2022
-
chore: Add source plugin benchmark (#415)
This adds a source plugin benchmark that runs in-memory. It allows for different scenarios to be set up, but currently comes with only two: one scenario without child tables, and one with child tables. Here are the results on my machine: ``` BenchmarkDefaultConcurrency BenchmarkDefaultConcurrency-8 1 11637 resources/s 12626 targetResources/s 73510696 B/op 1032412 allocs/op BenchmarkTablesWithChildrenDefaultConcurrency BenchmarkTablesWithChildrenDefaultConcurrency-8 1 544.6 resources/s 40606 targetResources/s 461292456 B/op 6689992 allocs/op PASS ``` The output is different from normal Go benchmarks, and I feel it better fits our use-case. It shows the actual `resources/s`, similar to what the CLI would show. It also shows a `targetResources/s`, an attempt to calculate the theoretical best value we could aim for. As the above results show, we are doing pretty well in the case with no child relations, but we are leaving a lot of room for improvement when child relations are involved. In theory, these could be sped up by almost 100x (this might use a lot of memory though, of course). In both cases it seems like we could also be doing better in terms of number of allocs and B/op, but that is pending further research. Closes cloudquery/cloudquery#3721
Configuration menu - View commit details
-
Copy full SHA for 962f3f7 - Browse repository at this point
Copy the full SHA 962f3f7View commit details -
feat: Support ${file:./path} expansion in spec (#418)
Per user request. we had a similar version of that in v0 but now it is generic and available to all specs and not only to a specific destination.
Configuration menu - View commit details
-
Copy full SHA for 58d7c44 - Browse repository at this point
Copy the full SHA 58d7c44View commit details -
fix: Increase GRPC message size limit to 50MiB (#419)
The default was 4MiB, and users have reported running into this limit on occasion. If 50MiB isn't enough, we'll probably need to implement some logic to drop messages at that point since they'll be too big to be useful for many databases anyway. Let's see if increasing the limit to 50MiB solves the problem first though. cloudquery/cloudquery#4834
Configuration menu - View commit details
-
Copy full SHA for a54c6ea - Browse repository at this point
Copy the full SHA a54c6eaView commit details -
chore(main): Release v1.6.0 (#412)
🤖 I have created a release *beep* *boop* --- ## [1.6.0](v1.5.3...v1.6.0) (2022-11-21) ### Features * Add option to plugin doc command to output tables as JSON ([#347](#347)) ([c1b4240](c1b4240)) * Support ${file:./path} expansion in spec ([#418](#418)) ([58d7c44](58d7c44)) ### Bug Fixes * Fix Destination testing suite ([#417](#417)) ([4771efa](4771efa)) * Increase GRPC message size limit to 50MiB ([#419](#419)) ([a54c6ea](a54c6ea)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Configuration menu - View commit details
-
Copy full SHA for c6655f9 - Browse repository at this point
Copy the full SHA c6655f9View commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff 1a2a9d9...c6655f9