fix: Don't log start-and-finish of relational tables#459
Merged
shimonp21 merged 1 commit intocloudquery:mainfrom Dec 3, 2022
shimonp21:fix_less_spammy_logs
Merged
fix: Don't log start-and-finish of relational tables#459shimonp21 merged 1 commit intocloudquery:mainfrom shimonp21:fix_less_spammy_logs
shimonp21 merged 1 commit intocloudquery:mainfrom
shimonp21:fix_less_spammy_logs
Conversation
Related to: cloudquery/cloudquery#5273 Logging every start and finish of relational tables fetches, means that the number of logs depends on the number of [parent] resources. for example, logs for fetching `aws_rds_*` (where there are many `aws_rds_engine_versions` resources, with `aws_rds_cluster_parameters` relations). All the relational-fetch logs are identical: ``` 2022-12-03T09:00:19Z INF fetch table finished client=615713231484:eu-north-1 errors=0 module=aws-src resources=5576 table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF table resolver started client=615713231484:eu-north-1 module=aws-src table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF fetch table finished client=615713231484:eu-north-1 errors=0 module=aws-src resources=5576 table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF table resolver started client=615713231484:eu-north-1 module=aws-src table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF fetch table finished client=615713231484:eu-north-1 errors=0 module=aws-src resources=5576 table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF table resolver started client=615713231484:eu-north-1 module=aws-src table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF fetch table finished client=615713231484:eu-north-1 errors=0 module=aws-src resources=5576 table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF table resolver started client=615713231484:eu-north-1 module=aws-src table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF fetch table finished client=615713231484:eu-north-1 errors=0 module=aws-src resources=5576 table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF table resolver started client=615713231484:eu-north-1 module=aws-src table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF fetch table finished client=615713231484:eu-north-1 errors=0 module=aws-src resources=5576 table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF table resolver started client=615713231484:eu-north-1 module=aws-src table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF fetch table finished client=615713231484:eu-north-1 errors=0 module=aws-src resources=5576 table=aws_rds_cluster_parameters 2022-12-03T09:00:19Z INF table resolver started client=615713231484:eu-north-1 module=aws-src table=aws_rds_cluster_parameters ``` This PR reduces the logs from 11700 lines (1.5MB) to 1300 lines (144 KB).
⏱️ Benchmark resultsComparing with 538211c
|
yevgenypats
approved these changes
Dec 3, 2022
Contributor
|
It looks ok but it does hide some information also Im curious why it logs the same line as every table can have different number of resources. Maybe we can sum up better the child tables or maybe those are not needed at all if we have metrics. |
Member
|
Is there any other kind of progress shown in the logs when fetching relations? Otherwise users might think CloudQuery hangs. I think this change is good, but we might need to add another indicator when fetching relations (if we don't have one already) |
kodiakhq bot
pushed a commit
that referenced
this pull request
Dec 6, 2022
🤖 I have created a release *beep* *boop* --- ## [1.11.0](v1.10.0...v1.11.0) (2022-12-05) ### Features * Add Support for net.IP in Faker ([#445](#445)) ([2deced1](2deced1)) ### Bug Fixes * **deps:** Update golang.org/x/exp digest to 6ab00d0 ([#449](#449)) ([b981e33](b981e33)) * **deps:** Update module github.com/avast/retry-go/v4 to v4.3.1 ([#450](#450)) ([e4116f1](e4116f1)) * **deps:** Update module google.golang.org/grpc to v1.51.0 ([#451](#451)) ([538211c](538211c)) * Don't log start-and-finish of relational tables ([#459](#459)) ([4d6eeca](4d6eeca)) * Sync Metrics for Long running plugins ([#455](#455)) ([3fecc61](3fecc61)) * TablesForSpec should only return top-level tables ([#456](#456)) ([ab7ca97](ab7ca97)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to: cloudquery/cloudquery#5273
Logging every start and finish of relational tables fetches, means that the number of logs depends on the number of [parent] resources.
for example, logs for fetching
aws_rds_*(where there are manyaws_rds_engine_versionsresources, withaws_rds_cluster_parametersrelations).All the relational-fetch logs are identical:
This PR reduces the logs from 11700 lines (1.5MB) to 1300 lines (144 KB).
Summary
Use the following steps to ensure your PR is ready to be reviewed
go fmtto format your code 🖊golangci-lint run🚨 (install golangci-lint here)