feat(aws): Add Support for fully customizable Cost Explorer Data syncing by bbernays · Pull Request #11185 · cloudquery/cloudquery

bbernays · 2023-06-01T18:08:49Z

Summary

closes #10805

…-table

github-actions · 2023-06-01T18:09:13Z

This PR has the following changes to source plugin(s) tables:

Table aws_alpha_costexplorer_cost_custom was added

plugins/source/aws/resources/services/costexplorer/cost_custom.go

plugins/source/aws/client/tableoptions/costexplorer.go

Co-authored-by: Kemal <223029+disq@users.noreply.github.com>

…-table

plugins/source/aws/client/tableoptions/table_options.go

…-table

disq · 2023-06-13T16:34:23Z

plugins/source/aws/resources/services/costexplorer/cost_custom.go

+)
+
+func CustomCost() *schema.Table {
+	tableName := "aws_costexplorer_cost_custom"


Suggested change

tableName := "aws_costexplorer_cost_custom"

tableName := "aws_alpha_costexplorer_cost_custom"

disq · 2023-06-13T16:34:47Z

plugins/source/aws/resources/services/costexplorer/cost_custom.go

+		Resolver: fetchCustom,
+		Title:    "AWS Cost Explorer costs based on custom inputs",
+		Description: `https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_GetCostAndUsage.html
+To sync this table you must set the 'use_paid_apis' option to 'true' in the AWS provider configuration as well as specify the request parameters in the 'table_options' attribute. `,


Suggested change

To sync this table you must set the 'use_paid_apis' option to 'true' in the AWS provider configuration as well as specify the request parameters in the 'table_options' attribute. `,

To sync this table you must set the 'use_paid_apis' option to 'true' in the AWS provider configuration as well as specify the request parameters in the 'table_options' attribute.

Please note that this table is considered **alpha** (experimental) and may have breaking changes or be removed in the future.

`,

plugins/source/aws/resources/services/costexplorer/cost_custom.go

…-table

…ys/cloudquery into custom-costexplorer-table

disq

LGTM with nits

plugins/source/aws/resources/services/costexplorer/cost_custom.go

Co-authored-by: Kemal <223029+disq@users.noreply.github.com>

bbernays · 2023-06-15T12:43:45Z

/gen sha=0567a4bb29c5189c798a79cf03bb53e2eb14d5d0 plugin=aws

🤖 I have created a release *beep* *boop* --- ## [18.4.0](plugins-source-aws-v18.3.0...plugins-source-aws-v18.4.0) (2023-06-15) ### This Release has the Following Changes to Tables - Table `aws_alpha_cloudwatch_metric_statistics` was added - Table `aws_alpha_cloudwatch_metrics` was added - Table `aws_alpha_costexplorer_cost_custom` was added - Table `aws_applicationautoscaling_scalable_targets`: column added with name `scalable_target_arn` and type `utf8` - Table `aws_appsync_graphql_apis`: column added with name `api_type` and type `utf8` - Table `aws_appsync_graphql_apis`: column added with name `dns` and type `json` - Table `aws_appsync_graphql_apis`: column added with name `merged_api_execution_role_arn` and type `utf8` - Table `aws_appsync_graphql_apis`: column added with name `owner_contact` and type `utf8` - Table `aws_appsync_graphql_apis`: column added with name `owner` and type `utf8` - Table `aws_appsync_graphql_apis`: column added with name `visibility` and type `utf8` - Table `aws_athena_work_group_query_executions`: column added with name `substatement_type` and type `utf8` - Table `aws_backup_protected_resources` was added - Table `aws_backup_vault_recovery_points`: column added with name `resource_name` and type `utf8` - Table `aws_cloudformation_stack_sets`: column added with name `regions` and type `list<item: utf8, nullable>` - Table `aws_cloudwatchlogs_log_groups`: column added with name `inherited_properties` and type `list<item: utf8, nullable>` - Table `aws_computeoptimizer_ebs_volume_recommendations`: column added with name `tags` and type `json` - Table `aws_computeoptimizer_ec2_instance_recommendations`: column added with name `external_metric_status` and type `json` - Table `aws_computeoptimizer_ec2_instance_recommendations`: column added with name `instance_state` and type `utf8` - Table `aws_computeoptimizer_ec2_instance_recommendations`: column added with name `tags` and type `json` - Table `aws_computeoptimizer_ecs_service_recommendations`: column added with name `tags` and type `json` - Table `aws_computeoptimizer_lambda_function_recommendations`: column added with name `tags` and type `json` - Table `aws_dynamodb_tables`: column added with name `deletion_protection_enabled` and type `bool` - Table `aws_ec2_hosts`: column added with name `host_maintenance` and type `utf8` - Table `aws_ec2_instance_connect_endpoints` was added - Table `aws_ec2_instances`: column added with name `current_instance_boot_mode` and type `utf8` - Table `aws_elasticache_replication_groups`: column added with name `cluster_mode` and type `utf8` - Table `aws_emr_cluster_instance_fleets`: column added with name `resize_specifications` and type `json` - Table `aws_frauddetector_event_types`: column added with name `event_orchestration` and type `json` - Table `aws_glue_database_tables`: column added with name `federated_table` and type `json` - Table `aws_glue_databases`: column added with name `federated_database` and type `json` - Table `aws_guardduty_detectors`: column added with name `features` and type `json` - Table `aws_iot_jobs`: column added with name `destination_package_versions` and type `list<item: utf8, nullable>` - Table `aws_iot_jobs`: column added with name `scheduled_job_rollouts` and type `json` - Table `aws_kafka_cluster_operations`: column added with name `vpc_connection_info` and type `json` - Table `aws_lambda_function_event_source_mappings`: column added with name `document_db_event_source_config` and type `json` - Table `aws_lightsail_disks`: column added with name `auto_mount_status` and type `utf8` - Table `aws_mwaa_environments`: column added with name `startup_script_s3_object_version` and type `utf8` - Table `aws_mwaa_environments`: column added with name `startup_script_s3_path` and type `utf8` - Table `aws_neptune_clusters`: column added with name `global_cluster_identifier` and type `utf8` - Table `aws_neptune_clusters`: column added with name `pending_modified_values` and type `json` - Table `aws_networkfirewall_firewall_policies`: column added with name `policy_variables` and type `json` - Table `aws_ram_resource_share_permissions`: column added with name `feature_set` and type `utf8` - Table `aws_ram_resource_share_permissions`: column added with name `permission_type` and type `utf8` - Table `aws_ram_resource_share_permissions`: column added with name `tags` and type `json` - Table `aws_rds_cluster_snapshots`: column added with name `storage_type` and type `utf8` - Table `aws_rds_clusters`: column added with name `io_optimized_next_allowed_modification_time` and type `timestamp[us, tz=UTC]` - Table `aws_rds_instances`: column added with name `read_replica_source_db_cluster_identifier` and type `utf8` - Table `aws_resiliencehub_app_version_resource_mappings`: column added with name `eks_source_name` and type `utf8` - Table `aws_resiliencehub_app_version_resources`: column added with name `additional_info` and type `json` - Table `aws_resiliencehub_app_version_resources`: column added with name `excluded` and type `bool` - Table `aws_resiliencehub_app_version_resources`: column added with name `parent_resource_name` and type `utf8` - Table `aws_resiliencehub_app_version_resources`: column added with name `source_type` and type `utf8` - Table `aws_sagemaker_models`: column added with name `deployment_recommendation` and type `json` - Table `aws_securityhub_hubs`: column added with name `control_finding_generator` and type `utf8` - Table `aws_timestream_tables`: column added with name `schema` and type `json` - Table `aws_wafv2_web_acls`: column added with name `association_config` and type `json` ### Features * **aws:** Add `aws_alpha_cloudwatch_metric*` tables to fetch Cloudwatch Metrics and statistics ([#11402](#11402)) ([07b76d2](07b76d2)) * **aws:** Add Support for EC2 Instance Connect Endpoints ([#11531](#11531)) ([73ae77b](73ae77b)) * **aws:** Add Support for fully customizable Cost Explorer Data syncing ([#11185](#11185)) ([04fd769](04fd769)), closes [#10805](#10805) * **aws:** Add support for Protected Resources in AWS Backup ([#11522](#11522)) ([12a1bc0](12a1bc0)) * **aws:** Table Options Validation ([#11548](#11548)) ([ea858fc](ea858fc)) ### Bug Fixes * **aws:** Upgrade all AWS SDK to latest version ([#11626](#11626)) ([71f787f](71f787f)) * **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 4d76231 ([#11532](#11532)) ([6f04233](6f04233)) * **deps:** Update github.com/cloudquery/arrow/go/v13 digest to d864719 ([#11611](#11611)) ([557a290](557a290)) * **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.10.6 ([#11473](#11473)) ([7272133](7272133)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

bbernays added 9 commits May 22, 2023 12:06

possible implementation

99feed1

Update table_options.go

ff18ed5

Create cost_custom.go

3448fce

Merge remote-tracking branch 'upstream/main' into custom-costexplorer…

f4b2513

…-table

Update cost_custom.go

c419cdb

Update cost_custom.go

418bcb2

Merge remote-tracking branch 'upstream/main' into custom-costexplorer…

42c6297

…-table

Update cost_custom.go

d1b0e1f

docs

404ed2a

cq-bot added aws labels Jun 1, 2023

vercel bot deployed to Preview June 1, 2023 18:11 View deployment

disq reviewed Jun 2, 2023

View reviewed changes

plugins/source/aws/resources/services/costexplorer/cost_custom.go Outdated Show resolved Hide resolved

disq reviewed Jun 2, 2023

View reviewed changes

plugins/source/aws/client/tableoptions/costexplorer.go Show resolved Hide resolved

bbernays and others added 2 commits June 2, 2023 08:24

Update plugins/source/aws/resources/services/costexplorer/cost_custom.go

9dbb3d0

Co-authored-by: Kemal <223029+disq@users.noreply.github.com>

Merge branch 'main' into custom-costexplorer-table

d60334e

vercel bot deployed to Preview June 5, 2023 19:47 View deployment

bbernays added 2 commits June 6, 2023 09:20

Merge remote-tracking branch 'upstream/main' into custom-costexplorer…

24699b6

…-table

Update cost_custom.go

adc0570

bbernays marked this pull request as ready for review June 6, 2023 16:25

bbernays requested review from hermanschaaf and yevgenypats as code owners June 6, 2023 16:25

bbernays mentioned this pull request Jun 6, 2023

feat: S3 Bucket storage information in cloudquery #11236

Closed

1 task

disq requested changes Jun 7, 2023

View reviewed changes

plugins/source/aws/client/tableoptions/table_options.go Outdated Show resolved Hide resolved

bbernays added 5 commits June 8, 2023 11:18

Merge remote-tracking branch 'upstream/main' into custom-costexplorer…

0d55f53

…-table

Update table_options.go

2558490

Update table_options_test.go

471a719

Update costexplorer.go

1cd7dca

Update table_options_test.go

0fb9e48

disq approved these changes Jun 8, 2023

View reviewed changes

bbernays mentioned this pull request Jun 13, 2023

Document _alpha_ tables in AWS Plugin #11519

Closed

disq requested changes Jun 13, 2023

View reviewed changes

bbernays and others added 2 commits June 14, 2023 10:44

Merge remote-tracking branch 'upstream/main' into custom-costexplorer…

c3ce576

…-table

Merge branch 'main' into custom-costexplorer-table

b5315bf

vercel bot deployed to Preview June 14, 2023 16:08 View deployment

bbernays added 3 commits June 14, 2023 12:41

Merge remote-tracking branch 'upstream/main' into custom-costexplorer…

4110df0

…-table

Merge branch 'custom-costexplorer-table' of https://github.com/bberna…

4133913

…ys/cloudquery into custom-costexplorer-table

keep in line with cw metrics implementation

7d7a5b9

bbernays requested a review from disq June 14, 2023 18:05

docs

ec64471

vercel bot deployed to Preview June 14, 2023 18:10 View deployment

disq approved these changes Jun 15, 2023

View reviewed changes

plugins/source/aws/resources/services/costexplorer/cost_custom.go Outdated Show resolved Hide resolved

plugins/source/aws/resources/services/costexplorer/cost_custom.go Show resolved Hide resolved

Apply suggestions from code review

03e6383

Co-authored-by: Kemal <223029+disq@users.noreply.github.com>

bbernays added the automerge Automatically merge once required checks pass label Jun 15, 2023

Merge branch 'main' into custom-costexplorer-table

0567a4b

vercel bot deployed to Preview June 15, 2023 12:26 View deployment

chore: Update code and docs

a8a0dad

vercel bot deployed to Preview June 15, 2023 12:55 View deployment

kodiakhq bot merged commit 04fd769 into cloudquery:main Jun 15, 2023

bbernays deleted the custom-costexplorer-table branch June 15, 2023 13:15

cq-bot mentioned this pull request Jun 15, 2023

chore(main): Release plugins-source-aws v18.4.0 #11530

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(aws): Add Support for fully customizable Cost Explorer Data syncing#11185

feat(aws): Add Support for fully customizable Cost Explorer Data syncing#11185
kodiakhq[bot] merged 27 commits intocloudquery:mainfrom
bbernays:custom-costexplorer-table

bbernays commented Jun 1, 2023

Uh oh!

github-actions bot commented Jun 1, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

disq Jun 13, 2023

Uh oh!

disq Jun 13, 2023

Uh oh!

Uh oh!

disq left a comment

Uh oh!

Uh oh!

Uh oh!

bbernays commented Jun 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	tableName := "aws_costexplorer_cost_custom"
	tableName := "aws_alpha_costexplorer_cost_custom"

Conversation

bbernays commented Jun 1, 2023

Summary

Uh oh!

github-actions bot commented Jun 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR has the following changes to source plugin(s) tables:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

disq Jun 13, 2023

Choose a reason for hiding this comment

Uh oh!

disq Jun 13, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

disq left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bbernays commented Jun 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Jun 1, 2023 •

edited

Loading