Skip to content
This repository was archived by the owner on Aug 16, 2022. It is now read-only.
This repository was archived by the owner on Aug 16, 2022. It is now read-only.

Cloudquery is silently missing resources #955

@dancrumb

Description

@dancrumb

Describe the bug

When I run cloudquery fetch, I expect it to fetch all resources it can.

For instance, I see in the logs

{"level":"info","instance_id":"37cccfe8-c7d4-4e99-b6bf-67af116f01d9","@module":"aws","Region":"us-east-1","account_id":"2265xxxxxxxx","client_id":"aws_ec2_security_groups:2265xxxxxxxx:us-east-1","count":0,"table":"aws_ec2_security_groups","timestamp":"2022-05-26T09:54:02.352-0500","time":"2022-05-26T09:54:02-05:00","message":"fetched successfully"}

However, this account has 92 security groups in that region.

The DB, accordingly, does not contain these groups.

I see this across multiple accounts and regions. I imagine that it also exists for different resources

Expected Behavior

Cloudquery should fetch all resources or show errors if it cannot

Steps to Reproduce

// Configuration AutoGenerated by CloudQuery CLI
cloudquery {
  plugin_directory = "./cq/providers"
  policy_directory = "./cq/policies"

  provider "aws" {
    version = "latest"
  }

  connection {
    username = "postgres"
    password = "pass"
    host     = "localhost"
    port     = 5432
    database = "postgres"
    sslmode  = "disable"
  }
}

// All Provider Configurations

provider "aws" {
  configuration {
    org {
      member_role_name = "CloudQueryFetchRole"
      admin_account "admin" {
        local_profile = "org"
      }
    }
  }

  // list of resources to fetch
  resources = [
    // just the default resources, but I'm showing that we definitely have security groups
    "ec2.security_groups",
    // ...
  ]
}

// Module Configurations
modules {
  // drift configuration block
  drift "drift-example" {
    // state block defines from where to access the state
    terraform {
      // backend: "local" or "s3"
      backend = "local"

      // local backend options
      // files: list of tfstate files
      files = ["/path/to.tfstate"]
    }
  }
}

Then (once the DB is up and running), just run cloudquery fetch

Possible Solution

No response

Provider and CloudQuery version

CQ: 0.24.0, AWS: 0.12.0

Additional Context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions