Skip to content

poor performance when talking to Postgres over a network #2462

@jimr6007

Description

@jimr6007

Describe the Bug

poor performance when talking to Postgres over a network

a very rough, non-scientific, observation/benchmark

CQ running on an m5xl 4 cores, 16 gb memory, 40 gb gp2 ebs
Postgres dockerized locally
pull approx. 700k resources over 10 AWS accounts
time: ~15 minutes

CQ running on the same m5xl 4 cores, 16 gb memory, 40 gb gp2 ebs
Postgres on a t2.small in the SAME REGION and SAME AZ as the ec2 instance where CQ is running
same CQ configs, pull approx. 700k resources over 10 AWS accounts
time: ~30 minutes

CQ running on the same m5xl 4 cores, 16 gb memory, 40 gb gp2 ebs
Postgres on a t2.small in the SAME REGION and DIFFERENT AZ as the ec2 instance where CQ is running
same CQ configs, pull approx. 700k resources over 10 AWS accounts
time: ~45 minutes

CQ running on the same m5xl 4 cores, 16 gb memory, 40 gb gp2 ebs
Postgres on a t2.small in the DIFFERENT REGION as the ec2 instance where CQ is running. (us-east-1 <--> us-west-1)
same CQ configs, pull approx. 700k resources over 10 AWS accounts
time: "forget it, you'll be there forever" :)

Expected Behavior

in big quotes :)
good/decent performance when pointed at a Postgres over a network, any case where Postgres is not "local"

CloudQuery Version

CQ=1.0.2
psql plugin = 1.0.0
aws plugin = 1.0.1

Debug Output

Steps to Reproduce

Also please share any other necessary configuration files
-->

Additional Context

References

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions