-
Notifications
You must be signed in to change notification settings - Fork 550
poor performance when talking to Postgres over a network #2462
Description
Describe the Bug
poor performance when talking to Postgres over a network
a very rough, non-scientific, observation/benchmark
CQ running on an m5xl 4 cores, 16 gb memory, 40 gb gp2 ebs
Postgres dockerized locally
pull approx. 700k resources over 10 AWS accounts
time: ~15 minutes
CQ running on the same m5xl 4 cores, 16 gb memory, 40 gb gp2 ebs
Postgres on a t2.small in the SAME REGION and SAME AZ as the ec2 instance where CQ is running
same CQ configs, pull approx. 700k resources over 10 AWS accounts
time: ~30 minutes
CQ running on the same m5xl 4 cores, 16 gb memory, 40 gb gp2 ebs
Postgres on a t2.small in the SAME REGION and DIFFERENT AZ as the ec2 instance where CQ is running
same CQ configs, pull approx. 700k resources over 10 AWS accounts
time: ~45 minutes
CQ running on the same m5xl 4 cores, 16 gb memory, 40 gb gp2 ebs
Postgres on a t2.small in the DIFFERENT REGION as the ec2 instance where CQ is running. (us-east-1 <--> us-west-1)
same CQ configs, pull approx. 700k resources over 10 AWS accounts
time: "forget it, you'll be there forever" :)
Expected Behavior
in big quotes :)
good/decent performance when pointed at a Postgres over a network, any case where Postgres is not "local"
CloudQuery Version
CQ=1.0.2
psql plugin = 1.0.0
aws plugin = 1.0.1
Debug Output
Steps to Reproduce
Also please share any other necessary configuration files
-->