feat/msp: allow enablement of logical replication features for Datastream#63092
Conversation
…oudsql-logicalreplication
a2fc5b4 to
bdabcfd
Compare
jac
left a comment
There was a problem hiding this comment.
lgtm
Thanks for the gcp docs links, they were massively helpful for reviewing!
| "USAGE", | ||
| })), | ||
| }) | ||
| // Unnecessary? |
There was a problem hiding this comment.
I would think so yeah
You are explicitly granting on p.Tables rather than ALL TABLES at time of grant creation.
As such there is no need to alter the default to give permission on newly created tables as they should instead be explicitly added to p.Tables
There was a problem hiding this comment.
Yeah, I was surprised to see it in the official guidance, though I guess it's a "quick and dirty" in case someone following the guide decides to add a new table
…oudsql-logicalreplication
2b568cd to
5c01f9b
Compare
5c01f9b to
c5161d0
Compare
|
@jac re-requesting review because there's been some significant changes since 🙏 |
|
Will leave the review to @jac as I focus on the EP work 🙏 |
|
My bazel is bust: https://sourcegraph.slack.com/archives/C04MYFW01NV/p1718661476825989 - if anyone wants to do |
…oudsql-logicalreplication
df09109 to
ed69144
Compare
Adds a new
postgreSQL.logicalReplicationconfiguration to allow MSP to generate prerequisite setup for integration with Datastream: https://cloud.google.com/datastream/docs/sources-postgresql. Integration with Datastream allows the Data Analytics team to self-serve data enrichment needs for the Telemetry V2 pipeline.Enabling this feature entails downtime (Cloud SQL instance restart), so enabling the logical replication feature at the Cloud SQL level (
cloudsql.logical_decoding) is gated behindpostgreSQL.logicalReplication: {}.Setting up the required stuff in Postgres is a bit complicated, requiring 3 Postgres provider instances:
A bit more context scattered here and there in the docstrings.
Beyond the Postgres configuration we also introduce some additional resources to enable easy Datastream configuration:
:5432from the range specified in 1, allowing a connection to the Cloud SQL instanceFrom there, data team can click-ops or manage the Datastream Stream and BigQuery destination on their own.
Closes CORE-165
Closes CORE-212
Sample config:
Test plan
https://github.com/sourcegraph/managed-services/pull/1569
Changelog
postgreSQL.logicalReplicationto enable Data Analytics team to replicate selected database tables into BigQuery.