Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

feat/msp: allow enablement of logical replication features for Datastream#63092

Merged
bobheadxi merged 25 commits into
mainfrom
msp-cloudsql-logicalreplication
Jul 5, 2024
Merged

feat/msp: allow enablement of logical replication features for Datastream#63092
bobheadxi merged 25 commits into
mainfrom
msp-cloudsql-logicalreplication

Conversation

@bobheadxi

@bobheadxi bobheadxi commented Jun 5, 2024

Copy link
Copy Markdown
Member

Adds a new postgreSQL.logicalReplication configuration to allow MSP to generate prerequisite setup for integration with Datastream: https://cloud.google.com/datastream/docs/sources-postgresql. Integration with Datastream allows the Data Analytics team to self-serve data enrichment needs for the Telemetry V2 pipeline.

Enabling this feature entails downtime (Cloud SQL instance restart), so enabling the logical replication feature at the Cloud SQL level (cloudsql.logical_decoding) is gated behind postgreSQL.logicalReplication: {}.

Setting up the required stuff in Postgres is a bit complicated, requiring 3 Postgres provider instances:

  1. The default admin one, authenticated with our admin user
  2. New: a workload identity provider, using Add support for GCP IAM impersonation cyrilgdn/terraform-provider-postgresql#448 / postgresql: use cyrilgdn/terraform-provider-postgresql#448 managed-services-platform-cdktf#11. This is required for creating a publication on selected tables, which requires being owner of said table. Because tables are created by application using e.g. auto-migrate, the workload identity is always the table owner, so we need to impersonate the IAM user
  3. New: a "replication user" which is created with the replication permission. Replication seems to not be a propagated permission so we need a role/user that has replication enabled.

A bit more context scattered here and there in the docstrings.

Beyond the Postgres configuration we also introduce some additional resources to enable easy Datastream configuration:

  1. Datastream Private Connection, which peers to the service private network
  2. Cloud SQL Proxy VM, which only allows connections to :5432 from the range specified in 1, allowing a connection to the Cloud SQL instance
  3. Datastream Connection Profile attached to 1

From there, data team can click-ops or manage the Datastream Stream and BigQuery destination on their own.

Closes CORE-165
Closes CORE-212

Sample config:

  resources:
    postgreSQL:
      databases:
        - "primary"
      logicalReplication:
        publications:
          - name: testing
            database: primary
            tables:
              - users

Test plan

https://github.com/sourcegraph/managed-services/pull/1569

Changelog

  • MSP services can now configure postgreSQL.logicalReplication to enable Data Analytics team to replicate selected database tables into BigQuery.

@cla-bot cla-bot Bot added the cla-signed label Jun 5, 2024
@bobheadxi bobheadxi force-pushed the msp-cloudsql-logicalreplication branch from a2fc5b4 to bdabcfd Compare June 7, 2024 20:01
@bobheadxi bobheadxi requested a review from jac June 7, 2024 20:43

@jac jac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Thanks for the gcp docs links, they were massively helpful for reviewing!

"USAGE",
})),
})
// Unnecessary?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would think so yeah

You are explicitly granting on p.Tables rather than ALL TABLES at time of grant creation.
As such there is no need to alter the default to give permission on newly created tables as they should instead be explicitly added to p.Tables

@bobheadxi bobheadxi Jun 13, 2024

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was surprised to see it in the official guidance, though I guess it's a "quick and dirty" in case someone following the guide decides to add a new table

@bobheadxi bobheadxi force-pushed the msp-cloudsql-logicalreplication branch from 2b568cd to 5c01f9b Compare June 14, 2024 00:45
@bobheadxi bobheadxi force-pushed the msp-cloudsql-logicalreplication branch from 5c01f9b to c5161d0 Compare June 14, 2024 00:46
@bobheadxi bobheadxi marked this pull request as ready for review June 17, 2024 21:14
@bobheadxi bobheadxi requested a review from jac June 17, 2024 21:58
@bobheadxi

Copy link
Copy Markdown
Member Author

@jac re-requesting review because there's been some significant changes since 🙏

@bobheadxi bobheadxi requested a review from a team June 17, 2024 22:06
@unknwon

unknwon commented Jun 17, 2024

Copy link
Copy Markdown
Contributor

Will leave the review to @jac as I focus on the EP work 🙏

@bobheadxi

Copy link
Copy Markdown
Member Author

My bazel is bust: https://sourcegraph.slack.com/archives/C04MYFW01NV/p1718661476825989 - if anyone wants to do sg bz configure and sg bz configure godeps for me, that would be nice 😆

@jac jac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

Comment thread dev/managedservicesplatform/stacks/cloudrun/cloudrun.go Outdated
@bobheadxi bobheadxi force-pushed the msp-cloudsql-logicalreplication branch from df09109 to ed69144 Compare June 19, 2024 20:43
@bobheadxi bobheadxi requested a review from jac July 4, 2024 01:30
@bobheadxi bobheadxi enabled auto-merge (squash) July 5, 2024 18:18
@bobheadxi bobheadxi merged commit 28348e7 into main Jul 5, 2024
@bobheadxi bobheadxi deleted the msp-cloudsql-logicalreplication branch July 5, 2024 18:24
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants