Skip to content

Support AWS Aurora/RDS PostgreSQL as source #5309

@oeyh

Description

@oeyh

Is your feature request related to a problem? Please describe.
As a Data Prepper user, I would like to have an rds source to load existing data and stream change events from RDS MySQL databases.

Describe the solution you'd like
For export (loading existing data), we can create a snapshot, export it to S3 and read the data from S3
For stream (streaming change events), we can create a Postgres logical replication stream to receive change events.

Describe alternatives you've considered (Optional)
Run SQL queries periodically through a JDBC driver to load existing and incremental data from the source database.

Additional context
Relevant issue for MySQL: #4561

Tasks

  • Export implementation
  • Stream implementation
  • Checkpointing in both export and stream
  • Secret rotation support
  • Add E2E acknowledge support
  • Add data type mapping
  • Add plugin metrics
  • Add aggregate metrics
  • Add integration tests

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    Done

    Status

    New

    Status

    No status

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions