Skip to content

Support AWS Aurora/RDS MySQL as source #4561

@oeyh

Description

@oeyh

Is your feature request related to a problem? Please describe.
As a Data Prepper user, I would like to have an rds source to load existing data and stream change events from RDS MySQL databases.

Describe the solution you'd like
For export (loading existing data), we can create a snapshot, export it to S3 and read the data from S3
For stream (streaming change events), we can connect to MySQL's binary log stream to receive change events.

Describe alternatives you've considered (Optional)
Run SQL queries periodically through a JDBC driver to load existing and incremental data from the source database.

Additional context
The feature shares similar ideas with existing dynamodb source and documentdb source.

Tasks

  • Project setup, source configurations, skeleton code
  • Export implementation - create snapshot and export to S3
  • Export implementation- read exported data files in S3
  • Stream implementation
  • Checkpointing in both export and stream
  • Pipeline configuration transformation template
  • Secret rotation support
  • Add E2E acknowledge support
  • Add data type mapping
  • Add plugin metrics
  • Add aggregate metrics
  • Add integration tests

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Status

Done

Status

New

Status

No status

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions