Skip to content

Add support for byte arrays and uuid arrays in datastream to sql template.#2174

Merged
liferoad merged 1 commit intoGoogleCloudPlatform:mainfrom
claudevdm:fix-uuid-bytea
Feb 13, 2025
Merged

Add support for byte arrays and uuid arrays in datastream to sql template.#2174
liferoad merged 1 commit intoGoogleCloudPlatform:mainfrom
claudevdm:fix-uuid-bytea

Conversation

@claudevdm
Copy link
Copy Markdown
Contributor

@claudevdm claudevdm commented Feb 12, 2025 •
edited
Loading

Fix reading ByteBuffer from Avro, inserting base64 encoded strings in… to postgres BYTEA columns and cast UUID arrays.

There were multiple issues:

  1. FormatDatastreamRecordToJson.java assumed the avro data could be cast as byte[], but they were in fact ByteBuffer types. This caused the entire record to fail on read/parse
  2. Jackson converts byte[] => base64 encoded strings. So we need to decode these strings in the postgres insert statement
  3. UUID arrays were not properly cast from strings
  4. Some tests werent marked with @test and therefore not running in FormatDatastreamRecordToJsonTest

Fix reading ByteBuffer from Avro, inserting base64 encoded strings in…
6a61bb7 
…to postgres BYTEA columns and cast UUID arrays.
@claudevdm claudevdm requested review from a team as code owners February 12, 2025 02:35
@claudevdm claudevdm changed the title Fix reading ByteBuffer from Avro, inserting base64 encoded strings in… Add support for byte arrays and uuid arrays in datastream to sql template. Feb 12, 2025
@codecov
Copy link
Copy Markdown

codecov bot commented Feb 12, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 58.33333% with 5 lines in your changes missing coverage. Please review.

Project coverage is 47.06%. Comparing base (40141f5) to head (6a61bb7).
Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
...tream/transforms/FormatDatastreamRecordToJson.java 44.44% 4 Missing and 1 partial ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main    #2174      +/-   ##
============================================
+ Coverage     46.96%   47.06%   +0.09%     
- Complexity     4047     4375     +328     
============================================
  Files           876      876              
  Lines         52194    52204      +10     
  Branches       5502     5505       +3     
============================================
+ Hits          24515    24568      +53     
+ Misses        25921    25876      -45     
- Partials       1758     1760       +2
Components Coverage Δ
spanner-templates 68.91% <ø> (+<0.01%) ⬆️
spanner-import-export 65.72% <ø> (+0.01%) ⬆️
spanner-live-forward-migration 76.50% <ø> (ø)
spanner-live-reverse-replication 78.77% <ø> (ø)
spanner-bulk-migration 87.94% <ø> (ø)
Files with missing lines Coverage Δ
...oogle/cloud/teleport/v2/utils/DatastreamToDML.java 9.09% <ø> (ø)
...oud/teleport/v2/utils/DatastreamToPostgresDML.java 58.44% <100.00%> (+1.68%) ⬆️
...tream/transforms/FormatDatastreamRecordToJson.java 57.09% <44.44%> (+14.51%) ⬆️

... and 1 file with indirect coverage changes

@liferoad liferoad added bug-fix improvement Making existing code better and removed bug-fix labels Feb 12, 2025
@liferoad liferoad merged commit 1a176fa into GoogleCloudPlatform:main Feb 13, 2025
31 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dhercher dhercher dhercher approved these changes

@liferoad liferoad liferoad approved these changes

@Deep1998 Deep1998 Awaiting requested review from Deep1998 Deep1998 is a code owner automatically assigned from GoogleCloudPlatform/spanner-migrations-team

@VardhanThigle VardhanThigle Awaiting requested review from VardhanThigle VardhanThigle is a code owner automatically assigned from GoogleCloudPlatform/spanner-migrations-team

Assignees

No one assigned

Labels

improvement Making existing code better size/M

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@claudevdm @dhercher @liferoad