Skip to content

[KafkaToBigQueryFlex Template]: Add support for Avro ENUM type and fix FLOAT type.#2029

Merged
an2x merged 9 commits intoGoogleCloudPlatform:mainfrom
an2x:avro-fix
Dec 2, 2024
Merged

[KafkaToBigQueryFlex Template]: Add support for Avro ENUM type and fix FLOAT type.#2029
an2x merged 9 commits intoGoogleCloudPlatform:mainfrom
an2x:avro-fix

Conversation

@an2x
Copy link
Copy Markdown
Member

@an2x an2x commented Nov 21, 2024

Using Avro format with an ENUM type field in the KafkaToBigQueryFlex template currently causes the following error:

Expected Avro schema types [STRING] for BigQuery STRING field operation, but received ENUM

Avro FLOAT type also wasn't handled correctly, resulting in the following error:

Expected Avro schema types [DOUBLE, INT] for BigQuery FLOAT64 field <...>, but received FLOAT

This PR should fix both.

@codecov
Copy link
Copy Markdown

codecov bot commented Nov 21, 2024

Codecov Report

Attention: Patch coverage is 0% with 12 lines in your changes missing coverage. Please review.

Project coverage is 45.58%. Comparing base (b04de34) to head (5dda9ba).
Report is 14 commits behind head on main.

Files with missing lines Patch % Lines
...gle/cloud/teleport/v2/utils/BigQueryAvroUtils.java 0.00% 12 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2029      +/-   ##
============================================
+ Coverage     45.42%   45.58%   +0.16%     
- Complexity     3678     3727      +49     
============================================
  Files           842      847       +5     
  Lines         49970    50198     +228     
  Branches       5261     5278      +17     
============================================
+ Hits          22697    22883     +186     
- Misses        25605    25642      +37     
- Partials       1668     1673       +5     
Components Coverage Δ
spanner-templates 67.06% <ø> (+0.33%) ⬆️
spanner-import-export 64.22% <ø> (-0.01%) ⬇️
spanner-live-forward-migration 75.91% <ø> (+0.02%) ⬆️
spanner-live-reverse-replication 76.87% <ø> (+0.22%) ⬆️
spanner-bulk-migration 86.61% <ø> (+0.23%) ⬆️
Files with missing lines Coverage Δ
...gle/cloud/teleport/v2/utils/BigQueryAvroUtils.java 0.00% <0.00%> (ø)

... and 34 files with indirect coverage changes

@pull-request-size pull-request-size bot added size/L and removed size/M labels Nov 22, 2024
@an2x an2x changed the title [KafkaToBigQueryFlex Template]: Add support for Avro ENUM type. [KafkaToBigQueryFlex Template]: Add support for Avro ENUM type and fix FLOAT type. Nov 22, 2024
@an2x an2x requested a review from Dippatel98 November 25, 2024 21:11
Dippatel98
Dippatel98 previously approved these changes Nov 25, 2024
Dippatel98
Dippatel98 previously approved these changes Nov 26, 2024
@an2x an2x merged commit 1c29788 into GoogleCloudPlatform:main Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants