[Do not merge] Validate Beam 2.63.0rc2 by Abacn · Pull Request #2177 · GoogleCloudPlatform/DataflowTemplates

Abacn · 2025-02-12T16:04:34Z

No description provided.

Abacn · 2025-02-12T20:45:52Z

org.tensorflow.proto does not support protobuf 4:

The latest release of

https://mvnrepository.com/artifact/org.tensorflow/proto

was in 2019.

This prevents classic Templates to use protobuf 4, that is now used by Beam 2.63.0

Abacn · 2025-02-12T23:29:39Z

sourcedb-to-spanner tests timing out

codecov · 2025-02-12T23:41:06Z

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Project coverage is 46.99%. Comparing base (b93cad8) to head (aaab86d).
Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
...apache/beam/sdk/io/gcp/spanner/LocalSpannerIO.java	0.00%	4 Missing ⚠️
...streamstopubsub/BigtableChangeStreamsToPubSub.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2177      +/-   ##
============================================
+ Coverage     46.97%   46.99%   +0.02%     
+ Complexity     4049     4046       -3     
============================================
  Files           876      874       -2     
  Lines         52193    52161      -32     
  Branches       5502     5498       -4     
============================================
- Hits          24517    24514       -3     
+ Misses        25920    25895      -25     
+ Partials       1756     1752       -4

Components	Coverage Δ
spanner-templates	`68.91% <ø> (-0.01%)`	⬇️
spanner-import-export	`65.72% <ø> (-0.02%)`	⬇️
spanner-live-forward-migration	`76.50% <ø> (ø)`
spanner-live-reverse-replication	`78.77% <ø> (ø)`
spanner-bulk-migration	`87.94% <ø> (ø)`

Files with missing lines	Coverage Δ
...e/cloud/teleport/templates/BigQueryToTFRecord.java	`19.69% <ø> (ø)`
...treamstopubsub/model/TestChangeStreamMutation.java	`0.00% <ø> (ø)`
...streamstopubsub/BigtableChangeStreamsToPubSub.java	`0.00% <0.00%> (ø)`
...apache/beam/sdk/io/gcp/spanner/LocalSpannerIO.java	`0.00% <0.00%> (ø)`

... and 3 files with indirect coverage changes

Abacn · 2025-02-13T15:36:40Z

all tests currently failing are PythonUDF tests.

Downloaded "surefire-integration-test-results.zip" manually, decompress and run grep -B 1 -nr 'error message' shows all test failures are PythonUDF tests, which are expected.

./python/target/surefire-reports/TEST-com.google.cloud.teleport.templates.python.PubSubToBigQueryYamlIT.xml-79-  <testcase name="testPubSubToBigQuery" classname="com.google.cloud.teleport.templates.python.PubSubToBigQueryYamlIT" time="964.957">
./python/target/surefire-reports/TEST-com.google.cloud.teleport.templates.python.PubSubToBigQueryYamlIT.xml:80:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_00_21-17100005783908586268?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_00_21-17100005783908586268?project=cloud-teleport-testing.
--
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.TextIOToBigQueryIT.xml-80-  <testcase name="testTextIOToBigQueryWithPythonUdf" classname="com.google.cloud.teleport.v2.templates.TextIOToBigQueryIT" time="1,532.708">
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.TextIOToBigQueryIT.xml:81:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_11-17125510468052756728?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_11-17125510468052756728?project=cloud-teleport-testing.
--
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToBigQueryIT.xml-82-  <testcase name="testPubsubToBigQueryWithPythonUdf" classname="com.google.cloud.teleport.v2.templates.PubSubToBigQueryIT" time="1,417.968">
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToBigQueryIT.xml:83:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_31-6459326777086093214?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_31-6459326777086093214?project=cloud-teleport-testing.
--
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToBigQueryAutoIT.xml-84-  <testcase name="testPubsubToBigQueryWithPythonUdf" classname="com.google.cloud.teleport.v2.templates.PubSubToBigQueryAutoIT" time="1,538.468">
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToBigQueryAutoIT.xml:85:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_31-14586942559005192948?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_31-14586942559005192948?project=cloud-teleport-testing.
--
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.TextToBigQueryStreamingIT.xml-79-  <testcase name="testTextToBigQueryWithPythonUDFs" classname="com.google.cloud.teleport.v2.templates.TextToBigQueryStreamingIT" time="1,536.09">
./v2/googlecloud-to-googlecloud/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.TextToBigQueryStreamingIT.xml:80:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_30-16711561795242901948?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_17_30-16711561795242901948?project=cloud-teleport-testing.
--
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.BigQueryToElasticsearchIT.xml-81-  <testcase name="testBigQueryToElasticsearchWithPythonUdf" classname="com.google.cloud.teleport.v2.elasticsearch.templates.BigQueryToElasticsearchIT" time="1,104.803">
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.BigQueryToElasticsearchIT.xml:82:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_30_14-16069209254424499837?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_30_14-16069209254424499837?project=cloud-teleport-testing.
--
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.GCSToElasticsearchIT.xml-83-  <testcase name="testElasticsearchCsvWithoutHeadersPython" classname="com.google.cloud.teleport.v2.elasticsearch.templates.GCSToElasticsearchIT" time="1,378.181">
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.GCSToElasticsearchIT.xml:84:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_32_47-15516695606867294538?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_32_47-15516695606867294538?project=cloud-teleport-testing.
--
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.PubSubToElasticsearchIT.xml-79-  <testcase name="testPubSubToElasticsearchWithPythonUdf" classname="com.google.cloud.teleport.v2.elasticsearch.templates.PubSubToElasticsearchIT" time="1,273.772">
./v2/googlecloud-to-elasticsearch/target/surefire-reports/TEST-com.google.cloud.teleport.v2.elasticsearch.templates.PubSubToElasticsearchIT.xml:80:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_32_03-1910802050061065714?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_19_32_03-1910802050061065714?project=cloud-teleport-testing.
--
./v2/pubsub-binary-to-bigquery/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubsubProtoToBigQueryIT.xml-79-  <testcase name="pubSubProtoToBigQueryWithPythonUdf" classname="com.google.cloud.teleport.v2.templates.PubsubProtoToBigQueryIT" time="1,186.835">
./v2/pubsub-binary-to-bigquery/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubsubProtoToBigQueryIT.xml:80:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_35_16-18294704747315617489?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_35_16-18294704747315617489?project=cloud-teleport-testing.
--
./v2/pubsub-to-mongodb/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToMongoDBIT.xml-79-  <testcase name="testPythonPubsubToMongoDB" classname="com.google.cloud.teleport.v2.templates.PubSubToMongoDBIT" time="1,104.131">
./v2/pubsub-to-mongodb/target/surefire-reports/TEST-com.google.cloud.teleport.v2.templates.PubSubToMongoDBIT.xml:80:    <error message="The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_56_06-2339488772302340482?project=cloud-teleport-testing." type="java.lang.RuntimeException">java.lang.RuntimeException: The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-12_18_56_06-2339488772302340482?project=cloud-teleport-testing.

Abacn · 2025-02-13T16:42:34Z

Kafka PR: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/13312637711

Spanner PR: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/13312639690

Abacn · 2025-02-13T17:57:26Z

Single test failed:

2025-02-13T17:20:34.0661569Z [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1,543.391 s <<< FAILURE! - in com.google.cloud.teleport.v2.templates.SpannerToSourceDbCustomTransformationIT
2025-02-13T17:20:34.0663779Z [ERROR] testCustomTransformation(com.google.cloud.teleport.v2.templates.SpannerToSourceDbCustomTransformationIT)  Time elapsed: 1,543.391 s  <<< FAILURE!
2025-02-13T17:20:34.0665084Z value of  : result.check if meets all conditions
2025-02-13T17:20:34.0665627Z expected  : CONDITION_MET
2025-02-13T17:20:34.0666021Z but was   : TIMEOUT
2025-02-13T17:20:34.0666472Z result was: TIMEOUT
2025-02-13T17:20:34.0669433Z 	at com.google.cloud.teleport.v2.templates.SpannerToSourceDbCustomTransformationIT.assertRowInMySQL(SpannerToSourceDbCustomTransformationIT.java:350)
2025-02-13T17:20:34.0672234Z 	at com.google.cloud.teleport.v2.templates.SpannerToSourceDbCustomTransformationIT.testCustomTransformation(SpannerToSourceDbCustomTransformationIT.java:166)

test infra issue, other spanner-to-sourcedb IT passed, ignore for now

Abacn · 2025-02-13T18:04:53Z

I see, test failure is actually relevant:

java.lang.NoClassDefFoundError: Could not initialize class com.google.spanner.v1.Session$LabelsDefaultEntryHolder
2025-02-13 12:35:35.242 EST
at com.google.spanner.v1.Session.internalGetLabels(Session.java:147)
2025-02-13 12:35:35.242 EST
at com.google.spanner.v1.Session.getSerializedSize(Session.java:490)
2025-02-13 12:35:35.242 EST
at com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:861)
2025-02-13 12:35:35.242 EST
at com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:641)
2025-02-13 12:35:35.242 EST
at com.google.spanner.v1.BatchCreateSessionsRequest.getSerializedSize(BatchCreateSessionsRequest.java:232)
2025-02-13 12:35:35.242 EST
at io.grpc.protobuf.lite.ProtoInputStream.available(ProtoInputStream.java:108)
2025-02-13 12:35:35.242 EST
at io.grpc.internal.MessageFramer.getKnownLength(MessageFramer.java:204)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread "grpc-default-executor-3"]
2025-02-13 12:35:35.242 EST
at com.google.spanner.v1.Session$LabelsDefaultEntryHolder.<clinit>(Session.java:132)
2025-02-13 12:35:35.242 EST
... 17 more

Abacn · 2025-02-13T19:49:39Z

Spanner PR:

First run:

     🧪 - MySQLSourceDbToSpannerSimpleIT | The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-13_09_34_20-15299568926532995869?project=cloud-teleport-testing.
     🧪 - SpannerToSourceDbCustomTransformationIT | value of  : result.check if meets all conditions

Second run:

🧪 - TimezoneIT | The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-13_10_19_21-17776592963417510888?project=cloud-teleport-testing.
     🧪 - PostgreSQLDataTypesIT | The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-13_10_51_31-2701783821855436363?project=cloud-teleport-testing.
     🧪 - SpannerChangeStreamToGcsSingleShardIT | The job failed before launch! For more information please check the job log at https://console.cloud.google.com/dataflow/jobs/us-central1/2025-02-13_10_17_33-14641481078103719277?project=cloud-teleport-testing.

example error:

2025-02-13 13:21:47.309 EST
Caused by: com.google.cloud.spanner.SpannerException: RESOURCE_EXHAUSTED: Timed out after waiting 60000ms for acquiring session. To mitigate error SessionPoolOptions#setAcquireSessionTimeout(Duration) to set a higher timeout or increase the number of sessions in the session pool.

different tests failed in two runs. proceed.

Kafka PR: passed

Abacn · 2025-02-18T20:32:48Z

For who going to upgrading Templates: remember to remove https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/2177/files#diff-21f0f2b4485d5b8359efec95a27f2bd574a0e46885f815cab18cc8eb847b9b69L706 (fixed by #2182) and re-generate requirements for Python templates

pull-request-size bot added the size/L label Feb 12, 2025

Abacn added ignore-for-release and removed size/L labels Feb 12, 2025

Abacn added 2 commits February 12, 2025 16:36

[Do not merge] Validate Beam 2.63.0rc1

2643e14

fixes

171b958

Abacn force-pushed the validate-beam-263rc1 branch from 1ee90bf to 171b958 Compare February 12, 2025 18:38

pull-request-size bot added the size/XL label Feb 12, 2025

spotless apply

6610531

Abacn added 2 commits February 12, 2025 21:32

Fix tests

118846c

Update tensorflow class

0f12564

Remove dependency pin managed by beam-gcp-bom

5c9bc02

Abacn changed the title ~~[Do not merge] Validate Beam 2.63.0rc1~~ [Do not merge] Validate Beam 2.63.0rc2 Feb 13, 2025

validate rc2

aaab86d

Abacn mentioned this pull request Feb 13, 2025

[Bug]: Flaky Spanner Integration test due to "Could not initialize class com.google.spanner.v1.Session$LabelsDefault" #2191

Closed

Abacn mentioned this pull request Feb 20, 2025

Upgrade Beam version to 2.63.0 #2206

Merged

Abacn closed this Feb 22, 2025

Abacn deleted the validate-beam-263rc1 branch February 22, 2025 02:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Do not merge] Validate Beam 2.63.0rc2#2177

[Do not merge] Validate Beam 2.63.0rc2#2177
Abacn wants to merge 7 commits intomainfrom
validate-beam-263rc1

Abacn commented Feb 12, 2025

Uh oh!

Abacn commented Feb 12, 2025

Uh oh!

Abacn commented Feb 12, 2025

Uh oh!

codecov bot commented Feb 12, 2025 •

edited

Loading

Uh oh!

Abacn commented Feb 13, 2025

Uh oh!

Abacn commented Feb 13, 2025 •

edited

Loading

Uh oh!

Abacn commented Feb 13, 2025

Uh oh!

Abacn commented Feb 13, 2025

Uh oh!

Abacn commented Feb 13, 2025

Uh oh!

Abacn commented Feb 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Abacn commented Feb 12, 2025

Uh oh!

Abacn commented Feb 12, 2025

Uh oh!

Abacn commented Feb 12, 2025

Uh oh!

codecov bot commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Abacn commented Feb 13, 2025

Uh oh!

Abacn commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Abacn commented Feb 13, 2025

Uh oh!

Abacn commented Feb 13, 2025

Uh oh!

Abacn commented Feb 13, 2025

Uh oh!

Abacn commented Feb 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov bot commented Feb 12, 2025 •

edited

Loading

Abacn commented Feb 13, 2025 •

edited

Loading