[BEAM-11968] Bump com.google.cloud:libraries-bom to 19.0.0 by Fokko · Pull Request #14218 · apache/beam

Fokko · 2021-03-12T16:04:28Z

Missing the isMtls() in google-http-client. Looks like it has been added since 1.38.0:

https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/http/HttpTransport.java#L136

Original stacktrace:

java.lang.RuntimeException: Encountered checked exception when constructing an instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
...
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:564)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
at java.lang.Thread.run (Thread.java:832)
Caused by: java.lang.NoSuchMethodError: 'boolean com.google.api.client.http.HttpTransport.isMtls()'
at com.google.api.services.storage.Storage$Builder.chooseEndpoint (Storage.java:11151)
at com.google.api.services.storage.Storage$Builder.<init> (Storage.java:11184)
at org.apache.beam.sdk.extensions.gcp.util.Transport.newStorageClient (Transport.java:97)
at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:104)
at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:93)
at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper (ProxyInvocationHandler.java:605)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault (ProxyInvocationHandler.java:546)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke (ProxyInvocationHandler.java:171)
at com.sun.proxy.$Proxy41.getGcsUtil (Unknown Source)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible (GcsPathValidator.java:83)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported (GcsPathValidator.java:53)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:291)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:564)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:217)
at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
...
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:564)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
at java.lang.Thread.run (Thread.java:832)

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-11968] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2021-03-12T16:22:30Z

Codecov Report

Merging #14218 (5a994f9) into master (0ca0088) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #14218      +/-   ##
==========================================
+ Coverage   83.41%   83.43%   +0.02%     
==========================================
  Files         469      918     +449     
  Lines       58721   117735   +59014     
==========================================
+ Hits        48985    98238   +49253     
- Misses       9736    19497    +9761

Impacted Files	Coverage Δ
...s/snippets/transforms/aggregation/combinevalues.py
...thon/apache_beam/io/azure/blobstoragefilesystem.py
...es/snippets/transforms/aggregation/cogroupbykey.py
...ild/srcs/sdks/python/apache_beam/runners/common.py
...examples/snippets/transforms/aggregation/sample.py
...e_beam/runners/portability/abstract_job_service.py
...beam/runners/interactive/display/pipeline_graph.py
...build/srcs/sdks/python/apache_beam/io/gcp/gcsio.py
.../python/apache_beam/io/gcp/tests/pubsub_matcher.py
...python/apache_beam/io/gcp/datastore/v1new/types.py
... and 1377 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0ca0088...5a994f9. Read the comment docs.

Missing the isMtls() in google-http-client. Looks like it has been added since 1.38.0: https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/http/HttpTransport.java#L136 Original stacktrace: ``` java.lang.RuntimeException: Encountered checked exception when constructing an instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:233) at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158) at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55) at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154) ... at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:564) at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282) at java.lang.Thread.run (Thread.java:832) Caused by: java.lang.NoSuchMethodError: 'boolean com.google.api.client.http.HttpTransport.isMtls()' at com.google.api.services.storage.Storage$Builder.chooseEndpoint (Storage.java:11151) at com.google.api.services.storage.Storage$Builder.<init> (Storage.java:11184) at org.apache.beam.sdk.extensions.gcp.util.Transport.newStorageClient (Transport.java:97) at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:104) at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:93) at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper (ProxyInvocationHandler.java:605) at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault (ProxyInvocationHandler.java:546) at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke (ProxyInvocationHandler.java:171) at com.sun.proxy.$Proxy41.getGcsUtil (Unknown Source) at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible (GcsPathValidator.java:83) at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported (GcsPathValidator.java:53) at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:291) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:564) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:217) at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158) at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55) at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154) ... at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:564) at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282) at java.lang.Thread.run (Thread.java:832) ```

kennknowles · 2021-04-01T18:27:48Z

Most of the failures appear to be a mocking situation that is checking how many times a trivial method is called. Probably the mocking should be more flexible or we could find a non-mock way to test. Do you feel able to go and adjust the mocks?

Fokko · 2021-04-01T20:06:09Z

@kennknowles I tried to debug the logs, I think it has to do with some missing licenses, but didn't get it running locally.

@pabloem do you have any pointers? :)

pabloem · 2021-04-15T20:14:40Z

sorry that I missed this. Rerunning things...

pabloem · 2021-04-16T18:51:58Z

Run Java PreCommit

pabloem · 2021-04-16T18:52:25Z

Run Java_Examples_Dataflow PreCommit

suztomo · 2021-04-19T14:47:10Z

I have #14527 for the libraries-bom 20.0.0, which brings google-http-client 1.39.2

pabloem · 2021-04-20T19:02:00Z

ok closing for the 20.0.0 update instead. @Fokko I hope that's fine ^^'

Fokko force-pushed the fd-bump-google branch from 053257e to 5a994f9 Compare March 21, 2021 11:05

aaltay requested a review from kennknowles April 1, 2021 17:57

pabloem closed this Apr 15, 2021

pabloem reopened this Apr 15, 2021

suztomo mentioned this pull request Apr 19, 2021

[BEAM-11205] Libraries-BOM 20.0.0 with gRPC and protobuf #14527

Merged

4 tasks

pabloem closed this Apr 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-11968] Bump com.google.cloud:libraries-bom to 19.0.0#14218

[BEAM-11968] Bump com.google.cloud:libraries-bom to 19.0.0#14218
Fokko wants to merge 1 commit intoapache:masterfrom
Fokko:fd-bump-google

Fokko commented Mar 12, 2021

Uh oh!

codecov bot commented Mar 12, 2021 •

edited

Loading

Uh oh!

kennknowles commented Apr 1, 2021

Uh oh!

Fokko commented Apr 1, 2021

Uh oh!

pabloem commented Apr 15, 2021

Uh oh!

pabloem commented Apr 16, 2021

Uh oh!

pabloem commented Apr 16, 2021

Uh oh!

suztomo commented Apr 19, 2021 •

edited

Loading

Uh oh!

pabloem commented Apr 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Fokko commented Mar 12, 2021

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

codecov bot commented Mar 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kennknowles commented Apr 1, 2021

Uh oh!

Fokko commented Apr 1, 2021

Uh oh!

pabloem commented Apr 15, 2021

Uh oh!

pabloem commented Apr 16, 2021

Uh oh!

pabloem commented Apr 16, 2021

Uh oh!

suztomo commented Apr 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pabloem commented Apr 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Mar 12, 2021 •

edited

Loading

suztomo commented Apr 19, 2021 •

edited

Loading