Skip to content

feat: Update spark connector pom.xml #445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jiangmichaellll merged 3 commits into from
Jan 11, 2021
Merged

feat: Update spark connector pom.xml #445

jiangmichaellll merged 3 commits into from
Jan 11, 2021

Conversation

@jiangmichaellll
Copy link
Contributor

@jiangmichaellll jiangmichaellll commented Jan 8, 2021 •
edited
Loading

  1. use auto service instead of auto service annotation in the processor. https://screenshot.googleplex.com/AZCi8QjvdQ9AdC2
  2. Add relocations to resolve grpc-netty-shaded issues.

@jiangmichaellll jiangmichaellll requested review from a team as code owners January 8, 2021 21:36
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Jan 8, 2021
@product-auto-label product-auto-label bot added the api: pubsublite Issues related to the googleapis/java-pubsublite API. label Jan 8, 2021
@codecov
Copy link

codecov bot commented Jan 8, 2021 •
edited
Loading

Codecov Report

Merging #445 (a4ed1a6) into master (5190989) will decrease coverage by 0.23%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #445      +/-   ##
============================================
- Coverage     71.32%   71.08%   -0.24%     
  Complexity      913      913              
============================================
  Files           167      167              
  Lines          4812     4831      +19     
  Branches        244      244              
============================================
+ Hits           3432     3434       +2     
- Misses         1241     1257      +16     
- Partials        139      140       +1
Impacted Files Coverage Δ Complexity Δ
...le/cloud/pubsublite/spark/PslContinuousReader.java 54.05% <0.00%> (-0.24%) 8.00% <0.00%> (ø%)
...m/google/cloud/pubsublite/spark/PslDataSource.java 0.00% <0.00%> (ø) 0.00% <0.00%> (ø%)
...loud/pubsublite/internal/wire/RoutingMetadata.java 0.00% <0.00%> (ø) 0.00% <0.00%> (ø%)
.../pubsublite/spark/PslContinuousInputPartition.java 0.00% <0.00%> (ø) 0.00% <0.00%> (ø%)
...ogle/cloud/pubsublite/internal/ServiceClients.java
...cloud/pubsublite/internal/wire/ServiceClients.java 0.00% <0.00%> (ø) 0.00% <0.00%> (?%)
...oud/pubsublite/internal/wire/PublisherBuilder.java 68.75% <0.00%> (+6.48%) 3.00% <0.00%> (ø%)
...ud/pubsublite/internal/wire/SubscriberBuilder.java 47.82% <0.00%> (+7.08%) 2.00% <0.00%> (ø%)
...le/cloud/pubsublite/spark/PslMicroBatchReader.java 77.50% <0.00%> (+7.73%) 12.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5190989...c780c14. Read the comment docs.

@jiangmichaellll
Copy link
Contributor Author

jiangmichaellll commented Jan 11, 2021 •
edited
Loading

I am going to try another suggestion from dataproc team. This PR makes it working in dataproc but restrains the fat jar with specific versions of tcnative that might not work with other images.

EDIT: the grpc netty shaded relocation works like a charm!

medb
medb suggested changes Jan 11, 2021
View reviewed changes
@jiangmichaellll jiangmichaellll merged commit 624b123 into master Jan 11, 2021
@jiangmichaellll jiangmichaellll deleted the jiangmichael-spark-correct-versions branch January 11, 2021 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dpcollins-google dpcollins-google dpcollins-google approved these changes

+1 more reviewer

@medb medb medb approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@dpcollins-google dpcollins-google

Labels

api: pubsublite Issues related to the googleapis/java-pubsublite API. cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jiangmichaellll @medb @dpcollins-google @manuelmenzella-google