Skip to content

Relocate Avro dependencies in Fn Harness JAR#29407

Closed
bvolpato wants to merge 1 commit intoapache:masterfrom
bvolpato:repackage-avro
Closed

Relocate Avro dependencies in Fn Harness JAR#29407
bvolpato wants to merge 1 commit intoapache:masterfrom
bvolpato:repackage-avro

Conversation

@bvolpato
Copy link
Copy Markdown
Contributor

@bvolpato bvolpato commented Nov 11, 2023

After #27851, user code that depends on versions newer than Avro 1.8.2 are having problems running on Dataflow.

For example in https://github.com/GoogleCloudPlatform/DataflowTemplates, where we moved on to Avro 1.11.3, there were incompatibility errors:

Caused by: java.io.InvalidClassException: org.apache.avro.specific.SpecificRecordBase; local class incompatible: stream classdesc serialVersionUID = -1463700717714793795, local class serialVersionUID = 189988654766568477

and

Caused by: java.lang.NoSuchMethodError: 'boolean org.apache.avro.generic.GenericRecord.hasField(java.lang.String)' com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.getMetadataIsDeleted(FormatDatastreamRecordToJson.java:258) com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.apply(FormatDatastreamRecordToJson.java:123) com.google.cloud.teleport.v2.transforms.FormatDatastreamRecordToJson.apply(FormatDatastreamRecordToJson.java:51) org.apache.beam.sdk.extensions.avro.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:610)

The root cause is that Avro classes are now being shipped along with the /opt/apache/beam/jars/beam-sdks-java-harness.jar, which wasn't the case before.

Relocating org.apache.avro to org.apache.beam.fn.harness.repackaged.org.apache.avro will solve the problem and allow users to control their Avro.

@github-actions github-actions bot added the java label Nov 11, 2023
@bvolpato
Copy link
Copy Markdown
Contributor Author

bvolpato commented Nov 11, 2023

@aromanenko-dev, was the inclusion of Avro intentional in the harness JAR?

@damccorm @Abacn I think this is a blocker for 2.52.0, as there will be pipeline failures if someone depends on anything after Avro 1.8.2 (which was the case for Dataflow Templates, and would require code refactoring).

@bvolpato bvolpato changed the title Relocate avro dependencies in harness jar Relocate Avro dependencies in harness jar Nov 11, 2023
@bvolpato bvolpato changed the title Relocate Avro dependencies in harness jar Relocate Avro dependencies in Fn Harness JAR Nov 11, 2023
@bvolpato
Copy link
Copy Markdown
Contributor Author

bvolpato commented Nov 11, 2023

Run Java PreCommit

1 similar comment
@bvolpato
Copy link
Copy Markdown
Contributor Author

bvolpato commented Nov 11, 2023

Run Java PreCommit

@bvolpato
Copy link
Copy Markdown
Contributor Author

bvolpato commented Nov 11, 2023

Java PreCommit is getting:

13:19:07 > Configure project :sdks:java:container:java21
13:19:07 docker-pull-licenses set, creating go-licenses
13:19:17 System Go installation: /snap/bin/go is go version go1.16.15 linux/amd64; Preparing to use /home/jenkins/go/bin/go1.21.4
13:19:17 /home/jenkins/go/pkg/mod/golang.org/dl@v0.0.0-20231107201224-8165dcd93058/go1.21.4/main.go:20:8: //go:build comment without // +build comment
13:19:17 
13:19:17 FAILURE: Build failed with an exception.
13:19:17 
13:19:17 * What went wrong:
13:19:17 Could not determine the dependencies of task ':sdks:java:container:goBuild'.
13:19:17 > Could not create task ':sdks:java:container:goPrepare'.
13:19:17    > Process 'command 'sh'' finished with non-zero exit value 1

Which I think is unrelated to the change.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Nov 11, 2023

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@bvolpato
Copy link
Copy Markdown
Contributor Author

bvolpato commented Nov 12, 2023

Run Java PreCommit

@bvolpato bvolpato mentioned this pull request Nov 13, 2023
Merged
@bvolpato
Copy link
Copy Markdown
Contributor Author

bvolpato commented Nov 13, 2023

Run Java PreCommit

@bvolpato
Copy link
Copy Markdown
Contributor Author

bvolpato commented Nov 13, 2023

Closing in favor of #29412

@bvolpato bvolpato closed this Nov 13, 2023
@bvolpato bvolpato mentioned this pull request Nov 13, 2023
Closed
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

java

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bvolpato