Skip to content

Make Beam expansion service shadowJar multi-release#33472

Merged
Abacn merged 1 commit intoapache:masterfrom
Abacn:multi-release-shadowjar
Jan 2, 2025
Merged

Make Beam expansion service shadowJar multi-release#33472
Abacn merged 1 commit intoapache:masterfrom
Abacn:multi-release-shadowjar

Conversation

@Abacn
Copy link
Copy Markdown
Contributor

@Abacn Abacn commented Dec 31, 2024

so it processes multi-release dependencies correctly

Fix #33471

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@Abacn Abacn force-pushed the multi-release-shadowjar branch from 94f3e01 to 4db5b67 Compare December 31, 2024 00:45
@github-actions github-actions bot removed the build label Dec 31, 2024
@Abacn Abacn mentioned this pull request Dec 31, 2024
Closed
17 tasks
kennknowles
kennknowles approved these changes Jan 2, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@kennknowles kennknowles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches the fix I had to do in some other places. Did it work for this jar? Perhaps we should make it default somehow... I don't know if there is a downside.

@Abacn
Copy link
Copy Markdown
Contributor Author

Abacn commented Jan 2, 2025

Yeah I think it's working. To test,

Creating a gradle project:

plugins {
    id 'java'
    id("com.github.johnrengelman.shadow") version "7.1.1"
}

repositories {
    mavenCentral()
    mavenLocal()
    maven {
        url 'https://repository.apache.org/content/groups/snapshots'
    }
}

dependencies {
    implementation("org.apache.beam:beam-sdks-java-io-expansion-service:2.63.0-SNAPSHOT")
}

def javaMainClass = "com.github.abacn.InspectClass"

task exec(type: Exec) {
    dependsOn build
    group = "Execution"
    description = "Run the main class with ExecTask"
    commandLine "java", "-classpath", sourceSets.main.runtimeClasspath.getAsPath(), javaMainClass
}

with a main class:

package com.github.abacn;

public class InspectClass {
  public static void main(String[] argv) throws ClassNotFoundException {
    String name = "org.xbill.DNS.spi.DnsjavaInetAddressResolverProvider";
    Class<?> clz = Class.forName(name);
    String path = clz.getProtectionDomain().getCodeSource().getLocation().getPath();
    System.out.println(name + ": " + path);
  }
}

Run with Java 21.

Before (run on nightly):

> Task :shadowPluginTest:exec FAILED
Exception in thread "main" java.lang.ClassNotFoundException: org.xbill.DNS.spi.DnsjavaInetAddressResolverProvider
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)

After:

> Task :shadowPluginTest:exec
org.xbill.DNS.spi.DnsjavaInetAddressResolverProvider: /Users/yathu/.m2/repository/org/apache/beam/beam-sdks-java-io-expansion-service/2.63.0-SNAPSHOT/beam-sdks-java-io-expansion-service-2.63.0-SNAPSHOT.jar

@Abacn Abacn marked this pull request as ready for review January 2, 2025 16:26
@Abacn
Copy link
Copy Markdown
Contributor Author

Abacn commented Jan 2, 2025

This matches the fix I had to do in some other places. Did it work for this jar? Perhaps we should make it default somehow... I don't know if there is a downside.

yeah I made it default to expansion services, where technically is run on different jre environment, and directly (without dependency resolution like maven or gradle)

Did a little search it seems multi-release jar is a debated, e.g. this blog https://blog.gradle.org/mrjars . So I ended up with only enabling it if upstream dependencies has used multi-released jar

imo the issue itself kind of indicated multi-release jar can bring problems.

@Abacn Abacn merged commit ee478ed into apache:master Jan 2, 2025
@Abacn Abacn deleted the multi-release-shadowjar branch January 2, 2025 16:31
@Abacn Abacn mentioned this pull request Jan 2, 2025
Merged
3 tasks
@kennknowles
Copy link
Copy Markdown
Member

kennknowles commented Jan 2, 2025 •
edited
Loading

This matches the fix I had to do in some other places. Did it work for this jar? Perhaps we should make it default somehow... I don't know if there is a downside.

yeah I made it default to expansion services, where technically is run on different jre environment, and directly (without dependency resolution like maven or gradle)

Did a little search it seems multi-release jar is a debated, e.g. this blog https://blog.gradle.org/mrjars . So I ended up with only enabling it if upstream dependencies has used multi-released jar

imo the issue itself kind of indicated multi-release jar can bring problems.

Good point. And thanks for the link to that article. It does seem like this could have the same impact on Beam users who build fat jars.

kennknowles added a commit that referenced this pull request Jan 3, 2025
@kennknowles
Merge pull request #33479: [Release-2.62] cherry-pick #33472 into rel…
88f1f3c 
…ease branch (Make Beam expansion service shadowJar multi-release)
stankiewicz pushed a commit to stankiewicz/beam that referenced this pull request Jan 16, 2025
@Abacn @stankiewicz
Make Beam expansion service shadowJar multi-release so it processes m…
57e32b8 
…ulti-release dependencies correctly (apache#33472)
@Abacn Abacn mentioned this pull request Jan 22, 2025
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kennknowles kennknowles kennknowles approved these changes

Assignees

No one assigned

Labels

extensions gcp io java sql

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: org.xbill.DNS.spi.DnsjavaInetAddressResolverProvider when running multi-lang Python jobs on Dataflow with Java 21

2 participants

@Abacn @kennknowles