[Bug]: org.xbill.DNS.spi.DnsjavaInetAddressResolverProvider when running multi-lang Python jobs on Dataflow with Java 21 #33471
Description
What happened?
Some of the Python pipelines (internal to Google) fails with following error when using Java SQL transform via multi-lang.
DEFAULT 2024-12-30T23:37:02.819320530Z Exception in thread "grpc-default-executor-0" java.util.ServiceConfigurationError: java.net.spi.InetAddressResolverProvider: Provider org.xbill.DNS.spi.DnsjavaInetAddressResolverProvider not found
DEFAULT 2024-12-30T23:37:02.819913526Z at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)
DEFAULT 2024-12-30T23:37:02.820028778Z at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1219)
DEFAULT 2024-12-30T23:37:02.820176017Z at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1228)
DEFAULT 2024-12-30T23:37:02.820271109Z at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)
DEFAULT 2024-12-30T23:37:02.820398047Z at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
DEFAULT 2024-12-30T23:37:02.820531718Z at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
DEFAULT 2024-12-30T23:37:02.820630146Z at java.base/java.util.ServiceLoader.findFirst(ServiceLoader.java:1812)
DEFAULT 2024-12-30T23:37:02.820734542Z at java.base/java.net.InetAddress.loadResolver(InetAddress.java:508)
DEFAULT 2024-12-30T23:37:02.820858964Z at java.base/java.net.InetAddress.resolver(InetAddress.java:488)
DEFAULT 2024-12-30T23:37:02.820959923Z at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1826)
DEFAULT 2024-12-30T23:37:02.821301769Z at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
DEFAULT 2024-12-30T23:37:02.821810837Z at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
DEFAULT 2024-12-30T23:37:02.821954953Z at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
DEFAULT 2024-12-30T23:37:02.822056706Z at org.apache.beam.vendor.grpc.v1p60p1.io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:632)
DEFAULT 2024-12-30T23:37:02.822204252Z at org.apache.beam.vendor.grpc.v1p60p1.io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
DEFAULT 2024-12-30T23:37:02.822299518Z at org.apache.beam.vendor.grpc.v1p60p1.io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
DEFAULT 2024-12-30T23:37:02.822406484Z at org.apache.beam.vendor.grpc.v1p60p1.io.grpc.grpclb.GrpclbNameResolver.doResolve(GrpclbNameResolver.java:63)
DEFAULT 2024-12-30T23:37:02.822509145Z at org.apache.beam.vendor.grpc.v1p60p1.io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
DEFAULT 2024-12-30T23:37:02.822601931Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
DEFAULT 2024-12-30T23:37:02.822688916Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
DEFAULT 2024-12-30T23:37:02.822788464Z at java.base/java.lang.Thread.run(Thread.java:1583)
The failure occurs when trying to startup the Java SDK Harness container in worker VMs.
I believe this is due to the recent Hadoop upgrade to 3.4.1: #33312
Which probably results in following bug being hit.
We probably need a fix similar to this: https://github.com/netsec-ethz/scion-java-packet-example/pull/1/files
Assigning to @Abacn to look into the fix and making this a 2.62.0 release blocker.
cc: @kennknowles
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner