Skip to content

[CI] GitHub Actions reverse lookup for hostname is inconsistent #10232

@lhotari

Description

@lhotari

Problem

A lot of builds are currently failing in CI. The problem seems to be inconsistency in reverse lookups for the hostname.

Problems such as

[ERROR] testPartitions(org.apache.pulsar.proxy.server.ProxyParserTest)  Time elapsed: 130.261 s  <<< FAILURE!                          
org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientE
xception: java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeoutException: connection timed out: /10.1.0.56:37577   
        at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1027)                                  
        at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:95)                                       
        at org.apache.pulsar.proxy.server.ProxyParserTest.testPartitions(ProxyParserTest.java:153)                                     
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                               
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                       
        at java.lang.reflect.Method.invoke(Method.java:498)                                                                            
        at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)                                    
        at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)                                               
        at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)                                                 
        at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)                                                 
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)                                                                    
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)                                             
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)                                             
        at java.lang.Thread.run(Thread.java:748)                                                                                       
Caused by: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.Completion
Exception: io.netty.channel.ConnectTimeoutException: connection timed out: /10.1.0.56:37577                                            
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)                                                
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)                                                     
        at org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:93)                                       
        ... 13 more                                                                                                                    
Caused by: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeou
tException: connection timed out: /10.1.0.56:37577                                                                                     
        at org.apache.pulsar.client.impl.ConnectionPool.lambda$null$9(ConnectionPool.java:216)                                         
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)                                  
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)                          
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)                                                          
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)                                
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)                                                   
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)                                       
        ... 1 more                                                                                                                     
Caused by: java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeoutException: connection timed out: /10.1.0.56:37577
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)                                          
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)                                        
        at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:661)                                                
        at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)                                        
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)                                             
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)                                   
        at org.apache.pulsar.common.util.netty.ChannelFutures.lambda$toCompletableFuture$0(ChannelFutures.java:58)                     
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)                            

Example of the problem

Here's a screenshot of a ssh debugging session where I used https://github.com/lhotari/github-actions-debug and https://github.com/lhotari/action-upterm to get a ssh shell in a GitHub Runner VM.

When doing nslookup for the short hostname (hostname -s) or long hostname (hostname -f), nslookup returns various IP addresses which aren't the IP address of the current host.

image

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bugThe PR fixed a bug or issue reported a bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions