Cannot connect to containers when using docker-in-docker #4596
Description
I'm using docker-in-docker inside of kubernetes in jenkins. My containers appear to start up fine, but the host/port never becomes accessible and so testcontainers cannot connect to them.
Everything works fine when I run the tests locally using docker desktop, it's only on when running in jenkins using docker-in-docker that they fail.
The container ip address is 172.17.0.2 but it's trying to connect to 172.17.0.1 which is the gateway. Is that normal?
I've tested this on both testcontainers 1.16.0 and 1.16.1 and they both fail. This could be related to #4594, except my problem happens on 1.16.0 as well. This also might be related to #4591 although I am not using TESTCONTAINERS_HOST_OVERRIDE
Here is my test case where I start up an elasticsearch container, log the ipAddress and hostAddress, and then run docker inspect and log the results:
class ElasticsearchContainerTest {
companion object {
private val logger = KontextLogger.logger {}
private const val ELASTICSEARCH_VERSION = "6.7.0"
private val container = ElasticsearchContainer(
DockerImageName.parse("{REDACTED CONTAINER MIRROR URL}/elasticsearch")
.withTag(ELASTICSEARCH_VERSION)
.asCompatibleSubstituteFor("docker.elastic.co/elasticsearch/elasticsearch")
).withLogConsumer(Slf4jLogConsumer(KontextLogger.logger {}))
@BeforeAll
@JvmStatic
fun beforeAll() {
container.start()
}
@AfterAll
@JvmStatic
fun afterAll() {
logger.error("Container IP address: ${container.containerIpAddress}")
logger.error("Container url: ${container.httpHostAddress}")
val dockerInspect = container.dockerClient.inspectContainerCmd(container.containerId).exec()
val mapper = ObjectMapper().disable(SerializationFeature.FAIL_ON_EMPTY_BEANS)
val jsonOutput = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(dockerInspect)
logger.error("docker inspect elasticsearch: $jsonOutput")
container.stop()
}
}
@Test
fun `check that container is running`() {
Assert.assertTrue("Container is not running", container.isRunning)
}
}
The test fails due to Timed out waiting for URL to be accessible (http://172.17.0.1:49154/ should return HTTP 200:
:integrationTest > 0 tests completed> :integrationTest > Executing test com.example.test.Elasticsea
ElasticsearchContainerTest > initializationError FAILED
org.testcontainers.containers.ContainerLaunchException: Container startup failed
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:336)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
at com.example.test.ElasticsearchContainerTest$Companion.beforeAll(ElasticsearchContainerTest.kt:28)
at com.example.test.ElasticsearchContainerTest.beforeAll(ElasticsearchContainerTest.kt)
Caused by:
org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
... 3 more
Caused by:
org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:525)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 4 more
Caused by:
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://172.17.0.1:49154/ should return HTTP 200)
at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:264)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
... 6 more
Here is the output of docker inspect:
{
"Args" : [ "eswrapper" ],
"Config" : {
"healthcheck" : null,
"AttachStderr" : false,
"AttachStdin" : false,
"AttachStdout" : false,
"Cmd" : [ "eswrapper" ],
"Domainname" : "",
"Entrypoint" : [ "/usr/local/bin/docker-entrypoint.sh" ],
"Env" : [ "discovery.type=single-node", "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ELASTIC_CONTAINER=true", "JAVA_HOME=/opt/jdk-12" ],
"ExposedPorts" : {
"9200/tcp" : { },
"9300/tcp" : { }
},
"Hostname" : "8d8c72fac5a9",
"Image" : "{REDACTED CONTAINER MIRROR URL}/open-source-mirror/dockerhub/elasticsearch:6.7.0",
"Labels" : {
"license" : "Elastic License",
"org.label-schema.build-date" : "20190305",
"org.label-schema.license" : "GPLv2",
"org.label-schema.name" : "elasticsearch",
"org.label-schema.schema-version" : "1.0",
"org.label-schema.url" : "https://www.elastic.co/products/elasticsearch",
"org.label-schema.vcs-url" : "https://github.com/elastic/elasticsearch",
"org.label-schema.vendor" : "Elastic",
"org.label-schema.version" : "6.7.0",
"org.testcontainers" : "true",
"org.testcontainers.sessionId" : "671d3a01-4780-4f87-af21-ac568bafdc6f"
},
"MacAddress" : null,
"NetworkDisabled" : null,
"OnBuild" : null,
"OpenStdin" : false,
"PortSpecs" : null,
"StdinOnce" : false,
"Tty" : false,
"User" : "",
"Volumes" : null,
"WorkingDir" : "/usr/share/elasticsearch",
"Healthcheck" : null
},
"Created" : "2021-10-20T20:26:52.934006535Z",
"Driver" : "overlay2",
"ExecDriver" : null,
"HostConfig" : {
"Binds" : [ ],
"BlkioWeight" : 0,
"BlkioWeightDevice" : null,
"BlkioDeviceReadBps" : null,
"BlkioDeviceWriteBps" : null,
"BlkioDeviceReadIOps" : null,
"BlkioDeviceWriteIOps" : null,
"MemorySwappiness" : null,
"NanoCpus" : 0,
"CapAdd" : null,
"CapDrop" : null,
"ContainerIDFile" : "",
"CpuPeriod" : 0,
"CpuRealtimePeriod" : 0,
"CpuRealtimeRuntime" : 0,
"CpuShares" : 0,
"CpuQuota" : 0,
"CpusetCpus" : "",
"CpusetMems" : "",
"Devices" : null,
"DeviceCgroupRules" : null,
"DeviceRequests" : null,
"DiskQuota" : null,
"Dns" : null,
"DnsOptions" : null,
"DnsSearch" : null,
"ExtraHosts" : [ ],
"GroupAdd" : null,
"IpcMode" : "shareable",
"Cgroup" : "",
"Links" : null,
"LogConfig" : {
"Type" : "json-file",
"Config" : { }
},
"LxcConf" : null,
"Memory" : 0,
"MemorySwap" : 0,
"MemoryReservation" : 0,
"KernelMemory" : 0,
"NetworkMode" : "default",
"OomKillDisable" : false,
"Init" : null,
"AutoRemove" : false,
"OomScoreAdj" : 0,
"PortBindings" : {
"9200/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ],
"9300/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ]
},
"Privileged" : false,
"PublishAllPorts" : false,
"ReadonlyRootfs" : false,
"RestartPolicy" : {
"MaximumRetryCount" : 0,
"Name" : ""
},
"Ulimits" : null,
"CpuCount" : 0,
"CpuPercent" : 0,
"IOMaximumIOps" : 0,
"IOMaximumBandwidth" : 0,
"VolumesFrom" : [ ],
"Mounts" : null,
"PidMode" : "",
"Isolation" : null,
"SecurityOpt" : null,
"StorageOpt" : null,
"CgroupParent" : "",
"VolumeDriver" : "",
"ShmSize" : 67108864,
"PidsLimit" : null,
"Runtime" : "runc",
"Tmpfs" : null,
"UTSMode" : "",
"UsernsMode" : "",
"Sysctls" : null,
"ConsoleSize" : [ 0, 0 ]
},
"HostnamePath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hostname",
"HostsPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hosts",
"LogPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457-json.log",
"Id" : "8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457",
"SizeRootFs" : null,
"Image" : "sha256:02982be5777dea7bbf9573e1993c936b8add775d303bafd638924cf9635e8c38",
"MountLabel" : "",
"Name" : "/focused_roentgen",
"RestartCount" : 0,
"NetworkSettings" : {
"Bridge" : "",
"SandboxID" : "786b67295fe24a75cd4edcf66b69383e72e2972f80bf6b6a801f9c020d14bf27",
"HairpinMode" : false,
"LinkLocalIPv6Address" : "",
"LinkLocalIPv6PrefixLen" : 0,
"Ports" : {
"9200/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49154"
}, {
"HostIp" : "::",
"HostPort" : "49154"
} ],
"9300/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49153"
}, {
"HostIp" : "::",
"HostPort" : "49153"
} ]
},
"SandboxKey" : "/run/user/1000/docker/netns/786b67295fe2",
"SecondaryIPAddresses" : null,
"SecondaryIPv6Addresses" : null,
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"PortMapping" : null,
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"MacAddress" : "02:42:ac:11:00:02",
"Networks" : {
"bridge" : {
"IPAMConfig" : null,
"Links" : null,
"Aliases" : null,
"NetworkID" : "aaeb65fa1c11b6e4e4350d4b68b34d38796f2170d26073bdb0282fe14f4dc6a6",
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"MacAddress" : "02:42:ac:11:00:02"
}
}
},
"Path" : "/usr/local/bin/docker-entrypoint.sh",
"ProcessLabel" : "",
"ResolvConfPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/resolv.conf",
"ExecIDs" : null,
"State" : {
"oomkilled" : false,
"pidLong" : 873,
"exitCodeLong" : 0,
"Status" : "running",
"Running" : true,
"Paused" : false,
"Restarting" : false,
"OOMKilled" : false,
"Dead" : false,
"Pid" : 873,
"ExitCode" : 0,
"Error" : "",
"StartedAt" : "2021-10-20T20:26:53.468541843Z",
"FinishedAt" : "0001-01-01T00:00:00Z",
"Health" : null
},
"Volumes" : null,
"VolumesRW" : null,
"Node" : null,
"Mounts" : [ ],
"GraphDriver" : {
"Name" : "overlay2",
"Data" : {
"RootDir" : null,
"DeviceId" : null,
"DeviceName" : null,
"DeviceSize" : null,
"dir" : null
}
},
"Platform" : "linux",
"Path" : "/usr/local/bin/docker-entrypoint.sh",
"Args" : [ "eswrapper" ],
"ProcessLabel" : "",
"Mounts" : [ ],
"Platform" : "linux",
"Config" : {
"Hostname" : "8d8c72fac5a9",
"Domainname" : "",
"User" : "",
"AttachStdin" : false,
"AttachStdout" : false,
"AttachStderr" : false,
"ExposedPorts" : {
"9200/tcp" : { },
"9300/tcp" : { }
},
"Tty" : false,
"OpenStdin" : false,
"StdinOnce" : false,
"Env" : [ "discovery.type=single-node", "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "ELASTIC_CONTAINER=true", "JAVA_HOME=/opt/jdk-12" ],
"Cmd" : [ "eswrapper" ],
"Image" : "{REDACTED CONAINER MIRROR URL}/open-source-mirror/dockerhub/elasticsearch:6.7.0",
"Volumes" : null,
"WorkingDir" : "/usr/share/elasticsearch",
"Entrypoint" : [ "/usr/local/bin/docker-entrypoint.sh" ],
"OnBuild" : null,
"Labels" : {
"license" : "Elastic License",
"org.label-schema.build-date" : "20190305",
"org.label-schema.license" : "GPLv2",
"org.label-schema.name" : "elasticsearch",
"org.label-schema.schema-version" : "1.0",
"org.label-schema.url" : "https://www.elastic.co/products/elasticsearch",
"org.label-schema.vcs-url" : "https://github.com/elastic/elasticsearch",
"org.label-schema.vendor" : "Elastic",
"org.label-schema.version" : "6.7.0",
"org.testcontainers" : "true",
"org.testcontainers.sessionId" : "671d3a01-4780-4f87-af21-ac568bafdc6f"
}
},
"Driver" : "overlay2",
"AppArmorProfile" : "",
"HostsPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hosts",
"HostnamePath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/hostname",
"GraphDriver" : {
"Data" : {
"LowerDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5-init/diff:/home/rootless/.local/share/docker/overlay2/1e71332ad2a429a4d06d157b6c4e2c4401da2b44b640d412557feb3c39452010/diff:/home/rootless/.local/share/docker/overlay2/1302968f5de6c81580eadb218555c0bc49bd3af7c766e77cbdb0ea9f02371d2a/diff:/home/rootless/.local/share/docker/overlay2/3b2b6c22a55172df81ac6af6f439aa3ec1cfe0576269d9fab8c25772e49a65ca/diff:/home/rootless/.local/share/docker/overlay2/42a8e273145d92d47a2c6827fa4d276800e98189da117ca69b25808ce1013edc/diff:/home/rootless/.local/share/docker/overlay2/be84a76283d29eac19bb5e262a4b26578218877d2302cea775fc57c29e195fb0/diff:/home/rootless/.local/share/docker/overlay2/05a94b779ac49607562a956de5a274e5cd000747ef966e612f156ba3f5018cb3/diff:/home/rootless/.local/share/docker/overlay2/57967e2c3b2027a4b67e5d97544d36b11573228ae195fb1cb0799b0ca954c669/diff",
"MergedDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/merged",
"UpperDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/diff",
"WorkDir" : "/home/rootless/.local/share/docker/overlay2/b90bba9ed9340bb782431671713047f5ef740d51818a6e82ae64d9246d8215a5/work"
},
"Name" : "overlay2"
},
"Image" : "sha256:02982be5777dea7bbf9573e1993c936b8add775d303bafd638924cf9635e8c38",
"Created" : "2021-10-20T20:26:52.934006535Z",
"Name" : "/focused_roentgen",
"RestartCount" : 0,
"NetworkSettings" : {
"Bridge" : "",
"SandboxID" : "786b67295fe24a75cd4edcf66b69383e72e2972f80bf6b6a801f9c020d14bf27",
"HairpinMode" : false,
"LinkLocalIPv6Address" : "",
"LinkLocalIPv6PrefixLen" : 0,
"Ports" : {
"9200/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49154"
}, {
"HostIp" : "::",
"HostPort" : "49154"
} ],
"9300/tcp" : [ {
"HostIp" : "0.0.0.0",
"HostPort" : "49153"
}, {
"HostIp" : "::",
"HostPort" : "49153"
} ]
},
"SandboxKey" : "/run/user/1000/docker/netns/786b67295fe2",
"SecondaryIPAddresses" : null,
"SecondaryIPv6Addresses" : null,
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"MacAddress" : "02:42:ac:11:00:02",
"Networks" : {
"bridge" : {
"IPAMConfig" : null,
"Links" : null,
"Aliases" : null,
"NetworkID" : "aaeb65fa1c11b6e4e4350d4b68b34d38796f2170d26073bdb0282fe14f4dc6a6",
"EndpointID" : "d2a62c1fbd8241342d7f782109a54fa9caf8fbe1a9d07408aaefa695abd82d87",
"Gateway" : "172.17.0.1",
"IPAddress" : "172.17.0.2",
"IPPrefixLen" : 16,
"IPv6Gateway" : "",
"GlobalIPv6Address" : "",
"GlobalIPv6PrefixLen" : 0,
"MacAddress" : "02:42:ac:11:00:02",
"DriverOpts" : null
}
}
},
"MountLabel" : "",
"State" : {
"Status" : "running",
"Running" : true,
"Paused" : false,
"Restarting" : false,
"OOMKilled" : false,
"Dead" : false,
"Pid" : 873,
"ExitCode" : 0,
"Error" : "",
"StartedAt" : "2021-10-20T20:26:53.468541843Z",
"FinishedAt" : "0001-01-01T00:00:00Z"
},
"ResolvConfPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/resolv.conf",
"ExecIDs" : null,
"HostConfig" : {
"Binds" : [ ],
"ContainerIDFile" : "",
"LogConfig" : {
"Type" : "json-file",
"Config" : { }
},
"NetworkMode" : "default",
"PortBindings" : {
"9200/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ],
"9300/tcp" : [ {
"HostIp" : "",
"HostPort" : ""
} ]
},
"RestartPolicy" : {
"Name" : "",
"MaximumRetryCount" : 0
},
"AutoRemove" : false,
"VolumeDriver" : "",
"VolumesFrom" : [ ],
"CapAdd" : null,
"CapDrop" : null,
"CgroupnsMode" : "host",
"Dns" : null,
"DnsOptions" : null,
"DnsSearch" : null,
"ExtraHosts" : [ ],
"GroupAdd" : null,
"IpcMode" : "shareable",
"Cgroup" : "",
"Links" : null,
"OomScoreAdj" : 0,
"PidMode" : "",
"Privileged" : false,
"PublishAllPorts" : false,
"ReadonlyRootfs" : false,
"SecurityOpt" : null,
"UTSMode" : "",
"UsernsMode" : "",
"ShmSize" : 67108864,
"Runtime" : "runc",
"ConsoleSize" : [ 0, 0 ],
"Isolation" : "",
"CpuShares" : 0,
"Memory" : 0,
"NanoCpus" : 0,
"CgroupParent" : "",
"BlkioWeight" : 0,
"BlkioWeightDevice" : null,
"BlkioDeviceReadBps" : null,
"BlkioDeviceWriteBps" : null,
"BlkioDeviceReadIOps" : null,
"BlkioDeviceWriteIOps" : null,
"CpuPeriod" : 0,
"CpuQuota" : 0,
"CpuRealtimePeriod" : 0,
"CpuRealtimeRuntime" : 0,
"CpusetCpus" : "",
"CpusetMems" : "",
"Devices" : null,
"DeviceCgroupRules" : null,
"DeviceRequests" : null,
"KernelMemory" : 0,
"KernelMemoryTCP" : 0,
"MemoryReservation" : 0,
"MemorySwap" : 0,
"MemorySwappiness" : null,
"OomKillDisable" : false,
"PidsLimit" : null,
"Ulimits" : null,
"CpuCount" : 0,
"CpuPercent" : 0,
"IOMaximumIOps" : 0,
"IOMaximumBandwidth" : 0,
"MaskedPaths" : [ "/proc/asound", "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ],
"ReadonlyPaths" : [ "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ]
},
"Id" : "8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457",
"LogPath" : "/home/rootless/.local/share/docker/containers/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457/8d8c72fac5a93471e0407a085e0b79236542b0ec01fc1f5d8c74024aa80b3457-json.log"
}
Some additional log lines which might help:
INFO org.testcontainers.containers.wait.strategy.HttpWaitStrategy - /focused_roentgen: Waiting for 120 seconds for URL: http://172.17.0.1:49154/ (where port 49154 maps to container port 9200)
Log output from the failed container:
[INFO ][o.e.e.NodeEnvironment ] [dP6fGhj] using [1] data paths, mounts [[/ (overlay)]], net usable_space [79.4gb], net total_space [242.2gb], types [overlay]
[INFO ][o.e.e.NodeEnvironment ] [dP6fGhj] heap size [990.7mb], compressed ordinary object pointers [true]
[INFO ][o.e.n.Node ] [dP6fGhj] node name derived from node ID [dP6fGhj8TES6gRpq72sI0Q]; set [node.name] to override
[INFO ][o.e.n.Node ] [dP6fGhj] version[6.7.0], pid[1], build[default/docker/8453f77/2019-03-21T15:32:29.844721Z], OS[Linux/5.4.0-1028-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[INFO ][o.e.n.Node ] [dP6fGhj] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-2290863965764670913, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [aggs-matrix-stats]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [analysis-common]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [ingest-common]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [ingest-geoip]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [ingest-user-agent]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [lang-expression]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [lang-mustache]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [lang-painless]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [mapper-extras]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [parent-join]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [percolator]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [rank-eval]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [reindex]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [repository-url]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [transport-netty4]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [tribe]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-ccr]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-core]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-deprecation]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-graph]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-ilm]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-logstash]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-ml]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-monitoring]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-rollup]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-security]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-sql]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-upgrade]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] loaded module [x-pack-watcher]
[INFO ][o.e.p.PluginsService ] [dP6fGhj] no plugins loaded
[INFO ][o.e.x.s.a.s.FileRolesStore] [dP6fGhj] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[INFO ][o.e.x.m.p.l.CppLogMessageHandler] [dP6fGhj] [controller/88] [Main.cc@109] controller (64 bit): Version 6.7.0 (Build d74ae2ac01b10d) Copyright (c) 2019 Elasticsearch BV
[INFO ][o.e.d.DiscoveryModule ] [dP6fGhj] using discovery type [single-node] and host providers [settings]
[INFO ][o.e.n.Node ] [dP6fGhj] initialized
[INFO ][o.e.n.Node ] [dP6fGhj] starting ...
[INFO ][o.e.t.TransportService ] [dP6fGhj] publish_address {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}
[INFO ][o.e.h.n.Netty4HttpServerTransport] [dP6fGhj] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[INFO ][o.e.n.Node ] [dP6fGhj] started
[2021-10-20T20:28:53.705Z] [2021-10-20T20:27:00,528][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [dP6fGhj] Failed to clear cache for realms [[]]
[INFO ][o.e.g.GatewayService ] [dP6fGhj] recovered [0] indices into cluster_state
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.watches] for index patterns [.watches*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[INFO ][o.e.c.m.MetaDataIndexTemplateService] [dP6fGhj] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[INFO ][o.e.l.LicenseService ] [dP6fGhj] license [2dcb2824-6c74-4c3a-b8ac-d06233221f44] mode [basic] - valid
When I run a similar test for postgres, it passes but that's only because the postgres testcontainer uses a log watching wait strategy and doesn't try to connect to the exposed port. If I try to actually connect to the postgres container it fails as well.
My kubernetes podspec is like this:
podSpec:
spec:
containers:
- name: main
image: <redacted, but my image is based on debian>
command:
- cat
tty: true
volumeMounts:
- mountPath: /var/run
name: docker-socket
env:
- name: TESTCONTAINERS_RYUK_DISABLED
value: true
- name: dind-daemon
image: docker:20.10.9-dind-rootless
securityContext:
privileged: true
readOnlyRootFilesystem: false
volumeMounts:
- name: docker-socket
mountPath: /run/user/1000
volumes:
- name: docker-socket
emptyDir: {}