Skip to content

[ci] Add PR pipeline for testing ipv6#140473

Merged
brianseeders merged 87 commits intoelastic:mainfrom
brianseeders:ipv6-pipelines
Mar 13, 2026
Merged

[ci] Add PR pipeline for testing ipv6#140473
brianseeders merged 87 commits intoelastic:mainfrom
brianseeders:ipv6-pipelines

Conversation

@brianseeders
Copy link
Copy Markdown
Contributor

@brianseeders brianseeders commented Jan 9, 2026

Overview

This PR is to support running most of our tests in an IPv6-only environment.

The running environment looks like this:

  • External IPv6 address, no external IPv4
  • NAT64 and DNS64 to that IPv4-only services on the open Internet are still reachable via IPv6
  • localhost resolves to the IPv6 loopback address only
  • Java/gradle is run with java.net.preferIPv6Addresses=true
  • ES test clusters started with "network.host": "_local:ipv6_" by default when java.net.preferIPv6Addresses=true

Stuff in this PR:

  • Add an optional PR pipeline for running tests with IPv6, can be enabled by adding test-ipv6 label
  • Add a weekly periodic pipeline for running the IPv6 tests on main branches weekly
  • Refactor tests all over the place that have "127.0.0.1", "localhost", etc hard-coded
  • Refactor tests all over the place that construct URLs using string concat, which fails when the host string is an IPv6 address, e.g. http://::1:80 needs to be http://[::1]:80

Skipped Tests

There are a few sets of tests that were skipped when running with IPv6:

.../azure/classic/AzureDiscoveryClusterFormationTests.java

I'm not really sure what was ultimately going on here. Possibly a problem with an Azure SDK. The responsible team will need to address.

Several yamlRestCompatTest

There are several YAML tests that needed to be updated to parse IPv6 IP addresses correctly. However, these tests check out the YAML that exists on older branches and uses it to run tests, so these tests have to be skipped until the YAML is updated on the old branches.

Any test using HdfsFixture.java

Hadoop doesn't support IPv6, so these tests fail. See comment in the code for more details.

Any test using Krb5kDcContainer.java

There's configuration for this fixture hard-coded to 127.0.0.1, and fixing it is more complicated than just changing that. The responsible team will have to take care of this one.

Running locally

Here is how I run the tests locally (we should add a command or something that does this for convenience):

export JAVA_TOOL_OPTIONS="-Djava.net.preferIPv6Addresses=true -Djava.net.preferIPv4Stack=false"
export _JAVA_OPTIONS="-Djava.net.preferIPv6Addresses=true -Djava.net.preferIPv4Stack=false"

./gradlew -Djava.net.preferIPv6Addresses=true -Djava.net.preferIPv4Stack=false <tasks>

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
test/framework/src/main/java/org/elasticsearch/transport/AbstractSimpleTransportTestCase.java (1)

3229-3240: ⚠️ Potential issue | 🟠 Major

Loopback normalization still misses the real formatted address.

serviceB.getLocalNode().getAddress().toString() includes the port, so neither ^::1$ nor ^\[::1\]$ matches values like [::1]:9300. In IPv6-only runs this assertion can still fail against the expanded loopback form this test is trying to tolerate. Normalize the host before formatting, or assert host and port separately.

Suggested fix
-            String address = serviceB.getLocalNode().getAddress().toString();
-            address = address.replaceFirst("^::1$", "0:0:0:0:0:0:0:1").replaceFirst("^\\[::1\\]$", "[0:0:0:0:0:0:0:1]");
+            final InetSocketAddress localAddress = serviceB.getLocalNode().getAddress().address();
+            final String hostAddress = localAddress.getAddress() instanceof Inet6Address && localAddress.getAddress().isLoopbackAddress()
+                ? "0:0:0:0:0:0:0:1"
+                : localAddress.getAddress().getHostAddress();

             assertThat(
                 channel.toString(),
                 allOf(
                     containsString("TcpTransportChannel"),
                     containsString('{' + ACTION + '}'),
                     containsString("TaskTransportChannel{task=" + task.getId() + '}'),
                     containsString("localAddress="),
-                    containsString(address)
+                    containsString(hostAddress),
+                    containsString(":" + localAddress.getPort())
                 )
             );
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@test/framework/src/main/java/org/elasticsearch/transport/AbstractSimpleTransportTestCase.java`
around lines 3229 - 3240, The assertion fails because
serviceB.getLocalNode().getAddress().toString() includes a port so the regexes
^::1$ and ^\[::1\]$ won't match; in AbstractSimpleTransportTestCase adjust the
test to normalize the host portion before formatting or compare host and port
separately: parse the address string returned by
serviceB.getLocalNode().getAddress().toString() to extract host and port,
normalize the host by replacing "::1" or "[::1]" with the expanded
"0:0:0:0:0:0:0:1" (preserving brackets when present), then recombine with the
port for the containsString(address) check (or assert containsString(host) and
containsString(":" + port) separately) so the loopback normalization works for
values like "[::1]:9300".
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In
`@test/framework/src/main/java/org/elasticsearch/transport/AbstractSimpleTransportTestCase.java`:
- Around line 3229-3240: The assertion fails because
serviceB.getLocalNode().getAddress().toString() includes a port so the regexes
^::1$ and ^\[::1\]$ won't match; in AbstractSimpleTransportTestCase adjust the
test to normalize the host portion before formatting or compare host and port
separately: parse the address string returned by
serviceB.getLocalNode().getAddress().toString() to extract host and port,
normalize the host by replacing "::1" or "[::1]" with the expanded
"0:0:0:0:0:0:0:1" (preserving brackets when present), then recombine with the
port for the containsString(address) check (or assert containsString(host) and
containsString(":" + port) separately) so the loopback normalization works for
values like "[::1]:9300".

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 231856b2-98c3-4984-9594-8eacfda74be3

📥 Commits

Reviewing files that changed from the base of the PR and between 654f069 and 7219652.

📒 Files selected for processing (5)
  • modules/repository-azure/src/test/java/org/elasticsearch/repositories/azure/AzureBlobContainerRetriesTests.java
  • test/framework/src/main/java/org/elasticsearch/repositories/blobstore/ESMockAPIBasedRepositoryIntegTestCase.java
  • test/framework/src/main/java/org/elasticsearch/transport/AbstractSimpleTransportTestCase.java
  • x-pack/plugin/core/src/test/java/org/elasticsearch/test/http/MockWebServer.java
  • x-pack/plugin/sql/sql-client/src/test/java/org/elasticsearch/xpack/sql/client/HttpClientRequestTests.java
🚧 Files skipped from review as they are similar to previous changes (2)
  • x-pack/plugin/core/src/test/java/org/elasticsearch/test/http/MockWebServer.java
  • modules/repository-azure/src/test/java/org/elasticsearch/repositories/azure/AzureBlobContainerRetriesTests.java

Copy link
Copy Markdown
Contributor

@jozala jozala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@brianseeders brianseeders merged commit 12a10a9 into elastic:main Mar 13, 2026
29 of 38 checks passed
@brianseeders
Copy link
Copy Markdown
Contributor Author

💔 Some backports could not be created

Status Branch Result
9.3
9.2 An unhandled error occurred. Please see the logs for details
8.19 The branch "elastic/8.19'" is invalid or doesn't exist

Manual backport

To create the backport manually run:

backport --pr 140473

Questions ?

Please refer to the Backport tool documentation

brianseeders added a commit to brianseeders/elasticsearch that referenced this pull request Mar 13, 2026
…ipv6 (elastic#140473)

(cherry picked from commit 12a10a9)

# Conflicts:
#	modules/reindex-management/src/internalClusterTest/java/org/elasticsearch/reindex/management/ReindexRelocationIT.java
#	modules/repository-azure/src/test/java/org/elasticsearch/repositories/azure/AzureBlobContainerRetriesTests.java
#	test/external-modules/apm-integration/src/javaRestTest/java/org/elasticsearch/test/apmintegration/MetricsApmIT.java
#	test/fixtures/aws-sts-fixture/src/main/java/fixture/aws/sts/AwsStsHttpFixture.java
#	x-pack/plugin/sql/sql-client/src/test/java/org/elasticsearch/xpack/sql/client/HttpClientRequestTests.java
brianseeders added a commit to brianseeders/elasticsearch that referenced this pull request Mar 13, 2026
…ipv6 (elastic#140473)

(cherry picked from commit 12a10a9)

# Conflicts:
#	modules/reindex-management/src/internalClusterTest/java/org/elasticsearch/reindex/management/ReindexRelocationIT.java
#	modules/repository-azure/src/test/java/org/elasticsearch/repositories/azure/AzureBlobContainerRetriesTests.java
#	rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/cat.nodes/10_basic.yml
#	test/external-modules/apm-integration/src/javaRestTest/java/org/elasticsearch/test/apmintegration/MetricsApmIT.java
#	test/fixtures/aws-sts-fixture/src/main/java/fixture/aws/sts/AwsStsHttpFixture.java
#	x-pack/plugin/inference/qa/inference-service-tests/src/javaRestTest/java/org/elasticsearch/xpack/inference/MlModelServer.java
#	x-pack/plugin/sql/sql-client/src/test/java/org/elasticsearch/xpack/sql/client/HttpClientRequestTests.java
@brianseeders
Copy link
Copy Markdown
Contributor Author

💚 All backports created successfully

Status Branch Result
9.2
8.19

Questions ?

Please refer to the Backport tool documentation

brianseeders added a commit to brianseeders/elasticsearch that referenced this pull request Mar 13, 2026
…ipv6 (elastic#140473)

(cherry picked from commit 12a10a9)

# Conflicts:
#	modules/reindex-management/src/internalClusterTest/java/org/elasticsearch/reindex/management/ReindexRelocationIT.java
#	modules/repository-azure/src/internalClusterTest/java/org/elasticsearch/repositories/azure/AzureBlobStoreRepositoryTests.java
#	modules/repository-azure/src/test/java/org/elasticsearch/repositories/azure/AzureBlobContainerRetriesTests.java
#	modules/repository-gcs/src/test/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageBlobContainerStatsTests.java
#	rest-api-spec/build.gradle
#	rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/cat.nodes/10_basic.yml
#	server/src/test/java/org/elasticsearch/transport/TransportInfoTests.java
#	test/external-modules/apm-integration/src/javaRestTest/java/org/elasticsearch/test/apmintegration/MetricsApmIT.java
#	test/fixtures/aws-sts-fixture/src/main/java/fixture/aws/sts/AwsStsHttpFixture.java
#	test/fixtures/s3-fixture/src/main/java/fixture/s3/S3HttpFixture.java
#	test/fixtures/url-fixture/src/main/java/fixture/url/URLFixture.java
#	x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/ssl/HttpClient5SslTests.java
#	x-pack/plugin/inference/qa/inference-service-tests/src/javaRestTest/java/org/elasticsearch/xpack/inference/MlModelServer.java
#	x-pack/plugin/sql/sql-client/src/test/java/org/elasticsearch/xpack/sql/client/HttpClientRequestTests.java
#	x-pack/qa/evil-tests/src/test/java/org/elasticsearch/xpack/security/authc/kerberos/SimpleKdcLdapServer.java
szybia added a commit to szybia/elasticsearch that referenced this pull request Mar 13, 2026
…elocations

* upstream/main: (72 commits)
  [Test] Randomly disable sequence numbers in CcrTimeSeriesDataStreamsIT (elastic#143930)
  Fix AsyncSearchIndexServiceTests.testCircuitBreaker failure (elastic#144058)
  Refine GenerativeIT some more, this time with accounting for some added (elastic#144220)
  ESQL: Physical Planning on the Lookup Node (elastic#143707)
  Mute org.elasticsearch.xpack.esql.CsvIT test {csv-spec:approximation.Approximate stats by with zero variance} elastic#144240
  Trigger counter metrics in test for delta temporality measurements (elastic#144193)
  fix capabiltiy approximation_v3 (elastic#144230)
  [ci] Add PR pipeline for testing ipv6 and fix tests not working with ipv6 (elastic#140473)
  update (elastic#144095)
  Make from/to optional in TBUCKET when Kibana timestamp filter is present (elastic#144057)
  Extract reroute behavior from create-index request classes (elastic#144140)
  ESQL: Fix release build only failures (elastic#144122)
  ES|QL query approximation: move sample correction to data node (elastic#144005)
  Add indexing pressure tracking to OTLP endpoints (elastic#144009)
  Fix replica writes after _seq_no doc values are pruned (elastic#144180)
  allow tests to configure supportsLoadingConfig (elastic#144061)
  [ES|QL] Unmute testGiantTextFieldInSubqueryIntermediateResultsWithSort (elastic#144126)
  [ESQL][DOCS] Add CPS page (unpublished for moment) (elastic#144206)
  ESQL: Forbid "load" unmapped_fields for certain commands (elastic#144115)
  Add CCS Remote Views Detection (elastic#143384)
  ...
ncordon pushed a commit to ncordon/elasticsearch that referenced this pull request Mar 16, 2026
michalborek pushed a commit to michalborek/elasticsearch that referenced this pull request Mar 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants