ESQL: Add spec-driven distributed integration tests for external sources#143336
Merged
costin merged 7 commits intoelastic:mainfrom Mar 1, 2026
Merged
ESQL: Add spec-driven distributed integration tests for external sources#143336costin merged 7 commits intoelastic:mainfrom
costin merged 7 commits intoelastic:mainfrom
Conversation
External source distributed execution has no end-to-end test coverage. Existing tests only assert plan structure at the unit level. This adds a multi-node integration test that runs the same csv-spec queries across all three distribution strategies (coordinator_only, round_robin, adaptive) and asserts identical results — any divergence flags a split assignment, exchange, or aggregation bug. - multi-node build.gradle: add S3/GCS fixtures, datasource plugins - ExternalDistributedClusters: 3-node cluster with S3 fixture wiring - ExternalDistributedSpecIT: parameterized test runner cross-producting external-basic specs with storage backends and distribution modes Developed using AI-assisted tooling
Collaborator
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
costin
added a commit
to costin/elasticsearch
that referenced
this pull request
Feb 28, 2026
Randomized tests that verify mathematical invariants of the split assignment algorithm: every split assigned exactly once, bounded load imbalance (no node exceeds ceil(splits/nodes)), deterministic output, and correct behavior at boundary conditions (single split, zero nodes, more nodes than splits). Builds on PR-D1 (elastic#143336) distributed integration test infrastructure. - ExternalDistributionPropertyTests: 11 property-based tests covering RoundRobinStrategy, AdaptiveStrategy, and CoordinatorOnlyStrategy Developed using AI-assisted tooling
b8b584e to
4e2f5e7
Compare
When test resources (iceberg-fixtures) are bundled inside a JAR by the internal-test-artifact plugin, the existing code asserted file: protocol URLs and failed. Expand S3FixtureUtils and AbstractExternalSourceSpecTestCase to enumerate and load fixtures from both filesystem and JAR sources. - S3FixtureUtils: added forEachFixtureEntry() and resolveLocalFixturesPath() utilities; loadFixturesFromResources() now dispatches to loadFixturesFromJar() - AbstractExternalSourceSpecTestCase: loadGcsFixtures(), generateCompressedFixtures(), resolveLocalFixturesPath() now use the shared utility instead of direct file walks Developed using AI-assisted tooling
4e2f5e7 to
fd9ddae
Compare
When test resources are bundled inside a JAR (CI), there is no filesystem path for the LOCAL storage backend. Exclude LOCAL from the test matrix when resolveLocalFixturesPath returns null. Also filter Azure SDK reactor-netty thread leaks in the distributed integration test suite. - AbstractExternalSourceSpecTestCase: BACKENDS list now computed at class-load time, excluding LOCAL when fixtures are in a JAR - ExternalDistributedSpecIT: added AzureReactorThreadFilter for reactor-http-nio and boundedElastic threads from Azure SDK Developed using AI-assisted tooling
ExternalDistributedClusters called ElasticsearchCluster.local().nodes(3) directly, which fails in serverless mode where changing the node count is not supported. Delegate to Clusters.testCluster() — the same pattern used by all other multi-node tests — so the serverless infrastructure can substitute its own cluster builder. - ExternalDistributedClusters: delegate to Clusters.testCluster() Developed using AI-assisted tooling
In serverless mode the datasource connector plugins (esql-datasource-s3, esql-datasource-gcs, etc.) are not installed, causing "Unsupported storage scheme" errors. Add a runtime probe that detects missing connectors and skips the test via assumeTrue. - ExternalDistributedSpecIT: override shouldSkipTest with connector check Developed using AI-assisted tooling
bpintea
approved these changes
Mar 1, 2026
costin
added a commit
that referenced
this pull request
Mar 1, 2026
* ESQL: Add spec-driven distributed integration tests for external sources External source distributed execution has no end-to-end test coverage. Existing tests only assert plan structure at the unit level. This adds a multi-node integration test that runs the same csv-spec queries across all three distribution strategies (coordinator_only, round_robin, adaptive) and asserts identical results — any divergence flags a split assignment, exchange, or aggregation bug. - multi-node build.gradle: add S3/GCS fixtures, datasource plugins - ExternalDistributedClusters: 3-node cluster with S3 fixture wiring - ExternalDistributedSpecIT: parameterized test runner cross-producting external-basic specs with storage backends and distribution modes Developed using AI-assisted tooling * ESQL: Add distribution property tests for external sources Randomized tests that verify mathematical invariants of the split assignment algorithm: every split assigned exactly once, bounded load imbalance (no node exceeds ceil(splits/nodes)), deterministic output, and correct behavior at boundary conditions (single split, zero nodes, more nodes than splits). Builds on PR-D1 (#143336) distributed integration test infrastructure. - ExternalDistributionPropertyTests: 11 property-based tests covering RoundRobinStrategy, AdaptiveStrategy, and CoordinatorOnlyStrategy Developed using AI-assisted tooling * ESQL: Handle JAR-packaged fixture resources in tests When test resources (iceberg-fixtures) are bundled inside a JAR by the internal-test-artifact plugin, the existing code silently skipped loading or asserted file: protocol URLs and failed. Expand S3FixtureUtils and AbstractExternalSourceSpecTestCase to enumerate and load fixtures from both filesystem and JAR sources. - S3FixtureUtils: added forEachFixtureEntry() and resolveLocalFixturesPath() utilities; loadFixturesFromResources() now dispatches to loadFixturesFromJar() - AbstractExternalSourceSpecTestCase: loadGcsFixtures(), generateCompressedFixtures(), resolveLocalFixturesPath() now use the shared utility instead of direct file walks Developed using AI-assisted tooling * ESQL: Skip LOCAL backend tests when fixtures are JAR-packaged When test resources are bundled inside a JAR (CI), there is no filesystem path for the LOCAL storage backend. Exclude LOCAL from the test matrix when resolveLocalFixturesPath returns null. Also filter Azure SDK reactor-netty thread leaks in the distributed integration test suite. - AbstractExternalSourceSpecTestCase: BACKENDS list now computed at class-load time, excluding LOCAL when fixtures are in a JAR - ExternalDistributedSpecIT: added AzureReactorThreadFilter for reactor-http-nio and boundedElastic threads from Azure SDK Developed using AI-assisted tooling * ESQL: Use Clusters.testCluster for distributed tests ExternalDistributedClusters called ElasticsearchCluster.local().nodes(3) directly, which fails in serverless mode where changing the node count is not supported. Delegate to Clusters.testCluster() — the same pattern used by all other multi-node tests — so the serverless infrastructure can substitute its own cluster builder. - ExternalDistributedClusters: delegate to Clusters.testCluster() Developed using AI-assisted tooling * ESQL: Skip external distributed tests when connectors unavailable In serverless mode the datasource connector plugins (esql-datasource-s3, esql-datasource-gcs, etc.) are not installed, causing "Unsupported storage scheme" errors. Add a runtime probe that detects missing connectors and skips the test via assumeTrue. - ExternalDistributedSpecIT: override shouldSkipTest with connector check Developed using AI-assisted tooling * ESQL: Address review feedback on distributed tests - ExternalDistributedClusters: use System.getProperty("java.io.tmpdir") instead of hardcoded "/tmp" for the fixtures path fallback - ExternalDistributedSpecIT: reuse framework AzureReactorThreadFilter, keep a small AzureSdkThreadFilter for the extra "azure-sdk-" prefix Developed using AI-assisted tooling
2 tasks
tballison
pushed a commit
to tballison/elasticsearch
that referenced
this pull request
Mar 3, 2026
…ces (elastic#143336) External source distributed execution has no end-to-end test coverage — existing tests only assert plan structure at the unit level. There is no verification that different distribution strategies produce identical results when running the same query. This adds a multi-node integration test that runs the same csv-spec queries across all three distribution strategies (coordinator_only, round_robin, adaptive) and asserts identical results. Any divergence immediately flags a split assignment, exchange, or aggregation bug. The test uses a 3-node cluster with S3/GCS fixtures and cross-products the existing `external-basic.csv-spec` tests with all storage backends and distribution modes. Part of elastic#143330 Developed using AI-assisted tooling
tballison
pushed a commit
to tballison/elasticsearch
that referenced
this pull request
Mar 3, 2026
…43341) * ESQL: Add spec-driven distributed integration tests for external sources External source distributed execution has no end-to-end test coverage. Existing tests only assert plan structure at the unit level. This adds a multi-node integration test that runs the same csv-spec queries across all three distribution strategies (coordinator_only, round_robin, adaptive) and asserts identical results — any divergence flags a split assignment, exchange, or aggregation bug. - multi-node build.gradle: add S3/GCS fixtures, datasource plugins - ExternalDistributedClusters: 3-node cluster with S3 fixture wiring - ExternalDistributedSpecIT: parameterized test runner cross-producting external-basic specs with storage backends and distribution modes Developed using AI-assisted tooling * ESQL: Add distribution property tests for external sources Randomized tests that verify mathematical invariants of the split assignment algorithm: every split assigned exactly once, bounded load imbalance (no node exceeds ceil(splits/nodes)), deterministic output, and correct behavior at boundary conditions (single split, zero nodes, more nodes than splits). Builds on PR-D1 (elastic#143336) distributed integration test infrastructure. - ExternalDistributionPropertyTests: 11 property-based tests covering RoundRobinStrategy, AdaptiveStrategy, and CoordinatorOnlyStrategy Developed using AI-assisted tooling * ESQL: Handle JAR-packaged fixture resources in tests When test resources (iceberg-fixtures) are bundled inside a JAR by the internal-test-artifact plugin, the existing code silently skipped loading or asserted file: protocol URLs and failed. Expand S3FixtureUtils and AbstractExternalSourceSpecTestCase to enumerate and load fixtures from both filesystem and JAR sources. - S3FixtureUtils: added forEachFixtureEntry() and resolveLocalFixturesPath() utilities; loadFixturesFromResources() now dispatches to loadFixturesFromJar() - AbstractExternalSourceSpecTestCase: loadGcsFixtures(), generateCompressedFixtures(), resolveLocalFixturesPath() now use the shared utility instead of direct file walks Developed using AI-assisted tooling * ESQL: Skip LOCAL backend tests when fixtures are JAR-packaged When test resources are bundled inside a JAR (CI), there is no filesystem path for the LOCAL storage backend. Exclude LOCAL from the test matrix when resolveLocalFixturesPath returns null. Also filter Azure SDK reactor-netty thread leaks in the distributed integration test suite. - AbstractExternalSourceSpecTestCase: BACKENDS list now computed at class-load time, excluding LOCAL when fixtures are in a JAR - ExternalDistributedSpecIT: added AzureReactorThreadFilter for reactor-http-nio and boundedElastic threads from Azure SDK Developed using AI-assisted tooling * ESQL: Use Clusters.testCluster for distributed tests ExternalDistributedClusters called ElasticsearchCluster.local().nodes(3) directly, which fails in serverless mode where changing the node count is not supported. Delegate to Clusters.testCluster() — the same pattern used by all other multi-node tests — so the serverless infrastructure can substitute its own cluster builder. - ExternalDistributedClusters: delegate to Clusters.testCluster() Developed using AI-assisted tooling * ESQL: Skip external distributed tests when connectors unavailable In serverless mode the datasource connector plugins (esql-datasource-s3, esql-datasource-gcs, etc.) are not installed, causing "Unsupported storage scheme" errors. Add a runtime probe that detects missing connectors and skips the test via assumeTrue. - ExternalDistributedSpecIT: override shouldSkipTest with connector check Developed using AI-assisted tooling * ESQL: Address review feedback on distributed tests - ExternalDistributedClusters: use System.getProperty("java.io.tmpdir") instead of hardcoded "/tmp" for the fixtures path fallback - ExternalDistributedSpecIT: reuse framework AzureReactorThreadFilter, keep a small AzureSdkThreadFilter for the extra "azure-sdk-" prefix Developed using AI-assisted tooling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
External source distributed execution has no end-to-end test coverage — existing tests only assert plan structure at the unit level. There is no verification that different distribution strategies produce identical results when running the same query.
This adds a multi-node integration test that runs the same csv-spec queries across all three distribution strategies (coordinator_only, round_robin, adaptive) and asserts identical results. Any divergence immediately flags a split assignment, exchange, or aggregation bug.
The test uses a 3-node cluster with S3/GCS fixtures and cross-products the existing
external-basic.csv-spectests with all storage backends and distribution modes.Part of #143330
Developed using AI-assisted tooling