ESQL: Add pluggable partition detection and virtual columns#143120
Merged
costin merged 7 commits intoelastic:mainfrom Feb 26, 2026
Merged
ESQL: Add pluggable partition detection and virtual columns#143120costin merged 7 commits intoelastic:mainfrom
costin merged 7 commits intoelastic:mainfrom
Conversation
Introduce a PartitionDetector SPI with Hive, template, and auto strategies so partition columns are discovered from file paths and injected as virtual columns into the schema and execution pages. - PartitionDetector, PartitionConfig, HivePartitionDetector, TemplatePartitionDetector, AutoPartitionDetector: pluggable detection - VirtualColumnInjector: injects constant blocks for partition values - ExternalSourceResolver: enriches schema with ReferenceAttribute partition columns (not FieldAttribute) preserving path order - GlobExpander: template-aware glob rewriting with filter hints - PartitionMetadata, FileSplit, SourceOperatorContext: order-preserving maps/sets via LinkedHashMap/LinkedHashSet with Maps preallocation - AsyncExternalSourceOperatorFactory, FileSourceFactory: wired virtual column injection into the read pipeline - FileSplitProvider, PartitionFilterHintExtractor: switch patterns - Config key strings extracted to PartitionConfig constants Developed using AI-assisted tooling
Collaborator
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
Collaborator
|
Hi @costin, I've created a changelog YAML for you. |
6 tasks
The rebase onto main dropped imports from FileSplitProvider and SourceOperatorContext that were introduced by the stage-2 commit. - FileSplitProvider: restored Expression, comparison type, and BiFunction imports needed by partition filter evaluation - SourceOperatorContext: restored Nullable and ExternalSplit imports, added missing split field to Builder Developed using AI-assisted tooling
Add @SuppressWarnings for the /**/ glob pattern in string literal that checkstyle misinterprets as an empty javadoc comment. Developed using AI-assisted tooling
bpintea
approved these changes
Feb 26, 2026
Contributor
bpintea
left a comment
There was a problem hiding this comment.
LGTM.
It'd be great to have qa tests added too, though.
| return doExpandCommaSeparated(pathList, provider, hints, hivePartitioning, null, null); | ||
| } | ||
|
|
||
| static FileSet expandCommaSeparated( |
| Map<String, Object> partitionValues, | ||
| BlockFactory blockFactory | ||
| ) { | ||
| if (fullOutput == null || fullOutput.isEmpty()) { |
Contributor
There was a problem hiding this comment.
Check.isTrue(Strings.hasText(fulOutput), "...") and .notNull() for the others.
| final class VirtualColumnInjector { | ||
|
|
||
| private final List<Attribute> fullOutput; | ||
| private final Set<String> partitionColumnNames; |
costin
added a commit
to costin/elasticsearch
that referenced
this pull request
Feb 26, 2026
Remove unused 6-arg expandCommaSeparated overload from GlobExpander. In VirtualColumnInjector, replace manual null/empty checks with Check.isTrue and Check.notNull, and demote partitionColumnNames from a field to a constructor-local variable. Developed using AI-assisted tooling
costin
added a commit
to costin/elasticsearch
that referenced
this pull request
Feb 26, 2026
Remove unused 6-arg expandCommaSeparated overload from GlobExpander. In VirtualColumnInjector, replace manual null/empty checks with Check.isTrue and Check.notNull, and demote partitionColumnNames from a field to a constructor-local variable. Developed using AI-assisted tooling
PeteGillinElastic
pushed a commit
to PeteGillinElastic/elasticsearch
that referenced
this pull request
Feb 27, 2026
…143120) Introduce a PartitionDetector SPI with Hive, template, and auto strategies so partition columns are discovered from file paths and injected as virtual columns into the schema and execution pages. PartitionDetector, PartitionConfig, HivePartitionDetector, TemplatePartitionDetector, AutoPartitionDetector: pluggable detection VirtualColumnInjector: injects constant blocks for partition values ExternalSourceResolver: enriches schema with ReferenceAttribute partition columns (not FieldAttribute) preserving path order GlobExpander: template-aware glob rewriting with filter hints PartitionMetadata, FileSplit, SourceOperatorContext: order-preserving maps/sets via LinkedHashMap/LinkedHashSet with Maps preallocation AsyncExternalSourceOperatorFactory, FileSourceFactory: wired virtual column injection into the read pipeline FileSplitProvider, PartitionFilterHintExtractor: switch patterns Config key strings extracted to PartitionConfig constants Relates elastic#142996 Developed using AI-assisted tooling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce a PartitionDetector SPI with Hive, template, and auto
strategies so partition columns are discovered from file paths and
injected as virtual columns into the schema and execution pages.
TemplatePartitionDetector, AutoPartitionDetector: pluggable detection
partition columns (not FieldAttribute) preserving path order
maps/sets via LinkedHashMap/LinkedHashSet with Maps preallocation
column injection into the read pipeline
Relates #142996
Developed using AI-assisted tooling