ESQL: CSV schema inference and parsing enhancements#144050
Merged
costin merged 2 commits intoelastic:mainfrom Mar 12, 2026
Merged
ESQL: CSV schema inference and parsing enhancements#144050costin merged 2 commits intoelastic:mainfrom
costin merged 2 commits intoelastic:mainfrom
Conversation
Collaborator
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
Collaborator
|
Hi @costin, I've created a changelog YAML for you. |
fcfdeab to
ad4f9a1
Compare
bpintea
approved these changes
Mar 11, 2026
|
|
||
| private static boolean hasTypeAnnotations(String[] columns) { | ||
| for (String column : columns) { | ||
| if (column.trim().contains(":")) { |
Contributor
There was a problem hiding this comment.
How about considering the escaping? I imagine column names could contain it otherwise, it's not reserved by the RFC.
| } | ||
| } | ||
|
|
||
| private List<Attribute> inferSchemaFromBatchReader(String headerLine) throws IOException { |
Contributor
There was a problem hiding this comment.
Might be worth sharing some code with inferSchemaFromSample.
487ea3d to
7fdcb73
Compare
Plain CSV headers without type annotations now trigger automatic schema inference instead of failing, unblocking all standard CSV files. Also fixes boolean case-sensitivity, datetime format flexibility, and numeric type alias recognition.
7fdcb73 to
4ffcdd4
Compare
Member
Author
|
Addressed both review comments:
|
szybia
added a commit
to szybia/elasticsearch
that referenced
this pull request
Mar 12, 2026
…elocations * upstream/main: (49 commits) CCS logging fixes (elastic#144070) Improve CPS cluster exclusion handling (elastic#143488) Remove snapshot condition now that node_reduce phase is in non-snapshot builds (elastic#144090) Drop deprecation warnings when updating a mapping in the cluster state applier (elastic#143884) (elastic#144040) Add ensureGreenAndNoInitializingShards helper (elastic#144044) Removed unnecessary applies_to blocks from deprecated query (elastic#144096) [CPS] Use single CrossProjectModeDecider instance (elastic#144030) Fix ESQL TS requests with LIMIT 0 (elastic#144031) ESQL: Remove `create` methods in aggs (elastic#144098) ES|QL: Refactor ChangeLimitOperator (elastic#144017) Add Paginated Hit Source Tests (elastic#142592) Fix test failure not preferred (elastic#144019) Remove serialization logic from EIS authorization response (elastic#144021) ESQL: CSV schema inference and parsing enhancements (elastic#144050) ESQL: Fix incorrectly optimized fork with nullify unmapped_fields (elastic#143030) Fix MMR release test using subqueries (elastic#144087) Refactoring `UserAgentPlugin` (elastic#140712) Drop non-finite samples in Prometheus remote write (elastic#144055) [TEST] Wait for internal inference indices to be created in authorization IT (elastic#143885) Disable ndjson datasource QA tests in release-tests (elastic#143992) ...
michalborek
pushed a commit
to michalborek/elasticsearch
that referenced
this pull request
Mar 23, 2026
Standard CSV files use plain column names without type annotations. Before this change, such headers caused a parsing failure, blocking all normal CSV files from being read. This adds sample-based schema inference for plain headers and fixes boolean case-sensitivity, datetime format flexibility, and numeric type alias recognition. Developed with AI-assisted tooling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Standard CSV files use plain column names without type annotations. Before this change, such headers caused a parsing failure, blocking all normal CSV files from being read.
This adds sample-based schema inference for plain headers and fixes boolean case-sensitivity, datetime format flexibility, and numeric type alias recognition.
Developed with AI-assisted tooling