Create SPI to allow prohibiting certain top-level mappings#132360
Merged
quux00 merged 20 commits intoelastic:mainfrom Sep 19, 2025
Merged
Create SPI to allow prohibiting certain top-level mappings#132360quux00 merged 20 commits intoelastic:mainfrom
quux00 merged 20 commits intoelastic:mainfrom
Conversation
16994be to
1ddcf5a
Compare
javanna
reviewed
Aug 21, 2025
javanna
reviewed
Aug 21, 2025
server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java
Show resolved
Hide resolved
javanna
reviewed
Aug 21, 2025
server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java
Outdated
Show resolved
Hide resolved
1ddcf5a to
64eddb6
Compare
1. One case with subobjects:false doesn't work as hoped:
// this does NOT return an error - it just silently "fails" to create the _project entry
PUT test333?error_trace=true
{
"mappings": {
"subobjects": false,
"properties" : {
"_project" : {
"type" : "object"
}
}
}
}
// YAY - this one fails with correct error message
PUT test333?error_trace=true
{
"mappings": {
"subobjects": false,
"properties": {
"_project": {
"type": "object",
"properties": {
"myfield": {
"type": "keyword"
}
}
}
}
}
}
2. Creating a runtime mapping is not detected and prevented:
PUT /rt-index
{
"mappings": {
"runtime": {
"_project": {
"type": "keyword",
"script": {
"source": "emit(doc['existing_field'].value + ' some additional text')"
}
}
}
}
}
…ected - basic manual tests now pass
…ssField can call it without a Mapper reference
This prevents _project or _project.foo mappings in query-time runtime mappings.
For example, these now fail with "Mapping rejected: [_project.foo]. No mappings of [_project] are allowed in order to avoid conflicts with project metadata tags in serverless"
GET /blogs/_search
{
"runtime_mappings": {
"_project": {
"type": "keyword",
"script": {
"source": "emit('somevalue')"
}
}
},
"query": {
"match_all": {}
},
"fields": ["_project"]
}
GET /blogs/_search
{
"runtime_mappings": {
"_project.foo": {
"type": "keyword",
"script": {
"source": "emit('somevalue')"
}
}
},
"query": {
"match_all": {}
},
"fields": ["_project.foo"]
}
…d.parseRuntimeFields. Removed checks from RootObjectMapper and IndexService. This checks both query-time and index-time runtime field mappings.
…ime; no query time runtime mapping tests
64eddb6 to
af4964f
Compare
javanna
reviewed
Aug 27, 2025
Contributor
javanna
left a comment
There was a problem hiding this comment.
solid work, I left some minor comments.
server/src/main/java/org/elasticsearch/index/mapper/RootObjectMapper.java
Outdated
Show resolved
Hide resolved
...c/main/java/org/elasticsearch/index/mapper/ServerlessRootObjectMapperNamespaceValidator.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/index/mapper/RootObjectMapperTests.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/elasticsearch/index/mapper/RootObjectMapperTests.java
Show resolved
Hide resolved
Collaborator
|
Pinging @elastic/es-search-foundations (Team:Search Foundations) |
ldematte
reviewed
Sep 1, 2025
server/src/main/java/org/elasticsearch/node/NodeConstruction.java
Outdated
Show resolved
Hide resolved
…it-_project-mappings
…it-_project-mappings
…it-_project-mappings
szybia
added a commit
to szybia/elasticsearch
that referenced
this pull request
Sep 19, 2025
* upstream/main: Turn NumericValues into functional interface (elastic#135068) Improve block loader for source only runtime fields of type keyword (elastic#135026) Mute org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT test {csv-spec:stats.StdDeviationGroupedAllTypes} elastic#135103 Mute org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT test {csv-spec:stats.StdDeviationWithLongs} elastic#135102 Mute org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT test {csv-spec:inlinestats.StdDevFilter} elastic#135101 Mute org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT test {csv-spec:stats.StdDevFilter} elastic#135100 Remove track_live_docs_in_memory_bytes feature flag (elastic#134900) Create SPI to allow prohibiting certain top-level mappings (elastic#132360) Only validate primary ids on release branches (elastic#135044) Added no-op support for project_routing query param to REST endpoints that will support cross-project search (elastic#134741) Fix race in FileSettingsServiceIT.testSettingsAppliedOnStart (elastic#134368)
elasticsearchmachine
pushed a commit
that referenced
this pull request
Sep 22, 2025
gmjehovich
pushed a commit
to gmjehovich/elasticsearch
that referenced
this pull request
Sep 22, 2025
`subobjects:auto` is guarded behind a feature flag, not enabled. elastic#132360 added tests without proper guarding. Fixes elastic#135134
DonalEvans
pushed a commit
to DonalEvans/elasticsearch
that referenced
this pull request
Sep 22, 2025
`subobjects:auto` is guarded behind a feature flag, not enabled. elastic#132360 added tests without proper guarding. Fixes elastic#135134
breskeby
pushed a commit
to breskeby/elasticsearch
that referenced
this pull request
Feb 11, 2026
… serverless (elastic#4596) In order to prepare for cross-project search where project tags (defined in cloud) can be used in search and ES|QL queries, we need to prohibit any Lucene or Elasticsearch runtime fields that start with _project, as that will be the prefix by which we identify project metadata tags in queries (stored in cluster state rather than a Lucene index). This PR leverages the RootObjectNamespaceValidator framework added to elasticsearch core repo in elastic#132360 by creating a `ServerlessRootObjectMapperNamespaceValidator` that prohibits any top level field (or "namespace") starting with `_project` both in standard index mappings and in runtime mappings.
breskeby
pushed a commit
to breskeby/elasticsearch
that referenced
this pull request
Feb 11, 2026
… serverless (elastic#4596) In order to prepare for cross-project search where project tags (defined in cloud) can be used in search and ES|QL queries, we need to prohibit any Lucene or Elasticsearch runtime fields that start with _project, as that will be the prefix by which we identify project metadata tags in queries (stored in cluster state rather than a Lucene index). This PR leverages the RootObjectNamespaceValidator framework added to elasticsearch core repo in elastic#132360 by creating a `ServerlessRootObjectMapperNamespaceValidator` that prohibits any top level field (or "namespace") starting with `_project` both in standard index mappings and in runtime mappings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The RootObjectNamespaceValidator allows prohibiting "namespaces" or top level fields in
Elasticsearch mappings: both "normal" index mappings and runtime mappings.
The same validator is used to check both index-time runtime mappings and query-time index mappings.
The primary target use case for this SPI is a feature in serverless to prohibit mappings like
_projector_project.my_sub_field, but the mechanism provided in this PR allows anypossible namespace detection and prevention.