Skip to content

Add runtime field section to Field Capabilities API#68904

Merged
cbuescher merged 3 commits intoelastic:masterfrom
cbuescher:field-caps-runtimefields
Feb 11, 2021
Merged

Add runtime field section to Field Capabilities API#68904
cbuescher merged 3 commits intoelastic:masterfrom
cbuescher:field-caps-runtimefields

Conversation

@cbuescher
Copy link
Copy Markdown
Member

@cbuescher cbuescher commented Feb 11, 2021

Currently runtime fields from search requests don't appear in the output of the
field capabilities API, but some consumer of runtime fields would like to see
runtime section just like they are defined in search requests reflected and
merged into the field capabilities output.
This change adds parsing of a "runtime_mappings" section equivallent to the one
on search requests to the _field_caps endpoint, passes this section down to
the shard level where any runtime fields defined here overwrite the mapping of
the targetet indices.

Closes #68117

Add runtime field section to Field Capabilities API
841a631 
Currently runtime fields from search requests don't appear in the output of the
field capabilities API, but some consumer of runtime fields would like to see
runtime section just like they are defined in search requests reflected and
merged into the field capabilities output.
This change adds parsing of a "runtime_mappings" section equivallent to the one
on search requests to the `_field_caps` endpoint, passes this section down to
the shard level where any runtime fields defined here overwrite the mapping of
the targetet indices.

Closes elastic#68117
@cbuescher cbuescher added >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types v8.0.0 v7.12.0 labels Feb 11, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Feb 11, 2021
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Feb 11, 2021

Pinging @elastic/es-search (Team:Search)

@cbuescher
Copy link
Copy Markdown
Member Author

cbuescher commented Feb 11, 2021

One caveat of this PR: I found that we do not propagate back exceptions thrown in the shard operation in TransportFieldCapabilitiesIndexAction to the calling nodes in a good way currently. If there are errors in the runtime_mappins section, e.g.

{
  "runtime_mappings": {
    "day_of_week": {
      "type": "some_bad_type",
      "script": {
        "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"
      }
    }
  }
}

The resulting MapperParsingException in that is thrown when parsing the runtime fields upon creating the SearchExecutionContext on the shard is dropped somewhere. Fixing this is a bit more involved so I would prefer doing this in a follow up.

jimczi
jimczi approved these changes Feb 11, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@jimczi jimczi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

javanna
javanna approved these changes Feb 11, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@javanna javanna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me, error handling looks like wasn't part of the initial design of field_caps and it now becomes more important as scripts may not compile etc. Although field_caps should not be used to validate scripts (and possibly we should not even need to compile them for field_caps), returning an empty response whenever there is an error does not seem like a good idea.

I'd say merge this one and let's discuss and work on the error handling as a follow-up?

server/src/test/java/org/elasticsearch/action/fieldcaps/FieldCapabilitiesRequestTests.java
request -> request.indexFilter(request.indexFilter() != null ? request.indexFilter().boost(2) : QueryBuilders.matchAllQuery())
);
mutators.add(request -> request.runtimeFields(Collections.singletonMap("other_key", "other_value")));

Copy link
Copy Markdown
Contributor

@javanna javanna Feb 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make sense to add a test for the toXContent output of the request if we don't test it yet?

Copy link
Copy Markdown
Member Author

@cbuescher cbuescher Feb 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, will do

Christoph Büscher and others added 2 commits February 11, 2021 15:06
@jimczi
Update docs/reference/search/field-caps.asciidoc
89647e6 
Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
add toXContentTest
0d81a87
@cbuescher cbuescher merged commit 3f267ad into elastic:master Feb 11, 2021
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this pull request Feb 11, 2021
Add runtime field section to Field Capabilities API (elastic#68904)
bb097d6 
Currently runtime fields from search requests don't appear in the output of the
field capabilities API, but some consumer of runtime fields would like to see
runtime section just like they are defined in search requests reflected and
merged into the field capabilities output.
This change adds parsing of a "runtime_mappings" section equivallent to the one
on search requests to the `_field_caps` endpoint, passes this section down to
the shard level where any runtime fields defined here overwrite the mapping of
the targetet indices.

Closes elastic#68117
cbuescher pushed a commit that referenced this pull request Feb 11, 2021
Add runtime field section to Field Capabilities API (#68904) (#68915)
96e3428 
Currently runtime fields from search requests don't appear in the output of the
field capabilities API, but some consumer of runtime fields would like to see
runtime section just like they are defined in search requests reflected and
merged into the field capabilities output.
This change adds parsing of a "runtime_mappings" section equivallent to the one
on search requests to the `_field_caps` endpoint, passes this section down to
the shard level where any runtime fields defined here overwrite the mapping of
the targetet indices.

Closes #68117
@przemekwitek przemekwitek mentioned this pull request Feb 15, 2021
Merged
@cbuescher cbuescher mentioned this pull request Feb 15, 2021
Closed
@przemekwitek przemekwitek mentioned this pull request Feb 19, 2021
Merged
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this pull request Mar 1, 2021
Throw IAE for runtime mappings in _field_caps for older nodes
afe9f01 
Typically we silently ignore nodes below a certain version that don't support a
specific feature when serializing requests between nodes. With runtime fields we
chose to throw errors instead when one of the nodes a search request is sent to
has a version that does not support the runtime fields feature. This change adds
the same behaviour for support of the "runtime_mappings" section introduced in elastic#68904
in the "_field_caps" API.

Relates to elastic#68904
@cbuescher cbuescher mentioned this pull request Mar 1, 2021
Merged
cbuescher pushed a commit that referenced this pull request Mar 1, 2021
Throw IAE for runtime mappings in _field_caps for older nodes (#69722)
d6aa939 
Typically we silently ignore nodes below a certain version that don't support a
specific feature when serializing requests between nodes. With runtime fields we
chose to throw errors instead when one of the nodes a search request is sent to
has a version that does not support the runtime fields feature. This change adds
the same behaviour for support of the "runtime_mappings" section introduced in #68904
in the "_field_caps" API.

Relates to #68904
cbuescher pushed a commit that referenced this pull request Mar 1, 2021
Throw IAE for runtime mappings in _field_caps for older nodes (#69722)
ae36a3f 
Typically we silently ignore nodes below a certain version that don't support a
specific feature when serializing requests between nodes. With runtime fields we
chose to throw errors instead when one of the nodes a search request is sent to
has a version that does not support the runtime fields feature. This change adds
the same behaviour for support of the "runtime_mappings" section introduced in #68904
in the "_field_caps" API.

Relates to #68904
cbuescher pushed a commit that referenced this pull request Mar 2, 2021
Throw IAE for runtime mappings in _field_caps for older nodes (#69722)
0a1ede2 
Typically we silently ignore nodes below a certain version that don't support a
specific feature when serializing requests between nodes. With runtime fields we
chose to throw errors instead when one of the nodes a search request is sent to
has a version that does not support the runtime fields feature. This change adds
the same behaviour for support of the "runtime_mappings" section introduced in #68904
in the "_field_caps" API.

Relates to #68904
@przemekwitek przemekwitek mentioned this pull request Mar 3, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jimczi jimczi jimczi approved these changes

+1 more reviewer

@javanna javanna javanna approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

>feature :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Meta label for search team v7.12.0 v8.0.0-alpha1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add a runtime field section to the Field Capabilities API

5 participants

@cbuescher @elasticmachine @javanna @jimczi @jakelandis