Allow specify dynamic templates in bulk request by dnhatn · Pull Request #69948 · elastic/elasticsearch

dnhatn · 2021-03-04T02:19:53Z

This change allows users to specify dynamic templates in a bulk request.

PUT myindex
{
  "mappings": {
    "dynamic_templates": [{
      "time_histograms": {
        "mapping": {
          "type": "histogram",
          "meta": {
            "unit": "s"
          }
        }
      }
    }]
  }
}

// Tells ES to use the dynamic template named `time_histograms` for field `response_times`
// in the index request
POST myindex/_bulk
{ "index": { "dynamic_templates": { "response_times": "time_histograms" } } }
{ "@timestamp": "2020-08-12", "response_times": { "values": [1, 10], "counts": [5, 1] }}

Closes #61939

elasticmachine · 2021-03-04T17:09:03Z

Pinging @elastic/es-search (Team:Search)

javanna

I left some questions. The approach looks good code-wise overall, my comments are mostly about the API.

Another one: does the matched "type" become the content of the dynamic_type variable? Or should it even? I am wondering myself if this should be considered and promoted as a type. To me it is more of an arbitrary name that defined dynamic templates can match against. Should we rather call it match_mapping_hint?

rest-api-spec/src/main/resources/rest-api-spec/test/bulk/11_type_hints.yml

server/src/main/java/org/elasticsearch/index/mapper/DynamicTemplate.java

server/src/main/java/org/elasticsearch/index/termvectors/TermVectorsService.java

dnhatn · 2021-03-09T21:26:21Z

@javanna Thank you for the review. I tweaked the API using "match_mapping_hint" and I found it's simpler and less confusing. I have updated the PR. Can you take another look?

axw · 2021-03-10T00:16:59Z

@dnhatn apologies if this is obvious: will this be immediately usable through an Ingest script processor (and if so how?), or would that come later?

dnhatn · 2021-03-10T02:18:48Z

can users use one of these types as hint in their bulk requests, in which case they won't have to do define anything on the dynamic template side of things? Or should hints always be arbitrary and explicitly defined in the dynamic template?

That's a good question. Let's discuss it after this iteration.

jimczi

The change looks great @dnhatn ! I left some comments but I think the feature is ready.

server/src/main/java/org/elasticsearch/action/index/IndexRequest.java

server/src/main/java/org/elasticsearch/index/mapper/ParseContext.java

server/src/main/java/org/elasticsearch/index/mapper/SourceToParse.java

rest-api-spec/src/main/resources/rest-api-spec/test/bulk/11_type_hints.yml

javanna

I left some more comments, nothing major though. I like how simple the implementation, thanks for iterating on it. Would it make sense to add docs as part of this PR or do you plan to do it as a followup?

rest-api-spec/src/main/resources/rest-api-spec/test/bulk/11_type_hints.yml

server/src/main/java/org/elasticsearch/action/index/IndexRequest.java

server/src/main/java/org/elasticsearch/index/mapper/ParseContext.java

server/src/main/java/org/elasticsearch/index/mapper/SourceToParse.java

server/src/test/java/org/elasticsearch/index/mapper/DocumentParserTests.java

server/src/test/java/org/elasticsearch/index/mapper/DynamicTemplateTests.java

server/src/main/java/org/elasticsearch/action/index/IndexRequest.java

server/src/main/java/org/elasticsearch/action/bulk/BulkRequestParser.java

server/src/internalClusterTest/java/org/elasticsearch/index/mapper/DynamicMappingIT.java

Co-authored-by: Adrien Grand <jpountz@gmail.com>

javanna

I left some minor comments, LGTM otherwise. Thanks for all the iterations.

docs/reference/docs/bulk.asciidoc

docs/reference/rest-api/common-parms.asciidoc

server/src/internalClusterTest/java/org/elasticsearch/index/mapper/DynamicMappingIT.java

dnhatn · 2021-04-08T16:43:02Z

Thanks everyone :).

Relates elastic#69948

Relates #69948

This change exposes the newly introduced parameter `dynamic_templates` in ingest. This parameter can be set by a set processor or a script processor. Relates #69948

This change exposes the newly introduced parameter dynamic_templates in ingest. This parameter can be set by a set processor or a script processor. Relates #69948

dnhatn force-pushed the dynamic-mapping-hint branch from 07aa9d1 to 6e80821 Compare March 4, 2021 16:31

dnhatn marked this pull request as ready for review March 4, 2021 17:08

dnhatn requested review from jimczi and jpountz March 4, 2021 17:08

dnhatn added the :Search Foundations/Mapping Index mappings, including merging and defining field types label Mar 4, 2021

elasticmachine added the Team:Search Meta label for search team label Mar 4, 2021

dnhatn added >enhancement and removed Team:Search Meta label for search team labels Mar 4, 2021

dnhatn added v8.0.0 v7.13.0 Team:Search Meta label for search team labels Mar 4, 2021

Add dynamic mapping type hint to bulk request

40e6a57

dnhatn force-pushed the dynamic-mapping-hint branch from e346e5a to 40e6a57 Compare March 4, 2021 20:09

jimczi requested review from javanna and removed request for jpountz March 5, 2021 09:28

javanna reviewed Mar 8, 2021

View reviewed changes

dnhatn added 6 commits March 8, 2021 10:25

Merge branch 'master' into dynamic-mapping-hint

eed16ef

Remove TODO

27a538b

Merge branch 'master' into dynamic-mapping-hint

6ee5210

match_mapping_hint

69a3849

fix doc

02b2e89

fix test

96795af

dnhatn requested a review from javanna March 9, 2021 21:26

jimczi approved these changes Mar 10, 2021

View reviewed changes

javanna suggested changes Mar 10, 2021

View reviewed changes

Merge branch 'master' into dynamic-mapping-hint

8f2c6c4

dnhatn and others added 2 commits April 6, 2021 22:39

Update docs/reference/docs/bulk.asciidoc

bf94a83

Co-authored-by: Adrien Grand <jpountz@gmail.com>

Allow new dynamic template format with old indices

60d11a4

javanna approved these changes Apr 8, 2021

View reviewed changes

docs/reference/docs/bulk.asciidoc Outdated Show resolved Hide resolved

docs/reference/rest-api/common-parms.asciidoc Show resolved Hide resolved

server/src/internalClusterTest/java/org/elasticsearch/index/mapper/DynamicMappingIT.java Show resolved Hide resolved

fix mapping syntax

d0010bd

dnhatn merged commit 5c99692 into elastic:master Apr 8, 2021

dnhatn deleted the dynamic-mapping-hint branch April 8, 2021 16:44

dnhatn added the backport pending label Apr 8, 2021

ebeahan mentioned this pull request Apr 8, 2021

[RFC] Label fields for additional types - Stage 0 elastic/ecs#1341

Closed

2 tasks

dnhatn mentioned this pull request Apr 9, 2021

Allow specify dynamic templates in bulk request #71518

Merged

dnhatn added a commit to dnhatn/elasticsearch that referenced this pull request Apr 10, 2021

Disable BWC tests

a9ec3ea

Relates elastic#69948

dnhatn mentioned this pull request Apr 10, 2021

Disable BWC tests #71536

Merged

dnhatn added a commit that referenced this pull request Apr 10, 2021

Disable BWC tests (#71536)

ee69f19

Relates #69948

dnhatn mentioned this pull request Apr 10, 2021

Adjust BWC version for dynamic templates in bulk requests #71537

Merged

dnhatn added a commit that referenced this pull request Apr 10, 2021

Adjust BWC version for dynamic templates in bulk requests (#71537)

8bef36d

Relates #69948

dnhatn removed the backport pending label Apr 10, 2021

dnhatn mentioned this pull request Apr 15, 2021

Expose dynamic_templates parameter in Ingest #71716

Merged

dnhatn mentioned this pull request Apr 19, 2021

Expose dynamic_templates parameter in Ingest #71856

Merged

dnhatn added a commit that referenced this pull request Apr 19, 2021

Expose dynamic_templates parameter in Ingest (#71716)

d6f176e

This change exposes the newly introduced parameter dynamic_templates in ingest. This parameter can be set by a set processor or a script processor. Relates #69948

stevejgordon mentioned this pull request Apr 21, 2021

7.13.0 Meta Ticket elastic/elasticsearch-net#5584

Closed

62 tasks

exekias mentioned this pull request Apr 30, 2021

Create a default mapping template for metrics #72536

Closed

tvernum mentioned this pull request May 4, 2021

[DOCS] Link from dynamic templates to bulk API is malformed #72659

Closed

axw mentioned this pull request May 5, 2021

[Change Proposal] Add support for explicitly defining dynamic_templates elastic/package-spec#171

Closed

2 tasks

jsoriano mentioned this pull request Jun 24, 2021

[Discuss] Avoiding duplication of ECS field definitions elastic/package-spec#63

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

axw mentioned this pull request Oct 18, 2021

Dotted field names that conflict with objects #63530

Closed

axw mentioned this pull request Feb 21, 2023

Dynamic mode to allow only dynamic mapping of fields with explicitly named template #93951

Closed

felixbarny mentioned this pull request Oct 22, 2025

OTLP: store units in mappings #134709

Merged

Conversation

dnhatn commented Mar 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Mar 4, 2021

Uh oh!

javanna left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dnhatn commented Mar 9, 2021

Uh oh!

axw commented Mar 10, 2021

Uh oh!

dnhatn commented Mar 10, 2021

Uh oh!

jimczi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

javanna left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

javanna left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dnhatn commented Apr 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dnhatn commented Mar 4, 2021 •

edited

Loading