Elasticsearch Version

9.2.4

Installed Plugins

No response

Java Version

bundled

OS Version

n/a

Problem Description

Summary

I encountered an issue where dynamic templates are being overridden by the automatic dense_vector mapping feature. When I index a float array with 128-4096 elements on a nested field, my dynamic template is completely ignored, and the field gets converted to dense_vector instead.

Affects: Elasticsearch 8.11.0+ (tested on 9.2.4)

What I Found

After digging into the code, I found there's a field name mismatch in the tracking logic:

• When a dynamic template is applied (DynamicFieldsBuilder.java:210), the field is marked using the short name (e.g., "embedding")

  • elasticsearch/server/src/main/java/org/elasticsearch/index/mapper/DynamicFieldsBuilder.java

    Line 210 in cc5852b

    context.markFieldAsAppliedFromTemplate(name);

• Later, when checking if a template was applied (DocumentParser.java:823 in postProcessDynamicArrayMapping), it checks using the full path (e.g., "text_embedding.embedding")

  • elasticsearch/server/src/main/java/org/elasticsearch/index/mapper/DocumentParser.java

    Line 823 in cc5852b

    || context.isFieldAppliedFromTemplate(fullFieldName)

• Since "embedding" != "text_embedding.embedding", the check fails and the automatic dense_vector conversion kicks in

The check context.isFieldAppliedFromTemplate(fullFieldName) is there to prevent this exact scenario, but it doesn't work due to the name mismatch.

It was introduced in PR #98512 ("Automatically map floats as dense vector"). If you read the PR along with this, it would be easier to track.

Additional Notes

• Using match instead of path_match has the same issue for nested fields
• Changing template type to integer works (template is respected), so I can confirm that the template matching itself is working
• Top-level fields with simple match patterns work fine

Related

This is an issue from elastic forum report https://discuss.elastic.co/t/dynamic-template-mapping-overridden-by-automatic-dense-vector-inference-for-float-arrays/385359

Steps to Reproduce

Reproduction

Here's a minimal example:

1. Create index with dynamic template:

PUT test_index
{
  "mappings": {
    "dynamic_templates": [
      {
        "embedding_as_float": {
          "path_match": "*.embedding",
          "mapping": {
            "type": "float",
            "index": false
          }
        }
      }
    ]
  }
}

2. Index a document with 150-element float array:

POST test_index/_doc
{
  "text_embedding": {
    "embedding": [
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
      0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
    ]
  }
}

3. Check the mapping:

GET test_index/_mapping
>>>output
{
  "test_index": {
    "mappings": {
      "dynamic_templates": [
        {
          "embedding_as_float": {
            "path_match": "*.embedding",
            "mapping": {
              "index": false,
              "type": "float"
            }
          }
        }
      ],
      "properties": {
        "text_embedding": {
          "properties": {
            "embedding": {
              "type": "dense_vector",
              "dims": 150,
              "index": true,
              "similarity": "cosine",
              "index_options": {
                "type": "int8_hnsw",
                "m": 16,
                "ef_construction": 100
              }
            }
          }
        }
      }
    }
  }
}

Expected vs Actual

Expected: text_embedding.embedding should be float with index: false per my template.

Actual: The field becomes dense_vector with dims: 150, index: true, completely ignoring my template.

Logs (if relevant)

No response