Metricbeat fails to publish elasticsearch metricbeat documents due to `require_alias` error

We observed in this deployment from @jasonrhodes https://admin.found.no/deployments/1df5d5b91b5a49fd9f9fa6fc2e93b51a that metricbeat was failing to publish ES monitoring docs with the following error:

```
Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.January, 27, 0, 22, 0, 50458769, time.Local), Meta:{"index":".monitoring-es-7-mb"}, Fields:{"agent":{"ephemeral_id":"9f635268-bace-41b6-a2bd-6f3eab0f19ab","hostname":"5d886d0ad862","id":"649e5cfd-66dc-4ab0-88da-12a66a65f81e","name":"5d886d0ad862","type":"metricbeat","version":"7.17.0"},"cluster_uuid":"1G-2qN3BS6-UcYX8I9DHpw","ecs":{"version":"1.12.0"},"enrich_coordinator_stats":{"executed_searches_total":0,"node_id":"MIcZUsjeSqC90MU5trc2wQ","queue_size":0,"remote_requests_current":0,"remote_requests_total":0},"event":{"dataset":"elasticsearch.enrich","duration":6114284,"module":"elasticsearch"},"host":{"name":"5d886d0ad862"},"interval_ms":10000,"metricset":{"name":"enrich","period":10000},"service":{"address":"https://5d886d0ad862:18669","type":"elasticsearch"},"timestamp":"2022-01-27T00:22:00.056Z","type":"enrich_coordinator_stats"}, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=404): {"type":"index_not_found_exception","reason":"no such index [.monitoring-es-7-mb-2022.01.27] and [require_alias] request flag is [true] and [.monitoring-es-7-mb-2022.01.27] is not an alias","index_uuid":"_na_","index":".monitoring-es-7-mb-2022.01.27"}, dropping event!
```

This is unexpected since the `.monitoring-es-7-*` indices are not expected to have aliases.

@lucasmoore provided the metricbeat config for the master instance of the deployment, which contain no reference to alias requirements.

<details>
<summary>metricbeat.yml</summary>

```
output.elasticsearch:
  hosts: ["http://containerhost:9244"]
  username: "elastic-observability-agent"
  password: "redacted"
  headers:
    X-Found-Cluster: d733939974c6401d95c05b2696664ac1
    X-Elastic-App-Auth: redacted


  ssl.verification_mode: full



setup.template.overwrite: false

metricbeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 30s

logging.level: info
logging.to_files: true
logging.json: true
logging.files:
  path: /app/logs/beats
  name: metricbeat.log
  keepfiles: 3



queue.disk:
  max_size: "1GB"
  read_ahead: 512
  write_ahead: 2048
```
</details>

<details>
<summary>modules.d/elasticsearch.yml</summary>

```
- module: elasticsearch
  metricsets:
    - ccr
    - enrich
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["https://${HOSTNAME:REDACTED}:REDACTED"]

  username: "ec-local-beats-monitor"


  password: "redacted"

  headers:


  ssl.verification_mode: none
  xpack.enabled: true
```
</details>

I found https://github.com/elastic/beats/pull/29879 which involves require_alias and was recently merged to 7.17. So at the moment I suspect it's what's causing the incorrect expectation, but I haven't gone over the PR in detail yet.

cc @kvch who opened the PR and may have additional context

Rel: https://github.com/elastic/beats/issues/29920 (probably same issue, but that issue seemed ECK focused so I didn't see the connection until I'd already opened this one)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metricbeat fails to publish elasticsearch metricbeat documents due to `require_alias` error #30044

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Metricbeat fails to publish elasticsearch metricbeat documents due to require_alias error #30044

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Metricbeat fails to publish elasticsearch metricbeat documents due to `require_alias` error #30044