Skip to content

[Ingest Manager] Updating a package immediately after a rollover fails #79370

@jonathan-buttner

Description

@jonathan-buttner

Updating a package immediately after performing a rollover on a data stream fails.

Steps to reproduce:

  1. Boot up a fresh kibana and elasticsearch
  2. Force install an older endpoint package: POST http://elastic:changeme@localhost:5601/api/ingest_manager/epm/packages/endpoint-0.16.0-dev.3?force=true
  3. Ingest some data that matches a data stream of the endpoint package

https://github.com/elastic/kibana/blob/9d07a677b89a723db55a3f307bfb09c7ae189de5/x-pack/test/functional/es_archives/endpoint/pipeline/dns/data.json.gz

node scripts/es_archiver.js --es-url http://elastic:changeme@localhost:9200 --kibana-url http://elastic:changeme@localhost:5601 load x-pack/test/functional/es_archives/endpoint/pipeline/dns --use-create

The above archive will populate data in the logs-endpoint.events.network-default data stream.

  1. Perform a manual rollover POST /logs-endpoint.events.network-default/_rollover
  2. Observe that the rolled over backing index does not have the value fields associated with the data_stream object:

image

  1. Navigate to the Ingest manager app to perform an upgrade or do
POST http://elastic:changeme@localhost:5601/api/ingest_manager/epm/packages/_bulk
{
  packages: ['endpoint']
}
  1. Observe the failure:

image

This is thrown here:

throw new Error(`data_stream values are missing from the index template ${indexName}`);

  1. Ingest more data into the data stream (can probably use the same archive from step 3).
  2. Observe that the backing index has the value fields populated under data_stream

image

  1. Navigate back to the ingest manager and you will likely see another error

image

Or after some time this:

image

Metadata

Metadata

Assignees

Labels

Team:Defend Workflows“EDR Workflows” sub-team of Security SolutionTeam:FleetTeam label for Observability Data Collection Fleet teambugFixes for quality problems that affect the customer experienceimpact:highAddressing this issue will have a high level of impact on the quality/strength of our product.v7.10.0

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions