I ran a benchmark that indexes apache logs into a single shard index. The ingest overhead is based on comparing the recently added ingest_took in the bulk response with the time it took for the bulk API to answer.

I ran several iterators. Indexing 300k, 600k and 1,2M document. For each volume I ran with different pipeles:

empty - A pipeline with no processors.
set - A pipeline that adds the current ingest time as a field to the document being processed.
grok - A pipeline that split up the apache log into several fields.
grok_and_geoip - Same as grok, but also adds geo information based on the client ip.

So in total I ran 12 iterations:

# docs	pipeline	ingest time overhead
300k	empty	12%
300k	set	21%
300k	grok	27%
300k	grok_and_geoip	37%
600k	empty	13%
600k	set	21%
600k	grok	27%
600k	grok_and_geoip	37%
1.2M	empty	13%
1.2M	set	21%
1.2M	grok	26%
1.2M	grok_and_geoip	37%

Each pipeline has a static tax on top of the indexing time and the time spent increases roughly linearly as more documents are being ingested. While ingesting these apache log with ingest, the used heap space overhead wasn't really noticeable.

The heap memory overhead of ingest is more tied to the size of the documents being pre-processed than to the amount of document being processed in a single bulk request. This is because ingest turns each document into a map of maps and because per bulk request, ingest processes a single document at the time. So the ingest heap overhead becomes noticeable when preprocessing larger documents. This is why the used heap overhead wasn't noticeable when benchmarking with logs. However when testing with documents that have ~400 fields, I have seen the used heap overhead being 20% to 30% higher then when indexing the same data without ingest enabled.

Benchmark the ingest plugin #14425

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions