Skip to content

file adapter runs out of memory on a very large write #6

@rlgomes

Description

@rlgomes

Here's a simple scenario that leads to the OOM:

./bin/juttle -e "read stochastic -source 'logs' -from :2014-01-01: -nhosts 100 -lpm 1000 | write file -file 'test.json'"

From a quick look at the code its waiting to accumulate all of the data points before writing them out to the underlying file instead of writing out each point or batches of points in order to avoid running out of memory.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions