@dmehra @demmer @dmajda @mstemm fixing the situation identified by issue #6 and I'll update the documentation here in this PR once the initial review for the change goes through.

I'm thinking that if we set the limit for the only existing format for write to be 100K right now then what we can do in a follow up PR is to add the ability to write file to specify an output format such as csv and jsonlines which would allow for incremental writing and never run out of memory.

@demmer initial approach of using the scheduler, doing this on a 1s interval seems a little "harsh" so I'm open to better suggestion but if we do things with a larger interval than the program would have to wait up to N additional seconds after finishing to actually complete and have the data flushed.

@rlgomes General note — in adapter-related PRs please tag @bkutil instead of (or in addition to) me. He is working more in the area so he should be able to give better feedback. Thanks.

the fix for issue #6 is currently to set a write limit so that we
simply limit writing out no more than 100K

Nice! So no more OOMs? IIUC, we still read & parse the whole file...

future work could be to include streamable writing formats such as CSV,
or JSONL (jsonlines: http://jsonlines.org/)

+10k - I agree with you that this is the way to go.

@bkutil taking care of the various suggestions all good catches. The issue with OOM'ing because we'd have an existing 1GB JSON file I'm not sure how we'd fix that at this point. I think another simple limitation would be that we'd fail to open files biger than 200MB ? impose some hard limit on the size of the file ? @demmer @dmajda what do you think of this ? I guess we'd also have to expose some way of overriding the limitation but at least this way we could avoid blowing things up in a nasty way.

As far as the limit on the size of the file, I guess you'll need to have two options: -limit which is a number of points and -maxSize in bytes. Seems icky but I don't see a better option.

But I'll document that maxSize is the file size while -limit is the number of points in said file.

@demmer @bkutil have a looksie I think this is much better at this point.

@demmer @bkutil another look at the current solution. I opted for not having a -flushInterval and instead doing a -flushFrequency (open to suggestions on a better name) where we only flush when we've hit a target amount of points and not just periodically checking if we have more things to flush or not. The -flushInterval option was leading to some absurdly complicated code due to the background callbacks having to synchronize their work with the final flush from the eof call.

Things to finish up:

• add tests for option validations that aren't covered and showing up as uncovered lines in the coverage report.

Overall this looks good to me now since the above comments are just a few minor suggestions.

Ok merging on green as I've taken care of all @demmer 's suggestions and I think this is ready for prime time.

@dmehra quick review on documentation additions: https://github.com/juttle/juttle/pull/11/files#diff-8cca8547a0a58a3918b9eb089cdda8a6

Doc review comment: how about a rephrase so defaults go into the "Required?" column, and don't mention -file because per #44 it will be renamed to -path anyway. Maybe like this:

`-bufferLimit`    | Maximum number of points to buffer in memory before each write to the specified file | No; defaults to 100000
`-maxFilesize`    | Maximum size of the file being written to, limited since the entire JSON needs to fit in memory | No; defaults to 200MB
`-flushFrequency` | How often (in number of points) to flush the points in memory out to the specified file | No; defaults to every 100 points

@dmehra fixed!