[BUG] Exporting a big file (>50MB) produces a V8 memory heap error #34
Description
Describe the bug
When a report is generated containing a lot of document then the Javascript v8 engine might suffers from a heap out of memory exception and crash.
I believe that it is caused by the fact that the export process keeps all content that should be exported in memory before writing all at once into the CSV file.
To Reproduce
Steps to reproduce the behavior:
- Go to 'Discover'
- Select a lot of documents and do not specify the columns that should be exported
- Click on 'Generate CSV'
- The dashboard application should crash
Expected behavior
Their should be almost no limit to the size of the exported CSV file and it should be possible to generate CSV file way bigger than the memory allocated to the Javascript heap.
Example of a trace
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
<--- Last few GCs --->
[1:0x41f1b90] 1220660 ms: Mark-sweep 786.3 (1233.8) -> 786.3 (1157.3) MB, 218.8 / 0.0 ms (average mu = 0.853, current mu = 0.000) last resort GC in old space requested
[1:0x41f1b90] 1221123 ms: Mark-sweep 786.3 (1157.3) -> 786.3 (1116.8) MB, 462.8 / 0.0 ms (average mu = 0.630, current mu = 0.000) last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x16ab3e6dbf1d]
Security context: 0x3a9704a9e6c1 <JSObject>
1: DoJoin(aka DoJoin) [0x3a9704a85e69] [native array.js:~87] [pc=0x16ab3eace969](this=0x07a6e6d826f1 <undefined>,l=0x0a2acd302271 <JSArray[10000]>,m=10000,A=0x07a6e6d828c9 <true>,w=0x0a66bb934901 <String[1]\: \n>,v=0x07a6e6d829a1 <false>)
2: Join(aka Join) [0x3a9704a85eb9] [native array.js:~112] [pc=0x16ab3eaaa4b8](this=0x07a6e6d826f1 <undefined>,l=0...
1: 0x8fb090 node::Abort() [/usr/share/kibana/bin/../node/bin/node]
2: 0x8fb0dc [/usr/share/kibana/bin/../node/bin/node]
3: 0xb0336e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
4: 0xb035a4 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
5: 0xef7602 [/usr/share/kibana/bin/../node/bin/node]
6: 0xf06e1f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/share/kibana/bin/../node/bin/node]
7: 0xed69cb v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
8: 0x11c5768 v8::internal::Runtime_StringBuilderJoin(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/share/kibana/bin/../node/bin/node]
9: 0x16ab3e6dbf1d
Host/Environment (please complete the following information):
- OS: Ubuntu
- Version: 20.10
- Docker version : 19.03.23
- Docker image : amazon/opendistro-for-elasticsearch-kibana:1.13.2