datastation icon indicating copy to clipboard operation
datastation copied to clipboard

Support CSV parallel encoding

Open dmtri35 opened this issue 3 years ago • 0 comments

  • Add a toggle in the UI to enable.
  • Add check if file is large enough to encode in parallel.
  • Each record will be passed into a channel where they will be consumed by a couple of goroutines.
  • Typically encode 3 times faster than normal for large files.

*Maybe we could do a generic goroutine and apply it to all the other file types and databases. This only involves changing the channel type.

dmtri35 avatar Apr 26 '22 13:04 dmtri35