datastation
datastation copied to clipboard
Support CSV parallel encoding
- Add a toggle in the UI to enable.
- Add check if file is large enough to encode in parallel.
- Each record will be passed into a channel where they will be consumed by a couple of goroutines.
- Typically encode 3 times faster than normal for large files.
*Maybe we could do a generic goroutine and apply it to all the other file types and databases. This only involves changing the channel type.