This PR adds Vector Builder implementations for each DataType, as well as high-level stream primitives for Iterables/AsyncIterables, node streams, and DOM streams.

edit: I've created a demo that transforms a CSV file/stream of JSON rows to an Arrow table in this repository: https://github.com/trxcllnt/csv-to-arrow-js

Builder API

The new Builder class exposes an API for sequentially appending (or setting into slots that have already been allocated) arbitrary JavaScript values that will be flushed to the same underlying Data chunk. The Builder class also supports specifying a list of null-value sentinels, or values that will be interpreted to indicate "null" should be written to the null bitmap instead of being written as a valid element.

Similar to the existing Vector API, Builder has a static Builder.new() method that will return the correct Builder subclass instance for the supplied DataType. Since the Builder constructor takes an options Object, this method also takes an Object:

import { Builder, Utf8 } from 'apache-arrow';

const utf8Builder = Builder.new({
    type: new Utf8(),
    nullValues: [null, 'n/a']
});

utf8Builder
    .append('hello')
    .append('n/a') // will be interpreted to mean `null`
    .append('world')
    .append(null);

const utf8Vector = utf8Builder.finish().toVector();

console.log(utf8Vector.toJSON());
// > ["hello", null, "world", null]

The Builder class has two methods for flushing the pending values to their underlying ArrayBuffer representations: flush(): Data<T> and toVector(): Vector<T> (toVector() calls flush() and creates a Vector instance from the returned Data instance).

Calling Builder.prototype.finish() will finalize a Builder instance. After this, no more values should be written to the Builder instance. This is a no-op on for most types, except the DictionaryBuilder, which flushes its internal dictionary and writes the values to the Dictionary type's dictionaryVector field.

Iterable and stream APIs

Creating and using Builders directly is a bit cumbersome, so we provide some high-level streaming APIs for automatically creating builders, appending values, and flushing chunks of a certain size:

Builder.throughIterable(options: IterableBuilderOptions<T, TNull>)
Builder.throughAsyncIterable(options: IterableBuilderOptions<T, TNull>)
Builder.throughDOM(options: BuilderTransformOptions<T, TNull>)
Builder.throughNode(options: BuilderDuplexOptions<T, TNull>)

Iterables and AsyncIterables

The static throughIterable and throughAsyncIterable methods take an options argument that indicates the Builder's type and null-value sentinels, and returns a function which accepts an Iterable or AsyncIterable, respectively, of values to transform:

import { Chunked, Builder, Utf8 } from 'apache-arrow';
const options = { type: new Utf8(), nullValues: [null, 'n/a'] };
const buildUtf8 = Builder.throughIterable(options);
const utf8Vector = Chunked.concat(...buildUtf8(['hello', 'n/a', 'world', null]));

The options argument can also specify a queueingStrategy and highWaterMark that control the chunking semantics:

• If the queueingStrategy is "count" (or is omitted), then the returned generator function will flush the Builder and yield a chunk once the number of values that have been written to the Builder reaches the value supplied for highWaterMark, regardless of how much memory the Builder has allocated.
• If the queueingStrategy is "bytes", then the returned generator function will flush the Builder and yield a new chunk once the Builder's byteLength field reaches or exceeds the value supplied for highWaterMark, regardless of how many elements the Builder contains.

Node and DOM Streams

In addition to the Iterable transform APIs, we can also create node and DOM transform streams with similar options:

import { Readable } from 'stream';
import { toArray } from 'ix/asynciterable/toarray';
import { Chunked, Builder, Utf8 } from 'apache-arrow';
const options = {
  type: new Utf8(),
  nullValues: [undefined, 'n/a'],
  queueingStrategy: 'bytes',
  highWaterMark: 64, // flush each chunk once 64 bytes have been written
};

const utf8Vector = Chunked.concat(await toArray(
  Readable
    .from(['hello', 'n/a', 'world', undefined])
    .pipe(Builder.throughNode(options))
));

Miscellaneous

• Updates most dependencies, updates TypeScript to v3.5.1 (and resolves Arrow TS Excessively Deep Type Instantiations #4452)
• Updates the BigInt compatibility type to use Object.setPrototypeOf(), yielding a 4x speedup
• Updates Int64 and Uint64 set routines to accept native bigint types if available
• Adds a docstring to the _InternalEmptyPlaceholderRecordBatch class added in ARROW-5396: [JS] Support files and streams with no record batches #4373