A couple of things I want to do that will benefit live streams/iterators:

Remove 'put' and 'del' events in favor of 'batch' event (just one event to listen to). This will also prepare us for hooks, especially a prewrite hook that would re-route db.put() to use a batch under the hood (and thus no longer emit a 'put' event).
In the 'batch' event, use "normalized" operation objects, meaning they should include encodings and other (default) options. And encodings should be objects here, rather than their string names.
The operation objects should include both the original input data (as today) and encoded data. Aka public data (what the public API received, e.g. db.batch()) and private data (what the private API received, e.g. db._batch()).

We can then support aligning the encoding of live data with the encoding of the iterator. E.g. if you do live(db, { valueEncoding: 'json' }) but then a db.put(key, value, { valueEncoding: 'buffer' }), the resulting 'batch' event includes the necessary information to transcode from buffer to json. Roughly like so:

const valueEncoding = db.valueEncoding('json')
const iterator = db.iterator({ valueEncoding })

db.on('batch', (operations) => {
  for (const op of operations) {
    if (op.valueEncoding.commonName === valueEncoding.commonName) {
      // Public data matches desired encoding
      const value = op.value
    } else if (op.valueEncoding.format === valueEncoding.commonName) {
      // Private data matches desired encoding
      const value = op.encodedValue
    } else {
      // Decode private data (one of view, buffer, utf8) to match desired encoding
      const transcoder = valueEncoding.createTranscoder(op.valueEncoding.format)
      const value = transcoder.decode(op.encodedValue)
    }
  }
})

That last step there (createTranscoder()) will just need a small utility method in level-transcoder:

Click to expand

function createTranscoder (format) {
  if (format === 'view') {
    return this.createViewTranscoder()
  } else if (format === 'buffer') {
    return this.createBufferTranscoder()
  } else if (format === 'utf8') {
    return this.createUTF8Transcoder()
  } else {
    throw new Error('nope')
  }
}

Uh oh!

Live streaming values from a level #222

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions