margaret

package module
v0.4.4-0...-1387e6e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 23, 2024 License: MIT Imports: 3 Imported by: 0

README

Margaret Go Reference Github Actions Tests Go Report Card REUSE status

Margaret is go-ssb's append-only log* provider, and greatly inspired by flumedb. Compatible with Go 1.13+.

margaret the log lady, 1989 edition

the project name is inspired by Twin Peaks's character Margaret aka the log lady

Margaret has the following facilities:

  • an append-only log interface .Append(interface{}), .Get(int64)
  • queries .Query(...QuerySpec) for retrieving ranges based on sequence numbers e.g. .Gt(int64), or limiting the amount of data returned .Limit(int64)
  • a variety of index mechanisms, both for categorizing log entries into buckets and for creating virtual logs (aka sublogs)

Margaret is one of a few key components that make the go implementation of ssb tick, for example:

  • ssb/sbot uses margaret for storing each peer's data
Log storage

Margaret outputs data according to the offset2 format, which is inspired by (but significantly differs from) flumelog-offset.

In brief: margaret stores the data of all logs in the three following files:

  • data stores the actual data (with a length-prefix before each entry)
  • ofst indexes the starting locations for each data entry in data
  • jrnl an integrity checking mechanism for all three files; a checksum of sorts, more details

More details

There are a few concepts that might be tough to digest for newcomers on first approach:

  • multilogs, a kind of tree-based index, where each leaf is a margaret.Log
    • in other words: it creates virtual sublogs that map to entries in an offset log (see log storage above)
  • margaret/indexes similar to leveldb indexes (arbitrary key-value stores)
  • sublogs (and rxLog/receiveLog/offsetLog and its equivalence to offset.log)
  • queries
  • zeroing out, or replacing, written data

For more on these concepts, visit the dev.scutttlebutt.nz portal for in-depth explanations.

* margaret is technically an append-based log, as there is support for both zeroing out and replacing items in the log after they have been written. Given the relative ubiquity of append-only logs & their uses, it's easier to just say append-only log.

Documentation

Index

Constants

View Source 
const (
	// SeqEmpty is the current sequence number of an empty log
	SeqEmpty int64 = -1

	// SeqErrored is returned if an operation (like Append) fails
	SeqErrored int64 = -2

	SeqSublogDeleted int64 = -255
)

Variables

View Source 
var ErrNulled = errors.New("margaret: Entry Nulled")
View Source 
var OOB oob

OOB is an out of bounds error

Functions

func IsErrNulled 

func IsErrNulled(err error) bool

func IsOutOfBounds 

func IsOutOfBounds(err error) bool

IsOutOfBounds returns whether a particular error is an out-of-bounds error

Types

type Alterer 

type Alterer interface {
	Null(int64) error

	Replace(int64, []byte) error
}

type Codec 

type Codec interface {
	// Marshal encodes a single value and returns the serialized byte slice.
	Marshal(value interface{}) ([]byte, error)

	// Unmarshal decodes and returns the value stored in data.
	Unmarshal(data []byte) (interface{}, error)

	NewDecoder(io.Reader) Decoder
	NewEncoder(io.Writer) Encoder
}

Codec marshals and unmarshals values and creates encoders and decoders

type Decoder 

type Decoder interface {
	Decode() (interface{}, error)
}

Decoder decodes values

type Encoder 

type Encoder interface {
	Encode(v interface{}) error
}

Encoder encodes values

type Log 

type Log interface {
	// Seq returns the current sequence number, which is also the number of entries in the log
	Seqer

	// Changes returns an observable that holds the current sequence number
	Changes() luigi.Observable

	// Get returns the entry with sequence number seq
	Get(seq int64) (interface{}, error)

	// Query returns a stream that is constrained by the passed query specification
	Query(...QuerySpec) (luigi.Source, error)

	// Append appends a new entry to the log
	Append(interface{}) (int64, error)
}

Log stores entries sequentially, which can be queried individually using Get or as streams using Query.

type Query 

type Query interface {
	// Gt makes the source return only items with sequence numbers > seq.
	Gt(seq int64) error
	// Gte makes the source return only items with sequence numbers >= seq.
	Gte(seq int64) error
	// Lt makes the source return only items with sequence numbers < seq.
	Lt(seq int64) error
	// Lte makes the source return only items with sequence numbers <= seq.
	Lte(seq int64) error
	// Limit makes the source return only up to n items.
	Limit(n int) error

	// Reverse makes the source return the lastest values first
	Reverse(yes bool) error

	// Live makes the source block at the end of the log and wait for new values
	// that are being appended.
	Live(bool) error

	// SeqWrap makes the source return values that contain both the item and its
	// sequence number, instead of the item alone.
	SeqWrap(bool) error
}

Query is the interface implemented by the concrete log implementations that collects the constraints of the query.

type QuerySpec 

type QuerySpec func(Query) error

QuerySpec is a constraint on the query.

func ErrorQuerySpec 

func ErrorQuerySpec(err error) QuerySpec

ErrorQuerySpec makes the log.Query call return the passed error.

func Gt 

func Gt(s int64) QuerySpec

Gt makes the source return only items with sequence numbers > seq.

func Gte 

func Gte(s int64) QuerySpec

Gte makes the source return only items with sequence numbers >= seq.

func Limit 

func Limit(n int) QuerySpec

Limit makes the source return only up to n items.

func Live 

func Live(live bool) QuerySpec

Live makes the source block at the end of the log and wait for new values that are being appended.

func Lt 

func Lt(s int64) QuerySpec

Lt makes the source return only items with sequence numbers < seq.

func Lte 

func Lte(s int64) QuerySpec

Lte makes the source return only items with sequence numbers <= seq.

func MergeQuerySpec 

func MergeQuerySpec(spec ...QuerySpec) QuerySpec

MergeQuerySpec collects several contraints and merges them into one.

func Reverse 

func Reverse(yes bool) QuerySpec

func SeqWrap 

func SeqWrap(wrap bool) QuerySpec

SeqWrap makes the source return values that contain both the item and its sequence number, instead of the item alone.

type SeqWrapper 

type SeqWrapper interface {
	Seqer

	// Value returns the item itself.
	Value() interface{}
}

SeqWrapper wraps a value to attach a sequence number to it.

func WrapWithSeq 

func WrapWithSeq(v interface{}, seq int64) SeqWrapper

WrapWithSeq wraps the value v to attach a sequence number to it.

type Seqer 

type Seqer interface {
	Seq() int64
}

Seqer returns the current sequence of a log

Source Files

View all Source files

Directories

Path Synopsis
codec
cbor
json
msgpack
badger
badger/test
mapidx
mapidx/test
mkv
mkv/test
test
test/all
internal
persist
persist/badger
persist/fs
persist/mkv
persist/sqlite
seqobsv
Package seqobsv wants to supply an observable value sepcialized for sequence numbers in append-only logs.
 Package seqobsv wants to supply an observable value sepcialized for sequence numbers in append-only logs.
mem
test
roaring
roaring/badger
roaring/badger/cmd/mbdump command
roaring/fs
roaring/mkv
roaring/sqlite
roaring/test
test
test/all
Package offset2 implements a margaret log as persisted sequence of data across multiple files.
 Package offset2 implements a margaret log as persisted sequence of data across multiple files.
test
all

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.