Skip to content

Add array/map iterators#403

Merged
klauspost merged 7 commits intotinylib:masterfrom
klauspost:map-arr-iters
Sep 17, 2025
Merged

Add array/map iterators#403
klauspost merged 7 commits intotinylib:masterfrom
klauspost:map-arr-iters

Conversation

@klauspost
Copy link
Collaborator

@klauspost klauspost commented Aug 27, 2025

Allows to receive iterators for arrays or maps, so inputs can be directly iterated over.

New functions in msgp:

// ReadArray returns an iterator that can be used to iterate over the elements
// of an array in the MessagePack data while being read by the provided Reader.
// The type parameter V specifies the type of the elements in the array.
// The returned iterator implements the iter.Seq[V] interface,
// allowing for sequential access to the array elements.
func ReadArray[T any](m *Reader, readFn func() (T, error)) iter.Seq2[T, error]

// WriteArray writes an array to the provided Writer.
// The writeFn parameter specifies the function to use to write each element of the array.
func WriteArray[T any](w *Writer, a []T, writeFn func(T) error) error

// ReadMap returns an iterator that can be used to iterate over the elements
// of a map in the MessagePack data while being read by the provided Reader.
// The type parameters K and V specify the types of the keys and values in the map.
// The returned iterator implements the iter.Seq2[K, V] interface,
// allowing for sequential access to the map elements.
// The returned function can be used to read any error that
// occurred during iteration when iteration is done.
func ReadMap[K, V any](m *Reader, readKey func() (K, error), readVal func() (V, error)) (iter.Seq2[K, V], func() error)

// WriteMap writes a map to the provided Writer.
// The writeKey and writeVal parameters specify the functions
// to use to write each key and value of the map.
func WriteMap[K comparable, V any](w *Writer, m map[K]V, writeKey func(K) error, writeVal func(V) error) error

// WriteMapSorted writes a map to the provided Writer.
// The keys of the map are sorted before writing.
// This provides deterministic output, but will allocate to sort the keys.
// The writeKey and writeVal parameters specify the functions
// to use to write each key and value of the map.
func WriteMapSorted[K cmp.Ordered, V any](w *Writer, m map[K]V, writeKey func(K) error, writeVal func(V) error) error 

// ReadArrayBytes returns an iterator that can be used to iterate over the elements
// of an array in the MessagePack data while being read by the provided Reader.
// The type parameter V specifies the type of the elements in the array.
// After the iterator is exhausted, the remaining bytes in the buffer
// and any error can be read by calling the returned function.
func ReadArrayBytes[T any](b []byte, readFn func([]byte) (T, []byte, error)) (iter.Seq[T], func() (remain []byte, err error))

// AppendArray writes an array to the provided buffer.
// The writeFn parameter specifies the function to use to write each element of the array.
// The returned buffer contains the encoded array.
// The function panics if the array is larger than math.MaxUint32 elements.
func AppendArray[T any](b []byte, a []T, writeFn func(b []byte, v T) []byte) []byte

// ReadMapBytes returns an iterator over key/value
// pairs from a MessagePack map encoded in b.
// The iterator yields K,V pairs, and this function also returns
// a closure to get the remaining bytes and any error.
func ReadMapBytes[K any, V any](b []byte,
	readK func([]byte) (K, []byte, error),
	readV func([]byte) (V, []byte, error)) (iter.Seq2[K, V], func() (remain []byte, err error))


// AppendMap writes a map to the provided buffer.
// The writeK and writeV parameters specify the functions to use to write each key and value of the map.
// The returned buffer contains the encoded map.
// The function panics if the map is larger than math.MaxUint32 elements.
func AppendMap[K comparable, V any](b []byte, m map[K]V,
	writeK func(b []byte, k K) []byte,
	writeV func(b []byte, v V) []byte) []byte

// AppendMapSorted writes a map to the provided buffer.
// Keys are sorted before writing.
// This provides deterministic output, but will allocate to sort the keys.
// The writeK and writeV parameters specify the functions to use to write each key and value of the map.
// The returned buffer contains the encoded map.
// The function panics if the map is larger than math.MaxUint32 elements.
func AppendMapSorted[K cmp.Ordered, V any](b []byte, m map[K]V,
	writeK func(b []byte, k K) []byte,
	writeV func(b []byte, v V) []byte) []byte

// DecoderFrom allows augmenting any type with a DecodeMsg method into a method
// that reads from Reader and returns a T.
// Provide an instance of T. This value isn't used.
// See ReadArray/ReadMap "struct" examples for usage.
func DecoderFrom[T any, PT DecodePtr[T]](r *Reader, _ T) func() (T, error)

// DecoderFromBytes allows augmenting any type with an UnmarshalMsg
// method into a method that reads from []byte and returns a T.
// Provide an instance of T. This value isn't used.
// See ReadArrayBytes or ReadMapBytes "struct" examples for usage.
func DecoderFromBytes[T any, PT UnmarshalPtr[T]](_ T) func([]byte) (T, []byte, error) 

// EncoderTo allows augmenting any type with an EncodeMsg
// method into a method that writes to Writer on each call.
// Provide an instance of T. This value isn't used.
// See ReadArray or ReadMap "struct" examples for usage.
func EncoderTo[T any, _ FlexibleEncoder[T]](w *Writer, _ T) func(T) error 

// EncoderToBytes allows augmenting any type with a MarshalMsg method into a method
// that reads from T and returns a []byte.
// Provide an instance of T. This value isn't used.
// See ReadArrayBytes or ReadMapBytes "struct" examples for usage.
func EncoderToBytes[T any, _ FlexibleMarshaler[T]](_ T) func([]byte, T) []byte

Examples included in godoc.

@klauspost
Copy link
Collaborator Author

(adding examples would probably also be a good idea)

@klauspost
Copy link
Collaborator Author

@philhofer Much more happy about the revised functions - and not too inconvenient to use.

@klauspost klauspost requested a review from philhofer August 30, 2025 13:47
@klauspost klauspost merged commit 924e018 into tinylib:master Sep 17, 2025
4 checks passed
@klauspost klauspost deleted the map-arr-iters branch September 17, 2025 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants