Skip to content

Handling custom serialization with MsgPack directly #4379

@jakirkham

Description

@jakirkham

Today we using things like extract_serialize to pull out objects MsgPack can't handle and serialize alongside it. In benchmarks we have done our extra handling code takes about 3x more time than MsgPack alone. An interesting idea to follow up on would be to see if we can add an ExtType or something to default encoding/decoding to handle out-of-band buffers and merely track where to insert them later. This would be analogous to how pickle works with out-of-band buffers. Though it may speed up serializing and deserializing by doing fewer passes over the data by leveraging MsgPack's own passes. In theory we could get up to a 4x speed up in serialization by following this strategy.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions