Skip to content

Store objects in "header+payload" format #2925

@roman-khimov

Description

@roman-khimov

Is your feature request related to a problem? Please describe.

I'm always frustrated when I look at how we allocate a lot of memory to fetch data from storage. #2316, #2178, #1398. #2814 also reminds about it.

Describe the solution you'd like

#1932 has some valid input for these problems, although it's localized to CLI, whereas I'm more concerned about SN. We can store/fetch objects in a bit different manner, serializing header and payload separately right at the FSTree level. Then object get operation would return a header and an io.ReaderCloser pointing to the file (if it's not too small). This can then be streamed to the outside as needed (but likely API changes are required as well).

Describe alternatives you've considered

Keep pulling MBs of data from the storage is no fun.

Metadata

Metadata

Assignees

No one assigned

    Labels

    I4No visible changesS1Highly significantU3RegularenhancementImproving existing functionalityneofs-storageStorage node application issuesperformanceMore of something per second

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions