Skip to content

[wip] etl: zero-copy fileDataProvider on mmap#19512

Closed
AskAlexSharov wants to merge 26 commits into
mainfrom
alex/etl_mmap_34
Closed

[wip] etl: zero-copy fileDataProvider on mmap#19512
AskAlexSharov wants to merge 26 commits into
mainfrom
alex/etl_mmap_34

Conversation

@AskAlexSharov

@AskAlexSharov AskAlexSharov commented Feb 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

for #19778
In PR:

  • inMemProvider return pointer to internal byte-slices
  • fileProvider return pointer to mmap
  • don't need copy/bytes.Clone anywhere - because data to external buffers
  • don't need

inMemProvider:

  ┌───────────────┬────────────────┬──────────────────┬─────────┐                                                                                                                   
  │ Key/Val Size  │ GetRef (ns/op) │ Get copy (ns/op) │ Speedup │                                                                                                                   
  ├───────────────┼────────────────┼──────────────────┼─────────┤                                                                                                                   
  │ key32_val32   │ 30,172         │ 49,856           │ 1.65x   │                                                                                                                   
  ├───────────────┼────────────────┼──────────────────┼─────────┤                                                                                                                   
  │ key32_val128  │ 30,607         │ 67,180           │ 2.20x   │                                                                                                                   
  ├───────────────┼────────────────┼──────────────────┼─────────┤                                                                                                                   
  │ key32_val256  │ 30,389         │ 92,183           │ 3.03x   │                                                                                                                   
  ├───────────────┼────────────────┼──────────────────┼─────────┤                                                                                                                   
  │ key32_val1024 │ 30,425         │ 240,858          │ 7.92x   │                                                                                                                 
  ├───────────────┼────────────────┼──────────────────┼─────────┤                                                                                                                   
  │ key64_val1024 │ 30,152         │ 251,985          │ 8.36x   │
  └───────────────┴────────────────┴──────────────────┴─────────┘

fileProvider:

 name                    old ns/op    new ns/op    delta                                                                        
  MmapZeroCopy               211643       120180     -43.20%
  BufioRead                  211643       120180     -43.20%                                                                     
                                                                                                                               
  name                    old B/op     new B/op     delta
  MmapZeroCopy               524386         370     -99.93%
  BufioRead                  524386         370     -99.93%

@AskAlexSharov AskAlexSharov mentioned this pull request Mar 10, 2026
Closed
@AskAlexSharov AskAlexSharov changed the title [wip] etl: zero-copy on mmap [wip] etl: zero-copy fileDataProvider on mmap Mar 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sudeepdino008 sudeepdino008 Awaiting requested review from sudeepdino008 sudeepdino008 will be requested when the pull request is marked ready for review sudeepdino008 is a code owner

Assignees

No one assigned

Labels

performance

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AskAlexSharov @Giulio2002