Use memory pools where needed to help reduce allocations#541
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces memory pools using sync.Pool to reduce memory allocations in performance-critical paths of the web crawler, particularly in body processing, XML parsing, and data copying operations.
Key changes:
- Optimized UUID generation to use
uuid.NewString()instead ofuuid.New().String() - Added
sync.Poolfor bufio.Reader instances in XML parsing - Added
sync.Poolfor bytes.Buffer instances in headless and general body processing - Added
sync.Poolfor byte slice buffers in the connection utility's copy operations
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
pkg/models/item.go |
Optimized UUID string generation to avoid intermediate allocation |
internal/pkg/postprocessor/extractor/xml.go |
Added bufio.Reader pool for XML parsing to reuse reader instances |
internal/pkg/archiver/headless/body.go |
Added bytes.Buffer pool for headless body processing with proper cleanup |
internal/pkg/archiver/general/body.go |
Added bytes.Buffer pool for MIME detection, but contains a critical double-Put bug |
internal/pkg/archiver/connutil/connutil.go |
Added byte slice pool for copy operations to avoid repeated 4KB allocations |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #541 +/- ##
==========================================
+ Coverage 56.48% 56.61% +0.13%
==========================================
Files 131 131
Lines 6581 6606 +25
==========================================
+ Hits 3717 3740 +23
- Misses 2489 2491 +2
Partials 375 375
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
willmhowes
left a comment
There was a problem hiding this comment.
Changes look good, and I compiled locally and successfully ran a test crawl as a sanity check!
No description provided.