-
Notifications
You must be signed in to change notification settings - Fork 45
Comparing changes
Open a pull request
base repository: projectdiscovery/utils
base: v0.6.1
head repository: projectdiscovery/utils
compare: v0.7.0
- 6 commits
- 5 files changed
- 2 contributors
Commits on Nov 15, 2025
-
test(httputil): add (bench) tests
Signed-off-by: Dwi Siswanto <git@dw1.io>
Configuration menu - View commit details
-
Copy full SHA for 6f3cae1 - Browse repository at this point
Copy the full SHA 6f3cae1View commit details -
fix(httputil): optimize
ResponseChainmemory usageto prevent OOM. `ResponseChain` currently suffers from unbounded memory growth under high-concurrency workloads, particularly when processing large responses or compression bombs. This manifests as OOM kills during nuclei scans with many concurrent requests. The root cause is threefold: `(*bytes.Buffer).ReadFrom()` over-allocates by doubling capacity when size is unknown, the buffer pool accumulates large buffers w/o bounds, and each `ResponseChain` pre-allocates a `fullResponse` buffer, even when unused. Introduce `limitedBuffer` wrapper to constrain buffer growth. This wrapper uses 32KB chunks and caps total capacity at `maxBodyRead`, preventing the 2x over-allocation behavior of `(*bytes.Buffer).ReadFrom()`. Reading now grows incrementally rather than speculatively. Implement semaphore-gated pooling for large buffers. Buffers under 512KB are pooled freely as most HTML responses fall in this range. Buffers at or above 512KB are limited to 20 pooled instances via semaphore. When the limit is reached, excess large buffers are discarded and reclaimed by GC. This prevents pool pollution from transient large responses while still enabling reuse during burst periods. Remove the pre-allocated `fullResponse` buffer from `ResponseChain` struct. Generate it lazily only when `FullResponse()` is called. This reduces per-instance memory footprint by one-third and eliminates waste when callers only need headers or body separately. Add runtime configuration via `SetMaxBodySize()`, `SetBufferSize()`, and `SetMaxLargeBuffers()` to allow tuning based on deployment characteristics. Increase default max body size from 4MB to 8MB to accommodate modern web apps. Also remove dependency on docker/go-units. Provide typed accessor methods `HeadersBytes()`, `HeadersString()`, `BodyBytes()`, `BodyString()`, and `FullResponseString()` for safe read-only access. These prevent callers from inadvertently retaining pooled buffers beyond their lifecycle. The `FullResponse()` method now returns a buffer that must be explicitly managed by the caller. This is a breaking change but necessary to support lazy generation semantics. Testing with nuclei workloads shows stable memory usage under sustained load where previously OOM would occur within minutes. ```bash go test -v -run "(BurstW(ithPoolExhaustion|orkload)|SustainedConcurrency|RapidCreateDestroy|ConcurrentReads|M(emoryPressure|ixedWorkload))$" -count 1 ./http/ ``` Signed-off-by: Dwi Siswanto <git@dw1.io>
Configuration menu - View commit details
-
Copy full SHA for c9deb70 - Browse repository at this point
Copy the full SHA c9deb70View commit details -
chore(httputil): satisfy lints
Signed-off-by: Dwi Siswanto <git@dw1.io>
Configuration menu - View commit details
-
Copy full SHA for d5adade - Browse repository at this point
Copy the full SHA d5adadeView commit details -
feat(httputil): rm
SetMaxBodySizefunc &maxBodyReadvarSigned-off-by: Dwi Siswanto <git@dw1.io>
Configuration menu - View commit details
-
Copy full SHA for cbe8953 - Browse repository at this point
Copy the full SHA cbe8953View commit details -
feat(httputil): adds
ResponseChain.maxBodySizefieldSigned-off-by: Dwi Siswanto <git@dw1.io>
Configuration menu - View commit details
-
Copy full SHA for d06a121 - Browse repository at this point
Copy the full SHA d06a121View commit details
Commits on Nov 18, 2025
-
Merge pull request #700 from projectdiscovery/dwisiswant0/fix/httputi…
…l/optimize-ResponseChain-memory-usage-to-prevent-OOM fix(httputil): optimize `ResponseChain` memory usage
Configuration menu - View commit details
-
Copy full SHA for 5314f45 - Browse repository at this point
Copy the full SHA 5314f45View commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff v0.6.1...v0.7.0