Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: projectdiscovery/utils
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.6.1
Choose a base ref
...
head repository: projectdiscovery/utils
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v0.7.0
Choose a head ref
  • 6 commits
  • 5 files changed
  • 2 contributors

Commits on Nov 15, 2025

  1. test(httputil): add (bench) tests

    Signed-off-by: Dwi Siswanto <git@dw1.io>
    dwisiswant0 committed Nov 15, 2025
    Configuration menu
    Copy the full SHA
    6f3cae1 View commit details
    Browse the repository at this point in the history
  2. fix(httputil): optimize ResponseChain memory usage

    to prevent OOM.
    
    `ResponseChain` currently suffers from unbounded
    memory growth under high-concurrency workloads,
    particularly when processing large responses
    or compression bombs. This manifests as OOM kills
    during nuclei scans with many concurrent requests.
    
    The root cause is threefold:
    `(*bytes.Buffer).ReadFrom()` over-allocates by
    doubling capacity when size is unknown, the buffer
    pool accumulates large buffers w/o bounds, and
    each `ResponseChain` pre-allocates a
    `fullResponse` buffer, even when unused.
    
    Introduce `limitedBuffer` wrapper to constrain
    buffer growth. This wrapper uses 32KB chunks and
    caps total capacity at `maxBodyRead`, preventing
    the 2x over-allocation behavior of
    `(*bytes.Buffer).ReadFrom()`. Reading now grows
    incrementally rather than speculatively.
    
    Implement semaphore-gated pooling for large
    buffers. Buffers under 512KB are pooled freely as
    most HTML responses fall in this range. Buffers at
    or above 512KB are limited to 20 pooled instances
    via semaphore. When the limit is reached, excess
    large buffers are discarded and reclaimed by GC.
    This prevents pool pollution from transient large
    responses while still enabling reuse during burst
    periods.
    
    Remove the pre-allocated `fullResponse` buffer
    from `ResponseChain` struct. Generate it lazily
    only when `FullResponse()` is called. This reduces
    per-instance memory footprint by one-third and
    eliminates waste when callers only need headers or
    body separately.
    
    Add runtime configuration via `SetMaxBodySize()`,
    `SetBufferSize()`, and `SetMaxLargeBuffers()` to
    allow tuning based on deployment characteristics.
    Increase default max body size from 4MB to 8MB to
    accommodate modern web apps. Also remove
    dependency on docker/go-units.
    
    Provide typed accessor methods `HeadersBytes()`,
    `HeadersString()`, `BodyBytes()`, `BodyString()`,
    and `FullResponseString()` for safe read-only
    access. These prevent callers from inadvertently
    retaining pooled buffers beyond their lifecycle.
    
    The `FullResponse()` method now returns a buffer
    that must be explicitly managed by the caller.
    This is a breaking change but necessary to support
    lazy generation semantics.
    
    Testing with nuclei workloads shows stable memory
    usage under sustained load where previously OOM
    would occur within minutes.
    
    ```bash
    go test -v -run "(BurstW(ithPoolExhaustion|orkload)|SustainedConcurrency|RapidCreateDestroy|ConcurrentReads|M(emoryPressure|ixedWorkload))$" -count 1 ./http/
    ```
    
    Signed-off-by: Dwi Siswanto <git@dw1.io>
    dwisiswant0 committed Nov 15, 2025
    Configuration menu
    Copy the full SHA
    c9deb70 View commit details
    Browse the repository at this point in the history
  3. chore(httputil): satisfy lints

    Signed-off-by: Dwi Siswanto <git@dw1.io>
    dwisiswant0 committed Nov 15, 2025
    Configuration menu
    Copy the full SHA
    d5adade View commit details
    Browse the repository at this point in the history
  4. feat(httputil): rm SetMaxBodySize func & maxBodyRead var

    Signed-off-by: Dwi Siswanto <git@dw1.io>
    dwisiswant0 committed Nov 15, 2025
    Configuration menu
    Copy the full SHA
    cbe8953 View commit details
    Browse the repository at this point in the history
  5. feat(httputil): adds ResponseChain.maxBodySize field

    Signed-off-by: Dwi Siswanto <git@dw1.io>
    dwisiswant0 committed Nov 15, 2025
    Configuration menu
    Copy the full SHA
    d06a121 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2025

  1. Merge pull request #700 from projectdiscovery/dwisiswant0/fix/httputi…

    …l/optimize-ResponseChain-memory-usage-to-prevent-OOM
    
    fix(httputil): optimize `ResponseChain` memory usage
    Mzack9999 authored Nov 18, 2025
    Configuration menu
    Copy the full SHA
    5314f45 View commit details
    Browse the repository at this point in the history
Loading