Bug Description

Reproducible By

Run the following code

async function main() {
    const p = []
    for (let i = 0; i < 1000; i++) {
        p.push(fetch('http://example.org/').then(re => console.log(re.status)));
    }
    await Promise.all(p);
}

main();

Expected Behavior

All promises to resolve. In particular I would expect undici to have some of the following behaviors:

• In cases like this where it is fetching from the same url I would expect undici to be
  • keeping an internal record of pending requests, if an identical request is already pending re-use the response across the multiple pending requests
  • keeping a cache of resolved requests and taking advantage of the expiry headers. Note that http://example.org/ has a 1 week expiry

  $ curl -I http://example.org/
  HTTP/1.1 200 OK
  Content-Encoding: gzip
  Accept-Ranges: bytes
  Age: 491770
  Cache-Control: max-age=604800
  Content-Type: text/html; charset=UTF-8
  Date: Wed, 08 May 2024 11:17:43 GMT
  Etag: "3147526947"
  Expires: Wed, 15 May 2024 11:17:43 GMT
  Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
  Server: ECAcc (nyd/D125)
  X-Cache: HIT
  Content-Length: 648
  

• In cases where different URLs are being fetched, limit the number of parallel requests taking place under the hood.

All of this should mean that for the above code - only 1 network request is actually made. Even better would be 0 network requests if another node process has already fetched http://example.org/ some time in the last week; and no promise rejections should occur from too many network requests trying to take place on my system at the same time.

Logs & Screenshots

Several of the promises reject with the error (this seems to consistently be the case for 1000+ fetch calls)

node:internal/deps/undici/undici:12344
    Error.captureStackTrace(err, this);
          ^

TypeError: fetch failed
    at node:internal/deps/undici/undici:12344:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 81)
    at async main (/home/jeswr/Documents/GitHub/jeswr/email_api_test/fetchCache.js:10:5) {
  cause: AggregateError
      at internalConnectMultiple (node:net:1114:18)
      at internalConnectMultiple (node:net:1177:5)
      at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
      at listOnTimeout (node:internal/timers:575:11)
      at process.processTimers (node:internal/timers:514:7) {
    code: 'ETIMEDOUT',
    [errors]: [
      Error: connect ETIMEDOUT 93.184.215.14:80
          at createConnectionError (node:net:1634:14)
          at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
          at listOnTimeout (node:internal/timers:575:11)
          at process.processTimers (node:internal/timers:514:7) {
        errno: -110,
        code: 'ETIMEDOUT',
        syscall: 'connect',
        address: '93.184.215.14',
        port: 80
      },
      Error: connect ENETUNREACH 2606:2800:21f:cb07:6820:80da:af6b:8b2c:80 - Local (:::0)
          at internalConnectMultiple (node:net:1176:40)
          at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
          at listOnTimeout (node:internal/timers:575:11)
          at process.processTimers (node:internal/timers:514:7) {
        errno: -101,
        code: 'ENETUNREACH',
        syscall: 'connect',
        address: '2606:2800:21f:cb07:6820:80da:af6b:8b2c',
        port: 80
      }
    ]
  }
}

Node.js v20.11.0

Environment

$ node -v
v20.11.0

Additional context

This issue was most recently prompted when I ran into o-development/ldo#43, but has been an ongoing pain point in several projects since migrating to use the node-native fetch available in Node 18+. For instance, it crops up in:

• e2e testing that does a lot of parallel fetch requests
• link-traversing query engines such as comunica