I'd like to propose a new "depcache" field in the import map, as an optimal solution to the waterfall problem for production optimization.

The core idea is to provide the ability for tools that generate import maps to also generate the metadata of the module graph as a sort of graph cache (dependency cache) at the same time - a map from URLs to their list of dependency specifiers.

With a populated depcache, as soon as a module is fetched, the depcache can be consulted, and the corresponding cached list of module dependencies preloaded in parallel immediately.

For example:

{
  "imports": {
    "main": "/main.js",
    "lazy-feature": "/lazy-feature.js",
    "dep1": "/lib/dep1.js",
    "dep2": "/lib/dep2.js"
  },
  "depcache": {
    "/main.js": ["dep1"],
    "/lazy-feature.js": ["dep1", "dep2"]
  }
}

Say the main app loads "main" on startup. This is resolved to /main.js, which the depcache allows us to see that we need to also load /lib/dep1.js immediately. This trace applies recursively to all requests as well. These known dependency requests are then made in parallel with the main request, possibly from the cache, the app thus loading with zero latency waterfalls applying.

Later on a dynamic import('lazy-feature') is executed. At this point, the depcache can again be consulted, to see that /lazy-feature.js will import both dep1 and dep2. Since /lib/dep1.js is already in the module map we do not need to send a new request, so we then immediately send out just two requests - one for /lazy-feature.js and one for /lib/dep2.js, again getting lazy loading with full immediate parallel requests, without any duplication and supporting far-future caching. No matter how deep the dependency tree, there is never a waterfall so long as there is a depcache entry.

Note that the unresolved dependency specifier is included in the depcache array. This allows the import map to remain the full source of truth for dependency resolution, and the depcache for eg a cached module doesn't go stale despite resolutions changing.

The alternative as mentioned in the readme of this spec is a more general preloading spec. I have discussed with various spec authors and implementors the more general preload spec over the past year or two, and have found there to be a lack of interest - mostly I think because for most web resources, 2-3 round trips is the maximum due to the nature of HTML / CSS development. I would argue that modules are actually unique in having the problem of potentially N levels deep dependency trees, where N can be over 10, thus the latency problem between these depths is truly unique to module graphs.

This depcache proposal seems to me the simplest path forward right now to solve this latency waterfall problem in a fully optimal way, given that a more general preload manifest is not getting traction, but I'm hoping by posting both this proposal and #208, we can drive these conversations forward to ensure that this production optimization solution is tackled, as it really needs to be now for modules.

I'd like to also be able to move forward with this or a similar proposal in both SystemJS and ES Module Shims. Depcache as specified here has worked very well in previous versions of SystemJS for many years, and I'd like to start shipping this feature or similar in both projects again soon as it is a critical performance necessity right now for users of these projects today. Both projects aim to track these standards so hopefully we can continue these discussions and continue to solve these problems optimally.