This issue is a placeholder to capture requirements - current pain points + desired enhancements - for a redesigned package registry. A number of discussions have happened on this topic privately; this issue is the place to centralize them as as well as bring more visibility and discoverability to them.

Summarizing from various private discussions:

• The current package registry works quite well but we foresee some potential scalability, performance, and reliability issues as the number of packages continues to grow.
  • We need to come up with a design that would allow the package registry process to start up in low and constant amount of time.
  • Ideally the size of the package registry image and the memory used by the package registry process would not grow proportionally with the number of packages being served by the registry.
    • Big packages should be supported Support ML trained models in integration packages package-spec#135.
  • The current package registry bears two distinct responsibilities under the hood: that of providing a search API for packages and that of acting as a static file server for packages. There might be an opportunity to decouple these responsibilities from each other to address the performance and scalability concerns mentioned above, but also to increase the reliability by delegating each responsibility to appropriate technologies.
• There will be a need for private / air-gapped package registries, especially for deployments that are subject to regulatory approval with very strict security/access requirements. Provide self-managed package-registry that backs integrations UI integrations#1178
• The registry will ideally support packages from multiple sources, not just Elastic-developed packages.
• The ability to serve as a source of truth for our marketing integrations page https://www.elastic.co/integrations
  • The ability to render landing pages and docs
  • Web-scale through a CDN if not already