Introduction

Instead of only exposing the AWS package in Fleet/Integrations list, all supported services inside AWS package should also have their own icons and displayed in Fleet/Integrations list in Kibana UI. Using EC2 as an example, under Integrations, we should have both AWS and EC2 icons:

Design

Here is my initial design. I split them into three parts for three repos I believe we need to make changes on: elastic/integrations, elastic/package-registry, and elastic/kibana.

Part 1(integration):

In elastic/integrations repo, for packages, we should introduce two new config options into manifest.yml files.
At the integration package level, we have a manifest.yml file that includes the icon that UI is using for the whole integration. We should add a config parameter expand_data_stream at the package level to indicate this package should be expanded to display all data streams.
For example, manifest.yml should look like this:

icons:
 - src: /img/logo_aws.svg
   title: logo aws
   size: 32x32
   type: image/svg+xml
expand_data_stream: true
policy_templates:
 - name: aws
   title: AWS logs and metrics
   description: Collect logs and metrics from AWS services
   inputs:
     - type: s3
	...
- type: aws/metrics
...

At the data stream level, we also have an existing manifest.yml file. We should add a section for icons and a section for description for each data stream so when expand_data_stream is true, these icons at the data stream level will be displayed in Kibana Fleet/Integrations list.

For example, manifest.yml file should look like this:

title: AWS EC2 metrics
release: beta
type: metrics
description: AWS EC2 Integration
icons:
 - src: /img/logo_aws_ec2.svg
   title: logo aws ec2
   size: 32x32
   type: image/svg+xml
streams:
 - input: aws/metrics
 ...

Part 2(Elastic Package Registry):

In the Elastic Package Registry(EPR), we should adjust the code to read the new config option expand_data_stream. When expand_data_stream is set to true, abstract data stream info and store them into the /search API endpoint result. I created a draft PR for this to show the changes to the /search API response.

For example with expand_data_stream: true in AWS package, /search API response will look like this:

[
  {
    "name": "aws",
    "title": "AWS",
    "version": "0.3.18",
    "release": "beta",
    "description": "AWS Integration",
    "type": "integration",
    "download": "/epr/aws/aws-0.3.18.zip",
    "path": "/package/aws/0.3.18",
    "icons": [
      {
        "src": "/img/logo_aws.svg",
        "path": "/package/aws/0.3.18/img/logo_aws.svg",
        "title": "logo aws",
        "size": "32x32",
        "type": "image/svg+xml"
      }
    ],
    "base_data_streams": [
      ...
      {
        "type": "metrics",
        "dataset": "aws.ec2_metrics",
        "title": "AWS EC2 metrics",
        "release": "beta",
        "description": "AWS EC2 Integration",
        "icons": [
          {
            "src": "../img/logo_aws_ec2.svg",
            "path": "",
            "title": "logo aws ec2",
            "size": "32x32",
            "type": "image/svg+xml"
          }
        ]
      },
      ...
    ]
  }
]

Part 3(Kibana Fleet UI):

Continue to take the response from the EPR's /search endpoint, also take into account the icons section under base_data_streams.

Follow up

Once we achieve the granular search for AWS, we can split README.md into each data stream/service. This way when the user clicks on an icon for a specific service, some dedicated documentation will show up instead of the full documentation.

@hbharding @jasonrhodes WDYT about this approach?

cc @sorantis @masci