Background

Before Elastic Agent and packages existed we introduced lightweight modules in Metricbeat to simplify the development of new metricsets and packages. The goal was to move to pure config defined modules as we have in Filebeat. Over time we have been making more and more use of it, for example in the AWS package. This is great as it reduces the number of code changes we have to make.

With the introduction of elastic packages, it has become even more important to have packages purely based on configurations because that is what is delivered as part of the package. Unfortunately many modules use lightweight modules but the packages we build also depend on the lightweight modules to be there, meaning the code/config files have to be contributed to the beats repository to work. This makes it easier to keep packages and modules in sync but it also means we keep adding to the beats repository. This issue is caused by running the migration script from Metricbeat to integrations.

Actions

1. For the old migrated data streams (from Bests) like rds: we should move the configuration part from Beats into integrations.
2. For all data streams: use ingest pipeline instead of rename processor.
3. For all new packages, we should add configuration directly into integrations.

Impacted Integrations (wip)

• aws dynamodb Remove lightweight modules from aws dynamodb data stream #3597
• aws ebs Remove lightweight modules from aws ebs data stream #3598
• aws lambda Remove lightweight modules from aws lambda data stream #3599
• aws natgateway Remove lightweight modules from aws natgateway data stream #3600
• aws s3_daily_storage Remove lightweight modules from aws s3_daily_storage data stream #3601
• aws s3_request Remove lightweight modules from aws s3_request data stream #3602
• aws sns Remove lightweight modules from aws sns data stream #3603
• aws sqs Remove lightweight modules from aws sqs data stream #3604
• aws transitgateway Remove lightweight modules from aws transitgateway data stream #3605
• aws usage Remove lightweight modules from aws usage data stream #3606
• aws kinesis [AWS] migrate AWS Kinesis module from beats to integrations  #3822
• azure_metrics container_instance Remove lightweight modules from azure_metrics container_instance data stream #3592
• azure_metrics container_registry Remove lightweight modules from azure_metrics container_registry data stream #3594
• azure_metrics container_service Remove lightweight modules from azure_metrics container_service data stream #3595
• azure_metrics database_account Remove lightweight modules from azure_metrics database_account data stream #3596
• gcp.billing [GCP] Remove lightweight module in gcp.billing data stream #3720
• gcp.compute [GCP] Remove lightweight module in gcp.compute data stream #3721
• gcp.firestore [GCP] Remove lightweight module in gcp.firestore data stream #3722
• gcp.storage [GCP] Remove lightweight module in gcp.storage data stream #4262
• gcp.pubsub [GCP] Remove lightweight module in gcp.pubsub data stream #4263
• gcp.loadbalancing [GCP] Remove lightweight module in gcp.loadbalancing data stream #4264
• gcp.dataproc [GCP] Remove lightweight module in gcp.dataproc data stream #4265