Test Environment Setup

Elastic package should be running the stack.

1. Go to the AWS directory and run elastic-package build.
2. Run elastic-package stack up -v -d --services package-registry to update the integration.

Datastreams

Add agent.id and check for cloud.availability_zone on migrated and merged TSDB data streams: #6421

• Usage
• ELB
• DynamoDB
• EBS
• ECS
• Kinesis
• Lambda
• Natgateway
• Redshift

Add metrics and dimensions:

• API Gateway

  • Dimensions: [AWS][API Gateway] Set dimension fields #6950
  • Metric type: [AWS][API Gateway] Fix metric type #6952

• EMR: [AWS][EMR] Update metric type and set dimensions fields #6964

• Billing - Important: TSDB was disabled on this data stream, because of this issue [TSDB] Error: the document timestamp X is outside of ranges of currently writable indices #7345

  • Add dimension fields: [AWS] Set dimensions fields of Billing, SQS and SNS data streams #6311
    • Verification of the count of documents (before & after TSDB enablement) in Discover Interface
    • Verify if field mapping is correct in the data stream template.
  • Adding metric_type mapping: [AWS] Add metric type to SNS, SQS and Billing data streams #6319
    • Verification of data in visualisation after enabling TSDB flag in kibana
    • Verify if field mapping is correct in the data stream template.

• Cloudwatch metrics

  • Add dimension fields: [AWS] [CloudWatch] Add dimensions metadata #6827
  • Adding metric_type mapping: [AWS Cloudwatch] Add metric_type #6999

• DynamoDB: [AWS] TSDB enablement for usage, ELB and DynamoDB data streams #6227

• EBS: [AWS][EBS] Enable TSDB support #6061

• EC2:

  • Add dimension fields: [AWS] Add dimensions to EC2 data stream. #7487
  • Adding metric_type mapping: [AWS][EC2] Add metric type to EC2 #7490

• ECS: [AWS][ECS] TSDB Enablement #5894

• ELB: [AWS] TSDB enablement for usage, ELB and DynamoDB data streams #6227

• Firewall metrics

  • Add dimension fields: [AWS] Add dimensions to firewall, transit gateway and vpn data streams #6339
    • Verification of the count of documents (before & after TSDB enablement) in Discover Interface
    • Verify if field mapping is correct in the data stream template.
  • Adding metric_type mapping: [AWS] Add metric type to VPN, Firewall and Transit Gateway data streams #6365
    • Verification of data in visualisation after enabling TSDB flag in kibana
    • Verify if field mapping is correct in the data stream template.

• Kinesis: [AWS Kinesis] TSDB Enablement #5864

• Lambda: [AWS][Lambda] TSDB Enablement #5970

• Natgateway: [AWS][Nategateway] TSDB Enablement #6101

• RDS: [AWS][RDS] Enable TSDB support #6078

• Redshift: [AWS][Redshift] TSDB Enablement #6082

• S3 daily storage & S3 request:

  • Dimensions: [AWS][S3] Add dimensions field to s3_daily_storage and s3_request #6666
  • Metrics: [AWS][S3] Add metric_type metadata to s3_daily_storage and s3_request data streams #6667

• S3 storage lens

  • Add dimension fields: [AWS] Add dimensions to S3 Storage Lens #6583
    • Verification of the count of documents (before & after TSDB enablement) in Discover Interface - documents every 24h, not easy to check
    • Verify if field mapping is correct in the data stream template.
  • Adding metric_type mapping: [AWS] Add metric type to S3 Storage Lens #6584
    • Verification of data in visualisation after enabling TSDB flag in kibana
    • Verify if field mapping is correct in the data stream template.

• SNS

  • Add dimension fields: [AWS] Set dimensions fields of Billing, SQS and SNS data streams #6311
    • Verification of the count of documents (before & after TSDB enablement) in Discover Interface
    • Verify if field mapping is correct in the data stream template.
  • Adding metric_type mapping: [AWS] Add metric type to SNS, SQS and Billing data streams #6319
    • Verification of data in visualisation after enabling TSDB flag in kibana
    • Verify if field mapping is correct in the data stream template.

• SQS

  • Add dimension fields: [AWS] Set dimensions fields of Billing, SQS and SNS data streams #6311
    • Verification of the count of documents (before & after TSDB enablement) in Discover Interface
    • Verify if field mapping is correct in the data stream template.
  • Adding metric_type mapping: [AWS] Add metric type to SNS, SQS and Billing data streams #6319
    • Verification of data in visualisation after enabling TSDB flag in kibana
    • Verify if field mapping is correct in the data stream template.

• Transitgateway

  • Add dimension fields: [AWS] Add dimensions to firewall, transit gateway and vpn data streams #6339
    • Verification of the count of documents (before & after TSDB enablement) in Discover Interface
    • Verify if field mapping is correct in the data stream template.
  • Adding metric_type mapping: [AWS] Add metric type to VPN, Firewall and Transit Gateway data streams #6365
    • Verification of data in visualisation after enabling TSDB flag in kibana
    • Verify if field mapping is correct in the data stream template.

• Usage: [AWS] TSDB enablement for usage, ELB and DynamoDB data streams #6227

• VPN

  • Add dimension fields: [AWS] Add dimensions to firewall, transit gateway and vpn data streams #6339
    • Verification of the count of documents (before & after TSDB enablement) in Discover Interface
    • Verify if field mapping is correct in the data stream template.
  • Adding metric_type mapping: [AWS] Add metric type to VPN, Firewall and Transit Gateway data streams #6365
    • Verification of data in visualisation after enabling TSDB flag in kibana
    • Verify if field mapping is correct in the data stream template.

Testing

For every data stream:

• The number of documents is the same before and after enabling TSDB
• The dashboards works before TSDB is enabled
• The dashboard works after TSDB is disabled

Tested:

• API Gateway
• Billing
  • Fix dimensions since documents are being overwritten: [AWS Billing] Add fingerprint dimension #6749
• Cloudwatch
• DynamoDB
• EBS
• EC2
• ECS
• ELB
• EMR
• Firewall
• Kinesis
• Lambda
• Nat Gateway
• RDS
• Redshift
• S3 daily storage
• S3 request
  • issue: [AWS Usage] Overlapping documents when enabling TSDB - no more dimensions available #6783 - not a blocker for TSDB
• S3 storage lens
• SNS
• SQS
• Transit Gateway
• Usage:
  • Issue: [AWS Usage] Overlapping documents when enabling TSDB - no more dimensions available #6783 - not considered a blocker for TSDB enablement.
• VPN

Enable TSDB by default

1. Billing, DynamoDB, EBS, ECS, ELB, Firewall, Kinesis, Lambda, NAT gateway, RDS, Redshift, S3 Storage Lens, SNS, SQS Transit Gateway, VPN: [AWS] Enable TSDB for some metric datasets - Part 1 #6782
2. Usage: [AWS Usage] Enable TSDB by default #6886
3. API Gateway and EMR: [AWS] Enable TSDB by default for API Gateway and EMR data streams #6969
4. Enable in Cloudwatch: [AWS][cloudwatch_metrics] Enable TSDB by default for cloudwatch metrics data stream #7262
5. Enable in EC2 metrics data stream: [AWS][EC2] Enable TSDB by default #7497