🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

All *_metrics data streams use the object mapping.

For example, here is the dimensions definition for EC2 metrics:

# packages/aws/data_stream/ec2_metrics/fields/fields.yml
- name: aws
  type: group
  fields:
    - name: dimensions
      type: group
      fields:
        - name: AutoScalingGroupName
          type: keyword
          dimension: true
          description: An Auto Scaling group is a collection of instances you define if you're using Auto Scaling.
        - name: ImageId
          type: keyword
          dimension: true
          description: This dimension filters the data you request for all instances running this Amazon EC2 Amazon Machine Image (AMI)
        - name: InstanceId
          type: keyword
          dimension: true
          description: Amazon EC2 instance ID
        - name: InstanceType
          type: keyword
          dimension: true
          description: This dimension filters the data you request for all instances running with this specified instance type.

Since ALL metrics data streams use this mapping, it probably makes sense to use the object type in the cloudwatch_metrics as well instead of the flattened type. We can't do the opposite since TSDB doesn't allow dimension: true on flattened type.

Looks good to me. Could you add a section in the description on how to test this locally. A use case that would prove that this uniformity solves the problem of data correlation?

Could you add a section in the description on how to test this locally. A use case that would prove that this uniformity solves the problem of data correlation?

I guess the main issue for users is (quoting @felixbarny) "[they] can’t build visualizations with flattened fields in Lens, because Kibana doesn’t recognize fields within flattened objects."

@MichaelKatsoulis, I updated the "how to test this locally" section accordingly.

I tested the upgrade from AWS integration 2.36.2 to 2.37.0 (the unreleased changes from this PR) with the following steps:

• Started a brand new local stack (8.17.0)
• Installed AWS integration 2.36.2 (with aws.dimensions as flattened)
• Started sending 1 document every 5 secs, including a field containing a sequence number
• Upgraded the AWS integration to 2.37.0 (unreleased, this PR)
• Waited for the rollout to take effect (checking settings.index.time_series | .start_time, .end_time)
• Checked that the the data stream didn't lose any sequence number

More details on selected steps.

Started sending 1 document every 5 secs

I used the following shell script:

sequence=0
while true
do
cat > metrics.json <<EOF
{
  "@timestamp": "$(date '+%Y-%m-%dT%H:%M:%S%z')",
    "aws": {
      "dimensions": {
        "name": "Maurizio Branca",
        "AutoScalingGroupName": "whatever"
      },
      "metric": {
        "cpu": 10,
        "sequence": $sequence
      }
  }
}  
EOF

((sequence++))

cat metrics.json | jq -c | es docs bulk -f - -i metrics-aws.cloudwatch_metrics-sdh5390
sleep 5
done

The scripts sends a document like the following every 5 secs:

{
  "@timestamp": "2024-12-31T00:14:58+0100",
  "aws": {
    "dimensions": {
      "name": "Maurizio Branca",
      "AutoScalingGroupName": "whatever"
    },
    "metric": {
      "cpu": 10,
      "sequence": 270
    }
  }
}

Waited for the rollout to take effect

Right after the upgrade, Fleet/ES creates a new -000002 index, but keeps writing to the -000001 index until the settings.index.time_series.end_time elapses.

Old index -000001:

{
  ".ds-metrics-aws.cloudwatch_metrics-sdh5390-2024.12.30-000001": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "1000",
            "ignore_dynamic_beyond_limit": "true"
          }
        },
        "hidden": "true",
        "time_series": {
          "end_time": "2024-12-30T23:05:37.000Z",
          "start_time": "2024-12-30T20:35:37.000Z"
        },

New index -000002:

{
  ".ds-metrics-aws.cloudwatch_metrics-sdh5390-2024.12.30-000002": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "1000",
            "ignore_dynamic_beyond_limit": "true"
          }
        },
        "hidden": "true",
        "time_series": {
          "end_time": "2024-12-30T23:35:37.000Z",
          "start_time": "2024-12-30T23:05:37.000Z"
        },

Checked that the the data stream didn't lose any sequence number

At 2024-12-30T23:05:37.000Z, ES started to write to the new index without losing a sequence number:

The testing methodology looks great!

Longer-term, I wish we had something similar to this for all integrations as automated upgrade tests. The only thing that should be required in an integration is to provide a templated sample document. The rest could be a generic infrastructure. WDYT @jsoriano?

Longer-term, I wish we had something similar to this for all integrations as automated upgrade tests.

We have an open issue about testing integration upgrades: elastic/elastic-package#1831

The only thing that should be required in an integration is to provide a templated sample document. The rest could be a generic infrastructure.

Some integrations start dummy services with docker compose to generate events. But for this case we'd be missing the validation on the sequence numbers.

@zmoog great approach to test this PR - #11883 (comment)! And thank you for sharing!

@tetianakravchenko, changelog updated!

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
25.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

💚 Build Succeeded

• Buildkite Build
• Commit: 3851a3aeb8ce5d7b31507edc49b7a19b765ea746

History

• 💚 Build #19924 succeeded 0b76887fd19baa7e07c6110d2723f48eef9694a4
• 💔 Build #19921 failed 7da35b0cbe0f3d9cbc4549fd81cf01db1705aae6
• 💚 Build #18721 succeeded 9641abeaa482d161b20f560096b3f04bb78f63d5

cc @zmoog

Package aws - 2.37.0 containing this change is available at https://epr.elastic.co/package/aws/2.37.0/