🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

I tested the upgrade from AWS integration 2.30.2 to 2.30.3 (the unreleased changes from this PR) with the following steps:

• Started a brand new local stack (8.15.4)
• Installed AWS integration 2.30.2 (with aws.dimensions as flattened)
• Started sending 1 document every 5 secs, including a field containing a sequence number
• Upgraded the AWS integration to 2.30.3 (unreleased, this PR)
• Waited for the rollout to take effect (checking settings.index.time_series | .start_time, .end_time)
• Checked that the the data stream didn't lose any sequence number

More details on selected steps.

Started sending 1 document every 5 secs

• create an API key on the local stack
• export env vars for the es tool.

Set up the es tool config:

export ELASTICSEARCH_ENDPOINTS="https://localhost:9200"
export ELASTICSEARCH_API_KEY="[redacted, event if it is not need since it's local]"

I used the following shell script:

sequence=0
while true
do
cat > metrics.json <<EOF
{
  "@timestamp": "$(date '+%Y-%m-%dT%H:%M:%S%z')",
    "aws": {
      "dimensions": {
        "name": "Maurizio Branca",
        "AutoScalingGroupName": "whatever"
      },
      "metric": {
        "cpu": 10,
        "sequence": $sequence
      }
  }
}  
EOF

((sequence++))

cat metrics.json | jq -c | es docs bulk -f - -i metrics-aws.cloudwatch_metrics-sdh5390
sleep 5
done

Results in:

2025/01/07 12:54:11 adding a new document: {"@timestamp":"2025-01-07T12:54:11+0100","aws":{"dimensions":{"name":"Maurizio Branca","AutoScalingGroupName":"whatever"},"metric":{"cpu":10,"sequence":0}}}
2025/01/07 12:54:11 closing bulk indexer
2025/01/07 12:54:11 Successfully indexed document 
2025/01/07 12:54:11 bulk indexer closed
2025/01/07 12:54:11 getting bulk indexer stats
2025/01/07 12:54:11 Stats: {NumAdded:1 NumFlushed:1 NumFailed:0 NumIndexed:0 NumCreated:1 NumUpdated:0 NumDeleted:0 NumRequests:1}

The scripts sends a document like the following every 5 secs:

{
  "@timestamp": "2024-12-31T00:14:58+0100",
  "aws": {
    "dimensions": {
      "name": "Maurizio Branca",
      "AutoScalingGroupName": "whatever"
    },
    "metric": {
      "cpu": 10,
      "sequence": 270
    }
  }
}

Upgraded the AWS integration to 2.30.3 (unreleased, this PR)

Upgrade the AWS integration package from 2.30.2 to 2.30.3.

Waited for the rollout to take effect

Right after the upgrade, Fleet/ES creates a new -000002 index, but keeps writing to the -000001 index until the settings.index.time_series.end_time elapses.

Old index -000001:

// GET metrics-aws.cloudwatch_metrics-sdh5390/_settings
{
  ".ds-metrics-aws.cloudwatch_metrics-sdh5390-2025.01.07-000001": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "1000",
            "ignore_dynamic_beyond_limit": "true"
          }
        },
        "hidden": "true",
        "time_series": {
          "end_time": "2025-01-07T12:33:16.000Z",
          "start_time": "2025-01-07T09:54:11.000Z"
        },

New index -000002:

// GET metrics-aws.cloudwatch_metrics-sdh5390/_settings
{
  ".ds-metrics-aws.cloudwatch_metrics-sdh5390-2025.01.07-000002": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "1000",
            "ignore_dynamic_beyond_limit": "true"
          }
        },
        "hidden": "true",
        "time_series": {
          "end_time": "2025-01-07T13:03:16.000Z",
          "start_time": "2025-01-07T12:33:16.000Z"
        },

Now I need to wait until 2025-01-07T12:33:16.000Z (UTC) to see if the ES smoothly transitions from index -000001 to -000002.

Checked that the the data stream didn't lose any sequence number

At 2025-01-07T12:33:16.000Z, ES successfully transitioned from index -000001 to -000002 and from flattened to object field mapping.

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
20.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

💚 Build Succeeded

• Buildkite Build
• Commit: 235d140

History

• 💔 Build #20092 failed 10f269f
• 💚 Build #20076 succeeded 63ae257

cc @zmoog

Package aws - 2.30.3 containing this change is available at https://epr.elastic.co/package/aws/2.30.3/