Skip to content
This repository was archived by the owner on Sep 17, 2024. It is now read-only.

chore: reduce AWS instances size#2477

Merged
mdelapenya merged 8 commits intoelastic:mainfrom
mdelapenya:adjust-aws-costs
May 6, 2022
Merged

chore: reduce AWS instances size#2477
mdelapenya merged 8 commits intoelastic:mainfrom
mdelapenya:adjust-aws-costs

Conversation

@mdelapenya
Copy link
Copy Markdown
Contributor

@mdelapenya mdelapenya commented May 5, 2022 •
edited
Loading

  • chore: reduce size for c5.4xlarge AWS instances
  • chore: reduce size for c5.2xlarge AWS instances
  • chore: reduce stack instance too

What does this PR do?

This PR reduces the size of the EC2 instances created in this pipeline:

c5.4xlarge -> t3.xlarge
c5.2xlarge -> t3.xlarge
a1.large = a1.large

More info on AWS instances: https://aws.amazon.com/ec2/instance-types/

Why is it important?

We want to reduce the Cloud costs, given the usage of the tests is increasing

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have run the Unit tests (make unit-test), and they are passing locally
  • I have run the End-2-End tests for the suite I'm working on, and they are passing locally
  • I have noticed new Go dependencies (run make notice in the proper directory)

@mdelapenya mdelapenya added Team:Automation Label for the Observability productivity team area:ci Anything related to the CI backport-v7.17.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify backport-v8.2.0 Automated backport with mergify labels May 5, 2022
@mdelapenya mdelapenya self-assigned this May 5, 2022
@mdelapenya mdelapenya requested a review from a team May 5, 2022 10:38
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented May 5, 2022 •
edited
Loading

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-05-06T11:41:21.396+0000

  • Duration: 92 min 1 sec

Test stats 🧪

Test Results
Failed 2
Passed 304
Skipped 0
Total 306

Test errors 2

Expand to view the tests failures

Initializing / End-To-End Tests / fleet_debian_amd64_apm_server / Deploying a default stand-alone agent with the Elastic APM integration – APM Integration
    Expand to view the error details

     Step the stand-alone agent is listed in Fleet as "online"

  • no stacktrace
Initializing / End-To-End Tests / fleet_debian_amd64_apm_server / Deploying a ubi8 stand-alone agent with the Elastic APM integration – APM Integration
    Expand to view the error details

     Step the stand-alone agent is listed in Fleet as "online"

  • no stacktrace

Steps errors 12

Expand to view the steps failures

Show only the first 10 steps failures

Shell Script
  • Took 3 min 16 sec . View more details here
  • Description: ssh -tt -o TCPKeepAlive=yes -o ServerAliveInterval=60 -o ServerAliveCountMax=10 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /var/lib/jenkins/workspace/PR-2477-8-ce6a1c5f-471e-436f-991b-35957f3b3d41/e2essh admin@3.141.97.225 -- 'sudo bash /home/admin/e2e-testing/.ci/scripts/functional-test.sh'
Archive the artifacts
  • Took 0 min 1 sec . View more details here
  • Description: [2022-05-06T12:27:03.247Z] Archiving artifacts script returned exit code 2
Docker login
  • Took 0 min 1 sec . View more details here
  • Description: set +x if command -v host 2>&1 > /dev/null; then host docker.elastic.co 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.elastic.co 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.elastic.co" 2>/dev/null
Docker login
  • Took 0 min 15 sec . View more details here
  • Description: set +x if command -v host 2>&1 > /dev/null; then host docker.elastic.co 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.elastic.co 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.elastic.co" 2>/dev/null
Executing ansible-playbook
  • Took 0 min 18 sec . View more details here
  • Description: /var/lib/jenkins/workspace/PR-2477-8-a70b7325-e3cd-4618-a0f6-09c2af58c0ce/.venv/bin/ansible-playbook --private-key="/var/lib/jenkins/workspace/PR-2477-8-a70b7325-e3cd-4618-a0f6-09c2af58c0ce/e2essh" --extra-vars="workspace=/var/lib/jenkins/workspace/PR-2477-8-a70b7325-e3cd-4618-a0f6-09c2af58c0ce/ runId=d739b604 sshPublicKey=/var/lib/jenkins/workspace/PR-2477-8-a70b7325-e3cd-4618-a0f6-09c2af58c0ce/e2essh.pub" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' /var/lib/jenkins/workspace/PR-2477-8-a70b7325-e3cd-4618-a0f6-09c2af58c0ce/.ci/ansible/playbook.yml -i "3.16.109.78," -t setup-node --extra-vars="buildURL=https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/PR-2477/8/ gitSha=ec09131ba7c905b83e052e5cabb9b8d373415918 build=8 repo=e2e-testing branch=pr-2477 type=ci stackRunner=52.14.89.186 nodeUser=ec2-user suite=fleet nodeLabel=sles15 nodeImage=ami-0f7cb53c916a75006 nodeInstanceType=t3.xlarge"
Sleep
  • Took 0 min 5 sec . View more details here
  • Description: 5
generate-cucumber-reports.sh
  • Took 0 min 0 sec . View more details here
  • Description: .ci/scripts/generate-cucumber-reports.sh
Executing ansible-playbook
  • Took 0 min 18 sec . View more details here
  • Description: /var/lib/jenkins/workspace/PR-2477-8-451a9957-e451-4d56-846b-4386b81f6d06/.venv/bin/ansible-playbook --private-key="/var/lib/jenkins/workspace/PR-2477-8-451a9957-e451-4d56-846b-4386b81f6d06/e2essh" --extra-vars="workspace=/var/lib/jenkins/workspace/PR-2477-8-451a9957-e451-4d56-846b-4386b81f6d06/ runId=c7305044 sshPublicKey=/var/lib/jenkins/workspace/PR-2477-8-451a9957-e451-4d56-846b-4386b81f6d06/e2essh.pub" --ssh-common-args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' /var/lib/jenkins/workspace/PR-2477-8-451a9957-e451-4d56-846b-4386b81f6d06/.ci/ansible/playbook.yml -i "18.188.225.176," -t setup-node --extra-vars="buildURL=https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/PR-2477/8/ gitSha=ec09131ba7c905b83e052e5cabb9b8d373415918 build=8 repo=e2e-testing branch=pr-2477 type=ci stackRunner=52.14.89.186 nodeUser=ec2-user suite=fleet nodeLabel=sles15 nodeImage=ami-0f7cb53c916a75006 nodeInstanceType=t3.xlarge"
Sleep
  • Took 0 min 5 sec . View more details here
  • Description: 5
Docker login
  • Took 0 min 18 sec . View more details here
  • Description: set +x if command -v host 2>&1 > /dev/null; then host docker.elastic.co 2>&1 > /dev/null fi if command -v dig 2>&1 > /dev/null; then dig docker.elastic.co 2>&1 > /dev/null fi docker login -u "${DOCKER_USER}" -p "${DOCKER_PASSWORD}" "docker.elastic.co" 2>/dev/null

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors 2

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Initializing / End-To-End Tests / fleet_debian_amd64_apm_server / Deploying a default stand-alone agent with the Elastic APM integration – APM Integration
  • Name: Initializing / End-To-End Tests / fleet_debian_amd64_apm_server / Deploying a ubi8 stand-alone agent with the Elastic APM integration – APM Integration

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

v1v
v1v reviewed May 6, 2022
View reviewed changes
.ci/.e2e-platforms.yaml
description: "Debian 10 AMD64, used specifically for running the stack"
image: "ami-0d90bed76900e679a"
instance_type: "c5.4xlarge"
instance_type: "t3.xlarge"
Copy link
Copy Markdown
Member

@v1v v1v May 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About 4x times cheaper: $0.1670 vs $0.68

@mdelapenya
Copy link
Copy Markdown
Contributor Author

mdelapenya commented May 6, 2022

For some reason fleet-server is not starting. Hopefully #2486 will mitigate that error and retry the setup

@mdelapenya mdelapenya merged commit 70fba3f into elastic:main May 6, 2022
mergify bot pushed a commit that referenced this pull request May 6, 2022
@mdelapenya @mergify
chore: reduce AWS instances size (#2477)
b08d49a 
* chore: reduce size for c5.4xlarge AWS instances

* chore: reduce size for c5.2xlarge AWS instances

* chore: reduce stack instance too

* fix: do not use ARM-based images

* chore: update automation for local development

(cherry picked from commit 70fba3f)
@mergify mergify bot mentioned this pull request May 6, 2022
Merged
mergify bot pushed a commit that referenced this pull request May 6, 2022
@mdelapenya @mergify
chore: reduce AWS instances size (#2477)
dc367e9 
* chore: reduce size for c5.4xlarge AWS instances

* chore: reduce size for c5.2xlarge AWS instances

* chore: reduce stack instance too

* fix: do not use ARM-based images

* chore: update automation for local development

(cherry picked from commit 70fba3f)

# Conflicts:
#	.ci/Makefile
@mergify mergify bot mentioned this pull request May 6, 2022
Merged
mergify bot pushed a commit that referenced this pull request May 6, 2022
@mdelapenya @mergify
chore: reduce AWS instances size (#2477)
5af11ac 
* chore: reduce size for c5.4xlarge AWS instances

* chore: reduce size for c5.2xlarge AWS instances

* chore: reduce stack instance too

* fix: do not use ARM-based images

* chore: update automation for local development

(cherry picked from commit 70fba3f)
@mergify mergify bot mentioned this pull request May 6, 2022
Merged
mdelapenya added a commit that referenced this pull request May 6, 2022
@mergify @mdelapenya
chore: reduce AWS instances size (#2477) (#2491)
f0d01c1 
* chore: reduce size for c5.4xlarge AWS instances

* chore: reduce size for c5.2xlarge AWS instances

* chore: reduce stack instance too

* fix: do not use ARM-based images

* chore: update automation for local development

(cherry picked from commit 70fba3f)

Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
mdelapenya added a commit that referenced this pull request May 6, 2022
@mergify @mdelapenya
chore: reduce AWS instances size (#2477) (#2493)
6b835bd 
* chore: reduce size for c5.4xlarge AWS instances

* chore: reduce size for c5.2xlarge AWS instances

* chore: reduce stack instance too

* fix: do not use ARM-based images

* chore: update automation for local development

(cherry picked from commit 70fba3f)

Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
mdelapenya added a commit that referenced this pull request May 9, 2022
@mergify @mdelapenya
chore: reduce AWS instances size (#2477) (#2492)
49a7fc9 
* chore: reduce size for c5.4xlarge AWS instances

* chore: reduce size for c5.2xlarge AWS instances

* chore: reduce stack instance too

* fix: do not use ARM-based images

* chore: update automation for local development

(cherry picked from commit 70fba3f)

# Conflicts:
#	.ci/Makefile

Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
@mdelapenya mdelapenya deleted the adjust-aws-costs branch May 9, 2022 10:25
mdelapenya added a commit to mdelapenya/e2e-testing that referenced this pull request May 9, 2022
@mdelapenya
Merge branch 'main' into unify-local-ci-dev-experience
ec4a52a 
* main:
  bump stack version 8.3.0-5c1ff35f (elastic#2494)
  chore: reduce AWS instances size (elastic#2477)
  chore: wait for the stack while provisioning (elastic#2486)
@mdelapenya mdelapenya mentioned this pull request May 10, 2022
Merged
8 tasks
mdelapenya added a commit to mdelapenya/e2e-testing that referenced this pull request May 22, 2022
@mdelapenya
Merge branch 'main' into 2338-bring-back-windows-support
7e07041 
* main:
  bump stack version 8.3.0-8ee1196f (elastic#2519)
  ci: daily pipeline to use main branch (elastic#2516)
  bump stack version 8.3.0-53513548 (elastic#2513)
  chore: run reaper daily at 1:30 am (elastic#2512)
  avoid aliases if no major.minor format (elastic#2501)
  bump stack version 8.3.0-d380914f (elastic#2505)
  chore: refactor cucumber tags (elastic#2496)
  chore: skip system-integration scenarios for Beats (elastic#2482)
  bump stack version 8.3.0-5c1ff35f (elastic#2494)
  chore: reduce AWS instances size (elastic#2477)
  chore: wait for the stack while provisioning (elastic#2486)
  bump stack version 8.3.0-6ba9f710 (elastic#2484)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

2 more reviewers

@v1v v1v v1v approved these changes

@kuisathaverat kuisathaverat kuisathaverat approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@mdelapenya mdelapenya

Labels

area:ci Anything related to the CI backport-v7.17.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify backport-v8.2.0 Automated backport with mergify Team:Automation Label for the Observability productivity team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mdelapenya @elasticmachine @v1v @kuisathaverat