[Filebeat] Integration tests in CI for AWS-S3 input by francescayeye · Pull Request #27491 · elastic/beats

francescayeye · 2021-08-19T08:34:10Z

Enhancement

What does this PR do?

#27199 introduced new integration tests that rely on terraform (https://github.com/elastic/beats/blob/master/x-pack/filebeat/input/awss3/input_integration_test.go)

We will run them in CI, similarly to what happens for Metricbeat (https://github.com/elastic/beats/blob/master/x-pack/metricbeat/Jenkinsfile.yml#L35-L47)

Why is it important?

Avoiding regression

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes #27469

Use cases

Screenshots

Logs

elasticmachine · 2021-08-19T08:34:31Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-08-19T10:45:11Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-09-06T08:35:52.815+0000
Duration: 178 min 24 sec
Commit: a2f2177

Test stats 🧪

Test	Results
Failed	0
Passed	55706
Skipped	5485
Total	61191

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	55706
Skipped	5485
Total	61191

francescayeye · 2021-08-19T11:51:20Z

there was an error in the directory name and terraform apply failed:

  {
    "displayDescription": "terraform apply -auto-approve",
    "displayName": "Terraform Apply on x-pack/filebeat/input/aws/_meta/terraform",
    "durationInMillis": 308,
    "id": "3743",
    "input": null,
    "result": "FAILURE",
    "startTime": "2021-08-19T09:39:03.038+0000",
    "state": "FINISHED",
    "type": "STAGE",
    "url": "https://beats-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/Beats/pipelines/beats/pipelines/PR-27491/runs/1/steps/3743/log"
  }

https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/pipelines/beats/pipelines/PR-27491/runs/1/steps/3743/log/

anyway the CI status was green

cc @v1v

jsoriano

Nice, added some suggestions.

jsoriano · 2021-08-19T15:43:32Z

x-pack/filebeat/Jenkinsfile.yml

+      withModule: true       ## run the ITs only if the changeset affects a specific module.
+      dirs:                  ## run the cloud tests for the given modules.
+        - "x-pack/filebeat/input/awss3/_meta/terraform" # needed by startCloudTestEnv in order to apply terraform
+        - "x-pack/filebeat/input/awss3" # the actual dir where tests reside


This looks a bit hacky, I would prefer if we don't need to add both directories.

I see some options:

Add a terraformDir config to this cloud step, so we can explicitly configure where is the terraform scenario and where the tests. This would have the advantage of allowing to use the same scenario for tests residing in multiple scenarios.

Add a terraform file where the tests reside, so we don't have to add both directories. This terraform file could use _meta/terraform as a module.

odify startCloudTestEnv to use ./_meta/terraform if there are no terraform files in the

fixes: This version of Terraform has an outdated GPG key and is unable to verify new provider releases

francescayeye · 2021-08-23T12:20:53Z

@v1v

[2021-08-23T10:43:45.606Z] Running in /var/lib/jenkins/workspace/PR-27491-3-e027cfdc-c85f-42af-a21e-a5769d4762dd/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/_meta/terraform
[2021-08-23T10:43:45.607Z] [Pipeline] {
[2021-08-23T10:43:45.628Z] [Pipeline] sh
[2021-08-23T10:43:45.920Z] + terraform init
[2021-08-23T10:43:45.920Z] 
[2021-08-23T10:43:45.920Z] Initializing the backend...
[2021-08-23T10:43:45.920Z] 
[2021-08-23T10:43:45.920Z] Initializing provider plugins...
[2021-08-23T10:43:45.920Z] - Checking for available provider plugins...
[2021-08-23T10:43:45.920Z] - Downloading plugin for provider "random" (hashicorp/random) 3.1.0...
[2021-08-23T10:43:46.496Z] - Downloading plugin for provider "aws" (hashicorp/aws) 3.55.0...
[2021-08-23T10:43:48.420Z] - Downloading plugin for provider "local" (hashicorp/local) 2.1.0...
[2021-08-23T10:43:49.367Z] 
[2021-08-23T10:43:49.367Z] The following providers do not have any version constraints in configuration,
[2021-08-23T10:43:49.367Z] so the latest version was installed.
[2021-08-23T10:43:49.367Z] 
[2021-08-23T10:43:49.367Z] To prevent automatic upgrades to new major versions that may contain breaking
[2021-08-23T10:43:49.367Z] changes, it is recommended to add version = "..." constraints to the
[2021-08-23T10:43:49.367Z] corresponding provider blocks in configuration, with the constraint strings
[2021-08-23T10:43:49.367Z] suggested below.
[2021-08-23T10:43:49.367Z] 
[2021-08-23T10:43:49.367Z] * provider.local: version = "~> 2.1"
[2021-08-23T10:43:49.367Z] * provider.random: version = "~> 3.1"
[2021-08-23T10:43:49.367Z] 
[2021-08-23T10:43:49.367Z] 
[2021-08-23T10:43:49.367Z] Warning: Provider source not supported in Terraform v0.12
[2021-08-23T10:43:49.367Z] 
[2021-08-23T10:43:49.367Z]   on main.tf line 3, in terraform:
[2021-08-23T10:43:49.367Z]    3:     aws = {
[2021-08-23T10:43:49.367Z]    4:       source  = "hashicorp/aws"
[2021-08-23T10:43:49.367Z]    5:       version = "~> 3.52"
[2021-08-23T10:43:49.367Z]    6:     }
[2021-08-23T10:43:49.368Z] 
[2021-08-23T10:43:49.368Z] A source was declared for provider aws. Terraform v0.12 does not support the
[2021-08-23T10:43:49.368Z] provider source attribute. It will be ignored.
[2021-08-23T10:43:49.368Z] 
[2021-08-23T10:43:49.368Z] Terraform has been successfully initialized!
[2021-08-23T10:43:49.368Z] 
[2021-08-23T10:43:49.368Z] You may now begin working with Terraform. Try running "terraform plan" to see
[2021-08-23T10:43:49.368Z] any changes that are required for your infrastructure. All Terraform commands
[2021-08-23T10:43:49.368Z] should now work.
[2021-08-23T10:43:49.368Z] 
[2021-08-23T10:43:49.368Z] If you ever set or change modules or backend configuration for Terraform,
[2021-08-23T10:43:49.368Z] rerun this command to reinitialize your working directory. If you forget, other
[2021-08-23T10:43:49.368Z] commands will detect it and remind you to do so if necessary.
[2021-08-23T10:43:49.378Z] [Pipeline] }
[2021-08-23T10:43:49.389Z] [Pipeline] // dir
[2021-08-23T10:43:49.409Z] [Pipeline] dir
[2021-08-23T10:43:49.409Z] Running in /var/lib/jenkins/workspace/PR-27491-3-e027cfdc-c85f-42af-a21e-a5769d4762dd/src/github.com/elastic/beats/x-pack/filebeat/input/awss3/_meta/terraform
[2021-08-23T10:43:49.410Z] [Pipeline] {
[2021-08-23T10:43:49.424Z] [Pipeline] sh
[2021-08-23T10:43:49.713Z] + terraform apply -auto-approve
[2021-08-23T10:43:53.030Z] 
[2021-08-23T10:43:53.030Z] Warning: Provider source not supported in Terraform v0.12
[2021-08-23T10:43:53.030Z] 
[2021-08-23T10:43:53.030Z]   on main.tf line 3, in terraform:
[2021-08-23T10:43:53.030Z]    3:     aws = {
[2021-08-23T10:43:53.030Z]    4:       source  = "hashicorp/aws"
[2021-08-23T10:43:53.030Z]    5:       version = "~> 3.52"
[2021-08-23T10:43:53.030Z]    6:     }
[2021-08-23T10:43:53.030Z] 
[2021-08-23T10:43:53.030Z] A source was declared for provider aws. Terraform v0.12 does not support the
[2021-08-23T10:43:53.030Z] provider source attribute. It will be ignored.
[2021-08-23T10:43:53.030Z] 
[2021-08-23T10:43:53.030Z] 
[2021-08-23T10:43:53.030Z] Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
[2021-08-23T10:43:53.030Z] 
[2021-08-23T10:43:53.030Z] Please see https://registry.terraform.io/providers/hashicorp/aws
[2021-08-23T10:43:53.030Z] for more information about providing credentials.
[2021-08-23T10:43:53.030Z] 
[2021-08-23T10:43:53.031Z] Error: NoCredentialProviders: no valid providers in chain. Deprecated.
[2021-08-23T10:43:53.031Z] 	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[2021-08-23T10:43:53.031Z] 
[2021-08-23T10:43:53.031Z] 
[2021-08-23T10:43:53.031Z]   on main.tf line 10, in provider "aws":
[2021-08-23T10:43:53.031Z]   10: provider "aws" {
[2021-08-23T10:43:53.031Z] 
[2021-08-23T10:43:53.031Z] 
[2021-08-23T10:43:53.040Z] [Pipeline] }
```

it seems that when cloud test fails at terraform steps the failure is not reported

v1v · 2021-08-23T13:30:31Z

Jenkinsfile

            // If it failed then cleanup without failing the build
            sh(label: 'Terraform Cleanup', script: ".ci/scripts/terraform-cleanup.sh ${folder}", returnStatus: true)
          }
+          error('startCloudTestEnv: terraform apply failed.')


Enforce the error is reported after the catch closure

…3-input

francescayeye · 2021-08-23T13:38:10Z

5bf9090

not sure it is correct: I can see that startCloudTestEnv is executed on dirs and target on directory, but I don't see directory defined in the reference from metricbeat: https://github.com/elastic/beats/blob/master/x-pack/metricbeat/Jenkinsfile.yml#L35-L47

if I can define dirs for terraform operations and directory for where to run the tests from then this change is not needed

v1v · 2021-08-24T14:49:22Z

@aspacca , we can syncup if needed, please poke me. 5bf9090 is half done, there are a couple of changes to be done:

terraformDir has not been defined in https://github.com/elastic/beats/blob/master/x-pack/metricbeat/Jenkinsfile.yml#L35-L47
the cloud step call in the pipeline needs to define terraformDir in https://github.com/elastic/beats/blob/master/Jenkinsfile#L1100

…3-input

Jenkinsfile

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

francescayeye · 2021-09-02T13:43:52Z

@v1v everything seems fine: we don't need to introduce terraformDir.

As far as I can see only problem left is that the step is triggered when github label aws is set while AWS credentials are setup only when the build is triggered by Jenkins.

It would be valuable in my opinion to add something like || labels.aws here https://github.com/elastic/beats/blob/master/Jenkinsfile#L842 (if possible).
Not an expert on groovy and I'm not sure how to add an extra param to pass to withCloudTestEnv that's already accepting a closure.

See https://github.com/elastic/apm-pipeline-library/blob/master/vars/matchesPrLabel.groovy

v1v · 2021-09-02T14:44:07Z

Jenkinsfile

+  // Allow AWS credentials when the build was configured to do so with:
+  //   - the cloudtests build parameters
+  //   - the aws github label
+  if (params.allCloudTests || params.awsCloudTests || matchesPrLabel(label: 'aws')) {


This should help to configure the AWS credentials when the aws github label was assigned to the Pull Request as explained in #27491 (comment)

v1v · 2021-09-03T07:55:02Z

There are some failures when applying the terraform, see the below screenshot

As far as I see it's related to this terraform resource:

beats/x-pack/metricbeat/module/aws/terraform.tf

Line 50 in 825bfb2

resource "aws_instance" "test" {

No idea whether it's a temporary glitch or something else

Let me rerun the build again

v1v · 2021-09-03T07:55:22Z

/test

francescayeye · 2021-09-03T09:58:05Z

/test

francescayeye · 2021-09-03T09:58:16Z

~~it seems it was temporary, now we have another one~~

francescayeye · 2021-09-06T08:35:23Z

/test

francescayeye · 2021-09-06T14:34:28Z

@v1v all good, let's merge?

v1v

Thanks @aspacca for making this happen!! 💯

* [Filebeat] Integration tests in CI for AWS-S3 input (cherry picked from commit 9d4597a)

* [Filebeat] Integration tests in CI for AWS-S3 input (cherry picked from commit 9d4597a) Co-authored-by: Andrea Spacca <andrea.spacca@elastic.co>

* master: (39 commits) [Heartbeat] Move JSON tests from python->go (elastic#27816) docs: simplify permissions for Dockerfile COPY (elastic#27754) Osquerybeat: Fix osquery logger plugin severy levels mapping (elastic#27789) [Filebeat] Update compatibility function to remove processor description on ES < 7.9.0 (elastic#27774) warn log entry and no validation failure when both queue_url and buck… (elastic#27612) libbeat/cmd/instance: ensure test config file has appropriate permissions (elastic#27178) [Heartbeat] Add httpcommon options to ZipURL (elastic#27699) Add a header round tripper option to httpcommon (elastic#27509) [Elastic Agent] Add validation to ensure certificate paths are absolute. (elastic#27779) Rename dashboards according to module.yml files for master (elastic#27749) Refactor vagrantfile, add scripts for provisioning with docker/kind (elastic#27726) Accept syslog dates with leading 0 (elastic#27775) [Filebeat] Add timezone config option to decode_cef and syslog input (elastic#27727) [Filebeat] Threatintel compatibility updates (elastic#27323) Add support for ephemeral containers in elastic agent dynamic provider (elastic#27707) [Filebeat] Integration tests in CI for AWS-S3 input (elastic#27491) Fix flakyness of TestFilestreamEmptyLine (elastic#27705) [Filebeat] kafka v2 using parsers (elastic#27335) Update Kafka version parsing / supported range (elastic#27720) Update Sarama to 1.29.1 (elastic#27717) ...

* [Filebeat] Integration tests in CI for AWS-S3 input

[Filebeat] Integration tests in CI for AWS-S3 input

8cd6025

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 19, 2021

francescayeye added the Team:Integrations Label for the Integrations team label Aug 19, 2021

francescayeye requested a review from jsoriano August 19, 2021 08:34

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 19, 2021

francescayeye requested a review from v1v August 19, 2021 08:34

francescayeye added the aws Enable builds in the CI for aws cloud testing label Aug 19, 2021

fix directory

68c009f

jsoriano reviewed Aug 19, 2021

View reviewed changes

Bump terraform version

289c85f

fixes: This version of Terraform has an outdated GPG key and is unable to verify new provider releases

francescayeye requested a review from a team as a code owner August 23, 2021 09:31

botelastic bot added the Team:Automation Label for the Observability productivity team label Aug 23, 2021

Fail terraform apply if any errors after retry

dab3a44

v1v reviewed Aug 23, 2021

View reviewed changes

Merge branch 'master' into Filebeat-Integration-tests-in-CI-for-AWS-S…

dc13f91

…3-input

Andrea Spacca added 2 commits August 25, 2021 15:39

Merge branch 'master' into Filebeat-Integration-tests-in-CI-for-AWS-S…

fc67dab

…3-input

rely on dirs

c1859de

cachedout approved these changes Aug 26, 2021

View reviewed changes

v1v reviewed Aug 26, 2021

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

Update Jenkinsfile

f27d69e

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

Configure aws credentials on a PR basis if 'aws' has been assigned

5348c48

See https://github.com/elastic/apm-pipeline-library/blob/master/vars/matchesPrLabel.groovy

v1v reviewed Sep 2, 2021

View reviewed changes

v1v mentioned this pull request Sep 3, 2021

CI: notify GH check status in cloud stages #27734

Merged

use t2.micro

a2f2177

francescayeye added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify labels Sep 6, 2021

v1v approved these changes Sep 6, 2021

View reviewed changes

francescayeye merged commit 9d4597a into elastic:master Sep 6, 2021

mergify bot pushed a commit that referenced this pull request Sep 6, 2021

[Filebeat] Integration tests in CI for AWS-S3 input (#27491)

c86ba3b

* [Filebeat] Integration tests in CI for AWS-S3 input (cherry picked from commit 9d4597a)

mergify bot mentioned this pull request Sep 6, 2021

[7.x](backport #27491) [Filebeat] Integration tests in CI for AWS-S3 input #27765

Merged

mergify bot pushed a commit that referenced this pull request Sep 6, 2021

[Filebeat] Integration tests in CI for AWS-S3 input (#27491)

b76859f

* [Filebeat] Integration tests in CI for AWS-S3 input (cherry picked from commit 9d4597a)

mergify bot mentioned this pull request Sep 6, 2021

[7.15](backport #27491) [Filebeat] Integration tests in CI for AWS-S3 input #27766

Merged

Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021

[Filebeat] Integration tests in CI for AWS-S3 input (elastic#27491)

cc7ffbf

* [Filebeat] Integration tests in CI for AWS-S3 input

Conversation

francescayeye commented Aug 19, 2021

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

Uh oh!

elasticmachine commented Aug 19, 2021

Uh oh!

elasticmachine commented Aug 19, 2021 • edited by jenkins-beats-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

francescayeye commented Aug 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

jsoriano Aug 19, 2021

Choose a reason for hiding this comment

Uh oh!

francescayeye commented Aug 23, 2021

Uh oh!

v1v Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

francescayeye commented Aug 23, 2021

Uh oh!

v1v commented Aug 24, 2021

Uh oh!

Uh oh!

francescayeye commented Sep 2, 2021

Uh oh!

v1v Sep 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

v1v commented Sep 3, 2021

Uh oh!

v1v commented Sep 3, 2021

Uh oh!

francescayeye commented Sep 3, 2021

Uh oh!

francescayeye commented Sep 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

francescayeye commented Sep 6, 2021

Uh oh!

francescayeye commented Sep 6, 2021

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

elasticmachine commented Aug 19, 2021 •

edited by jenkins-beats-ci bot

Loading

francescayeye commented Aug 19, 2021 •

edited

Loading

v1v Sep 2, 2021 •

edited

Loading

francescayeye commented Sep 3, 2021 •

edited

Loading