Skip to content

Fix Google Cloud Function configuration file issues#22156

Merged
kvch merged 2 commits intoelastic:masterfrom
kvch:fix-functionbeat-gcp-config-file
Oct 26, 2020
Merged

Fix Google Cloud Function configuration file issues#22156
kvch merged 2 commits intoelastic:masterfrom
kvch:fix-functionbeat-gcp-config-file

Conversation

@kvch
Copy link
Copy Markdown
Contributor

@kvch kvch commented Oct 26, 2020 •
edited
Loading

What does this PR do?

This PR adds a new function to to cfgfile to set the path to the configuration file of a Beat. This fixes the issue on GCP with Functionbeat.

Why is it important?

ATM Functionbeat cannot run on GCP.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
    - [ ] I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes #20864

@kvch kvch added needs_backport PR is waiting to be backported to other branches. Team:Services (Deprecated) Label for the former Integrations-Services team labels Oct 26, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 26, 2020

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 26, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 26, 2020

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Pull request #22156 opened]

  • Reason: Aborted from #2

  • Start Time: 2020-10-26T16:01:49.855+0000

  • Duration: 4 min 25 sec

  • Commit: 6d38569

Steps errors 2

Expand to view the steps failures

  • Name: Error signal

    • Description: tar: step failled with error null

    • Duration: 0 min 0 sec

    • Start Time: 2020-10-26T16:06:10.050+0000

    • log

  • Name: General Build Step

    • Description:

    • Duration: 0 min 0 sec

    • Start Time: 2020-10-26T16:06:12.261+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-10-26T16:04:05.543Z]  > git clean -fdx # timeout=10
[2020-10-26T16:04:05.559Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2020-10-26T16:04:05.559Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2020-10-26T16:04:05.562Z]  > git fetch --no-tags --progress --prune -- git@github.com:elastic/beats.git +refs/pull/22156/head:refs/remotes/origin/PR-22156 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2020-10-26T16:04:06.664Z] Merging remotes/origin/master commit d36a5106da49ccce2ae1d4a5f53397e73ea8417c into PR head commit 6d38569c326bb3aa69f94fe186c1f1c70ab08550
[2020-10-26T16:04:06.652Z]  > git config core.sparsecheckout # timeout=10
[2020-10-26T16:04:06.656Z]  > git checkout -f 6d38569c326bb3aa69f94fe186c1f1c70ab08550 # timeout=15
[2020-10-26T16:04:08.443Z] Merge succeeded, producing a6d43d5e27d85f69afa5f87efb38d7ae9dd2751e
[2020-10-26T16:04:08.443Z] Checking out Revision a6d43d5e27d85f69afa5f87efb38d7ae9dd2751e (PR-22156)
[2020-10-26T16:04:08.658Z] Commit message: "Merge commit 'd36a5106da49ccce2ae1d4a5f53397e73ea8417c' into HEAD"
[2020-10-26T16:04:08.658Z] First time build. Skipping changelog.
[2020-10-26T16:04:08.658Z] Cleaning workspace
[2020-10-26T16:04:08.913Z]  > git --version # timeout=10
[2020-10-26T16:04:08.925Z]  > git --version # 'git version 2.17.1'
[2020-10-26T16:04:08.933Z] fatal: bad object e7f6515e61e4f984ab47003862b6b71fc101ea31
[2020-10-26T16:04:09.947Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2020-10-26T16:04:08.115Z]  > git remote # timeout=10
[2020-10-26T16:04:08.119Z]  > git config --get remote.origin.url # timeout=10
[2020-10-26T16:04:08.123Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2020-10-26T16:04:08.127Z]  > git merge d36a5106da49ccce2ae1d4a5f53397e73ea8417c # timeout=10
[2020-10-26T16:04:08.423Z]  > git rev-parse HEAD^{commit} # timeout=10
[2020-10-26T16:04:08.431Z]  > git config core.sparsecheckout # timeout=10
[2020-10-26T16:04:08.439Z]  > git checkout -f a6d43d5e27d85f69afa5f87efb38d7ae9dd2751e # timeout=15
[2020-10-26T16:04:08.645Z]  > git rev-parse --verify HEAD # timeout=10
[2020-10-26T16:04:08.657Z] Resetting working tree
[2020-10-26T16:04:08.658Z]  > git reset --hard # timeout=10
[2020-10-26T16:04:08.839Z]  > git clean -fdx # timeout=10
[2020-10-26T16:04:10.655Z] + git fetch https://****:****@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2020-10-26T16:05:18.472Z] Archiving artifacts
[2020-10-26T16:05:19.221Z] + git rev-parse HEAD
[2020-10-26T16:05:19.590Z] + git rev-parse HEAD
[2020-10-26T16:05:19.906Z] + git rev-parse origin/pr/22156
[2020-10-26T16:05:19.954Z] [INFO] githubEnv: Found Git Build Cause: pr
[2020-10-26T16:05:20.142Z] Masking supported pattern matches of $GITHUB_TOKEN
[2020-10-26T16:05:20.950Z] [WARN] githubApiCall: The REST API call https://api.github.com/repos/elastic/beats/pulls/22156/reviews return 0 elements
[2020-10-26T16:05:20.999Z] [INFO] githubPrCheckApproved: Title: Fix Google Cloud Function configuration file issues - User: kvch - Author Association: MEMBER
[2020-10-26T16:05:22.418Z] [INFO] stashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-10-26T16:05:22.479Z] [INFO] stashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-10-26T16:05:22.875Z] + tar --version
[2020-10-26T16:05:23.281Z] + tar --exclude=source.tgz -czf source.tgz .
[2020-10-26T16:06:07.587Z] Sending interrupt signal to process
[2020-10-26T16:06:10.013Z] Terminated
[2020-10-26T16:06:10.013Z] tar: source.tgz: Wrote only 4096 of 10240 bytes
[2020-10-26T16:06:10.013Z] tar: Child returned status 143
[2020-10-26T16:06:10.013Z] tar: Error is not recoverable: exiting now
[2020-10-26T16:06:10.022Z] script returned exit code 2
[2020-10-26T16:06:10.045Z] [INFO] source.tgz was not compressed or archived : null
[2020-10-26T16:06:10.144Z] Stage "Lint" skipped due to earlier failure(s)
[2020-10-26T16:06:10.223Z] Stage "Build&Test" skipped due to earlier failure(s)
[2020-10-26T16:06:12.229Z] [INFO] unstashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2020-10-26T16:06:12.254Z] [INFO] unstashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2020-10-26T16:06:12.350Z] Error when executing always post condition:
[2020-10-26T16:06:12.356Z] com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
[2020-10-26T16:06:12.356Z] {
[2020-10-26T16:06:12.356Z]   "code" : 404,
[2020-10-26T16:06:12.356Z]   "errors" : [ {
[2020-10-26T16:06:12.356Z]     "domain" : "global",
[2020-10-26T16:06:12.356Z]     "message" : "No such object: beats-ci-temp/Beats/beats/PR-22156-1/source/source.tgz",
[2020-10-26T16:06:12.356Z]     "reason" : "notFound"
[2020-10-26T16:06:12.356Z]   } ],
[2020-10-26T16:06:12.356Z]   "message" : "No such object: beats-ci-temp/Beats/beats/PR-22156-1/source/source.tgz"
[2020-10-26T16:06:12.356Z] }
[2020-10-26T16:06:12.356Z] 	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
[2020-10-26T16:06:12.356Z] 	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
[2020-10-26T16:06:12.356Z] 	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
[2020-10-26T16:06:12.356Z] 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
[2020-10-26T16:06:12.356Z] 	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1056)
[2020-10-26T16:06:12.356Z] 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
[2020-10-26T16:06:12.356Z] 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
[2020-10-26T16:06:12.356Z] 	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
[2020-10-26T16:06:12.356Z] 	at com.google.jenkins.plugins.util.RequestCallable$1.call(RequestCallable.java:46)
[2020-10-26T16:06:12.356Z] 	at com.google.jenkins.plugins.util.Executor$Default.execute(Executor.java:125)
[2020-10-26T16:06:12.356Z] Caused: com.google.jenkins.plugins.util.NotFoundException
[2020-10-26T16:06:12.356Z] 	at com.google.jenkins.plugins.util.Executor$Default.execute(Executor.java:132)
[2020-10-26T16:06:12.356Z] 	at com.google.jenkins.plugins.util.Executor.execute(Executor.java:49)
[2020-10-26T16:06:12.356Z] 	at com.google.jenkins.plugins.storage.DownloadStep.resolveBucketPath(DownloadStep.java:419)
[2020-10-26T16:06:12.356Z] 	at com.google.jenkins.plugins.storage.DownloadStep.perform(DownloadStep.java:210)
[2020-10-26T16:06:12.356Z] Caused: java.io.IOException: Exception while performing download
[2020-10-26T16:06:12.356Z] 	at com.google.jenkins.plugins.storage.DownloadStep.perform(DownloadStep.java:223)
[2020-10-26T16:06:12.356Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-10-26T16:06:12.356Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-10-26T16:06:12.356Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-10-26T16:06:12.356Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-10-26T16:06:12.356Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-10-26T16:06:12.356Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-10-26T16:06:12.356Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-10-26T16:06:12.356Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-10-26T16:06:12.356Z] 
[2020-10-26T16:06:13.518Z] Running on worker-1244230 in /var/lib/jenkins/workspace/Beats_beats_PR-22156
[2020-10-26T16:06:13.630Z] [INFO] getVaultSecret: Getting secrets
[2020-10-26T16:06:13.706Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-10-26T16:06:15.775Z] + chmod 755 generate-build-data.sh
[2020-10-26T16:06:15.775Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22156/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22156/runs/1 ABORTED 264528
[2020-10-26T16:06:15.775Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22156/runs/1/steps/?limit=10000 -o steps-info.json
[2020-10-26T16:06:18.025Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22156/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-10-26T16:06:18.721Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2020-10-26T16:06:20.165Z] Retry 2/3 exited 22, retrying in 2 seconds...
[2020-10-26T16:06:22.414Z] Retry 3/3 exited 22, no more retries left.
[2020-10-26T16:06:22.414Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22156/runs/1/log/ -o pipeline-log.txt
[2020-10-26T16:06:22.580Z] Click here to forcibly terminate running steps

michalpristas
michalpristas approved these changes Oct 26, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@michalpristas michalpristas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok. any tests related to this?

@kvch kvch merged commit 13a195a into elastic:master Oct 26, 2020
@kvch kvch mentioned this pull request Oct 26, 2020
Merged
3 tasks
@kvch kvch removed the needs_backport PR is waiting to be backported to other branches. label Oct 26, 2020
kvch added a commit to kvch/beats that referenced this pull request Oct 26, 2020
@kvch
Fix Google Cloud Function configuration file issues (elastic#22156)
ede4fc8 
## What does this PR do?

This PR adds a new function to to `cfgfile` to set the path to the configuration file of a Beat. This fixes the issue on GCP with Functionbeat.

## Why is it important?

ATM Functionbeat cannot run on GCP.

## Related issues

Closes elastic#20864

(cherry picked from commit 13a195a)
@kvch kvch added the v7.11.0 label Oct 26, 2020
@kvch kvch mentioned this pull request Oct 26, 2020
Merged
3 tasks
kvch added a commit to kvch/beats that referenced this pull request Oct 26, 2020
@kvch
Fix Google Cloud Function configuration file issues (elastic#22156)
1db261c 
## What does this PR do?

This PR adds a new function to to `cfgfile` to set the path to the configuration file of a Beat. This fixes the issue on GCP with Functionbeat.

## Why is it important?

ATM Functionbeat cannot run on GCP.

## Related issues

Closes elastic#20864

(cherry picked from commit 13a195a)
@kvch kvch added the v7.10.0 label Oct 26, 2020
kvch added a commit that referenced this pull request Oct 26, 2020
@kvch
Fix Google Cloud Function configuration file issues (#22156) (#22157)
036c74c 
## What does this PR do?

This PR adds a new function to to `cfgfile` to set the path to the configuration file of a Beat. This fixes the issue on GCP with Functionbeat.

## Why is it important?

ATM Functionbeat cannot run on GCP.

## Related issues

Closes #20864

(cherry picked from commit 13a195a)
kvch added a commit that referenced this pull request Oct 26, 2020
@kvch
Fix Google Cloud Function configuration file issues (#22156) (#22158)
8ce8fa9 
## What does this PR do?

This PR adds a new function to to `cfgfile` to set the path to the configuration file of a Beat. This fixes the issue on GCP with Functionbeat.

## Why is it important?

ATM Functionbeat cannot run on GCP.

## Related issues

Closes #20864

(cherry picked from commit 13a195a)
v1v added a commit to v1v/beats that referenced this pull request Oct 26, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/beats-tes…
1605aef 
…ter-commit

* upstream/master: (25 commits)
  [CI] set env variable for the params (elastic#22143)
  Fix zeek connection pipeline (elastic#22151)
  Fix Google Cloud Function configuration file issues (elastic#22156)
  Remove old TODO on kubernetes node update (elastic#22074)
  [CI] Enable winlogbeat (elastic#22142)
  [CI] support windows-10 (elastic#19804)
  Use default config when creating the input (elastic#22126)
  Change x509 mappings from file. to tls.server. (elastic#22097)
  Add fleet settings image (elastic#22065)
  Edit 7.9.3 changelog (elastic#22117)
  Edit 6.8.13 release notes (elastic#22120)
  Incorporate librpm fix feedback (elastic#22098)
  [libbeat] Add more disk queue unit tests and fix a size-check bug (elastic#22107)
  docs: move kerberos include (elastic#22109)
  Check context.Canceled and fix s3 input config (elastic#22036)
  Add max_number_of_messages into aws filebeat fileset vars (elastic#22057)
  Remove suricata.eve.timestamp alias (elastic#22095)
  [Ingest Manager] Use symlink path for reexecutions (elastic#21835)
  chore: use ubuntu 18 as linux agent (elastic#22084)
  docs: Prepare Changelog for 7.9.3 (elastic#22073) (elastic#22075)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 27, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/support-f…
9c3d147 
…laky-test-analyser

* upstream/master:
  Add new licence status: expired (elastic#22180)
  [filebeat][okta] Make cursor optional for okta and update docs (elastic#22091)
  Add documentation of filestream input (elastic#21615)
  [Ingest Manager] Skip flaky gateway tests elastic#22177
  [CI] set env variable for the params (elastic#22143)
  Fix zeek connection pipeline (elastic#22151)
  Fix Google Cloud Function configuration file issues (elastic#22156)
  Remove old TODO on kubernetes node update (elastic#22074)
v1v added a commit to v1v/beats that referenced this pull request Oct 27, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/support-f…
0757395 
…laky-test-analyser

* upstream/master:
  Add new licence status: expired (elastic#22180)
  [filebeat][okta] Make cursor optional for okta and update docs (elastic#22091)
  Add documentation of filestream input (elastic#21615)
  [Ingest Manager] Skip flaky gateway tests elastic#22177
  [CI] set env variable for the params (elastic#22143)
  Fix zeek connection pipeline (elastic#22151)
  Fix Google Cloud Function configuration file issues (elastic#22156)
  Remove old TODO on kubernetes node update (elastic#22074)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@michalpristas michalpristas michalpristas approved these changes

Assignees

No one assigned

Labels

Team:Services (Deprecated) Label for the former Integrations-Services team v7.10.0 v7.11.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Functionbeat for GCP fails to find functionbeat.yml file - no index pattern seen in Kibana

3 participants

@kvch @elasticmachine @michalpristas