Skip to content

agent cloudbeat support#179

Merged
ph merged 7 commits intoelastic:mainfrom
oren-zohar:cloudbeat-agent-support
Mar 29, 2022
Merged

agent cloudbeat support#179
ph merged 7 commits intoelastic:mainfrom
oren-zohar:cloudbeat-agent-support

Conversation

@oren-zohar
Copy link
Copy Markdown
Contributor

@oren-zohar oren-zohar commented Mar 7, 2022 •
edited by ghost
Loading

What does this PR do?

Adds cloudbeat support to the agent.

Why is it important?

As part of the Cloud Security Posture team work to add our own beat to collect cloud posture information and send it to ES, we need to add the relevant configuration changes in order to support it in the agent.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Mar 7, 2022

This pull request does not have a backport label. Could you fix it @oren-zohar? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip label Mar 7, 2022
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Mar 7, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-03-29T15:08:46.753+0000

  • Duration: 39 min 12 sec

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages.

  • run integration tests : Run the Elastic Agent Integration tests.

  • run end-to-end tests : Generate the packages and run the E2E Tests.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@oren-zohar oren-zohar marked this pull request as ready for review March 7, 2022 18:26
@oren-zohar oren-zohar mentioned this pull request Mar 7, 2022
Merged
6 tasks
@ph ph added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Mar 7, 2022
@ph
Copy link
Copy Markdown
Contributor

ph commented Mar 7, 2022

/test

@narph narph added the ci:end-to-end Enable the e2e-testing label Mar 8, 2022
@jlind23 jlind23 requested a review from a team March 9, 2022 15:53
ph
ph suggested changes Mar 14, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@ph ph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small changes around the initial arguments.

internal/spec/cloudbeat.yml Outdated
"-E", "setup.ilm.enabled=false",
"-E", "setup.template.enabled=false",
"-E", "management.enabled=true",
"-E", "logging.level=debug"
Copy link
Copy Markdown
Contributor

@ph ph Mar 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can remove this, I will double-check the other existing spec,

internal/spec/cloudbeat.yml Outdated
- inputs
- output

when: length(${inputs}) > 0 and hasKey(${output}, 'elasticsearch')
Copy link
Copy Markdown
Contributor

@ph ph Mar 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason why you only support Elasticsearch?, since you are using the libbeat framework you should have access to all of the outputs? see https://github.com/elastic/elastic-agent/blob/main/internal/spec/apm-server.yml#L37

@jlind23 jlind23 requested a review from a team March 15, 2022 13:01
@ph ph requested a review from narph March 15, 2022 13:07
@ph
Copy link
Copy Markdown
Contributor

ph commented Mar 15, 2022

This look good to me, but @narph Would having their packages added into our dev-tool yaml definition would make our build fail until the artifacts exits?

@ph
Copy link
Copy Markdown
Contributor

ph commented Mar 18, 2022

@oren-zohar The changes for the spec files lgtm, the main branch is the 8.2 release are you planning to release for 8.2?

@oren-zohar
Copy link
Copy Markdown
Contributor Author

oren-zohar commented Mar 21, 2022

@ph yes I think we should merge it. Everything we work on should be hidden behind a feature flag, so it will not affect the release AFAIK.

@narph
Copy link
Copy Markdown
Contributor

narph commented Mar 22, 2022

This look good to me, but @narph Would having their packages added into our dev-tool yaml definition would make our build fail until the artifacts exits?

It might cause some issues, maybe the artifacts can come first, I am not sure what is the progress on that.

@ph , shouldn't introducing a new spec cause a regeneration of the supported.go file? @oren-zohar have you ran mage buildspec?

@oren-zohar
Copy link
Copy Markdown
Contributor Author

oren-zohar commented Mar 22, 2022 •
edited
Loading

It might cause some issues, maybe the artifacts can come first, I am not sure what is the progress on that.

That makes sense, @DaveSys911 wdyt?

@ph , shouldn't introducing a new spec cause a regeneration of the supported.go file? @oren-zohar have you ran mage buildspec?

@narph I was thinking the same, we did run the buildspec script but it didn't change supported.go for some reason.

@eyalkraft
Copy link
Copy Markdown
Contributor

eyalkraft commented Mar 23, 2022

@oren-zohar , @yashtewari says pulling from main will fix the buildspec script

@oren-zohar
Copy link
Copy Markdown
Contributor Author

oren-zohar commented Mar 24, 2022

@narph @ph added the missing part in supported.go e2e fails but I'm not sure if it's related (also fails in main)

@oren-zohar oren-zohar requested a review from ph March 28, 2022 11:07
@ph
Copy link
Copy Markdown
Contributor

ph commented Mar 29, 2022

Just to clarify what I've said on slack, let me clarify the elastic agent behavior.

  1. When an input is present in the Elastic Agent.
  2. The agent looks on the disk if the archive is present.
  3. When the archive is not present it will fetch it over the artifact api.

When the Elastic Agent is built from the release manager, the manager is responsible for building everything independently and assembling the distribution. It makes sure that the artifacts is present inside the archive or the docker container.

Now, the files in dev-tools are required only in development when we are trying to build Elastic Agent locally, it's for an ease of development, I believe this is not required for cloudbeat and that would add an unnecessary link between our code.

Only the changes in the spec files are needed.

@ph
Copy link
Copy Markdown
Contributor

ph commented Mar 29, 2022

Change lgtm, waiting for ci to complete to merge.

@ph ph merged commit 71616dc into elastic:main Mar 29, 2022
@ph
Copy link
Copy Markdown
Contributor

ph commented Mar 29, 2022

Merged, I've reworded the commit message.

@eyalkraft eyalkraft mentioned this pull request Mar 31, 2022
Merged
4 tasks
@oren-zohar oren-zohar deleted the cloudbeat-agent-support branch April 6, 2022 11:18
@oren-zohar oren-zohar mentioned this pull request Apr 27, 2022
Merged
6 tasks
@tetianakravchenko tetianakravchenko mentioned this pull request Jul 14, 2022
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@narph narph Awaiting requested review from narph

1 more reviewer

@ph ph ph approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@oren-zohar oren-zohar

Labels

backport-skip ci:end-to-end Enable the e2e-testing Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@oren-zohar @elasticmachine @ph @narph @eyalkraft