Skip to content

Add new shipper output type#31281

Merged
rdner merged 17 commits intoelastic:mainfrom
rdner:shipper-client
Apr 27, 2022
Merged

Add new shipper output type#31281
rdner merged 17 commits intoelastic:mainfrom
rdner:shipper-client

Conversation

@rdner
Copy link
Copy Markdown
Member

@rdner rdner commented Apr 13, 2022 •
edited
Loading

What does this PR do?

This will allow us to start experimenting with the new shipper architecture.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
    - [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

THERE ARE NO TESTS WHATSOEVER

The tests for the actual gRPC communication will be added later.

Closes elastic/elastic-agent-shipper#22

@rdner
Add new shipper output type
73aecb0 
This will allow us to start experimenting with the new shipper architecture.
@rdner rdner self-assigned this Apr 13, 2022
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 13, 2022
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 13, 2022 •
edited by jenkins-beats-ci bot
Loading

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

kvch
kvch reviewed Apr 13, 2022
View reviewed changes
@cmacknz cmacknz requested a review from faec April 13, 2022 14:17
@cmacknz cmacknz added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Apr 13, 2022
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 13, 2022
kvch
kvch reviewed Apr 14, 2022
View reviewed changes
@rdner rdner marked this pull request as ready for review April 14, 2022 12:19
@rdner rdner requested a review from a team as a code owner April 14, 2022 12:19
@rdner rdner requested review from leehinman and removed request for a team April 14, 2022 12:19
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Apr 14, 2022

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@rdner rdner requested a review from kvch April 14, 2022 12:20
@rdner
Copy link
Copy Markdown
Member Author

rdner commented Apr 14, 2022

/test

cmacknz
cmacknz reviewed Apr 14, 2022
View reviewed changes
cmacknz
cmacknz reviewed Apr 14, 2022
View reviewed changes
belimawr
belimawr approved these changes Apr 14, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@belimawr belimawr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

kvch
kvch suggested changes Apr 21, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@kvch kvch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not expose backoff settings and convert the documentation page to a README.md file.

belimawr
belimawr approved these changes Apr 26, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@belimawr belimawr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but see @kvch's comment about not exposing the backoff settings.

@rdner
Copy link
Copy Markdown
Member Author

rdner commented Apr 26, 2022

@belimawr I addressed all comments including the one about the backoff configuration. I planned to re-request reviews after the CI passed.

@rdner
Copy link
Copy Markdown
Member Author

rdner commented Apr 26, 2022

/test

1 similar comment
@rdner
Copy link
Copy Markdown
Member Author

rdner commented Apr 27, 2022

/test

@rdner
Copy link
Copy Markdown
Member Author

rdner commented Apr 27, 2022

/test

@rdner
Copy link
Copy Markdown
Member Author

rdner commented Apr 27, 2022

I have strange failures in the e2e tests, I'm pretty sure my added feature should not break anything existing:


[2022-04-27T13:22:15.695Z] --- Failed steps:
[2022-04-27T13:22:15.695Z] 
[2022-04-27T13:22:15.695Z]   Scenario: Logs collection from running pod # features/elastic_agent_standalone.feature:9
[2022-04-27T13:22:15.695Z]     Then "elastic-agent" collects events with "kubernetes.pod.name:a-pod" # features/elastic_agent_standalone.feature:12
[2022-04-27T13:22:15.695Z]       Error: context deadline exceeded
[2022-04-27T13:22:15.695Z] 
[2022-04-27T13:22:15.695Z]   Scenario: Logs collection from a pod with an init container # features/elastic_agent_standalone.feature:14
[2022-04-27T13:22:15.695Z]     Then "elastic-agent" collects events with "kubernetes.container.name:init-container" # features/elastic_agent_standalone.feature:17
[2022-04-27T13:22:15.695Z]       Error: context deadline exceeded
[2022-04-27T13:22:15.695Z] 
[2022-04-27T13:22:15.695Z]   Scenario: Logs collection from short-living cronjobs # features/elastic_agent_standalone.feature:22
[2022-04-27T13:22:15.695Z]     Then "elastic-agent" collects events with "kubernetes.container.name:cronjob-container" # features/elastic_agent_standalone.feature:26
[2022-04-27T13:22:15.695Z]       Error: context deadline exceeded
[2022-04-27T13:22:15.696Z] 
[2022-04-27T13:22:15.696Z]   Scenario: Logs collection from failing pod # features/elastic_agent_standalone.feature:28
[2022-04-27T13:22:15.696Z]     Then "elastic-agent" collects events with "kubernetes.pod.name:a-failing-pod" # features/elastic_agent_standalone.feature:31
[2022-04-27T13:22:15.696Z]       Error: context deadline exceeded
[2022-04-27T13:22:15.696Z] 
[2022-04-27T13:22:15.696Z]   Scenario: Metrics collection configured from targeted Redis Pod # features/elastic_agent_standalone.feature:33
[2022-04-27T13:22:15.696Z]     Then "elastic-agent" collects events with "kubernetes.pod.name:redis" # features/elastic_agent_standalone.feature:36
[2022-04-27T13:22:15.696Z]       Error: context deadline exceeded
[2022-04-27T13:22:15.696Z] 
[2022-04-27T13:22:15.696Z] 
[2022-04-27T13:22:15.696Z] 5 scenarios (5 failed)
[2022-04-27T13:22:15.696Z] 22 steps (16 passed, 5 failed, 1 skipped)

@cmacknz would you mind if I merged it regardless the failures?

@cmacknz
Copy link
Copy Markdown
Member

cmacknz commented Apr 27, 2022

Some of the failing E2E tests (https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/main/876/testReport/) are listed as flaky: https://github.com/elastic/e2e-testing/issues

The packaging step is being triggered because of the changes to go.mod here.

@cmacknz
Copy link
Copy Markdown
Member

cmacknz commented Apr 27, 2022

The exact same test set is failing on main so I don't think they are caused by your change: https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/main/876/#showFailuresLink

@cmacknz
Copy link
Copy Markdown
Member

cmacknz commented Apr 27, 2022

I think it is safe to merge this given your change didn't introduce the E2E test failures.

@rdner rdner merged commit 755cb8a into elastic:main Apr 27, 2022
@rdner rdner deleted the shipper-client branch April 27, 2022 14:36
kush-elastic pushed a commit to kush-elastic/beats that referenced this pull request May 2, 2022
@rdner @kush-elastic
Add new shipper output type (elastic#31281)
fe95a3c 
This will allow us to start experimenting with the new shipper architecture.
chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
@rdner @chrisberkhout
Add new shipper output type (#31281)
ba40ad6 
This will allow us to start experimenting with the new shipper architecture.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@faec faec faec left review comments

@belimawr belimawr belimawr approved these changes

@cmacknz cmacknz cmacknz approved these changes

@leehinman leehinman Awaiting requested review from leehinman leehinman is a code owner automatically assigned from elastic/elastic-agent-data-plane

+1 more reviewer

@kvch kvch kvch approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@rdner rdner

Labels

Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team v8.3.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement and document the basic experimental shipper output type

6 participants

@rdner @elasticmachine @cmacknz @belimawr @kvch @faec