Skip to content

[Ingest Manager] Agent atomic installer#21745

Merged
michalpristas merged 4 commits intoelastic:masterfrom
michalpristas:agent-atomic-unpack
Oct 13, 2020
Merged

[Ingest Manager] Agent atomic installer#21745
michalpristas merged 4 commits intoelastic:masterfrom
michalpristas:agent-atomic-unpack

Conversation

@michalpristas
Copy link
Copy Markdown
Contributor

@michalpristas michalpristas commented Oct 13, 2020 •
edited
Loading

What does this PR do?

This PR makes unpacking an "atomic" operation by unpacking to temporary location and then moving it to the correct place, this should solve some rare issues which manifests when agent got killed during installation of beats and results in half-unpacked archive to be understood as correct installation

I tested this change on darwin/linux/windows and using local e2e test suite.

Fixes: #21718

Why is it important?

Makes agent more reliable

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@michalpristas michalpristas self-assigned this Oct 13, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 13, 2020

Pinging @elastic/ingest-management (Team:Ingest Management)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 13, 2020
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Oct 13, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21745 updated]

  • Start Time: 2020-10-13T13:00:04.778+0000

  • Duration: 45 min 45 sec

Test stats 🧪

Test Results
Failed 0
Passed 1394
Skipped 4
Total 1398

@michalpristas
Copy link
Copy Markdown
Contributor Author

michalpristas commented Oct 13, 2020

/package

@ph
Copy link
Copy Markdown
Contributor

ph commented Oct 13, 2020

@ruflin OK to backport this to 7.10?

@ruflin
Copy link
Copy Markdown
Contributor

ruflin commented Oct 13, 2020

@ph I'm not sure we should backport this to 7.10 even though I agree it would be nice to have. This mainly fixes an issue about a partial upgrade? Is it a bug or an enhancement? ;-)

@michalpristas
Copy link
Copy Markdown
Contributor Author

michalpristas commented Oct 13, 2020

@ruflin this also fixes issue we see when agent is installed, enrolled and restarted quickly. on windows we see metricbeat is not a win32 application error sometimes.

@michalpristas michalpristas added the needs_backport PR is waiting to be backported to other branches. label Oct 13, 2020
@ph
Copy link
Copy Markdown
Contributor

ph commented Oct 13, 2020 •
edited
Loading

Well, the win32 thing seems like a a bug fix to me. :)

Should we have some delay/backoff when starting the subprocess?

@ph
Copy link
Copy Markdown
Contributor

ph commented Oct 13, 2020

@michalpristas Can you link the issue for the bug in that PR description?

@michalpristas
Copy link
Copy Markdown
Contributor Author

michalpristas commented Oct 13, 2020

@ph we added sync when unpacking, which should get rid of text file busy and then this one together with flipped enroll-install-restart flow should help us get rid of issue described here: #21718

@ph
Copy link
Copy Markdown
Contributor

ph commented Oct 13, 2020

Thanks @michalpristas for the clarification, @ruflin I think its worth to have it shipped in 7.10.

blakerouse
blakerouse approved these changes Oct 13, 2020
View reviewed changes
Copy link
Copy Markdown
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@michalpristas michalpristas merged commit fff5f6a into elastic:master Oct 13, 2020
michalpristas added a commit to michalpristas/beats that referenced this pull request Oct 14, 2020
@michalpristas
[Ingest Manager] Agent atomic installer (elastic#21745)
f82723d 
[Ingest Manager] Agent atomic installer (elastic#21745)
michalpristas added a commit to michalpristas/beats that referenced this pull request Oct 14, 2020
@michalpristas
[Ingest Manager] Agent atomic installer (elastic#21745)
2b26bfc 
[Ingest Manager] Agent atomic installer (elastic#21745)
This was referenced Oct 14, 2020
Merged
Merged
v1v added a commit to v1v/beats that referenced this pull request Oct 14, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/download-…
2bc6d54 
…dependencies-goals

* upstream/master: (46 commits)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  typofix for dns timeout configuration option (elastic#21069)
  chore: create CI artifacts for DEV usage (elastic#21645)
  [Ingest Manager] Atomic installed forgotten changelog elastic#21780
  [Ingest Manager] Agent atomic installer (elastic#21745)
  Add missing configuration annotations (elastic#21736)
  [Filebeat] Add check for context.DeadlineExceeded error (elastic#21732)
  Remove kafka partition ISR from Metricbeat docs (elastic#21709)
  Skip flaky test with oauth2 config in httpjson input (elastic#21749)
  Fix for azure retrieve resource by ids (elastic#21711)
  Update obs app links (elastic#21682)
  fix: update fleet test suite name (elastic#21738)
  Remove dot from file.extension value in Auditbeat FIM (elastic#21644)
  Fix leaks with metadata processors (elastic#16349)
  Add istiod metricset (elastic#21519)
  [Ingest Manager] Change Sync/Close call order (elastic#21735)
  ...
v1v added a commit to v1v/beats that referenced this pull request Oct 14, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
e60b7d6 
…ne-2.0-arm

* upstream/master: (93 commits)
  Fix non-windows fields on system/filesystem (elastic#21758)
  disable TestReceiveEventsAndMetadata/TestSocketCleanup/TestReceiveNewEventsConcurrently in Windows (elastic#21750)
  Use badger code from upstream repository (elastic#21705)
  Disable writes sync in persistent cache (elastic#21754)
  Make API address and Shard ID required in Cloud Foundry settings (elastic#21759)
  [CI] Support skip-ci label (elastic#21377)
  Increase recommended memory when deploying in Cloud foundry (elastic#21755)
  typofix for dns timeout configuration option (elastic#21069)
  chore: create CI artifacts for DEV usage (elastic#21645)
  [Ingest Manager] Atomic installed forgotten changelog elastic#21780
  [Ingest Manager] Agent atomic installer (elastic#21745)
  Add missing configuration annotations (elastic#21736)
  [Filebeat] Add check for context.DeadlineExceeded error (elastic#21732)
  Remove kafka partition ISR from Metricbeat docs (elastic#21709)
  Skip flaky test with oauth2 config in httpjson input (elastic#21749)
  Fix for azure retrieve resource by ids (elastic#21711)
  Update obs app links (elastic#21682)
  fix: update fleet test suite name (elastic#21738)
  Remove dot from file.extension value in Auditbeat FIM (elastic#21644)
  Fix leaks with metadata processors (elastic#16349)
  ...
michalpristas added a commit that referenced this pull request Oct 14, 2020
@michalpristas
Cherry-pick #21745 to 7.x: Agent atomic installer (#21779)
6f284fb 
* [Ingest Manager] Syncing unpacked files (#21706)

[Ingest Manager] Syncing unpacked files (#21706)

* [Ingest Manager] Change Sync/Close call order (#21735)

[Ingest Manager] Change Sync/Close call order (#21735)

* [Ingest Manager] Agent atomic installer (#21745)

[Ingest Manager] Agent atomic installer (#21745)
michalpristas added a commit that referenced this pull request Oct 14, 2020
@michalpristas
Cherry-pick #21745 to 7.10: Agent atomic installer (#21778)
2890b7e 
* [Ingest Manager] Syncing unpacked files (#21706)

[Ingest Manager] Syncing unpacked files (#21706)

* [Ingest Manager] Change Sync/Close call order (#21735)

[Ingest Manager] Change Sync/Close call order (#21735)

* [Ingest Manager] Agent atomic installer (#21745)

[Ingest Manager] Agent atomic installer (#21745)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@blakerouse blakerouse blakerouse approved these changes

Assignees

@michalpristas michalpristas

Labels

enhancement needs_backport PR is waiting to be backported to other branches. review

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Elastic Agent] Partial extracted beat result in failure to spawn beat

5 participants

@michalpristas @elasticmachine @ph @ruflin @blakerouse