Skip to content

Thread Buildup Fix - Reduce total active threads to only agent threads#277

Merged
Zakaria-Kofiro merged 3 commits intomasterfrom
zkofiro/thread-buildup-fix
Oct 25, 2023
Merged

Thread Buildup Fix - Reduce total active threads to only agent threads#277
Zakaria-Kofiro merged 3 commits intomasterfrom
zkofiro/thread-buildup-fix

Conversation

@Zakaria-Kofiro
Copy link
Collaborator

@Zakaria-Kofiro Zakaria-Kofiro commented Oct 25, 2023

Thread Buildup Fix - Reduce total active threads to only agent threads

Currently, when running a job, each agent instance is spinning up numerous additional threads per user thread, resulting in a total of up to 10x-15x times the amount of active threads requested. This resulted in an overhead of unnecessary active threads during job runs, and in some cases, caused agents to crash for certain EC2 instance types when ran long enough:

Screenshot 2023-10-25 at 10 25 14 AM

The root cause of these unnecessary additional threads was found to be due to a call from the LogUtil class to AmazonUtil creating a new HttpClient object each time it was called and subsequently spinning up around 7-8 helper threads in the process for each call for each agent/user thread (see above). The logging util class is used throughout the agent code.

This is fixed by moving the HttpClient object to be a static member of AmazonUtil, so that it is created once and reused for all calls. This greatly improves the performance of jobs by now limiting the total number of threads run to only agent threads:

Screenshot 2023-10-25 at 10 41 28 AM

Please make sure these check boxes are checked before submitting

  • ** Squashed Commits **
  • ** All Tests Passed ** - mvn clean test -P default

** PR review process **

  • Requires one +1 from a reviewer
  • Repository owners will merge your PR once it is approved.

@kevin-mcgoldrick
Copy link
Collaborator

@Zakaria-Kofiro Can you make it final?

@Zakaria-Kofiro Zakaria-Kofiro force-pushed the zkofiro/thread-buildup-fix branch from 52b2ac9 to 807301d Compare October 25, 2023 17:46
@Zakaria-Kofiro Zakaria-Kofiro marked this pull request as ready for review October 25, 2023 17:50
@Zakaria-Kofiro Zakaria-Kofiro merged commit 700ca15 into master Oct 25, 2023
@Zakaria-Kofiro Zakaria-Kofiro deleted the zkofiro/thread-buildup-fix branch October 25, 2023 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants