Thread Buildup Fix - Reduce total active threads to only agent threads#277
Merged
Zakaria-Kofiro merged 3 commits intomasterfrom Oct 25, 2023
Merged
Thread Buildup Fix - Reduce total active threads to only agent threads#277Zakaria-Kofiro merged 3 commits intomasterfrom
Zakaria-Kofiro merged 3 commits intomasterfrom
Conversation
Collaborator
|
@Zakaria-Kofiro Can you make it |
52b2ac9 to
807301d
Compare
kevin-mcgoldrick
approved these changes
Oct 25, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thread Buildup Fix - Reduce total active threads to only agent threads
Currently, when running a job, each agent instance is spinning up numerous additional threads per user thread, resulting in a total of up to 10x-15x times the amount of active threads requested. This resulted in an overhead of unnecessary active threads during job runs, and in some cases, caused agents to crash for certain EC2 instance types when ran long enough:
The root cause of these unnecessary additional threads was found to be due to a call from the
LogUtilclass toAmazonUtilcreating a newHttpClientobject each time it was called and subsequently spinning up around 7-8 helper threads in the process for each call for each agent/user thread (see above). The logging util class is used throughout the agent code.This is fixed by moving the
HttpClientobject to be a static member ofAmazonUtil, so that it is created once and reused for all calls. This greatly improves the performance of jobs by now limiting the total number of threads run to only agent threads:Please make sure these check boxes are checked before submitting
mvn clean test -P default** PR review process **