-
Notifications
You must be signed in to change notification settings - Fork 5k
[DSIP-35][Alert] Refactor the alert thread model #15931
Copy link
Copy link
Closed
Labels
Description
Search before asking
- I had searched in the DSIP and found no similar DSIP.
Motivation
The alert server is designed as HA, and we use single thread in the active server to loop alert event and send alert event to remote e.g HTTP/Slack/Email...
The current thread model has some problems
- Once an alert is blocked due to the remote server network block, the whole alert server will be blocked.
- The throughput of the AlertServer depends on the main thread on the active server, and due to the event sending is io blocking operation, so the throughput is very small.
Design Detail
This issue aims to improve the throughput of the alert server.
In the new thread model, there will exist three kinds of threads in the alert server
- Event fetch thread
- Event loop thread
- Evend sending threads
Each kind of threads will not block others.
Compatibility, Deprecation, and Migration Plan
Compatibility with latest version.
Test Plan
Test by UT.
Code of Conduct
- I agree to follow this project's Code of Conduct
Reactions are currently unavailable