Skip to content

[DSIP-35][Alert] Refactor the alert thread model #15931

@ruanwenjun

Description

@ruanwenjun

Search before asking

  • I had searched in the DSIP and found no similar DSIP.

Motivation

The alert server is designed as HA, and we use single thread in the active server to loop alert event and send alert event to remote e.g HTTP/Slack/Email...

image

The current thread model has some problems

  • Once an alert is blocked due to the remote server network block, the whole alert server will be blocked.
  • The throughput of the AlertServer depends on the main thread on the active server, and due to the event sending is io blocking operation, so the throughput is very small.

Design Detail

This issue aims to improve the throughput of the alert server.

image

In the new thread model, there will exist three kinds of threads in the alert server

  • Event fetch thread
  • Event loop thread
  • Evend sending threads

Each kind of threads will not block others.

Compatibility, Deprecation, and Migration Plan

Compatibility with latest version.

Test Plan

Test by UT.

Code of Conduct

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions