Skip to content

Pipeline: use notify instead of polling for ExchangeReceiver#9073

Merged
ti-chi-bot[bot] merged 40 commits intopingcap:masterfrom
SeaRise:notify_for_receiver
Sep 27, 2024
Merged

Pipeline: use notify instead of polling for ExchangeReceiver#9073
ti-chi-bot[bot] merged 40 commits intopingcap:masterfrom
SeaRise:notify_for_receiver

Conversation

@SeaRise
Copy link
Contributor

@SeaRise SeaRise commented May 22, 2024

What problem does this PR solve?

Issue Number: ref #8869

Problem Summary:

What is changed and how it works?

The cpu usage of wait reactor is reduced to 0 in tpch50.
c72a55fd-0cd5-4fe2-87e2-beb5e5631d0e

Check List

Tests

Tested tpch50, no performance regression

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 22, 2024
@SeaRise SeaRise changed the title Pipeline: use notify instead of polling for ExchangeReceiver DNM: Pipeline: use notify instead of polling for ExchangeReceiver May 22, 2024
@SeaRise SeaRise requested a review from windtalker September 4, 2024 02:52
@windtalker windtalker mentioned this pull request Sep 9, 2024
12 tasks
: queue(log_, std::forward<Args>(args)...)
{}

void registerTask(TaskPtr && task) override { queue.registerPipeReadTask(std::move(task)); }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe rename it to registerPipelineReadTask?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because this is an override of NotifyFuture::registerTask, it cannot be renamed

@SeaRise SeaRise requested a review from windtalker September 20, 2024 06:32
Copy link
Contributor

@windtalker windtalker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 23, 2024
private:
using QueueImpl = LooseBoundedMPMCQueue<ReceivedMessagePtr>;
// these are unbounded queues.
std::unique_ptr<QueueImpl> queue = std::make_unique<QueueImpl>(std::numeric_limits<size_t>::max());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why use unique_ptr? Also, how about letting MSGChannel inherits LooseBoundedMPMCQueue<ReceivedMessagePtr>?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ref #7963,
Because LooseBoundedMPMCQueue includes locks, it must use unique_ptr or shared_ptr somewhere.

void registerPipeWriteTask(TaskPtr && task) { queue.registerPipeWriteTask(std::move(task)); }

private:
GRPCRecvQueue<ReceivedMessagePtr> queue;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about directly letting GRPCRecvNotifyQueue inherit GRPCRecvQueue<ReceivedMessagePtr>? If so, these functions do not need to be written, only the registerTask function is required.

Signed-off-by: gengliqi <gengliqiii@gmail.com>
@ti-chi-bot ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 27, 2024
Signed-off-by: gengliqi <gengliqiii@gmail.com>
Signed-off-by: gengliqi <gengliqiii@gmail.com>
Copy link
Contributor

@gengliqi gengliqi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Sep 27, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gengliqi, windtalker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [gengliqi,windtalker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 27, 2024
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Sep 27, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-09-23 03:00:55.990452415 +0000 UTC m=+1448525.730876363: ☑️ agreed by windtalker.
  • 2024-09-27 10:48:38.150357477 +0000 UTC m=+6273.570570488: ☑️ agreed by gengliqi.

@SeaRise
Copy link
Contributor Author

SeaRise commented Sep 27, 2024

/retest

Signed-off-by: gengliqi <gengliqiii@gmail.com>
@gengliqi
Copy link
Contributor

/retest

@ti-chi-bot ti-chi-bot bot merged commit 92e8dac into pingcap:master Sep 27, 2024
@SeaRise SeaRise deleted the notify_for_receiver branch July 29, 2025 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants