Skip to content

[EP] Implement send/recv#1236

Merged
UNIDY2002 merged 13 commits intomainfrom
sunxun/send-recv
Dec 22, 2025
Merged

[EP] Implement send/recv#1236
UNIDY2002 merged 13 commits intomainfrom
sunxun/send-recv

Conversation

@UNIDY2002
Copy link
Copy Markdown
Collaborator

@UNIDY2002 UNIDY2002 commented Dec 18, 2025 •
edited
Loading

Description

Part of #1225.

This PR adds support for torch.distributed.send / recv, and also supports torch.distributed.batch_isend_irecv.

Type of Change

  • Types
    • Bug fix
    • New feature
      • Transfer Engine
      • Mooncake Store
      • Mooncake EP
      • Integration
      • P2P Store
      • Python Wheel
    • Breaking change
    • CI/CD
    • Documentation update
    • Other

How Has This Been Tested?

python -m unittest mooncake-wheel.tests.test_mooncake_backend_p2p_cpu

E2E test in SGLang also verifies the correctness.

Checklist

  • I have performed a self-review of my own code.
  • I have updated the documentation.
  • I have added tests to prove my changes are effective.

@gemini-code-assist
Copy link
Copy Markdown
Contributor

gemini-code-assist Bot commented Dec 18, 2025

Summary of Changes

Hello @UNIDY2002, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates essential point-to-point send and recv capabilities into the Mooncake backend. This enables direct, ordered tensor exchanges between specific ranks, a fundamental requirement for many distributed machine learning algorithms. The implementation leverages existing transfer engine mechanisms for data movement and introduces a store-based control plane to ensure reliable and ordered message delivery.

Highlights

  • P2P Communication API: Implemented send and recv methods in MooncakeBackend for point-to-point tensor communication, supporting single-tensor operations as required by torch.distributed P2POp.
  • Worker Infrastructure Update: Modified mooncake_worker.cuh to include c10d::Store and added P2P-specific metadata (store, backendIndex, p2pSendSeq, p2pRecvSeq) to TransferGroupMeta for managing message ordering and control messages.
  • Data Transfer Logic: Developed the core send and recv logic in mooncake_backend.cpp, handling both CPU and CUDA tensors, utilizing RDMA for efficient data movement, and employing a c10d::Store for control message synchronization to ensure reliable and ordered delivery.
  • Unit Tests: Introduced new CPU-based unit tests (test_mooncake_backend_p2p_cpu.py) to validate the P2P functionality, covering ring communication and guaranteed message ordering between ranks.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist Bot reviewed Dec 18, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements point-to-point send and recv operations for the Mooncake backend. The implementation is synchronous, with communication and synchronization handled via RDMA writes and a distributed key-value store. New Python tests are added to verify the functionality for the CPU backend, including ring communication and message ordering.

My review focuses on performance and correctness. I've identified a few areas for improvement:

  • The send operation contains a busy-wait loop that can be made more efficient.
  • The CUDA stream synchronization in both send and recv can be made more precise to avoid potential performance impacts on other operations.
  • The synchronization logic in the new tests can be made more robust by using dist.barrier().

Overall, this is a good addition, and with the suggested changes, it will be more robust and performant.

Comment thread mooncake-ep/src/mooncake_backend.cpp Outdated
Comment thread mooncake-ep/src/mooncake_backend.cpp Outdated
Comment thread mooncake-wheel/tests/test_mooncake_backend_p2p_cpu.py
Comment thread mooncake-wheel/tests/test_mooncake_backend_p2p_cpu.py
@UNIDY2002 UNIDY2002 marked this pull request as ready for review December 22, 2025 05:21
@UNIDY2002
Fix
2e0617b
@UNIDY2002
Merge remote-tracking branch 'origin/main' into sunxun/send-recv [ski…
a7ae4f6 
…p ci]

# Conflicts:
#	.github/workflows/ci.yml
#	mooncake-ep/include/mooncake_backend.h
#	mooncake-ep/include/mooncake_worker.cuh
#	mooncake-ep/src/mooncake_backend.cpp
@UNIDY2002
Debug
6113975
@UNIDY2002
Debug
db2ad8b
@UNIDY2002
Format
efff79b
@UNIDY2002 UNIDY2002 requested a review from ympcMark December 22, 2025 08:23
ympcMark
ympcMark approved these changes Dec 22, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@ympcMark ympcMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.

@UNIDY2002 UNIDY2002 merged commit 32e60d6 into main Dec 22, 2025
23 of 29 checks passed
@UNIDY2002 UNIDY2002 deleted the sunxun/send-recv branch December 22, 2025 09:30
nickyc975 pushed a commit to nickyc975/Mooncake that referenced this pull request Jan 11, 2026
@UNIDY2002 @nickyc975
[EP] Implement send/recv (kvcache-ai#1236)
b9b0b07
@UNIDY2002 UNIDY2002 mentioned this pull request Jan 22, 2026
Closed
JasonZhang517 pushed a commit to JasonZhang517/Mooncake that referenced this pull request Feb 9, 2026
@UNIDY2002
[EP] Implement send/recv (kvcache-ai#1236)
356b948
@UNIDY2002 UNIDY2002 mentioned this pull request Mar 9, 2026
Closed
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ympcMark ympcMark ympcMark approved these changes

@ShangmingCai ShangmingCai Awaiting requested review from ShangmingCai ShangmingCai is a code owner

@stmatengss stmatengss Awaiting requested review from stmatengss stmatengss is a code owner

@ykwd ykwd Awaiting requested review from ykwd ykwd is a code owner

@Ann-1024 Ann-1024 Awaiting requested review from Ann-1024 Ann-1024 is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

run-ci

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@UNIDY2002 @ympcMark