Motivation

This PR improves the robustness and flexibility of the RDMA transport layer and fixes several critical bugs encountered in AMD AINIC network environments and edge cases.

Key Goals

• Fix GID Selection: Resolve issues where incorrect GID indices caused connection failures in environments with different network configurations
• Enhance CPU Support: Enable RDMA device search and usage when memory is located on the CPU
• Improve Stability: Fix segmentation faults related to object lifecycle management
• Optimize for AINIC: Fine-tune Queue Pair (QP) parameters and notification mechanisms for better compatibility with AMD AINIC hardware
• Remove SRQ: Refine notification mechanism, Move from Shared Receive Queues (SRQ) to per-Queue Pair (QP) posting for improved compatibility (for AINIC) and stability

Technical Details

RDMA / IBVerbs Provider

Auto GID Detection (src/application/transport/rdma/providers/ibverbs/ibverbs.cpp):

• Implemented auto-detection mechanism for gidIdx when configured to -1
• Iterates through the GID table and selects the best index (prioritizing: IPv4 > Global IPv6 > Link Local)

Correct Attribute Usage:

• Fixed maxMsgsNum, maxCqeNum, maxMsgSge params
• Updated QP initialization attributes
• Correctly sets sgid_index during connection establishment using the detected local GID index

Error Handling:

• Added SYSCALL_RETURN_ZERO checks to ibv_modify_qp and other critical calls
• Added badWr parameter to ibv_post_send instead of nullptr, which was causing post send error on AINIC driver(ionic_datapath.c):

static int ionic_post_send_common(struct ionic_ibdev *dev,
				  struct ionic_vcq *vcq,
				  struct ionic_cq *cq,
				  struct ionic_qp *qp,
				  struct ib_send_wr *wr,
				  struct ib_send_wr **bad)
{
	unsigned long irqflags;
	bool notify = false;
	int spend, rc = 0;
	u16 old_prod;

	ionic_lat_trace(dev->lats, application);
	ionic_stat_incr(dev->stats, post_send);

	if (!bad)
		return -EINVAL;

	if (!qp->has_sq) {
		*bad = wr;
		return -EINVAL;
	}
...

RDMA Backend (src/io/rdma/backend_impl.cpp)

Flexible Notification Mechanism (include/mori/io/backend.hpp):

• Added enableNotification configuration option to RdmaBackendConfig
• Allows disabling transfer completion notifications when not needed for compatibility or performance reasons
• When disabled, skips RdmaNotifyTransfer calls and related RECV operations

CPU Topology Search:

• Added support for MemoryLocationType::CPU in RdmaManager::Search
• Implemented round-robin strategy for device selection to prevent failures
• Note: Current implementation uses simple round-robin; more sophisticated selection logic planned for future work

Memory Safety:

• Replaced free() with delete for C++ objects (msg->meta, msg) to ensure proper destructor calls
• Added mutex protection for status messages in TransferStatus::SetMessage

Test Plan

• RDMA Connectivity: Verified connection establishment in environments requiring specific GID indices.
• CPU Memory: Tested RDMA operations with CPU-resident memory.
• Stability: Ran existing benchmarks and workloads to verify the fix for segmentation faults (-O3 related).

Test Result

• ✅ Pass: GID auto-detection correctly identifies IPv4 GIDs
• ✅ Pass: Configurable notification mechanism works correctly
• ✅ Pass: CPU memory operations work correctly
• ✅ Pass: No segmentation faults in tests

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.