Skip to content

ut_sendmsg#1452

Merged
kmesh-bot merged 20 commits intokmesh-net:mainfrom
wxnzb:ut_sendmsg
Sep 19, 2025
Merged

ut_sendmsg#1452
kmesh-bot merged 20 commits intokmesh-net:mainfrom
wxnzb:ut_sendmsg

Conversation

@wxnzb
Copy link
Copy Markdown
Contributor

@wxnzb wxnzb commented Jul 24, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:


Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @wxnzb, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on enhancing the test coverage for BPF programs by introducing a dedicated unit test for the sendmsg functionality. The new test validates the BPF program's ability to intercept and modify network messages, ensuring that it correctly embeds predefined metadata into the message payload. This improves the robustness and reliability of the BPF codebase.

Highlights

  • New BPF Map for Socket Management: I've introduced a new BPF_MAP_TYPE_SOCKHASH map named map_of_kmesh_socket in bpf/kmesh/workload/sendmsg.c. This map is designed to store socket-related information, keyed by struct bpf_sock_tuple, and will be crucial for managing socket states within BPF programs.
  • Introduction of sendmsg BPF Unit Test: I've added a new unit test specifically for the sendmsg BPF program. This involves creating test/bpf_ut/workload_sendmsg_test.c which includes the original sendmsg.c with a mocked bpf_sk_storage_get helper, allowing for isolated testing of the sendmsg logic.
  • Comprehensive Go Test Integration: The new testSendMsg function has been integrated into test/bpf_ut/bpftest/workload_test.go. This Go test sets up a simulated network environment, attaches the sendmsg_prog to a sockhash map, and verifies that the BPF program correctly modifies outgoing messages by encoding specific metadata (like IP and port) into the payload using a TLV (Type-Length-Value) format.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a unit test for the sendmsg BPF program. The changes include adding the BPF test file, Go test logic, and Makefile rules. The review identifies a critical issue with a BPF map name mismatch that would cause the test to fail, a high-severity issue with a potential race condition in the test, and several medium-severity issues related to maintainability and code clarity, such as duplicated flags in the Makefile and use of magic numbers.

Comment on lines +10 to +16
struct {
__uint(type, BPF_MAP_TYPE_SOCKHASH);
__type(key, struct bpf_sock_tuple);
__type(value, __u32);
__uint(max_entries, MAP_SIZE_OF_MANAGER);
__uint(map_flags, 0);
} map_of_kmesh_socket SEC(".maps");
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The BPF map is named map_of_kmesh_socket, but the Go test code in test/bpf_ut/bpftest/workload_test.go:806 attempts to access it using the name km_socket. This mismatch will cause the test to fail with a nil map object when trying to get its file descriptor.

To ensure consistency with other BPF maps accessed from Go tests in this file (e.g., km_manage) and to fix the test, it's recommended to rename this map to km_socket.

struct {
    __uint(type, BPF_MAP_TYPE_SOCKHASH);
    __type(key, struct bpf_sock_tuple);
    __type(value, __u32);
    __uint(max_entries, MAP_SIZE_OF_MANAGER);
    __uint(map_flags, 0);
} km_socket SEC(".maps");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, why define a map here, we already have one in bpf/kmesh/workload/sockops.c

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to redefine the map in sendmsg.c because each eBPF object file (.o) only contains its own maps. The Go test loads workload_sendmsg_test.o, so it can only access maps defined in that file. Even though the same map exists in sockops.c, it’s in a different ELF file, so the test can’t see it.

By redefining it as km_socket in sendmsg.c, the map exists in the test’s ELF file and the Go test can access it.

I think this solution works for the unit test. Do you think it’s reasonable, or do you have a better approach?

t.Fatalf("Failed to start TCP server: %v", err)
}
defer listener.Close()
go func() {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The test starts a server in a goroutine but doesn't wait for it to finish. This can lead to flaky tests where the main test function exits before the goroutine has completed its checks, potentially missing errors reported by t.Error or t.Errorf.

To fix this, you should use a sync.WaitGroup:

  1. Add var wg sync.WaitGroup and wg.Add(1) before starting the goroutine.
  2. Add defer wg.Done() at the beginning of the goroutine.
  3. Add wg.Wait() at the end of the workFunc, after the client has sent data, to wait for the server goroutine to finish.

Comment on lines +51 to +53
WORKLOAD_SENDMSG_FLAGS = -I$(ROOT_DIR)/bpf/kmesh/ -I$(ROOT_DIR)/bpf/kmesh/workload/include -I$(ROOT_DIR)/api/v2-c
workload_sendmsg_test.o: workload_sendmsg_test.c
$(QUIET) $(CLANG) $(CLANG_FLAGS) $(WORKLOAD_SENDMSG_FLAGS) -c $< -o $@
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The WORKLOAD_SENDMSG_FLAGS variable is a duplicate of XDP_FLAGS. To improve maintainability and avoid redundancy, you can reuse XDP_FLAGS for the workload_sendmsg_test.o target.

workload_sendmsg_test.o: workload_sendmsg_test.c
	$(QUIET) $(CLANG) $(CLANG_FLAGS) $(XDP_FLAGS) -c $< -o $@

Comment on lines +827 to +833
clientPort := 12345
serverPort := 54321
serverSocket := localIP + ":" + strconv.Itoa(serverPort)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

These port numbers are magic numbers. For better readability and maintainability, please define them as named constants, for example testClientPort and testServerPort.

Comment on lines +851 to +894
if n < 16 {
t.Errorf("Received data too short for TLV format")
return
}
if buf[0] != 0x01 {
t.Errorf("Unexpected TLV Type: got %x, want 0x01", buf[0])
}
length := binary.BigEndian.Uint32(buf[1:5])
if length != 6 {
t.Errorf("Unexpected TLV Length: got %d, want 6", length)
}

ip := net.IPv4(buf[5], buf[6], buf[7], buf[8])
expectedIP := net.IPv4(8, 8, 8, 8)
if !ip.Equal(expectedIP) {
t.Errorf("Unexpected TLV IP: got %v, want %v", ip, expectedIP)
}

port := binary.BigEndian.Uint16(buf[9:11])
if port != 53 {
t.Errorf("Unexpected TLV Port: got %d, want 53", port)
}

if buf[11] != 0xFE {
t.Errorf("Missing or wrong TLV EndTag: got %x, want 0xFE", buf[11])
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The TLV validation logic uses several magic numbers (e.g., 16, 0x01, 6, 0xFE). These correspond to values defined in bpf/kmesh/workload/sendmsg.c. To improve readability and consistency, please define these values as constants in the Go test as well. For example:

const (
    tlvOrgDstAddrType   = 0x01
    tlvPayloadType      = 0xfe
    tlvIPv4DataLen      = 6
    expectedMinRecvSize = 16
)

@hzxuzhonghu hzxuzhonghu requested a review from Copilot July 25, 2025 06:13
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces unit tests for the sendmsg BPF program, adding test infrastructure to validate TLV (Type-Length-Value) encoding functionality when data is sent through waypoint connections.

Key changes:

  • Added sendmsg unit test with mock storage and TLV validation
  • Extended BPF test framework to support SkMsg program type
  • Added build configuration for the new test file

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
test/bpf_ut/workload_sendmsg_test.c Creates mock BPF test environment with storage data for sendmsg testing
test/bpf_ut/bpftest/workload_test.go Implements comprehensive sendmsg test with TLV format validation
test/bpf_ut/bpftest/bpf_test.go Adds SkMsg program type support to test framework
test/bpf_ut/Makefile Adds build target for workload_sendmsg_test.o
bpf/kmesh/workload/sendmsg.c Adds sockhash map definition for socket management

})
startLogReader(coll)
// Get the sockhash and the program
sockMap := coll.Maps["km_socket"]
Copy link

Copilot AI Jul 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The map name "km_socket" doesn't match the map defined in sendmsg.c which is "map_of_kmesh_socket". This will cause the test to fail as sockMap will be nil.

Suggested change
sockMap := coll.Maps["km_socket"]
sockMap := coll.Maps["map_of_kmesh_socket"]

Copilot uses AI. Check for mistakes.
{
void *storage = NULL;
storage = bpf_sk_storage_get(map, sk, value, flags);
if (!storage && map == &map_of_sock_storage) {
Copy link

Copilot AI Jul 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The map reference "map_of_sock_storage" is not defined or included in this test file, which will cause a compilation error.

Copilot uses AI. Check for mistakes.
Port: clientPort,
},
Timeout: 2 * time.Second,
}).Dial("tcp4", serverSocket)
Copy link

Copilot AI Jul 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The defer statement should be placed after the error check for the Dial operation. If Dial fails, conn will be nil and calling Close() on it will panic.

Suggested change
}).Dial("tcp4", serverSocket)
}).Dial("tcp4", serverSocket)
if err != nil {
t.Fatal(err)
}

Copilot uses AI. Check for mistakes.
}
t.Logf("Server received data: % x", buf[:n])
//check
if n < 16 {
Copy link

Copilot AI Jul 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The magic number 16 should be defined as a constant or calculated based on the TLV structure (1 byte type + 4 bytes length + 4 bytes IP + 2 bytes port + 1 byte end tag = 12 bytes minimum, not 16).

Suggested change
if n < 16 {
const MinTLVSize = 12 // 1 byte type + 4 bytes length + 4 bytes IP + 2 bytes port + 1 byte end tag
if n < MinTLVSize {

Copilot uses AI. Check for mistakes.
@LiZhenCheng9527
Copy link
Copy Markdown
Contributor

Run make gen beforn push your commit if you add or change bpf map.
And can you provide the result of sendmsg unit test?

wxnzb added 5 commits August 4, 2025 11:51
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Copy link
Copy Markdown
Member

@hzxuzhonghu hzxuzhonghu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just find it missed submit comments

Comment on lines +10 to +16
struct {
__uint(type, BPF_MAP_TYPE_SOCKHASH);
__type(key, struct bpf_sock_tuple);
__type(value, __u32);
__uint(max_entries, MAP_SIZE_OF_MANAGER);
__uint(map_flags, 0);
} map_of_kmesh_socket SEC(".maps");
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, why define a map here, we already have one in bpf/kmesh/workload/sockops.c

$(QUIET) $(CLANG) $(CLANG_FLAGS) $(TC_FLAGS) -c $< -o $@

WORKLOAD_SENDMSG_FLAGS = -I$(ROOT_DIR)/bpf/kmesh/ -I$(ROOT_DIR)/bpf/kmesh/workload/include -I$(ROOT_DIR)/api/v2-c
workload_sendmsg_test.o: workload_sendmsg_test.c
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest we moving these test building to a script and keep makefile simple

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest we moving these test building to a script and keep makefile simple我建议我们将这些测试构建移动到脚本中并使 makefile 保持简单

Should all the previous include directives be combined into a single bash script?

}
localIP := get_local_ipv4(t)
clientPort := 12345
serverPort := 54321
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of using a fixed port, we can let os randomly choose which free port by calling net.Listen(:0) ?

t.Errorf("Unexpected TLV Port: got %d, want 53", port)
}

if buf[11] != 0xFE {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you abstract a utility function to parse tlv protocol? So we can resue it in other tests as well

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to verify the IP, port, and the correctness of the format specifically, so I feel it’s not really reusable. Do you think it’s necessary to abstract it?If abstracted, the message, IP, and port need to be passed as parameters. IPv4 and IPv6 should share the logic first when needed, right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you abstract a utility function to parse tlv protocol? So we can resue it in other tests as well你能抽象出一个实用函数来解析 TLV 协议吗?这样我们就可以在其他测试中重复使用它了。

can you abstract a utility function to parse tlv protocol? So we can resue it in other tests as well你能抽象出一个实用函数来解析 TLV 协议吗?这样我们就可以在其他测试中重复使用它了。

Yes, I’ve abstracted the TLV parsing logic into ParseTLVMessage and ValidateTLVMessage in a new file tlv_utils.go, so it can be reused in other tests.

return storage;
}

#define bpf_sk_storage_get mock_bpf_sk_storage_get
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you elaborate a little bit how this function can be used

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I will add some comments.

wxnzb added 2 commits August 7, 2025 16:34
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
@codecov
Copy link
Copy Markdown

codecov bot commented Aug 7, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 38.35%. Comparing base (bf64ca4) to head (dddab4c).
⚠️ Report is 56 commits behind head on main.
see 7 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7193230...dddab4c. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wxnzb
Copy link
Copy Markdown
Contributor Author

wxnzb commented Aug 24, 2025

Run make gen beforn push your commit if you add or change bpf map.如果您添加或更改 bpf 映射,请在推送提交之前运行 make gen。 And can you provide the result of sendmsg unit test?您能提供 sendmsg 单元测试的结果吗?

截图 2025-08-24 18-53-01

wxnzb added 3 commits August 24, 2025 19:48
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
__type(value, __u32);
__uint(max_entries, MAP_SIZE_OF_MANAGER);
__uint(map_flags, 0);
} map_of_kmesh_sendmsg SEC(".maps");
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a comment before this struct to indicate this is used for test only

Copy link
Copy Markdown
Member

@hzxuzhonghu hzxuzhonghu Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wxnzb is there a way to move this out from this file, i donot want to make it load and attached to kernel when we run kmesh.

Copy link
Copy Markdown
Member

@hzxuzhonghu hzxuzhonghu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait ths map will pollute the dualengine, actually we only need this for test. Is there other way to only load the bpf for test

@wxnzb wxnzb mentioned this pull request Sep 11, 2025
6 tasks
@hzxuzhonghu
Copy link
Copy Markdown
Member

please update

@kmesh-bot
Copy link
Copy Markdown
Collaborator

Adding label do-not-merge/contains-merge-commits because PR contains merge commits, which are not allowed in this repository.
Use git rebase to reapply your commits on top of the target branch. Detailed instructions for doing so can be found here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
@wxnzb
Copy link
Copy Markdown
Contributor Author

wxnzb commented Sep 18, 2025

Wait ths map will pollute the dualengine, actually we only need this for test. Is there other way to only load the bpf for test
等等,这个映射会污染双引擎,实际上我们只需要这个用于测试。还有其他方法可以只加载 bpf 进行测试吗?

Thanks for the feedback! You're absolutely right about the map pollution concern.

I've addressed this by implementing conditional compilation to ensure the map is only compiled during testing:

Changes made:

  1. Added conditional compilation in sendmsg.c:

    #ifdef KMESH_UNIT_TEST
    struct {
        __uint(type, BPF_MAP_TYPE_SOCKHASH);
        __type(key, struct bpf_sock_tuple);
        __type(value, __u32);
        __uint(max_entries, MAP_SIZE_OF_MANAGER);
        __uint(map_flags, 0);
    } map_of_kmesh_sendmsg SEC(".maps");
    #endif
  2. Updated test Makefile to define KMESH_UNIT_TEST only for test compilation:

    WORKLOAD_SENDMSG_FLAGS = ... -DKMESH_UNIT_TEST

Result:

  • Production builds: The map is completely excluded from dualengine
  • Test builds: Include the necessary sockhash map for sk_msg program attachment
  • Clean separation: Zero pollution of production code

This approach ensures the BPF map is only loaded during testing, completely addressing the dualengine pollution concern while maintaining full test functionality.

Signed-off-by: sweeywu <sweetwx6@gmail.com>
@kmesh-bot kmesh-bot added size/L and removed size/XL labels Sep 18, 2025
Signed-off-by: sweeywu <sweetwx6@gmail.com>
@kmesh-bot kmesh-bot added size/XL and removed size/L labels Sep 18, 2025
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Signed-off-by: sweeywu <sweetwx6@gmail.com>
Copy link
Copy Markdown
Member

@hzxuzhonghu hzxuzhonghu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you

@kmesh-bot
Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hzxuzhonghu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hzxuzhonghu
Copy link
Copy Markdown
Member

/lgtm

@kmesh-bot kmesh-bot added the lgtm label Sep 19, 2025
@kmesh-bot kmesh-bot merged commit 0847367 into kmesh-net:main Sep 19, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants