Skip to content

Option to build tests#727

Merged
elazarg merged 2 commits into
vbpf:mainfrom
Alan-Jowett:option_to_build_tests
Oct 14, 2024
Merged

Option to build tests#727
elazarg merged 2 commits into
vbpf:mainfrom
Alan-Jowett:option_to_build_tests

Conversation

@Alan-Jowett

@Alan-Jowett Alan-Jowett commented Oct 14, 2024

Copy link
Copy Markdown
Contributor

This pull request introduces a new option to conditionally build and run tests in the CI workflow and modifies the CMake configuration to support this option. The changes are primarily focused on enhancing the build process and improving the flexibility of the CI pipeline.

CI Workflow Enhancements:

CMake Configuration Updates:

  • CMakeLists.txt: Introduced a new option VERIFIER_ENABLE_TESTS to control the inclusion of test-related targets and dependencies. Wrapped test-related configuration and target definitions in conditional statements based on this option. [1] [2] [3] [4] [5] [6] [7] [8]

Summary by CodeRabbit

  • New Features

    • Introduced a flexible build configuration that allows conditional execution of tests during the CI process for both Ubuntu and Windows builds.
    • Added a new option to enable or disable tests in the build process.
  • Bug Fixes

    • Enhanced the build process by ensuring that test executables are only included when the appropriate option is enabled.
  • Documentation

    • Updated documentation to reflect the new testing configuration options available in the build process.

@Alan-Jowett

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Oct 14, 2024

Copy link
Copy Markdown
✅ Actions performed

Full review triggered.

Signed-off-by: Alan Jowett <alan.jowett@microsoft.com>
@Alan-Jowett Alan-Jowett force-pushed the option_to_build_tests branch from 036dd88 to 866964d Compare October 14, 2024 00:27
@coderabbitai

coderabbitai Bot commented Oct 14, 2024

Copy link
Copy Markdown

Walkthrough

The changes introduce a new build matrix configuration in the .github/workflows/build.yml file for both build_ubuntu and build_windows jobs, adding a build_tests parameter to control the execution of test-related steps. Additionally, the CMakeLists.txt file is updated to include a new option, VERIFIER_ENABLE_TESTS, which governs the inclusion of tests in the build process. This option allows for conditional compilation and linking of various test executables based on its value.

Changes

File Path Change Summary
.github/workflows/build.yml Added build_tests: [true, false] to build_ubuntu and build_windows; updated cmake commands to include -DVERIFIER_ENABLE_TESTS=${{matrix.build_tests}}; conditioned test execution on matrix.build_tests.
CMakeLists.txt Added option VERIFIER_ENABLE_TESTS (default ON); conditionally added executables (check, tests, run_yaml, conformance_check); set properties and compile options for test executables; linked against necessary libraries.

Poem

In the meadow where we play,
New tests bloom in bright array.
With builds that flex and twist,
We hop along, we can't resist!
CMake sings, workflows dance,
In this code, we take a chance! 🐇✨


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Outside diff range comments (3)
CMakeLists.txt (3)

Line range hint 98-107: LGTM: Conditional compilation of test files

The conditional compilation of test files is a good practice. It reduces compilation time and binary size when tests are not needed.

One suggestion for improvement:

While using file(GLOB ...) for test files is generally acceptable, consider explicitly listing the test files instead. This approach provides better control over which files are included and makes it easier to spot when a new test file is added or removed. For example:

if(VERIFIER_ENABLE_TESTS)
set(ALL_TEST
    "./src/test/test_conformance.cpp"
    "./src/test/test_marshal.cpp"
    "./src/test/test_print.cpp"
    "./src/test/test_verify.cpp"
    "./src/test/test_wto.cpp"
    "./src/test/test_yaml.cpp"
)
endif()

This approach is more explicit and less prone to accidentally including or excluding files.


Line range hint 133-151: LGTM: Conditional setting of test executable properties

The conditional setting of target properties for test executables is well implemented. This ensures that these properties are only set when tests are enabled, maintaining a clean build configuration.

One suggestion for improvement:

Consider using a variable for the RUNTIME_OUTPUT_DIRECTORY instead of repeating "${CMAKE_BINARY_DIR}/.." for each executable. This would make it easier to change the output directory for all test executables in the future if needed. For example:

if(VERIFIER_ENABLE_TESTS)
    set(TEST_OUTPUT_DIR "${CMAKE_BINARY_DIR}/..")
    set_target_properties(check tests run_yaml conformance_check
        PROPERTIES
        RUNTIME_OUTPUT_DIRECTORY "${TEST_OUTPUT_DIR}")

    add_dependencies(tests conformance_check)
endif()

This approach reduces repetition and makes the configuration more maintainable.


Line range hint 179-200: LGTM: Conditional compilation and linking of test executables

The conditional setting of compilation flags and library linkage for test executables is well implemented. This ensures that these settings are only applied when tests are enabled, maintaining a clean build configuration.

One suggestion for improvement:

Consider grouping the compile options for each executable together, followed by the link libraries. This can improve readability and make it easier to manage the settings for each executable. For example:

if(VERIFIER_ENABLE_TESTS)
    # Compile options
    foreach(target check tests)
        target_compile_options(${target} PRIVATE ${COMMON_FLAGS})
        target_compile_options(${target} PUBLIC "$<$<CONFIG:DEBUG>:${DEBUG_FLAGS}>")
        target_compile_options(${target} PUBLIC "$<$<CONFIG:RELEASE>:${RELEASE_FLAGS}>")
        target_compile_options(${target} PUBLIC "$<$<CONFIG:SANITIZE>:${SANITIZE_FLAGS}>")
    endforeach()

    # Link libraries
    target_link_libraries(check PRIVATE ebpfverifier)
    target_link_libraries(tests PRIVATE 
        ebpfverifier
        bpf_conformance
        ${CMAKE_DL_LIBS}
        Threads::Threads
        Catch2::Catch2WithMain
    )
    target_link_libraries(run_yaml PRIVATE ebpfverifier)
    target_link_libraries(conformance_check PRIVATE ebpfverifier)
endif()

This approach reduces repetition and groups related settings together, making the configuration more maintainable.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE

📥 Commits

Files that changed from the base of the PR and between a476ecd and 866964d.

📒 Files selected for processing (2)
  • .github/workflows/build.yml (4 hunks)
  • CMakeLists.txt (8 hunks)
🧰 Additional context used
🔇 Additional comments (6)
.github/workflows/build.yml (3)

21-21: Excellent addition of build_tests matrix!

The introduction of the build_tests parameter with [true, false] options in the matrix strategy is a great improvement. This change allows for more flexible CI configurations, potentially reducing build times when tests are not required.


58-58: Excellent consistency in Windows build configuration!

The addition of build_tests: [true, false] to the Windows build matrix mirrors the change made in the Ubuntu build job. This ensures consistency across different OS builds and provides the same flexibility for conditional test execution in Windows environments.


21-21: Overall, excellent improvements to the CI workflow!

The changes introduced in this PR significantly enhance the flexibility of the CI pipeline by allowing conditional building and execution of tests. The implementation is consistent across both Ubuntu and Windows jobs, which is commendable.

Key improvements:

  1. Introduction of build_tests matrix parameter.
  2. Conditional CMake configuration for tests.
  3. Conditional execution of test steps.

These changes will likely lead to more efficient CI runs when tests are not required. The code is well-structured and the changes are implemented consistently.

A few minor suggestions have been made for improved readability and consistency, but these are not critical. Great work on enhancing the CI workflow!

Also applies to: 41-41, 45-45, 49-49, 58-58, 73-73, 77-77

CMakeLists.txt (3)

8-9: LGTM: New option for enabling tests

The addition of the VERIFIER_ENABLE_TESTS option is a good practice. It allows for flexibility in the build process, enabling or disabling test compilation as needed. The default value of ON maintains backwards compatibility with existing build processes.


124-129: LGTM: Conditional addition of test executables

The conditional addition of test executables (check, tests, run_yaml, conformance_check) is well implemented. This approach ensures that these executables are only built when tests are enabled, which helps to reduce compilation time and disk usage when tests are not needed.

The variety of test executables suggests a comprehensive testing strategy, covering different aspects of the project's functionality and conformance.


Line range hint 1-201: Overall assessment: Well-implemented conditional test compilation

The changes introduced in this CMakeLists.txt file significantly improve the flexibility and efficiency of the build system by introducing conditional compilation for test-related components. The new VERIFIER_ENABLE_TESTS option allows users to easily control whether tests are built, which can lead to faster build times and reduced resource usage when tests are not needed.

Key improvements:

  1. Conditional inclusion of the Catch2 testing framework
  2. Conditional compilation of test source files
  3. Conditional addition and configuration of test executables
  4. Conditional inclusion of the bpf_conformance subdirectory
  5. Conditional setting of compilation flags and library linkage for test executables

These changes are consistently applied throughout the file and follow CMake best practices. The default setting of ON for VERIFIER_ENABLE_TESTS ensures backwards compatibility with existing build processes.

While the implementation is generally very good, there are a few minor suggestions for improvement in the individual comments above, mainly focused on reducing repetition and improving maintainability.

Great job on implementing this feature! It significantly enhances the project's build system flexibility.

Comment thread .github/workflows/build.yml Outdated
Comment thread .github/workflows/build.yml Outdated
Comment thread .github/workflows/build.yml Outdated
Comment thread CMakeLists.txt
@coveralls

coveralls commented Oct 14, 2024

Copy link
Copy Markdown

Coverage Status

coverage: 90.292% (+0.001%) from 90.291%
when pulling ec09d71 on Alan-Jowett:option_to_build_tests
into a476ecd on vbpf:main.

@elazarg

elazarg commented Oct 14, 2024

Copy link
Copy Markdown
Collaborator

@Alan-Jowett feel free to dismiss coderabbit comments you deem irrelevant or wrong

@elazarg

elazarg commented Oct 14, 2024

Copy link
Copy Markdown
Collaborator

Two comments:

  1. Is it possible/sensible to give names to the tests option instead of true/false?
  2. If you do any other change, please add new line at the end of CMakeLists.txt

Signed-off-by: Alan Jowett <alan.jowett@microsoft.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🧹 Outside diff range comments (3)
CMakeLists.txt (3)

Line range hint 98-107: LGTM with suggestion: Conditional inclusion of test files

The conditional inclusion of test files based on VERIFIER_ENABLE_TESTS is good. It ensures that test-related files are only processed when tests are enabled.

Consider replacing file(GLOB ...) with explicit file listings. While GLOB is convenient, it can lead to build system issues if new files are added without re-running CMake. Here's an example of how you could refactor this:

if(VERIFIER_ENABLE_TESTS)
    set(ALL_TEST
        src/test/test_conformance.cpp
        src/test/test_marshal.cpp
        src/test/test_print.cpp
        src/test/test_verify.cpp
        src/test/test_wto.cpp
        src/test/test_yaml.cpp
    )
endif()

This approach provides better control over which files are included in the build and ensures that CMake always picks up changes to the file list.


Line range hint 133-151: LGTM: Setting properties for test executables

The addition of runtime output directory settings for test executables and the dependency between 'tests' and 'conformance_check' are good practices.

For consistency and easier maintenance, consider using a loop or a function to set the runtime output directory for all test executables. For example:

if(VERIFIER_ENABLE_TESTS)
    foreach(test_exec check tests run_yaml conformance_check)
        set_target_properties(${test_exec}
            PROPERTIES
            RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/..")
    endforeach()

    add_dependencies(tests conformance_check)
endif()

This approach reduces repetition and makes it easier to apply the same settings to all test executables consistently.


Line range hint 179-200: LGTM: Conditional compilation and linking for test executables

The conditional setting of compile options and linking of libraries for test executables is well-implemented. It ensures that these settings are only applied when tests are enabled, which is consistent with the VERIFIER_ENABLE_TESTS option.

To improve readability and maintainability, consider creating a function to apply common settings to all test executables. For example:

if(VERIFIER_ENABLE_TESTS)
    function(configure_test_executable target)
        target_compile_options(${target} PRIVATE ${COMMON_FLAGS})
        target_compile_options(${target} PUBLIC "$<$<CONFIG:DEBUG>:${DEBUG_FLAGS}>")
        target_compile_options(${target} PUBLIC "$<$<CONFIG:RELEASE>:${RELEASE_FLAGS}>")
        target_compile_options(${target} PUBLIC "$<$<CONFIG:SANITIZE>:${SANITIZE_FLAGS}>")
        target_link_libraries(${target} PRIVATE ebpfverifier)
    endfunction()

    configure_test_executable(check)
    configure_test_executable(tests)
    target_link_libraries(tests PRIVATE bpf_conformance ${CMAKE_DL_LIBS} Threads::Threads Catch2::Catch2WithMain)

    configure_test_executable(run_yaml)
    configure_test_executable(conformance_check)
endif()

This approach centralizes the common configuration and makes it easier to maintain consistent settings across all test executables.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE

📥 Commits

Files that changed from the base of the PR and between 866964d and ec09d71.

📒 Files selected for processing (2)
  • .github/workflows/build.yml (4 hunks)
  • CMakeLists.txt (8 hunks)
🧰 Additional context used
📓 Learnings (1)
CMakeLists.txt (1)
Learnt from: Alan-Jowett
PR: vbpf/ebpf-verifier#727
File: CMakeLists.txt:18-25
Timestamp: 2024-10-14T15:06:59.489Z
Learning: In the ebpf-verifier project, focus code reviews on changes introduced in the PR and avoid commenting on existing code that was not modified.
🔇 Additional comments (4)
.github/workflows/build.yml (3)

21-21: LGTM! Great addition for flexible CI configurations.

The introduction of the target parameter with values tests and library in the matrix strategy is a good approach. This allows for more granular control over the CI process, potentially speeding up builds by separating library and test compilations.


49-49: LGTM! Proper conditional execution of tests.

The addition of the condition if: ${{matrix.target == 'tests'}} to the "Run unit tests" step is correct and aligns well with the new matrix configuration. This ensures that tests are only executed when explicitly targeted, potentially saving CI time for library-only builds.


53-53: LGTM! Consistent conditional execution of memory leak tests.

The addition of the condition if: ${{matrix.target == 'tests'}} to the "Test for memory leaks" step is correct and consistent with the previous test execution step. This ensures that memory leak tests are only run when necessary, further optimizing the CI process for different build targets.

CMakeLists.txt (1)

Line range hint 1-200: Summary: Well-implemented conditional test compilation

Overall, the changes to implement conditional test compilation based on the VERIFIER_ENABLE_TESTS option are well-done and consistent throughout the CMakeLists.txt file. The modifications allow for more flexible build configurations, which can be particularly useful in CI/CD pipelines or deployments where tests are not needed.

Key points:

  1. The new VERIFIER_ENABLE_TESTS option is clear and set to ON by default, maintaining backwards compatibility.
  2. Test-related components (Catch2, test files, executables, and compile/link settings) are consistently wrapped in conditional blocks.
  3. The changes follow CMake best practices, with a few suggestions for minor improvements in code organization and clarity.

One area that requires verification is the conditional inclusion of the bpf_conformance subdirectory. Please ensure that bpf_conformance is not required by the main project outside of testing contexts.

Comment thread .github/workflows/build.yml
Comment thread .github/workflows/build.yml
Comment thread CMakeLists.txt
Comment thread CMakeLists.txt
Comment thread CMakeLists.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants