Skip to content

Support CTest#10216

Merged
Buristan merged 9 commits intotarantool:masterfrom
ligurio:ligurio/run-tests-by-cmake
Dec 18, 2024
Merged

Support CTest#10216
Buristan merged 9 commits intotarantool:masterfrom
ligurio:ligurio/run-tests-by-cmake

Conversation

@ligurio ligurio requested a review from Buristan July 9, 2024 07:32
@ligurio ligurio force-pushed the ligurio/run-tests-by-cmake branch 2 times, most recently from ef3b9ee to 9f5bc59 Compare July 9, 2024 07:57
@ligurio ligurio marked this pull request as draft July 9, 2024 07:57
ligurio added a commit to ligurio/nanodata that referenced this pull request Jul 17, 2024
The proposed patch a bit ugly: fuzzing tests will be run
sequentionally after functional testing and this will make a job
longer. This will be reworked after introducing CTest support [1],
these tests will be executed in parallel with other tests.

1. tarantool#10216

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci
ligurio added a commit to ligurio/nanodata that referenced this pull request Jul 17, 2024
The proposed patch a bit ugly: fuzzing tests will be run
sequentionally after functional testing and this will make a job
longer. This will be reworked after introducing CTest support [1],
these tests will be executed in parallel with other tests.

1. tarantool#10216

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci
ligurio added a commit to ligurio/nanodata that referenced this pull request Jul 17, 2024
The proposed patch a bit ugly: fuzzing tests will be run
sequentionally after functional testing and this will make a job
longer. This will be reworked after introducing CTest support [1],
these tests will be executed in parallel with other tests.

1. tarantool#10216

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci
ligurio added a commit to ligurio/nanodata that referenced this pull request Jul 17, 2024
The proposed patch a bit ugly: fuzzing tests will be run
sequentionally after functional testing and this will make a job
longer. This will be reworked after introducing CTest support [1],
these tests will be executed in parallel with other tests.

1. tarantool#10216

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci
ligurio added a commit to ligurio/nanodata that referenced this pull request Jul 17, 2024
The proposed patch a bit ugly: fuzzing tests will be run
sequentionally after functional testing and this will make time
of jobs execution longer. This will be reworked after introducing
CTest support [1], these tests will be executed in parallel with
other tests.

The proposed test was added to workflow with debug build, because
this workflow runs in a short CI, and to workflow with enabled
ASAN because AddressSanitizer can help to find additional problems
related to incorrect work with memory.

1. tarantool#10216

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci
ligurio added a commit to ligurio/nanodata that referenced this pull request Jul 17, 2024
The proposed patch a bit ugly: fuzzing tests will be run
sequentionally after functional testing and this will make time
of jobs execution longer. This will be reworked after introducing
CTest support [1], these tests will be executed in parallel with
other tests.

The proposed test was added to workflow with debug build, because
this workflow runs in a short CI, and to workflow with enabled
ASAN because AddressSanitizer can help to find additional problems
related to incorrect work with memory.

1. tarantool#10216

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci
ligurio added a commit to ligurio/nanodata that referenced this pull request Jul 17, 2024
The proposed patch a bit ugly: fuzzing tests will be run
sequentionally after functional testing and this will make time
of jobs execution longer. This will be reworked after introducing
CTest support [1], these tests will be executed in parallel with
other tests.

The proposed test was added to workflow with debug build, because
this workflow runs in a short CI, and to workflow with enabled
ASAN because AddressSanitizer can help to find additional problems
related to incorrect work with memory.

1. tarantool#10216

NO_CHANGELOG=ci
NO_DOC=ci
NO_TEST=ci
@ligurio ligurio force-pushed the ligurio/run-tests-by-cmake branch 6 times, most recently from 81afeda to 30ba8eb Compare August 2, 2024 09:56
@ligurio ligurio force-pushed the ligurio/run-tests-by-cmake branch 6 times, most recently from 4672e1b to 8b883dd Compare August 16, 2024 14:23
@coveralls
Copy link

coveralls commented Aug 16, 2024

Coverage Status

coverage: 87.354% (-0.04%) from 87.398%
when pulling 7d0117c on ligurio:ligurio/run-tests-by-cmake
into d5d2291
on tarantool:master
.

ligurio added a commit to tarantool/small that referenced this pull request Aug 20, 2024
The patch introduces a CMake function create_test and replaces
add_test() with create_test(). Needed for assigning test labels
in the following commit.

Needed for tarantool/tarantool#10216
ligurio added a commit to tarantool/small that referenced this pull request Aug 20, 2024
locker pushed a commit to tarantool/small that referenced this pull request Aug 20, 2024
The patch introduces a CMake function create_test and replaces
add_test() with create_test(). Needed for assigning test labels
in the following commit.

Needed for tarantool/tarantool#10216
locker pushed a commit to tarantool/small that referenced this pull request Aug 20, 2024
@ligurio ligurio force-pushed the ligurio/run-tests-by-cmake branch from d783924 to b37b2b6 Compare October 8, 2024 17:51
@ligurio
Copy link
Member Author

ligurio commented Oct 8, 2024

Rebased to master branch after merging commits in PR #10655.

@ligurio ligurio force-pushed the ligurio/run-tests-by-cmake branch from b37b2b6 to 289a74e Compare October 11, 2024 14:38
@ligurio ligurio force-pushed the ligurio/run-tests-by-cmake branch from 289a74e to 54b41eb Compare October 30, 2024 19:05
@ligurio ligurio changed the title Support CTest [WIP] Support CTest Oct 31, 2024
@ligurio ligurio marked this pull request as ready for review October 31, 2024 06:31
@ligurio
Copy link
Member Author

ligurio commented Nov 20, 2024

(rebased to master with merged #10725)

@ligurio
Copy link
Member Author

ligurio commented Nov 27, 2024

(rebased to master with merged #10778)

@ligurio
Copy link
Member Author

ligurio commented Dec 16, 2024

(rebased to draft PR #10824)

Copy link
Collaborator

@Buristan Buristan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sergey,
Thanks for the fixes!
LGTM!

Copy link
Contributor

@Totktonada Totktonada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code owner review request is due to the test/config-luatest/CMakeLists.txt file. I don't mind to add it.

It is not possible to get a number of parallel jobs in CMake
passed by the user with option "--parallel", but CMake allows to
pass a number of parallel jobs with environment variable
CMAKE_BUILD_PARALLEL_LEVEL [1] on configuration phase. The value
set by that environment variable is used as a number of CPU
threads when it was not specified by user. Number of CPU threads
detected using builtin CMake function [2].

NOTE: CMAKE_BUILD_PARALLEL_LEVEL has been added in a version 3.12.
In earlier CMake versions environment variable is ignored.

1. https://cmake.org/cmake/help/latest/envvar/CMAKE_BUILD_PARALLEL_LEVEL.html
2. https://cmake.org/cmake/help/latest/module/ProcessorCount.html

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build
Needed for the following commits.

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build
Needed for the following commit.

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build
The patch introduces CTest support, see [1] and [2]: the CMake
target `test` was replaced by builtin CMake target that executes
ctest-based tests. The target `test` that runs `test-run.py` was
renamed to `test-all`. Build and test steps remains the same
as before:

$ cmake -S . -B build
$ cmake --build build --parallel
$ cmake --build build --target test

One could use `ctest` tool instead:

$ ctest --print-labels
Test project <snipped>
All Labels:
  LuaJIT-tests
  PUC-Rio-Lua-5.1-tests
  app
  app-luatest
  app-tap
  box
  box-luatest
  box-py
  <snipped>

$ ctest                          # Run all tests.
$ ctest -L app-luatest           # Run tests by label.
$ ctest --rerun-fail             # Run only the tests
                                 # that failed previously.
$ ctest --verbose                # Print details to stdout.
$ ctest --output-on-failure      # Print details to stdout
                                 # on test failure only.

The possible benefits of using CTest are:

- test execution triggers rebuilding of dependencies automatically
- close integration with a build system options
  (ASAN/UBSAN/Debug/Release etc.) and without setting builddir
  manually
- versatile tool for running different kind of tests (functional,
  performance, fuzzing tests) made using different test frameworks
  and libraries: TAP, luatest, diff-based tests, Google Benchmark,
  LuaJIT-based and PUC Rio Lua-based tests etc.
- no more stderr and stdout suppressing
- easy access to a raw command line for running tests manually
  for debug purposes (see `ctest --verbose`)

Note that CMake targets test-unit, test-unit-force, test-func,
test-func-force, test-all, 1mops_write_perftest,
box_select_perftest, tarantoolgh-7089-vclock-copy_perftest,
uri_escape_unescape_perftest, column_scan_perftest,
tuple_perftest, bps_tree_perftest, light_perftest, small_perftest,
memtx_perftest are still operating for backward compatibility.

Note that it is not possible to add dependencies to `add_test()`
in CMake, see [3]. CMake 3.7 introduces FIXTURES_REQUIRED [4] and
FIXTURES_SETUP [5], but these test properties cannot be used - the
currently supported CMake version is lower. This is workarounded
by the introduced macro `add_test_suite_target` that adds
a CMake-test for each testsuite that executes a target that builds
test dependencies.

1. https://cmake.org/cmake/help/latest/manual/ctest.1.html
2. https://cmake.org/cmake/help/book/mastering-cmake/chapter/Testing%20With%20CMake%20and%20CTest.html
3. https://gitlab.kitware.com/cmake/cmake/-/issues/8774
4. https://cmake.org/cmake/help/latest/prop_test/FIXTURES_REQUIRED.html
5. https://cmake.org/cmake/help/latest/prop_test/FIXTURES_SETUP.html
6. tarantool/luajit@9f1137b

Related to tarantool#5000.

NO_CHANGELOG=testing
NO_DOC=testing
The patch add CTest support for performance tests. Now it is
possible to execute these tests using `ctest` tool with labels
`performance-lua`, `performance-c` and `performance`.

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build
The patch add CTest support for fuzzing tests. Now it is
possible to execute these tests using `ctest` tool with labels
`fuzzing-c` and `fuzzing`.

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build
The patch add CTest support for fuzzing tests written in Lua.
Now it is possible to execute these tests using `ctest` tool with
labels `fuzzing-lua` and `fuzzing`.

Follows up commit 33670ea ("test/fuzz: add an engine fuzzing test").

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build
The patch add support of luatest's tests by CTest.

The environment variable `VARDIR`, used by luatest [1], is set to
`/tmp/t`, the same value used by test-run.py.

The CMake target `tarantool` is not used a pseudotest "*-deps",
because target `box_generate_lua_sources`, required for building
tarantool and that builds Lua source files, being run in parallel
can corrupt generated files.

The tests in luatest suites can be executed by `ctest`
using labels app-luatest, box-luatest, engine-luatest,
metrics-luatest, replication-luatest, sql-luatest, vinyl-luatest.

1. test-run/lib/luatest/luatest/server.lua.

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build
The patch introduce a target `test-force-ctest` that runs
functional regression tests and this target is used in
GHA workflow `static_build_cmake_linux`.

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build
@ligurio
Copy link
Member Author

ligurio commented Dec 18, 2024

(rebased to a master branch)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Enables all tests for a pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants