Skip to content

cmake: update cpm move third party cli11 xxhash to cpm.#1911

Closed
bruno-at-bareos wants to merge 19 commits intobareos:masterfrom
bruno-at-bareos:dev/bruno/update-cpm-cl11
Closed

cmake: update cpm move third party cli11 xxhash to cpm.#1911
bruno-at-bareos wants to merge 19 commits intobareos:masterfrom
bruno-at-bareos:dev/bruno/update-cpm-cl11

Conversation

@bruno-at-bareos
Copy link
Contributor

@bruno-at-bareos bruno-at-bareos commented Aug 1, 2024

Update CPM, move CLI11 and xxHASH to CPM with updates

Related to https://github.com/bareos/internal/issues/86

  • Add a build-flag to CMake that enables CPMLicenseManager, adds a target to (re-)build the third-party licenses file (and disables all other targets)
  • Use that to check for changes just like we do for the json schema for the configuration.
  • Update pr-tool to merge third party licenses file into main LICENSE file

Please check

  • Short description and the purpose of this PR is present above this paragraph
  • Your name is present in the AUTHORS file (optional)

If you have any questions or problems, please give a comment in the PR.

Helpful documentation and best practices

Checklist for the reviewer of the PR (will be processed by the Bareos team)

Make sure you check/merge the PR using devtools/pr-tool to have some simple automated checks run and a proper changelog record added.

General
  • Is the PR title usable as CHANGELOG entry?
  • Purpose of the PR is understood
  • Commit descriptions are understandable and well formatted
  • Required backport PRs have been created
  • Correct milestone is set
  • Update license see CPMLicence integration
Source code quality
  • Source code changes are understandable
  • Variable and function names are meaningful
  • Code comments are correct (logically and spelling)
  • Required documentation changes are present and part of the PR
Tests
  • Decision taken that a test is required (if not, then remove this paragraph)
  • The choice of the type of test (unit test or systemtest) is reasonable
  • Testname matches exactly what is being tested
  • On a fail, output of the test leads quickly to the origin of the fault

@bruno-at-bareos bruno-at-bareos added this to the 24.0.0 milestone Aug 1, 2024
@pstorz pstorz changed the title cmake: update cpm move thrid party cli11 xxhash to cpm. cmake: update cpm move third party cli11 xxhash to cpm. Aug 5, 2024
@bruno-at-bareos bruno-at-bareos force-pushed the dev/bruno/update-cpm-cl11 branch from abd4986 to 2e6b53b Compare August 5, 2024 12:41
@bruno-at-bareos
Copy link
Contributor Author

bruno-at-bareos commented Aug 5, 2024

@sebsura I'm stuck with add_library(xxHash::xxhash ALIAS xxhash) that alias when using -DUSE_SYSTEM_XXHASH seems to exist and bareos build locally correctly, and almost all tests pass (one failed due to a change in cli11).

but when used with CPM it is missing and then failed

Entering /bareos/git/bareos/contrib/dir-plugins
-- Configuring done (1.4s)
CMake Error at core/src/lib/CMakeLists.txt:154 (target_link_libraries):
  Target "bareos" links to:

    xxHash::xxhash

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

@bruno-at-bareos
Copy link
Contributor Author

We should also add / try CPMLicenses
This might lead to the cleanest solution for managing licences.

@bruno-at-bareos
Copy link
Contributor Author

@tuxmaster5000 the introduction of this PR will removed the -DUSE_SYSTEM_XXX variable CPM will simply search local system package if present. As you were one of the writer of those cmake variable, would you mind to test this PR, and eventually share your feedback?
Thanks.

@bruno-at-bareos
Copy link
Contributor Author

We will need help from @arogge to setup central cache and also the whole offline containers.
build well locally.

Comment on lines +65 to +73
set(XXHASH_ENABLE_DISPATCH
OFF
CACHE INTERNAL ""
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's really bad as it will break xxHash performance.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not sure about which flags and options you use when you introduce xxHash,
it is now up to us to bring them to this new tool. we have used the example given at cpm to integrate xxHash.
did you remember or have you some notes about the flags needed for xxHash ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we have an additional question here: before we build xxHash as static lib, now do we don't just link to it ?
So the performance is not related to us, but the people build libxxhash ?

Comment on lines +44 to +57
CPMFindPackage(
NAME CPMLicenses.cmake
GITHUB_REPOSITORY cpm-cmake/CPMLicenses.cmake
VERSION 0.0.5
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I like the idea of this, I'm not sure it won't work for us.
At least you will get different results depending on which packages were downloaded and which one were present locally.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In term of workflow, what I can imagine, is that any new 3rd party that will enter the code, will be introduced by adding it here. So one time make write-licences will be done and the generated License file will be created, the author will then pick the licence and will add it to the main repo in global License file.
It is quite rare that a component change its license and we will miss that one.
So whatever local or download module present the generated content may change, it is still up to the author of the PR to bring the license. We may want a checkbox in case of new component introduction.
We can use dplcompat to test the workflow and see how it goes?

Comment on lines 19 to 21

message("Entering ${CMAKE_CURRENT_SOURCE_DIR}")

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that's the only thing left here, we can probably also remove the CMakeLists.txt and the directory.

@tuxmaster5000
Copy link
Contributor

@bruno-at-bareos I have tested the last cpm stuff from the master which will use fmt as the first one.
It will break the compatibility with Fedora. I think the fmt version from cpm is to old. Fedora comes with 10.0.0 and cpm will try to use 6.2.1( as far as I read the code right).
Compiling the master against 10.0.0 of Fedora fails with:

[ 54%] Building CXX object core/src/stored/CMakeFiles/bareossd.dir/device_resource.cc.o
cd /home/<user>/GitHub/bareos/build-root/core/src/stored && /usr/bin/ccache base_dir=/home/<user>/GitHub/bareos hash_dir=true namespace=bareos /usr/lib64/ccache/c++ -DFMT_HEADER_ONLY=1 -DHAVE_GCC=1 -D_FILE_OFFSET_BITS=64 -Dbareossd_EXPORTS -I/usr/include/tirpc -I/home/<user>/GitHub/bareos/core/src -I/home/<user>/GitHub/bareos/build-root/core/src -fdebug-prefix-map=/home/<user>/GitHub/bareos=/usr/src/bareos -fmacro-prefix-map=/home/<user>/GitHub/bareos=/usr/src/bareos -Wsuggest-override -Wformat -Werror=format-security -Wno-invalid-offsetof -Werror -Wall -Wextra -O2 -g  -std=gnu++17 -fPIC -MD -MT core/src/stored/CMakeFiles/bareossd.dir/device_resource.cc.o -MF CMakeFiles/bareossd.dir/device_resource.cc.o.d -o CMakeFiles/bareossd.dir/device_resource.cc.o -c /home/<user>/GitHub/bareos/core/src/stored/device_resource.cc
In file included from /usr/include/c++/13/string:51,
                 from ../../../../core/src/lib/message_destination_info.h:28,
                 from ../../../../core/src/lib/message.h:36,
                 from ../../../../core/src/include/baconfig.h:34,
                 from ../../../../core/src/include/bareos.h:125,
                 from ../../../../core/src/lib/parse_conf.h:27,
                 from ../../../../core/src/stored/device_resource.cc:24:
In static member function 'static _Up* std::__copy_move<_IsMove, true, std::random_access_iterator_tag>::__copy_m(_Tp*, _Tp*, _Up*) [with _Tp = unsigned int; _Up = unsigned int; bool _IsMove = false]',
    inlined from '_OI std::__copy_move_a2(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:506:30,
    inlined from '_OI std::__copy_move_a1(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:533:42,
    inlined from '_OI std::__copy_move_a(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:540:31,
    inlined from '_OI std::copy(_II, _II, _OI) [with _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:633:7,
    inlined from 'static _ForwardIterator std::__uninitialized_copy<true>::__uninit_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:147:27,
    inlined from '_ForwardIterator std::uninitialized_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:185:15,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::grow(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:952:28,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:927:39,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:926:24,
    inlined from 'void fmt::v10::detail::buffer<T>::try_resize(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:918:16,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::resize(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:1023:63,
    inlined from 'void fmt::v10::detail::bigint::assign(UInt) [with UInt = long unsigned int; typename std::enable_if<(std::is_same<UInt, long unsigned int>::value || std::is_same<UInt, __int128 unsigned>::value), int>::type <anonymous> = 0]' at /usr/include/fmt/format.h:3096:19,
    inlined from 'void fmt::v10::detail::bigint::operator=(Int) [with Int = int]' at /usr/include/fmt/format.h:3117:11,
    inlined from 'void fmt::v10::detail::bigint::assign_pow10(int)' at /usr/include/fmt/format.h:3190:32:
/usr/include/c++/13/bits/stl_algobase.h:437:30: error: 'void* __builtin_memmove(void*, const void*, long unsigned int)' forming offset 4 is out of the bounds [0, 4] [-Werror=array-bounds=]
  437 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
      |             ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In static member function 'static _Up* std::__copy_move<_IsMove, true, std::random_access_iterator_tag>::__copy_m(_Tp*, _Tp*, _Up*) [with _Tp = unsigned int; _Up = unsigned int; bool _IsMove = false]',
    inlined from '_OI std::__copy_move_a2(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:506:30,
    inlined from '_OI std::__copy_move_a1(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:533:42,
    inlined from '_OI std::__copy_move_a(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:540:31,
    inlined from '_OI std::copy(_II, _II, _OI) [with _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:633:7,
    inlined from 'static _ForwardIterator std::__uninitialized_copy<true>::__uninit_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:147:27,
    inlined from '_ForwardIterator std::uninitialized_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:185:15,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::grow(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:952:28,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:927:39,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:926:24,
    inlined from 'void fmt::v10::detail::buffer<T>::try_resize(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:918:16,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::resize(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:1023:63,
    inlined from 'void fmt::v10::detail::bigint::assign(UInt) [with UInt = long unsigned int; typename std::enable_if<(std::is_same<UInt, long unsigned int>::value || std::is_same<UInt, __int128 unsigned>::value), int>::type <anonymous> = 0]' at /usr/include/fmt/format.h:3096:19,
    inlined from 'void fmt::v10::detail::bigint::operator=(Int) [with Int = int]' at /usr/include/fmt/format.h:3117:11,
    inlined from 'void fmt::v10::detail::bigint::assign_pow10(int)' at /usr/include/fmt/format.h:3197:11:
/usr/include/c++/13/bits/stl_algobase.h:437:30: error: 'void* __builtin_memmove(void*, const void*, long unsigned int)' forming offset 4 is out of the bounds [0, 4] [-Werror=array-bounds=]
  437 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
      |             ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In static member function 'static _Up* std::__copy_move<_IsMove, true, std::random_access_iterator_tag>::__copy_m(_Tp*, _Tp*, _Up*) [with _Tp = unsigned int; _Up = unsigned int; bool _IsMove = false]',
    inlined from '_OI std::__copy_move_a2(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:506:30,
    inlined from '_OI std::__copy_move_a1(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:533:42,
    inlined from '_OI std::__copy_move_a(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:540:31,
    inlined from '_OI std::copy(_II, _II, _OI) [with _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:633:7,
    inlined from 'static _ForwardIterator std::__uninitialized_copy<true>::__uninit_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:147:27,
    inlined from '_ForwardIterator std::uninitialized_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:185:15,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::grow(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:952:28,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:927:39,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:926:24,
    inlined from 'void fmt::v10::detail::buffer<T>::try_resize(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:918:16,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::resize(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:1023:63,
    inlined from 'void fmt::v10::detail::bigint::assign(UInt) [with UInt = long unsigned int; typename std::enable_if<(std::is_same<UInt, long unsigned int>::value || std::is_same<UInt, __int128 unsigned>::value), int>::type <anonymous> = 0]' at /usr/include/fmt/format.h:3096:19,
    inlined from 'void fmt::v10::detail::bigint::operator=(Int) [with Int = int]' at /usr/include/fmt/format.h:3117:11,
    inlined from 'void fmt::v10::detail::format_dragon(basic_fp<__int128 unsigned>, unsigned int, int, buffer<char>&, int&)' at /usr/include/fmt/format.h:3291:11:
/usr/include/c++/13/bits/stl_algobase.h:437:30: error: 'void* __builtin_memmove(void*, const void*, long unsigned int)' forming offset 4 is out of the bounds [0, 4] [-Werror=array-bounds=]
  437 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
      |             ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In static member function 'static _Up* std::__copy_move<_IsMove, true, std::random_access_iterator_tag>::__copy_m(_Tp*, _Tp*, _Up*) [with _Tp = unsigned int; _Up = unsigned int; bool _IsMove = false]',
    inlined from '_OI std::__copy_move_a2(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:506:30,
    inlined from '_OI std::__copy_move_a1(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:533:42,
    inlined from '_OI std::__copy_move_a(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:540:31,
    inlined from '_OI std::copy(_II, _II, _OI) [with _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:633:7,
    inlined from 'static _ForwardIterator std::__uninitialized_copy<true>::__uninit_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:147:27,
    inlined from '_ForwardIterator std::uninitialized_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:185:15,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::grow(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:952:28,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:927:39,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:926:24,
    inlined from 'void fmt::v10::detail::buffer<T>::try_resize(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:918:16,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::resize(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:1023:63,
    inlined from 'void fmt::v10::detail::bigint::assign(UInt) [with UInt = long unsigned int; typename std::enable_if<(std::is_same<UInt, long unsigned int>::value || std::is_same<UInt, __int128 unsigned>::value), int>::type <anonymous> = 0]' at /usr/include/fmt/format.h:3096:19,
    inlined from 'void fmt::v10::detail::bigint::operator=(Int) [with Int = int]' at /usr/include/fmt/format.h:3117:11,
    inlined from 'void fmt::v10::detail::format_dragon(basic_fp<__int128 unsigned>, unsigned int, int, buffer<char>&, int&)' at /usr/include/fmt/format.h:3294:19:
/usr/include/c++/13/bits/stl_algobase.h:437:30: error: 'void* __builtin_memmove(void*, const void*, long unsigned int)' forming offset 4 is out of the bounds [0, 4] [-Werror=array-bounds=]
  437 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
      |             ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In static member function 'static _Up* std::__copy_move<_IsMove, true, std::random_access_iterator_tag>::__copy_m(_Tp*, _Tp*, _Up*) [with _Tp = unsigned int; _Up = unsigned int; bool _IsMove = false]',
    inlined from '_OI std::__copy_move_a2(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:506:30,
    inlined from '_OI std::__copy_move_a1(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:533:42,
    inlined from '_OI std::__copy_move_a(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:540:31,
    inlined from '_OI std::copy(_II, _II, _OI) [with _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:633:7,
    inlined from 'static _ForwardIterator std::__uninitialized_copy<true>::__uninit_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:147:27,
    inlined from '_ForwardIterator std::uninitialized_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:185:15,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::grow(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:952:28,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:927:39,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:926:24,
    inlined from 'void fmt::v10::detail::buffer<T>::try_resize(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:918:16,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::resize(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:1023:63,
    inlined from 'void fmt::v10::detail::bigint::assign(UInt) [with UInt = long unsigned int; typename std::enable_if<(std::is_same<UInt, long unsigned int>::value || std::is_same<UInt, __int128 unsigned>::value), int>::type <anonymous> = 0]' at /usr/include/fmt/format.h:3096:19,
    inlined from 'void fmt::v10::detail::bigint::operator=(Int) [with Int = int]' at /usr/include/fmt/format.h:3117:11,
    inlined from 'void fmt::v10::detail::format_dragon(basic_fp<__int128 unsigned>, unsigned int, int, buffer<char>&, int&)' at /usr/include/fmt/format.h:3310:17:
/usr/include/c++/13/bits/stl_algobase.h:437:30: error: 'void* __builtin_memmove(void*, const void*, long unsigned int)' forming offset 4 is out of the bounds [0, 4] [-Werror=array-bounds=]
  437 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
      |             ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In static member function 'static _Up* std::__copy_move<_IsMove, true, std::random_access_iterator_tag>::__copy_m(_Tp*, _Tp*, _Up*) [with _Tp = unsigned int; _Up = unsigned int; bool _IsMove = false]',
    inlined from '_OI std::__copy_move_a2(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:506:30,
    inlined from '_OI std::__copy_move_a1(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:533:42,
    inlined from '_OI std::__copy_move_a(_II, _II, _OI) [with bool _IsMove = false; _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:540:31,
    inlined from '_OI std::copy(_II, _II, _OI) [with _II = unsigned int*; _OI = unsigned int*]' at /usr/include/c++/13/bits/stl_algobase.h:633:7,
    inlined from 'static _ForwardIterator std::__uninitialized_copy<true>::__uninit_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:147:27,
    inlined from '_ForwardIterator std::uninitialized_copy(_InputIterator, _InputIterator, _ForwardIterator) [with _InputIterator = unsigned int*; _ForwardIterator = unsigned int*]' at /usr/include/c++/13/bits/stl_uninitialized.h:185:15,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::grow(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:952:28,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:927:39,
    inlined from 'void fmt::v10::detail::buffer<T>::try_reserve(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:926:24,
    inlined from 'void fmt::v10::detail::buffer<T>::try_resize(size_t) [with T = unsigned int]' at /usr/include/fmt/core.h:918:16,
    inlined from 'void fmt::v10::basic_memory_buffer<T, SIZE, Allocator>::resize(size_t) [with T = unsigned int; long unsigned int SIZE = 32; Allocator = std::allocator<unsigned int>]' at /usr/include/fmt/format.h:1023:63,
    inlined from 'void fmt::v10::detail::bigint::assign(UInt) [with UInt = long unsigned int; typename std::enable_if<(std::is_same<UInt, long unsigned int>::value || std::is_same<UInt, __int128 unsigned>::value), int>::type <anonymous> = 0]' at /usr/include/fmt/format.h:3096:19,
    inlined from 'void fmt::v10::detail::bigint::operator=(Int) [with Int = int]' at /usr/include/fmt/format.h:3117:11,
    inlined from 'void fmt::v10::detail::format_dragon(basic_fp<__int128 unsigned>, unsigned int, int, buffer<char>&, int&)' at /usr/include/fmt/format.h:3317:11:
/usr/include/c++/13/bits/stl_algobase.h:437:30: error: 'void* __builtin_memmove(void*, const void*, long unsigned int)' forming offset 4 is out of the bounds [0, 4] [-Werror=array-bounds=]
  437 |             __builtin_memmove(__result, __first, sizeof(_Tp) * _Num);
      |             ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors
gmake[2]: *** [core/src/stored/CMakeFiles/bareossd.dir/build.make:244: core/src/stored/CMakeFiles/bareossd.dir/device_resource.cc.o] Error 1
gmake[2]: Leaving directory '/home/<user>/GitHub/bareos/build-root'
gmake[1]: *** [CMakeFiles/Makefile2:3578: core/src/stored/CMakeFiles/bareossd.dir/all] Error 2
gmake[1]: Leaving directory '/home/<user>/GitHub/bareos/build-root'
gmake: *** [Makefile:146: all] Error 2

@bruno-at-bareos
Copy link
Contributor Author

I have tested the last cpm stuff from the master which will use fmt as the first one.
It will break the compatibility with Fedora. I think the fmt version from cpm is to old. Fedora comes with 10.0.0 and cpm will try to use 6.2.1( as far as I read the code right).

Sorry if I wasn't clear enough, only this branch is valid for tests with CPM, not the actual master branch. BTW fmt versioning is a beast :-)

@tuxmaster5000
Copy link
Contributor

I think in my case it was an dirty build. (A total clean build will let build the master using the local fmt of Fedora 39)
But it looks the PR is out of sync(against the master):

checking file cmake/BareosCpmPackages.cmake
Hunk #2 FAILED at 53.
1 out of 2 hunks FAILED
checking file cmake/BareosCpmPackages.cmake
Hunk #1 FAILED at 53.
1 out of 1 hunk FAILED
checking file cmake/BareosCpmPackages.cmake
Hunk #1 FAILED at 45.
1 out of 1 hunk FAILED
checking file cmake/BareosCpmPackages.cmake
Hunk #1 FAILED at 32.
1 out of 1 hunk FAILED
checking file cmake/BareosCpmPackages.cmake
Hunk #1 FAILED at 32.
Hunk #2 FAILED at 39.
Hunk #3 FAILED at 60.
3 out of 3 hunks FAILED
checking file cmake/BareosCpmPackages.cmake
Hunk #1 FAILED at 55.
1 out of 1 hunk FAILED
checking file cmake/BareosCpmPackages.cmake
Hunk #1 FAILED at 55.
1 out of 1 hunk FAILED

I have tried this path:
https://patch-diff.githubusercontent.com/raw/bareos/bareos/pull/1911.patch

@bruno-at-bareos
Copy link
Contributor Author

@arogge I've some questions that appears when I switch back to CPMAddPackages.

Such combination for the moment are failing the cmake, while the cache is filled.

CPM_DOWNLOAD_ALL=ON
CPM_SOURCE_CACHE=~/.cache/CPM/build/
CPM_LOCAL_PACKAGES_ONLY=ON
CPM_USE_LOCAL_PACKAGES=ON

FETCHCONTENT_FULLY_DISCONNECTED=OFF

I developed a scenario to test the different cases

start with a container which doesn't have cli11,fmt,xxhash system package installed,

declare a cache location with
export CPM_SOURCE_CACHE=~/.cache/CPM/build/
as state in cmake/BareosCpmPackage, if CPM_DOWNLOAD_ALL is ON

  • we set to OFF CPM_USE_LOCAL_PACKAGES CPM_LOCAL_PACKAGES_ONLY and FETCHCONTENT_FULLY_DISCONNECTED as it doesn't make sense to keep them ON
  • we expect that the cache will be filled up with all packages.
-- CPM: Adding package CLI11@2.4.2 (v2.4.2 to ~/.cache/CPM/build/cli11/c4f61952e7941a2f0a3f95cdb246a26bc84a0afe)
-- CPM: Adding package CPMLicenses.cmake@0.0.5 (v0.0.5 to ~/.cache/CPM/build/cpmlicenses.cmake/b671285eb3cdd3619425823d97670674bbf8ce04)
-- CPM: Adding package fmt@6.2.1 (10.2.1 to ~/.cache/CPM/build/fmt/c3ebf53335b44df838d9982d1a0e35afe190e34f)
-- Version: 10.2.1
-- Build type:
-- CPM: Adding package xxHash@0.8.2 (v0.8.2 to ~/.cache/CPM/build/xxhash/bd2cc5037bfb339335d743b9f59f6d54b3977030)
-- Populating xxhash

so far so good.

Now we want to benefit from that cache and as such use CPM_LOCAL_PACKAGES_ONLY=ON

But then the cmake produce error with find_package (almost expected) but finally still find the package as it is in the cache.

CMake Error at ~/.cache/CPM/build/cpm/CPM_0.40.1.cmake:725 (message):
  CPM: CLI11 not found via find_package(CLI11 2.4.2)
Call Stack (most recent call first):
  cmake/BareosCpmPackages.cmake:46 (CPMAddPackage)
  CMakeLists.txt:201 (include)
-- CPM: Adding package CLI11@2.4.2 (v2.4.2 at ~/.cache/CPM/build/cli11/c4f61952e7941a2f0a3f95cdb246a26bc84a0afe)

CMake Error at ~/.cache/CPM/build/cpm/CPM_0.40.1.cmake:725 (message):
  CPM: CPMLicenses.cmake not found via find_package(CPMLicenses.cmake 0.0.5)
Call Stack (most recent call first):
  cmake/BareosCpmPackages.cmake:53 (CPMAddPackage)
  CMakeLists.txt:201 (include)
-- CPM: Adding package CPMLicenses.cmake@0.0.5 (v0.0.5 at ~/.cache/CPM/build/cpmlicenses.cmake/b671285eb3cdd3619425823d97670674bbf8ce04)

CMake Error at ~/.cache/CPM/build/cpm/CPM_0.40.1.cmake:725 (message):
  CPM: fmt not found via find_package(fmt 6.2.1)
Call Stack (most recent call first):
  cmake/BareosCpmPackages.cmake:59 (CPMAddPackage)
  CMakeLists.txt:201 (include)
-- Version: 10.2.1
-- Build type:
-- CPM: Adding package fmt@6.2.1 (10.2.1 at ~/.cache/CPM/build/fmt/c3ebf53335b44df838d9982d1a0e35afe190e34f)

CMake Error at ~/.cache/CPM/build/cpm/CPM_0.40.1.cmake:725 (message):
  CPM: xxHash not found via find_package(xxHash 0.8.2)
Call Stack (most recent call first):
  cmake/BareosCpmPackages.cmake:67 (CPMAddPackage)
  CMakeLists.txt:201 (include)
-- CPM: Adding package xxHash@0.8.2 (v0.8.2 at ~/.cache/CPM/build/xxhash/bd2cc5037bfb339335d743b9f59f6d54b3977030)
xxHash local configuration included
XXHASH_SRC_DIR ~/.cache/CPM/build/xxhash/bd2cc5037bfb339335d743b9f59f6d54b3977030

I would have expected that if find_package doesn't find a local system, the addpackage find the cache (what it do) but not finish in errors.

Changing CPM_USE_LOCAL_PACKAGES doesn't help neither.

I don't know how to handle the case in cmake : Failure, but we found a workaround :-)

Maybe there's an issue with CPM ? Or I overlooked the usage of centralized cache, any though ?

- remove from third-party
- move USE_SYSTEM_CLI11 to BareosCpmPackages.cmake
- remove set_target_properties IMPORTED_GLOBAL as TRUE by default
  with find/add_librairy
- gtests: remove extra third \n in expected cli help output
  with CLI11 2.4.2 cli --help will output no more extra line

Signed-off-by: Bruno Friedmann <bruno.friedmann@bareos.com>
- remove from third-party
- move `USE_SYSTEM_XXHASH` to BareosCpmPackages
- remove set_target_properties IMPORTED_GLOBAL as TRUE by default
  with find/add_library
- CPMFindPackage allow to use local/system package if present
  in the system.
- Removed usage of cmake parameter USE_SYSTEM_FMT, USE_SYSTEM_CL11
  USE_SYSTEM_FMT

Signed-off-by: Bruno Friedmann <bruno.friedmann@bareos.com>
- CPMFindPackage allow to use local/system package if present
  in the system.
- Removed usage of cmake parameter USE_SYSTEM_FMT, USE_SYSTEM_CL11
  USE_SYSTEM_FMT

Signed-off-by: Bruno Friedmann <bruno.friedmann@bareos.com>
- allow retrieval of the CPM third-party modules licenses
- use `make write-licenses` to create output bundle file in
  "${CMAKE_CURRENT_BINARY_DIR}/LICENSES_third_party.txt"

Signed-off-by: Bruno Friedmann <bruno.friedmann@bareos.com>
Signed-off-by: Bruno Friedmann <bruno.friedmann@bareos.com>
Signed-off-by: Bruno Friedmann <bruno.friedmann@bareos.com>
-Dlicenses-only will allow to activate CPM modules and
`make write-licenses` to create the third party licenses.

Signed-off-by: Bruno Friedmann <bruno.friedmann@bareos.com>
@bruno-at-bareos bruno-at-bareos force-pushed the dev/bruno/update-cpm-cl11 branch from 1c7f8f5 to 50f8e41 Compare September 9, 2024 14:09
to be used as reference for pr-tools license update.

Signed-off-by: Bruno Friedmann <bruno.friedmann@bareos.com>
@arogge
Copy link
Member

arogge commented Oct 7, 2024

I'll close this one, as we probably go with the replacement PR #1963

@arogge arogge closed this Oct 7, 2024
@bruno-at-bareos bruno-at-bareos deleted the dev/bruno/update-cpm-cl11 branch November 25, 2024 10:27
@arogge arogge removed this from the 24.0.0 milestone Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants