Does UNIX include MSYS? The documentation isn't clear about that, it only mentions CYGWIN [0]. The only bad case I can think of is #14365, when targeting MSVC, so I think it would be good to have the pkg-config detection enabled for MSYS.

[0] https://cmake.org/cmake/help/latest/variable/UNIX.html

I've made a PR to make these CMake flags transparent: #14417
→ Yes, MSYS is UNIX: https://github.com/curl/curl/actions/runs/10265251756/job/28401088397?pr=14417#step:16:43

Using UNIX here just applies the pattern already used in our Find* modules.
Once we know more about the nature of side-effects, we can update all of these
pkg-config uses with conditions giving the best result in most cases.
Including vcpkg, which supports pkg-config on native Windows.

#14365 looks like an explictly misconfigured env, though still awaiting feedback.
#14405 doesn't say about an explicit configuration, also awaiting feedback.

An earlier failed attempt to tackle this: #14140. With no answers and adding to
the confusion. My concern is also cross-builds, which broke a CI elsewhere.

#14365 resolved by deleting the local/manual PKG_CONFIG_PATH setting.

An earlier failed attempt to tackle this: #14140. With no answers and adding to
the confusion. My concern is also cross-builds, which broke a CI elsewhere.

Which answers were missing?
I mean I often do cross-builds when updating the vcpkg port of CURL, also targeting mingw, also with static linkage. If pkg-config comes with the wrong personality for the target, or gets the wrong environment for the target, then it is not CURL'S responsibility to fix it. This doesn't scale. Garbage in, garbage out.

Does UNIX include MSYS?

This means: the limited environment which doesn't understand C:/ or C:\.
UNIX doesn't include MinGW.

It's good that vcpkg is cleared, and AFAIK it's the only platform besides UNIX which is known to tolerate (or support) pkg-config out of the box. Excluding the cases of explicit misconfigurations, there is at least one known case which doesn't play well with it: cross-builds run on platforms which have native pkg-config configured by default. A common example is Windows builds from Linux or macOS.

It doesn't seem good behaviour to make these build-cases suddenly broken, and make them require some tweak to make it work again. What's still unclear is how to define/detect the affected build cases. It's also unclear what is the best way to opt-in-to / opt-out-from pkg-config if defaults don't behave as expected.

It's also unclear what is the best way to opt-in-to / opt-out-from pkg-config if defaults don't behave as expected.

I had a clear opinion/suggestion for CMake builds, but I failed to convince you to that fixing problems in CMake builds should be done with CMake features alone.

A common example is Windows builds from Linux or macOS.

This is a normal cross-build scenario. Even with autotools, you would be expected to setup either the enviroment or to use a wrapper which does it for you. I don't see anything special.

It's also unclear what is the best way to opt-in-to / opt-out-from pkg-config if defaults don't behave as expected.

I had a clear opinion/suggestion for CMake builds, but I failed to convince you to that fixing problems in CMake builds should be done with CMake features alone.

That solution broke on CMake internals.

IMO a pkg-config-specific solution is fine for a pkg-config-specific problem,
as long as it solves the opt-in/opt-out issue. A CMake knob would be better,
if there is any.

A common example is Windows builds from Linux or macOS.

This is a normal cross-build scenario. Even with autotools, you would be expected to setup either the enviroment or to use a wrapper which does it for you. I don't see anything special.

The envs, userbase, expectations, build options and the histories are quite
different for CMake and autotools.

pkg-config is also GCC-family- and Unix-biased, while CMake supports
MSVC and Windows, which don't fit into the pkg-config concept by default.

Perhaps that's the reason why pkg-config support in CMake feels like an
afterthought and neither CMake itself, nor most (?) projects have a clear idea
or patterns how to use it accurately. While in autotools, it's a natural
underpinning for dependency detection.

Either way there is/was great efforts put into syncing cmake/autotools, but
IMO it's not helpful to bring on the big hammer and break stuff for CMake on
accounts that autotools always worked a certain way.

Hence my seeking actual answers to these questions. If there are none or
the questions are wrong, anyone is free to propose alternate PRs, till we
try figuring it out step by step.

The FindGnuTLS thing is a cmake bug and won't go away quickly, even when the fix is simple.
It is mitigated by the CMAKE_USE_GNUTLS gate, but I understand that more users want to keep that gate open today. (But these user probably wouldn't use CMAKE_DISABLE_FIND_PACKAGE_PkgConfig anyways? IDK.)

There isn't just a CMake side to the pkg-config problems. There are also many pc files with problems, at least for transitive usage requirements.

While in autotools, it's a natural underpinning for dependency detection.

I'm still patching many up-to-date autotools projects to use pkg-config instead of hardcoded library searches. Indicating that projects don't care about static linkage, or are unaware of different binary names, or that they lack contributions.

I'm glad to see curl better fill Requires now. The vpckg port has an extra script to unroll link libraries, and I would love to drop it. OTOH we still need to unroll the the dependencies as long as there is curl-config, and as long as not all dependecies come with pkg-config.

I'm glad to see curl better fill Requires now. The vpckg port has an extra script to unroll link libraries, and I would love to drop it. OTOH we still need to unroll the the dependencies as long as there is curl-config, and as long as not all dependecies come with pkg-config.

/OFF

Can you be more specific?:

• is there something in curl Requires that still needs fixing?
• is there something we can do with curl-config to improve it?

Sadly Requires also broke some stuff over at Debian, but I think it should be okay by now.

pkg-config is clearly fragile everywhere, but it just reinforces that we need to figure
out how to toggle it and avoid it in (new) problematic cases by default.

pkg-config is clearly fragile everywhere

I don't think any system for this can be introduced without being fragile since it requires that the info provided is correctly setup and filled in, and usually the ones providing the info isn't themselves using it so it is bound to end up wrong every now and then.

I'm sure libcurl's pkg-config fits that pattern as well.

• is there something in curl Requires that still needs fixing

I didn't find time to validate it so far so the port just doesn't use it. (Curl release must be adopted quickly for security concerns, and at that time your (positive) work has the consequence that updating the patches in the port is more difficult than usual. Between releases time goes into housekeeping for other peoples' contributions. And that time isn't available for upstreaming useful stuff such as exporting CURL components to CMake config. Which is necessary to make CURL config a full replacement for FindCURL.cmake.)

• is there something we can do with curl-config to improve it?

Drop it for using pkg-config? AFAIU this is what pkg-config was invented for. (I know that some <foo>-config delegate to pkg-config today, but I don't know now if curl-config is already in the same boat. And that approach might have it is own traps.)

#14405 turned out to be an env with an explicit PKG_CONFIG_LIBDIR setting. Causing OpenSSL header/lib mixup after pkg-config detection (libidn2). There could be other factors leading to this outcome, but deleting that setting fixed the build.

One more related update: #13947 plans to switch system IDN on by default on Windows and Apple. The consequence is that libidn2 will no longer be detected by default for these targets, also avoiding the issue fixed by this PR in default builds.

Update regarding #14405: Even though the env had a manual PKG_CONFIG_LIBDIR, another contributing factor appears to be Strawberry Perl. It's now a build dependency for OpenSSL, and it comes with a Perl (also via MS-DOS batch wrapper) implementation of pkg-config (usually in C:/Strawberry/perl/bin), libidn2, a copy of OpenSSL and other packages, and also a mingw toolchain. With its c/include getting into the include path, it can cause confusion.

This may change the situation a little bit because this could be present in envs and in PATH on many machines.

(It's present on AppVeyor CI runners, but our MSVC jobs were not affected.)

Perhaps this condition could replace if(UNIX): if(NOT MSVC OR VCPKG_TOOLCHAIN), plus require cross-compilations to setup PKG_CONFIG_PATH/PKG_CONFIG_LIBDIR/PKG_CONFIG_SYSROOT_DIR accurately (or to empty to disable pkg-config.) I can't see a situation where MSVC mixes well with pkg-config outside of vcpkg.

Let me know what you think.

Perhaps this condition could replace if(UNIX): if(NOT MSVC OR VCPKG_TOOLCHAIN), plus require cross-compilations to setup PKG_CONFIG_PATH/PKG_CONFIG_LIBDIR/PKG_CONFIG_SYSROOT_DIR accurately (or to empty to disable pkg-config.) I can't see a situation where MSVC mixes well with pkg-config outside of vcpkg.

No. All those variables (incl. UNIX) bind effects tightly to variables which merely have a correlation with the observations, but are not the root cause for failure or success. And they take control completely out of user's hands. Given the dislike for CMAKE_DISABLE_FIND_PACKAGE_PkgConfig, UNIX should be replaced with a cache variable like CURL_USE_PKGCONFIG. Use the correlations for the variable's default.

In the meantime I did https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9709: Guard all use of pkg_check_modules in CMake. CMAKE_DISABLE_FIND_PACKAGE_PkgConfig should work in the future.

Strawberry Perl. It's now a build dependency for OpenSSL, and it comes with a Perl (also via MS-DOS batch wrapper) implementation of pkg-config (usually in C:/Strawberry/perl/bin), libidn2, a copy of OpenSSL and other packages, and also a mingw toolchain. With its c/include getting into the include path, it can cause confusion

A known pain. Note that you don't get completely rid of it with UNIX. There are people who target Android (or embedded Linux) from Windows.

Perhaps this condition could replace if(UNIX): if(NOT MSVC OR VCPKG_TOOLCHAIN), plus require cross-compilations to setup PKG_CONFIG_PATH/PKG_CONFIG_LIBDIR/PKG_CONFIG_SYSROOT_DIR accurately (or to empty to disable pkg-config.) I can't see a situation where MSVC mixes well with pkg-config outside of vcpkg.

No. All those variables (incl. UNIX) bind effects tightly to variables which merely have a correlation with the observations, but are not the root cause for failure or success. And they take control completely out of user's hands. Given the dislike for CMAKE_DISABLE_FIND_PACKAGE_PkgConfig, UNIX should be replaced with a cache variable like CURL_USE_PKGCONFIG. Use the correlations for the variable's default.

I agree CURL_USE_PKGCONFIG would be nicer, but is it a legit option within Find* modules? They are supposed to be distributed in curl binary distributions and executed outside curl's own CMake build process, where CURL_USE_PKGCONFIG is not defined.

In the meantime I did https://gitlab.kitware.com/cmake/cmake/-/merge_requests/9709: Guard all use of pkg_check_modules in CMake. CMAKE_DISABLE_FIND_PACKAGE_PkgConfig should work in the future.

Nice!

In the meantime we learned (Thanks @micahsnyder!) that -DPKG_CONFIG_EXECUTABLE={""|none|OFF} also works to disable pkg-config.

A known pain. Note that you don't get completely rid of it with UNIX. There are people who target Android (or embedded Linux) from Windows.

I'm aware. It's a condition that was used in curl for a decade. This doesn't in itself make it accurate, but there were apparently no reports about issues, so either everyone managed to work it around or it did not cause grave problems. On the other hand, omitting the condition triggered two bug reports in a week.

I agree CURL_USE_PKGCONFIG would be nicer, but is it a legit option within Find* modules? They are supposed to be distributed in curl binary distributions and executed outside curl's own CMake build process, where CURL_USE_PKGCONFIG is not defined.

Why shouldn't it be legit? The find modules should be seen as private implementation details. CURL ships CMake config. If it has to ship and use extra find modules, they should be scoped to finding transitive depedencies, and they shouldn't be allowed to find something signitificantly different then used during the build. Implementation-wise, CURL CMake config would set CURL_USE_PKGCONFIG before calling find_dependendency, like it has to set CMAKE_MODULE_PATH. And restore before existing the scope (unless using another cache variable).

OK, so we must replicate the CURL_USE_PKGCONFIG initialization inside curl-config.cmake, once we start shipping any local Find module.

Do you think there anything special to do in the restore-before-existing-scope step there? (I know very little about variable caching)