Build with opentelemetry support (on unix) by h-vetinari · Pull Request #1048 · conda-forge/arrow-cpp-feedstock

h-vetinari · 2023-05-05T01:51:43Z

No description provided.

conda-forge-webservices · 2023-05-05T01:51:48Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

h-vetinari · 2023-05-06T22:29:44Z

Looks like either opentelemetry or arrow need an #include <winsock2.h> or #include <ws2tcpip.h> somewhere

xhochy · 2023-05-08T08:05:01Z

@h-vetinari You might get better error messages if you deactivate the unity builds. The error message triggers a gut feeling of "oh, maybe there is a macro defined twice".

h-vetinari · 2023-05-11T21:35:30Z

You might get better error messages if you deactivate the unity builds.

Interestingly, this gets much further now (didn't know that knob). It now fails with missing symbols while trying to create arrow_acero:

[322/458] Linking CXX shared library release\arrow_acero.dll
FAILED: release/arrow_acero.dll release/arrow_acero.lib 
cmd.exe /C "cd . && D:\bld\apache-arrow_1683618300204\_build_env\Library\bin\cmake.exe -E vs_link_dll --intdir=src\arrow\acero\CMakeFiles\arrow_acero_shared.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\link.exe /nologo @CMakeFiles\arrow_acero_shared.rsp  /out:release\arrow_acero.dll /implib:release\arrow_acero.lib /pdb:release\arrow_acero.pdb /dll /version:1200.0 /machine:x64  /NODEFAULTLIB:LIBCMT /INCREMENTAL:NO  && cd ."
LINK: command "C:\PROGRA~1\MICROS~2\2022\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\HostX64\x64\link.exe /nologo @CMakeFiles\arrow_acero_shared.rsp /out:release\arrow_acero.dll /implib:release\arrow_acero.lib /pdb:release\arrow_acero.pdb /dll /version:1200.0 /machine:x64 /NODEFAULTLIB:LIBCMT /INCREMENTAL:NO /MANIFEST /MANIFESTFILE:release\arrow_acero.dll.manifest" failed (exit code 1120) with the following output:
   Creating library release\arrow_acero.lib and object release\arrow_acero.exp
project_node.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::UnwrapSpan(class arrow::util::tracing::SpanDetails *)" (?UnwrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@@Z)
swiss_join.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::UnwrapSpan(class arrow::util::tracing::SpanDetails *)" (?UnwrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@@Z)
aggregate_node.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::UnwrapSpan(class arrow::util::tracing::SpanDetails *)" (?UnwrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@@Z)
exec_plan.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::UnwrapSpan(class arrow::util::tracing::SpanDetails *)" (?UnwrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@@Z)
filter_node.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::UnwrapSpan(class arrow::util::tracing::SpanDetails *)" (?UnwrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@@Z)
hash_join.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::UnwrapSpan(class arrow::util::tracing::SpanDetails *)" (?UnwrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@@Z)
project_node.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::RewrapSpan(class arrow::util::tracing::SpanDetails *,class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span>)" (?RewrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@V4567@@Z)
source_node.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::RewrapSpan(class arrow::util::tracing::SpanDetails *,class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span>)" (?RewrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@V4567@@Z)
swiss_join.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::RewrapSpan(class arrow::util::tracing::SpanDetails *,class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span>)" (?RewrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@V4567@@Z)
util.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::RewrapSpan(class arrow::util::tracing::SpanDetails *,class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span>)" (?RewrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@V4567@@Z)
aggregate_node.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::RewrapSpan(class arrow::util::tracing::SpanDetails *,class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span>)" (?RewrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@V4567@@Z)
exec_plan.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::RewrapSpan(class arrow::util::tracing::SpanDetails *,class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span>)" (?RewrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@V4567@@Z)
filter_node.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::RewrapSpan(class arrow::util::tracing::SpanDetails *,class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span>)" (?RewrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@V4567@@Z)
hash_join.cc.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::RewrapSpan(class arrow::util::tracing::SpanDetails *,class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span>)" (?RewrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@V4567@@Z)

release\arrow_acero.dll : fatal error LNK1120: 2 unresolved externals

The __cdecl arrow::internal::tracing::UnwrapSpan(class arrow::util::tracing::SpanDetails *) makes it look like this comes from arrow somehow (I'm guessing there are no shared windows builds with OTEL in upstream CI). Do we even need tracing in conda-forge? Isn't that usually for sanitizer instrumentation?

xhochy · 2023-05-12T11:27:54Z

The UnwrapScan function is probably missing an ARROW_EXPORT declaration: https://github.com/apache/arrow/blob/cdefbb8f4b4183b29fbcdb014af1f6fc0030475c/cpp/src/arrow/util/tracing_internal.h#LL125C1-L125C1

Although it is internal, it is used across library boundaries and thus it needs to be exported. Thus should be addressed upstream.

h-vetinari · 2023-05-13T03:52:15Z

Good analysis @xhochy. The build now succeeded, but we run into something else when trying to load the library:

import: 'pyarrow'
[libprotobuf ERROR D:\bld\libprotobuf-split_1670986007159\work\src\google\protobuf\descriptor_database.cc:642] File already exists in database: opentelemetry/proto/common/v1/common.proto
[libprotobuf FATAL D:\bld\libprotobuf-split_1670986007159\work\src\google\protobuf\descriptor.cc:1986] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 
Traceback (most recent call last):
  File "D:\bld\apache-arrow_1683937204901\test_tmp\run_test.py", line 2, in <module>
    import pyarrow
  File "D:\bld\apache-arrow_1683937204901\_test_env\lib\site-packages\pyarrow\__init__.py", line 65, in <module>
    import pyarrow.lib as _lib
ImportError: DLL load failed while importing lib: A dynamic link library (DLL) initialization routine failed.

I haven't seen this before, but it seems there's some double-loading(?) of proto information into a protobuf-internal database?

If so, it sounds like arrow shouldn't be re-loading the otel proto definitions?

CC @lidavidm

h-vetinari · 2023-05-13T04:04:32Z

If so, it sounds like arrow shouldn't be re-loading the otel proto definitions?

To top things off, it looks like there's a third potential source for this (at least in conda-forge), in addition to opentelemetry & arrow: https://github.com/conda-forge/proto-opentelemetry-proto-feedstock

The opentelemetry lib includes this as a build dependency(?), which seems wrong somehow? Shouldn't it be a host-dependence?

Also, if there's a need to match the proto-opentelemetry-proto version here, I think this should become a run-export of libopentelemetry-cpp, rather than duplicating the version ("0.19") here and then trying to keep it in sync?

xhochy · 2023-05-15T07:43:59Z

File already exists in database: opentelemetry/proto/common/v1/common.proto
2023-05-15T06:23:18.8525852Z [libprotobuf FATAL

This happens if two libraries load the same protobuf definition in a different, conflicting version. We have the same problem with all the onnx-* packages. The workaround for this is build all (except at most one) with static protobuf so that each library gets its own protobuf (global) namespace.

h-vetinari · 2023-05-15T07:48:08Z

Thanks for the input!

This happens if two libraries load the same protobuf definition in a different, conflicting version.

Can we not ensure that they get the same protobuf definitions? There's even a specific feedstock for those.

h-vetinari · 2023-05-15T07:51:39Z

BTW: The recent commits also showed that the unity build was directly implicated in the windows failures. Switching it back on lead to failure again. Should I raise an issue for this upstream?

lidavidm · 2023-05-15T12:19:03Z

The opentelemetry lib includes this as a build dependency(?), which seems wrong somehow? Shouldn't it be a host-dependence?

The Protobuf definitions are only needed to generate code, not at runtime. But the version needs to be matched, so if there's a better way...

This happens if two libraries load the same protobuf definition in a different, conflicting version. We have the same problem with all the onnx-* packages. The workaround for this is build all (except at most one) with static protobuf so that each library gets its own protobuf (global) namespace.

Hmm, something seems wrong. Arrow itself shouldn't be loading the OpenTelemetry Protobuf generated code. Either we actually built a bundled copy of OpenTelemetry (doesn't seem so) or we still statically linked OpenTelemetry somehow? (Shouldn't be possible?) Or possibly multiple OpenTelemetry libraries included the Protobuf generated code?

BTW: The recent commits also showed that the unity build was directly implicated in the windows failures. Switching it back on lead to failure again. Should I raise an issue for this upstream?

Yes, please do.

conda-forge-admin · 2024-11-25T00:39:34Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe/meta.yaml:

ℹ️ The recipe is not parsable by parser conda-souschef (grayskull). This parser is not currently used by conda-forge, but may be in the future. We are collecting information to see which recipes are compatible with grayskull.
ℹ️ The recipe is not parsable by parser conda-recipe-manager. The recipe can only be automatically migrated to the new v1 format if it is parseable by conda-recipe-manager.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/12921137857. Examine the logs at this URL for more detail.}

…nda-forge-pinning 2025.01.22.17.59.35

h-vetinari · 2025-01-23T04:35:28Z

Alrighty, close to two years later, we're finally there! 🥳

(well, except windows, as always...)

h-vetinari force-pushed the otel branch from 4612cd2 to 1880bf5 Compare May 7, 2023 09:21

h-vetinari force-pushed the otel branch from 115414d to e65846f Compare May 9, 2023 07:39

h-vetinari mentioned this pull request May 15, 2023

Enable test suite of libarrow #1058

Merged

h-vetinari mentioned this pull request Jun 16, 2023

Dependency TODOs: enablements & clean-ups #944

Open

30 tasks

h-vetinari force-pushed the otel branch from be3a505 to 6735737 Compare December 1, 2023 01:46

h-vetinari mentioned this pull request Dec 4, 2023

Close out migration for libopentelemetry_cpp112 conda-forge/conda-forge-pinning-feedstock#5221

Merged

h-vetinari force-pushed the otel branch from cef824b to defb88c Compare November 25, 2024 00:38

h-vetinari mentioned this pull request Dec 9, 2024

[C++] Allow building against system opentelemetry apache/arrow#44982

Closed

h-vetinari force-pushed the otel branch 2 times, most recently from 045a9e2 to 6a3f9c4 Compare December 10, 2024 17:12

h-vetinari mentioned this pull request Dec 15, 2024

[C++] ENH: enable building against system opentelemetry on windows apache/arrow#45033

Open

h-vetinari changed the title ~~WIP: Build with opentelemetry support~~ Build with opentelemetry support (on unix) Dec 27, 2024

h-vetinari force-pushed the otel branch 2 times, most recently from 8ffa52b to 8fe1f02 Compare December 28, 2024 08:02

h-vetinari mentioned this pull request Dec 29, 2024

[C++] Failing arrow-flight-test when enabling opentelemetry apache/arrow#45120

Open

h-vetinari force-pushed the otel branch from 8fe1f02 to 0b9d375 Compare January 23, 2025 00:05

h-vetinari added 2 commits January 23, 2025 13:08

build with opentelemetry support on unix

24637d2

ignore gflags run-export also on linux

93b9017

MNT: Re-rendered with conda-build 25.1.1, conda-smithy 3.45.3, and co…

90ef48d

…nda-forge-pinning 2025.01.22.17.59.35

h-vetinari force-pushed the otel branch from 0b9d375 to 90ef48d Compare January 23, 2025 02:36

h-vetinari marked this pull request as ready for review January 23, 2025 04:34

h-vetinari requested review from cpcloud, jakirkham, jreback, kou, kszucs, leifwalsh, nealrichardson, pcmoritz, pearu, pitrou, raulcd, robertnishihara, siddharthteotia, wesm and xhochy as code owners January 23, 2025 04:34

h-vetinari merged commit 11a8899 into conda-forge:main Jan 23, 2025

h-vetinari deleted the otel branch January 23, 2025 04:37

h-vetinari mentioned this pull request Feb 7, 2026

Enable opentelemetry support on windows #1924

Draft

Uh oh!

Conversation

h-vetinari commented May 5, 2023

Uh oh!

conda-forge-webservices bot commented May 5, 2023

Uh oh!

h-vetinari commented May 6, 2023

Uh oh!

xhochy commented May 8, 2023

Uh oh!

h-vetinari commented May 11, 2023

Uh oh!

xhochy commented May 12, 2023

Uh oh!

h-vetinari commented May 13, 2023

Uh oh!

h-vetinari commented May 13, 2023

Uh oh!

xhochy commented May 15, 2023

Uh oh!

h-vetinari commented May 15, 2023

Uh oh!

h-vetinari commented May 15, 2023

Uh oh!

lidavidm commented May 15, 2023

Uh oh!

conda-forge-admin commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

h-vetinari commented Jan 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

conda-forge-admin commented Nov 25, 2024 •

edited

Loading