Skip to content

[chore](thirdparty) Fix the linkage errors for librdkafka #17181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
BiteTheDDDDt merged 1 commit into from
Feb 28, 2023
Merged

[chore](thirdparty) Fix the linkage errors for librdkafka #17181

BiteTheDDDDt merged 1 commit into from
Feb 28, 2023

Conversation

@adonis0147
Copy link
Contributor

@adonis0147 adonis0147 commented Feb 27, 2023 •
edited
Loading

Proposed changes

Issue Number: close #xxx

Problem summary

Fix the linkage errors. See the following.

$ clang --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

$ /usr/bin/../bin/clang++ -L/Programs/doris/thirdparty/installed/lib -lssl -lcrypto -lzstd -lz -lsasl2 -shared -dynamiclib -Wl,-install_name,/Programs/doris/thirdparty/installed/lib/librdkafka++.1.dylib RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o -o librdkafka++.1.dylib -L../src -lrdkafka
ar rcs librdkafka++.a RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o
Undefined symbols for architecture arm64:
  "_GSS_C_NT_HOSTBASED_SERVICE", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
      _gssapi_client_mech_step in libsasl2.a(gssapi.o)
  "_GSS_C_NT_USER_NAME", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
  "_GSS_C_SEC_CONTEXT_SASL_SSF", referenced from:
      _gssapi_get_ssf in libsasl2.a(gssapi.o)
  "_gss_accept_sec_context", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
  "_gss_acquire_cred", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
  "_gss_canonicalize_name", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
  "_gss_compare_name", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
  "_gss_delete_sec_context", referenced from:
      _sasl_gss_free_context_contents in libsasl2.a(gssapi.o)
      _gssapi_client_mech_step in libsasl2.a(gssapi.o)
  "_gss_display_name", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
      _gssapi_client_mech_step in libsasl2.a(gssapi.o)
  "_gss_display_status", referenced from:
      _sasl_gss_seterror_ in libsasl2.a(gssapi.o)
  "_gss_import_name", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
      _gssapi_client_mech_step in libsasl2.a(gssapi.o)
  "_gss_init_sec_context", referenced from:
      _gssapi_client_mech_step in libsasl2.a(gssapi.o)
  "_gss_inquire_context", referenced from:
      _gssapi_client_mech_step in libsasl2.a(gssapi.o)
  "_gss_inquire_sec_context_by_oid", referenced from:
      _gssapi_get_ssf in libsasl2.a(gssapi.o)
  "_gss_release_buffer", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
      _gssapi_server_mech_ssfcap in libsasl2.a(gssapi.o)
      _gssapi_server_mech_ssfreq in libsasl2.a(gssapi.o)
      _sasl_gss_seterror_ in libsasl2.a(gssapi.o)
      _sasl_gss_encode in libsasl2.a(gssapi.o)
      _gssapi_decode_packet in libsasl2.a(gssapi.o)
      _gssapi_client_mech_step in libsasl2.a(gssapi.o)
      ...
  "_gss_release_buffer_set", referenced from:
      _gssapi_get_ssf in libsasl2.a(gssapi.o)
  "_gss_release_cred", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
      _sasl_gss_free_context_contents in libsasl2.a(gssapi.o)
  "_gss_release_name", referenced from:
      _gssapi_server_mech_authneg in libsasl2.a(gssapi.o)
      _sasl_gss_free_context_contents in libsasl2.a(gssapi.o)
  "_gss_unwrap", referenced from:
      _gssapi_server_mech_ssfreq in libsasl2.a(gssapi.o)
      _gssapi_decode_packet in libsasl2.a(gssapi.o)
      _gssapi_client_mech_step in libsasl2.a(gssapi.o)
  "_gss_wrap", referenced from:
      _gssapi_server_mech_ssfcap in libsasl2.a(gssapi.o)
      _sasl_gss_encode in libsasl2.a(gssapi.o)
      _gssapi_client_mech_step in libsasl2.a(gssapi.o)
  "_gss_wrap_size_limit", referenced from:
      _gssapi_wrap_sizes in libsasl2.a(gssapi.o)
  "_krb5_gss_register_acceptor_identity", referenced from:
      _gssapiv2_server_plug_init in libsasl2.a(gssapi.o)
ld: symbol(s) not found for architecture arm64

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@adonis0147
Copy link
Contributor Author

adonis0147 commented Feb 27, 2023

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 27, 2023 •
edited
Loading

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.42 seconds
stream load tsv: 479 seconds loaded 74807831229 Bytes, about 148 MB/s
stream load json: 37 seconds loaded 2358488459 Bytes, about 60 MB/s
stream load orc: 69 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 27 seconds loaded 861443392 Bytes, about 30 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230228052655_clickbench_pr_105406.html

@morningman
Copy link
Contributor

morningman commented Feb 28, 2023

run p0

2 similar comments
@adonis0147
Copy link
Contributor Author

adonis0147 commented Feb 28, 2023

run p0

@adonis0147
Copy link
Contributor Author

adonis0147 commented Feb 28, 2023

run p0

@adonis0147
Copy link
Contributor Author

adonis0147 commented Feb 28, 2023

run buildall

BiteTheDDDDt
BiteTheDDDDt approved these changes Feb 28, 2023
View reviewed changes
Copy link
Contributor

@BiteTheDDDDt BiteTheDDDDt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BiteTheDDDDt BiteTheDDDDt merged commit cf7e97d into apache:master Feb 28, 2023
@adonis0147 adonis0147 deleted the thirdparty branch February 28, 2023 14:06
yagagagaga pushed a commit to yagagagaga/doris that referenced this pull request Mar 9, 2023
@adonis0147 @yagagagaga
[chore](thirdparty) Fix the linkage errors for librdkafka (apache#17181)
005cf15 
 Fix the linkage errors for librdkafka
@morningman morningman mentioned this pull request Apr 9, 2023
Merged
5 tasks
morningman added a commit that referenced this pull request Apr 9, 2023
@morningman
[deps](libhdfs) add official hadoop libhdfs for x86 (#17435) (#18494)
70abac3 
cherry-pick #17435 and #17181
This is the first step to introduce official hadoop libhdfs to Doris. Because the current hdfs client libhdfs3 lacks some important feature and is hard to maintain.

Download the hadoop 3.3.4 binary from hadoop website: https://hadoop.apache.org/releases.html Extract libs and headers which are used for libhdfs, and pack them into hadoop_lib_3.3.4-x86.tar.gz Upload it to https://github.com/apache/doris-thirdparty/releases/tag/hadoop-libs-3.3.4

TODO:
The hadoop libs for arm is missing, we need to find a way to build it
mongo360 pushed a commit to mongo360/doris that referenced this pull request Jul 12, 2023
@morningman
[deps](libhdfs) add official hadoop libhdfs for x86 (apache#17435) (a…
b5ea516 
…pache#18494)

cherry-pick apache#17435 and apache#17181
This is the first step to introduce official hadoop libhdfs to Doris. Because the current hdfs client libhdfs3 lacks some important feature and is hard to maintain.

Download the hadoop 3.3.4 binary from hadoop website: https://hadoop.apache.org/releases.html Extract libs and headers which are used for libhdfs, and pack them into hadoop_lib_3.3.4-x86.tar.gz Upload it to https://github.com/apache/doris-thirdparty/releases/tag/hadoop-libs-3.3.4

TODO:
The hadoop libs for arm is missing, we need to find a way to build it
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

Assignees

No one assigned

Labels

dev/1.2.4-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@adonis0147 @hello-stephen @morningman @BiteTheDDDDt