[libtorch] create a new port#17199
Merged
BillyONeal merged 90 commits intomicrosoft:masterfrom Jan 18, 2023
luncliff:port/libtorch
Merged
[libtorch] create a new port#17199BillyONeal merged 90 commits intomicrosoft:masterfrom luncliff:port/libtorch
BillyONeal merged 90 commits intomicrosoft:masterfrom
luncliff:port/libtorch
Conversation
Contributor
Author
|
This PR will remain as a shortcut to related PRs until some necessary ports become ready.
|
JackBoosY
reviewed
Apr 12, 2021
Contributor
JackBoosY
left a comment
There was a problem hiding this comment.
Please note to use ./vcpkg x-add-version libtorch and commit changes.
Contributor
Author
Sure. I'll leave the part until some dependencies are ready. 👍 |
This was referenced May 23, 2021
Contributor
|
@luncliff Any news? |
Contributor
Author
Contributor
|
First, how I can help? |
Contributor
Author
Great. I will update note about related works within days. ONNX users can check #18073 |
Contributor
|
Please ping me if you need any help. |
There was a problem hiding this comment.
This is a new experimental fast check for PR issues. Please let us know if this bot is helpful!
After committing all other changes, the version database must be updated
git add -u && git commit
git checkout 1909fbc4eb8ef30017a79dce50c0f232d149a051 -- versions
./vcpkg x-add-version --allDiff
diff --git a/versions/baseline.json b/versions/baseline.json
index 5bf9f84..59a9d13 100644
--- a/versions/baseline.json
+++ b/versions/baseline.json
@@ -4646,7 +4646,7 @@
},
"onnx": {
"baseline": "1.9.0",
- "port-version": 0
+ "port-version": 1
},
"onnx-optimizer": {
"baseline": "0.2.6",
diff --git a/versions/o-/onnx.json b/versions/o-/onnx.json
index 5f4fc6e..44c1b4f 100644
--- a/versions/o-/onnx.json
+++ b/versions/o-/onnx.json
@@ -1,5 +1,10 @@
{
"versions": [
+ {
+ "git-tree": "ea25f2179598a99237671a57e2b57d0554c7ecb0",
+ "version-semver": "1.9.0",
+ "port-version": 1
+ },
{
"git-tree": "b53c9c9e969928def925c57ea5ddcdfb09293693",
"version-semver": "1.9.0",* https://github.com/houseroad/foxi * install the project's copyright (MIT) * pytorch requires `foxi_loader` The CMake target will be renamed to `onnxifi_*` for convenience.
This was referenced Sep 11, 2021
Checked the protject's CI logs. It turned out onnx/onnx_proto are ALWAYS static. Specify it in CMakeLists.txt because vcpkg configures `BUILD_SHARED_LIBS=ON` If the triplet requires it. There are no `ONNXIFI_ENABLE_EXT=ON` case. Removed the misused build options in portfile. Add port feature `protobuf-lite` which is in build option.
* remove SHARED for `onnxifi_wrapper` and `onnxifi_dummy`
* also remove redundant part in patch files
* use `link_libraries` to vcpkg installed folder * future work may use library names without `find_library` * update versions JSON to use `version-semver`
* Caffe2 is exported when BUILD_SHARED_LIBS
Contributor
Author
|
Stucked in Windows link errors |
Contributor
Author
|
Visual Studio 2019 succeeded but Visual Studio 2022 didn't. set(VCPKG_VISUAL_STUDIO_PATH "C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community") |
Contributor
Author
|
@Cheney-W , would you test this? #17199 (comment) |
Contributor
Of course, I'm testing. |
Contributor
|
@luncliff I can't reproduce this issue in |
Contributor
Author
|
confirmed the build succeeded last night. I will update the ci.baseline after testing other platforms.
Get Outlook for iOS<https://aka.ms/o0ukef>
…________________________________
From: Cheney Wang ***@***.***>
Sent: Tuesday, December 13, 2022 7:37:54 PM
To: microsoft/vcpkg ***@***.***>
Cc: Park DongHa ***@***.***>; Mention ***@***.***>
Subject: Re: [microsoft/vcpkg] [libtorch] create a new port (#17199)
@luncliff<https://github.com/luncliff> I can't reproduce this issue in VS2022 17.3.7 and VS2019.
Also, I found the bug of Visual Studio<https://developercommunity.visualstudio.com/t/thread_local-causing-fatal-error-LNK1161/10199441> has been fixed and will release in VS2022 17.4.3.
However, because our CI currently uses VS2022 17.4.1, I think we can add libtorch:x64-windows=fail to ci.baseline.txt to avoid affecting the pr merge.
—
Reply to this email directly, view it on GitHub<#17199 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADLSYE6ANVCX4JKAKAN4JILWNBGYFANCNFSM42WPHK6A>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
* fails with Visual Studio 2022 17.4.2 * requires 17.4.3+
Cheney-W
previously approved these changes
Dec 19, 2022
Member
|
@luncliff thanks for all the effort you've put on this PR. Next VM update will be around (2023/01/10) and we'll be able to test this in CI and hopefully merge it. |
Contributor
|
Waiting for #28895 merged. |
Member
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
vicroms
requested changes
Jan 17, 2023
BillyONeal
requested changes
Jan 17, 2023
Member
BillyONeal
left a comment
There was a problem hiding this comment.
The caffe2 conflict is still present. I think just making caffe2 an empty port that depends on libtorch should work now that you made it work on Windows?
Thanks! ❤️
Installing 26/26 libtorch:x64-linux...
Building libtorch[core,xnnpack]:x64-linux...
-- Note: libtorch only supports dynamic library linkage. Building dynamic library.
-- Downloading https://github.com/pytorch/pytorch/archive/v1.12.1.tar.gz -> pytorch-pytorch-v1.12.1.tar.gz...
-- Extracting source /home/bion/vcpkg/downloads/pytorch-pytorch-v1.12.1.tar.gz
-- Applying patch pytorch-pr-85958.patch
-- Applying patch fix-cmake.patch
-- Applying patch fix-fbgemm-include.patch
-- Applying patch fix-c10-glog.patch
-- Applying patch use-flatbuffers2.patch
-- Applying patch fix-windows.patch
-- Using source at /home/bion/vcpkg/buildtrees/libtorch/src/v1.12.1-d15308d103.clean
-- Using flatc: /home/bion/vcpkg/installed/x64-linux/tools/flatbuffers/flatc
-- Using protoc: /home/bion/vcpkg/installed/x64-linux/tools/protobuf/protoc
-- Setting up python virtual environmnent...
-- Installing python packages: typing-extensions;pyyaml;numpy
-- Setting up python virtual environmnent...finished.
-- Using Python3: /home/bion/vcpkg/buildtrees/libtorch/x64-linux-venv/bin/python
-- Configuring x64-linux-dbg
-- Configuring x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Building x64-linux-dbg
-- Building x64-linux-rel
-- Installing: /home/bion/vcpkg/packages/libtorch_x64-linux/share/libtorch/copyright
-- Performing post-build validation
Stored binary cache: "/home/bion/.cache/vcpkg/archives/21/211408e045c75aac65a724e61051d81797b74bbf68df490cb6e0e1da3e1b2725.zip"
error: The following files are already installed in /home/bion/vcpkg/installed/x64-linux and are in conflict with libtorch:x64-linux
Installed by caffe2:x64-linux
include/caffe2/operators/shape_op.h
include/caffe2/operators/reducer_functors.h
include/caffe2/operators/reduction_ops.h
include/caffe2/operators/relu_op.h
include/caffe2/operators/remove_data_blocks_op.h
include/caffe2/operators/replace_nan_op.h
include/caffe2/operators/reshape_op.h
include/caffe2/operators/resize_op.h
include/caffe2/operators/reverse_packed_segs_op.h
include/caffe2/operators/roi_pool_op.h
include/caffe2/operators/rowmul_op.h
include/caffe2/operators/scale_op.h
include/caffe2/operators/segment_reduction_op.h
include/caffe2/operators/sequence_ops.h
include/caffe2/operators/rank_loss_op.h
include/caffe2/operators/sinusoid_position_encoding_op.h
include/caffe2/operators/softmax_op.h
include/caffe2/operators/softmax_with_loss_op.h
include/caffe2/operators/softplus_op.h
include/caffe2/operators/space_batch_op.h
include/caffe2/operators/sparse_to_dense_mask_op.h
include/caffe2/operators/sparse_to_dense_op.h
include/caffe2/operators/spatial_batch_norm_op.h
include/caffe2/operators/spatial_softmax_with_loss_op.h
include/caffe2/operators/square_root_divide_op.h
include/caffe2/operators/stop_gradient.h
include/caffe2/operators/string_ops.h
include/caffe2/operators/one_hot_ops.h
include/caffe2/operators/lengths_top_k_op.h
include/caffe2/operators/load_save_op.h
include/caffe2/operators/local_response_normalization_op.h
include/caffe2/operators/loss_op.h
include/caffe2/operators/lpnorm_op.h
include/caffe2/operators/lstm_unit_op.h
include/caffe2/operators/map_ops.h
include/caffe2/operators/margin_ranking_criterion_op.h
include/caffe2/operators/matmul_op.h
include/caffe2/operators/merge_id_lists_op.h
include/caffe2/operators/multi_class_accuracy_op.h
include/caffe2/operators/no_default_engine_op.h
include/caffe2/operators/normalize_op.h
include/caffe2/operators/summarize_op.h
include/caffe2/operators/operator_fallback_gpu.h
include/caffe2/operators/order_switch_ops.h
include/caffe2/operators/pack_rnn_sequence_op.h
include/caffe2/operators/pack_segments.h
include/caffe2/operators/pad_op.h
include/caffe2/operators/partition_ops.h
include/caffe2/operators/perplexity_op.h
include/caffe2/operators/piecewise_linear_transform_op.h
include/caffe2/operators/pool_op.h
include/caffe2/operators/prefetch_op.h
include/caffe2/operators/prelu_op.h
include/caffe2/operators/prepend_dim_op.h
include/caffe2/utils/proto_utils.h
include/caffe2/transforms/common_subexpression_elimination.h
include/caffe2/transforms/conv_to_nnpack_transform.h
include/caffe2/transforms/pattern_net_transform.h
include/caffe2/transforms/single_op_transform.h
include/caffe2/utils/cast.h
include/caffe2/utils/cblas.h
include/caffe2/utils/conversions.h
include/caffe2/utils/cpu_neon.h
include/caffe2/utils/cpuid.h
include/caffe2/utils/fixed_divisor.h
include/caffe2/utils/math-detail.h
include/caffe2/utils/math.h
include/caffe2/utils/murmur_hash3.h
include/caffe2/sgd/rmsprop_op.h
include/caffe2/utils/signal_handler.h
include/caffe2/utils/simple_queue.h
include/caffe2/utils/smart_tensor_printer.h
include/caffe2/utils/string_utils.h
include/caffe2/utils/threadpool/ThreadPool.h
include/caffe2/utils/threadpool/ThreadPoolCommon.h
include/caffe2/utils/threadpool/WorkersPool.h
include/caffe2/utils/threadpool/pthreadpool.h
include/caffe2/utils/zmq_helper.h
include/caffe2/video/video_decoder.h
include/caffe2/video/video_input_op.h
include/caffe2/video/video_io.h
include/caffe2/proto/caffe2.pb.h
include/caffe2/operators/tensor_protos_db_input.h
include/caffe2/operators/text_file_reader_utils.h
include/caffe2/operators/tile_op.h
include/caffe2/operators/top_k.h
include/caffe2/operators/transpose_op.h
include/caffe2/operators/tt_linear_op.h
include/caffe2/operators/utility_ops.h
include/caffe2/operators/while_op.h
include/caffe2/operators/zero_gradient_op.h
include/caffe2/perfkernels/common.h
include/caffe2/perfkernels/cvtsh_ss_bugfix.h
include/caffe2/perfkernels/embedding_lookup.h
include/caffe2/perfkernels/typed_axpy.h
include/caffe2/operators/lengths_tile_op.h
include/caffe2/queue/blobs_queue.h
include/caffe2/queue/blobs_queue_db.h
include/caffe2/queue/queue_ops.h
include/caffe2/queue/rebatching_queue.h
include/caffe2/queue/rebatching_queue_ops.h
include/caffe2/sgd/adagrad_op.h
include/caffe2/sgd/adam_op.h
include/caffe2/sgd/ftrl_op.h
include/caffe2/sgd/iter_op.h
include/caffe2/sgd/learning_rate_functors.h
include/caffe2/sgd/learning_rate_op.h
include/caffe2/sgd/momentum_sgd_op.h
include/caffe2/distributed/redis_store_handler.h
include/caffe2/core/scope_guard.h
include/caffe2/core/static_tracepoint.h
include/caffe2/core/static_tracepoint_elfx86.h
include/caffe2/core/stats.h
include/caffe2/core/tensor.h
include/caffe2/core/timer.h
include/caffe2/core/transform.h
include/caffe2/core/types.h
include/caffe2/core/workspace.h
include/caffe2/cuda_rtc/common_rtc.h
include/caffe2/db/create_db_op.h
include/caffe2/distributed/file_store_handler.h
include/caffe2/distributed/file_store_handler_op.h
include/caffe2/core/qtensor_serialization.h
include/caffe2/distributed/redis_store_handler_op.h
include/caffe2/distributed/store_handler.h
include/caffe2/distributed/store_ops.h
include/caffe2/experiments/operators/fully_connected_op_decomposition.h
include/caffe2/experiments/operators/fully_connected_op_prune.h
include/caffe2/experiments/operators/fully_connected_op_sparse.h
include/caffe2/experiments/operators/funhash_op.h
include/caffe2/experiments/operators/sparse_funhash_op.h
include/caffe2/experiments/operators/sparse_matrix_reshape_op.h
include/caffe2/experiments/operators/tt_contraction_op.h
include/caffe2/experiments/operators/tt_pad_op.h
include/caffe2/image/image_input_op.h
include/caffe2/core/graph.h
include/caffe2/core/blob.h
include/caffe2/core/blob_serialization.h
include/caffe2/core/blob_serializer_base.h
include/caffe2/core/blob_stats.h
include/caffe2/core/common.h
include/caffe2/core/common_cudnn.h
include/caffe2/core/common_gpu.h
include/caffe2/core/common_omp.h
include/caffe2/core/context.h
include/caffe2/core/context_gpu.h
include/caffe2/core/db.h
include/caffe2/core/event.h
include/caffe2/core/flags.h
include/caffe2/image/transform_gpu.h
include/caffe2/core/init.h
include/caffe2/core/logging.h
include/caffe2/core/macros.h
include/caffe2/core/memonger.h
include/caffe2/core/net.h
include/caffe2/core/net_simple.h
include/caffe2/core/observer.h
include/caffe2/core/operator.h
include/caffe2/core/operator_gradient.h
include/caffe2/core/operator_schema.h
include/caffe2/core/plan_executor.h
include/caffe2/core/qtensor.h
include/caffe2/operators/free_op.h
include/caffe2/operators/cross_entropy_op.h
include/caffe2/operators/dataset_ops.h
include/caffe2/operators/distance_op.h
include/caffe2/operators/do_op.h
include/caffe2/operators/dropout_op.h
include/caffe2/operators/elementwise_linear_op.h
include/caffe2/operators/elementwise_logical_ops.h
include/caffe2/operators/elementwise_op_test.h
include/caffe2/operators/elu_op.h
include/caffe2/operators/feed_blob_op.h
include/caffe2/operators/filler_op.h
include/caffe2/operators/find_duplicate_elements_op.h
include/caffe2/operators/find_op.h
include/caffe2/operators/counter_ops.h
include/caffe2/operators/fully_connected_op.h
include/caffe2/operators/given_tensor_fill_op.h
include/caffe2/operators/gru_unit_op.h
include/caffe2/operators/h_softmax_op.h
include/caffe2/operators/half_float_ops.h
include/caffe2/operators/if_op.h
include/caffe2/operators/im2col_op.h
include/caffe2/operators/index_hash_ops.h
include/caffe2/operators/instance_norm_op.h
include/caffe2/operators/layer_norm_op.h
include/caffe2/operators/leaky_relu_op.h
include/caffe2/operators/lengths_reducer_ops.h
include/caffe2/operators/clip_op.h
include/caffe2/mpi/mpi_common.h
include/caffe2/mpi/mpi_ops.h
include/caffe2/operators/accumulate_op.h
include/caffe2/operators/accuracy_op.h
include/caffe2/operators/apmeter_op.h
include/caffe2/operators/batch_box_cox_op.h
include/caffe2/operators/batch_gather_ops.h
include/caffe2/operators/batch_matmul_op.h
include/caffe2/operators/boolean_mask_ops.h
include/caffe2/operators/boolean_unmask_ops.h
include/caffe2/operators/cast_op.h
include/caffe2/operators/channel_shuffle_op.h
include/caffe2/core/allocator.h
include/caffe2/operators/concat_split_op.h
include/caffe2/operators/conv_op.h
include/caffe2/operators/conv_op_cache_cudnn.h
include/caffe2/operators/conv_op_impl.h
include/caffe2/operators/conv_op_shared.h
include/caffe2/operators/conv_pool_op_base.h
include/caffe2/operators/conv_transpose_op.h
include/caffe2/operators/conv_transpose_op_impl.h
include/caffe2/operators/conv_transpose_op_mobile.h
include/caffe2/operators/conv_transpose_op_mobile_impl.h
include/caffe2/operators/conv_transpose_unpool_op_base.h
include/caffe2/operators/cosine_embedding_criterion_op.h
error: Please ensure you're using the latest port files with `git pull` and `vcpkg update`.
Then check for known issues at:
https://github.com/microsoft/vcpkg/issues?q=is%3Aissue+is%3Aopen+in%3Atitle+libtorch
You can submit a new issue at:
https://github.com/microsoft/vcpkg/issues/new?template=report-package-build-failure.md&title=[libtorch]+Build+error
Include '[libtorch] Build error' in your bug report title, the following version information in your bug description, and attach any relevant failure logs from above.
vcpkg-tool version: 2022-12-14-7ae0d8527fb488fde10a89c2813802dc9b03b6f9
vcpkg-scripts version: 3fd29f0d3 2023-01-17 (29 minutes ago)
* update baseline
Cheney-W
approved these changes
Jan 18, 2023
vicroms
approved these changes
Jan 18, 2023
BillyONeal
approved these changes
Jan 18, 2023
Member
BillyONeal
left a comment
There was a problem hiding this comment.
This ROCKS. Thank you so so much for getting through this slog!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does your PR fix? Fixes
Resolves #16319
Resolves #15763
Resolves #5209
Resolves #6345
This is a rework of #8318. It tried to download the existing releases at the moment, but this PR will build from the source.
References
Related Works
Some required PRs are still open and in progress, but I think it's time to start the draft.
Wish I can get more feedback and opinions for the
libtorchport.find_package[fbgemm] fix find_package #20111onnx[onnx] create a new port #18073foxi[onnx] update ONNXIFI build output and host dependencies #20112foxi[foxi] create a new port #21786Which triplets are supported/not supported? Have you updated the CI baseline?
Supports 2 triplets. Out of source build is not available in
x64-windows.x64-osxx64-linuxDoes your PR follow the maintainer guide?
Yes, but I will check it regularly.