[protobuf] Upgrade third_party/protobuf to 22.x by jtattermusch · Pull Request #32606 · grpc/grpc

jtattermusch · 2023-03-13T11:43:16Z

The very non-trivial upgrade of third_party/protobuf to 22.x
This PR strives to be as small as possible and many changes that were compatible with protobuf 21.x and didn't have to be merged atomically with the upgrade were already merged.
Due to the complexity of the upgrade, this PR wasn't created automatically by a tool, but manually. Subsequent upgraded of third_party/protobuf with our OSS release script should work again once this change is merged.

This is best reviewed commit-by-commit, I tried to group changes in logical areas.

Notable changes:

the upgrade of third_party/protobuf submodule, the bazel protobuf dependency itself
upgrade of UPB dependency to 22.x (in the past, we used to always upgrade upb to "main", but upb now has release branch as well). UPB needs to be upgraded atomically with protobuf since there's a de-facto circular dependency (new protobuf depends on new upb, which depends on new protobuf for codegen).
some protobuf and upb bazel rules are now aliases, so extract_metadata_from_bazel_xml.py and gen_upb_api_from_bazel_xml.py had to be modified to be able to follow aliases and reach the actual aliased targets.
some protobuf public headers were renamed, so especially src/compiler needed to be updated to use the new headers.
protobuf and upb now both depend on utf8_range project, so since we bundle upb with grpc in some languages, we now have to bundle utf8_range as well (hence changes in build for python, PHP, objC, cmake etc).
protoc now depends on absl and utf8_range (previously protobuf had absl dependency, but not for the codegen part), so python's make_grpcio_tools.py required partial rewrite to be able to handle those dependencies in the grpcio_tools build.
many updates and fixes required for C++ distribtests (currently they all pass, but we'll probably need to follow up, make protobuf's and grpc's handling of dependencies more aligned and revisit the distribtests)
bunch of other changes mostly due to overhaul of protobuf's and upb's internal build layout.

TODOs:

[DONE] make sure IWYU and clang_tidy_code pass
create a list of followups (e.g. work to reenable the few tests I had to disable and to remove workaround I had to use)
[DONE in cl/523706129] figure out problem(s) with internal import

- most linux and windows C++ distribtests - follow may be needed: gRPC_PROTOBUF_PACKAGE_TYPE=MODULE does not correctly declare absl dependency - add dependency protobuf on absl in external project distribtest - fix aarch64 crosscompile distribtest - fix grpc_android test build in a similar way as cmake distribtests

… distribtest

due to protobuf bumping their required versions here: protocolbuffers/protobuf#10652

…-warning-option

…port

sampajano · 2023-04-12T20:55:16Z

@sampajano the changes to ObjC are pretty minor, but can you please doublecheck anyway? (there's a commit to bump required target versions for objc examples and a change to add utf8_range include.)

@jtattermusch Thanks for checking. Sorry i was OOO until today. ObjC changes LGTM thanks :)

The very non-trivial upgrade of third_party/protobuf to 22.x This PR strives to be as small as possible and many changes that were compatible with protobuf 21.x and didn't have to be merged atomically with the upgrade were already merged. Due to the complexity of the upgrade, this PR wasn't created automatically by a tool, but manually. Subsequent upgraded of third_party/protobuf with our OSS release script should work again once this change is merged. This is best reviewed commit-by-commit, I tried to group changes in logical areas. Notable changes: - the upgrade of third_party/protobuf submodule, the bazel protobuf dependency itself - upgrade of UPB dependency to 22.x (in the past, we used to always upgrade upb to "main", but upb now has release branch as well). UPB needs to be upgraded atomically with protobuf since there's a de-facto circular dependency (new protobuf depends on new upb, which depends on new protobuf for codegen). - some protobuf and upb bazel rules are now aliases, so ` extract_metadata_from_bazel_xml.py` and `gen_upb_api_from_bazel_xml.py` had to be modified to be able to follow aliases and reach the actual aliased targets. - some protobuf public headers were renamed, so especially `src/compiler` needed to be updated to use the new headers. - protobuf and upb now both depend on utf8_range project, so since we bundle upb with grpc in some languages, we now have to bundle utf8_range as well (hence changes in build for python, PHP, objC, cmake etc). - protoc now depends on absl and utf8_range (previously protobuf had absl dependency, but not for the codegen part), so python's make_grpcio_tools.py required partial rewrite to be able to handle those dependencies in the grpcio_tools build. - many updates and fixes required for C++ distribtests (currently they all pass, but we'll probably need to follow up, make protobuf's and grpc's handling of dependencies more aligned and revisit the distribtests) - bunch of other changes mostly due to overhaul of protobuf's and upb's internal build layout. TODOs: - [DONE] make sure IWYU and clang_tidy_code pass - create a list of followups (e.g. work to reenable the few tests I had to disable and to remove workaround I had to use) - [DONE in cl/523706129] figure out problem(s) with internal import --------- Co-authored-by: Craig Tiller <ctiller@google.com>

ghost · 2023-08-06T20:19:44Z

Anyone got any idea what I need to do to update my flow to work with gRPC v1.55.0 and later. I think the issue is this protobuf upgrade and I found one mention of CMake changes required so now do a find_package(protobuf CONFIG REQUIRED) but that hasn't helped.

The error I'm getting is an undefined reference:

 undefined reference to `google::protobuf::internal::ThreadSafeArena::thread_cache_'

and my CMakeLists.txt looks like:

project(genesee_protobuf LANGUAGES CXX)
cmake_minimum_required(VERSION 3.15)

find_package(protobuf CONFIG REQUIRED)
find_package(gRPC CONFIG REQUIRED)
find_package(Threads REQUIRED)

set(PROTO_FILES
    ${CMAKE_CURRENT_LIST_DIR}/genesee_protobuf/discovery/discovery.proto
    ${CMAKE_CURRENT_LIST_DIR}/genesee_protobuf/backdoor_memory_interface/backdoor_memory_interface.proto
    ${CMAKE_CURRENT_LIST_DIR}/genesee_protobuf/trace/trace.proto
)

set(PROTO_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})
set(PROTO_IMPORT_DIR ${CMAKE_CURRENT_LIST_DIR})

add_library(genesee_protobuf ${PROTO_FILES})
target_link_libraries(genesee_protobuf PUBLIC protobuf::libprotobuf gRPC::grpc gRPC::grpc++)
target_include_directories(genesee_protobuf PUBLIC "$<BUILD_INTERFACE:${PROTO_BINARY_DIR}>")

#
# Compile protobuf and grpc files in genesee_protobuf target to cpp
#
get_target_property(grpc_cpp_plugin_location gRPC::grpc_cpp_plugin LOCATION)

protobuf_generate(
    TARGET genesee_protobuf
    IMPORT_DIRS ${PROTO_IMPORT_DIR}
    PROTOC_OUT_DIR ${PROTO_BINARY_DIR}
    LANGUAGE cpp)

protobuf_generate(
    TARGET genesee_protobuf
    LANGUAGE grpc
    IMPORT_DIRS ${PROTO_IMPORT_DIR}
    PROTOC_OUT_DIR ${PROTO_BINARY_DIR}
    GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc
    PLUGIN "protoc-gen-grpc=${grpc_cpp_plugin_location}")

My main application uses the shared library spat out by the above.

github-actions bot added lang/C# lang/c++ lang/core lang/ObjC lang/Python labels Mar 13, 2023

grpc-checks bot added bloat/medium per-call-memory/neutral per-channel-memory/neutral labels Mar 13, 2023

jtattermusch force-pushed the protobuf_upgrade_march2 branch 8 times, most recently from c2b8786 to 310881f Compare March 28, 2023 18:47

jtattermusch force-pushed the protobuf_upgrade_march2 branch 6 times, most recently from 02c96f8 to 9f1e6fe Compare April 5, 2023 07:42

jtattermusch force-pushed the protobuf_upgrade_march2 branch 2 times, most recently from fd191ad to ef3cd38 Compare April 6, 2023 14:26

jtattermusch added the release notes: yes Indicates if PR needs to be in release notes label Apr 6, 2023

github-actions bot added the PR title format label Apr 6, 2023

jtattermusch changed the title ~~Protobuf upgrade march2~~ [protobuf] Upgrade third_party/protobuf to 22.x Apr 6, 2023

github-actions bot removed the PR title format label Apr 6, 2023

jtattermusch force-pushed the protobuf_upgrade_march2 branch 2 times, most recently from 966909e to e330883 Compare April 10, 2023 18:34

jtattermusch and others added 16 commits April 12, 2023 16:00

run make_grpcio_tools.py

73775e4

python bazel distribtests: add pip_deps from UPB to fix

32029d8

workaround issues with protobuf's pkgconfig to fix cmake_as_pkgconfig…

42b08d3

… distribtest

update Makefiles in C++ examples to accept protobuf 22.x+ as well

f74b7a5

bump required versions for objC examples

cc13b44

due to protobuf bumping their required versions here: protocolbuffers/protobuf#10652

hack: fix binder transport apk build by explicitly using -Wno-unknown…

024f507

…-warning-option

remove unused WKT headers

3eb54e2

add protobuf external dep to grpc++_codegen_proto and grpc_plugin_sup…

ea22a7c

…port

add protobuf external dep to few more targets

2ebc624

FOLLOWUP NEEDED: temporarily disable some python bazel distribtests

faa64e8

Update fix_build_deps.py

32f6ef0

more fixes to fix_build_deps.py

b764001

add upb_collections_lib dep

486ba1b

add IWYU pragmas

6c26742

run iwyu and clang-format code

6d8f429

jtattermusch force-pushed the protobuf_upgrade_march2 branch from 74f2ed1 to 6d8f429 Compare April 12, 2023 15:05

jtattermusch merged commit 0f1afec into grpc:master Apr 12, 2023

copybara-service bot added the imported Specifies if the PR has been imported to the internal repository label Apr 13, 2023

ejona86 mentioned this pull request Apr 17, 2023

Upgrade Protobuf Java to 3.22.3 (aka 22.3) grpc/grpc-java#10045

Merged

pgeler mentioned this pull request Jul 16, 2023

add grpc package version 1.54.3 and 1.48.4 conan-io/conan-center-index#18544

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[protobuf] Upgrade third_party/protobuf to 22.x#32606

[protobuf] Upgrade third_party/protobuf to 22.x#32606
jtattermusch merged 35 commits intogrpc:masterfrom
jtattermusch:protobuf_upgrade_march2

jtattermusch commented Mar 13, 2023 •

edited

Loading

Uh oh!

sampajano commented Apr 12, 2023

Uh oh!

ghost commented Aug 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

jtattermusch commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sampajano commented Apr 12, 2023

Uh oh!

ghost commented Aug 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jtattermusch commented Mar 13, 2023 •

edited

Loading