Skip to content

feature: Protobuf upgrade#1672

Merged
thanethomson merged 25 commits intomainfrom
feature/proto-upgrade
Nov 29, 2023
Merged

feature: Protobuf upgrade#1672
thanethomson merged 25 commits intomainfrom
feature/proto-upgrade

Conversation

@thanethomson
Copy link
Contributor

@thanethomson thanethomson commented Nov 21, 2023

The primary contribution to #1330.

Must be merged by way of a merge commit (i.e. please do not squash).

Individual PRs have already been reviewed.

✔️ Manual run of the E2E nightlies on the feature/proto-upgrade branch: https://github.com/cometbft/cometbft/actions/runs/7025265160


PR checklist

  • Tests written/updated
  • Changelog entry added in .changelog (we use unclog to manage our changelog)
  • Updated relevant documentation (docs/ or spec/) and code comments

mzabaluev and others added 18 commits April 21, 2023 13:13
* proto: Add versioned cometbft proto files

Rename the packages, placing them under the top-level cometbft package.
The version suffixes denote evolution of message types across
successive CometBFT releases, without binding versioned packages
to any particular release. I.e. messages that have not evolved from
the baseline remain located in .v1 even as other messages' definitions
are taken from .v2 and onwards.
The source releases for versioning as of this commit are the following:
- v0.34.27
- v0.37.0
- main, to become v0.38.x

* Regenerate *.pb.go in the api folder

Replace the *.pb.go files generated from tendermint domain protos
with those generated from the versioned protos under cometbft.

* Move and adapt supporting code alongside *.pb.go

Two types of changes here:
- Adapted manually-written files adding methods to generated Go types.
- Added aliases for multi-versioned type definitions in their
  "parent" domain package that gets the importer the latest definition
   of each message type in the domain.

* Remove go_package directives from tendermint protos

---------

Co-authored-by: Thane Thomson <connect@thanethomson.com>
The service in the rpc.grpc package has been deprecated in v0.38.x.
Copy the deprecation notice present in that branch.
* Add first draft of ADR-103

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Add rollout strategy

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Expand on problem with manual diffs

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Clarify that code generators generate breaking code from non-breaking proto changes

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Clarify that consolidation only applies to currently maintained versions

Signed-off-by: Thane Thomson <connect@thanethomson.com>

---------

Signed-off-by: Thane Thomson <connect@thanethomson.com>
Also update the generated *.pb.go file that is affected by the change.
* proto: abci/v4 package with renamed enums

Rename all enum types and their value names defined in abci
to correspond to buf naming guidelines: no enum definitions nested
in message definitions, value names prefixed with their enum type name,
the zero value has suffix _UNDEFINED.
The only exception to the latter is CHECK_TX_TYPE_NEW that has
to be grandfathered in because of the semantics of the wire value.

Provide versioned definitions of all types that included the enums.

* proto: _UNKNOWN suffix for zero enum values

The _UNSPECIFIED convention is a recommendation and the suffix
can be overridden in lint settings.
Following the precedent in existing enum definitions,
which would otherwise need a new version and cause a lot of knock-on
versioning churn just for this reason, it's better to continue with
_UNKNOWN as the common suffix throughout CometBFT proto files.

* api: generate Go code for abci/v4

* Update abci aliases and add-on methods to v4

Change the names of aliases in the abci package to match the
new, less ugly naming convention derived directly from proto files.

* proto: enum lints and overrides

Enable the lints verifying correctness of enum naming.

* Fix expected abci-cli output in docs and tests

Stringization of the status enum has changed.

* proto: redefine CheckTxType with proper zero value

v4.CheckTxType is redefined to have 0 as the unspecified value, to
conform to the best protobuf practices. The valid values are reassigned
starting from 1.

* proto: Rename the CHECK_TX_TYPE_NEW value

Rename to CHECK_TX_TYPE_CHECK to reduce the possibility of confusion
with the old code CHECK_TX_TYPE_NEW (which was 0, but now mapped to
the unknown value)

Co-authored-by: Sergio Mena <sergio@informal.systems>
proto: copy latest changes in CometBFT 0.38

Replicate protobuf changes made for CometBFT 0.38.0 into
the appropriate versions of cometbft.* packages.
This only touches existing packages, not the newly added service
packages which already have versioning in the main branch.

proto: replicate services.* packages to cometbft

Copy over the definitions from tendermint packages to cometbft
namespace and rename references.
The versioning for the proto packages themselves is already done
in the main branch.

proto: redefine RequestExtendVote in abci.v4

Need to change a field to v4.Misbehavior.

api: regenerate *.pb.go files from protobuf

Regenerate api. Change the RequestExtendVote alias
to point to v4 package.
Add aliases for HasProposalBlockPart.

consensus: update msgs.go to main and fix imports
consensus: update msgs_test.go to main
consensus: use aliased consts in state_test.go

grpc: fix import in BlockResultsService client

Regenerate mocks.

mempool: set the Type field in CheckTx request

Rather than using the default which is now
CHECK_TX_TYPE_UNKNOWN, pass the type explicitly.
Before the enum normalization, the default value was the same as
New.

mempool: panic on unexpected type in ABCI CheckTx

Backstop an unexpected value of RequestCheckTx.type with a panic
rather than silently doing nothing.

Small change to mempool test infra

All RequestCheckTx created must have filed `Type` set

changelog for the new CheckTx type field behavior

Add the note that the Type field can no longer be omitted from the
proto-generated RequestCheckTx struct, retroactively linked to the
GitHub issue about enum renaming changes.
…attern (#1510)

* proto: change version suffixes vN -> v1betaN

* api: regenerate from .v1betaN protos

* Rename import paths to api modules to v1betaN

* grpc: fix import path in pruning_service.go

* proto: update lint ignore paths

Follow the renamed packages.

* Update .changelog for ADR-107 versioning changes

No new entries created, since the previous unreleased entries describe
all the upcoming changes and only need to be updated.

* .changelog: Updated proto package name to cometbft

The changelog entry was merged into main and needed to be
updated for the feature branch where we have renamed the proto
packages.

* Fix redundant import aliases for api packages

* p2p: fix lints and restore formatting

* e2e: fix revive lint

* Fix sundry golangci lints

* exclude goconst lints for test files

* types: lint fix, formatting in vote_set.go

* state/indexer: string constants for SQL queries

---------

Co-authored-by: Sergio Mena <sergio@informal.systems>
* proto: correct Go package paths in service protos

In the new layout, the generated files are under /api.

* api: regenerate *.pb.go files for services
* proto: set lint category to DEFAULT in buf.yaml

Enable all lints in the default category and override them for legacy
proto packages.

* proto: unique response types in BlockResultService

Give the GetLatestBlockResults method its own response type
GetLatestBlockResultsResponse, as enforced by the buf linting rule
RPC_REQUEST_RESPONSE_UNIQUE.

* proto: rename ABCI service, requests and responses

Renaming to satisfy buf lint rules:
- RPC_REQUEST_STANDARD_NAME
- RPC_RESPONSE_STANDARD_NAME
- SERVICE_SUFFIX

* proto: state.v1beta4.ABCIResponsesInfo

New version to refer to the renamed FinalizeBlockResponse message
from ABCI.

* api: generated state/v1beta4 for ABCIResponsesInfo

Also update the unversioned alias for ABCIResponsesInfo to the v1beta4
definition.

* docs: fix links to ABCI proto file in the guide

* proto: rename fields in block_results service

Rename txs_results fields in response messages to tx_results,
to harmonize with FinalizeBlockResponse in ABCI.

* proto: rename field in ABCIResponsesInfo

In state.v1beta4.ABCIResponsesInfo, shorten the response_finalize_block
field name to finalize_block.

* docs/guides: update request/response struct names

* spec: rename ABCI request and response types

Update to the recent renaming in protos, also replace the odd language
like "call to `RequestCheckTx`" with "call to `CheckTx`".

* docs/app-dev: renamed ABCI response type
* spec: update URLs to ABCI protobuf

With proto renaming and versioning, it should point to the latest
versioned package under the cometbft.abci.

* proto: remove outdated go_package references

In the legacy tendermint.* protos, there should not be go_package
references because we no longer use them to generate Go code.

* spec: replace mentions of `Request*`/`Response*`

Replace with `*Request`, `*Response`, or the exact name of the message
in question.
* Merge branch main into feature/proto-upgrade

* test/e2e: update ABCI request names
* Split /api into separate go mod

* Move Wrapper definition from p2p to types

It's used in a lot of modules that don't need to depend on p2p
for this.

* blocksync: moved proto aliases from api package

* consensus: move proto aliases from api package

* privval: move proto aliases from api package

* state: move proto aliases from api package

* types: move proto aliases from api package

* test/e2e: modify Dockerfile for api/go.*

Copy api/go.mod and api/go.sum into the image build.

---------

Co-authored-by: glnro <lauren@informal.systems>
Add redirections for the /api in-tree module to go.mod files
of all modules that require /api.
Signed-off-by: Thane Thomson <connect@thanethomson.com>
* proto: Promote abci, types, crypto, version to v1

* proto: Promote blocksync to v1

* proto: Promote consensus, libs/bits to v1

* proto: Promote mempool to v1

* proto: Promote p2p to v1

* proto: promote privval to v1

* proto: Promote service packages back to v1

* proto: promote state package to v1

* proto: promote statesync to v1

* proto: Promote store to v1

* proto: Eliminate v1beta* packages redundant to v1

* api: Re-generate *.pb.go files after v1 promotion

* api: Promote handwritten code to v1

* Update proto imports to v1 in the rest of the code

* Update changelog entry 495-proto-version-packages

Update for the 1.0.0 release plan and drop the mention of 0.39.
Explain the new structure of the protobuf files and document the
mapping between version suffixes and CometBFT releases.
@thanethomson thanethomson marked this pull request as ready for review November 28, 2023 22:14
@thanethomson thanethomson requested review from a team as code owners November 28, 2023 22:14
@thanethomson thanethomson requested a review from a team November 28, 2023 22:14
@thanethomson thanethomson added enhancement New feature or request breaking A breaking change and removed conflicts labels Nov 28, 2023
Copy link
Collaborator

@melekes melekes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…es (#1707)

* Update changelog entries on versioned protos

Change outdated references to v1beta* packages that now need to
point to v1.
Provide an example of enum renaming.

* abci: Update the link to the proto file to v1
melekes and others added 2 commits November 29, 2023 19:04
* add missing docs to protos

apply the same linting rules used by Cosmos SDK

* format protos with `make proto-format`

* buf: exclude only COMMENT_FIELD linter

* remove tendermint from ignore_only

since it's already globally ignored

* Apply suggestions from code review

Co-authored-by: Thane Thomson <connect@thanethomson.com>

---------

Co-authored-by: Thane Thomson <connect@thanethomson.com>
* Makefile: Ensure local usage of latest Buf release

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* make proto-gen

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* ci: Always check that generated code is as expected

Signed-off-by: Thane Thomson <connect@thanethomson.com>

---------

Signed-off-by: Thane Thomson <connect@thanethomson.com>
@thanethomson thanethomson merged commit c5f766b into main Nov 29, 2023
@thanethomson thanethomson deleted the feature/proto-upgrade branch November 29, 2023 18:00
damiannolan pushed a commit to 01builders/cometbft that referenced this pull request Apr 11, 2025
## Description

This Pr took the diff files from 0.34.x -> 0.34.x-celestia and applied
the changes on top of 0.38.x branch of comet.


This contains 95% of the changes. 

Missing Features that will be PRed into this branch
- cat mempool 
- priority mempool

---------

Co-authored-by: chatton <github.qpeyb@simplelogin.fr>
Co-authored-by: Marko Baricevic <markobaricevic@Markos-MacBook-Pro.local>
Co-authored-by: Marko Baricevic <markobaricevic@Markos-MacBook-Pro.fritz.box>
Co-authored-by: julienrbrt <julien@rbrt.fr>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking A breaking change enhancement New feature or request

Projects

No open projects
Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants