Skip to content

state: add more tests for block validation#3674

Merged
ebuchman merged 46 commits intodevelopfrom
issues/2589-block-validation
Jun 21, 2019
Merged

state: add more tests for block validation#3674
ebuchman merged 46 commits intodevelopfrom
issues/2589-block-validation

Conversation

@thanethomson
Copy link
Contributor

@thanethomson thanethomson commented May 21, 2019

Attempts to address #2589

  • Updated all relevant documentation in docs
  • Updated all code comments where relevant
  • Wrote tests
  • Updated CHANGELOG_PENDING.md

@thanethomson thanethomson self-assigned this May 21, 2019
@codecov-io
Copy link

codecov-io commented May 21, 2019

Codecov Report

Merging #3674 into develop will increase coverage by 0.17%.
The diff coverage is 40%.

@@             Coverage Diff             @@
##           develop    #3674      +/-   ##
===========================================
+ Coverage    63.95%   64.12%   +0.17%     
===========================================
  Files          241      242       +1     
  Lines        19996    19963      -33     
===========================================
+ Hits         12788    12801      +13     
+ Misses        6162     6122      -40     
+ Partials      1046     1040       -6
Impacted Files Coverage Δ
state/services.go 50% <ø> (+25%) ⬆️
state/validation.go 84.48% <100%> (+18.09%) ⬆️
types/errors.go 33.33% <33.33%> (ø)
types/validator_set.go 76.53% <50%> (ø) ⬆️
consensus/reactor.go 70.94% <0%> (-2.12%) ⬇️
blockchain/reactor.go 71.49% <0%> (-1.87%) ⬇️
blockchain/pool.go 80.26% <0%> (-1.32%) ⬇️
consensus/state.go 79.85% <0%> (-0.12%) ⬇️
libs/db/remotedb/grpcdb/server.go 0% <0%> (ø) ⬆️
... and 7 more

The required tests already appear to be implemented (implicitly) through
the TestValidateBlockHeader test.
Having the tests littered with helper functions makes them less easily
readable imho, so I've pulled them out into a separate file. This also
makes it easier to see what helper functions are available during
testing, so we minimize the chance of duplication when writing new
tests.
@thanethomson
Copy link
Contributor Author

@ebuchman, could you please clarify:

/*
	TODO(#2589):
	- test unmarshalling BlockParts that are too big into a Block that
		(note this logic happens in the consensus, not in the validation here).
	- test making blocks from the types.MaxXXX functions works/fails as expected
*/
func TestValidateBlockSize(t *testing.T) {
}

The first test request's sentence is incomplete, and looks like a note that this test should actually happen in the consensus package.

With regard to the second requested test, it appears as though this is better tested by testing the state/execution.go#BlockExecutor.CreateProposalBlock method, is it not?

This refactors all of the state package's tests into a state_test
package, so as to keep any usage of the state package's internal methods
explicit.

Any internal methods/constants used by tests are now explicitly exported
in state/export_test.go
@thanethomson thanethomson marked this pull request as ready for review June 2, 2019 23:06
@thanethomson thanethomson changed the title [WIP] state: add more tests for block validation state: add more tests for block validation Jun 2, 2019
@thanethomson
Copy link
Contributor Author

So I've marked this PR as R4R since I think it increases test coverage a little, but now it also refactors the state package to separate its tests from the package (i.e. tests are now in a state_test package). All private functionality used by the tests is now made public in the state/export_test.go file so that the private methods are visible publicly exclusively during testing.

This is a stop-gap measure until such time that we can refactor the tests to remove the dependence on these private methods altogether.

If there are additional tests we should add here, let's please put up a specific issue describing exactly what we want to test in detail?

const ValSetCheckpointInterval = valSetCheckpointInterval

// UpdateState is an alias for updateState exported from execution.go,
// exclusively and explicitly for testing. TODO: Remove.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the TODO only related to this function? or all functions in this package?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All functions in this file need to eventually be refactored away. In our tests, we shouldn't be depending on private methods inside packages.

We could have one big TODO for the whole file instead, but when using certain IDEs the IDE pops up with the description of the method. That'll hopefully discourage people from using them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add TODO to every function then?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's already a TODO for each function. Are you asking if we should remove the TODOs and rather just have one in the file somewhere? We can certainly do that, and include a note about why dependence on the functions/constants needs to be removed.

AppHash: nil,
})

// save validators to db for 2 heights
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why 2 heights?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure actually, I didn't write the original code. @ebuchman - do you perhaps know the reasoning behind this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this a little deeper, I suspect this comment is just an artifact of old code and should probably be removed. It doesn't make sense in context.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's because SaveState persists the validator sets for heights 1 and 2 when the LastBlockHeight is 0

(note this logic happens in the consensus, not in the validation here).
- test making blocks from the types.MaxXXX functions works/fails as expected
*/
func TestValidateBlockSize(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this being tracked somewhere else?

@ebuchman ebuchman merged commit 228bba7 into develop Jun 21, 2019
@ebuchman ebuchman deleted the issues/2589-block-validation branch June 21, 2019 21:29
unclezoro pushed a commit to unclezoro/tendermint that referenced this pull request Sep 6, 2019
* Expose priv validators for use in testing

* Generalize block header validation test past height 1

* Remove ineffectual assignment

* Remove redundant SaveState call

* Reorder comment for clarity

* Use the block executor ApplyBlock function instead of implementing a stripped-down version of it

* Remove commented-out code

* Remove unnecessary test

The required tests already appear to be implemented (implicitly) through
the TestValidateBlockHeader test.

* Allow for catching of specific error types during TestValidateBlockCommit

* Make return error testable

* Clean up and add TestValidateBlockCommit code

* Fix formatting

* Extract function to create a new mock test app

* Update comment for clarity

* Fix comment

* Add skeleton code for evidence-related test

* Allow for addressing priv val by address

* Generalize test beyond a single validator

* Generalize TestValidateBlockEvidence past first height

* Reorder code to clearly separate tests and utility code

* Use a common constant for stop height for testing in state/validation_test.go

* Refactor errors to resemble existing conventions

* Fix formatting

* Extract common helper functions

Having the tests littered with helper functions makes them less easily
readable imho, so I've pulled them out into a separate file. This also
makes it easier to see what helper functions are available during
testing, so we minimize the chance of duplication when writing new
tests.

* Remove unused parameter

* Remove unused parameters

* Add field keys

* Remove unused height constant

* Fix typo

* Fix incorrect return error

* Add field keys

* Use separate package for tests

This refactors all of the state package's tests into a state_test
package, so as to keep any usage of the state package's internal methods
explicit.

Any internal methods/constants used by tests are now explicitly exported
in state/export_test.go

* Refactor: extract helper function to make, validate, execute and commit a block

* Rename state function to makeState

* Remove redundant constant for number of validators

* Refactor mock evidence registration into TestMain

* Remove extraneous nVals variable

* Replace function-level TODOs with file-level TODO and explanation

* Remove extraneous comment

* Fix linting issues brought up by GolangCI (pulled in from latest merge from develop)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants