fixes #3800

* Remove go func {}()

closes #357

- Remove go func(){}() that caused race condiditon

- To reproduce
	- add -race in make file to `install_abci`
	- Remove `CGO_ENABLED=0` & add -race to `install`

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* remove -race

* fix data race

also, reorder callbacks similarly to socket client

…3608)

* docs: go built-in guide

* fix package imports, add badger db, simplify Query

* newTendermint function

* working example

* finish the first guide

* add one more note

* add the second Golang guide - external ABCI app

* fix typos

* Remove db from tendemrint in favor of tendermint/tm-cmn

- remove db from `libs`
- update dependancy, there have been no breaking changes in the updated deps
	- https://github.com/grpc/grpc-go/releases
	- https://github.com/golang/protobuf/releases

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* changelog add

* gofmt

* more gofmt

* ADR TOC in readme.md

* Added A TOC to the Readme.md of ADR Section

- Added table of contents to the Readme of the architecture section.
	- Easier to traverse and when you know what is there.
	- If the Adr's become viewable online it would help guide the user

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* add tm-cmn to subprojects

* normalize word

* rpc: make max_body_bytes and max_header_bytes configurable

* update changelog pending

* use byte buffer pool to decreass allocs

* wrap to put buffer in defer

* wapper defer

* add dependency

* remove Gopkg,*

* add change log

* implement broadcast_duplicate_vote endpoint

* fix test_cover

* address comments

* address comments

* Update abci/example/kvstore/persistent_kvstore.go

Co-Authored-By: mossid <torecursedivine@gmail.com>

* Update rpc/client/main_test.go

Co-Authored-By: mossid <torecursedivine@gmail.com>

* address comments in progress

* reformat the code

* make linter happy

* make tests pass

* replace BroadcastDuplicateVote with BroadcastEvidence

* fix test

* fix endpoint name

* improve doc

* fix TestBroadcastEvidenceDuplicateVote

* Update rpc/core/evidence.go

Co-Authored-By: Thane Thomson <connect@thanethomson.com>

* add changelog entry

* fix TestBroadcastEvidenceDuplicateVote

* mempool: make max_msg_bytes configurable

* apply suggestions from code review

* update changelog pending

* apply suggestions from code review again

#3825)

* rpc: return err if page is incorrect (less than 0 or greater than total pages)

Fixes #3813

* fix rpc_test

* go routines in blockchain reactor

* Added reference to the go routine diagram

* Initial commit

* cleanup

* Undo testing_logger change, committed by mistake

* Fix the test loggers

* pulled some fsm code into pool.go

* added pool tests

* changes to the design

added block requests under peer

moved the request trigger in the reactor poolRoutine, triggered now by a ticker

in general moved everything required for making block requests smarter in the poolRoutine

added a simple map of heights to keep track of what will need to be requested next

added a few more tests

* send errors to FSM in a different channel than blocks

send errors (RemovePeer) from switch on a different channel than the
one receiving blocks
renamed channels
added more pool tests

* more pool tests

* lint errors

* more tests

* more tests

* switch fast sync to new implementation

* fixed data race in tests

* cleanup

* finished fsm tests

* address golangci comments :)

* address golangci comments :)

* Added timeout on next block needed to advance

* updating docs and cleanup

* fix issue in test from previous cleanup

* cleanup

* Added termination scenarios, tests and more cleanup

* small fixes to adr, comments and cleanup

* Fix bug in sendRequest()

If we tried to send a request to a peer not present in the switch, a
missing continue statement caused the request to be blackholed in a peer
that was removed and never retried.

While this bug was manifesting, the reactor kept asking for other
blocks that would be stored and never consumed. Added the number of
unconsumed blocks in the math for requesting blocks ahead of current
processing height so eventually there will be no more blocks requested
until the already received ones are consumed.

* remove bpPeer's didTimeout field

* Use distinct err codes for peer timeout and FSM timeouts

* Don't allow peers to update with lower height

* review comments from Ethan and Zarko

* some cleanup, renaming, comments

* Move block execution in separate goroutine

* Remove pool's numPending

* review comments

* fix lint, remove old blockchain reactor and duplicates in fsm tests

* small reorg around peer after review comments

* add the reactor spec

* verify block only once

* review comments

* change to int for max number of pending requests

* cleanup and godoc

* Add configuration flag fast sync version

* golangci fixes

* fix config template

* move both reactor versions under blockchain

* cleanup, golint, renaming stuff

* updated documentation, fixed more golint warnings

* integrate with behavior package

* sync with master

* gofmt

* add changelog_pending entry

* move to improvments

* suggestion to changelog entry

* Renamed wire.go to codec.go

- Wire was the previous name of amino
- Codec describes the file better than `wire` & `amino`

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* ide error

* rename amino.go to codec.go

cleanup to add linter

    grpc change:
        https://godoc.org/google.golang.org/grpc#WithContextDialer
        https://godoc.org/google.golang.org/grpc#WithDialer
        grpc/grpc-go#2627
    prometheous change:
        due to UninstrumentedHandler, being deprecated in the future
    empty branch = empty if or else statement
        didn't delete them entirely but commented
        couldn't find a reason to have them
    could not replicate the issue #3406
        but if want to keep it commented then we should comment out the if statement as well

to the types package

Paritally Fixes #3584

* p2p: fix false-positive error logging when stopping connections

This changeset fixes two types of false-positive errors occurring during
connection shutdown.

The first occurs when the process invokes FlushStop() or Stop() on a
connection. While the previous behavior did properly wait for the sendRoutine
to finish, it did not notify the recvRoutine that the connection was shutting
down. This would cause the recvRouting to receive and error when reading and
log this error. The changeset fixes this by notifying the recvRoutine that
the connection is shutting down.

The second occurs when the connection is terminated (gracefully) by the other side.
The recvRoutine would get an EOF error during the read, log it, and stop the connection
with an error. The changeset detects EOF and gracefully shuts down the connection.

* bring back the comment about flushing

* add changelog entry

* listen for quitRecvRoutine too

* we have to call stopForError

Otherwise peer won't be removed from the peer set and maybe readded
later.

…ds (#3834)

* Do not write 'Couldn't connect to any seeds' if there are no seeds

* changelog

* remove privValUpgrade

* Fix typo in changelog

* Update CHANGELOG_PENDING.md

Co-Authored-By: Marko <marbar3778@yahoo.com>

I'm setting up all peers dynamically by calling dial_peers, so p2p.seeds in configs is empty, and I'm seeing error log a lot in logs.

…3838)

* add abci grpc kotlin guide

* Update docs/guides/kotlin.md

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update docs/guides/kotlin.md

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update docs/guides/kotlin.md

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update kotlin.md

* node: allow replacing existing p2p.Reactor(s)

using [`CustomReactors`
option](https://godoc.org/github.com/tendermint/tendermint/node#CustomReactors).
Warning: beware of accidental name clashes. Here is the list of existing
reactors: MEMPOOL, BLOCKCHAIN, CONSENSUS, EVIDENCE, PEX.

* check the absence of "CUSTOM" prefix

* merge 2 tests

* add doc.go to node package

    Add gocritic as a linter

    The linting is not complete, but should i complete in this PR or in a following.

    23 files have been touched so it may be better to do in a following PR


Commits:

* Add gocritic to linting

- Added gocritic to linting

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* gocritic

* pr comments

* remove switch in cmdBatch

* tm-cmn to tm-db

* go.mod changes

* go.mod changes

* more go.mod

* fix tm-db

* ci fix, pending change

- Replace the previous intersect call, which was called at each query condition, with a map intersection.
- Replace fmt.Sprintf with string()

closes: #3076

Benchmarks

```
Old
goos: darwin
goarch: amd64
pkg: github.com/tendermint/tendermint/state/txindex/kv
BenchmarkTxSearch-4   	     200	 103641206 ns/op	 7998416 B/op	   71171 allocs/op
PASS
ok  	github.com/tendermint/tendermint/state/txindex/kv	26.019s

New
goos: darwin
goarch: amd64
pkg: github.com/tendermint/tendermint/state/txindex/kv
BenchmarkTxSearch-4   	    1000	  38615024 ns/op	13515226 B/op	  166460 allocs/op
PASS
ok  	github.com/tendermint/tendermint/state/txindex/kv	53.618s
```

~62% performance improvement

Commits:

* Refactor tx search

* Add pending changelog entry

* Add tx search benchmarking

* remove intermediate hashes list

also reset timer in BenchmarkTxSearch
and fix other benchmark

* fix import

* Add test cases

* Fix searching

* Replace fmt.Sprintf with string

* Update state/txindex/kv/kv.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Rename params

* Cleanup

* Check error in benchmarks

* changelog updates

* pr comments

Refs #3262

* config: move max_msg_bytes into mempool section

It was incorrectly placed in fastsync section during
4d7cd80#diff-092cdc48047eeb4c0bca311a2e1b8ae6
merge.

Fixes #3868

* add changelog entry

#3871)

* consensus: reduce "Error attempting to add vote" message severity (Error -> Info)

Fixes #3839

* add missing changelog entry

This PR is related to #3107 and a continuation of #3351

It is important to emphasise that in the privval original design, client/server and listening/dialing roles are inverted and do not follow a conventional interaction.

Given two hosts A and B:

    Host A is listener/client
    Host B is dialer/server (contains the secret key)
    When A requires a signature, it needs to wait for B to dial in before it can issue a request.
    A only accepts a single connection and any failure leads to dropping the connection and waiting for B to reconnect.

The original rationale behind this design was based on security.

    Host B only allows outbound connections to a list of whitelisted hosts.
    It is not possible to reach B unless B dials in. There are no listening/open ports in B.

This PR results in the following changes:

    Refactors ping/heartbeat to avoid previously existing race conditions.
    Separates transport (dialer/listener) from signing (client/server) concerns to simplify workflow.
    Unifies and abstracts away the differences between unix and tcp sockets.
    A single signer endpoint implementation unifies connection handling code (read/write/close/connection obj)
    The signer request handler (server side) is customizable to increase testability.
    Updates and extends unit tests

A high level overview of the classes is as follows:

Transport (endpoints): The following classes take care of establishing a connection

    SignerDialerEndpoint
    SignerListeningEndpoint
    SignerEndpoint groups common functionality (read/write/timeouts/etc.)

Signing (client/server): The following classes take care of exchanging request/responses

    SignerClient
    SignerServer

This PR also closes #3601

Commits:

* refactoring - work in progress

* reworking unit tests

* Encapsulating and fixing unit tests

* Improve tests

* Clean up

* Fix/improve unit tests

* clean up tests

* Improving service endpoint

* fixing unit test

* fix linter issues

* avoid invalid cache values (improve later?)

* complete implementation

* wip

* improved connection loop

* Improve reconnections + fixing unit tests

* addressing comments

* small formatting changes

* clean up

* Update node/node.go

Co-Authored-By: jleni <juan.leni@zondax.ch>

* Update privval/signer_client.go

Co-Authored-By: jleni <juan.leni@zondax.ch>

* Update privval/signer_client_test.go

Co-Authored-By: jleni <juan.leni@zondax.ch>

* check during initialization

* dropping connecting when writing fails

* removing break

* use t.log instead

* unifying and using cmn.GetFreePort()

* review fixes

* reordering and unifying drop connection

* closing instead of signalling

* refactored service loop

* removed superfluous brackets

* GetPubKey can return errors

* Revert "GetPubKey can return errors"

This reverts commit 68c06f1.

* adding entry to changelog

* Update CHANGELOG_PENDING.md

Co-Authored-By: jleni <juan.leni@zondax.ch>

* Update privval/signer_client.go

Co-Authored-By: jleni <juan.leni@zondax.ch>

* Update privval/signer_dialer_endpoint.go

Co-Authored-By: jleni <juan.leni@zondax.ch>

* Update privval/signer_dialer_endpoint.go

Co-Authored-By: jleni <juan.leni@zondax.ch>

* Update privval/signer_dialer_endpoint.go

Co-Authored-By: jleni <juan.leni@zondax.ch>

* Update privval/signer_dialer_endpoint.go

Co-Authored-By: jleni <juan.leni@zondax.ch>

* Update privval/signer_listener_endpoint_test.go

Co-Authored-By: jleni <juan.leni@zondax.ch>

* updating node.go

* review fixes

* fixes linter

* fixing unit test

* small fixes in comments

* addressing review comments

* addressing review comments 2

* reverting suggestion

* Update privval/signer_client_test.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update privval/signer_client_test.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* Update privval/signer_listener_endpoint_test.go

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* do not expose brokenSignerDialerEndpoint

* clean up logging

* unifying methods

shorten test time
signer also drops

* reenabling pings

* improving testability + unit test

* fixing go fmt + unit test

* remove unused code

* Addressing review comments

* simplifying connection workflow

* fix linter/go import issue

* using base service quit

* updating comment

* Simplifying design + adjusting names

* fixing linter issues

* refactoring test harness + fixes

* Addressing review comments

* cleaning up

* adding additional error check

* Add Schedule:

    + The schedule is a data structure used to determine the optimal
      schedule of requests to the optimal set of peers in order to perform
      fast-sync as fast and efficiently as possible.

* Add some doc strings

* fix golangci

* Add Schedule:

    + The schedule is a data structure used to determine the optimal
      schedule of requests to the optimal set of peers in order to perform
      fast-sync as fast and efficiently as possible.

* Add some doc strings

* remove globals from tests

…3877)

Fix #3868 (comment)

Commits:

* mempool: make `max_tx_bytes` configurable instead of `max_msg_bytes`

* update CHANGELOG_PENDING

* apply suggestions from code review

* init adr 042: state sync

* link to blockchain reactor

* brapse proposal

* formatting

* response to feedback

* Update docs/architecture/adr-042-state-sync.md

Co-Authored-By: Aditya <adityasripal@gmail.com>

* Update security models and more

* clarify compression

* typo

* Update docs/architecture/adr-042-state-sync.md

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* (1/2) of replace errors.go with github.com/pkg/errors

ref #3862

- step one in removing instances of errors.go in favor of github.com/pkg/errors

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* gofmt

* add in /store

* init of (2/2) common errors

* Remove instances of cmn.Error (2/2)

- Replace usage of cmnError and errorWrap
- ref #3862

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* comment wording

* simplify IsErrXXX functions

* log panic along with stopping the MConnection

* Add test for bcBlockRequestMessage ValidateBasic method

* Add test for bcNoBlockResponseMessage ValidateBasic method

* Add test for bcStatusRequestMessage ValidateBasic method

* Add test for bcStatusResponseMessage ValidateBasic method

* Add blockchain v1 reactor ValidateBasic tests

* Add test for NewRoundStepMessage ValidateBasic method

* Add test for NewValidBlockMessage ValidateBasic method

* Test BlockParts Size

* Import cmn package

* Add test for ProposalPOLMessage ValidateBasic method

* Add test for BlockPartMessage ValidateBasic method

* Add test for HasVoteMessage ValidateBasic method

* Add test for VoteSetMaj23Message ValidateBasic method

* Add test for VoteSetBitsMessage ValidateBasic method

* Fix linter errors

* Improve readability

* Add test for BaseConfig ValidateBasic method

* Add test for RPCConfig ValidateBasic method

* Add test for P2PConfig ValidateBasic method

* Add test for MempoolConfig ValidateBasic method

* Add test for FastSyncConfig ValidateBasic method

* Add test for ConsensusConfig ValidateBasic method

* Add test for InstrumentationConfig ValidateBasic method

* Add test for BlockID ValidateBasic method

* Add test for SignedHeader ValidateBasic method

* Add test for MockGoodEvidence and MockBadEvidence ValidateBasic methods

* Remove debug logging

Co-Authored-By: Marko <marbar3778@yahoo.com>

* Update MempoolConfig field

* Test a single struct field at a time, for maintainability

Fixes #2740

…3887)

* add abci grpc java guide

* fix grammar and spelling

* store: register block amino, not just crypto

Fixes #3893

* update changelog

* Initiat commit of lite client with
with weak subjectivity ADR

* Apply suggestions from code review

Co-Authored-By: Marko <marbar3778@yahoo.com>

* Update docs/architecture/adr-044-lite-client-with-weak-subjectivity.md

Co-Authored-By: Christopher Goes <cwgoes@pluranimity.org>

* Apply suggestions from code review

Co-Authored-By: Christopher Goes <cwgoes@pluranimity.org>

* Apply suggestions from code review

Co-Authored-By: Anca Zamfir <ancazamfir@users.noreply.github.com>

* fix typo and format the code block

* address cwgoes comments

* add Positive/Negative points

* change status to accepted

* Apply suggestions from code review

Co-Authored-By: Christopher Goes <cwgoes@pluranimity.org>

* link the spec and explain headers caching

* manually swagging

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* three definitions with polymorphism

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* added blockchain and block

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* low quality generation, commit, block_response and validators

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* genesis and consensus states endpoints

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* fix indentation

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* consensus parameters

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* fix indentation

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* add height to consensus parameters endpoint

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* unconfirmed_txs and num_unconfirmed_txs

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* add missing query parameter

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* add ABCI queries

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* added index document for swagger documentation

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* add missing routes

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* contract tests added on CCI

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* contract tests job should be in the test suite

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* simplify requirements to test

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* typo

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* build is a prerequisite to start localnet

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* reduce nodejs size, move goodman to get_tools, add docs, fix comments

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* Update scripts/get_tools.sh

That's cleaner, thanks!

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* xz not supported by cci image, let's keep it simple

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* REMOVE-indirect debug of CCI paths

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* dirty experiment, volume is empty but binary has been produced

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* dirty experiment, volume is empty but binary has been produced

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* dirty experiment going on

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* locally works, CCI have difficulties with second layaer containers volumes

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* restore experiment, use machine instead of docker for contract tests

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* simplify a bit

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* rollback on machine golang

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* Document the changes

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* Changelog

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

* comments

Signed-off-by: Karoly Albert Szabo <szabo.karoly.a@gmail.com>

*    Cleanup go-wire mentions
*    remove Roadmap.md
*    build_race in makefile was failing

ref #2790

When using the RPC client in my test suite (with -race enabled), I do a lot of Subscribe/Unsubscribe operations, at some point (randomly) the race detector returns the following warning:

WARNING: DATA RACE
Read at 0x00c0009dbe30 by goroutine 31:
runtime.mapiterinit()
/usr/local/go/src/runtime/map.go:804 +0x0
github.com/tendermint/tendermint/rpc/client.(*WSEvents).redoSubscriptionsAfter()
/go/pkg/mod/github.com/tendermint/tendermint@v0.31.5/rpc/client/httpclient.go:364 +0xc0
github.com/tendermint/tendermint/rpc/client.(*WSEvents).eventListener()
/go/pkg/mod/github.com/tendermint/tendermint@v0.31.5/rpc/client/httpclient.go:393 +0x3c6 

Turns out that the redoSubscriptionAfter is not protecting the access to subscriptions.

The following change protects the read access to the subscription map behind the mutex

Fixes some typos in the docs.

Fixes the classification of some changelog items re Features vs. Improvements

* Test nodejs image

Signed-off-by: Marko Baricevic <marbar3778@yahoo.com>

* minor changes

* v in version

* version in swagger.yaml

* updates lite cline spec based on Zarko's and my previous specs

* updates spec incorporating comments from Aug 8 meeting

* added check of validators

* Update docs/spec/consensus/light-client.md

Co-Authored-By: Anca Zamfir <ancazamfir@users.noreply.github.com>

* Update docs/spec/consensus/light-client.md

Co-Authored-By: Anca Zamfir <ancazamfir@users.noreply.github.com>

* Update docs/spec/consensus/light-client.md

Co-Authored-By: Anca Zamfir <ancazamfir@users.noreply.github.com>

* Update docs/spec/consensus/light-client.md

Co-Authored-By: Anca Zamfir <ancazamfir@users.noreply.github.com>

* fix typos & improve formatting

Co-Authored-By: Anca Zamfir <ancazamfir@users.noreply.github.com>

* Update docs/spec/consensus/light-client.md

Co-Authored-By: Anton Kaliaev <anton.kalyaev@gmail.com>

* incorporated some of Anca's and Anton's comments

* addressed all current comments

* highlight assumptions/verification conditions

* added remark on checks

* Update docs/spec/consensus/light-client.md

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

* Update docs/spec/consensus/light-client.md

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

* Update docs/spec/consensus/light-client.md

Co-Authored-By: Ethan Buchman <ethan@coinculture.info>

* fixes

* addressed Ethan's comments from Aug 18

* trustlevel

* description of 3 docs fixed

* added comment on justification asked for by Anca

* bla

* removed trustlevel for adjacent headers. Added bisection order.

* added comment in bisection

* fixed comment in checksupport

* added comment on bisection timeout