Often, the test fails only in circleCI.
The same error does not occur in local.
The following are examples.
But the error is not repeated and occurred randomly.
TestMempoolProgressAfterCreateEmptyBlocksInterval in test_cover
--- FAIL: TestMempoolProgressAfterCreateEmptyBlocksInterval (0.29s)
panic: Timeout expired while waiting for new activity on the channel [recovered]
panic: Timeout expired while waiting for new activity on the channel
goroutine 1027 [running]:
testing.tRunner.func1.1(0x1130d40, 0x15127d0)
/usr/local/go/src/testing/testing.go:941 +0x5d0
testing.tRunner.func1(0xc001076240)
/usr/local/go/src/testing/testing.go:944 +0x600
panic(0x1130d40, 0x15127d0)
/usr/local/go/src/runtime/panic.go:973 +0x3e3
github.com/tendermint/tendermint/consensus.ensureNewEventOnChannel(0xc0014787e0)
/go/src/github.com/tendermint/tendermint/consensus/common_test.go:677 +0x143
github.com/tendermint/tendermint/consensus.TestMempoolProgressAfterCreateEmptyBlocksInterval(0xc001076240)
/go/src/github.com/tendermint/tendermint/consensus/mempool_test.go:58 +0x46c
testing.tRunner(0xc001076240, 0x13ddf90)
/usr/local/go/src/testing/testing.go:992 +0x1ec
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1043 +0x661
FAIL github.com/tendermint/tendermint/consensus 2.003s
FAIL
https://app.circleci.com/pipelines/github/line/tendermint/180/workflows/eb9f0815-ee92-406e-bac8-29d93c1e4395/jobs/1738
TestDontExhaustMaxActiveIDs in test_cover
error switch=0 peer=MockPeer err="unmarshal to mempool.Message failed after 0 bytes (EOF while reading prefix bytes.): 010203"
E[2020-06-08|04:48:31.290] Error decoding message validator=0 src=MockPeer chId=48 msg=null err="unmarshal to mempool.Message failed after 0 bytes (EOF while reading prefix bytes.): 010203" bytes="\u0001\u0002\u0003"
E[2020-06-08|04:48:31.290] Stopping peer for error switch=0 peer=MockPeer err="unmarshal to mempool.Message failed after 0 bytes (EOF while reading prefix bytes.): 010203"
E[2020-06-08|04:48:31.293] Error decoding message validator=0 src=MockPeer chId=48 msg=null err="unmarshal to mempool.Message failed after 0 bytes (EOF while reading prefix bytes.): 010203" bytes="\u0001\u0002\u0003"
E[2020-06-08|04:48:31.293] Stopping peer for error switch=0 peer=MockPeer err="unmarshal to mempool.Message failed after 0 bytes (EOF while reading prefix bytes.): 010203"
…
goroutine 1523 [runnable]:
crypto/ed25519/internal/edwards25519.FeCombine(0xc00212f8a0, 0xff7af5ab69e77867, 0x12dcb327206a54, 0xfff74f7867b53278, 0xffe9abae69ebe6ac, 0xffe9344ed5d17010, 0x399b2045bd4c6, 0x2439589c62070c, 0xffebb05936a2e0b4, 0x170a0f9a7fffcc, ...)
/usr/local/go/src/crypto/ed25519/internal/edwards25519/edwards25519.go:260 +0x357
crypto/ed25519/internal/edwards25519.FeSquare(0xc00212f8a0, 0xc00212f8c8)
/usr/local/go/src/crypto/ed25519/internal/edwards25519/edwards25519.go:477 +0x47
crypto/ed25519/internal/edwards25519.FeInvert(0xc00212f9e0, 0xc00212fb48)
/usr/local/go/src/crypto/ed25519/internal/edwards25519/edwards25519.go:545 +0x376
crypto/ed25519/internal/edwards25519.(*ExtendedGroupElement).ToBytes(0xc00212faf8, 0xc00212fa78)
/usr/local/go/src/crypto/ed25519/internal/edwards25519/edwards25519.go:711 +0xac
crypto/ed25519.newKeyFromSeed(0xc00212fc38, 0x40, 0x40, 0xc00125e020, 0x20, 0x20)
/usr/local/go/src/crypto/ed25519/ed25519.go:120 +0x194
crypto/ed25519.NewKeyFromSeed(...)
/usr/local/go/src/crypto/ed25519/ed25519.go:101
golang.org/x/crypto/ed25519.NewKeyFromSeed(...)
/go/pkg/mod/golang.org/x/crypto@v0.0.0-20200406173513-056763e48d71/ed25519/ed25519_go113.go:60
github.com/tendermint/tendermint/crypto/ed25519.genPrivKey(0x13f3b20, 0xc0000db770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/tendermint/tendermint/crypto/ed25519/ed25519.go:108 +0x14b
github.com/tendermint/tendermint/crypto/ed25519.GenPrivKey(...)
/go/src/github.com/tendermint/tendermint/crypto/ed25519/ed25519.go:97
github.com/tendermint/tendermint/p2p/mock.NewPeer(0x0, 0x0, 0x0, 0xc002190b90)
/go/src/github.com/tendermint/tendermint/p2p/mock/peer.go:30 +0x162
github.com/tendermint/tendermint/mempool.TestDontExhaustMaxActiveIDs(0xc001c06000)
/go/src/github.com/tendermint/tendermint/mempool/reactor_test.go:239 +0x104
testing.tRunner(0xc001c06000, 0x12d4fc8)
/usr/local/go/src/testing/testing.go:992 +0x1ec
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1043 +0x661
FAIL github.com/tendermint/tendermint/mempool 301.022s
FAIL
https://app.circleci.com/pipelines/github/line/tendermint/187/workflows/12acb5e0-f7dd-44c0-890e-3c463c573ab8/jobs/1791
TestMempoolNoProgressUntilTxsAvailable in test_cover
--- FAIL: TestMempoolNoProgressUntilTxsAvailable (0.30s)
panic: Timeout expired while waiting for new activity on the channel [recovered]
panic: Timeout expired while waiting for new activity on the channel
goroutine 734 [running]:
testing.tRunner.func1.1(0x112fca0, 0x1511030)
/usr/local/go/src/testing/testing.go:941 +0x5d0
testing.tRunner.func1(0xc000fc2240)
/usr/local/go/src/testing/testing.go:944 +0x600
panic(0x112fca0, 0x1511030)
/usr/local/go/src/runtime/panic.go:973 +0x3e3
github.com/tendermint/tendermint/consensus.ensureNewEventOnChannel(0xc0019ba360)
/go/src/github.com/tendermint/tendermint/consensus/common_test.go:677 +0x143
github.com/tendermint/tendermint/consensus.TestMempoolNoProgressUntilTxsAvailable(0xc000fc2240)
/go/src/github.com/tendermint/tendermint/consensus/mempool_test.go:40 +0x468
testing.tRunner(0xc000fc2240, 0x13dcb48)
/usr/local/go/src/testing/testing.go:992 +0x1ec
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1043 +0x661
FAIL github.com/tendermint/tendermint/consensus 1.751s
FAIL
https://app.circleci.com/pipelines/github/line/tendermint/187/workflows/f0e22d4b-abad-4cf9-82d7-393446327692/jobs/1790
pex test fail in test_cover
FAIL github.com/tendermint/tendermint/p2p/pex 21.610s
FAIL
https://app.circleci.com/pipelines/github/line/tendermint/187/workflows/a13b060a-179c-4dac-a5be-d3b1477fbf49/jobs/1788
Often, the test fails only in circleCI.
The same error does not occur in local.
The following are examples.
But the error is not repeated and occurred randomly.
TestMempoolProgressAfterCreateEmptyBlocksInterval in test_cover
https://app.circleci.com/pipelines/github/line/tendermint/180/workflows/eb9f0815-ee92-406e-bac8-29d93c1e4395/jobs/1738
TestDontExhaustMaxActiveIDs in test_cover
https://app.circleci.com/pipelines/github/line/tendermint/187/workflows/12acb5e0-f7dd-44c0-890e-3c463c573ab8/jobs/1791
TestMempoolNoProgressUntilTxsAvailable in test_cover
https://app.circleci.com/pipelines/github/line/tendermint/187/workflows/f0e22d4b-abad-4cf9-82d7-393446327692/jobs/1790
pex test fail in test_cover
https://app.circleci.com/pipelines/github/line/tendermint/187/workflows/a13b060a-179c-4dac-a5be-d3b1477fbf49/jobs/1788