increase timeout threshold by liamsi · Pull Request #267 · celestiaorg/celestia-core

liamsi · 2021-04-05T16:21:12Z

migling with these timeouts to see if that fixes the CI fails #241 (comment)

liamsi · 2021-04-05T16:27:47Z

Still happening:

--- FAIL: TestStateLockPOLUnlockOnUnknownBlock (0.27s)
panic: Timeout expired while waiting for NewTimeout event [recovered]
	panic: Timeout expired while waiting for NewTimeout event

goroutine 77108 [running]:
testing.tRunner.func1.1(0x1ea1400, 0xc003d93d20)
	/opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1072 +0x46a
testing.tRunner.func1(0xc047aa9500)
	/opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1075 +0x636
panic(0x1ea1400, 0xc003d93d20)
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/panic.go:975 +0x47a
github.com/lazyledger/lazyledger-core/consensus.ensureNewEvent(0xc01c0fa720, 0x1, 0x0, 0x5f5e100, 0x22456c7, 0x32)
	/home/runner/work/lazyledger-core/lazyledger-core/consensus/common_test.go:486 +0x43f
github.com/lazyledger/lazyledger-core/consensus.ensureNewTimeout(...)
	/home/runner/work/lazyledger-core/lazyledger-core/consensus/common_test.go:524
github.com/lazyledger/lazyledger-core/consensus.TestStateLockPOLUnlockOnUnknownBlock(0xc047aa9500)
	/home/runner/work/lazyledger-core/lazyledger-core/consensus/state_test.go:830 +0xb25
testing.tRunner(0xc047aa9500, 0x23917d0)
	/opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1123 +0x203
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1168 +0x5bc
FAIL	github.com/lazyledger/lazyledger-core/consensus	81.675s

liamsi · 2021-04-05T16:42:51Z

Now TestBlockPoolBasic fails. No race, and the others in question just pass?

liamsi · 2021-04-05T17:01:54Z

Now there is this:

long output

SIGQUIT: quit
PC=0x47b0d9 m=5 sigcode=0

goroutine 0 [idle]:
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc002353800, 0x0, 0x0, 0x7fffffff, 0x7f91899eace0, 0x0, 0x0, ...)
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/traceback.go:189 +0x2b9
runtime.scanstack(0xc002353800, 0xc000053e98)
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/mgcmark.go:844 +0x17c
runtime.markroot.func1()
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/mgcmark.go:245 +0xc6
runtime.markroot(0xc000053e98, 0xbfe)
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/mgcmark.go:218 +0x310
runtime.gcDrain(0xc000053e98, 0xb)
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/mgcmark.go:1109 +0x118
runtime.gcBgMarkWorker.func2()
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/mgc.go:1979 +0x191
runtime.systemstack(0x7f9180000020)
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/proc.go:1116

goroutine 20 [GC worker (idle)]:
runtime.systemstack_switch()
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/asm_amd64.s:330 fp=0xc00005af60 sp=0xc00005af58 pc=0x48c800
runtime.gcBgMarkWorker(0xc000052800)
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/mgc.go:1945 +0x1be fp=0xc00005afd8 sp=0xc00005af60 pc=0x4392de
runtime.goexit()
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00005afe0 sp=0xc00005afd8 pc=0x48e5e1
created by runtime.gcBgMarkStartWorkers
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/mgc.go:1839 +0x77

goroutine 1 [chan receive, 7 minutes]:
testing.(*T).Run(0xc000104f00, 0x220d1c7, 0x13, 0x2391790, 0x1)
	/opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1169 +0x5f4
testing.runTests.func1(0xc000104f00)
	/opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1439 +0xa7
testing.tRunner(0xc000104f00, 0xc0010ffba0)
	/opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1123 +0x203
testing.runTests(0xc0002632a0, 0x30dbc80, 0x47, 0x47, 0xc012ee25a9989878, 0x6ff5b3a530, 0x32364e0, 0x0)
	/opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1437 +0x613
testing.(*M).Run(0xc0000f0780, 0x0)
	/opt/hostedtoolcache/go/1.15.10/x64/src/testing/testing.go:1345 +0x3b4
github.com/lazyledger/lazyledger-core/consensus.TestMain(0xc0000f0780)
	/home/runner/work/lazyledger-core/lazyledger-core/consensus/replay_test.go:44 +0x177
main.main()
	_testmain.go:265 +0x372

goroutine 1574 [chan receive, 8 minutes]:
github.com/lazyledger/lazyledger-core/libs/pubsub.(*Server).loop(0xc0002367e0, 0xc001e05bc0, 0xc001e05bf0)
	/home/runner/work/lazyledger-core/lazyledger-core/libs/pubsub/pubsub.go:324 +0xb9
created by github.com/lazyledger/lazyledger-core/libs/pubsub.(*Server).OnStart
	/home/runner/work/lazyledger-core/lazyledger-core/libs/pubsub/pubsub.go:310 +0x73

goroutine 7 [select, 9 minutes]:
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc00002f8c0)
	/home/runner/go/pkg/mod/github.com/ipfs/go-log@v1.0.4/writer/writer.go:71 +0x1b6
created by github.com/ipfs/go-log/writer.NewMirrorWriter
	/home/runner/go/pkg/mod/github.com/ipfs/go-log@v1.0.4/writer/writer.go:36 +0x13a

goroutine 31909 [runnable]:
github.com/dgraph-io/ristretto/z.(*AllocatorPool).freeupAllocators(0xc0234d1360)
	/home/runner/go/pkg/mod/github.com/dgraph-io/ristretto@v0.0.4-0.20210122082011-bb5d392ed82d/z/allocator.go:383 +0x265
created by github.com/dgraph-io/ristretto/z.NewAllocatorPool
	/home/runner/go/pkg/mod/github.com/dgraph-io/ristretto@v0.0.4-0.20210122082011-bb5d392ed82d/z/allocator.go:323 +0x139

goroutine 9 [runnable]:
go.opencensus.io/stats/view.(*worker).reportUsage(0xc0000f0200)
	/home/runner/go/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:364 +0xfe
go.opencensus.io/stats/view.(*worker).start(0xc0000f0200)
	/home/runner/go/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:280 +0x20b
created by go.opencensus.io/stats/view.init.0
	/home/runner/go/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:34 +0xb2

goroutine 1572 [chan receive, 9 minutes]:
github.com/lazyledger/lazyledger-core/libs/pubsub.(*Server).loop(0xc0002362a0, 0xc001e04000, 0xc001e04030)
	/home/runner/work/lazyledger-core/lazyledger-core/libs/pubsub/pubsub.go:324 +0xb9
created by github.com/lazyledger/lazyledger-core/libs/pubsub.(*Server).OnStart
	/home/runner/work/lazyledger-core/lazyledger-core/libs/pubsub/pubsub.go:310 +0x73

goroutine 18386 [chan receive, 8 minutes]:
github.com/lazyledger/lazyledger-core/libs/pubsub.(*Server).loop(0xc00014ec40, 0xc00364b560, 0xc00364b590)
	/home/runner/work/lazyledger-core/lazyledger-core/libs/pubsub/pubsub.go:324 +0xb9
created by github.com/lazyledger/lazyledger-core/libs/pubsub.(*Server).OnStart
	/home/runner/work/lazyledger-core/lazyledger-core/libs/pubsub/pubsub.go:310 +0x73

goroutine 1571 [chan receive, 9 minutes]:
github.com/lazyledger/lazyledger-core/libs/pubsub.(*Server).loop(0xc000093c70, 0xc0025a5200, 0xc0025a5230)
	/home/runner/work/lazyledger-core/lazyledger-core/libs/pubsub/pubsub.go:324 +0xb9
created by github.com/lazyledger/lazyledger-core/libs/pubsub.(*Server).OnStart
	/home/runner/work/lazyledger-core/lazyledger-core/libs/pubsub/pubsub.go:310 +0x73

goroutine 1573 [chan receive, 9 minutes]:
github.com/lazyledger/lazyledger-core/libs/pubsub.(*Server).loop(0xc0002364d0, 0xc001e04de0, 0xc001e04e10)
	/home/runner/work/lazyledger-core/lazyledger-core/libs/pubsub/pubsub.go:324 +0xb9
created by github.com/lazyledger/lazyledger-core/libs/pubsub.(*Server).OnStart
	/home/runner/work/lazyledger-core/lazyledger-core/libs/pubsub/pubsub.go:310 +0x73

goroutine 34 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7f91b1f48e88, 0x72, 0x72)
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0000f0a18, 0x72, 0x0, 0x0, 0x2135b1d)
	/opt/hostedtoolcache/go/1.15.10/x64/src/internal/poll/fd_poll_runtime.go:87 +0xe6
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.15.10/x64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0000f0a00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/hostedtoolcache/go/1.15.10/x64/src/internal/poll/fd_unix.go:394 +0x2fe
net.(*netFD).accept(0xc0000f0a00, 0x0, 0x0, 0x0)
	/opt/hostedtoolcache/go/1.15.10/x64/src/net/fd_unix.go:172 +0x58
net.(*TCPListener).accept(0xc00000ea40, 0x0, 0x0, 0x0)
	/opt/hostedtoolcache/go/1.15.10/x64/src/net/tcpsock_posix.go:139 +0x50
net.(*TCPListener).Accept(0xc00000ea40, 0x0, 0x0, 0x0, 0x0)
	/opt/hostedtoolcache/go/1.15.10/x64/src/net/tcpsock.go:261 +0x8e
github.com/lazyledger/lazyledger-core/p2p.(*MultiplexTransport).acceptPeers(0xc0003cd7a0)
	/home/runner/work/lazyledger-core/lazyledger-core/p2p/transport.go:266 +0x96
created by github.com/lazyledger/lazyledger-core/p2p.(*MultiplexTransport).Listen
	/home/runner/work/lazyledger-core/lazyledger-core/p2p/transport.go:259 +0x191

goroutine 39 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7f91b1f48da0, 0x72, 0x72)
	/opt/hostedtoolcache/go/1.15.10/x64/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0000f0d98, 0x72, 0x0, 0x0, 0x2135b1d)
	/opt/hostedtoolcache/go/1.15.10/x64/src/internal/poll/fd_poll_runtime.go:87 +0xe6
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.15.10/x64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0000f0d80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/hostedtoolcache/go/1.15.10/x64/src/internal/poll/fd_unix.go:394 +0x2fe
net.(*netFD).accept(0xc0000f0d80, 0x0, 0x8, 0xc00005b608)
	/opt/hostedtoolcache/go/1.15.10/x64/src/net/fd_unix.go:172 +0x58
net.(*TCPListener).accept(0xc00000f0e0, 0x211765e, 0xc001219e40, 0x213c224)
	/opt/hostedtoolcache/go/1.15.10/x64/src/net/tcpsock_posix.go:139 +0x50
net.(*TCPListener).Accept(0xc00000f0e0, 0x0, 0x0, 0x0, 0x0)
	/opt/hostedtoolcache/go/1.15.10/x64/src/net/tcpsock.go:261 +0x8e
*** Test killed: ran too long (9m0s).
FAIL	github.com/lazyledger/lazyledger-core/consensus	545.984s

liamsi · 2021-04-05T17:19:07Z

@marbar3778 feel free to merge that into your branch. It seems to fix some of the previously failing tests in the consensus package. But as I'm not really sure what is / was going (tests now seem to consistently pass on CI too), feel free to close and ignore it as well.

Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 3.3.0 to 4.0.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v3.3.0...v4.0.0) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

liamsi added 2 commits April 5, 2021 18:19

increase timeout threshold

58abfe8

increase timeout even further

c691da4

liamsi requested a review from tac0turtle April 5, 2021 17:15

liamsi marked this pull request as ready for review April 5, 2021 17:15

liamsi requested a review from evan-forbes as a code owner April 5, 2021 17:15

tac0turtle merged commit d0f7fdb into marko/db_changes Apr 5, 2021

tac0turtle deleted the ismail/marko/db_changes_increase_timeouts branch April 5, 2021 17:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

increase timeout threshold#267

increase timeout threshold#267
tac0turtle merged 2 commits intomarko/db_changesfrom
ismail/marko/db_changes_increase_timeouts

liamsi commented Apr 5, 2021

Uh oh!

liamsi commented Apr 5, 2021

Uh oh!

liamsi commented Apr 5, 2021

Uh oh!

liamsi commented Apr 5, 2021

Uh oh!

liamsi commented Apr 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

liamsi commented Apr 5, 2021

Uh oh!

liamsi commented Apr 5, 2021

Uh oh!

liamsi commented Apr 5, 2021

Uh oh!

liamsi commented Apr 5, 2021

Uh oh!

liamsi commented Apr 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants