-
Notifications
You must be signed in to change notification settings - Fork 4.1k
Persistent crash in Raft #3896
Copy link
Copy link
Closed
Description
I was trying to start the first of the three nodes in a local cluster (the other 2 nodes were down), and I got a panic. It looks like a Replica's mu.raftGroup is somehow nil. Perhaps it's because of the errors that precede the panic?
This is consistently reproducible with my set of 'data' directories, which I've packaged up here:
https://www.dropbox.com/s/qzw3d8eh62ujhqx/bad_data.tgz?dl=0
Below is the Cockroach log output. Let me know if you need any other info.
/Users/cdo/gocode/bin/cockroach start --insecure --stores=ssd=data1 --addr=127.0.0.1:26001 --pgaddr=127.0.0.1:15001 --gossip=127.0.0.1:26001,127.0.0.2:26002,127.0.0.3:26003
I0113 09:45:43.061175 31993 cli/start.go:141 build Vers: go1.5.2
I0113 09:45:43.061271 31993 cli/start.go:142 build Tag: v0.1-alpha-359-g80581b9
I0113 09:45:43.061282 31993 cli/start.go:143 build Time: 2016/01/13 14:45:22
I0113 09:45:43.061290 31993 cli/start.go:144 build Deps: github.com/VividCortex/ewma:c34099b489e4ac33ca8d8c5f9d29d6eeaf69f2ed github.com/biogo/store:3b4c041f52c224ee4a44f5c8b150d003a40643a0 github.com/cockroachdb/c-lz4:c40aaae2fc50293eb8750b34632bc3efe813e23f github.com/cockroachdb/c-protobuf:4feb192131ea08dfbd7253a00868ad69cbb61b81 github.com/cockroachdb/c-rocksdb:b7fb7bddcb55be35eacdf67e9e2c931083ce02c4 github.com/cockroachdb/c-snappy:5c6d0932e0adaffce4bfca7bdf2ac37f79952ccf github.com/cockroachdb/cockroach:80581b9b16187c17d4728669b96fc1feb9ac092c github.com/codahale/hdrhistogram:954f16e8b9ef0e5d5189456aa4c1202758e04f17 github.com/coreos/etcd:b35ab33045197fb31d22212e9e429e6cf72ea3db github.com/elazarl/go-bindata-assetfs:57eb5e1fc594ad4b0b1dbea7b286d299e0cb43c2 github.com/gogo/protobuf:c57e439bad574c2e0877ff18d514badcfced004d github.com/google/btree:cc6329d4279e3f025a53a83c397d2339b5705c45 github.com/julienschmidt/httprouter:21439ef4d70ba4f3e2a5ed9249e7b03af4019b40 github.com/lib/pq:2d785ad24be7a038e2fc35424b993645a0025425 github.com/mattn/go-runewidth:e882a96ec18dd43fa283187b66af74497c9101c0 github.com/montanaflynn/stats:2c10aa99e7ec8c4607d4427ec7a1a60fcdfce85f github.com/olekukonko/tablewriter:48dc4474bcf3e0134e9a64222207b1e020f171e9 github.com/peterh/liner:3f1c20449d1836aa4cbe38731b96f95cdf89634d github.com/rcrowley/go-metrics:7839c01b09d2b1d7068034e5fe6e423f6ac5be22 github.com/spf13/cobra:97f831a4fd01378ad85840fed4f43824bc3dd102 github.com/spf13/pflag:7f60f83a2c81bc3c3c0d5297f61ddfa68da9d3b7 golang.org/x/crypto:d67eb63455fa4d6fca5802332d86f1f204017e00 golang.org/x/net:76365a41624aa608f888301bf60fabecaf928b5a golang.org/x/text:cf4986612c83df6c55578ba198316d1684a9a287 gopkg.in/yaml.v1:9f9df34309c04878acc86042b16630b0f696e1de
I0113 09:45:43.061370 31993 server/context.go:186 1 storage engine(s) specified
I0113 09:45:43.061392 31993 cli/start.go:176 starting cockroach node
W0113 09:45:43.061412 31993 server/server.go:102 running in insecure mode, this is strongly discouraged. See --insecure and --certs.
I0113 09:45:43.063136 31993 storage/engine/rocksdb.go:106 opening rocksdb instance at "data1"
I0113 09:45:43.063152 31993 gossip/gossip.go:916 starting client to 127.0.0.2:26002
W0113 09:45:43.063480 31993 rpc/client.go:317 dial tcp 127.0.0.2:26002: getsockopt: connection refused
I0113 09:45:43.093544 31993 server/node.go:293 initialized store store=1:1 ([ssd]=data1): {Capacity:249779191808 Available:220275060736 RangeCount:0}
I0113 09:45:43.093736 31993 server/node.go:220 node ID 1 initialized
I0113 09:45:43.093870 31993 gossip/gossip.go:218 setting node descriptor node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:26001" > attrs:<>
W0113 09:45:43.094056 31993 storage/replica.go:1576 could not acquire lease for range gossip: range 1 was not found
W0113 09:45:43.094227 31993 storage/replica.go:1576 could not acquire lease for range gossip: range 1 was not found
W0113 09:45:43.094595 31993 storage/store.go:639 error gossiping system config: range 1 was not found
W0113 09:45:43.094613 31993 storage/store.go:620 error gossiping first range data: range 1 was not found
I0113 09:45:43.101676 31993 gossip/gossip.go:267 read 3 gossip host(s) for bootstrapping from persistent storage
I0113 09:45:43.101704 31993 server/node.go:389 connecting to gossip network to verify cluster ID...
I0113 09:45:43.101721 31993 server/node.go:406 node connected via gossip and verified as part of cluster "3ce7a37d-7215-41a2-9411-318829882572"
I0113 09:45:43.101772 31993 server/node.go:258 Started node with [[ssd]=data1] engine(s) and attributes []
I0113 09:45:43.101828 31993 server/server.go:262 starting http server at 127.0.0.1:26001
I0113 09:45:43.101942 31993 sql/pgwire/server.go:76 starting postgres server at 127.0.0.1:15001
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x46f7634]
goroutine 51 [running]:
github.com/coreos/etcd/raft.(*RawNode).Tick(0x0)
/Users/cdo/gocode/src/github.com/coreos/etcd/raft/rawnode.go:115 +0x14
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
/Users/cdo/gocode/src/github.com/cockroachdb/cockroach/storage/store.go:1604 +0x790
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820214a80, 0xc8203717e0)
/Users/cdo/gocode/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:89 +0x52
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/Users/cdo/gocode/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:90 +0x62
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels