reading specific rows never returns #18199
Description
Bug report
We are running a 3 nodes cluster under v1.0.5
Node 1:
I170904 17:40:50.553387 303137322 util/log/clog.go:910 [config] file created at: 2017/09/04 17:40:50
I170904 17:40:50.553387 303137322 util/log/clog.go:910 [config] running on machine: online-cockroach-a
I170904 17:40:50.553387 303137322 util/log/clog.go:910 [config] binary: CockroachDB CCL v1.0.5 (linux amd64, built 2017/08/24 17:43:46, go1.8.3)
I170904 17:40:50.553387 303137322 util/log/clog.go:910 [config] arguments: [/home/lefty/cockroach start --store /data/1/cockroach --store /data/2/cockroach --store /data/3/cockroach --host=0.0.0.0 --advertise-host=10.91.208.131 --certs-dir /home/lefty/cockroach_certs --http-host 0.0.0.0 --http-port 8080 --join 10.91.151.188 --join 10.91.155.209]
Node 2:
I170904 17:44:10.433957 250568015 util/log/clog.go:910 [config] file created at: 2017/09/04 17:44:10
I170904 17:44:10.433957 250568015 util/log/clog.go:910 [config] running on machine: online-cockroach-b
I170904 17:44:10.433957 250568015 util/log/clog.go:910 [config] binary: CockroachDB CCL v1.0.5 (linux amd64, built 2017/08/24 17:43:46, go1.8.3)
I170904 17:44:10.433957 250568015 util/log/clog.go:910 [config] arguments: [/home/lefty/cockroach start --store /data/1/cockroach --store /data/2/cockroach --store /data/3/cockroach --host=0.0.0.0 --advertise-host=10.91.151.188 --certs-dir /home/lefty/cockroach_certs --http-host 0.0.0.0 --http-port 8080 --join 10.91.208.131 --join 10.91.155.209]
Node 3:
I170904 17:44:34.696304 36972590 util/log/clog.go:910 [config] file created at: 2017/09/04 17:44:34
I170904 17:44:34.696304 36972590 util/log/clog.go:910 [config] running on machine: online-cockroach-c
I170904 17:44:34.696304 36972590 util/log/clog.go:910 [config] binary: CockroachDB CCL v1.0.5 (linux amd64, built 2017/08/24 17:43:46, go1.8.3)
I170904 17:44:34.696304 36972590 util/log/clog.go:910 [config] arguments: [/home/lefty/cockroach start --store /data/1/cockroach --store /data/2/cockroach --store /data/3/cockroach --host=0.0.0.0 --advertise-host=10.91.155.209 --certs-dir /home/lefty/certs --http-host 0.0.0.0 --http-port 8080 --join 10.91.208.131 --join 10.91.151.188]
Some of our queries stopped returning results, even simple queries such as
SELECT id from users where id = '2970788235'; ( id is the primary key of the table )
We tried those queries from both our java code and cockroach cli.
While this happens for specific rows, the server continues to work fine when reading others.
Not sure it is related but our cockroach logs output a lot of
W170904 17:30:35.746218 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746247 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746295 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746343 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746392 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746445 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746506 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746536 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746565 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746605 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746642 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746693 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746739 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746777 241 storage/gc_queue.go:417 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] unable to resolve intents of committed txn on gc: context deadline exceeded
W170904 17:30:35.746897 302617246 storage/gc_queue.go:879 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] push of txn id=ac2d8029 key=/Table/52/1/"5320052451#1500557009361"/0 rw=false pri=0.01513283 iso=SERIALIZABLE stat=PE
NDING epo=0 ts=1500557007.703880494,0 orig=0.000000000,0 max=0.000000000,0 wto=false rop=false seq=3 failed: context deadline exceeded
W170904 17:30:35.746907 302617051 storage/gc_queue.go:879 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] push of txn id=e925a948 key=/Table/52/1/"5319772163#1500556927647"/0 rw=false pri=0.00135358 iso=SERIALIZABLE stat=PE
NDING epo=0 ts=1500556918.902844582,0 orig=0.000000000,0 max=0.000000000,0 wto=false rop=false seq=4 failed: context deadline exceeded
W170904 17:30:35.746912 302617057 storage/gc_queue.go:879 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] push of txn id=e93d4ffc key=/Table/52/1/"5320670056#1500972905215"/0 rw=false pri=0.03541256 iso=SERIALIZABLE stat=PE
NDING epo=0 ts=1500972901.427410919,0 orig=0.000000000,0 max=0.000000000,0 wto=false rop=false seq=3 failed: context deadline exceeded
W170904 17:30:35.746933 302617247 storage/gc_queue.go:879 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] push of txn id=541e71ce key=/Table/52/1/"5320052451#1500557180174"/0 rw=false pri=0.02997946 iso=SERIALIZABLE stat=PE
NDING epo=0 ts=1500557104.540700487,0 orig=0.000000000,0 max=0.000000000,0 wto=false rop=false seq=5 failed: context deadline exceeded
W170904 17:30:35.746947 302617189 storage/gc_queue.go:879 [gc,n1,s2,r10122/3:/Table/52/1/"53{19312…-20860…}] push of txn id=aae047d9 key=/Table/52/1/"5320063883#1500516197839"/0 rw=false pri=0.01194207 iso=SERIALIZABLE stat=PE
NDING epo=0 ts=1500516197.750966587,0 orig=0.000000000,0 max=0.000000000,0 wto=false rop=false seq=3 failed: context deadline exceeded