I180801 21:47:39.648271 1 util/log/clog.go:1134 [config] file created at: 2018/08/01 21:47:39 I180801 21:47:39.648271 1 util/log/clog.go:1134 [config] running on machine: srv8-sv I180801 21:47:39.648271 1 util/log/clog.go:1134 [config] binary: CockroachDB CCL v2.0.4 (x86_64-unknown-linux-gnu, built 2018/07/16 20:25:32, go1.10) I180801 21:47:39.648271 1 util/log/clog.go:1134 [config] arguments: [/opt/cockroachdb/cockroach start --locality=datacenter=SV --cache=25% --max-sql-memory=25% --insecure --store /data/cockroachdb --advertise-host 10.64.2.25 --join 10.64.2.17 --join 10.64.2.19 --join 10.64.2.21 --join 10.64.2.23 --join 10.64.3.17 --join 10.64.3.19 --join 10.64.3.21 --join 10.64.3.23 --join 10.64.3.25 --join 10.64.4.17 --join 10.64.4.19 --join 10.64.4.21 --join 10.64.4.23 --join 10.64.4.25] I180801 21:47:39.648271 1 util/log/clog.go:1134 line format: [IWEF]yymmdd hh:mm:ss.uuuuuu goid file:line msg utf8=✓ W180801 21:47:39.648271 1 cli/start.go:909 RUNNING IN INSECURE MODE! - Your cluster is open for any client that can access . - Any user, even root, can log in without providing a password. - Any user, connecting as root, can read or write any data in your cluster. - There is no network encryption nor authentication, and thus no confidentiality. Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v2.0/secure-a-cluster.html I180801 21:47:39.648501 1 cli/start.go:923 CockroachDB CCL v2.0.4 (x86_64-unknown-linux-gnu, built 2018/07/16 20:25:32, go1.10) I180801 21:47:39.749933 1 server/config.go:330 available memory from cgroups (8.0 EiB) exceeds system memory 31 GiB, using system memory I180801 21:47:39.749994 1 server/config.go:430 system total memory: 31 GiB I180801 21:47:39.750194 1 server/config.go:432 server configuration: max offset 500000000 cache size 7.8 GiB SQL memory pool size 7.8 GiB scan interval 10m0s scan max idle time 200ms event log enabled true I180801 21:47:39.750259 1 cli/start.go:796 process identity: uid 73 euid 73 gid 73 egid 73 I180801 21:47:39.750291 1 cli/start.go:461 starting cockroach node I180801 21:47:39.751624 15 storage/engine/rocksdb.go:552 opening rocksdb instance at "/data/cockroachdb/cockroach-temp640197922" I180801 21:47:40.051752 15 server/server.go:734 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled I180801 21:47:40.053006 15 storage/engine/rocksdb.go:552 opening rocksdb instance at "/data/cockroachdb" I180801 21:47:40.933503 15 server/config.go:538 [n?] 1 storage engine initialized I180801 21:47:40.933542 15 server/config.go:541 [n?] RocksDB cache size: 7.8 GiB I180801 21:47:40.933561 15 server/config.go:541 [n?] store 0: RocksDB, max size 0 B, max open file limit 10000 W180801 21:47:40.944628 15 gossip/gossip.go:1293 [n?] no incoming or outgoing connections I180801 21:47:40.952837 96 gossip/client.go:129 [n17] started gossip client to 10.64.2.17:26257 I180801 21:47:43.944130 15 server/node.go:507 [n17] initialized store [n17,s17]: disk (capacity=724 GiB, available=624 GiB, used=6.3 GiB, logicalBytes=26 GiB), ranges=9355, leases=0, writes=0.00, bytesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=19783464.00 pMax=54905834.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00} I180801 21:47:43.944149 15 server/node.go:347 [n17] node ID 17 initialized I180801 21:47:43.944215 15 gossip/gossip.go:333 [n17] NodeDescriptor set to node_id:17 address: attrs:<> locality: > ServerVersion: I180801 21:47:43.944312 15 storage/stores.go:331 [n17] read 14 node addresses from persistent storage I180801 21:47:43.944443 15 server/node.go:648 [n17] connecting to gossip network to verify cluster ID... I180801 21:47:43.944456 15 server/node.go:673 [n17] node connected via gossip and verified as part of cluster "4cd9c226-665f-42e1-9345-0c47f2d70fc5" I180801 21:47:43.944472 15 server/node.go:441 [n17] node=17: started with [=/data/cockroachdb] engine(s) and attributes [] I180801 21:47:43.944530 15 server/server.go:1430 [n17] starting http server at srv8-sv:8080 I180801 21:47:43.944539 15 server/server.go:1431 [n17] starting grpc/postgres server at srv8-sv:26257 I180801 21:47:43.944547 15 server/server.go:1432 [n17] advertising CockroachDB node at 10.64.2.25:26257 W180801 21:47:43.944722 15 sql/jobs/registry.go:300 [n17] canceling all jobs due to liveness failure I180801 21:47:44.804308 550 storage/raft_transport.go:459 [n17] raft transport stream to node 13 established I180801 21:47:44.806031 552 storage/raft_transport.go:459 [n17] raft transport stream to node 15 established I180801 21:47:44.940410 551 storage/raft_transport.go:459 [n17] raft transport stream to node 24 established I180801 21:47:44.940736 553 storage/raft_transport.go:459 [n17] raft transport stream to node 26 established W180801 21:47:44.944965 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:47:44.971864 536 storage/raft_transport.go:459 [n17] raft transport stream to node 22 established I180801 21:47:44.994430 336 storage/raft_transport.go:459 [n17] raft transport stream to node 18 established I180801 21:47:45.294636 543 storage/raft_transport.go:459 [n17] raft transport stream to node 20 established I180801 21:47:45.339662 542 storage/raft_transport.go:459 [n17] raft transport stream to node 27 established I180801 21:47:45.394731 528 storage/raft_transport.go:459 [n17] raft transport stream to node 21 established W180801 21:47:45.945096 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:47:45.949532 674 storage/raft_transport.go:459 [n17] raft transport stream to node 16 established I180801 21:47:46.005389 595 storage/raft_transport.go:459 [n17] raft transport stream to node 14 established W180801 21:47:46.945271 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:47:47.240247 627 storage/raft_transport.go:459 [n17] raft transport stream to node 23 established I180801 21:47:47.523825 628 storage/raft_transport.go:459 [n17] raft transport stream to node 25 established W180801 21:47:47.945478 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:47:48.112992 696 storage/raft_transport.go:459 [n17] raft transport stream to node 19 established W180801 21:47:48.445191 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:47:48.445200 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:47:48.945695 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:47:49.945903 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:47:50.946045 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:47:51.946248 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:47:52.945444 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:47:52.945451 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:47:52.946332 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:47:53.945125 500 server/status/runtime.go:219 [n17] runtime stats: 545 MiB RSS, 370 goroutines, 130 MiB/49 MiB/238 MiB GO alloc/idle/total, 222 MiB/263 MiB CGO alloc/total, 0.00cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (11x) W180801 21:47:53.946409 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:47:53.974286 480 storage/store.go:4516 [n17,s17] sstables (read amplification = 4): 0 [ 4M 1 ]: 4M 4 [ 4M 3 ]: 4M 714K 16K 5 [ 272M 55 ]: 12M 10M[2] 9M[10] 8M[3] 7M[2] 6M[4] 5M[6] 4M[3] 3M[10] 2M[6] 1M[2] 664K 547K 457K 433K 371K 287K 6 [ 6G 130 ]: 100M[36] 99M[6] 98M[2] 97M[2] 96M[2] 95M[3] 94M 93M 92M[2] 91M[2] 88M 84M 59M 18M 17M 15M 12M 11M[3] 9M[2] 7M 6M 5M[2] 4M[7] 3M[4] 2M[3] 1M[7] 968K 893K 797K 717K 683K 589K 555K 519K[2] 482K 474K 424K 423K 419K 398K 390K[2] 324K 313K 312K 288K 263K 164K 156K 149K 146K 137K 134K 125K 110K 98K 91K 80K 53K 26K 22K I180801 21:47:53.974359 480 storage/store.go:4517 [n17,s17] ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 3.68 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 8.2 0 1 0.448 0 0 L4 3/0 4.42 MB 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 L5 55/1 272.43 MB 0.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0 L6 130/1 5.91 GB 0.0 0.1 0.0 0.1 0.1 0.0 0.0 10.3 15.2 14.0 7 1 7.226 0 20K Sum 189/2 6.19 GB 0.0 0.1 0.0 0.1 0.1 0.0 0.0 28.5 14.3 13.6 8 2 3.837 0 20K Int 0/0 0.00 KB 0.0 0.1 0.0 0.1 0.1 0.0 0.0 28.5 14.3 13.6 8 2 3.837 0 20K Uptime(secs): 13.9 total, 13.9 interval Flush(GB): cumulative 0.004, interval 0.004 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.10 GB write, 7.52 MB/s write, 0.11 GB read, 7.90 MB/s read, 7.7 seconds Interval compaction: 0.10 GB write, 7.52 MB/s write, 0.11 GB read, 7.90 MB/s read, 7.7 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count estimated_pending_compaction_bytes: 0 B W180801 21:47:54.946623 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:47:55.946838 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:47:56.946992 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:47:57.445695 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:47:57.445702 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:47:57.947150 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:47:58.947238 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:47:59.265449 901 storage/replica_raftstorage.go:726 [n17,s17,r23/51:/System/{NodeLive…-tsd}] applying Raft snapshot at index 39268166 (id=5568b8d7, encoded size=478762, 2 rocksdb batches, 2 log entries) I180801 21:47:59.380323 901 storage/replica_raftstorage.go:732 [n17,s17,r23/51:/System/{NodeLive…-tsd}] applied Raft snapshot in 115ms [clear=3ms batch=0ms entries=0ms commit=111ms] W180801 21:47:59.947493 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:00.947749 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:01.946184 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:01.946204 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:01.947910 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:02.948103 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:48:03.946166 500 server/status/runtime.go:219 [n17] runtime stats: 628 MiB RSS, 369 goroutines, 198 MiB/38 MiB/288 MiB GO alloc/idle/total, 246 MiB/293 MiB CGO alloc/total, 975.82cgo/sec, 0.28/0.07 %(u/s)time, 0.00 %gc (1x) W180801 21:48:03.948237 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:04.948447 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:05.948652 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:06.446828 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:06.446851 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 21:48:06.831276 28 gossip/gossip.go:1307 [n17] node has connected to cluster via gossip I180801 21:48:06.831859 28 storage/stores.go:350 [n17] wrote 14 node addresses to persistent storage W180801 21:48:06.948829 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:07.949044 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:08.949194 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:09.949306 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:10.944895 954 server/server.go:1267 The server appears to be unable to contact the other nodes in the cluster. Please try - starting the other nodes, if you haven't already - double-checking that the '--join' and '--host' flags are set up correctly - running the 'cockroach init' command if you are trying to initialize a new cluster If problems persist, please see https://www.cockroachlabs.com/docs/v2.0/cluster-setup-troubleshooting.html. W180801 21:48:10.947430 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:10.947521 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:10.949479 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:11.949640 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:12.949808 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:48:13.945978 500 server/status/runtime.go:219 [n17] runtime stats: 643 MiB RSS, 369 goroutines, 216 MiB/22 MiB/289 MiB GO alloc/idle/total, 263 MiB/310 MiB CGO alloc/total, 474.90cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:48:13.950016 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:14.950306 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:15.448224 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:15.448244 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:15.950510 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:16.950738 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:17.950964 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:18.951126 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:19.949094 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:19.949126 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:19.951293 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:20.951616 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:21.951822 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:22.952009 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:48:23.945061 500 server/status/runtime.go:219 [n17] runtime stats: 660 MiB RSS, 371 goroutines, 234 MiB/5.1 MiB/289 MiB GO alloc/idle/total, 269 MiB/318 MiB CGO alloc/total, 481.83cgo/sec, 0.33/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:48:23.952101 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:24.449939 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:24.449962 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:24.952364 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:25.952494 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:26.952624 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:27.952868 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:28.950593 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:28.950614 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:28.953022 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:29.953192 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:30.953426 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:31.953592 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:32.953748 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:33.451300 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:33.451322 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 21:48:33.946011 500 server/status/runtime.go:219 [n17] runtime stats: 673 MiB RSS, 372 goroutines, 139 MiB/92 MiB/294 MiB GO alloc/idle/total, 282 MiB/332 MiB CGO alloc/total, 462.57cgo/sec, 0.33/0.08 %(u/s)time, 0.00 %gc (2x) W180801 21:48:33.953915 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:34.954008 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:35.954247 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:36.954462 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:37.951985 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:37.952006 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:37.954672 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:38.954776 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:39.954989 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:40.955190 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:41.955324 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:42.452539 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:42.452561 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:42.955554 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:43.903995 474 storage/replica.go:1502 [n17,s17,r1/36:/{Min-System/}] have been waiting 1m0s attempting to acquire lease I180801 21:48:43.944964 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (1m3s: infos 14/6566 sent/received, bytes 6383B/1235469B sent/received) gossip server (0/3 cur/max conns, infos 14/6566 sent/received, bytes 6383B/1235469B sent/received) W180801 21:48:43.945067 349 storage/replica.go:1502 [n17,s17,r1/36:/{Min-System/}] have been waiting 1m0s attempting to acquire lease W180801 21:48:43.945153 347 kv/dist_sender.go:1305 [range-lookup=/System/"system-version/",range-lookup=/Meta2/System/"system-version/\x00",n17] have been waiting 1m0s sending RPC to r1 (currently pending: [(n17,s17):36]) for batch: Scan [/Meta1/System/"system-version/\x00\x00",/Meta2/"") I180801 21:48:43.945760 500 server/status/runtime.go:219 [n17] runtime stats: 697 MiB RSS, 372 goroutines, 147 MiB/94 MiB/303 MiB GO alloc/idle/total, 294 MiB/347 MiB CGO alloc/total, 465.71cgo/sec, 0.34/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:48:43.946464 383 storage/replica.go:1502 [n17,s17,r1/36:/{Min-System/}] have been waiting 1m0s attempting to acquire lease W180801 21:48:43.946542 382 kv/dist_sender.go:1305 [ts-poll,range-lookup=/System/tsd/cr.node.build.timestamp/17/10s/2018-08-01T21:00:00Z,n17] have been waiting 1m0s sending RPC to r1 (currently pending: [(n17,s17):36]) for batch: Scan [/Meta2/System/tsd/cr.node.build.timestamp/17/10s/2018-08-01T21:00:00Z,/System/"") W180801 21:48:43.955723 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:43.957101 61 storage/replica.go:1502 [n17,s17,r1/36:/{Min-System/}] have been waiting 1m0s attempting to acquire lease W180801 21:48:43.957144 60 kv/dist_sender.go:1305 [summaries,range-lookup=/System/StatusNode/17,n17] have been waiting 1m0s sending RPC to r1 (currently pending: [(n17,s17):36]) for batch: Scan [/Meta2/System/StatusNode/17/NULL,/System/"") W180801 21:48:44.956028 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:45.956180 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:46.952958 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:46.952974 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:46.956286 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:47.956437 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:48.956614 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:49.956855 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:50.957027 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:51.453546 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:51.453568 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:51.957233 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:52.957385 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:48:53.946621 500 server/status/runtime.go:219 [n17] runtime stats: 703 MiB RSS, 374 goroutines, 162 MiB/83 MiB/303 MiB GO alloc/idle/total, 298 MiB/352 MiB CGO alloc/total, 466.97cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:48:53.957639 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:54.957871 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:55.954225 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:48:55.954248 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:48:55.958035 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:56.958260 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:57.958496 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:58.958695 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:48:59.958890 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:00.454392 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:00.454410 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:00.959120 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:01.959365 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:02.959574 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:49:03.945942 500 server/status/runtime.go:219 [n17] runtime stats: 716 MiB RSS, 374 goroutines, 171 MiB/74 MiB/303 MiB GO alloc/idle/total, 312 MiB/367 MiB CGO alloc/total, 498.03cgo/sec, 0.35/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:49:03.959742 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:04.954603 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:04.954627 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:04.959880 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:05.960164 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:06.960380 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:07.960592 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:08.960749 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:09.454867 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:09.454891 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:09.960935 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:10.961312 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:11.961540 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:12.961760 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:49:13.945561 500 server/status/runtime.go:219 [n17] runtime stats: 725 MiB RSS, 374 goroutines, 188 MiB/60 MiB/304 MiB GO alloc/idle/total, 324 MiB/381 MiB CGO alloc/total, 534.91cgo/sec, 0.36/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:49:13.946048 1037 storage/replica.go:1502 [n17,s17,r1/36:/{Min-System/}] have been waiting 1m0s attempting to acquire lease W180801 21:49:13.946101 1036 kv/dist_sender.go:1305 [txn=064ad6e3,range-lookup=/Table/2/1/0/"system"/3/1,n17] have been waiting 1m0s sending RPC to r1 (currently pending: [(n17,s17):36]) for batch: Scan [/Meta2/Table/2/1/0/"system"/3/1/NULL,/System/"") W180801 21:49:13.955024 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:13.955037 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:13.961865 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:14.962118 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:15.962318 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:16.962540 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:17.962787 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:18.455266 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:18.455293 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:18.963254 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:19.963497 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:20.963791 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:21.964070 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:22.955676 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:22.955713 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:22.964220 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:49:23.945890 500 server/status/runtime.go:219 [n17] runtime stats: 740 MiB RSS, 374 goroutines, 198 MiB/54 MiB/307 MiB GO alloc/idle/total, 329 MiB/386 MiB CGO alloc/total, 461.79cgo/sec, 0.33/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:49:23.964345 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:24.964522 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:25.964724 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:26.964888 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:27.455907 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:27.455934 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:27.965103 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:28.965261 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:29.965485 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:30.965827 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:31.956148 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:31.956173 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:31.966043 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:32.966227 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:49:33.945917 500 server/status/runtime.go:219 [n17] runtime stats: 758 MiB RSS, 374 goroutines, 205 MiB/50 MiB/309 MiB GO alloc/idle/total, 342 MiB/400 MiB CGO alloc/total, 493.90cgo/sec, 0.33/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:49:33.966361 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:34.966647 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:35.966902 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:36.456408 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:36.456433 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:36.967127 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:37.967339 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:38.967576 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:39.967841 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:40.956634 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:40.956657 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:40.968029 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:41.968132 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:42.968280 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:49:43.944703 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (2m3s: infos 18/13262 sent/received, bytes 9087B/2208509B sent/received) gossip server (0/3 cur/max conns, infos 18/13262 sent/received, bytes 9087B/2208509B sent/received) I180801 21:49:43.946232 500 server/status/runtime.go:219 [n17] runtime stats: 708 MiB RSS, 374 goroutines, 197 MiB/56 MiB/310 MiB GO alloc/idle/total, 294 MiB/355 MiB CGO alloc/total, 419.99cgo/sec, 0.45/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:49:43.968463 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:44.968663 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:45.457046 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:45.457071 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:45.968830 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:46.968969 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:47.969208 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:48.969361 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:49.957250 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:49.957273 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:49.969588 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:50.969754 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:51.969990 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:52.970431 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:53.594074 1452 storage/replica.go:1502 [timeSeriesMaintenance,n17,s17,r3068/27:/System/tsd/cr.node.sql.mem.d…] have been waiting 1m0s attempting to acquire lease W180801 21:49:53.594221 1453 storage/node_liveness.go:501 [n17,s17,r3068/27:/System/tsd/cr.node.sql.mem.d…] slow heartbeat took 60.0s E180801 21:49:53.594233 1453 storage/replica_range_lease.go:282 [n17,s17,r3068/27:/System/tsd/cr.node.sql.mem.d…] context canceled I180801 21:49:53.945813 500 server/status/runtime.go:219 [n17] runtime stats: 723 MiB RSS, 372 goroutines, 215 MiB/43 MiB/312 MiB GO alloc/idle/total, 300 MiB/362 MiB CGO alloc/total, 531.52cgo/sec, 0.38/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:49:53.970565 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:54.457831 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:54.457854 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:54.970742 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:55.970899 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:56.971135 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:57.971368 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:58.958389 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:49:58.958418 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:49:58.971518 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:49:59.971668 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:00.971923 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:01.972154 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:02.972306 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:03.459045 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:03.459070 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 21:50:03.945852 500 server/status/runtime.go:219 [n17] runtime stats: 733 MiB RSS, 372 goroutines, 222 MiB/37 MiB/313 MiB GO alloc/idle/total, 313 MiB/376 MiB CGO alloc/total, 435.20cgo/sec, 0.34/0.10 %(u/s)time, 0.00 %gc (1x) W180801 21:50:03.972410 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:04.972530 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:05.972641 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:06.972770 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:07.959432 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:07.959461 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:07.972925 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:08.973086 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:09.973260 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:10.973456 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:11.973615 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:12.460094 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:12.460117 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:12.973775 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:50:13.945845 500 server/status/runtime.go:219 [n17] runtime stats: 751 MiB RSS, 372 goroutines, 226 MiB/36 MiB/315 MiB GO alloc/idle/total, 328 MiB/392 MiB CGO alloc/total, 468.80cgo/sec, 0.24/0.07 %(u/s)time, 0.00 %gc (1x) W180801 21:50:13.973931 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:14.974235 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:15.974459 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:16.960842 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:16.960865 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:16.974610 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:17.974842 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:18.974996 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:19.975163 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:20.975334 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:21.461341 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:21.461358 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:21.975548 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:22.975701 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:50:23.945881 500 server/status/runtime.go:219 [n17] runtime stats: 763 MiB RSS, 372 goroutines, 236 MiB/28 MiB/318 MiB GO alloc/idle/total, 333 MiB/398 MiB CGO alloc/total, 503.60cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:50:23.975795 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:24.975949 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:25.961634 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:25.961645 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:25.976113 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:26.976269 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:27.976465 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:28.976634 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:29.976809 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:30.462221 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:30.462241 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:30.976971 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:31.977124 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:32.977281 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:50:33.945919 500 server/status/runtime.go:219 [n17] runtime stats: 780 MiB RSS, 374 goroutines, 238 MiB/33 MiB/325 MiB GO alloc/idle/total, 346 MiB/412 MiB CGO alloc/total, 467.40cgo/sec, 0.32/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:50:33.977396 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:34.962825 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:34.962853 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:34.977653 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:50:35.513473 435 storage/replica_proposal.go:201 [n17,s17,r9777/9:/System/tsd/cr.node.sql.mem.c…] new range lease repl=(n17,s17):9 seq=5 start=1533160232.638791410,0 epo=43 pro=1533160232.638823557,0 following repl=(n15,s15):2 seq=4 start=1533155833.422429483,0 epo=236 pro=1533155833.422432210,0 W180801 21:50:35.977840 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:36.977989 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:37.978216 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:38.978365 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:39.463517 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:39.463551 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:39.978520 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:40.978711 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:41.978874 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:42.979011 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:50:43.945105 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (3m3s: infos 21/21910 sent/received, bytes 11115B/3541515B sent/received) gossip server (0/3 cur/max conns, infos 21/21910 sent/received, bytes 11115B/3541515B sent/received) I180801 21:50:43.945882 500 server/status/runtime.go:219 [n17] runtime stats: 787 MiB RSS, 374 goroutines, 237 MiB/34 MiB/325 MiB GO alloc/idle/total, 358 MiB/425 MiB CGO alloc/total, 467.00cgo/sec, 0.36/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:50:43.963791 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:43.963823 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:43.979148 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:44.979286 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:45.979511 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:46.979653 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:47.979844 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:48.464063 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:48.464089 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:48.979986 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:49.980152 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:50.980331 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:51.980576 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:52.964272 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:52.964296 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:52.980723 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:50:53.945037 500 server/status/runtime.go:219 [n17] runtime stats: 802 MiB RSS, 374 goroutines, 242 MiB/29 MiB/325 MiB GO alloc/idle/total, 362 MiB/430 MiB CGO alloc/total, 548.44cgo/sec, 0.36/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:50:53.980898 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:54.981063 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:55.981273 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:56.981462 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:57.464452 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:50:57.464466 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:50:57.981691 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:58.981844 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:50:59.982030 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:00.982217 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:01.964639 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:01.964665 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:01.982385 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:02.982554 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:51:03.945030 500 server/status/runtime.go:219 [n17] runtime stats: 820 MiB RSS, 374 goroutines, 239 MiB/33 MiB/327 MiB GO alloc/idle/total, 374 MiB/442 MiB CGO alloc/total, 475.00cgo/sec, 0.34/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:51:03.982699 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:04.982997 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:05.983211 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:06.464836 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:06.464859 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:06.983360 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:07.983548 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:08.983648 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:09.983876 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:10.965071 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:10.965104 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:10.984121 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:11.984335 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:12.984492 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:51:13.945355 500 server/status/runtime.go:219 [n17] runtime stats: 828 MiB RSS, 374 goroutines, 236 MiB/35 MiB/327 MiB GO alloc/idle/total, 378 MiB/447 MiB CGO alloc/total, 451.29cgo/sec, 0.36/0.10 %(u/s)time, 0.00 %gc (1x) W180801 21:51:13.984697 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:14.984928 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:15.465349 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:15.465385 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:15.985112 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:16.985322 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:17.985539 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:18.985652 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:19.965618 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:19.965653 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:19.985909 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:20.986100 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:21.986303 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:22.986457 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:51:23.945989 500 server/status/runtime.go:219 [n17] runtime stats: 836 MiB RSS, 374 goroutines, 238 MiB/34 MiB/327 MiB GO alloc/idle/total, 389 MiB/459 MiB CGO alloc/total, 497.88cgo/sec, 0.34/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:51:23.986575 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:24.465879 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:24.465902 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:24.986737 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:25.986966 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:26.987150 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:27.987374 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:28.966123 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:28.966147 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:28.987623 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:29.987819 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:30.988047 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:31.988357 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:32.639532 3000 storage/node_liveness.go:501 [n17,s17,r9777/9:/System/tsd/cr.node.sql.mem.c…] slow heartbeat took 56.9s E180801 21:51:32.639559 3000 storage/replica_range_lease.go:282 [n17,s17,r9777/9:/System/tsd/cr.node.sql.mem.c…] context canceled W180801 21:51:32.988501 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:33.466765 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:33.466786 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 21:51:33.945968 500 server/status/runtime.go:219 [n17] runtime stats: 848 MiB RSS, 374 goroutines, 240 MiB/31 MiB/327 MiB GO alloc/idle/total, 400 MiB/470 MiB CGO alloc/total, 465.80cgo/sec, 0.35/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:51:33.988622 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:34.988830 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:51:35.633033 423 storage/replica_proposal.go:201 [n17,s17,r410/27:/System/tsd/cr.node.sql.mem.a…] new range lease repl=(n17,s17):27 seq=5 start=1533160292.639183562,0 epo=44 pro=1533160292.639228223,0 following repl=(n24,s24):36 seq=4 start=1533155235.550035908,0 epo=28 pro=1533155235.550040878,0 W180801 21:51:35.989017 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:36.989190 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:37.967397 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:37.967417 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:37.989408 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:38.989559 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:39.989758 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:40.989925 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:41.990341 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:42.467641 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:42.467665 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:42.991155 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:51:43.944654 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (4m3s: infos 27/29959 sent/received, bytes 15171B/4758667B sent/received) gossip server (0/3 cur/max conns, infos 27/29959 sent/received, bytes 15171B/4758667B sent/received) I180801 21:51:43.945551 500 server/status/runtime.go:219 [n17] runtime stats: 795 MiB RSS, 374 goroutines, 230 MiB/40 MiB/327 MiB GO alloc/idle/total, 348 MiB/419 MiB CGO alloc/total, 450.91cgo/sec, 0.36/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:51:43.991273 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:44.991584 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:45.991769 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:46.967794 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:46.967802 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:46.991895 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:47.992058 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:48.992207 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:49.992377 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:50.992547 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:51.468013 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:51.468045 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:51.992794 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:52.992980 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:51:53.945915 500 server/status/runtime.go:219 [n17] runtime stats: 805 MiB RSS, 374 goroutines, 241 MiB/31 MiB/327 MiB GO alloc/idle/total, 352 MiB/423 MiB CGO alloc/total, 461.49cgo/sec, 0.33/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:51:53.993155 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:54.993420 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:55.968308 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:51:55.968336 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:51:55.993586 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:56.993720 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:57.993886 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:58.994057 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:51:59.994329 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:00.468485 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:00.468493 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:00.994531 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:01.994752 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:02.994936 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:52:03.946380 500 server/status/runtime.go:219 [n17] runtime stats: 851 MiB RSS, 387 goroutines, 169 MiB/133 MiB/368 MiB GO alloc/idle/total, 364 MiB/437 MiB CGO alloc/total, 486.98cgo/sec, 0.38/0.09 %(u/s)time, 0.00 %gc (2x) W180801 21:52:03.995085 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:04.968718 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:04.968744 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:04.995529 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:05.995688 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:06.995952 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:07.996111 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:08.996264 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:09.468985 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:09.469018 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:09.996465 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:10.996862 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:11.997112 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:12.997223 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:52:13.945943 500 server/status/runtime.go:219 [n17] runtime stats: 859 MiB RSS, 386 goroutines, 175 MiB/128 MiB/368 MiB GO alloc/idle/total, 376 MiB/449 MiB CGO alloc/total, 468.82cgo/sec, 0.34/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:52:13.969159 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:13.969171 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:13.997368 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:14.997555 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:15.997730 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:16.997999 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:17.998169 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:18.469342 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:18.469367 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:18.998310 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:19.998540 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:20.998819 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:21.999057 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:22.969552 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:22.969588 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:22.999175 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:52:23.945094 500 server/status/runtime.go:219 [n17] runtime stats: 869 MiB RSS, 386 goroutines, 181 MiB/123 MiB/368 MiB GO alloc/idle/total, 379 MiB/453 MiB CGO alloc/total, 453.23cgo/sec, 0.36/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:52:23.999395 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:24.999664 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure E180801 21:52:25.631876 4189 server/admin.go:916 query execution canceled: context canceled E180801 21:52:25.631902 4189 server/admin.go:997 rpc error: code = Internal desc = An internal server error has occurred. Please check your CockroachDB logs for more details. E180801 21:52:25.718084 4129 server/status.go:836 [n17,status] context canceled W180801 21:52:25.999914 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:27.000105 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:27.469816 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:27.469849 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:28.000274 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:29.000418 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:30.000626 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:31.000962 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:31.970144 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:31.970170 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:32.001236 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:32.639934 3791 storage/node_liveness.go:501 [n17,s17,r410/27:/System/tsd/cr.node.sql.mem.a…] slow heartbeat took 56.8s E180801 21:52:32.639962 3791 storage/replica_range_lease.go:282 [n17,s17,r410/27:/System/tsd/cr.node.sql.mem.a…] context canceled W180801 21:52:33.001403 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:52:33.945928 500 server/status/runtime.go:219 [n17] runtime stats: 883 MiB RSS, 382 goroutines, 238 MiB/73 MiB/368 MiB GO alloc/idle/total, 390 MiB/465 MiB CGO alloc/total, 515.37cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:52:34.001560 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:35.001748 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:36.001968 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:36.470755 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:36.470775 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:37.002197 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:38.002305 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:39.002393 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:40.002552 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:40.971403 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:40.971423 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:41.002837 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:42.003082 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure E180801 21:52:42.378674 4597 server/admin.go:916 query execution canceled: context canceled E180801 21:52:42.378710 4597 server/admin.go:997 rpc error: code = Internal desc = An internal server error has occurred. Please check your CockroachDB logs for more details. E180801 21:52:42.378872 4523 server/status.go:836 [n17,status] context canceled W180801 21:52:43.003233 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:52:43.945058 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (5m3s: infos 29/38334 sent/received, bytes 16523B/6010123B sent/received) gossip server (0/3 cur/max conns, infos 29/38334 sent/received, bytes 16523B/6010123B sent/received) I180801 21:52:43.945828 500 server/status/runtime.go:219 [n17] runtime stats: 891 MiB RSS, 372 goroutines, 241 MiB/71 MiB/368 MiB GO alloc/idle/total, 401 MiB/477 MiB CGO alloc/total, 468.40cgo/sec, 0.34/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:52:44.003497 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:45.003750 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:45.472085 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:45.472106 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:46.003915 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:47.004012 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:48.004198 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:49.004383 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:49.972781 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:49.972811 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:50.004534 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:51.004785 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:52.005033 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:53.005193 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:52:53.946140 500 server/status/runtime.go:219 [n17] runtime stats: 899 MiB RSS, 372 goroutines, 240 MiB/71 MiB/368 MiB GO alloc/idle/total, 404 MiB/480 MiB CGO alloc/total, 459.79cgo/sec, 0.35/0.07 %(u/s)time, 0.00 %gc (1x) W180801 21:52:54.005410 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:54.473342 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:54.473363 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:55.005566 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:56.005727 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:57.005892 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:58.006047 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:52:58.973992 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:52:58.974014 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:52:59.006208 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:00.006395 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:01.006628 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:02.006832 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:03.007009 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:03.474404 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:03.474420 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 21:53:03.946022 500 server/status/runtime.go:219 [n17] runtime stats: 909 MiB RSS, 373 goroutines, 240 MiB/71 MiB/368 MiB GO alloc/idle/total, 416 MiB/492 MiB CGO alloc/total, 463.80cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:53:04.007227 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:05.007387 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:06.007636 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:07.007804 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:07.975012 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:07.975033 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:08.007986 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:09.008093 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:10.008313 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:11.008551 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:12.008772 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:12.475670 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:12.475696 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:13.008919 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:53:13.945111 500 server/status/runtime.go:219 [n17] runtime stats: 913 MiB RSS, 372 goroutines, 260 MiB/54 MiB/368 MiB GO alloc/idle/total, 419 MiB/495 MiB CGO alloc/total, 454.23cgo/sec, 0.36/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:53:14.009031 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:15.009259 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:16.009415 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:16.976189 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:16.976209 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:17.009674 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:18.009867 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:19.010063 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:20.010282 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:21.010475 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:21.476978 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:21.476998 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:22.010826 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:23.010996 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:53:23.946104 500 server/status/runtime.go:219 [n17] runtime stats: 921 MiB RSS, 372 goroutines, 146 MiB/148 MiB/368 MiB GO alloc/idle/total, 430 MiB/506 MiB CGO alloc/total, 479.66cgo/sec, 0.40/0.09 %(u/s)time, 0.00 %gc (2x) W180801 21:53:24.011245 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:25.011480 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:25.977643 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:25.977673 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:26.011711 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:27.011891 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:28.012010 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:29.012127 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:30.012219 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:30.477943 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:30.477950 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:31.012350 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:32.012632 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:33.012817 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:53:33.945589 500 server/status/runtime.go:219 [n17] runtime stats: 929 MiB RSS, 372 goroutines, 148 MiB/146 MiB/368 MiB GO alloc/idle/total, 440 MiB/517 MiB CGO alloc/total, 479.62cgo/sec, 0.27/0.07 %(u/s)time, 0.00 %gc (1x) W180801 21:53:34.012993 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:34.978647 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:34.978669 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:35.013244 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:36.013515 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:37.013689 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:38.013822 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:39.014001 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:39.479229 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:39.479249 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:40.014219 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:41.014479 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:42.014668 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:43.014845 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:53:43.944912 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (6m3s: infos 30/48133 sent/received, bytes 17199B/7594332B sent/received) gossip server (0/3 cur/max conns, infos 30/48133 sent/received, bytes 17199B/7594332B sent/received) I180801 21:53:43.945853 500 server/status/runtime.go:219 [n17] runtime stats: 942 MiB RSS, 372 goroutines, 147 MiB/148 MiB/368 MiB GO alloc/idle/total, 450 MiB/528 MiB CGO alloc/total, 460.39cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:53:43.979520 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:43.979528 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:44.014941 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:45.015023 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:46.015140 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:47.015289 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:48.015407 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:48.480036 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:48.480057 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:49.015580 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:50.015677 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:51.015804 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:52.015936 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:52.980594 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:52.980617 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:53.016110 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:53:53.945132 500 server/status/runtime.go:219 [n17] runtime stats: 898 MiB RSS, 372 goroutines, 253 MiB/60 MiB/368 MiB GO alloc/idle/total, 394 MiB/482 MiB CGO alloc/total, 331.52cgo/sec, 0.67/0.04 %(u/s)time, 0.00 %gc (0x) W180801 21:53:54.016263 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:55.016452 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:56.016664 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:57.016817 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:57.481211 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:53:57.481232 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:53:58.016996 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:53:59.017180 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:00.017548 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:01.017763 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:01.981913 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:01.981935 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:02.018030 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:03.018182 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:54:03.946067 500 server/status/runtime.go:219 [n17] runtime stats: 913 MiB RSS, 372 goroutines, 274 MiB/40 MiB/368 MiB GO alloc/idle/total, 405 MiB/493 MiB CGO alloc/total, 498.37cgo/sec, 0.43/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:54:04.018281 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:05.018482 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:06.018769 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:06.482426 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:06.482446 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:07.018983 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:08.019084 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:09.019276 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:10.019630 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:10.983068 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:10.983088 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:11.019792 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:12.020034 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:13.020194 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:54:13.946260 500 server/status/runtime.go:219 [n17] runtime stats: 922 MiB RSS, 372 goroutines, 263 MiB/51 MiB/369 MiB GO alloc/idle/total, 416 MiB/505 MiB CGO alloc/total, 461.59cgo/sec, 0.37/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:54:14.020378 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:15.020563 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:15.483729 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:15.483750 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:16.020771 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:17.021077 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:18.021234 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:19.021368 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:19.984321 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:19.984344 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:20.021528 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:21.021754 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:22.021923 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:23.022080 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:54:23.946130 500 server/status/runtime.go:219 [n17] runtime stats: 928 MiB RSS, 372 goroutines, 261 MiB/52 MiB/369 MiB GO alloc/idle/total, 419 MiB/509 MiB CGO alloc/total, 456.80cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:54:24.022289 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:24.485011 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:24.485033 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:25.022580 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:26.022788 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:27.022945 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:28.023053 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:28.985583 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:28.985604 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:29.023166 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:30.023383 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:31.023547 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:32.023728 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:33.023875 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:33.486141 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:33.486161 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 21:54:33.946441 500 server/status/runtime.go:219 [n17] runtime stats: 934 MiB RSS, 372 goroutines, 258 MiB/55 MiB/369 MiB GO alloc/idle/total, 429 MiB/520 MiB CGO alloc/total, 449.89cgo/sec, 0.32/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:54:34.024130 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:35.024356 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:36.024553 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:37.024678 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:37.986765 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:37.986787 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:38.024838 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:39.025044 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:40.025392 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:41.025536 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:42.025771 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:42.487332 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:42.487354 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:43.025881 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:54:43.944986 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (7m3s: infos 33/55789 sent/received, bytes 19227B/8669086B sent/received) gossip server (0/3 cur/max conns, infos 33/55789 sent/received, bytes 19227B/8669086B sent/received) I180801 21:54:43.946104 500 server/status/runtime.go:219 [n17] runtime stats: 940 MiB RSS, 372 goroutines, 258 MiB/56 MiB/369 MiB GO alloc/idle/total, 432 MiB/523 MiB CGO alloc/total, 481.51cgo/sec, 0.33/0.10 %(u/s)time, 0.00 %gc (1x) W180801 21:54:44.026031 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:45.026302 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:46.026430 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:46.987641 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:46.987648 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:47.026526 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:48.026693 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:49.026949 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:50.027224 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:51.027365 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:51.488383 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:51.488404 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:52.027597 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:53.027790 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:54:53.945962 500 server/status/runtime.go:219 [n17] runtime stats: 948 MiB RSS, 372 goroutines, 261 MiB/53 MiB/369 MiB GO alloc/idle/total, 443 MiB/534 MiB CGO alloc/total, 490.91cgo/sec, 0.37/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:54:54.028057 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:55.028293 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:55.989069 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:54:55.989092 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:54:56.028520 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:57.028793 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:58.028961 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:54:59.029171 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:00.029408 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:00.489412 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:00.489422 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:01.029569 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:02.029840 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:03.030000 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:55:03.946016 500 server/status/runtime.go:219 [n17] runtime stats: 954 MiB RSS, 372 goroutines, 260 MiB/53 MiB/369 MiB GO alloc/idle/total, 454 MiB/545 MiB CGO alloc/total, 438.70cgo/sec, 0.34/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:55:04.030162 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:04.990170 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:04.990201 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:05.030387 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:06.030544 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:07.030686 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:08.030915 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:09.031085 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:09.491001 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:09.491033 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:10.031302 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:11.031529 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:12.031736 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:13.031910 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:55:13.945983 500 server/status/runtime.go:219 [n17] runtime stats: 964 MiB RSS, 372 goroutines, 264 MiB/51 MiB/369 MiB GO alloc/idle/total, 456 MiB/548 MiB CGO alloc/total, 452.20cgo/sec, 0.37/0.10 %(u/s)time, 0.00 %gc (1x) W180801 21:55:13.991396 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:13.991406 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:14.032080 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:15.032352 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:16.032522 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:17.032668 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:18.032824 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:18.492026 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:18.492047 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:19.033009 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:20.033225 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:21.033622 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:22.033826 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:22.992624 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:22.992649 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:23.033975 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:55:23.946057 500 server/status/runtime.go:219 [n17] runtime stats: 972 MiB RSS, 372 goroutines, 268 MiB/47 MiB/369 MiB GO alloc/idle/total, 466 MiB/558 MiB CGO alloc/total, 471.29cgo/sec, 0.38/0.07 %(u/s)time, 0.00 %gc (1x) W180801 21:55:24.034191 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:25.034570 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:26.034761 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:27.034962 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:27.492922 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:27.492930 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:28.035125 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:29.035283 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:30.035591 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:31.035795 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:55:31.832748 456 storage/replica_proposal.go:201 [n17,s17,r2191/21:/System/tsd/cr.store.lastupda…] new range lease repl=(n17,s17):21 seq=4 start=1533160531.158338950,0 epo=45 pro=1533160531.158371957,0 following repl=(n27,s27):22 seq=3 start=1533150771.469972480,0 epo=32 pro=1533150771.469977776,0 W180801 21:55:31.993554 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:31.993577 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:32.035953 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:33.036108 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:55:33.978044 500 server/status/runtime.go:219 [n17] runtime stats: 978 MiB RSS, 374 goroutines, 272 MiB/45 MiB/369 MiB GO alloc/idle/total, 476 MiB/569 MiB CGO alloc/total, 517.34cgo/sec, 0.38/0.09 %(u/s)time, 0.00 %gc (2x) W180801 21:55:34.036319 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:35.036630 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:36.036792 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:36.493752 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:36.493784 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:37.036964 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:38.037102 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:39.037301 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:40.037618 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:40.994011 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:40.994034 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:41.037812 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:42.038003 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:43.038211 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:55:43.944609 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (8m3s: infos 35/64623 sent/received, bytes 20579B/10024753B sent/received) gossip server (0/3 cur/max conns, infos 35/64623 sent/received, bytes 20579B/10024753B sent/received) I180801 21:55:43.945071 500 server/status/runtime.go:219 [n17] runtime stats: 989 MiB RSS, 374 goroutines, 272 MiB/43 MiB/369 MiB GO alloc/idle/total, 488 MiB/581 MiB CGO alloc/total, 487.10cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (0x) W180801 21:55:44.038400 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:45.038657 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:45.494296 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:45.494330 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:46.038883 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:47.039042 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:48.039169 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:49.039340 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:49.994551 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:49.994581 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:50.039552 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:51.039729 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:52.039924 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:53.040077 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:55:53.945323 500 server/status/runtime.go:219 [n17] runtime stats: 936 MiB RSS, 374 goroutines, 257 MiB/58 MiB/369 MiB GO alloc/idle/total, 426 MiB/517 MiB CGO alloc/total, 492.49cgo/sec, 0.37/0.08 %(u/s)time, 0.00 %gc (2x) W180801 21:55:54.040280 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:54.494816 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:54.494850 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:55.040494 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:56.040675 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:57.040826 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:58.040949 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:55:58.995014 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:55:58.995037 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:55:59.041102 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:00.041366 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:01.041591 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:02.041840 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:03.042035 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:03.495290 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:03.495322 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 21:56:03.959390 500 server/status/runtime.go:219 [n17] runtime stats: 944 MiB RSS, 374 goroutines, 276 MiB/41 MiB/369 MiB GO alloc/idle/total, 437 MiB/527 MiB CGO alloc/total, 462.05cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:56:04.042255 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:05.042438 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:06.042649 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:07.042863 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:07.995577 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:07.995612 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:08.042980 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:09.043124 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:10.043373 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:11.043627 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:12.043852 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:12.495721 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:12.495730 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:13.044006 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:56:13.945063 500 server/status/runtime.go:219 [n17] runtime stats: 952 MiB RSS, 374 goroutines, 272 MiB/43 MiB/369 MiB GO alloc/idle/total, 439 MiB/531 MiB CGO alloc/total, 450.94cgo/sec, 0.31/0.08 %(u/s)time, 0.00 %gc (0x) W180801 21:56:14.044202 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:15.044436 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:16.044679 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:16.995897 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:16.995921 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:17.044863 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:18.044986 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:19.045198 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:20.045391 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:21.045571 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:21.496105 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:21.496130 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:22.045740 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:23.045901 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:56:23.956133 500 server/status/runtime.go:219 [n17] runtime stats: 962 MiB RSS, 374 goroutines, 265 MiB/49 MiB/369 MiB GO alloc/idle/total, 450 MiB/542 MiB CGO alloc/total, 473.59cgo/sec, 0.34/0.10 %(u/s)time, 0.00 %gc (1x) W180801 21:56:24.046193 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:25.046347 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:25.996371 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:25.996398 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:26.046456 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:27.046699 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:28.046877 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:29.047042 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:30.047214 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:30.496611 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:30.496648 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:31.047507 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:31.158977 7591 storage/node_liveness.go:501 [n17,s17,r2191/21:/System/tsd/cr.store.lastupda…] slow heartbeat took 59.3s E180801 21:56:31.159008 7591 storage/replica_range_lease.go:282 [n17,s17,r2191/21:/System/tsd/cr.store.lastupda…] context canceled W180801 21:56:32.053626 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:33.053790 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:56:33.945953 500 server/status/runtime.go:219 [n17] runtime stats: 966 MiB RSS, 372 goroutines, 268 MiB/46 MiB/369 MiB GO alloc/idle/total, 460 MiB/552 MiB CGO alloc/total, 464.47cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:56:34.054053 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:34.997261 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:34.997282 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:35.054297 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:36.054489 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:37.054906 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:38.055128 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:39.055281 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:39.497855 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:39.497877 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:40.055516 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:41.055666 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:42.055862 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:43.056019 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:56:43.945007 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (9m3s: infos 38/70003 sent/received, bytes 22607B/10831006B sent/received) gossip server (0/3 cur/max conns, infos 38/70003 sent/received, bytes 22607B/10831006B sent/received) I180801 21:56:43.946346 500 server/status/runtime.go:219 [n17] runtime stats: 976 MiB RSS, 372 goroutines, 265 MiB/49 MiB/369 MiB GO alloc/idle/total, 462 MiB/555 MiB CGO alloc/total, 483.39cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:56:43.998136 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:43.998144 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:44.058484 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:45.058734 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:46.058888 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:47.059015 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:48.059198 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:48.498946 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:48.498977 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:49.059542 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:50.059727 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:51.059894 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:52.060067 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:52.999585 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:52.999607 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:53.060211 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:56:53.946087 500 server/status/runtime.go:219 [n17] runtime stats: 984 MiB RSS, 372 goroutines, 267 MiB/48 MiB/369 MiB GO alloc/idle/total, 472 MiB/565 MiB CGO alloc/total, 477.01cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:56:54.060371 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:55.060589 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:56.060759 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:57.060926 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:57.500229 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:56:57.500250 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:56:58.061045 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:56:59.061268 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:00.061460 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:01.061628 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:02.000778 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:02.000801 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:02.061812 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:03.061985 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:57:03.946125 500 server/status/runtime.go:219 [n17] runtime stats: 990 MiB RSS, 372 goroutines, 257 MiB/57 MiB/369 MiB GO alloc/idle/total, 482 MiB/576 MiB CGO alloc/total, 440.40cgo/sec, 0.32/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:57:04.062190 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:05.062508 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:06.062702 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:06.501443 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:06.501465 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:07.062925 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:08.063066 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:09.063217 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:10.063425 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:11.002060 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:11.002082 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:11.063737 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:12.063951 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:13.064034 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:57:13.945055 500 server/status/runtime.go:219 [n17] runtime stats: 997 MiB RSS, 372 goroutines, 252 MiB/61 MiB/369 MiB GO alloc/idle/total, 484 MiB/578 MiB CGO alloc/total, 474.14cgo/sec, 0.34/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:57:14.064183 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:15.064410 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:15.502656 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:15.502678 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:16.064590 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:17.064733 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:18.064887 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:19.065064 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:20.003174 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:20.003194 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:20.065291 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:21.065558 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:22.065726 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:23.065901 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:57:23.945309 500 server/status/runtime.go:219 [n17] runtime stats: 1001 MiB RSS, 372 goroutines, 254 MiB/58 MiB/369 MiB GO alloc/idle/total, 493 MiB/588 MiB CGO alloc/total, 461.89cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:57:24.066109 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:24.503990 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:24.504024 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:25.066328 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:26.066480 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:27.066664 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:28.066836 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:29.004606 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:29.004627 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:29.067017 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:30.067231 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:31.067410 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:32.067647 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:33.067832 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:33.505125 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:33.505145 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 21:57:33.945863 500 server/status/runtime.go:219 [n17] runtime stats: 1009 MiB RSS, 372 goroutines, 250 MiB/62 MiB/369 MiB GO alloc/idle/total, 502 MiB/597 MiB CGO alloc/total, 433.38cgo/sec, 0.32/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:57:34.068005 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:35.068230 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:36.068391 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:37.068605 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:38.005655 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:38.005676 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:38.068761 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:39.068970 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:40.069136 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:41.069358 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:42.069640 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:42.506289 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:42.506313 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:43.069871 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:57:43.945100 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (10m3s: infos 39/75311 sent/received, bytes 23283B/11689511B sent/received) gossip server (0/3 cur/max conns, infos 39/75311 sent/received, bytes 23283B/11689511B sent/received) I180801 21:57:43.946456 500 server/status/runtime.go:219 [n17] runtime stats: 1015 MiB RSS, 372 goroutines, 259 MiB/54 MiB/369 MiB GO alloc/idle/total, 503 MiB/598 MiB CGO alloc/total, 505.68cgo/sec, 0.37/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:57:44.070079 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:45.070290 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:46.070467 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:47.006600 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:47.006607 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:47.070621 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:48.070842 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:49.071054 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:50.071259 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:51.071585 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:51.507295 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:51.507325 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:52.071820 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:53.071999 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:57:53.946038 500 server/status/runtime.go:219 [n17] runtime stats: 966 MiB RSS, 372 goroutines, 259 MiB/54 MiB/369 MiB GO alloc/idle/total, 450 MiB/554 MiB CGO alloc/total, 432.91cgo/sec, 0.45/0.07 %(u/s)time, 0.00 %gc (1x) I180801 21:57:53.995161 480 storage/store.go:4516 [n17,s17] sstables (read amplification = 3): 4 [ 11M 4 ]: 5M[2] 710K 16K 5 [ 270M 55 ]: 12M 10M[2] 9M[8] 8M[4] 7M[3] 6M[4] 5M[7] 4M[2] 3M[10] 2M[6] 1M[2] 664K 547K 457K 433K 371K 287K 6 [ 6G 130 ]: 100M[36] 99M[5] 98M[3] 97M[2] 96M[2] 95M[3] 94M 93M 92M[2] 91M[2] 88M 84M 59M 18M 17M 15M 12M 11M[3] 9M[2] 7M 6M 5M[2] 4M[7] 3M[4] 2M[3] 1M[7] 968K 893K 797K 717K 683K 589K 555K 519K[2] 482K 474K 424K 423K 419K 398K 390K[2] 324K 313K 312K 288K 263K 164K 156K 149K 146K 137K 134K 125K 110K 98K 91K 80K 53K 26K 22K I180801 21:57:53.995255 480 storage/store.go:4517 [n17,s17] ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.4 4 6 0.677 0 0 L4 4/0 10.87 MB 0.7 0.0 0.0 0.0 0.0 0.0 0.0 1.3 8.6 6.2 5 3 1.760 1032K 186K L5 55/0 270.02 MB 0.6 0.1 0.0 0.1 0.1 0.0 0.0 15.4 10.4 10.4 10 1 10.101 517K 0 L6 130/0 5.91 GB 0.0 0.2 0.0 0.2 0.2 0.0 0.0 10.4 15.3 14.0 14 2 7.133 639K 661K Sum 189/0 6.19 GB 0.0 0.4 0.1 0.3 0.4 0.0 0.0 14.0 10.9 10.8 34 12 2.809 2190K 847K Int 0/0 0.00 KB 0.0 0.3 0.0 0.2 0.3 0.0 0.0 11.6 9.9 9.9 26 10 2.604 2190K 827K Uptime(secs): 613.9 total, 600.0 interval Flush(GB): cumulative 0.025, interval 0.022 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.35 GB write, 0.59 MB/s write, 0.36 GB read, 0.60 MB/s read, 33.7 seconds Interval compaction: 0.25 GB write, 0.43 MB/s write, 0.25 GB read, 0.43 MB/s read, 26.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count estimated_pending_compaction_bytes: 0 B W180801 21:57:54.072145 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:55.072381 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:56.008049 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:57:56.008070 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:57:56.072569 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:57.072788 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:58.073035 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:57:59.073287 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:00.073491 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:00.508440 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:00.508459 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:01.073640 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:02.073741 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:03.073980 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:58:03.946343 500 server/status/runtime.go:219 [n17] runtime stats: 978 MiB RSS, 372 goroutines, 259 MiB/55 MiB/369 MiB GO alloc/idle/total, 460 MiB/564 MiB CGO alloc/total, 468.19cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:58:04.074187 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:05.008747 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:05.008754 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:05.074445 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:06.074652 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:07.074864 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:08.074985 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:09.075194 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:09.509095 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:09.509105 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:10.075380 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:11.075563 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:12.075739 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:13.075884 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:58:13.945218 500 server/status/runtime.go:219 [n17] runtime stats: 984 MiB RSS, 372 goroutines, 260 MiB/53 MiB/369 MiB GO alloc/idle/total, 462 MiB/566 MiB CGO alloc/total, 448.54cgo/sec, 0.34/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:58:14.009606 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:14.009626 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:14.076036 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:15.076200 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:16.076390 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:17.076611 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:18.076729 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:18.510191 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:18.510212 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:19.076964 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:20.077072 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:21.077264 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:22.077421 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:23.010771 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:23.010800 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:23.077579 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:58:23.946013 500 server/status/runtime.go:219 [n17] runtime stats: 992 MiB RSS, 372 goroutines, 263 MiB/51 MiB/369 MiB GO alloc/idle/total, 471 MiB/576 MiB CGO alloc/total, 448.87cgo/sec, 0.37/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:58:24.077777 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:25.077941 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:26.078081 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:27.078270 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:27.511420 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:27.511490 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:28.078419 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:29.078577 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:30.078682 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:31.078840 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:32.011715 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:32.011722 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:32.078988 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:33.079161 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:58:33.945962 500 server/status/runtime.go:219 [n17] runtime stats: 998 MiB RSS, 372 goroutines, 266 MiB/49 MiB/369 MiB GO alloc/idle/total, 481 MiB/586 MiB CGO alloc/total, 448.10cgo/sec, 0.34/0.07 %(u/s)time, 0.00 %gc (1x) W180801 21:58:34.079332 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:35.079565 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:36.079665 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:36.512336 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:36.512358 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:37.079876 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:38.080037 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:39.080262 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:40.080445 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:41.013032 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:41.013053 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:41.080658 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:42.080846 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:43.081001 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:58:43.944984 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (11m3s: infos 41/80310 sent/received, bytes 24635B/12479234B sent/received) gossip server (0/3 cur/max conns, infos 41/80310 sent/received, bytes 24635B/12479234B sent/received) I180801 21:58:43.946104 500 server/status/runtime.go:219 [n17] runtime stats: 1005 MiB RSS, 372 goroutines, 264 MiB/50 MiB/369 MiB GO alloc/idle/total, 482 MiB/588 MiB CGO alloc/total, 475.69cgo/sec, 0.34/0.10 %(u/s)time, 0.00 %gc (1x) W180801 21:58:44.081143 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:45.081358 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:45.513685 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:45.513706 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:46.081570 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:47.081680 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:48.081840 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:49.082085 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:50.014275 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:50.014297 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:50.082245 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:51.082461 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:52.082679 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:53.082864 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:58:53.945999 500 server/status/runtime.go:219 [n17] runtime stats: 1011 MiB RSS, 372 goroutines, 263 MiB/51 MiB/369 MiB GO alloc/idle/total, 491 MiB/597 MiB CGO alloc/total, 437.50cgo/sec, 0.34/0.07 %(u/s)time, 0.00 %gc (1x) W180801 21:58:54.083086 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:54.514842 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:54.514865 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:55.083274 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:56.083601 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:57.083817 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:58.083944 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:58:59.015226 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:58:59.015245 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:58:59.084128 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:00.084292 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:01.084512 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:02.084687 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:03.084871 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:03.515864 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:03.515885 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 21:59:03.946157 500 server/status/runtime.go:219 [n17] runtime stats: 1019 MiB RSS, 372 goroutines, 265 MiB/50 MiB/369 MiB GO alloc/idle/total, 500 MiB/604 MiB CGO alloc/total, 463.90cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:59:04.085128 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:05.085423 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:06.085713 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:07.085979 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:08.016435 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:08.016455 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:08.086225 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:09.086489 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:10.086687 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:11.086791 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:12.086903 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:12.516678 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:12.516684 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:13.087024 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:59:13.945103 500 server/status/runtime.go:219 [n17] runtime stats: 1023 MiB RSS, 372 goroutines, 265 MiB/50 MiB/369 MiB GO alloc/idle/total, 501 MiB/606 MiB CGO alloc/total, 479.74cgo/sec, 0.29/0.07 %(u/s)time, 0.00 %gc (1x) W180801 21:59:14.087129 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:15.087391 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:16.087645 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:17.017273 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:17.017295 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:17.087807 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:18.087955 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:19.088136 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:20.088384 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:21.088586 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:21.517914 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:21.517935 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:22.088759 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:23.088916 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:59:23.945970 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 265 MiB/49 MiB/369 MiB GO alloc/idle/total, 510 MiB/607 MiB CGO alloc/total, 489.57cgo/sec, 0.36/0.10 %(u/s)time, 0.00 %gc (1x) W180801 21:59:24.089098 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:25.089323 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:26.018507 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:26.018528 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:26.089528 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:27.089732 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:28.089925 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:29.090106 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:30.090323 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:30.519130 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:30.519152 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:31.090552 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:32.090740 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:33.090920 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:59:33.946079 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 264 MiB/50 MiB/369 MiB GO alloc/idle/total, 519 MiB/616 MiB CGO alloc/total, 424.89cgo/sec, 0.33/0.08 %(u/s)time, 0.00 %gc (1x) W180801 21:59:34.091093 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:35.019790 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:35.019811 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:35.091238 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:36.091388 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:37.091632 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:38.091793 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:39.092012 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:39.520499 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:39.520520 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:40.092243 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:41.092475 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:42.092705 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:43.092881 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:59:43.944599 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (12m3s: infos 43/84694 sent/received, bytes 25987B/13108523B sent/received) gossip server (0/3 cur/max conns, infos 43/84694 sent/received, bytes 25987B/13108523B sent/received) I180801 21:59:43.945178 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 264 MiB/50 MiB/369 MiB GO alloc/idle/total, 519 MiB/616 MiB CGO alloc/total, 466.23cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:59:44.020804 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:44.020812 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:44.093075 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:45.093236 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:46.093455 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:47.093726 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:48.093882 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:48.521272 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:48.521289 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:49.094136 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:50.094392 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:51.094599 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:52.094784 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:53.021584 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:53.021598 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:53.095001 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 21:59:53.946123 500 server/status/runtime.go:219 [n17] runtime stats: 981 MiB RSS, 372 goroutines, 261 MiB/53 MiB/369 MiB GO alloc/idle/total, 465 MiB/569 MiB CGO alloc/total, 436.17cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 21:59:54.095225 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:55.095658 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:56.095970 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:57.096194 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:57.521894 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 21:59:57.521904 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 21:59:58.096407 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 21:59:59.096575 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:00.096830 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:01.097172 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:02.022451 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:02.022473 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:02.097832 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:03.098047 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:00:03.946536 500 server/status/runtime.go:219 [n17] runtime stats: 985 MiB RSS, 372 goroutines, 259 MiB/54 MiB/369 MiB GO alloc/idle/total, 475 MiB/571 MiB CGO alloc/total, 443.58cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:00:04.098199 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:05.098511 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:06.098817 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:06.523169 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:06.523192 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:07.099132 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:08.099388 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:09.099613 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:10.099826 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:11.023879 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:11.023900 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:11.100085 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:12.100260 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:13.100466 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:00:13.946047 500 server/status/runtime.go:219 [n17] runtime stats: 990 MiB RSS, 372 goroutines, 263 MiB/50 MiB/367 MiB GO alloc/idle/total, 476 MiB/573 MiB CGO alloc/total, 489.02cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:00:14.100648 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:15.100961 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:15.524514 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:15.524536 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:16.101130 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:17.101334 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:18.101529 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:19.101794 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:20.025177 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:20.025205 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:20.101993 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:21.102163 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:00:21.217600 406 storage/replica_proposal.go:201 [n17,s17,r10021/6:/System/tsd/cr.store.leases.e…] new range lease repl=(n17,s17):6 seq=4 start=1533160819.678428993,0 epo=46 pro=1533160819.678459419,0 following repl=(n27,s27):4 seq=3 start=1533154957.786123095,0 epo=32 pro=1533154957.786127650,0 W180801 22:00:22.102335 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:23.102570 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:00:23.945963 500 server/status/runtime.go:219 [n17] runtime stats: 998 MiB RSS, 374 goroutines, 266 MiB/47 MiB/367 MiB GO alloc/idle/total, 485 MiB/582 MiB CGO alloc/total, 506.20cgo/sec, 0.38/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:00:24.102773 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:24.525748 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:24.525770 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:25.102991 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:26.103139 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:27.103322 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:28.103571 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:29.025992 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:29.026025 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:29.103770 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:30.103886 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:31.104037 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:32.104189 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:33.104333 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:33.526251 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:33.526278 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 22:00:33.946054 500 server/status/runtime.go:219 [n17] runtime stats: 1002 MiB RSS, 374 goroutines, 268 MiB/46 MiB/367 MiB GO alloc/idle/total, 495 MiB/592 MiB CGO alloc/total, 487.50cgo/sec, 0.35/0.08 %(u/s)time, 0.00 %gc (1x) W180801 22:00:34.104496 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:35.104682 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:36.104784 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:37.104943 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:38.026468 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:38.026504 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:38.105100 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:39.105249 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:40.105423 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:41.105615 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:42.105786 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:42.526693 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:42.526742 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:43.105943 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:00:43.944842 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (13m3s: infos 49/89369 sent/received, bytes 30043B/13796221B sent/received) gossip server (0/3 cur/max conns, infos 49/89369 sent/received, bytes 30043B/13796221B sent/received) I180801 22:00:43.948339 500 server/status/runtime.go:219 [n17] runtime stats: 1010 MiB RSS, 374 goroutines, 270 MiB/44 MiB/367 MiB GO alloc/idle/total, 496 MiB/594 MiB CGO alloc/total, 447.99cgo/sec, 0.37/0.09 %(u/s)time, 0.00 %gc (2x) W180801 22:00:44.106104 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:45.106227 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:46.106392 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:47.026903 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:47.026911 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:47.106518 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:48.106656 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:49.106851 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:50.107020 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:51.107251 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:51.527132 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:51.527158 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:52.107397 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:53.107581 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:00:53.945004 500 server/status/runtime.go:219 [n17] runtime stats: 1016 MiB RSS, 374 goroutines, 142 MiB/149 MiB/367 MiB GO alloc/idle/total, 505 MiB/603 MiB CGO alloc/total, 443.65cgo/sec, 0.32/0.08 %(u/s)time, 0.00 %gc (1x) W180801 22:00:54.107759 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:55.107958 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:56.027359 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:00:56.027391 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:00:56.108126 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:57.108277 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:58.108497 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:00:59.108726 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:00.109014 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:00.527538 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:00.527548 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:01.109287 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:02.109459 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:03.109669 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:01:03.945503 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 374 goroutines, 147 MiB/144 MiB/367 MiB GO alloc/idle/total, 514 MiB/612 MiB CGO alloc/total, 458.98cgo/sec, 0.33/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:01:04.109900 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:05.027804 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:05.027828 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:05.110106 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:06.110336 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:07.110614 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:08.110901 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:09.111082 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:09.528014 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:09.528038 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:10.111318 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:11.111500 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:12.111660 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:13.111790 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:01:13.945039 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 374 goroutines, 146 MiB/146 MiB/367 MiB GO alloc/idle/total, 514 MiB/613 MiB CGO alloc/total, 432.42cgo/sec, 0.35/0.08 %(u/s)time, 0.00 %gc (1x) W180801 22:01:14.028285 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:14.028310 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:14.112048 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:15.112277 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:16.112491 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:17.112695 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:18.112894 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:18.528492 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:18.528520 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:19.113106 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:19.679072 11238 storage/node_liveness.go:501 [n17,s17,r10021/6:/System/tsd/cr.store.leases.e…] slow heartbeat took 58.4s E180801 22:01:19.679098 11238 storage/replica_range_lease.go:282 [n17,s17,r10021/6:/System/tsd/cr.store.leases.e…] context canceled W180801 22:01:20.113323 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:21.113492 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:22.113635 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:23.028818 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:23.028828 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:23.113806 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:01:23.945205 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 146 MiB/145 MiB/367 MiB GO alloc/idle/total, 523 MiB/622 MiB CGO alloc/total, 468.09cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:01:24.114141 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:25.114444 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:26.114655 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:27.114870 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:27.529810 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:27.529832 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:28.115081 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:29.115332 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:30.115575 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:31.115843 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:32.030496 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:32.030519 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:32.116044 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:33.116257 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:01:33.946054 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 152 MiB/141 MiB/367 MiB GO alloc/idle/total, 524 MiB/623 MiB CGO alloc/total, 449.67cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:01:34.116519 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:35.116769 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:36.116885 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:36.531225 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:36.531246 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:37.117082 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:38.117314 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:39.117556 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:40.117739 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:41.031881 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:41.031902 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:41.117980 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:42.118134 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:43.118302 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:01:43.944937 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (14m3s: infos 51/93866 sent/received, bytes 31395B/14467470B sent/received) gossip server (0/3 cur/max conns, infos 51/93866 sent/received, bytes 31395B/14467470B sent/received) I180801 22:01:43.946262 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 152 MiB/141 MiB/367 MiB GO alloc/idle/total, 532 MiB/631 MiB CGO alloc/total, 443.29cgo/sec, 0.36/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:01:44.118519 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:45.118620 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:45.532759 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:45.532780 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:46.118869 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:47.119010 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:48.119151 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:49.119282 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:50.033580 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:50.033602 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:50.119507 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:51.119658 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:52.119742 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:53.119914 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:01:53.946365 500 server/status/runtime.go:219 [n17] runtime stats: 1009 MiB RSS, 372 goroutines, 258 MiB/54 MiB/367 MiB GO alloc/idle/total, 480 MiB/598 MiB CGO alloc/total, 357.50cgo/sec, 0.61/0.06 %(u/s)time, 0.00 %gc (0x) W180801 22:01:54.120145 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:54.533926 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:54.533932 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:55.120319 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:56.120487 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:57.120654 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:58.120802 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:01:59.034590 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:01:59.034612 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:01:59.120935 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:00.121134 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:01.121350 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:02.121555 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:03.121856 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:03.535127 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:03.535148 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 22:02:03.945975 500 server/status/runtime.go:219 [n17] runtime stats: 1017 MiB RSS, 372 goroutines, 268 MiB/45 MiB/367 MiB GO alloc/idle/total, 489 MiB/607 MiB CGO alloc/total, 481.32cgo/sec, 0.46/0.07 %(u/s)time, 0.00 %gc (1x) W180801 22:02:04.122023 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:05.122332 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:06.122592 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:07.122748 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:08.035764 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:08.035785 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:08.122932 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:09.123150 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:10.123345 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:11.123563 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:12.123755 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:12.540759 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:12.540777 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:13.123982 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:02:13.946143 500 server/status/runtime.go:219 [n17] runtime stats: 1023 MiB RSS, 372 goroutines, 264 MiB/49 MiB/368 MiB GO alloc/idle/total, 491 MiB/609 MiB CGO alloc/total, 450.10cgo/sec, 0.34/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:02:14.124125 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:15.124308 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:16.124526 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:17.041438 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:17.041460 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:17.124769 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:18.124995 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:19.125284 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:20.125555 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:21.125709 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:21.541978 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:21.541999 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:22.125877 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:23.126048 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:02:23.946065 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 266 MiB/48 MiB/368 MiB GO alloc/idle/total, 502 MiB/620 MiB CGO alloc/total, 466.80cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:02:24.126198 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:25.126368 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:26.042402 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:26.042414 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:26.126612 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:27.126806 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:28.126955 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:29.127249 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:30.127536 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:30.543012 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:30.543033 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:31.127736 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:32.127850 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:33.128048 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:02:33.945474 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 266 MiB/48 MiB/368 MiB GO alloc/idle/total, 512 MiB/630 MiB CGO alloc/total, 450.52cgo/sec, 0.33/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:02:34.128259 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:35.043785 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:35.043807 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:35.128474 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:36.128679 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:37.128890 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:38.129050 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:39.129218 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:39.544531 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:39.544553 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:40.129371 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:41.129532 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:42.129670 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:43.129814 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:02:43.944960 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (15m3s: infos 52/99288 sent/received, bytes 32071B/15342282B sent/received) gossip server (0/3 cur/max conns, infos 52/99288 sent/received, bytes 32071B/15342282B sent/received) I180801 22:02:43.946044 500 server/status/runtime.go:219 [n17] runtime stats: 1008 MiB RSS, 372 goroutines, 269 MiB/10 MiB/333 MiB GO alloc/idle/total, 515 MiB/634 MiB CGO alloc/total, 490.78cgo/sec, 0.37/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:02:44.045124 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:44.045145 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:44.130050 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:45.130185 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:46.130402 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:47.130627 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:48.130796 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:48.545913 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:48.545937 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:49.130939 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:50.131195 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:51.131400 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:52.131606 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:53.046442 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:53.046462 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:53.131890 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:02:53.946158 500 server/status/runtime.go:219 [n17] runtime stats: 1019 MiB RSS, 372 goroutines, 269 MiB/9.2 MiB/333 MiB GO alloc/idle/total, 524 MiB/644 MiB CGO alloc/total, 497.30cgo/sec, 0.34/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:02:54.132199 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:55.132419 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:56.132617 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:57.132839 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:57.547114 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:02:57.547136 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:02:58.133125 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:02:59.133254 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:00.133530 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:01.133789 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:02.047711 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:02.047732 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:02.134116 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:03.134317 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:03:03.946278 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 268 MiB/10 MiB/333 MiB GO alloc/idle/total, 534 MiB/652 MiB CGO alloc/total, 448.10cgo/sec, 0.35/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:03:04.134627 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:05.134951 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:06.135095 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:06.548368 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:06.548389 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:07.135268 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:08.135529 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:09.135724 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:10.135872 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:11.049037 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:11.049060 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:11.136086 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:12.136362 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:13.136661 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:03:13.945164 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 144 MiB/110 MiB/333 MiB GO alloc/idle/total, 535 MiB/653 MiB CGO alloc/total, 449.34cgo/sec, 0.40/0.08 %(u/s)time, 0.00 %gc (2x) W180801 22:03:14.136822 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:15.136993 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:15.549657 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:15.549680 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:16.137193 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:17.137381 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:18.137543 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:19.137725 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:20.050133 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:20.050139 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:20.137876 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:21.138069 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:22.138302 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:23.138580 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:03:23.946361 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 273 MiB/5.8 MiB/333 MiB GO alloc/idle/total, 544 MiB/654 MiB CGO alloc/total, 450.46cgo/sec, 0.32/0.10 %(u/s)time, 0.00 %gc (0x) W180801 22:03:24.138823 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:24.550698 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:24.550720 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:25.139033 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:26.139248 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:27.139528 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:28.139709 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:29.051259 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:29.051280 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:29.139896 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:30.140058 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:31.140255 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:32.140414 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:33.140613 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:33.551907 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:33.551928 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 22:03:33.984449 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 272 MiB/7.3 MiB/333 MiB GO alloc/idle/total, 545 MiB/656 MiB CGO alloc/total, 443.10cgo/sec, 0.38/0.07 %(u/s)time, 0.00 %gc (2x) W180801 22:03:34.140851 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:35.141046 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:36.141233 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:37.141393 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:38.052244 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:38.052254 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:38.141561 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:39.141793 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:40.141972 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:41.142130 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:42.142376 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:42.552814 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:42.552835 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:43.142601 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:03:43.945110 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (16m3s: infos 53/104728 sent/received, bytes 32747B/16222967B sent/received) gossip server (0/3 cur/max conns, infos 53/104728 sent/received, bytes 32747B/16222967B sent/received) I180801 22:03:43.945927 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 267 MiB/12 MiB/333 MiB GO alloc/idle/total, 554 MiB/665 MiB CGO alloc/total, 448.13cgo/sec, 0.32/0.10 %(u/s)time, 0.00 %gc (0x) W180801 22:03:44.142888 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:45.143098 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:46.143303 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:47.053266 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:47.053273 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:47.143529 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:48.143770 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:49.143959 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:50.144183 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:51.144303 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:51.553707 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:51.553721 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:52.144488 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:53.144627 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:03:53.946105 500 server/status/runtime.go:219 [n17] runtime stats: 994 MiB RSS, 373 goroutines, 267 MiB/12 MiB/333 MiB GO alloc/idle/total, 500 MiB/617 MiB CGO alloc/total, 461.99cgo/sec, 0.38/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:03:54.144903 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:55.145086 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:56.054363 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:03:56.054385 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:03:56.145322 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:57.145476 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:58.145625 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:03:59.145775 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:00.146041 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:00.554915 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:00.554930 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:01.146192 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:02.146408 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:03.146593 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:04:03.946037 500 server/status/runtime.go:219 [n17] runtime stats: 1008 MiB RSS, 372 goroutines, 268 MiB/10 MiB/333 MiB GO alloc/idle/total, 511 MiB/628 MiB CGO alloc/total, 461.80cgo/sec, 0.35/0.08 %(u/s)time, 0.00 %gc (1x) W180801 22:04:04.146832 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:05.055554 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:05.055574 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:05.146984 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:06.147407 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:07.147618 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:08.147800 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:09.148072 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:09.556116 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:09.556137 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:10.148355 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:11.148631 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:12.148835 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:13.149047 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:04:13.945890 500 server/status/runtime.go:219 [n17] runtime stats: 1014 MiB RSS, 372 goroutines, 270 MiB/8.7 MiB/333 MiB GO alloc/idle/total, 512 MiB/630 MiB CGO alloc/total, 454.11cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:04:14.056665 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:14.056687 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:14.149257 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:15.149480 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:16.149668 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:17.149849 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:18.150009 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:18.557070 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:18.557079 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:19.150197 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:20.150396 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:21.150665 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:22.150882 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:23.057645 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:23.057667 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:23.151158 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:04:23.945161 500 server/status/runtime.go:219 [n17] runtime stats: 1024 MiB RSS, 372 goroutines, 266 MiB/14 MiB/333 MiB GO alloc/idle/total, 522 MiB/641 MiB CGO alloc/total, 473.33cgo/sec, 0.35/0.08 %(u/s)time, 0.00 %gc (1x) W180801 22:04:24.151383 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:25.151611 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:26.151838 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:27.152105 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:27.558337 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:27.558358 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:28.152287 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:29.152459 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:30.152679 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:31.152901 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:32.058665 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:32.058674 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:32.153146 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:33.153314 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:04:33.946585 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 263 MiB/15 MiB/333 MiB GO alloc/idle/total, 531 MiB/651 MiB CGO alloc/total, 484.25cgo/sec, 0.34/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:04:34.153563 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:35.153857 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:36.154073 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:36.559275 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:36.559297 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:37.154493 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:38.154671 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:39.154897 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:40.155156 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:41.059949 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:41.059963 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:41.155326 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:42.155628 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:43.155928 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:04:43.945056 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (17m3s: infos 54/109977 sent/received, bytes 33423B/17068331B sent/received) gossip server (0/3 cur/max conns, infos 54/109977 sent/received, bytes 33423B/17068331B sent/received) I180801 22:04:43.946442 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 260 MiB/18 MiB/333 MiB GO alloc/idle/total, 532 MiB/652 MiB CGO alloc/total, 452.20cgo/sec, 0.35/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:04:44.156122 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:45.156311 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:45.560911 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:45.560967 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:46.156536 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:47.156753 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:48.157004 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:49.157257 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:50.061558 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:50.061579 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:50.157491 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:51.157665 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:52.157763 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:53.157886 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:04:53.945085 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 256 MiB/21 MiB/333 MiB GO alloc/idle/total, 542 MiB/661 MiB CGO alloc/total, 453.55cgo/sec, 0.26/0.06 %(u/s)time, 0.00 %gc (1x) W180801 22:04:54.158029 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:54.562273 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:54.562294 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:55.158264 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:56.158481 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:57.158693 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:58.158839 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:04:59.062868 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:04:59.062888 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:04:59.158955 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:00.159201 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:01.159432 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:02.159715 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:03.159865 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:03.563419 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:03.563491 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 22:05:03.946045 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 261 MiB/17 MiB/333 MiB GO alloc/idle/total, 544 MiB/664 MiB CGO alloc/total, 477.46cgo/sec, 0.35/0.08 %(u/s)time, 0.00 %gc (1x) W180801 22:05:04.160064 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:05.160198 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:06.160435 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:07.160677 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:08.064052 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:08.064075 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:08.160853 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:09.161047 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:10.161322 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:11.161520 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:12.161740 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:12.564691 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:12.564712 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:13.161917 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:05:13.946062 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 257 MiB/21 MiB/333 MiB GO alloc/idle/total, 552 MiB/673 MiB CGO alloc/total, 477.80cgo/sec, 0.37/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:05:14.162092 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:15.162369 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:16.162531 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:17.065339 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:17.065362 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:17.162798 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:18.163016 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:19.163227 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:20.163493 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:21.163656 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:21.565800 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:21.565816 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:22.164319 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:23.164514 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:05:23.946144 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 257 MiB/21 MiB/333 MiB GO alloc/idle/total, 561 MiB/674 MiB CGO alloc/total, 439.70cgo/sec, 0.35/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:05:24.164728 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:25.164961 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:26.066360 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:26.066382 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:26.165183 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:27.165337 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:28.165545 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:29.165727 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:30.165877 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:30.566929 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:30.566950 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:31.166081 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:32.166321 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:33.166456 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:05:33.946071 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 256 MiB/23 MiB/333 MiB GO alloc/idle/total, 562 MiB/675 MiB CGO alloc/total, 442.20cgo/sec, 0.31/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:05:34.166677 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:35.067654 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:35.067679 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:35.166860 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:36.167015 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:37.167237 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:38.167566 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:39.167727 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:39.568273 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:39.568294 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:40.167952 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:41.168179 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:42.168465 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:43.168620 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:05:43.945045 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (18m3s: infos 56/114677 sent/received, bytes 34775B/17760983B sent/received) gossip server (0/3 cur/max conns, infos 56/114677 sent/received, bytes 34775B/17760983B sent/received) I180801 22:05:43.945955 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 257 MiB/22 MiB/333 MiB GO alloc/idle/total, 570 MiB/684 MiB CGO alloc/total, 468.01cgo/sec, 0.36/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:05:44.068847 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:44.068874 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:44.168924 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:45.169146 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:46.169298 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:47.169482 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:48.169641 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:48.569198 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:48.569208 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:49.169874 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:50.170299 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:51.170566 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:52.170856 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:53.069572 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:53.069581 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:53.171059 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:05:53.946034 500 server/status/runtime.go:219 [n17] runtime stats: 1010 MiB RSS, 372 goroutines, 254 MiB/24 MiB/333 MiB GO alloc/idle/total, 516 MiB/629 MiB CGO alloc/total, 477.10cgo/sec, 0.41/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:05:54.171358 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:55.171873 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:56.172024 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:57.172192 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:57.570224 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:05:57.570245 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:05:58.172353 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:05:59.172497 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:00.172771 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:01.173020 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:02.070758 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:02.070780 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:02.173183 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:03.173381 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:06:03.946123 500 server/status/runtime.go:219 [n17] runtime stats: 1014 MiB RSS, 372 goroutines, 255 MiB/22 MiB/333 MiB GO alloc/idle/total, 527 MiB/640 MiB CGO alloc/total, 495.49cgo/sec, 0.35/0.08 %(u/s)time, 0.00 %gc (1x) W180801 22:06:04.173572 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:05.173795 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:06.174040 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:06.571367 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:06.571393 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:07.174207 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:08.174420 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:09.174676 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:10.174863 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:11.072065 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:11.072086 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:11.175170 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:12.175376 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:13.175557 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:06:13.946000 500 server/status/runtime.go:219 [n17] runtime stats: 1022 MiB RSS, 372 goroutines, 257 MiB/21 MiB/333 MiB GO alloc/idle/total, 529 MiB/642 MiB CGO alloc/total, 541.80cgo/sec, 0.37/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:06:14.175736 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:15.175903 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:15.572839 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:15.572856 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:16.176075 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:17.176251 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:18.176434 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:19.176599 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:20.073526 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:20.073559 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:20.176775 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:21.176953 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:22.177150 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:23.177282 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:06:23.945109 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 258 MiB/19 MiB/333 MiB GO alloc/idle/total, 538 MiB/652 MiB CGO alloc/total, 476.53cgo/sec, 0.36/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:06:24.177428 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:24.574004 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:24.574015 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:25.177760 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:26.177922 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:27.178093 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:28.178243 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:29.074620 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:29.074644 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:29.178349 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:30.178498 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:06:30.791038 461 storage/replica_proposal.go:201 [n17,s17,r7389/3:/System/tsd/cr.node.sql.mem.i…] new range lease repl=(n17,s17):3 seq=4 start=1533161186.387633525,0 epo=47 pro=1533161186.387671942,0 following repl=(n27,s27):5 seq=3 start=1533150387.001043176,0 epo=32 pro=1533150387.001047679,0 W180801 22:06:31.178666 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:32.178829 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:33.179008 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:33.575189 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:33.575211 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 22:06:33.945046 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 374 goroutines, 260 MiB/17 MiB/333 MiB GO alloc/idle/total, 540 MiB/654 MiB CGO alloc/total, 510.50cgo/sec, 0.34/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:06:34.179167 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:35.179335 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:36.179584 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:37.179754 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:38.075387 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:38.075415 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:38.179944 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:39.180109 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:40.180309 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:41.180469 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:42.180673 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:42.575630 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:42.575664 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:43.180872 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:06:43.945033 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (19m3s: infos 62/118732 sent/received, bytes 38831B/18339584B sent/received) gossip server (0/3 cur/max conns, infos 62/118732 sent/received, bytes 38831B/18339584B sent/received) I180801 22:06:43.946052 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 374 goroutines, 263 MiB/16 MiB/333 MiB GO alloc/idle/total, 549 MiB/662 MiB CGO alloc/total, 476.16cgo/sec, 0.38/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:06:44.181072 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:45.181279 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:46.181430 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:47.075819 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:47.075828 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:47.181607 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:48.181807 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:49.181989 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:50.182173 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:51.182414 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:51.576062 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:51.576088 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:52.182582 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:53.182801 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:06:53.946151 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 374 goroutines, 261 MiB/16 MiB/333 MiB GO alloc/idle/total, 558 MiB/671 MiB CGO alloc/total, 478.70cgo/sec, 0.34/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:06:54.182962 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:55.183160 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:56.076262 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:06:56.076288 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:06:56.183408 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:57.183566 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:58.183795 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:06:59.183951 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:00.184083 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:00.576548 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:00.576573 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:01.184281 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:02.184594 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:03.184787 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:07:03.946098 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 374 goroutines, 263 MiB/15 MiB/333 MiB GO alloc/idle/total, 559 MiB/672 MiB CGO alloc/total, 482.80cgo/sec, 0.35/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:07:04.185534 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:05.076763 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:05.076790 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:05.185785 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:06.186028 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:07.186195 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:08.186510 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:09.186692 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:09.577037 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:09.577063 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:10.186886 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:11.187033 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:12.187211 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:13.187371 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:07:13.946001 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 374 goroutines, 260 MiB/18 MiB/333 MiB GO alloc/idle/total, 568 MiB/681 MiB CGO alloc/total, 453.80cgo/sec, 0.36/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:07:14.077307 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:14.077332 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:14.187602 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:15.187810 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:16.188112 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:17.188311 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:18.188433 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:18.577962 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:18.578000 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:19.188602 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:20.188784 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:21.188962 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:22.189272 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:23.078161 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:23.078186 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:23.189427 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:07:23.946146 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 374 goroutines, 260 MiB/18 MiB/333 MiB GO alloc/idle/total, 577 MiB/690 MiB CGO alloc/total, 432.69cgo/sec, 0.34/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:07:24.189591 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:25.189757 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:26.189900 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:26.388367 15759 storage/node_liveness.go:501 [n17,s17,r7389/3:/System/tsd/cr.node.sql.mem.i…] slow heartbeat took 55.2s E180801 22:07:26.388390 15759 storage/replica_range_lease.go:282 [n17,s17,r7389/3:/System/tsd/cr.node.sql.mem.i…] context canceled W180801 22:07:27.190069 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:27.578769 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:27.578791 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:28.190307 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:29.190545 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:30.190736 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:31.191004 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:32.079493 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:32.079515 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:32.191187 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:33.191355 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:07:33.945951 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 262 MiB/16 MiB/333 MiB GO alloc/idle/total, 578 MiB/691 MiB CGO alloc/total, 476.21cgo/sec, 0.37/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:07:34.191728 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:35.191977 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:36.192131 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:36.580151 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:36.580172 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:37.192322 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:38.192635 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:39.192814 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:40.193087 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:41.080880 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:41.080911 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:41.193285 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:42.193496 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:43.193654 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:07:43.944742 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (20m3s: infos 63/124278 sent/received, bytes 39507B/19248871B sent/received) gossip server (0/3 cur/max conns, infos 63/124278 sent/received, bytes 39507B/19248871B sent/received) I180801 22:07:43.946653 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 264 MiB/14 MiB/333 MiB GO alloc/idle/total, 586 MiB/692 MiB CGO alloc/total, 475.87cgo/sec, 0.37/0.10 %(u/s)time, 0.00 %gc (1x) W180801 22:07:44.193858 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:45.194019 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:45.581618 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:45.581639 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:46.194189 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:47.194303 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:48.194568 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:49.194730 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:50.082338 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:50.082368 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:50.194918 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:51.195148 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:52.195362 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:53.195595 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:07:53.946303 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 264 MiB/14 MiB/333 MiB GO alloc/idle/total, 532 MiB/653 MiB CGO alloc/total, 460.91cgo/sec, 0.39/0.09 %(u/s)time, 0.00 %gc (1x) I180801 22:07:53.995339 480 storage/store.go:4516 [n17,s17] sstables (read amplification = 4): 0 [ 4M 1 ]: 4M 4 [ 12M 4 ]: 6M 5M 1M 16K 5 [ 273M 55 ]: 12M 10M[2] 9M[9] 8M[4] 7M[2] 6M[4] 5M[7] 4M[2] 3M[11] 2M[5] 1M[2] 664K 547K 457K 433K 371K 287K 6 [ 6G 130 ]: 100M[36] 99M[5] 98M[3] 97M[2] 96M[2] 95M[3] 94M 93M 92M[2] 91M[2] 88M 84M 59M 18M 17M 15M 12M 11M[3] 9M[2] 7M 6M 5M[2] 4M[7] 3M[4] 2M[3] 1M[7] 968K 893K 797K 717K 683K 589K 555K 519K[2] 482K 474K 424K 423K 419K 398K 390K[2] 324K 313K 312K 288K 263K 164K 156K 149K 146K 137K 134K 125K 110K 98K 91K 80K 53K 26K 22K I180801 22:07:53.995425 480 storage/store.go:4517 [n17,s17] ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 4.48 MB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 6.3 8 11 0.702 0 0 L4 4/0 11.87 MB 0.7 0.1 0.0 0.0 0.1 0.0 0.0 1.3 9.2 6.6 9 5 1.752 1841K 333K L5 55/0 273.27 MB 0.8 0.2 0.0 0.2 0.2 0.0 0.0 14.8 10.7 10.5 20 2 9.774 1376K 61K L6 130/0 5.91 GB 0.0 0.2 0.0 0.2 0.2 0.0 0.0 10.4 15.3 14.0 14 2 7.133 639K 661K Sum 190/0 6.20 GB 0.0 0.5 0.1 0.4 0.5 0.1 0.0 10.6 10.1 10.1 50 20 2.515 3857K 1056K Int 0/0 0.00 KB 0.0 0.1 0.0 0.1 0.1 0.0 0.0 6.6 8.4 8.9 17 8 2.073 1667K 208K Uptime(secs): 1213.9 total, 600.0 interval Flush(GB): cumulative 0.047, interval 0.022 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.50 GB write, 0.42 MB/s write, 0.49 GB read, 0.42 MB/s read, 50.3 seconds Interval compaction: 0.14 GB write, 0.25 MB/s write, 0.14 GB read, 0.23 MB/s read, 16.6 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count estimated_pending_compaction_bytes: 0 B W180801 22:07:54.195821 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:54.582724 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:54.582734 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:55.196056 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:56.196314 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:57.196692 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:58.196873 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:07:59.083392 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:07:59.083413 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:07:59.197007 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:00.197172 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:01.197330 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:02.197601 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:03.197763 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:03.584021 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:03.584042 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 22:08:03.946072 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 264 MiB/14 MiB/333 MiB GO alloc/idle/total, 541 MiB/663 MiB CGO alloc/total, 505.61cgo/sec, 0.37/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:08:04.198004 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:05.198168 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:06.198325 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:07.198554 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:08.084507 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:08.084527 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:08.198730 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:09.198911 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:10.199100 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:11.199214 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:12.199500 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:12.587336 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:12.587368 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:13.199632 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:08:13.946084 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 259 MiB/18 MiB/333 MiB GO alloc/idle/total, 543 MiB/664 MiB CGO alloc/total, 453.40cgo/sec, 0.38/0.08 %(u/s)time, 0.00 %gc (1x) W180801 22:08:14.199800 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:15.199995 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:16.200165 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:17.088054 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:17.088076 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:17.200432 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:18.200589 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:19.200745 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:20.200976 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:21.201231 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:21.588721 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:21.588742 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:22.201507 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:23.201619 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:08:23.946368 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 256 MiB/21 MiB/333 MiB GO alloc/idle/total, 552 MiB/674 MiB CGO alloc/total, 472.19cgo/sec, 0.34/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:08:24.201774 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:25.202001 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:26.089339 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:26.089360 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:26.202196 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:27.202494 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:28.202789 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:29.203088 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:30.203337 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:30.589914 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:30.589934 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:31.203597 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:32.203818 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:33.203999 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:08:33.946439 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 257 MiB/21 MiB/333 MiB GO alloc/idle/total, 553 MiB/675 MiB CGO alloc/total, 451.50cgo/sec, 0.33/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:08:34.204169 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:35.090754 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:35.090775 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:35.204377 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:36.204523 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:37.204734 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:38.204968 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:39.205243 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:39.591357 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:39.591377 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:40.205529 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:41.205856 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:42.206060 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:43.206211 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:08:43.944739 481 gossip/gossip.go:488 [n17] gossip status (ok, 15 nodes) gossip client (1/3 cur/max conns) 13: 10.64.2.17:26257 (21m3s: infos 65/128936 sent/received, bytes 40859B/19945235B sent/received) gossip server (0/3 cur/max conns, infos 65/128936 sent/received, bytes 40859B/19945235B sent/received) I180801 22:08:43.946915 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 252 MiB/25 MiB/333 MiB GO alloc/idle/total, 561 MiB/684 MiB CGO alloc/total, 440.68cgo/sec, 0.34/0.08 %(u/s)time, 0.00 %gc (1x) W180801 22:08:44.091959 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:44.091985 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:44.206373 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:45.206530 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:46.206642 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:47.206827 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:48.207037 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:48.592631 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:48.592653 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:49.207344 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:50.207599 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:51.207776 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:52.208067 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:53.093235 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:53.093266 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:53.208224 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:08:53.945138 500 server/status/runtime.go:219 [n17] runtime stats: 1.0 GiB RSS, 372 goroutines, 249 MiB/28 MiB/333 MiB GO alloc/idle/total, 570 MiB/693 MiB CGO alloc/total, 457.57cgo/sec, 0.35/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:08:54.208394 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:55.208680 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:56.208844 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:57.209030 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:57.593879 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:08:57.593899 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:08:58.209251 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:08:59.209416 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:00.209711 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:01.209829 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:02.094741 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:09:02.094783 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:09:02.210039 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:03.210201 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:09:03.945963 500 server/status/runtime.go:219 [n17] runtime stats: 1.1 GiB RSS, 372 goroutines, 246 MiB/32 MiB/334 MiB GO alloc/idle/total, 571 MiB/694 MiB CGO alloc/total, 478.47cgo/sec, 0.34/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:09:04.210371 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:05.210606 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:06.210755 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:06.595369 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:09:06.595391 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:09:07.210919 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:08.211171 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:09.211335 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:10.211609 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:11.096041 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:09:11.096070 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:09:11.211771 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:12.211973 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:13.212120 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:09:13.946249 500 server/status/runtime.go:219 [n17] runtime stats: 1.1 GiB RSS, 372 goroutines, 237 MiB/73 MiB/369 MiB GO alloc/idle/total, 580 MiB/704 MiB CGO alloc/total, 478.89cgo/sec, 0.37/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:09:14.212345 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:15.212543 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:15.596709 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:09:15.596731 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:09:16.212695 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:17.212926 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:18.213166 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:19.213347 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:20.097368 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:09:20.097389 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:09:20.213639 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:21.213893 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:22.214105 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:23.214261 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure I180801 22:09:23.946128 500 server/status/runtime.go:219 [n17] runtime stats: 1.1 GiB RSS, 372 goroutines, 231 MiB/78 MiB/369 MiB GO alloc/idle/total, 589 MiB/713 MiB CGO alloc/total, 507.01cgo/sec, 0.37/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:09:24.214411 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:24.597861 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:09:24.597877 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:09:25.214656 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:26.214811 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:27.214900 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:28.215136 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:29.098550 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:09:29.098571 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:09:29.215319 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:30.215509 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:31.215752 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:32.216056 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:33.216233 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:33.599199 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:09:33.599219 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded I180801 22:09:33.946052 500 server/status/runtime.go:219 [n17] runtime stats: 1.1 GiB RSS, 372 goroutines, 229 MiB/80 MiB/369 MiB GO alloc/idle/total, 589 MiB/714 MiB CGO alloc/total, 458.30cgo/sec, 0.34/0.09 %(u/s)time, 0.00 %gc (1x) W180801 22:09:34.216436 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:35.216607 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:36.216740 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:37.220384 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure W180801 22:09:38.099937 343 storage/node_liveness.go:501 [n17,hb] slow heartbeat took 4.5s W180801 22:09:38.099969 343 storage/node_liveness.go:438 [n17,hb] failed node liveness heartbeat: context deadline exceeded W180801 22:09:38.220583 344 sql/jobs/registry.go:300 canceling all jobs due to liveness failure