High Service Latency: SQL in Cockroach DB v2.0.2 #26830
Description
Hi Team,
I am having very high 'Service Latency: SQL' on my Cockroach DB from a couple of days and I am not able to find the cause.
The application is during about 50 inserts/updates per second and has a 6 node cluster in Digital Ocean. Total records in the system are around 22M.
Please do find the debug info, that may help.
Node2# free -m
total used free shared buff/cache available
Mem: 32174 13746 261 216 18166 17788
Swap: 0 0 0
Node2# top
top - 15:25:02 up 49 days, 6:41, 1 user, load average: 0.65, 0.57, 0.63
Tasks: 156 total, 1 running, 155 sleeping, 0 stopped, 0 zombie
%Cpu(s): 18.5 us, 1.9 sy, 0.0 ni, 74.9 id, 4.0 wa, 0.0 hi, 0.3 si, 0.5 st
KiB Mem : 32946972 total, 357452 free, 14073752 used, 18515768 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 18217832 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10610 root 20 0 47.357g 0.013t 26480 S 6.2 42.0 26681:01 cockroach
1 root 20 0 185160 5040 3268 S 0.0 0.0 0:30.06 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 2:05.38 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root 20 0 0 0 0 S 0.0 0.0 10:11.21 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:00.57 migration/0
10 root rt 0 0 0 0 S 0.0 0.0 0:14.38 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:13.54 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:00.58 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 11:16.16 ksoftirqd/1
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
16 root rt 0 0 0 0 S 0.0 0.0 0:12.37 watchdog/2
17 root rt 0 0 0 0 S 0.0 0.0 0:00.74 migration/2
18 root 20 0 0 0 0 S 0.0 0.0 2:06.27 ksoftirqd/2
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H
21 root rt 0 0 0 0 S 0.0 0.0 0:12.84 watchdog/3
22 root rt 0 0 0 0 S 0.0 0.0 0:00.49 migration/3
23 root 20 0 0 0 0 S 0.0 0.0 2:00.40 ksoftirqd/3
25 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H
26 root rt 0 0 0 0 S 0.0 0.0 0:13.15 watchdog/4
27 root rt 0 0 0 0 S 0.0 0.0 0:00.46 migration/4
28 root 20 0 0 0 0 S 0.0 0.0 1:59.53 ksoftirqd/4
30 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0H
31 root rt 0 0 0 0 S 0.0 0.0 0:13.33 watchdog/5
32 root rt 0 0 0 0 S 0.0 0.0 0:00.47 migration/5
33 root 20 0 0 0 0 S 0.0 0.0 1:59.90 ksoftirqd/5
35 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0H
36 root rt 0 0 0 0 S 0.0 0.0 0:13.02 watchdog/6
37 root rt 0 0 0 0 S 0.0 0.0 0:00.88 migration/6
38 root 20 0 0 0 0 S 0.0 0.0 1:59.48 ksoftirqd/6
40 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/6:0H
Node3# free -m
total used free shared buff/cache available
Mem: 32174 13190 280 192 18703 18364
Swap: 0 0 0
Node3#top - 15:24:11 up 49 days, 6:41, 1 user, load average: 0.61, 0.81, 0.86
Tasks: 155 total, 1 running, 154 sleeping, 0 stopped, 0 zombie
%Cpu(s): 16.1 us, 1.7 sy, 0.0 ni, 77.7 id, 3.9 wa, 0.0 hi, 0.2 si, 0.3 st
KiB Mem : 32946972 total, 270080 free, 13514464 used, 19162428 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 18798336 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10296 root 20 0 35.452g 0.012t 21772 S 12.5 40.3 14836:24 cockroach
6153 root 20 0 40520 3624 3028 R 6.2 0.0 0:00.01 top
1 root 20 0 185220 4668 2832 S 0.0 0.0 0:27.76 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.08 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 1:53.68 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root 20 0 0 0 0 S 0.0 0.0 9:10.20 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:00.73 migration/0
10 root rt 0 0 0 0 S 0.0 0.0 0:15.12 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:13.91 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:00.70 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 10:00.63 ksoftirqd/1
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
16 root rt 0 0 0 0 S 0.0 0.0 0:12.78 watchdog/2
17 root rt 0 0 0 0 S 0.0 0.0 0:00.70 migration/2
18 root 20 0 0 0 0 S 0.0 0.0 1:49.24 ksoftirqd/2
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H
21 root rt 0 0 0 0 S 0.0 0.0 0:12.91 watchdog/3
22 root rt 0 0 0 0 S 0.0 0.0 0:00.73 migration/3
23 root 20 0 0 0 0 S 0.0 0.0 1:49.52 ksoftirqd/3
25 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H
26 root rt 0 0 0 0 S 0.0 0.0 0:12.14 watchdog/4
27 root rt 0 0 0 0 S 0.0 0.0 0:00.63 migration/4
28 root 20 0 0 0 0 S 0.0 0.0 1:50.19 ksoftirqd/4
30 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:0H
31 root rt 0 0 0 0 S 0.0 0.0 0:12.91 watchdog/5
32 root rt 0 0 0 0 S 0.0 0.0 0:00.66 migration/5
33 root 20 0 0 0 0 S 0.0 0.0 1:49.96 ksoftirqd/5
35 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:0H
36 root rt 0 0 0 0 S 0.0 0.0 0:12.88 watchdog/6
37 root rt 0 0 0 0 S 0.0 0.0 0:01.00 migration/6
38 root 20 0 0 0 0 S 0.0 0.0 1:51.36 ksoftirqd/6
# cockroach version Build Tag: v2.0.2 Build Time: 2018/05/21 14:55:20 Distribution: CCL Platform: linux amd64 (x86_64-unknown-linux-gnu) Go Version: go1.10 C Compiler: gcc 6.3.0 Build SHA-1: fa2b2bfd802ebe8150030a14b0a36c25fef90b91 Build Type: release
Also I am unable to get 'profile?debug' has req. is timing out on the server when I issue wget for the same.
cockroach-last500line-node2.log
cockroach-last500line-node3.log