Bug #74660: teuthology fio Input/output error - nvme-of - Ceph

Actions

Copy link

Bug #74660

open

teuthology fio Input/output error

Added by Vallari Agrawal about 2 months ago. Updated 9 days ago.

Status:

New

Priority:

Normal

Assignee:

Vallari Agrawal

Target version:

% Done:

Source:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Tags (freeform):

Merge Commit:

Fixed In:

Released In:

Upkeep Timestamp:

Description

Fio failed with return code 8 and io_u error on multiple devices (on main branch)

2026-01-29T16:11:42.619 INFO:tasks.workunit.client.0.trial011.stdout:fio: pid=207843, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-01-29T16:11:42.703 INFO:tasks.workunit.client.0.trial011.stderr:fio: io_u error on file /dev/nvme28n85: Input/output error: write offset=715374592, buflen=28672
2026-01-29T16:11:42.705 INFO:tasks.workunit.client.0.trial011.stdout:fio: pid=207866, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-01-29T16:11:42.724 INFO:tasks.workunit.client.0.trial011.stderr:fio: io_u error on file /dev/nvme32n31: Input/output error: write offset=293437440, buflen=53248
2026-01-29T16:11:42.726 INFO:tasks.workunit.client.0.trial011.stdout:fio: pid=207838, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-01-29T16:11:42.735 INFO:tasks.workunit.client.0.trial011.stderr:fio: io_u error on file /dev/nvme16n19: Input/output error: read offset=548081664, buflen=8192
2026-01-29T16:11:42.738 INFO:tasks.workunit.client.0.trial011.stdout:fio: pid=207837, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-01-29T16:11:42.759 INFO:tasks.workunit.client.0.trial011.stderr:fio: io_u error on file /dev/nvme8n19: Input/output error: read offset=47222784, buflen=53248
2026-01-29T16:11:42.762 INFO:tasks.workunit.client.0.trial011.stdout:fio: pid=208678, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-01-29T16:11:42.767 INFO:tasks.workunit.client.0.trial011.stderr:fio: io_u error on file /dev/nvme4n77: Input/output error: write offset=803434496, buflen=24576
2026-01-29T16:11:42.773 INFO:tasks.workunit.client.0.trial011.stdout:fio: pid=208679, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-01-29T16:11:42.778 INFO:tasks.workunit.client.0.trial011.stderr:fio: io_u error on file /dev/nvme24n43: Input/output error: write offset=534425600, buflen=12288
2026-01-29T16:11:42.782 INFO:tasks.workunit.client.0.trial011.stdout:fio: pid=208705, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-01-29T16:11:42.808 INFO:tasks.workunit.client.0.trial011.stderr:fio: io_u error on file /dev/nvme12n89: Input/output error: read offset=353738752, buflen=8192
2026-01-29T16:11:42.809 INFO:tasks.workunit.client.0.trial011.stdout:fio: pid=207846, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
...
2026-01-29T16:14:20.094 INFO:tasks.workunit.client.0.trial011.stdout:job-/dev/nvme16n18: (groupid=0, jobs=32): err= 5 (file:io_u.c:1889, func=io_u error, error=Input/output error): pid=207835: Thu Jan 29 16:14:20 2026
2026-01-29T16:14:20.094 INFO:tasks.workunit.client.0.trial011.stdout:  read: IOPS=2431, BW=58.8MiB/s (61.6MB/s)(68.9GiB/1200006msec)
2026-01-29T16:14:20.094 INFO:tasks.workunit.client.0.trial011.stdout:    clat (usec): min=59, max=22439k, avg=2029.97, stdev=23181.20
2026-01-29T16:14:20.094 INFO:tasks.workunit.client.0.trial011.stdout:     lat (usec): min=59, max=22439k, avg=2030.03, stdev=23181.20
2026-01-29T16:14:20.094 INFO:tasks.workunit.client.0.trial011.stdout:    clat percentiles (usec):
2026-01-29T16:14:20.094 INFO:tasks.workunit.client.0.trial011.stdout:     |  1.00th=[  269],  5.00th=[  318], 10.00th=[  359], 20.00th=[  424],
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:     | 30.00th=[  482], 40.00th=[  545], 50.00th=[  635], 60.00th=[  971],
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:     | 70.00th=[ 2089], 80.00th=[ 3458], 90.00th=[ 5145], 95.00th=[ 7177],
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:     | 99.00th=[12387], 99.50th=[14877], 99.90th=[20841], 99.95th=[23725],
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:     | 99.99th=[31589]
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:   bw (  KiB/s): min= 9607, max=319377, per=100.00%, avg=63289.62, stdev=839.46, samples=73010
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:   iops        : min=  488, max= 9522, avg=2554.36, stdev=23.98, samples=73010
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:  write: IOPS=2430, BW=58.8MiB/s (61.6MB/s)(68.9GiB/1200006msec); 0 zone resets
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:    clat (usec): min=857, max=23424k, avg=10668.62, stdev=49390.47
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:     lat (usec): min=862, max=23424k, avg=10700.30, stdev=49390.62
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:    clat percentiles (usec):
2026-01-29T16:14:20.095 INFO:tasks.workunit.client.0.trial011.stdout:     |  1.00th=[ 2409],  5.00th=[ 4293], 10.00th=[ 5080], 20.00th=[ 5997],
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:     | 30.00th=[ 7242], 40.00th=[ 8455], 50.00th=[ 9503], 60.00th=[10814],
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:     | 70.00th=[12387], 80.00th=[14222], 90.00th=[17433], 95.00th=[20317],
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:     | 99.00th=[27132], 99.50th=[30016], 99.90th=[36439], 99.95th=[39584],
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:     | 99.99th=[48497]
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:   bw (  KiB/s): min=14122, max=307451, per=100.00%, avg=63296.73, stdev=756.03, samples=73005
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:   iops        : min=  698, max= 8938, avg=2554.30, stdev=18.95, samples=73005
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:  lat (usec)   : 100=0.03%, 250=0.24%, 500=16.34%, 750=11.57%, 1000=1.97%
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:  lat (msec)   : 2=4.73%, 4=8.97%, 10=32.36%, 20=20.96%, 50=2.82%
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:  lat (msec)   : 100=0.01%, 250=0.01%, 1000=0.01%, >=2000=0.01%
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:  cpu          : usr=0.29%, sys=0.07%, ctx=6110333, majf=0, minf=80345
2026-01-29T16:14:20.096 INFO:tasks.workunit.client.0.trial011.stdout:  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:     complete  : 0=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:     issued rwts: total=2917347,2916931,0,0 short=0,0,0,0 dropped=0,0,0,0
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:     latency   : target=0, window=0, percentile=100.00%, depth=1
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:Run status group 0 (all jobs):
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:   READ: bw=58.8MiB/s (61.6MB/s), 58.8MiB/s-58.8MiB/s (61.6MB/s-61.6MB/s), io=68.9GiB (74.0GB), run=1200006-1200006msec
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:  WRITE: bw=58.8MiB/s (61.6MB/s), 58.8MiB/s-58.8MiB/s (61.6MB/s-61.6MB/s), io=68.9GiB (74.0GB), run=1200006-1200006msec
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:Disk stats (read/write):
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:  nvme16n18: ios=89164/88840, merge=0/0, ticks=195286/965418, in_queue=1160704, util=93.16%
2026-01-29T16:14:20.097 INFO:tasks.workunit.client.0.trial011.stdout:  nvme4n83: ios=98273/98568, merge=0/0, ticks=183671/1011696, in_queue=1195367, util=95.98%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme16n19: ios=81930/81777, merge=0/0, ticks=160396/858479, in_queue=1018875, util=93.56%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme32n31: ios=79959/79951, merge=0/0, ticks=158007/860118, in_queue=1018125, util=93.63%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme36n79: ios=94340/94406, merge=0/0, ticks=188568/1006921, in_queue=1195489, util=96.17%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme40n33: ios=98481/97897, merge=0/0, ticks=183239/1012105, in_queue=1195344, util=96.18%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme36n55: ios=97222/97125, merge=0/0, ticks=187078/1008425, in_queue=1195503, util=96.37%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme20n14: ios=93910/93981, merge=0/0, ticks=171303/984569, in_queue=1155872, util=93.19%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme16n55: ios=80718/81232, merge=0/0, ticks=157322/860661, in_queue=1017983, util=94.00%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme4n30: ios=93811/93094, merge=0/0, ticks=175169/980937, in_queue=1156106, util=93.52%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme32n30: ios=87443/87455, merge=0/0, ticks=198300/962599, in_queue=1160899, util=94.11%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme12n89: ios=81110/81453, merge=0/0, ticks=157127/855981, in_queue=1013108, util=94.18%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme16n33: ios=98329/98561, merge=0/0, ticks=180506/1014906, in_queue=1195412, util=97.04%
2026-01-29T16:14:20.098 INFO:tasks.workunit.client.0.trial011.stdout:  nvme36n24: ios=88939/89374, merge=0/0, ticks=197005/964630, in_queue=1161635, util=94.28%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme20n3: ios=99155/99026, merge=0/0, ticks=180050/1015480, in_queue=1195530, util=97.16%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme24n77: ios=98161/97913, merge=0/0, ticks=180275/1015194, in_queue=1195469, util=97.40%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme8n38: ios=89843/88565, merge=0/0, ticks=196450/964172, in_queue=1160622, util=94.62%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme28n31: ios=98520/98078, merge=0/0, ticks=183937/1011425, in_queue=1195362, util=97.72%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme28n85: ios=79253/79310, merge=0/0, ticks=164040/854037, in_queue=1018077, util=95.68%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme28n34: ios=94077/93869, merge=0/0, ticks=174216/981586, in_queue=1155802, util=94.74%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme12n10: ios=92354/92996, merge=0/0, ticks=179315/976618, in_queue=1155933, util=94.83%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme4n63: ios=97687/97311, merge=0/0, ticks=183888/1011489, in_queue=1195377, util=98.14%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme28n55: ios=96073/95763, merge=0/0, ticks=183529/1011967, in_queue=1195496, util=98.34%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme20n54: ios=95507/94700, merge=0/0, ticks=175858/980050, in_queue=1155908, util=95.35%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme32n54: ios=86797/86095, merge=0/0, ticks=200642/960456, in_queue=1161098, util=95.92%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme8n29: ios=96308/96198, merge=0/0, ticks=189827/1005644, in_queue=1195471, util=98.98%
2026-01-29T16:14:20.099 INFO:tasks.workunit.client.0.trial011.stdout:  nvme8n19: ios=79405/79317, merge=0/0, ticks=163430/854960, in_queue=1018390, util=97.17%
2026-01-29T16:14:20.100 INFO:tasks.workunit.client.0.trial011.stdout:  nvme4n77: ios=80021/80313, merge=0/0, ticks=162519/855744, in_queue=1018263, util=97.15%
2026-01-29T16:14:20.100 INFO:tasks.workunit.client.0.trial011.stdout:  nvme24n43: ios=82095/81682, merge=0/0, ticks=161157/856774, in_queue=1017931, util=97.18%
2026-01-29T16:14:20.100 INFO:tasks.workunit.client.0.trial011.stdout:  nvme12n58: ios=94419/94410, merge=0/0, ticks=177433/978462, in_queue=1155895, util=96.20%
2026-01-29T16:14:20.100 INFO:tasks.workunit.client.0.trial011.stdout:  nvme16n9: ios=97086/97877, merge=0/0, ticks=188606/1006962, in_queue=1195568, util=99.60%
2026-01-29T16:14:20.100 INFO:tasks.workunit.client.0.trial011.stdout:  nvme28n19: ios=99211/99264, merge=0/0, ticks=176457/1018995, in_queue=1195452, util=99.78%
2026-01-29T16:14:20.100 INFO:tasks.workunit.client.0.trial011.stderr:+ '[' 8 -ne 0 ']'
2026-01-29T16:14:20.101 INFO:tasks.workunit.client.0.trial011.stdout:[nvmeof.fio]: fio failed!

https://pulpito.ceph.com/vallariag-2026-01-29_15:22:39-nvmeof-main-distro-default-trial/26316/

Actions

Copy link

Updated by Vallari Agrawal about 2 months ago

Description updated (diff)

Actions

Copy link

Updated by Vallari Agrawal about 2 months ago

Subject changed from fio Input/output error to teuthology fio Input/output error

Actions

Copy link

Updated by Vallari Agrawal about 2 months ago · Edited

Observing https://pulpito.ceph.com/vallariag-2026-02-04_07:27:21-nvmeof-main-distro-default-trial/33796/

$ grep ":fio:" teuthology.log.11
2026-02-04T08:01:58.750 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme44n2: Input/output error: read offset=127537152, buflen=8192
2026-02-04T08:01:58.758 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72886, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:01:58.759 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme52n2: Input/output error: write offset=255967232, buflen=20480
2026-02-04T08:01:58.766 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72862, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:01:58.766 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme20n2: Input/output error: write offset=840749056, buflen=57344
2026-02-04T08:01:58.767 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme28n2: Input/output error: read offset=320696320, buflen=4096
2026-02-04T08:01:58.771 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme16n2: Input/output error: write offset=675028992, buflen=8192
2026-02-04T08:01:58.772 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72866, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:01:58.774 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72869, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:01:58.776 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72881, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:01:58.780 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme64n2: Input/output error: read offset=802365440, buflen=8192
2026-02-04T08:01:58.780 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme48n2: Input/output error: write offset=239034368, buflen=4096
2026-02-04T08:01:58.783 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme4n2: Input/output error: read offset=842371072, buflen=16384
2026-02-04T08:01:58.784 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72876, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:01:58.784 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72865, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:01:58.788 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72879, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:01:58.800 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme24n2: Input/output error: read offset=784076800, buflen=28672
2026-02-04T08:01:58.803 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72860, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:02:21.613 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme8n3: Input/output error: write offset=665735168, buflen=8192
2026-02-04T08:02:21.614 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72867, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:02:21.614 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme32n3: Input/output error: write offset=1016147968, buflen=8192
2026-02-04T08:02:21.617 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72890, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:02:21.648 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme16n3: Input/output error: write offset=825405440, buflen=32768
2026-02-04T08:02:21.649 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72872, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:02:21.655 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme20n3: Input/output error: write offset=282124288, buflen=12288
2026-02-04T08:02:21.656 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72877, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:02:21.704 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme36n3: Input/output error: write offset=930676736, buflen=45056
2026-02-04T08:02:21.704 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72880, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:02:21.759 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme48n3: Input/output error: write offset=533045248, buflen=4096
2026-02-04T08:02:21.759 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72873, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:02:21.776 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme52n3: Input/output error: write offset=535871488, buflen=8192
2026-02-04T08:02:21.776 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72891, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:02:21.808 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme64n3: Input/output error: write offset=854343680, buflen=4096
2026-02-04T08:02:21.808 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72883, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:47.827 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme64n1: Input/output error: read offset=647192576, buflen=40960
2026-02-04T08:07:47.835 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72889, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:47.865 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme12n1: Input/output error: read offset=836255744, buflen=12288
2026-02-04T08:07:47.868 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72875, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:47.868 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme32n1: Input/output error: write offset=653799424, buflen=32768
2026-02-04T08:07:47.873 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72884, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:47.892 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme48n1: Input/output error: read offset=519376896, buflen=4096
2026-02-04T08:07:47.897 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72871, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:47.905 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme56n1: Input/output error: write offset=514629632, buflen=4096
2026-02-04T08:07:47.911 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72861, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:47.920 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme36n1: Input/output error: read offset=493207552, buflen=57344
2026-02-04T08:07:47.923 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72885, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:56.628 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme4n4: Input/output error: write offset=726396928, buflen=32768
2026-02-04T08:07:56.630 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72887, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:56.662 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme12n4: Input/output error: write offset=389734400, buflen=24576
2026-02-04T08:07:56.666 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72870, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:56.667 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme8n4: Input/output error: read offset=683814912, buflen=20480
2026-02-04T08:07:56.669 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72874, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:56.684 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme20n4: Input/output error: write offset=219742208, buflen=24576
2026-02-04T08:07:56.685 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72868, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:56.732 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme28n4: Input/output error: read offset=341274624, buflen=20480
2026-02-04T08:07:56.733 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72888, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:56.749 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme36n4: Input/output error: write offset=28143616, buflen=12288
2026-02-04T08:07:56.750 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72882, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:56.764 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme40n4: Input/output error: write offset=889032704, buflen=4096
2026-02-04T08:07:56.764 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72864, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:56.795 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme48n4: Input/output error: read offset=799129600, buflen=4096
2026-02-04T08:07:56.795 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72878, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error
2026-02-04T08:07:56.812 INFO:tasks.workunit.client.0.trial020.stderr:fio: io_u error on file /dev/nvme52n4: Input/output error: read offset=359665664, buflen=16384
2026-02-04T08:07:56.812 INFO:tasks.workunit.client.0.trial020.stdout:fio: pid=72863, err=5/file:io_u.c:1889, func=io_u error, error=Input/output error

Before first batch of errors:
nvmeof.a was removed and mon was stopped at the same time:

2026-02-04T08:01:43.190 INFO:tasks.nvmeof.[nvmeof.thrasher]:kill nvmeof.a
2026-02-04T08:01:43.190 DEBUG:teuthology.orchestra.run.trial040:> ceph orch daemon rm nvmeof.nvmeof.a
...
2026-02-04T08:01:43.770 INFO:tasks.mon_thrash.[nvmeof.thrasher.mon_thrasher]:killing mon.c
2026-02-04T08:01:43.770 INFO:tasks.cephadm.mon.c:Stopping mon.c...
2026-02-04T08:01:43.770 DEBUG:teuthology.orchestra.run.trial118:> sudo systemctl stop ceph-b99e93ee-019d-11f1-9e96-d404e6e7d460@mon.c

Before second batch of errors:

2026-02-04T08:01:59.745 INFO:tasks.nvmeof.[nvmeof.thrasher]:kill nvmeof.b
2026-02-04T08:01:59.745 DEBUG:teuthology.orchestra.run.trial115:> ceph orch daemon stop nvmeof.nvmeof.b
...
2026-02-04T08:02:00.054 INFO:tasks.nvmeof.[nvmeof.thrasher]:kill nvmeof.d
2026-02-04T08:02:00.055 DEBUG:teuthology.orchestra.run.trial163:> ceph orch daemon rm nvmeof.nvmeof.d

Before third batch of errors:

2026-02-04T08:07:32.917 INFO:tasks.nvmeof.[nvmeof.thrasher]:kill nvmeof.a
2026-02-04T08:07:32.917 DEBUG:teuthology.orchestra.run.trial040:> ceph orch daemon rm nvmeof.nvmeof.a
...
2026-02-04T08:07:33.476 INFO:tasks.mon_thrash.[nvmeof.thrasher.mon_thrasher]:killing mon.c
2026-02-04T08:07:33.476 INFO:tasks.cephadm.mon.c:Stopping mon.c...
2026-02-04T08:07:33.477 DEBUG:teuthology.orchestra.run.trial118:> sudo systemctl stop ceph-b99e93ee-019d-11f1-9e96-d404e6e7d460@mon.c

Before forth batch of errors:

2026-02-04T08:07:33.476 INFO:tasks.mon_thrash.[nvmeof.thrasher.mon_thrasher]:killing mon.c
2026-02-04T08:07:33.476 INFO:tasks.cephadm.mon.c:Stopping mon.c...
2026-02-04T08:07:33.477 DEBUG:teuthology.orchestra.run.trial118:> sudo systemctl stop ceph-b99e93ee-019d-11f1-9e96-d404e6e7d460@mon.c
...
2026-02-04T08:07:48.830 INFO:tasks.nvmeof.[nvmeof.thrasher]:kill nvmeof.b
2026-02-04T08:07:48.830 DEBUG:teuthology.orchestra.run.trial115:> ceph orch daemon rm nvmeof.nvmeof.b

Actions

Copy link

Updated by Vallari Agrawal 23 days ago

Assignee set to Vallari Agrawal

Actions

Copy link

Updated by Vallari Agrawal 9 days ago

https://pulpito.ceph.com/yuriw-2026-03-09_21:07:04-nvmeof-wip-rocky10-branch-of-the-day-2026-03-09-1773078259-distro-default-trial/96657
https://pulpito.ceph.com/yuriw-2026-03-09_21:07:04-nvmeof-wip-rocky10-branch-of-the-day-2026-03-09-1773078259-distro-default-trial/96667

Actions

Copy link

Updated by Vallari Agrawal 9 days ago

Replication steps:

I think it's a combination of "ceph orch rm nvmeof.b" and "systemctl stop mon.c" at same second (try doing two iterations of this)

1. remove nvmeof.b and stop mon.b (first mon, then gateway, almost at same second)
2. systemctl start mon.b
3. remove nvmeof.c & nvmeof.d and stop mon.b (first mon then gateways, almost at same second)

On node1, I ran:
    1  ceph -s
    2  ceph orch ps
    3  ceph nvme-gw show mypool mygroup1
    4  ceph orch ps --dameon-type nvmeof
    5  ceph orch ps --daemon-type nvmeof
    6  ceph daemon rm nvmeof.mypool.mygroup1.fio-vallari-cluster-node1.rcidzv
    7  ceph orch daemon rm nvmeof.mypool.mygroup1.fio-vallari-cluster-node1.rcidzv
    8  ceph orch ps --daemon-type nvmeof
    9  ceph orch daemon rm nvmeof.mypool.mygroup1.fio-vallari-cluster-node4.akbgku
   10  ceph orch ps --daemon-type nvmeof
   11  ceph orch daemon rm nvmeof.mypool.mygroup1.fio-vallari-cluster-node1.gtwqzy
   12  ceph orch daemon rm nvmeof.mypool.mygroup1.fio-vallari-cluster-node2.hdxptj
   13  ceph orch ps --daemon-type nvmeof
   14  ceph orch ps --daemon-type mon
   15  ceph -s
   16  ceph orch ps --daemon-type nvmeof
   17  ceph orch daemon rm nvmeof.mypool.mygroup1.fio-vallari-cluster-node1.hvyjxt
On node3:
    1  systemctl | grep mon
    2  systemctl stop ceph-7c9ae176-01c9-11f1-bb87-02002d88170e@mon.fio-vallari-cluster-node3.service
    3  systemctl | grep mon
    4  systemctl start ceph-7c9ae176-01c9-11f1-bb87-02002d88170e@mon.fio-vallari-cluster-node3.service
    5  systemctl | grep mon
    6  systemctl start ceph-7c9ae176-01c9-11f1-bb87-02002d88170e@mon.fio-vallari-cluster-node3.service
    7  date
    8  systemctl stop ceph-7c9ae176-01c9-11f1-bb87-02002d88170e@mon.fio-vallari-cluster-node3.service
    9  systemctl start ceph-7c9ae176-01c9-11f1-bb87-02002d88170e@mon.fio-vallari-cluster-node3.service
   10  systemctl stop ceph-7c9ae176-01c9-11f1-bb87-02002d88170e@mon.fio-vallari-cluster-node3.service
   11  history | less
   12  export HISTTIMEFORMAT="%F %T " 
   13  history | less
and fio on initator with
sh fio.sh

Actions

Copy link