This repository was archived by the owner on Oct 16, 2020. It is now read-only.
This repository was archived by the owner on Oct 16, 2020. It is now read-only.
[AWS EBS] NVMe sometimes fails to attach #2371
Description
Issue Report
Container Linux Version
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1632.3.0
VERSION_ID=1632.3.0
BUILD_ID=2018-02-14-0338
PRETTY_NAME="Container Linux by CoreOS 1632.3.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
Environment
- AWS EC2
m5.xlargeinstance - Encrypted EBS gp2 volume attached as
/dev/xvdba
Expected Behavior
Every single AWS EBS NVMe volume gets attached to the nodes.
Actual Behavior
Some volumes fail to be brought up by the kernel:
[ 2753.800168] pci 0000:00:1e.0: [1d0f:8061] type 00 class 0x010802
[ 2753.800270] pci 0000:00:1e.0: reg 0x10: [mem 0x00000000-0x00003fff]
[ 2753.801205] pci 0000:00:1e.0: BAR 0: assigned [mem 0xc0004000-0xc0007fff]
[ 2753.803879] nvme nvme2: pci function 0000:00:1e.0
[ 2753.805987] nvme 0000:00:1e.0: enabling device (0000 -> 0002)
[ 2753.919589] nvme nvme2: failed to mark controller live
[ 2753.920704] nvme nvme2: Removing after probe failure status: 0
Some similar volumes actually attached successfully:
$ sudo nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 vol0ff923da7ac79719f Amazon Elastic Block Store 1 0.00 B / 107.37 GB 512 B + 0 B 1.0
/dev/nvme1n1 vol031fc070a0ea9b7b7 Amazon Elastic Block Store 1 0.00 B / 42.95 GB 512 B + 0 B 1.0
$ sudo nvme id-ctrl /dev/nvme0n1
NVME Identify Controller:
vid : 0x1d0f
ssvid : 0x1d0f
sn : vol0ff923da7ac79719f
mn : Amazon Elastic Block Store
fr : 1.0
rab : 32
ieee : dc02a0
cmic : 0
mdts : 6
cntlid : 0
ver : 0
rtd3r : 0
rtd3e : 0
oaes : 0
oacs : 0
acl : 4
aerl : 0
frmw : 0x3
lpa : 0
elpe : 0
npss : 1
avscc : 0x1
apsta : 0
wctemp : 0
cctemp : 0
mtfa : 0
hmpre : 0
hmmin : 0
tnvmcap : 0
unvmcap : 0
rpmbs : 0
sqes : 0x66
cqes : 0x44
nn : 1
oncs : 0
fuses : 0
fna : 0
vwc : 0x1
awun : 0
awupf : 0
nvscc : 0
acwu : 0
sgls : 0
subnqn :
ps 0 : mp:0.01W operational enlat:1000000 exlat:1000000 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-
ps 1 : mp:0.00W operational enlat:0 exlat:0 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-