Bug #74827
openTEST_mon_features fails due to hardcoded feature count - missing nvmeof_beacon_diff
0%
Description
jayaprakash-2026-02-06_12:54:34-rados-jaya-bs-testing-05-02-2026-distro-default-trial/38067
2026-02-06T13:11:39.742 INFO:tasks.workunit.client.0.trial192.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: jq_success: printf %s '.monmap.features.persistent | length == 12'
2026-02-06T13:11:39.742 INFO:tasks.workunit.client.0.trial192.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: jq_success: sed 's/'\''/'\''\\'\'''\''/g'
2026-02-06T13:11:39.742 INFO:tasks.workunit.client.0.trial192.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: jq_success: filter_escaped='.monmap.features.persistent | length == 12'
2026-02-06T13:11:39.743 INFO:tasks.workunit.client.0.trial192.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: jq_success: echo '
2026-02-06T13:11:39.743 INFO:tasks.workunit.client.0.trial192.stderr:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":3,"fsid":"cb0a356a-9fc6-4525-beb4-42bf357ebcf4","modified":"2026-02-06T13:11:35.230941Z","created":"2026-02-06T13:11:04.782143Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7127","nonce":0}]},"addr":"127.0.0.1:7127/0","public_addr":"127.0.0.1:7127/0","priority":0,"weight":0,"time_added":"2026-02-06T13:11:04:782138+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7128","nonce":0}]},"addr":"127.0.0.1:7128/0","public_addr":"127.0.0.1:7128/0","priority":0,"weight":0,"time_added":"2026-02-06T13:11:04:782141+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7129","nonce":0}]},"addr":"127.0.0.1:7129/0","public_addr":"127.0.0.1:7129/0","priority":0,"weight":0,"time_added":"2026-02-06T13:11:30:226153+0000","crush_location":[]}]}}'
2026-02-06T13:11:39.743 INFO:tasks.workunit.client.0.trial192.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: jq_success: jq '.monmap.features.persistent | length == 12'
2026-02-06T13:11:39.746 INFO:tasks.workunit.client.0.trial192.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: jq_success: ret=false
2026-02-06T13:11:39.746 INFO:tasks.workunit.client.0.trial192.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: jq_success: [[ false == \t\r\u\e ]]
2026-02-06T13:11:39.746 INFO:tasks.workunit.client.0.trial192.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: jq_success: [[ -n "" ]]
2026-02-06T13:11:39.746 INFO:tasks.workunit.client.0.trial192.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: jq_success: [[ false == \"\" ]]
2026-02-06T13:11:39.746 INFO:tasks.workunit.client.0.trial192.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: jq_success: return 1
2026-02-06T13:11:39.746 INFO:tasks.workunit.client.0.trial192.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh:285: TEST_mon_features: return 1
Trace
2026-02-06T13:11:39.865 DEBUG:teuthology.orchestra.run:got remote process result: 1
2026-02-06T13:11:39.865 INFO:tasks.workunit.client.0.trial192.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']'
2026-02-06T13:11:39.865 INFO:tasks.workunit.client.0.trial192.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0
2026-02-06T13:11:39.865 INFO:tasks.workunit.client.0.trial192.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 1
2026-02-06T13:11:39.866 INFO:tasks.workunit:Stopping ['mon'] on client.0...
2026-02-06T13:11:39.866 DEBUG:teuthology.orchestra.run.trial192:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2026-02-06T13:11:40.219 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_ceph-c_72ccb6b54df50310bea4d2e7b9e76a5b32ea647b/qa/tasks/workunit.py", line 144, in task
_spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'),
File "/home/teuthworker/src/git.ceph.com_ceph-c_72ccb6b54df50310bea4d2e7b9e76a5b32ea647b/qa/tasks/workunit.py", line 292, in _spawn_on_all_clients
with parallel() as p:
File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_ceph-c_72ccb6b54df50310bea4d2e7b9e76a5b32ea647b/qa/tasks/workunit.py", line 433, in _run_tests
remote.run(
File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/orchestra/remote.py", line 575, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/orchestra/run.py", line 461, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed (workunit test mon/misc.sh) on trial192 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=72ccb6b54df50310bea4d2e7b9e76a5b32ea647b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh'
Updated by Radoslaw Zarzynski about 1 month ago
From the failure log:
{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_di
ff"], "optional":[]}
Python agrees it isn't 12.
>>> len(["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]) 13
nvmeof_beacon_diff looks suspicious.
Updated by Radoslaw Zarzynski about 1 month ago
The name of this feature is:
} else if (f == FEATURE_NVMEOF_BEACON_DIFF) {
return "nvmeof_beacon_diff";
This has been merged in https://github.com/ceph/ceph/pull/65563/changes but the PR doesn't alter the TEST_mon_features.
Updated by Laura Flores about 1 month ago
- Subject changed from qa/standalone/mon/misc.sh TEST_mon_features failed to TEST_mon_features fails due to hardcoded feature count - missing nvmeof_beacon_diff
Updated by Laura Flores about 1 month ago
- Has duplicate Bug #74846: TEST_mon_features fails due to hardcoded feature count - missing nvmeof_beacon_diff added
Updated by Nitzan Mordechai about 1 month ago
/a/yaarit-2026-02-10_02:34:33-rados-wip-rocky10-branch-of-the-day-2026-02-09-1770676549-distro-default-trial/42349
Updated by Leonid Chernin about 1 month ago
adding nvmeof_beacon_diff to persistent bits - is not my code, we never edit this misc.sh file.
I think when Sam did some fixes related to the introducing the beacon diff feature added this code.
I will modify the test in misc.sh
Updated by Leonid Chernin about 1 month ago
added PR that was merged to main
https://github.com/ceph/ceph/pull/67307
please re-test
Updated by Nitzan Mordechai about 1 month ago
/a/nmordech-2026-02-17_05:46:21-rados-wip-rocky10-branch-of-the-day-2026-02-13-1771001916-distro-default-trial/51934
Updated by Laura Flores about 1 month ago
- Status changed from New to Resolved
- Pull request ID set to 67307
Updated by Radoslaw Zarzynski about 1 month ago
- Status changed from Resolved to Pending Backport
Updated by Upkeep Bot about 1 month ago
- Copied to Backport #74988: tentacle: TEST_mon_features fails due to hardcoded feature count - missing nvmeof_beacon_diff added
Updated by Upkeep Bot about 1 month ago
- Tags (freeform) set to backport_processed
Updated by Upkeep Bot about 1 month ago
- Merge Commit set to b53f0f81740fcfc70440e97b7d373685fa466b71
- Fixed In set to v20.3.0-5268-gb53f0f8174
- Upkeep Timestamp set to 2026-02-18T00:57:01+00:00
Updated by Nitzan Mordechai 28 days ago
/a/nmordech-2026-02-21_21:55:34-rados-wip-rocky10-branch-of-the-day-2026-02-21-1771692874-tentacle-distro-default-trial/63234
Updated by Sridhar Seshasayee 26 days ago
/a/skanta-2026-02-22_05:18:48-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/63358
Updated by Lee Sanders 26 days ago
/a/skanta-2026-02-07_14:54:11-rados-wip-bharath5-testing-2026-02-06-2052-distro-default-trial/39489
Updated by Nitzan Mordechai 26 days ago
/a/nmordech-2026-02-23_13:43:25-rados-wip-rocky10-branch-of-the-day-2026-02-23-1771833711-distro-default-trial/66807
Updated by Jaya Prakash 25 days ago
/a/yuriw-2026-02-21_00:21:18-rados-wip-yuri3-testing-2026-02-19-1610-tentacle-distro-default-trial/62192
Updated by Kamoltat (Junior) Sirivadhna 24 days ago
Although https://github.com/ceph/ceph/pull/67307 has been merged, we are still seeing it main with this failure:
2026-02-25T16:42:50.832 INFO:tasks.workunit.client.0.trial005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: jq_success: jq '.monmap.features.persistent[]|select(. == "tentacle")' 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: jq_success: ret='"tentacle"' 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: jq_success: [[ "tentacle" == \t\r\u\e ]] 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: jq_success: [[ -n "nvmeof_beacon_diff" ]] 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: jq_success: [[ "tentacle" == \"\n\v\m\e\o\f\_\b\e\a\c\o\n\_\d\i\f\f\" ]] 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: jq_success: return 1 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh:284: TEST_mon_features: return 1 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh:31: run: return 1 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: main: code=1 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/misc 1 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/misc 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2026-02-25T16:42:50.834 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/misc KILL 2026-02-25T16:42:50.835 INFO:tasks.workunit.client.0.trial005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T16:42:50.835 INFO:tasks.workunit.client.0.trial005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T16:42:50.835 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T16:42:50.835 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T16:42:50.835 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T16:42:50.941 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T16:42:50.942 INFO:tasks.workunit.client.0.trial005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T16:42:50.942 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T16:42:50.943 INFO:tasks.workunit.client.0.trial005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T16:42:50.944 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T16:42:50.944 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T16:42:50.944 INFO:tasks.workunit.client.0.trial005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T16:42:50.945 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T16:42:50.946 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T16:42:50.946 INFO:tasks.workunit.client.0.trial005.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T16:42:50.946 INFO:tasks.workunit.client.0.trial005.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core
Updated by Laura Flores 23 days ago
/a/nmordech-2026-02-25_11:36:23-rados-wip-rocky10-branch-of-the-day-2026-02-24-1771941190-distro-default-trial/70144
Updated by Sridhar Seshasayee 20 days ago
/a/sseshasa-2026-02-26_14:56:45-rados-wip-sseshasa-testing-2026-02-26-1772100687-distro-default-trial/72454
Updated by Nitzan Mordechai 20 days ago
/a/yaarit-2026-02-26_20:20:34-rados-wip-rocky10-branch-of-the-day-2026-02-26-1772108951-distro-default-trial/73036
Updated by Nitzan Mordechai 19 days ago
/a/yaarit-2026-03-03_02:44:53-rados-wip-rocky10-branch-of-the-day-2026-03-02-1772457124-tentacle-distro-default-trial/77782
Updated by Nitzan Mordechai 18 days ago
/a/yaarit-2026-03-04_01:35:25-rados-wip-rocky10-branch-of-the-day-2026-03-03-1772543694-tentacle-distro-default-trial/80540
Updated by Nitzan Mordechai 17 days ago
/a/yaarit-2026-03-05_03:07:52-rados-wip-rocky10-branch-of-the-day-2026-03-04-1772624857-tentacle-distro-default-trial/86762
Updated by Laura Flores 16 days ago
/a/yaarit-2026-03-05_02:43:32-rados-wip-rocky10-branch-of-the-day-2026-03-04-1772633736-distro-default-trial/86399
Updated by Naveen Naidu 16 days ago
/a/yuriw-2026-03-02_18:34:01-rados-wip-yuri3-testing-2026-03-02-1622-distro-default-trial/76734
Updated by Nitzan Mordechai 14 days ago
/a/yuriw-2026-03-07_15:37:55-rados-wip-rocky10-branch-of-the-day-2026-03-06-1772840606-distro-default-trial/93227
Updated by Nitzan Mordechai 12 days ago
/a/yuriw-2026-03-09_20:52:18-rados-wip-rocky10-branch-of-the-day-2026-03-09-1773078259-distro-default-trial/95674
Updated by Nitzan Mordechai 11 days ago
/a/skanta-2026-03-08_04:44:53-rados-wip-bharath5-testing-2026-03-07-1422-distro-default-trial/94087
Updated by Nitzan Mordechai 11 days ago
/a/yuriw-2026-03-09_21:03:52-rados-wip-rocky10-branch-of-the-day-2026-03-09-1773079353-tentacle-distro-default-trial/96783
Updated by Aishwarya Mathuria 10 days ago
/a/skanta-2026-03-07_15:39:05-rados-wip-bharath4-testing-2026-03-05-1456-tentacle-distro-default-trial/93062
Updated by Sridhar Seshasayee 8 days ago
/a/skanta-2026-03-04_23:53:38-rados-wip-bharath1-testing-2026-03-04-1011-distro-default-trial/85643
Updated by Nitzan Mordechai 5 days ago
/a/nmordech-2026-03-17_06:15:36-rados-wip-rocky10-branch-of-the-day-2026-03-16-1773712510-distro-default-trial/103750
Updated by Laura Flores 2 days ago
/a/nmordech-2026-03-18_09:51:02-rados-wip-rocky10-branch-of-the-day-2026-03-18-1773820469-distro-default-trial/107433