Project

General

Profile

Actions

Bug #73838

closed

LazyOmapStatsTest::wait_for_active_clean -

Added by Nitzan Mordechai 4 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Tags (freeform):
backport_processed
Fixed In:
v20.3.0-4609-g74227f6f49
Released In:
Upkeep Timestamp:
2025-12-15T19:03:59+00:00

Description

/a/nmordech-2025-11-12_16:07:54-rados-wip-rocky10-branch-of-the-day-2025-11-10-1762829866-distro-default-smithi/8599258

the test shows:

2025-11-12T19:02:12.941 INFO:teuthology.orchestra.run.smithi093.stdout:pool 'lazy_omap_test_pool' created
2025-11-12T19:02:12.946 INFO:teuthology.orchestra.run.smithi093.stdout:Querying pool id
2025-11-12T19:02:12.947 INFO:teuthology.orchestra.run.smithi093.stdout:Found pool ID: 2
2025-11-12T19:02:12.978 INFO:teuthology.orchestra.run.smithi093.stdout:Created payload with 2000 keys of 445 bytes each. Total size in bytes = 890000
2025-11-12T19:02:12.978 INFO:teuthology.orchestra.run.smithi093.stdout:Waiting for active+clean
2025-11-12T19:02:13.230 INFO:tasks.ceph.mgr.x.smithi093.stderr:2025-11-12T19:02:13.228+0000 7fd1c52fd6c0 -1 mgr.server reply reply (22) Invalid argument unparseable JSON
2025-11-12T19:02:13.230 INFO:teuthology.orchestra.run.smithi093.stderr:Failed to get ! Error: 22 Invalid argument
2025-11-12T19:02:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 22

the mgr side shows:

2025-11-12T19:02:13.228+0000 7fd1c52fd6c0  1 -- [v2:172.21.15.93:6824/2003521395,v1:172.21.15.93:6825/2003521395] <== client.4189 172.21.15.93:0/4235421012 2 ==== mgr_command(tid 1: ) ==== 24+0+0 (secure 0 0 0) 0x55efb094a340 con 0x55efb0828800
2025-11-12T19:02:13.228+0000 7fd1c52fd6c0 -1 mgr.server reply reply (22) Invalid argument unparseable JSON 

Looking on the recent changes, we added std::move to the call of get_output inside do..while loop. First iterator is fine, we moved the JSON, but now we have an empty string, second iterator we are passing empty JSON, and that's the issue

void LazyOmapStatsTest::wait_for_active_clean()
{
  cout << "Waiting for active+clean" << endl;

  int index = -1;
  boost::regex reg(
      "\n" 
      R"((PG_STAT[\s\S]*))" 
      "\n +\n[0-9]");
  string command = R"({"prefix": "pg dump"})";
  int num_not_clean;
  do {
    string dump_output = get_output(std::move(command), true);
    if (index == -1) {
      boost::regex ireg(
          "\n" 
          R"((PG_STAT\s.*))" 
          "\n");
      index = find_index(dump_output, ireg, "STATE");
    }
    boost::smatch match;
    boost::regex_search(dump_output, match, reg, boost::match_not_dot_newline);
    istringstream buffer(match[1].str());
    string line;
    num_not_clean = 0;
    while (std::getline(buffer, line)) {
      if (line.compare(0, 1, "P") == 0) continue;
      boost::char_separator<char> sep{" "};
      boost::tokenizer<boost::char_separator<char>> tok(line, sep);
      vector<string> tokens(tok.begin(), tok.end());
      num_not_clean += tokens.at(index).compare("active+clean");
    }
    cout << "." << flush;
    this_thread::sleep_for(chrono::milliseconds(250));
  } while (num_not_clean);

  cout << endl;
}


Related issues 3 (0 open3 closed)

Copied to RADOS - Backport #74223: squid: LazyOmapStatsTest::wait_for_active_clean - RejectedNitzan MordechaiActions
Copied to RADOS - Backport #74224: tentacle: LazyOmapStatsTest::wait_for_active_clean - RejectedNitzan MordechaiActions
Copied to RADOS - Backport #74225: reef: LazyOmapStatsTest::wait_for_active_clean - RejectedNitzan MordechaiActions
Actions #1

Updated by Nitzan Mordechai 4 months ago

  • Description updated (diff)
Actions #2

Updated by Nitzan Mordechai 4 months ago

  • Status changed from New to In Progress
  • Assignee set to Nitzan Mordechai
Actions #3

Updated by Nitzan Mordechai 4 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 66277
Actions #4

Updated by Aishwarya Mathuria 4 months ago

  • Tags set to main-failures

/a/teuthology-2025-11-16_20:00:21-rados-main-distro-default-smithi/8605386

Actions #5

Updated by Radoslaw Zarzynski 4 months ago

scrub note: PR approved.

Actions #6

Updated by Sridhar Seshasayee 4 months ago

/a/skanta-2025-11-13_10:26:04-rados-wip-bharath3-testing-2025-11-12-2038-distro-default-smithi/
[8601370, 8601384]

Actions #7

Updated by Laura Flores 4 months ago

/a/lflores-2025-11-19_18:47:12-rados-wip-lflores-testing-2-2025-11-19-1711-distro-default-smithi/8613485

Actions #8

Updated by Laura Flores 4 months ago

/a/skanta-2025-11-18_05:30:29-rados-wip-bharath10-testing-2025-11-18-0557-distro-default-smithi/8609043

Actions #9

Updated by Radoslaw Zarzynski 4 months ago

scrub note: the PR went into QA.

Actions #10

Updated by Laura Flores 4 months ago

/a/skanta-2025-11-22_15:10:56-rados-wip-lflores-testing-4-2025-11-21-1738-distro-default-smithi/8620139

Actions #11

Updated by Radoslaw Zarzynski 4 months ago

scrub note: still in QA.

Actions #12

Updated by Kamoltat (Junior) Sirivadhna 4 months ago

bug-scrub: bump

Actions #13

Updated by Laura Flores 4 months ago

/a/lflores-2025-12-02_17:29:40-rados-wip-lflores-testing-4-2025-12-01-1527-distro-default-smithi/8635972

Actions #14

Updated by Aishwarya Mathuria 4 months ago

/a/yuriw-2025-12-03_15:44:36-rados-wip-yuri5-testing-2025-12-02-1256-distro-default-smithi/8639592
/a/yuriw-2025-12-03_15:44:36-rados-wip-yuri5-testing-2025-12-02-1256-distro-default-smithi/8639540

Actions #15

Updated by Radoslaw Zarzynski 3 months ago

scrub note: still in QA.

Actions #16

Updated by Sridhar Seshasayee 3 months ago

/a/skanta-2025-12-03_02:50:04-rados-wip-bharath5-testing-2025-12-02-1511-distro-default-smithi/8638390

Fix is yet to be merged as of writing this.

Actions #17

Updated by Radoslaw Zarzynski 3 months ago

  • Backport set to tentacle,squid,reef

Merged!

Actions #18

Updated by Radoslaw Zarzynski 3 months ago

  • Status changed from Fix Under Review to Pending Backport
Actions #19

Updated by Upkeep Bot 3 months ago

  • Merge Commit set to 74227f6f49fb37340ff8b0cae7e9bf8c0bd2612e
  • Fixed In set to v20.3.0-4609-g74227f6f49
  • Upkeep Timestamp set to 2025-12-15T19:03:59+00:00
Actions #20

Updated by Upkeep Bot 3 months ago

  • Copied to Backport #74223: squid: LazyOmapStatsTest::wait_for_active_clean - added
Actions #21

Updated by Upkeep Bot 3 months ago

  • Copied to Backport #74224: tentacle: LazyOmapStatsTest::wait_for_active_clean - added
Actions #22

Updated by Upkeep Bot 3 months ago

  • Copied to Backport #74225: reef: LazyOmapStatsTest::wait_for_active_clean - added
Actions #23

Updated by Upkeep Bot 3 months ago

  • Tags (freeform) set to backport_processed
Actions #24

Updated by Nitzan Mordechai 3 months ago

https://github.com/ceph/ceph/pull/65014 was never backported to tentacle,squid,reef - that error doesn't appear over there

Actions #25

Updated by Radoslaw Zarzynski 2 months ago

  • Status changed from Pending Backport to Resolved
  • Backport deleted (tentacle,squid,reef)
Actions

Also available in: Atom PDF