Project

General

Profile

Actions
Copy link

Bug #48899

closed

api_list: LibRadosList.EnumerateObjects and LibRadosList.EnumerateObjectsSplit failed

Added by Neha Ojha about 5 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

100%

Source:
Backport:
octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

2021-01-15T07:38:23.743 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: [ RUN      ] LibRadosList.EnumerateObjects
2021-01-15T07:38:23.743 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.0.0-9032-ga83a843c/rpm/el8/BUILD/ceph-16.0.0-9032-ga83a843c/src/test/librados/list.cc:418: Failure
2021-01-15T07:38:23.743 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: Value of: set_pg_num(&s_cluster, pool_name, 11).empty()
2021-01-15T07:38:23.744 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list:   Actual: false
2021-01-15T07:38:23.744 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: Expected: true
2021-01-15T07:38:23.745 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: [  FAILED  ] LibRadosList.EnumerateObjects (3668253 ms)
2021-01-15T07:38:23.745 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: [ RUN      ] LibRadosList.EnumerateObjectsSplit
2021-01-15T07:38:23.746 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.0.0-9032-ga83a843c/rpm/el8/BUILD/ceph-16.0.0-9032-ga83a843c/src/test/librados/list.cc:466: Failure
2021-01-15T07:38:23.746 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: Value of: set_pg_num(&s_cluster, pool_name, 11).empty()
2021-01-15T07:38:23.746 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list:   Actual: false
2021-01-15T07:38:23.747 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: Expected: true
2021-01-15T07:38:23.747 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: [  FAILED  ] LibRadosList.EnumerateObjectsSplit (3668757 ms)
2021-01-15T07:38:23.748 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list: [----------] 6 tests from LibRadosList (8148482 ms total)
2021-01-15T07:38:23.748 INFO:tasks.workunit.client.0.smithi093.stdout:                 api_list:

/a/nojha-2021-01-15_01:03:38-rados-wip-38730-2021-01-14-distro-basic-smithi/5787064

Related issues 2 (0 open — 2 closed)

Related to RADOS - Bug #48918: valgrind issue in PgScrubber::dump: Use of uninitialised value of size 8ResolvedBrad Hubbard

Actions
Copied to RADOS - Backport #55581: octopus: api_list: LibRadosList.EnumerateObjects and LibRadosList.EnumerateObjectsSplit failedRejectedActions
Actions
Copy link
 #1

Updated by Neha Ojha about 5 years ago

  • Related to Bug #42706: LibRadosList.EnumerateObjectsSplit fails added
Actions
Copy link
 #2

Updated by Neha Ojha about 5 years ago

EnumerateObjectsSplit and EnumerateObjects rely on a healthy cluster. When this happens, this is the state of the cluster

2021-01-19T19:08:40.680 INFO:tasks.workunit.client.0.smithi067.stdout:                 api_list: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.0.0-9142-gb27b97a4/rpm/el8/BUILD/ceph-16.0.0-9142-gb27b97a4/src/test/librados/list.cc:418: Failure
2021-01-19T19:08:40.681 INFO:tasks.workunit.client.0.smithi067.stdout:                 api_list: Value of: set_pg_num(&s_cluster, pool_name, 11).empty()
2021-01-19T19:08:40.682 INFO:tasks.workunit.client.0.smithi067.stdout:                 api_list:   Actual: false

  cluster:
    id:     747fc241-429e-42af-9946-99e189a0e8cc
    health: HEALTH_WARN
            Reduced data availability: 10 pgs stale
            Degraded data redundancy: 2/162 objects degraded (1.235%), 2 pgs degraded, 10 pgs undersized
            2 pool(s) do not have an application enabled
            45 slow ops, oldest one blocked for 11695 sec, daemons [osd.2,osd.4,osd.5] have slow ops.

  services:
    mon: 3 daemons, quorum a,b,c (age 3h)
    mgr: x(active, since 3h), standbys: y
    osd: 8 osds: 6 up (since 3h), 6 in (since 3h)

  data:
    pools:   8 pools, 201 pgs
    objects: 81 objects, 120 MiB
    usage:   347 MiB used, 540 GiB / 540 GiB avail
    pgs:     2/162 objects degraded (1.235%)
             191 active+clean
             8   stale+active+undersized
             2   stale+active+undersized+degraded

ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP  META     AVAIL    %USE  VAR   PGS  STATUS  TYPE NAME         
-1         0.70312         -  540 GiB  347 MiB  167 MiB   0 B  180 MiB  540 GiB  0.06  1.00    -          root default      
-5         0.35156         -  270 GiB  134 MiB   48 MiB   0 B   86 MiB  270 GiB  0.05  0.77    -              host smithi027
 0    ssd  0.08789   1.00000   90 GiB   37 MiB  7.9 MiB   0 B   30 MiB   90 GiB  0.04  0.65   49      up          osd.0     
 1    ssd  0.08789         0      0 B      0 B      0 B   0 B      0 B      0 B     0     0    0    down          osd.1     
 2    ssd  0.08789   1.00000   90 GiB   51 MiB   20 MiB   0 B   31 MiB   90 GiB  0.06  0.88   59      up          osd.2     
 3    ssd  0.08789   1.00000   90 GiB   45 MiB   20 MiB   0 B   26 MiB   90 GiB  0.05  0.79   68      up          osd.3     
-3         0.35156         -  270 GiB  213 MiB  120 MiB   0 B   93 MiB  270 GiB  0.08  1.23    -              host smithi144
 4    ssd  0.08789   1.00000   90 GiB   89 MiB   56 MiB   0 B   34 MiB   90 GiB  0.10  1.55   64      up          osd.4     
 5    ssd  0.08789   1.00000   90 GiB   73 MiB   44 MiB   0 B   29 MiB   90 GiB  0.08  1.26   85      up          osd.5     
 6    ssd  0.08789   1.00000   90 GiB   51 MiB   20 MiB   0 B   31 MiB   90 GiB  0.06  0.88   57      up          osd.6     
 7    ssd  0.08789         0      0 B      0 B      0 B   0 B      0 B      0 B     0     0   10    down          osd.7

2 OSDs are down due to https://tracker.ceph.com/issues/48918

/a/nojha-2021-01-19_18:03:54-rados:verify-master-distro-basic-smithi/5803214

Actions
Copy link
 #3

Updated by Neha Ojha about 5 years ago

  • Related to Bug #48918: valgrind issue in PgScrubber::dump: Use of uninitialised value of size 8 added
Actions
Copy link
 #4

Updated by Neha Ojha about 5 years ago

  • Related to deleted (Bug #42706: LibRadosList.EnumerateObjectsSplit fails)
Actions
Copy link
 #5

Updated by Neha Ojha about 5 years ago

  • Status changed from New to Resolved
Actions
Copy link
 #6

Updated by Laura Flores almost 4 years ago

  • Status changed from Resolved to Pending Backport
  • Backport set to octopus

/a/lflores-2022-05-04_18:59:38-rados-wip-55077-octopus-distro-default-smithi/6820998

2022-05-07T11:13:36.888 INFO:teuthology.orchestra.run.smithi032.stdout:{"fsid":"d6d89093-a7e1-4c30-a5f5-12a9975c9e46","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":4420,"monmap":{"epoch":1,"min_mon_release_name":"octopus","num_mons":3},"osdmap":{"epoch":2767,"num_osds":8,"num_up_osds":8,"osd_up_since":1651918726,"num_in_osds":8,"osd_in_since":1651918660,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":88},{"state_name":"active+remapped+backfilling","count":1}],"num_pgs":89,"num_pools":4,"num_objects":20,"data_bytes":2053,"bytes_used":8786190336,"bytes_avail":764307922944,"bytes_total":773094113280,"misplaced_objects":2,"misplaced_total":60,"misplaced_ratio":0.033333333333333333},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2022-05-07T11:04:47.228921+0000","services":{}},"progress_events":{}}

...

2022-05-07T11:47:42.399 INFO:tasks.workunit.client.0.smithi032.stdout:                 api_list: [ RUN      ] LibRadosList.EnumerateObjectsSplit
2022-05-07T11:47:42.399 INFO:tasks.workunit.client.0.smithi032.stdout:                 api_list: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.16-87-g7f781e26/rpm/el8/BUILD/ceph-15.2.16-87-g7f781e26/src/test/librados/list.cc:466: Failure
2022-05-07T11:47:42.400 INFO:tasks.workunit.client.0.smithi032.stdout:                 api_list: Value of: set_pg_num(&s_cluster, pool_name, 11).empty()
2022-05-07T11:47:42.400 INFO:tasks.workunit.client.0.smithi032.stdout:                 api_list:   Actual: false
2022-05-07T11:47:42.400 INFO:tasks.workunit.client.0.smithi032.stdout:                 api_list: Expected: true
2022-05-07T11:47:42.400 INFO:tasks.workunit.client.0.smithi032.stdout:                 api_list: [  FAILED  ] LibRadosList.EnumerateObjectsSplit (3608345 ms)

Actions
Copy link
 #7

Updated by Upkeep Bot almost 4 years ago

  • Copied to Backport #55581: octopus: api_list: LibRadosList.EnumerateObjects and LibRadosList.EnumerateObjectsSplit failed added
Actions
Copy link
 #9

Updated by Konstantin Shalygin almost 3 years ago

  • Status changed from Pending Backport to Resolved
  • % Done changed from 0 to 100
Actions
Copy link

Also available in: Atom PDF