Skip to content

qa/config/rados: add dispatch delay testing params#40593

Merged
jdurgin merged 1 commit intoceph:masterfrom
ideepika:wip-new-testing-params
Apr 28, 2021
Merged

qa/config/rados: add dispatch delay testing params#40593
jdurgin merged 1 commit intoceph:masterfrom
ideepika:wip-new-testing-params

Conversation

@ideepika
Copy link
Member

@ideepika ideepika commented Apr 5, 2021

these parameters have proven to catch some of the uncaught bugs,
adopting them will help in preventing more such hard to debug bugs.

Signed-off-by: Deepika Upadhyay dupadhya@redhat.com

Checklist

  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

@ideepika ideepika force-pushed the wip-new-testing-params branch from b344165 to fc43089 Compare April 5, 2021 13:17
@jdurgin
Copy link
Member

jdurgin commented Apr 6, 2021

How much time does this add to runs?

@ideepika
Copy link
Member Author

ideepika commented Apr 6, 2021 •
edited
Loading

How much time does this add to runs?

with @athanatos 's patch it took around 20-30 min(seems normal)
verifying that in these run, will do analysis once the rerun finishes with baselines
https://pulpito.ceph.com/ideepika-2021-04-06_14:26:42-rados-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/
https://pulpito.ceph.com/ideepika-2021-04-05_14:32:13-rados-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/

@neha-ojha
Copy link
Member

neha-ojha commented Apr 6, 2021

@ideepika have you reviewed all the failures in #40593 (comment)?

for example https://pulpito.ceph.com/ideepika-2021-04-06_14:26:42-rados-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/6024548/ seems related to this change based on the sentry event history https://sentry.ceph.com/organizations/ceph/issues/5540/events/bec66e879e0e4d7cb83431ebcd4f0f1c/events/?project=2

https://pulpito.ceph.com/ideepika-2021-04-05_14:32:13-rados-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/6022260/ seems related too

Here's a baseline rados run from yesterday for your reference - https://pulpito.ceph.com/nojha-2021-04-05_22:22:02-rados-master-distro-basic-smithi/

@batrick
Copy link
Member

batrick commented Apr 10, 2021

https://tracker.ceph.com/issues/50208#note-2

@ideepika ideepika force-pushed the wip-new-testing-params branch from fc43089 to 9d55c93 Compare April 12, 2021 13:30
@ideepika
Copy link
Member Author

ideepika commented Apr 13, 2021

new param tests:
https://pulpito.ceph.com/ideepika-2021-04-12_22:48:32-rados-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/
https://pulpito.ceph.com/ideepika-2021-04-12_22:13:05-rados:thrash-erasure-code-isa-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/
https://pulpito.ceph.com/ideepika-2021-04-12_22:34:43-rados:thrash-erasure-code-shec-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/
https://pulpito.ceph.com/ideepika-2021-04-12_22:36:57-rados:thrash-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/

neha-ojha
neha-ojha approved these changes Apr 16, 2021
View reviewed changes
Copy link
Member

@neha-ojha neha-ojha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding these options to the thrash suites makes sense.

@ideepika
Copy link
Member Author

ideepika commented Apr 19, 2021 •
edited
Loading 

| with    |         |          |  | without |         |          |  |
| ------- | ------- | -------- |  | ------- | ------- | -------- |  |
|         | runtime | duration |  |         | runtime | duration |  |
|         | 0:46:43 | 0:38:22  |  |         | 0:22:52 | 0:10:49  |  |
|         | 0:34:05 | 0:23:10  |  |         | 0:21:35 | 0:13:08  |  |
|         | 0:31:26 | 0:21:46  |  |         | 0:33:42 | 0:22:18  |  |
|         | 0:44:41 | 0:34:05  |  |         | 0:35:38 | 0:25:02  |  |
|         | 0:32:22 | 0:21:26  |  |         | 0:36:02 | 0:25:17  |  |
|         | 0:22:09 | 0:10:43  |  |         | 0:36:42 | 0:25:26  |  |
|         | 0:31:58 | 0:22:00  |  |         | 0:36:42 | 0:25:28  |  |
|         | 0:49:17 | 0:38:06  |  |         | 0:36:18 | 0:25:34  |  |
|         | 0:41:26 | 0:33:37  |  |         | 0:40:11 | 0:28:31  |  |
|         | 1:03:43 | 0:57:34  |  |         | 0:35:44 | 0:29:30  |  |
|         | 0:27:12 | 0:21:12  |  |         | 0:45:22 | 0:30:56  |  |
|         | 0:36:32 | 0:24:52  |  |         | 0:38:35 | 0:31:08  |  |
|         | 0:39:58 | 0:29:04  |  |         | 0:40:03 | 0:31:20  |  |
|         | 0:41:33 | 0:29:06  |  |         | 0:40:45 | 0:32:25  |  |
|         | 0:40:16 | 0:28:43  |  |         | 0:40:11 | 0:33:19  |  |
|         | 0:48:07 | 0:40:46  |  |         | 0:42:15 | 0:34:38  |  |
|         | 0:22:27 | 0:10:24  |  |         | 0:42:14 | 0:34:54  |  |
|         | 0:23:39 | 0:10:35  |  |         | 0:45:11 | 0:37:52  |  |
| average | 0:37:39 | 0:27:32  |  |         | 0:37:13 | 0:27:39  |  |

seems not to add up any additional runtime:
https://pulpito.ceph.com/ideepika-2021-04-19_04:50:07-rados-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi
https://pulpito.ceph.com/ideepika-2021-04-12_22:48:32-rados-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi

is limited to rados:thrash sub-suite, should keep noise away for other components.

qa/config/rados: add dispatch delay testing params
b2c2a43 
these parameters have proven to catch some of the uncaught bugs such as:
https://tracker.ceph.com/issues/48417, adopting them will help in
preventing more such hard to debug bugs.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
@ideepika ideepika force-pushed the wip-new-testing-params branch from 9d55c93 to b2c2a43 Compare April 19, 2021 06:58
@jdurgin
Copy link
Member

jdurgin commented Apr 28, 2021

jenkins test api

@jdurgin jdurgin merged commit 0e273e6 into ceph:master Apr 28, 2021
@ideepika ideepika deleted the wip-new-testing-params branch April 29, 2021 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@athanatos athanatos athanatos approved these changes

@jdurgin jdurgin jdurgin approved these changes

@neha-ojha neha-ojha neha-ojha approved these changes

Assignees

No one assigned

Labels

core wip-deepika-testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ideepika @jdurgin @neha-ojha @batrick @athanatos