Skip to content

test/c_write_operations: set RETURNVEC on expected cmpext failuer#50210

Merged
yuriw merged 1 commit intoceph:mainfrom
NitzanMordhai:wip-nitzan-c-write-operations-inject-sockert
May 19, 2023
Merged

test/c_write_operations: set RETURNVEC on expected cmpext failuer#50210
yuriw merged 1 commit intoceph:mainfrom
NitzanMordhai:wip-nitzan-c-write-operations-inject-sockert

Conversation

@NitzanMordhai
Copy link
Contributor

@NitzanMordhai NitzanMordhai commented Feb 22, 2023 •
edited
Loading

We are sending cmpext request that return error and rval error. In order to get reliably rval return for cmpext, we need to set RETURNVEC flag, so we record in pg log in case of reconnect during socket injection. Without that flag, the re-request of the op won't have the rval set, and we will end with an unmatch error.

Fixes: https://tracker.ceph.com/issues/21592
Signed-off-by: Nitzan Mordechai nmordec@redhat.com

Contribution Guidelines

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows

@NitzanMordhai NitzanMordhai requested a review from a team February 23, 2023 13:21
@rzarzynski rzarzynski self-requested a review February 27, 2023 18:56
rzarzynski
rzarzynski approved these changes Feb 27, 2023
View reviewed changes
src/test/librados/c_write_operations.cc

ASSERT_EQ(-MAX_ERRNO - 1, val);

// do it 1000 times to make sure we are hitting
Copy link
Contributor

@rzarzynski rzarzynski Feb 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to make sure – a purist would argue it's about increasing the likelihood ;-).

Copy link
Contributor Author

@NitzanMordhai NitzanMordhai Feb 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rzarzynski indeed, we want to make sure we will hit on some point the socket failure injection and testing the error return value

src/test/librados/c_write_operations.cc Outdated
rados_write_op_cmpext(op, "four", 4, 0, &val);
rados_write_op_write(op, "six ", 4, 0);
std::string const s = "test_" + std::to_string(i);
ASSERT_EQ(-MAX_ERRNO - 1, rados_write_op_operate(op, ioctx, s.c_str(), NULL, LIBRADOS_OPERATION_RETURNVEC));
Copy link
Contributor

@rzarzynski rzarzynski Feb 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: too long line.

nit: NULL instead of nullptr in changed code?

ACK for LIBRADOS_OPERATION_RETURNVEC.

Copy link
Contributor Author

@NitzanMordhai NitzanMordhai Feb 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all other calls to rados_write_op_operate using NULL and not nullptr, do you think we should consider changing it?

Copy link
Contributor

@rzarzynski rzarzynski Feb 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won't run atilt for that :-).

@NitzanMordhai NitzanMordhai force-pushed the wip-nitzan-c-write-operations-inject-sockert branch from 97cfd6d to 80a531c Compare February 28, 2023 06:34
@ljflores
Copy link
Member

ljflores commented Mar 29, 2023

Hey @NitzanMordhai, I see a related failure with this one:

/a/yuriw-2023-03-28_22:45:07-rados-wip-yuri7-testing-2023-03-28-0942-distro-default-smithi/7224707

2023-03-29T09:45:51.704 INFO:tasks.workunit.client.0.smithi161.stdout:   api_c_write_operations: [ RUN      ] LibRadosCWriteOps.CmpExt
2023-03-29T09:45:51.704 INFO:tasks.workunit.client.0.smithi161.stdout:   api_c_write_operations: /build/ceph-18.0.0-3159-ga2027c62/src/test/librados/c_write_operations.cc:271: Failure
2023-03-29T09:45:51.704 INFO:tasks.workunit.client.0.smithi161.stdout:   api_c_write_operations: Expected equality of these values:
2023-03-29T09:45:51.704 INFO:tasks.workunit.client.0.smithi161.stdout:   api_c_write_operations:   -4095 - 1
2023-03-29T09:45:51.704 INFO:tasks.workunit.client.0.smithi161.stdout:   api_c_write_operations:     Which is: -4096
2023-03-29T09:45:51.704 INFO:tasks.workunit.client.0.smithi161.stdout:   api_c_write_operations:   rados_write_op_operate(op, ioctx, s.c_str(), __null, LIBRADOS_OPERATION_RETURNVEC)
2023-03-29T09:45:51.704 INFO:tasks.workunit.client.0.smithi161.stdout:   api_c_write_operations:     Which is: -4095
2023-03-29T09:45:51.704 INFO:tasks.workunit.client.0.smithi161.stdout:   api_c_write_operations: [  FAILED  ] LibRadosCWriteOps.CmpExt (3412 ms)

There are several more instances on this link: https://pulpito.ceph.com/yuriw-2023-03-28_22:45:07-rados-wip-yuri7-testing-2023-03-28-0942-distro-default-smithi/

@ljflores ljflores removed the needs-qa label Mar 31, 2023
@NitzanMordhai
Copy link
Contributor Author

NitzanMordhai commented Apr 24, 2023

@ljflores sorry, missed that notification, since i changed the file target the offset was changed and thats the reason we are seeing -4096 and not -4095, going to change that.

@NitzanMordhai
test/c_write_operations: set RETURNVEC on expected cmpext failuer
7fa4f28 
We are sending cmpext request that return error and rval error.
In order to get reliably rval return for cmpext, we need to set RETURNVEC
flag so we record in pg log in case of reconnect during socket injection.
without that flag the re-request of the op won't have the rval set
and we will end with unmatch error.

Fixes: https://tracker.ceph.com/issues/21592
Signed-off-by: Nitzan Mordechai <nmordec@redhat.com>
@NitzanMordhai NitzanMordhai force-pushed the wip-nitzan-c-write-operations-inject-sockert branch from 80a531c to 7fa4f28 Compare April 24, 2023 07:49
@ljflores
Copy link
Member

ljflores commented May 18, 2023

Rados suite review: https://tracker.ceph.com/projects/rados/wiki/MAIN#httpstrellocomclM1xjBe01744-wip-yuri-testing-2023-05-10-0917

@yuriw yuriw merged commit 7e0522f into ceph:main May 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rzarzynski rzarzynski rzarzynski approved these changes

Assignees

No one assigned

Labels

core needs-qa tests wip-yuri-testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@NitzanMordhai @ljflores @rzarzynski @yuriw @neha-ojha