This add unit tests in order to cover the code present in `./objectstore/`
This also updates current unit tests with respect to the recent changes
and fixes some tests that were already broken.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

- update unit test devices/raw/test_list.py:
  e5e4296 missed the corresponding unit test update.
  This commit updates it.

- fix unit tests when run as non-root user:
  When unit tests are run as non-root user, it fails for many reasons
  such as 'permission denied' etc...
  This commit addresses that by adding missing fixture `is_root`
  and using pyfakefs when it's needed.

- drop test_writer_uses_log_on_unicodeerror test:
  This test is broken for a while. Since we don't make test against py2,
  let's drop it.

- test_arg_validators cleanup:
  This makes use of `self.validator` defined in `setup_method()`
  rather than redefining a new object in each tests.
  Also, this migrates test_files_are_missing to pyfakefs

- test_migrate cleanup:
  This makes use of `is_root` fixture insead of using mock.patch
  on `os.getuid`

- test_activate cleanup:
    - removes the fixture monkeypatch from test_no_data_uuid() given
      that it's not used in this test.
    - remove the fixture is_root from test_activate_all()

- simple.TestActivate.test_no_data_uuid fix:
  This fixes the following issue:

  ```
  ________________________ TestActivate.test_no_data_uuid ________________________

  self = <test_activate.TestActivate object at 0x7f6b1885f1c0>
  factory = <class 'ceph_volume.tests.conftest.Factory'>, is_root = None
  capture = <ceph_volume.tests.conftest.Capture object at 0x7f6b17f59340>
  fake_filesystem = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f6b17125190>

      def test_no_data_uuid(self, factory, is_root, capture, fake_filesystem):
          fake_filesystem.create_file('/tmp/json-config', contents='{}')
          args = factory(osd_id='0', osd_fsid='1234', json_config='/tmp/json-config')
          with pytest.raises(RuntimeError):
  >           activate.Activate([]).activate(args)

  /home/jenkins-build/build/workspace/ceph-volume-pr/src/ceph-volume/ceph_volume/tests/devices/simple/test_activate.py:12:
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  a = (<ceph_volume.devices.simple.activate.Activate object at 0x7f6b16fcd8b0>, <ceph_volume.tests.conftest.Factory object at 0x7f6b16fcd160>)
  kw = {}

      @wraps(func)
      def is_root(*a, **kw):
          if not os.getuid() == 0 and not os.environ.get('CEPH_VOLUME_SKIP_NEEDS_ROOT', False):
  >           raise exceptions.SuperUserError()
  E           ceph_volume.exceptions.SuperUserError: This command needs to be executed with sudo or as root

  /home/jenkins-build/build/workspace/ceph-volume-pr/src/ceph-volume/ceph_volume/decorators.py:15: SuperUserError
  ```

  Even though we use the fixture `is_root`, it doesn't seem to work.
  Using @patch() instead fixes this issue.

- address 'PytestRemovedIn8Warning' messages:
  pretty self-explanatory:
  ```
  PytestRemovedIn8Warning: Support for nose tests is deprecated and will be removed in a future release.
  ```

  ```
  To remove this warning, rename it to `setup_method(self)`
  ```

  so this commit renames some `def setup(self)` to `def setup_method(self)` as suggested.

- add missing unit tests:
  This adds some unit test updates missed by commits
  0985e20 and bd5e1a8

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

This is causing issues in ceph-volume unit tests (Still some details
not very clear so I can't explain for sure why for now).
Let's run the tests serially for now until I figure out what's wrong.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

When run make check(arm64) on omani series servers, eg.
https://jenkins.ceph.com/job/ceph-pull-requests-arm64/54751/.
Shows:
python: timeout after 30 seconds
INFO     root:call_wrappers.py:284 Non-zero exit code 124 from /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/cephadm/.tox/py3/bin/python -c for i in range(1000000): print(i, flush=True)

Fixes: https://tracker.ceph.com/issues/65355

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Instruct readers to use "mkdir /mnt/cephfs1" to create a mountpoint
before using "ceph-fuse" to mount a filesystem, if "/mnt/cephfs1"
doesn't already exist. cf.
ceph#56831 (comment)

Signed-off-by: Zac Dover <zac.dover@proton.me>

Remove what I think are vestigial configuration parameters setting
`basepython` in tox.ini. These may have been useful during the change
from python 2 to 3, but I don't see them serving any useful purpose
at this current time. In fact they now interfere with the ability to
globally override the basepython version of all testenv's at once.
So let's just remove them. We can always add it back if we find
an issue in the future (and document it!).

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Remove what I think are vestigial configuration parameters setting
`basepython` in tox.ini. These may have been useful during the change
from python 2 to 3, but I don't see them serving any useful purpose
at this current time. In fact they now interfere with the ability to
globally override the basepython version of all testenv's at once.
So let's just remove them. We can always add it back if we find
an issue in the future (and document it!).

Signed-off-by: John Mulligan <jmulligan@redhat.com>

This is specifically being added with the orch/cephadm suite
in mind, where coming up with a viable ignorelist has proved
difficult. The orch testing does a lot of actions that can
cause thigns like an OSD or MON daemon to be down very
briefly, and I've found the vast majority of the time we
really don't want to fail the test when these pop up as cephadm
testing really only benefits from catching the CEPHADM_ errors/
warnings rather than eveyr possible one. Rather than continuing to
play whack-a-mole with the errors in the logs, this
patch should allow us to limit what we fail on to at
least get the suite in a good spot again. We can always
phase out the uses of this new "log-only_match" option
later in a more controlled way, and adding it shouldn't
affect log scraping for any of the tests that aren't
facing a similar issue.

Signed-off-by: Adam King <adking@redhat.com>

Rather than failing for any instance of
[ERR], [WRN], or [SEC]. The orch/cephadm suite
does a lot of stuff that can cause these various
warnings to breifly appear. Trying to catch all
cases has been difficult and the suite has been
red for some time. This patch makes it so it
instead only matches log messages that
include CEPHADM_ on top of having [ERR], [WRN],
or [SEC] as those warnings have been the ones
that have actually lead us to cephadm bugs, while
the others are pretty much always just noise in
these tests. This patch does not apply this
to the mds_upgrade_sequence, nfs, or rbd-iscsi
sections as those are symlinked from other suites
and I didn't want to affect those suites tests
directly with this change.

Signed-off-by: Adam King <adking@redhat.com>

Check if the PROMETHEUS_API_HOST value has the ip addr of the host if
not replace hostnmame with ip addr

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

When sanitizer is enabled, unittest_osdscrub shows

```
=================================================================
==1633952==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 28 byte(s) in 1 object(s) allocated from:
    #0 0xaaaab4e108e0 in malloc (/root/ceph/build/bin/unittest_osdscrub+0x1ed08e0) (BuildId: b3cfa2137be96d75535beecf0f2500cec10c7550)
    #1 0xffffa8cac2f8 in __res_context_send resolv/./resolv/res_send.c:334:9
    #2 0xffffa8ca9c54 in __res_context_query resolv/./resolv/res_query.c:216:6
    #3 0xffffa8caa4a8 in __res_context_querydomain resolv/./resolv/res_query.c:625:9
    #4 0xffffa8caa4a8 in __res_context_search resolv/./resolv/res_query.c:381:9
    #5 0xffffa8caaa20 in context_search_common resolv/./resolv/res_query.c:550:16
    #6 0xffffa8caaa20 in res_nsearch resolv/./resolv/res_query.c:563:10
    #7 0xffffabbf1f64 in ceph::ResolvHWrapper::res_nsearch(__res_state*, char const*, int, int, unsigned char*, int) /root/ceph/src/common/dns_resolve.cc:37:10
    #8 0xffffabbf6574 in ceph::DNSResolver::resolve_srv_hosts(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::DNSResolver::SRV_Protocol, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::DNSResolver::Record, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::DNSResolver::Record> > >*) /root/ceph/src/common/dns_resolve.cc:295:19
    #9 0xffffac8edaf0 in MonMap::init_with_dns_srv(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, std::ostream&) /root/ceph/src/mon/MonMap.cc:935:36
    #10 0xffffac8eeec8 in MonMap::build_initial(ceph::common::CephContext*, bool, std::ostream&) /root/ceph/src/mon/MonMap.cc:1014:20
    #11 0xffffac85beb0 in MonClient::build_initial_monmap() /root/ceph/src/mon/MonClient.cc:93:18
    #12 0xaaaab4e50d98 in TestOSDScrub_scrub_time_permit_Test::TestBody() /root/ceph/src/test/osd/TestOSDScrub.cc:73:6
    #13 0xaaaab4f655b0 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10
    #14 0xaaaab4f16264 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14
    #15 0xaaaab4ec6ca8 in testing::Test::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2680:5
    #16 0xaaaab4ec8bec in testing::TestInfo::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2858:11
    #17 0xaaaab4eca1ec in testing::TestSuite::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:3012:28
    #18 0xaaaab4ee5fb0 in testing::internal::UnitTestImpl::RunAllTests() /root/ceph/src/googletest/googletest/src/gtest.cc:5723:44
    #19 0xaaaab4f6f4c4 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10
    #20 0xaaaab4f1d4bc in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14
    #21 0xaaaab4ee5428 in testing::UnitTest::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:5306:10
    #22 0xaaaab4e4b790 in RUN_ALL_TESTS() /root/ceph/src/googletest/googletest/include/gtest/gtest.h:2486:46
    #23 0xaaaab4e49dbc in main /root/ceph/src/test/unit.cc:45:10
    #24 0xffffa8bc73f8 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #25 0xffffa8bc74c8 in __libc_start_main csu/../csu/libc-start.c:392:3
    #26 0xaaaab4d9972c in _start (/root/ceph/build/bin/unittest_osdscrub+0x1e5972c) (BuildId: b3cfa2137be96d75535beecf0f2500cec10c7550)

-----------------------------------------------------
Suppressions used:
  count      bytes template
      1         45 ^MallocExtension::Initialize
-----------------------------------------------------

SUMMARY: AddressSanitizer: 28 byte(s) leaked in 1 allocation(s).
```

1. 'res_ninit/res_nquery' memory should be freed.

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

2. free MonClient using Monclient::shutdown

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

Remove a GPG key that was lost in a server upgrade, and remove the text
that describes it.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Removed "minutely" from retention policy dropdown

Fixes: https://tracker.ceph.com/issues/65493

Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Fixes: https://tracker.ceph.com/issues/65499

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Fixes https://tracker.ceph.com/issues/65207

- Moves "features" section in rbd image create form to "Advanced" section.
- makes rbd configuration section to be expanded by default rather than
  being collapsed as it has only single section. This will improve user experience as it will not
require two clicks.
- updates e2e test

Signed-off-by: Afreen <afreen23.git@gmail.com>

Fixes: https://tracker.ceph.com/issues/64970
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Fixes: https://tracker.ceph.com/issues/65271
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

As the thrash tests were introduced, some options were disabled
until the tests are stabilized.

Re-enable chance_down option (default is 0.4) to detect bugs on restart.

Since it will probably take few iterations before thrash and recovery tests ('default.yaml')
will pass successfully, add anoter 'simple.yaml' which should remain stable.

Fixes: https://tracker.ceph.com/issues/65130

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

… phase

If exit/unlock the barrier before entering the next phase, it is
possible for the next request to exit the barrier at the same time, and
enters the next phase first, causing reorder issues.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Fixes https://tracker.ceph.com/issues/65506

Rgw roles list page is crashing due to these new fields coming from api but not supported in dashboard frontend.

This will also fix failing CI tests.

Signed-off-by: Afreen <afreen23.git@gmail.com>

Parameter "timeout" is accepted by LocalRemote.run() but the method
doesn't do anything about it besides accepting it. Thus, this parameter
has no effect.

In LocalRemote.run(), pass this parameter to LocalRemoteProcess.wait()
and from this method pass it to subprocess.Popen.communicate(). Thus,
command will be terminated by subprocess module at seconds specified by
"timeout" parameter. IOW, "timeout" parameter will have an effect.

Fixes: https://tracker.ceph.com/issues/65533
Signed-off-by: Rishabh Dave <ridave@redhat.com>

This issue was exposed by ceph API test failure. Link to this failure - https://jenkins.ceph.com/job/ceph-api/72373/

Copying traceback below from here https://jenkins.ceph.com/job/ceph-api/72373/consoleFull#-2010121386c212b007-e891-4176-9ee7-2f60eca393b7 -

2024-04-15 12:32:34,808.808 INFO:__main__:> ../src/vstart.sh -n
--nolockdep
Using guessed paths
/home/jenkins-build/build/workspace/ceph-api/build/lib/
['/home/jenkins-build/build/workspace/ceph-api/qa',
'/home/jenkins-build/build/workspace/ceph-api/build/lib/cython_modules/lib.3',
'/home/jenkins-build/build/workspace/ceph-api/src/pybind']
Traceback (most recent call last):
  File
"/home/jenkins-build/build/workspace/ceph-api/build/../qa/tasks/vstart_runner.py",
line 1552, in <module>
    exec_test()
  File
"/home/jenkins-build/build/workspace/ceph-api/build/../qa/tasks/vstart_runner.py",
line 1402, in exec_test
    remote.run(args=args, env=vstart_env, timeout=(3 * 60))
  File
"/home/jenkins-build/build/workspace/ceph-api/build/../qa/tasks/vstart_runner.py",
line 452, in run
    return self._do_run(**kwargs)
  File
"/home/jenkins-build/build/workspace/ceph-api/build/../qa/tasks/vstart_runner.py",
line 491, in _do_run
    proc.wait(timeout)
  File
"/home/jenkins-build/build/workspace/ceph-api/build/../qa/tasks/vstart_runner.py",
line 252, in wait
    out, err = self.subproc.communicate(timeout=timeout)
  File "/usr/lib/python3.10/subprocess.py", line 1152, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.10/subprocess.py", line 2004, in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
  File "/usr/lib/python3.10/subprocess.py", line 1196, in _check_timeout
    raise TimeoutExpired(
subprocess.TimeoutExpired: Command '../src/vstart.sh -n --nolockdep'
timed out after 180 seconds

Signed-off-by: Rishabh Dave <ridave@redhat.com>

The function is typically invoked on client errors like NoSuchBucket. Logging these errors with level 1 may initially suggest a significant issue, when in fact it's just a client error. Consider raising the logging level to 20 for better clarity.

Fixes: https://tracker.ceph.com/issues/65469
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Some versions of lvm put a dot at the end.

Fixes: https://tracker.ceph.com/issues/65481
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

It happens implicitly on Ubuntu, but not on CentOS Stream 8.

Fixes: https://tracker.ceph.com/issues/65481
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Following clang-tidy's advice.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

using std::ssize() to avoid a narrowing conversion when using
the size of a container as a parameter to a function expecting
an int64_t.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Otherwise you cannot use LocalDaemon.check_status.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

This mirrors teuthology and makes it possible to check the exit status of a
daemon.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

That autocommit is properly turned off and that commits via context managers
work as expected.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

SQLite3 and python's binding were both committing transactions at unintended
points. Turn it off and stop using executescript.

Fixes: https://tracker.ceph.com/issues/65494
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

We call back into be_scan_list repeatedly for the same object in deep
scrub if the object is too large/has too many omap entries to be done
in a single call.  This causes us to wastefully call
ObjectStore::getattrs and stat multiple times.

This has been a long standing, but mostly harmless bug (aside from the
extra metadata traffic).  However, 1a4d3f0, between reef and squid,
switched ScrubMap::object::attrs to a map<string, bufferlist> from a
map<string, bufferptr>.  This should have been harmless, except that
the ObjectStore::getattrs overload for that type uses

      aset[i->first].append(i->second);

rather than

      aset.emplace(i.first.c_str(), i.second);

to populate the map.  Combined with this bug, that means that if we
need 7 calls into be_scan_list to deep scrub an object, we'll end up
with attributes concatenated 7 times each.  This didn't cause visible
problems with squid/main testing since all of the replicas would end
up doing the same thing and they'd still decode ok, but it did become
visible during reef->squid upgrade testing as the older osds sent
maps with the correct contents.

The next commit will fix ObjectStore::getattrs to clear the map.

Fixes: https://tracker.ceph.com/issues/65185
Signed-off-by: Samuel Just <sjust@redhat.com>

Passing in a non-empty map would otherwise exhibit quite unexpected
behavior.  For the bufferptr overload, any preexisting entries would
not be overwritten due to how std::map::emplace behaves.  For the
bufferlist overload, it would result in appending to any pre-existing
entries.

The prior commit cleans up one such inadvertent caller which resulted
in the below bug.

Fixes: https://tracker.ceph.com/issues/65185
Signed-off-by: Samuel Just <sjust@redhat.com>

…rams_t::user_modify

Before this change OpsExecuter::user_modify was maintained in OpsExecuter::do_write_op.
However, osd_op_params->user_modify was not updated when used in OpsExecuter::prepare_transaction

Fixes: https://tracker.ceph.com/issues/65672
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Not all commands return JSON, like `dirfrag split`.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Reproduces: https://tracker.ceph.com/issues/65603
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Credit to Leonid for first noticing this.

Fixes: https://tracker.ceph.com/issues/65603
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

This is an optimization to obviate repeated calls to acquire_locks.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

And handle inode becoming quiesced after op is created.

Fixes: https://tracker.ceph.com/issues/65603
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Fixes: https://tracker.ceph.com/issues/65698
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Rendering the dashboards with showMultiCluster=True allows for
them to work with multiple clusters storing their metrics in a single
Prometheus instance. This works via the cluster label and that functionality
already existed. This just fixes some inconsistencies in applying the label
filters.

Additionally this contains updates to the tests to have them succeed with
with both configurations and avoid the introduction of regressions in
regards to multiCluster in the future.

There also are some consistency cleanups here and there:
 * `datasource` was not used consistently
 * `cluster` label_values are determined from `ceph_health_status`
 * `job` template and filters on this label were removed to align multi cluster
    support solely via the `cluster` label
 * `ceph_hosts` filter now uses label_values from any ceph_metadata metrici
    to now show all instance values, but those of hosts with some Ceph
    component / daemon.
 *  Enable showMultiCluster=True since `cluster` label is now always present,
    via ceph#54964

Improves: https://tracker.ceph.com/issues/64321
Signed-off-by: Christian Rohmann <christian.rohmann@inovex.de>

Fixes: https://tracker.ceph.com/issues/65218

Signed-off-by: cloudbehl <cloudbehl@gmail.com>

Otherwise it is impossible to make rbd snapshots

Fixes: https://tracker.ceph.com/issues/65691

Signed-off-by: Adam King <adking@redhat.com>

As stated in the commit-message line, this corrects the typo "cepg" to
the correct string "ceph".

This typo was discovered by https://github.com/test-erik and this was
brought to our attention way back in
ceph#50420.

Signed-off-by: Zac Dover <zac.dover@proton.me>

…or ECANCELED(ConcurrentModification) errors.

An ECANCELED error coming from our write of the bucket instance metadat is a common error for metadata writes on secondary zones, because secondary write races with metadata sync from the write that is forwarded to the master zone

Signed-off-by: kchheda3 <kchheda3@bloomberg.net>

Bumps [@applitools/eyes-cypress](https://github.com/applitools/eyes.sdk.javascript1/tree/HEAD/js/packages/eyes-cypress) from 3.22.5 to 3.44.4.
- [Commits](https://github.com/applitools/eyes.sdk.javascript1/commits/HEAD/js/packages/eyes-cypress)

---
updated-dependencies:
- dependency-name: "@applitools/eyes-cypress"
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>