src/test: Wip rgw bilog autotrim tests by TRYTOBE8TME · Pull Request #45053 · ceph/ceph

TRYTOBE8TME · 2022-02-16T10:32:40Z

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

TRYTOBE8TME · 2022-02-16T10:34:17Z

Sorry, will have to close PR #44758 because of mistake done in rebasing.

cbodley

i'm not sure why there are two separate test cases here?

cbodley · 2022-02-16T15:54:00Z

src/test/rgw/rgw_multi/tests.py

+     # bilog showing objects after reshard
+     zone.zone.cluster.admin(['bilog', 'list',
+         '--bucket', test_bucket.name,
+         '--gen=1'])


thanks, these tests should be explicit about --gen when listing bilogs. can you please update the bilog_list() function to accept an optional gen argument, instead of making the raw call to cluster.admin() here?

and instead of hard-coding --gen=1 here, we should be using log generation numbers that we saw in bucket layout output

Got it! I'll update this.

cbodley · 2022-02-16T16:02:50Z

src/test/rgw/rgw_multi/tests.py

+    print('1.')
+    print(json.loads(first_check))


ok, this at least prints the bucket layout output. but we really want this test to verify that the output matches the expectations i described in #44758 (comment)

so here, before the initial reshard, we should verify that the json array in logs only has a single entry

after the reshard, you can check that it has a second entry, etc

cbodley · 2022-02-17T21:20:51Z

src/test/rgw/rgw_multi/tests.py

    cmd = ['bilog', 'list', '--bucket', bucket] + (args or [])
    cmd += ['--tenant', config.tenant, '--uid', user.name] if config.tenant else []
+    if gen:
+        cmd += ['--gen', gen]


sorry, i didn't realize this already took args = None. instead of adding another gen = None, i think it's easier for callers to just pass that as args = ['--gen', gen]. does that make sense?

Yup, we can use that too. But wouldn't keeping gen as a separate parameter be good choice for the future tests?

i guess it depends on what we change in the future. if we just add more arguments, it's easier to pass those in args too instead of changing this bilog_list() helper function

cbodley · 2022-02-21T18:49:09Z

src/test/rgw/rgw_multi/tests.py

+    # verify the bucket has empty bilog
+    test_bilog = bilog_list(zone.zone, test_bucket.name)
+    assert(len(test_bilog) == 0)


okay; radosgw-admin bilog list without a --gen will list the most recent generation, which is what we want to do here

however - the only time this test writes objects is during generation 0, before any reshards happen. that means that we won't write any bilog entries to later generations, so this call to bilog_list() doesn't actually verify that bilog_autotrim() did anything to this last bilog generation

can you please add some extra objects after the last reshard so that the last bilog generation isn't empty before that bilog_autotrim()?

cbodley · 2022-02-21T18:50:49Z

can you help me understand what the two separate test cases are testing? it seems like we could just combine them into a single test

TRYTOBE8TME · 2022-02-23T08:02:21Z

can you help me understand what the two separate test cases are testing? it seems like we could just combine them into a single test

So, basically what I'm thinking is test_bucket_reshard_index_log_trim_checking mainly focuses on bucket layout logs adding followed by bilog autotrim which deletes the log entries. And the second test test_bucket_reshard_index_log_trim mainly focuses on using the --gen parameter exclusively. My main focus was to address both things/topics separately through different tests.

If you want we can combine these into one or should we keep it separate?

cbodley · 2022-02-23T14:06:47Z

If you want we can combine these into one or should we keep it separate?

if the tests are performing the same operations (uploads, reshards) but checking different things (bucket layout, bi list), they should just be one test that does all of those checks

TRYTOBE8TME · 2022-04-11T18:30:09Z

@cbodley and @adamemerson can any one of you please do a final review?

cbodley

looks good, it's passing for me. can you please squash the commits? then i'll merge it into wip-rgw-multisite-reshard

This test includes checking radosgw-admin bucket layout command along with bilog autrotrim on a resharded bucket. Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

cbodley · 2022-04-14T15:34:52Z

thanks @TRYTOBE8TME, nice work. merged into wip-rgw-multisite-reshard

cbodley · 2022-04-26T19:07:53Z

seeing this test fail a lot on wip-rgw-multisite-reshard:

======================================================================
FAIL: test_multi.test_bucket_reshard_index_log_trim
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/cbodley/ceph/build/venv/lib/python3.10/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/cbodley/ceph/build/venv/lib/python3.10/site-packages/nose/util.py", line 620, in newfunc
    return func(*arg, **kw)
  File "/home/cbodley/ceph/src/test/rgw/rgw_multi/tests.py", line 1389, in test_bucket_reshard_index_log_trim
    assert(len(json_obj_5['layout']['logs']) == 1)
AssertionError:
-------------------- >> begin captured logging << --------------------



rgw_multi.tests: DEBUG: running cmd: /home/cbodley/ceph/src/test/rgw/test-rgw-call.sh call_rgw_admin c1 bilog autotrim --debug-rgw=0 --debug-ms=0
rgw_multi.tests: DEBUG: command returned status=0 stdout=
rgw_multi.tests: DEBUG: running cmd: /home/cbodley/ceph/src/test/rgw/test-rgw-call.sh call_rgw_admin c1 bucket layout --bucket wvwdhp-45 --debug-rgw=0 --debug
-ms=0
rgw_multi.tests: DEBUG: command returned status=0 stdout={                                                                                          [27/42036]
    "layout": {
        "resharding": "None",
        "current_index": {
            "gen": 2,
            "layout": {
                "type": "Normal",
                "normal": {
                    "num_shards": 3,
                    "hash_type": "Mod"
                }
            }
        },
        "logs": [
            {
                "gen": 1,
                "layout": {
                    "type": "InIndex",
                    "in_index": {
                        "gen": 1,
                        "layout": {
                            "num_shards": 3,
                            "hash_type": "Mod"
                        }
                    }
                }
            },
            {
                "gen": 2,
                "layout": {
                    "type": "InIndex",
                    "in_index": {
                        "gen": 2,
                        "layout": {
                            "num_shards": 3,
                            "hash_type": "Mod"
                        }
                    }
                }
            }
        ]
    }
}

TRYTOBE8TME · 2022-04-27T13:23:41Z

seeing this test fail a lot on wip-rgw-multisite-reshard:

I'll take a look at it.

cbodley · 2022-05-05T17:48:36Z

i rebuilt the latest wip-rgw-multisite-reshard today, and haven't been able to reproduce that failure ¯\_(ツ)_/¯

TRYTOBE8TME requested a review from cbodley February 16, 2022 10:32

github-actions bot added rgw tests labels Feb 16, 2022

TRYTOBE8TME changed the title ~~Wip rgw bilog autotrim test~~ src/test: Wip rgw bilog autotrim tests Feb 16, 2022

cbodley reviewed Feb 16, 2022

View reviewed changes

TRYTOBE8TME force-pushed the wip-rgw-bilog-autotrim-test branch from 60f910d to e96b920 Compare February 17, 2022 20:43

TRYTOBE8TME requested a review from cbodley February 17, 2022 20:45

TRYTOBE8TME marked this pull request as ready for review February 18, 2022 07:19

cbodley reviewed Feb 21, 2022

View reviewed changes

TRYTOBE8TME force-pushed the wip-rgw-bilog-autotrim-test branch from e96b920 to 7363681 Compare February 21, 2022 18:16

cbodley reviewed Feb 21, 2022

View reviewed changes

TRYTOBE8TME mentioned this pull request Feb 23, 2022

src/test: Test case for bilog trim across reshard event #44758

Closed

14 tasks

TRYTOBE8TME force-pushed the wip-rgw-bilog-autotrim-test branch from 7363681 to faab6a7 Compare February 23, 2022 07:08

TRYTOBE8TME force-pushed the wip-rgw-bilog-autotrim-test branch from faab6a7 to 539d3a3 Compare February 24, 2022 19:31

TRYTOBE8TME requested a review from cbodley February 24, 2022 19:32

TRYTOBE8TME force-pushed the wip-rgw-bilog-autotrim-test branch from 539d3a3 to 60b47b2 Compare February 24, 2022 19:34

TRYTOBE8TME force-pushed the wip-rgw-bilog-autotrim-test branch from 60b47b2 to 3a1b604 Compare April 8, 2022 05:46

cbodley approved these changes Apr 12, 2022

View reviewed changes

src/test: Addition of new bilog trim test

35737fa

This test includes checking radosgw-admin bucket layout command along with bilog autrotrim on a resharded bucket. Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

TRYTOBE8TME force-pushed the wip-rgw-bilog-autotrim-test branch from 3a1b604 to 35737fa Compare April 14, 2022 08:32

cbodley closed this Apr 14, 2022

Conversation

TRYTOBE8TME commented Feb 16, 2022

Checklist

Uh oh!

TRYTOBE8TME commented Feb 16, 2022

Uh oh!

cbodley left a comment

Choose a reason for hiding this comment

Uh oh!

cbodley Feb 16, 2022

Choose a reason for hiding this comment

Uh oh!

TRYTOBE8TME Feb 16, 2022

Choose a reason for hiding this comment

Uh oh!

cbodley Feb 16, 2022

Choose a reason for hiding this comment

Uh oh!

cbodley Feb 17, 2022

Choose a reason for hiding this comment

Uh oh!

TRYTOBE8TME Feb 21, 2022

Choose a reason for hiding this comment

Uh oh!

cbodley Feb 21, 2022

Choose a reason for hiding this comment

Uh oh!

cbodley Feb 21, 2022

Choose a reason for hiding this comment

Uh oh!

cbodley commented Feb 21, 2022

Uh oh!

TRYTOBE8TME commented Feb 23, 2022

Uh oh!

cbodley commented Feb 23, 2022

Uh oh!

TRYTOBE8TME commented Apr 11, 2022

Uh oh!

cbodley left a comment

Choose a reason for hiding this comment

Uh oh!

cbodley commented Apr 14, 2022

Uh oh!

cbodley commented Apr 26, 2022

Uh oh!

TRYTOBE8TME commented Apr 27, 2022

Uh oh!

cbodley commented May 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants