Skip to content

qa/rgw: add multisite suite to configure and run multisite tests#14688

Merged
cbodley merged 14 commits intoceph:masterfrom
cbodley:wip-rgw-multi-suite
May 19, 2017
Merged

qa/rgw: add multisite suite to configure and run multisite tests#14688
cbodley merged 14 commits intoceph:masterfrom
cbodley:wip-rgw-multi-suite

Conversation

@cbodley
Copy link
Contributor

@cbodley cbodley commented Apr 20, 2017

work in progress for a multisite test suite in teuthology

TODO:

@cbodley
Copy link
Contributor Author

cbodley commented May 2, 2017

three-zone tests are currently failing, testing #14915 as a potential fix

@cbodley cbodley force-pushed the wip-rgw-multi-suite branch from 4e1684e to 9b588f2 Compare May 5, 2017 04:05
@cbodley cbodley force-pushed the wip-rgw-multi-suite branch 2 times, most recently from 17c4e13 to 753071a Compare May 15, 2017 20:29
log.info('Removing apache config...')
for client in clients_to_create_as:
cluster_name, daemon_type, client_id = teuthology.split_role(client)
client_with_cluster = '.'.join((cluster_name, daemon_type, client_id))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add some logging?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what would you like to see logged here? just that it's getting the right value for client_with_cluster? here's a snippet from a teuthology.log that shows this in action:

INFO:tasks.rgw.client.0:Stopped
INFO:teuthology.orchestra.run.mira114:Running: 'rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock'
INFO:tasks.rgw:Stopping apache...
INFO:tasks.rgw:Removing apache config...
INFO:teuthology.orchestra.run.mira114:Running: 'rm -f /home/ubuntu/cephtest/apache/apache.ceph.client.0.conf && rm -f /home/ubuntu/cephtest/apache/htdocs.ceph.client.0/rgw.fcgi'
INFO:tasks.rgw:Cleaning up apache directories...
INFO:teuthology.orchestra.run.mira114:Running: 'rm -rf /home/ubuntu/cephtest/apache/tmp.ceph.client.0 && rmdir /home/ubuntu/cephtest/apache/htdocs.ceph.client.0'
INFO:teuthology.orchestra.run.mira114:Running: 'rmdir /home/ubuntu/cephtest/apache'
DEBUG:teuthology.run_tasks:Unwinding manager ceph

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted the cluster name and client id.
I see from the logs the client id is already logged and I can find out what is cluster name from the teuthology node.
No need to add more logging

# separate region info so only clients are keys in config
regions = config['regions']
del config['regions']
regions = config.pop('regions', {})
Copy link
Member

@oritwas oritwas May 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe it is time to change it to zonegroups :)
It can be in a separate pr ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, i have a https://github.com/cbodley/ceph/commits/wip-rgw-agent-cleanup branch that removes this stuff

return period, realm_epoch, num_shards, markers

def meta_sync_status(zone):
for _ in range(60):
Copy link
Member

@oritwas oritwas May 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a comment why 60?
I see it used a few times maybe a const?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, i just settled on a total of 5 minutes. i'll make this configurable along with the sleep in set_master_zone()

zonegroup.master_zone = zone
# wait for reconfiguration, so that later metadata requests go to the new master
time.sleep(5)
time.sleep(15)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe it should be configurable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea! then we can keep it at 5 for test_multi.py, and raise it as much as we need to for teuthology

@cbodley cbodley force-pushed the wip-rgw-multi-suite branch 3 times, most recently from 1907d99 to b18c20a Compare May 16, 2017 15:01
cbodley added 7 commits May 17, 2017 14:48
also removes unnecessary tuple parens

Signed-off-by: Casey Bodley <cbodley@redhat.com>
this makes the 'compression type' setting global to all gateways, and
makes the setting visible to other tasks in ctx.rgw.compression_type

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley cbodley force-pushed the wip-rgw-multi-suite branch from b18c20a to b39b314 Compare May 17, 2017 18:49
cbodley added 7 commits May 18, 2017 13:37
Signed-off-by: Casey Bodley <cbodley@redhat.com>
if multisite tests are going to run in teuthology, they can't loop forever

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
changes default level from info to debug

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley cbodley force-pushed the wip-rgw-multi-suite branch from b39b314 to de836ee Compare May 18, 2017 17:38
@cbodley
Copy link
Contributor Author

cbodley commented May 18, 2017

the latest run of the full rgw suite with --subset=1/6 --filter frontend/civetweb.yaml was all green: http://pulpito.ceph.com/cbodley-2017-05-17_14:49:58-rgw-master---basic-smithi/

the rgw:multisite suite pretty much always fails with valgrind errors (and osd crashes on shutdown), but the multisite tests themselves are passing:
http://pulpito.ceph.com/cbodley-2017-05-16_10:46:23-rgw:multisite-master---basic-smithi/

INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
INFO:tasks.rgw_multisite_tests:Ran 11 tests in 921.748s
INFO:tasks.rgw_multisite_tests:
INFO:tasks.rgw_multisite_tests:OK

@cbodley cbodley changed the title [DNM] qa/rgw: add multisite suite to configure and run multisite tests qa/rgw: add multisite suite to configure and run multisite tests May 18, 2017
@cbodley cbodley merged commit cff53b2 into ceph:master May 19, 2017
@cbodley cbodley deleted the wip-rgw-multi-suite branch May 19, 2017 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants