KAFKA-9123 Test a large number of replicas#7621
Conversation
…of one test as a trial
hachikuji
left a comment
There was a problem hiding this comment.
Thanks, left a few small comments.
| kafka_topic_script = self.path.script("kafka-topics.sh", node) | ||
|
|
||
| cmd = kafka_topic_script + " " | ||
| cmd += "--zookeeper %(zk_connect)s --delete --topic %(topic)s " % { |
There was a problem hiding this comment.
Perhaps we may as well use --bootstrap-server since delete_topic is a new addition?
| @cluster(num_nodes=12) | ||
| @parametrize(topic_count=500, partition_count=34, replication_factor=3) | ||
| def test_produce_consume(self, topic_count, partition_count, replication_factor): | ||
| t0 = time.time() |
There was a problem hiding this comment.
nit: maybe use meaningful names? e.g. topic_creation_start
Even better would be to add some kind of timed function
There was a problem hiding this comment.
Do we have any timer utilities in the python tests?
There was a problem hiding this comment.
I don't know of any, but I haven't checked. I thought it might be straightforward to add one, but it's up to you.
| "replication-factor": replication_factor, | ||
| "configs": {"min.insync.replicas": 1} | ||
| } | ||
| self.kafka.create_topic(topic_cfg, describe=False) |
There was a problem hiding this comment.
Not for this patch, but we should do a KIP to add support for batch topic creation.
hachikuji
left a comment
There was a problem hiding this comment.
LGTM, assuming tests pass. Just had one question.
| for i in range(topic_count): | ||
| topic = "topic-%04d" % i | ||
| self.logger.info("Deleting topic %s" % topic) | ||
| self.kafka.delete_topic(topic) |
There was a problem hiding this comment.
Adding batch deletion here would be useful also. This has caused problems in Kafka previously.
There was a problem hiding this comment.
Yes, definitely. Exposing the batch create/delete through kafka-topics.sh would be nice and help out a lot in the tests (since our kafka fixture uses the CLIs).
Two tests using 50k replicas on 8 brokers: * Do a rolling restart with clean shutdown, delete topics * Run produce bench and consumer bench on a subset of topics Reviewed-By: David Jacot <djacot@confluent.io>, Vikas Singh <vikas@confluent.io>, Jason Gustafson <jason@confluent.io>
This PR adds two new system tests to exercise the system with a large number of replicas.
The first test creates 500 topics with 34 partitions and x3 replication for a total of 51,000 replicas. This is done across 8 brokers which is 6375 replicas per broker. Once the topics have been created and verified, a controlled shutdown of each broker is performed. Finally, each of the topics is deleted.
The other test runs produce and consume benchmark utilities against a small number of topics.