Skip to content

KAFKA-7832 Use automatic RPC generation in CreateTopics#5972

Merged
cmccabe merged 1 commit into
apache:trunkfrom
cmccabe:KAFKA-7609-createtopics
Feb 4, 2019
Merged

KAFKA-7832 Use automatic RPC generation in CreateTopics#5972
cmccabe merged 1 commit into
apache:trunkfrom
cmccabe:KAFKA-7609-createtopics

Conversation

@cmccabe

@cmccabe cmccabe commented Nov 29, 2018

Copy link
Copy Markdown
Contributor

Use KAFKA-7609 RPC generation in CreateTopics. This was split off of KAFKA-7609

@cmccabe cmccabe changed the title Use KAFKA-7609 RPC generation in CreateTopics KAFKA-7832 Use automatic RPC generation in CreateTopics Jan 16, 2019
@cmccabe cmccabe force-pushed the KAFKA-7609-createtopics branch 2 times, most recently from b526dea to 606c5c0 Compare January 16, 2019 19:43

@junrao junrao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmccabe : Thanks for the PR. A few comments below.

Also, the 4 test failures like the following seem related to the PR?

kafka.api.AuthorizerIntegrationTest > testCreateTopicAuthorizationWithClusterCreate FAILED
kafka.api.AuthorizerIntegrationTest > testAuthorizationWithTopicExisting FAILED

Comment thread clients/src/main/java/org/apache/kafka/common/requests/CreateTopicsRequest.java Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should return TOPIC_AUTHORIZATION_FAILED?

@cmccabe cmccabe Jan 22, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we keep this timeout since the default timeout is 0 and could fail a request prematurely?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Let's set the default timeout to something like a minute, so that we don't get bogus failures in unit tests. (This shouldn't matter for AdminClient since AC always explicitly sets the timeout.)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we keep the 1000 timeout?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above, let's set it in CreateTopicsRequest.json

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we keep the 1000 timeout?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we keep the 10000 timeout?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the partition index start at 0?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't really work as a general pattern. What I mean is that for ElectPreferredReplicas there is no analog of NewTopic: The AdminClient API is just a Collection<TopicPartition>, so there's no handy class in which to put the conversion code. It might be worth having a consistent place to put these conversion functions (for AdminClient protocol messages, at least); either a single common class (AdminClientProtocolUtil), or a ElectPreferredReplicasProtocolUtil etc. The latter would have one benefit: With the use of nested classes there's the possibility for imports like import org.apache.kafka.common.message.ElectPreferredLeadersResponseData.Result; to collide when another protocol uses a nested class named Result, and using outer class qualified type names makes the code ugly. Having a ElectPreferredReplicasProtocolUtil etc would avoid this.

@cmccabe cmccabe Jan 22, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that you can't use exactly the same pattern for TopicPartition, since it's a more generic class, not an AdminClient class at all. Probably the best place to put the conversion function is ElectPreferredLeaderElectionRequest.java, since it's relevant to that request.

As a side note, with TopicPartition, the conversion code is pretty simple, right? Something like partitions.map(p -> new ElectPreferredLeadersResponseData.Result.setIndex(p.partition()).setTopic(p.topic()) So you could consider doing it inlline, at least in theory (I haven't looked at the code to see if this makes sense).

NewTopic was more involved since there are multiple options like whether we explicitly choose a partition layout, or just a number of replicas, etc.

dhruvilshah3 added a commit to dhruvilshah3/kafka that referenced this pull request Jan 18, 2019
@junrao

junrao commented Jan 22, 2019

Copy link
Copy Markdown
Contributor

Also, in CreateTopicsRequest.json, I am wondering if we should mark the topic name field as mapKey to enforce uniqueness?

@cmccabe

cmccabe commented Jan 23, 2019

Copy link
Copy Markdown
Contributor Author

Good idea. I have marked the topic name field as mapKey to enforce uniqueness. This should be ready to go.

@cmccabe cmccabe force-pushed the KAFKA-7609-createtopics branch from 606c5c0 to ae2e8a1 Compare January 23, 2019 18:07

@junrao junrao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmccabe : Thanks for the updated patch. Just a minor comment below. There also seem a few compilation errors in the tests?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since results is a set, is that check still needed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be clear, it's a multiset. This was done specifically to preserve the existing behavior where duplicate entries caused us to return an error code, rather than triggering a deserialization error (which would show up as a disconnection)

@junrao junrao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cmccabe : Thanks for the explanation. LGTM. Once the compilation errors are fixed and the tests pass, you can merge the PR.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a coding standard to structure the L815-816? I wonder if we could merge both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason for having two lines is to avoid a line which is too long.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we merge L213-214?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@abbccdda

Copy link
Copy Markdown

Great work Colin! The protocol iteration will be much simpler with this change introduced. Just have some minor comments.

@cmccabe cmccabe force-pushed the KAFKA-7609-createtopics branch 2 times, most recently from 5e03aa3 to 7396ffc Compare January 26, 2019 00:58
@cmccabe

cmccabe commented Jan 26, 2019

Copy link
Copy Markdown
Contributor Author

rebased on trunk

@cmccabe

cmccabe commented Jan 31, 2019

Copy link
Copy Markdown
Contributor Author

retest this please

@cmccabe cmccabe force-pushed the KAFKA-7609-createtopics branch from 7396ffc to 83e42e5 Compare February 1, 2019 23:53
@cmccabe

cmccabe commented Feb 1, 2019

Copy link
Copy Markdown
Contributor Author

rebase on trunk

@cmccabe cmccabe merged commit e2e8bdb into apache:trunk Feb 4, 2019
pengxiaolong pushed a commit to pengxiaolong/kafka that referenced this pull request Jun 14, 2019
Reviewers: Jun Rao <junrao@gmail.com>, Tom Bentley <tbentley@redhat.com>, Boyang Chen <bchen11@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants