Skip to content

changefeedccl: better validation check for sarama config#118890

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
wenyihu6:test-obsarama
Feb 9, 2024
Merged

changefeedccl: better validation check for sarama config#118890
craig[bot] merged 1 commit intocockroachdb:masterfrom
wenyihu6:test-obsarama

Conversation

@wenyihu6
Copy link
Copy Markdown
Contributor

@wenyihu6 wenyihu6 commented Feb 7, 2024

Previously, we implement our own validation check for sarama config which may
have missed certain cases. This patch changes an additional check by adding
sarama's own exported validation function.

Release note: none
Epic: none

@wenyihu6 wenyihu6 requested a review from a team as a code owner February 7, 2024 15:55
@wenyihu6 wenyihu6 requested review from jayshrivastava and removed request for a team February 7, 2024 15:55
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@wenyihu6
Copy link
Copy Markdown
Contributor Author

wenyihu6 commented Feb 7, 2024

pkg/ccl/changefeedccl/sink_kafka.go line 211 at r1 (raw file):

	// where our call to Flush() would block forever.
	if (c.Flush.Bytes > 0 || c.Flush.Messages > 1) && c.Flush.Frequency == 0 {
		return errors.New("Flush.Frequency must be > 0 when Flush.Bytes > 0 or Flush.Messages > 1")

Sarama does the check for this here - it checks for c.Flush.Messages > 0 rather than c.Flush.Messages > 1 which I thought makes more sense.

https://github.com/IBM/sarama/blob/25c9c1a880e385781e1a39b49f8e7239e3d5e729/config.go#L593-L595

Copy link
Copy Markdown
Collaborator

@rharding6373 rharding6373 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: The sarama config validator is a nice find!

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @wenyihu6)


pkg/ccl/changefeedccl/sink_kafka.go line 211 at r1 (raw file):

Previously, wenyihu6 (Wenyi Hu) wrote…

Sarama does the check for this here - it checks for c.Flush.Messages > 0 rather than c.Flush.Messages > 1 which I thought makes more sense.

https://github.com/IBM/sarama/blob/25c9c1a880e385781e1a39b49f8e7239e3d5e729/config.go#L593-L595

👍

@wenyihu6
Copy link
Copy Markdown
Contributor Author

wenyihu6 commented Feb 8, 2024

pkg/ccl/changefeedccl/sink_kafka.go line 211 at r1 (raw file):

Previously, rharding6373 (Rachael Harding) wrote…

👍

I just realized that sarama doesn't return an error in this case but just outputs a warning https://github.com/IBM/sarama/blob/25c9c1a880e385781e1a39b49f8e7239e3d5e729/config.go#L594. I will the validation back. It does return error for improper client id though. I will add a test for this in the other PR.

@blathers-crl
Copy link
Copy Markdown

blathers-crl bot commented Feb 8, 2024

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@wenyihu6 wenyihu6 requested a review from rharding6373 February 8, 2024 18:15
Copy link
Copy Markdown
Collaborator

@rharding6373 rharding6373 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @wenyihu6)


pkg/ccl/changefeedccl/sink_kafka.go line 211 at r1 (raw file):

Previously, wenyihu6 (Wenyi Hu) wrote…

I just realized that sarama doesn't return an error in this case but just outputs a warning https://github.com/IBM/sarama/blob/25c9c1a880e385781e1a39b49f8e7239e3d5e729/config.go#L594. I will the validation back. It does return error for improper client id though. I will add a test for this in the other PR.

Could you add a note in the comment to the effect of "Note that the sarama config validator only logs an error in this case, so we check explicitly."

Previously, we implement our own validation check for sarama config which may
have missed certain cases. This patch changes an additional check by adding
sarama's own exported validation function.

Release note: none
Epic: none
@wenyihu6
Copy link
Copy Markdown
Contributor Author

wenyihu6 commented Feb 8, 2024

pkg/ccl/changefeedccl/sink_kafka.go line 211 at r1 (raw file):

Previously, rharding6373 (Rachael Harding) wrote…

Could you add a note in the comment to the effect of "Note that the sarama config validator only logs an error in this case, so we check explicitly."

Done.

@wenyihu6
Copy link
Copy Markdown
Contributor Author

wenyihu6 commented Feb 9, 2024

TFTRs!!

bors r=rharding6373

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Feb 9, 2024

Build succeeded:

@craig craig bot merged commit 69525a2 into cockroachdb:master Feb 9, 2024
@wenyihu6 wenyihu6 deleted the test-obsarama branch February 18, 2025 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants