KAFKA-2570: commit offsets on rebalance/close when auto-commit is enabled#257
KAFKA-2570: commit offsets on rebalance/close when auto-commit is enabled#257hachikuji wants to merge 1 commit into
Conversation
|
kafka-trunk-git-pr #588 FAILURE |
|
Build failure seems to be caused by KAFKA-2555, which (correctly) introduced the possibility of UnknownConsumerIdException being raised from commitSync(). I'll update the test to allow for this possibility. |
b11b0cc to
e6fafef
Compare
|
kafka-trunk-git-pr #591 FAILURE |
|
This time failure looks like transient failure from KAFKA-2058. |
There was a problem hiding this comment.
Add a TODO mentioning that this should be removed after we persist in ZK.
|
LGTM overall, could you add a unit test for auto commit? |
e6fafef to
28357e7
Compare
|
@guozhangwang Ready for another look when you have a chance. |
|
kafka-trunk-git-pr #615 FAILURE |
|
The test failure looks to be unrelated. I've added a comment on KAFKA-2587 to see if that jira needs to be reopened. |
…r new components (apache#257) TICKET = N/A LI_DESCRIPTION = Modified github actions to generate JARs and run tests for new components. EXIT_CRITERIA = Never. We want to have custom github actions for publishing the artifact for linkedin. In 2.4, we only used to upload JARs for clients and core subprojects into the artifactory. This is not enough to build kafka, and we observed a failure when building it internally. This review adds the JARs for storage, raft, server-common, storage:api subprojects. We also add these subprojects (and also shell used for CLI tools and generator used for generating request and response classes) to the unit test and integration test github actions. We also switch from using uploadArchive to publish as recommended. Kafka 3.0 moved to using the maven-publish plugin, that provides the publish target. uploadArchive is an alias that points to it, and has been kept for backwards compatibility. According to the README.md file, publish is the recommended target.
…r new components (apache#257) TICKET = N/A LI_DESCRIPTION = Modified github actions to generate JARs and run tests for new components. EXIT_CRITERIA = Never. We want to have custom github actions for publishing the artifact for linkedin. In 2.4, we only used to upload JARs for clients and core subprojects into the artifactory. This is not enough to build kafka, and we observed a failure when building it internally. This review adds the JARs for storage, raft, server-common, storage:api subprojects. We also add these subprojects (and also shell used for CLI tools and generator used for generating request and response classes) to the unit test and integration test github actions. We also switch from using uploadArchive to publish as recommended. Kafka 3.0 moved to using the maven-publish plugin, that provides the publish target. uploadArchive is an alias that points to it, and has been kept for backwards compatibility. According to the README.md file, publish is the recommended target.
…s for new components (apache#257) TICKET = N/A LI_DESCRIPTION = Modified github actions to generate JARs and run tests for new components. EXIT_CRITERIA = Never. We want to have custom github actions for publishing the artifact for linkedin. In 2.4, we only used to upload JARs for clients and core subprojects into the artifactory. This is not enough to build kafka, and we observed a failure when building it internally. This review adds the JARs for storage, raft, server-common, storage:api subprojects. We also add these subprojects (and also shell used for CLI tools and generator used for generating request and response classes) to the unit test and integration test github actions. We also switch from using uploadArchive to publish as recommended. Kafka 3.0 moved to using the maven-publish plugin, that provides the publish target. uploadArchive is an alias that points to it, and has been kept for backwards compatibility. According to the README.md file, publish is the recommended target.
…s for new components (apache#257) TICKET = N/A LI_DESCRIPTION = Modified github actions to generate JARs and run tests for new components. EXIT_CRITERIA = Never. We want to have custom github actions for publishing the artifact for linkedin. In 2.4, we only used to upload JARs for clients and core subprojects into the artifactory. This is not enough to build kafka, and we observed a failure when building it internally. This review adds the JARs for storage, raft, server-common, storage:api subprojects. We also add these subprojects (and also shell used for CLI tools and generator used for generating request and response classes) to the unit test and integration test github actions. We also switch from using uploadArchive to publish as recommended. Kafka 3.0 moved to using the maven-publish plugin, that provides the publish target. uploadArchive is an alias that points to it, and has been kept for backwards compatibility. According to the README.md file, publish is the recommended target.
…rom outside the package. (apache#257) Co-authored-by: Tetiana Fioshkina <tetiana.fioshkina@hpe.com>
…che#257) By sequentually downloading all files to merge to a temporal directory and use the local files for the merging process, we avoid having all fetched objects open while merging. This should avoid having too many connections open to S3. The disk used is cleaned after each merge run.
…che#257) By sequentually downloading all files to merge to a temporal directory and use the local files for the merging process, we avoid having all fetched objects open while merging. This should avoid having too many connections open to S3. The disk used is cleaned after each merge run.
…pache#257) Refreshed snapshot from the kafka format sweep review. Five language-engine PRs landed in this session: - PR apache#253 — Annotation paren anchor for continuation slots - PR apache#254 — StructuralBody2x catalog class for inline lambda bodies - PR apache#255 — Lambda body as continuation enclosing scope - PR apache#256 — Inline-lambda body close brace catalog (+0 / +projectDelta) - PR apache#257 — emittedLineStarts restricted to OWNER's prefix slot Sweep stats (post-apache#257): files scanned : 5,834 files changed : 1,027 (17.6%) [initial PR #6: 1,038] byte delta : -10,877 [initial PR #6: -6,661] reformat time : 7.2s total, 1.23 ms/file The larger byte delta vs. the initial sweep reflects PR apache#257 fixing the AsyncKafkaConsumerTest-style "body at col 48" pattern across multiple files — body statements no longer indent to the inflated close-paren-line column, so each affected line shrinks by 30+ characters. Remaining patterns deferred to subsequent PRs: - Chain-dot over-application on Class.STATIC_FIELD / new X() (AdminClientConfig L273/L282, WriteTxnMarkersRequest L135, TxnOffsetCommitRequest L156/L159) - Partial-shift inconsistencies (StreamsResetter, ClientUtilsTest, RequestResponseTest, LeaveGroupRequest*, AlterReplicaLogDirs) - Trailing-whitespace cleanups on blank lines (incidental noise)
…pache#258) Refreshed snapshot. Six language-engine PRs landed this session: - PR apache#253 — Annotation paren anchor for continuation slots - PR apache#254 — StructuralBody2x catalog class for inline lambda bodies - PR apache#255 — Lambda body as continuation enclosing scope - PR apache#256 — Inline-lambda body close brace catalog (+0 / +projectDelta) - PR apache#257 — emittedLineStarts restricted to OWNER's prefix slot - PR apache#258 — alignmentTargets restricted to OWNER's prefix slot Sweep stats (post-apache#258): files scanned : 5,834 files changed : 1,000 (17.1%) [initial PR #6: 1,038] byte delta : -10,630 [initial PR #6: -6,661] reformat time : 7.1s total, 1.22 ms/file Remaining patterns (deferred): - Outer chain-dot receiver-anchor variations (TxnOffsetCommitRequest L156/L159, RequestResponseTest L3373+, etc.) — separate path from PR apache#258's inner-arg fix - Small continuation normalizations of non-standard source (StreamsResetter, ClientUtilsTest, LeaveGroupRequestTest) - Trailing-whitespace cleanups on blank lines (incidental noise)
…pache#259) Refreshed snapshot. Seven language-engine PRs landed this session: - PR apache#253 — Annotation paren anchor for continuation slots - PR apache#254 — StructuralBody2x catalog class for inline lambda bodies - PR apache#255 — Lambda body as continuation enclosing scope - PR apache#256 — Inline-lambda body close brace catalog - PR apache#257 — emittedLineStarts restricted to OWNER's prefix slot - PR apache#258 — alignmentTargets restricted to OWNER's prefix slot - PR apache#259 — Chain wrap preservation when siblings agree on column Sweep stats (post-apache#259): files scanned : 5,834 files changed : 990 (17.0%) [initial PR #6: 1,038] byte delta : -10,685 [initial PR #6: -6,661] reformat time : ~7s total Most called-out files from the user's original review now fully closed: - ShareConsumerTest (body + close brace) - ProducerIdExpirationTest - DeleteAclsResponse - CustomQuotaCallbackTest - AsyncKafkaConsumerTest (body indent regression) - AdminClientConfig L273/L282 - WriteTxnMarkersRequest L135 - TxnOffsetCommitRequest L156-159 - RequestResponseTest chain shifts - AlterReplicaLogDirsRequestTest (cascade-closed) Remaining residuals (~5 files, small): - StreamsResetter L580 (single `||` continuation normalisation) - ClientUtilsTest L79/L143 (continuation normalisation) - LeaveGroupRequestTest L55 (continuation shift) - LeaveGroupResponseTest (trailing-ws on blank lines) - ReassignPartitionsUnitTest (single line) These are mostly the formatter normalising slightly non-standard source — not really bugs.
…pache#260) Refreshed snapshot. Eight language-engine PRs landed this session: - PR apache#253 — Annotation paren anchor for continuation slots - PR apache#254 — StructuralBody2x catalog class for inline lambda bodies - PR apache#255 — Lambda body as continuation enclosing scope - PR apache#256 — Inline-lambda body close brace catalog - PR apache#257 — emittedLineStarts restricted to OWNER's prefix slot - PR apache#258 — alignmentTargets restricted to OWNER's prefix slot - PR apache#259 — Chain wrap preservation when siblings agree on column - PR apache#260 — Binary OPERATOR_PREFIX consults outermost binary's anchor Sweep stats (post-apache#260): files scanned : 5,834 files changed : 989 (17.0%) [initial PR #6: 1,038] byte delta : -10,663 reformat time : ~7s total Almost all called-out files from the user's original review now fully closed. Remaining residuals (<5 files) are correct normalisations of slightly non-standard source or trailing whitespace inside comment content (intentionally out of scope for this engine version per user design choice).
…pache#261) Refreshed snapshot. Nine language-engine PRs landed this session: - PR apache#253 — Annotation paren anchor for continuation slots - PR apache#254 — StructuralBody2x catalog class for inline lambda bodies - PR apache#255 — Lambda body as continuation enclosing scope - PR apache#256 — Inline-lambda body close brace catalog - PR apache#257 — emittedLineStarts restricted to OWNER's prefix slot - PR apache#258 — alignmentTargets restricted to OWNER's prefix slot - PR apache#259 — Chain wrap preservation when siblings agree on column - PR apache#260 — Binary OPERATOR_PREFIX consults outermost binary's anchor - PR apache#261 — Nested continuation slots use innermost enclosing scope Sweep stats (post-apache#261): files scanned : 5,834 files changed : 986 (16.9%) [initial PR #6: 1,038] byte delta : -11,137 [initial PR #6: -6,661] reformat time : ~7s total PR #7 review comments status: - Comments 1+2 closed (PR apache#261) - Comments 3 + 4/5 deferred (workbench-first investigation; diagnosis captured in language-engine memory project_pr7_deferred_comments_2026_05_17.md)
No description provided.