Skip to content

KAFKA-2570: commit offsets on rebalance/close when auto-commit is enabled#257

Closed
hachikuji wants to merge 1 commit into
apache:trunkfrom
hachikuji:KAFKA-2570
Closed

KAFKA-2570: commit offsets on rebalance/close when auto-commit is enabled#257
hachikuji wants to merge 1 commit into
apache:trunkfrom
hachikuji:KAFKA-2570

Conversation

@hachikuji

Copy link
Copy Markdown
Contributor

No description provided.

@asfbot

asfbot commented Sep 28, 2015

Copy link
Copy Markdown

kafka-trunk-git-pr #588 FAILURE
Looks like there's a problem with this pull request

@hachikuji

Copy link
Copy Markdown
Contributor Author

Build failure seems to be caused by KAFKA-2555, which (correctly) introduced the possibility of UnknownConsumerIdException being raised from commitSync(). I'll update the test to allow for this possibility.

@asfbot

asfbot commented Sep 29, 2015

Copy link
Copy Markdown

kafka-trunk-git-pr #591 FAILURE
Looks like there's a problem with this pull request

@hachikuji

Copy link
Copy Markdown
Contributor Author

This time failure looks like transient failure from KAFKA-2058.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a TODO mentioning that this should be removed after we persist in ZK.

@guozhangwang

Copy link
Copy Markdown
Contributor

LGTM overall, could you add a unit test for auto commit?

@hachikuji

Copy link
Copy Markdown
Contributor Author

@guozhangwang Ready for another look when you have a chance.

@asfbot

asfbot commented Sep 29, 2015

Copy link
Copy Markdown

kafka-trunk-git-pr #615 FAILURE
Looks like there's a problem with this pull request

@hachikuji

Copy link
Copy Markdown
Contributor Author

The test failure looks to be unrelated. I've added a comment on KAFKA-2587 to see if that jira needs to be reopened.

@asfgit asfgit closed this in 5e769cc Sep 30, 2015
wyuka added a commit to wyuka/kafka that referenced this pull request Jan 27, 2022
…r new components (apache#257)

TICKET = N/A
LI_DESCRIPTION = Modified github actions to generate JARs and run tests for new components.
EXIT_CRITERIA = Never. We want to have custom github actions for publishing the artifact for linkedin.

In 2.4, we only used to upload JARs for clients and core subprojects into the artifactory. This is not enough to build kafka, and we observed a failure when building it internally.

This review adds the JARs for storage, raft, server-common, storage:api subprojects.
We also add these subprojects (and also shell used for CLI tools and generator used for generating request and response classes) to the unit test and integration test github actions.

We also switch from using uploadArchive to publish as recommended. Kafka 3.0 moved to using the maven-publish plugin, that provides the publish target. uploadArchive is an alias that points to it, and has been kept for backwards compatibility. According to the README.md file, publish is the recommended target.
wyuka added a commit to wyuka/kafka that referenced this pull request Mar 4, 2022
…r new components (apache#257)

TICKET = N/A
LI_DESCRIPTION = Modified github actions to generate JARs and run tests for new components.
EXIT_CRITERIA = Never. We want to have custom github actions for publishing the artifact for linkedin.

In 2.4, we only used to upload JARs for clients and core subprojects into the artifactory. This is not enough to build kafka, and we observed a failure when building it internally.

This review adds the JARs for storage, raft, server-common, storage:api subprojects.
We also add these subprojects (and also shell used for CLI tools and generator used for generating request and response classes) to the unit test and integration test github actions.

We also switch from using uploadArchive to publish as recommended. Kafka 3.0 moved to using the maven-publish plugin, that provides the publish target. uploadArchive is an alias that points to it, and has been kept for backwards compatibility. According to the README.md file, publish is the recommended target.
wyuka added a commit to wyuka/kafka that referenced this pull request Mar 28, 2022
…s for new components (apache#257)

TICKET = N/A
LI_DESCRIPTION = Modified github actions to generate JARs and run tests for new components.
EXIT_CRITERIA = Never. We want to have custom github actions for publishing the artifact for linkedin.

In 2.4, we only used to upload JARs for clients and core subprojects into the artifactory. This is not enough to build kafka, and we observed a failure when building it internally.

This review adds the JARs for storage, raft, server-common, storage:api subprojects.
We also add these subprojects (and also shell used for CLI tools and generator used for generating request and response classes) to the unit test and integration test github actions.

We also switch from using uploadArchive to publish as recommended. Kafka 3.0 moved to using the maven-publish plugin, that provides the publish target. uploadArchive is an alias that points to it, and has been kept for backwards compatibility. According to the README.md file, publish is the recommended target.
wyuka added a commit to wyuka/kafka that referenced this pull request Jun 16, 2022
…s for new components (apache#257)

TICKET = N/A
LI_DESCRIPTION = Modified github actions to generate JARs and run tests for new components.
EXIT_CRITERIA = Never. We want to have custom github actions for publishing the artifact for linkedin.

In 2.4, we only used to upload JARs for clients and core subprojects into the artifactory. This is not enough to build kafka, and we observed a failure when building it internally.

This review adds the JARs for storage, raft, server-common, storage:api subprojects.
We also add these subprojects (and also shell used for CLI tools and generator used for generating request and response classes) to the unit test and integration test github actions.

We also switch from using uploadArchive to publish as recommended. Kafka 3.0 moved to using the maven-publish plugin, that provides the publish target. uploadArchive is an alias that points to it, and has been kept for backwards compatibility. According to the README.md file, publish is the recommended target.
udaynpusa pushed a commit to mapr/kafka that referenced this pull request Jan 30, 2024
…rom outside the package. (apache#257)

Co-authored-by: Tetiana Fioshkina <tetiana.fioshkina@hpe.com>
davide-armand pushed a commit to aiven/kafka that referenced this pull request Dec 1, 2025
…che#257)

By sequentually downloading all files to merge to a temporal directory
and use the local files for the merging process, we avoid having all
fetched objects open while merging.
This should avoid having too many connections open to S3.

The disk used is cleaned after each merge run.
jeqo added a commit to aiven/kafka that referenced this pull request Jan 16, 2026
…che#257)

By sequentually downloading all files to merge to a temporal directory
and use the local files for the merging process, we avoid having all
fetched objects open while merging.
This should avoid having too many connections open to S3.

The disk used is cleaned after each merge run.
traceyyoshima added a commit to traceyyoshima/kafka that referenced this pull request May 17, 2026
…pache#257)

Refreshed snapshot from the kafka format sweep review. Five
language-engine PRs landed in this session:

- PR apache#253 — Annotation paren anchor for continuation slots
- PR apache#254 — StructuralBody2x catalog class for inline lambda bodies
- PR apache#255 — Lambda body as continuation enclosing scope
- PR apache#256 — Inline-lambda body close brace catalog (+0 / +projectDelta)
- PR apache#257 — emittedLineStarts restricted to OWNER's prefix slot

Sweep stats (post-apache#257):
  files scanned : 5,834
  files changed : 1,027 (17.6%)   [initial PR #6: 1,038]
  byte delta    : -10,877          [initial PR #6: -6,661]
  reformat time : 7.2s total, 1.23 ms/file

The larger byte delta vs. the initial sweep reflects PR apache#257
fixing the AsyncKafkaConsumerTest-style "body at col 48" pattern
across multiple files — body statements no longer indent to the
inflated close-paren-line column, so each affected line shrinks
by 30+ characters.

Remaining patterns deferred to subsequent PRs:
  - Chain-dot over-application on Class.STATIC_FIELD / new X()
    (AdminClientConfig L273/L282, WriteTxnMarkersRequest L135,
    TxnOffsetCommitRequest L156/L159)
  - Partial-shift inconsistencies (StreamsResetter, ClientUtilsTest,
    RequestResponseTest, LeaveGroupRequest*, AlterReplicaLogDirs)
  - Trailing-whitespace cleanups on blank lines (incidental noise)
traceyyoshima added a commit to traceyyoshima/kafka that referenced this pull request May 17, 2026
…pache#258)

Refreshed snapshot. Six language-engine PRs landed this session:

- PR apache#253 — Annotation paren anchor for continuation slots
- PR apache#254 — StructuralBody2x catalog class for inline lambda bodies
- PR apache#255 — Lambda body as continuation enclosing scope
- PR apache#256 — Inline-lambda body close brace catalog (+0 / +projectDelta)
- PR apache#257 — emittedLineStarts restricted to OWNER's prefix slot
- PR apache#258 — alignmentTargets restricted to OWNER's prefix slot

Sweep stats (post-apache#258):
  files scanned : 5,834
  files changed : 1,000 (17.1%)   [initial PR #6: 1,038]
  byte delta    : -10,630          [initial PR #6: -6,661]
  reformat time : 7.1s total, 1.22 ms/file

Remaining patterns (deferred):
  - Outer chain-dot receiver-anchor variations (TxnOffsetCommitRequest
    L156/L159, RequestResponseTest L3373+, etc.) — separate path from
    PR apache#258's inner-arg fix
  - Small continuation normalizations of non-standard source
    (StreamsResetter, ClientUtilsTest, LeaveGroupRequestTest)
  - Trailing-whitespace cleanups on blank lines (incidental noise)
traceyyoshima added a commit to traceyyoshima/kafka that referenced this pull request May 17, 2026
…pache#259)

Refreshed snapshot. Seven language-engine PRs landed this session:

- PR apache#253 — Annotation paren anchor for continuation slots
- PR apache#254 — StructuralBody2x catalog class for inline lambda bodies
- PR apache#255 — Lambda body as continuation enclosing scope
- PR apache#256 — Inline-lambda body close brace catalog
- PR apache#257 — emittedLineStarts restricted to OWNER's prefix slot
- PR apache#258 — alignmentTargets restricted to OWNER's prefix slot
- PR apache#259 — Chain wrap preservation when siblings agree on column

Sweep stats (post-apache#259):
  files scanned : 5,834
  files changed : 990 (17.0%)     [initial PR #6: 1,038]
  byte delta    : -10,685          [initial PR #6: -6,661]
  reformat time : ~7s total

Most called-out files from the user's original review now fully
closed:
  - ShareConsumerTest (body + close brace)
  - ProducerIdExpirationTest
  - DeleteAclsResponse
  - CustomQuotaCallbackTest
  - AsyncKafkaConsumerTest (body indent regression)
  - AdminClientConfig L273/L282
  - WriteTxnMarkersRequest L135
  - TxnOffsetCommitRequest L156-159
  - RequestResponseTest chain shifts
  - AlterReplicaLogDirsRequestTest (cascade-closed)

Remaining residuals (~5 files, small):
  - StreamsResetter L580 (single `||` continuation normalisation)
  - ClientUtilsTest L79/L143 (continuation normalisation)
  - LeaveGroupRequestTest L55 (continuation shift)
  - LeaveGroupResponseTest (trailing-ws on blank lines)
  - ReassignPartitionsUnitTest (single line)

These are mostly the formatter normalising slightly non-standard
source — not really bugs.
traceyyoshima added a commit to traceyyoshima/kafka that referenced this pull request May 17, 2026
…pache#260)

Refreshed snapshot. Eight language-engine PRs landed this session:

- PR apache#253 — Annotation paren anchor for continuation slots
- PR apache#254 — StructuralBody2x catalog class for inline lambda bodies
- PR apache#255 — Lambda body as continuation enclosing scope
- PR apache#256 — Inline-lambda body close brace catalog
- PR apache#257 — emittedLineStarts restricted to OWNER's prefix slot
- PR apache#258 — alignmentTargets restricted to OWNER's prefix slot
- PR apache#259 — Chain wrap preservation when siblings agree on column
- PR apache#260 — Binary OPERATOR_PREFIX consults outermost binary's anchor

Sweep stats (post-apache#260):
  files scanned : 5,834
  files changed : 989 (17.0%)     [initial PR #6: 1,038]
  byte delta    : -10,663
  reformat time : ~7s total

Almost all called-out files from the user's original review now
fully closed. Remaining residuals (<5 files) are correct
normalisations of slightly non-standard source or trailing
whitespace inside comment content (intentionally out of scope for
this engine version per user design choice).
traceyyoshima added a commit to traceyyoshima/kafka that referenced this pull request May 17, 2026
…pache#261)

Refreshed snapshot. Nine language-engine PRs landed this session:

- PR apache#253 — Annotation paren anchor for continuation slots
- PR apache#254 — StructuralBody2x catalog class for inline lambda bodies
- PR apache#255 — Lambda body as continuation enclosing scope
- PR apache#256 — Inline-lambda body close brace catalog
- PR apache#257 — emittedLineStarts restricted to OWNER's prefix slot
- PR apache#258 — alignmentTargets restricted to OWNER's prefix slot
- PR apache#259 — Chain wrap preservation when siblings agree on column
- PR apache#260 — Binary OPERATOR_PREFIX consults outermost binary's anchor
- PR apache#261 — Nested continuation slots use innermost enclosing scope

Sweep stats (post-apache#261):
  files scanned : 5,834
  files changed : 986 (16.9%)     [initial PR #6: 1,038]
  byte delta    : -11,137          [initial PR #6: -6,661]
  reformat time : ~7s total

PR #7 review comments status:
  - Comments 1+2 closed (PR apache#261)
  - Comments 3 + 4/5 deferred (workbench-first investigation; diagnosis
    captured in language-engine memory project_pr7_deferred_comments_2026_05_17.md)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants