test 3.0-action#2
Closed
lmr3796 wants to merge 22 commits into
Closed
Conversation
This commit is a squash of: - [LI-HOTFIX] Add a Github Actions workflow to trigger build and test upon PR and push (linkedin#106) - [LI-HOTFIX] Add a Github Actions workflow to trigger build and run all tests upon a tag creation/push (linkedin#107) - [LI-HOTFIX] Enable the Github Actions workflow to upload the built archive and disable all Travis actions (linkedin#108) - [LI-HOTFIX] Updating description of the CI flow (linkedin#114) - Add rat to CI build to make it consistent with PR (linkedin#173) And partially picked up the rat set up from - Adding licence and rat check (linkedin#159) TICKET = N/A LI_DESCRIPTION = LIKAFKA-34202 EXIT_CRITERIA = When migrating off Github Actions for CI [LI-HOTFIX] Add rat to CI build to make it consistent with PR (linkedin#173)
EXIT_CRITERIA = When working branch changed or rebase finished
LI_DESCRIPTION = The tests has been flaky and causing the build very hard to pass. TICKET=N/A EXIT_CRITERIA = When upstream tests become very stable
Squashed commits: - [LI-HOTFIX] add Bintray support to LinkedIn Kafka Github - [LI-HOTFIX] Migrate bintray publish to JFrog (linkedin#132) - [LI-HOTFIX] Use JFrog Api key instead of password (linkedin#139) TICKET = N/A LI_DESCRIPTION = DEPENG-2065. This is advised by dep engineering EXIT_CRITERIA = When not using JFrog for publishing
LI infra doesn't support Scala 2.13 yet. So pin to 2.12 TICKET = N/A LI_DESCRIPTION = N/A EXIT_CRITERIA = When LI infra supports Scala 2.13
This is a squash of 2 commits, performed because the 2 units are too tangled: - 7f6ada8 [LI-HOTFIX] LIKAFKA-4425 and LIKAFKA-4424 - add quota metrics log - 4e1a1ad [LI-HOTFIX] Log request quota metrics to kafka-quota-metrics.log TICKET = LIKAFKA-4425, LIKAFKA-4424, LIKAFKA-18357 EXIT_CRITERIA = MANUAL ["when quota metrics log is no longer needed"] LI_DESCRIPTION = Original commit messages: ---------------- [LI-HOTFIX] LIKAFKA-4425 and LIKAFKA-4424 - add quota metrics log This patch has 2 changes: - Added a background thread that dumps the quota metrics into the log every 30 seconds. - Also changed the delayQueue sensor into an absolute counter --------------- [LI-HOTFIX] Log request quota metrics to kafka-quota-metrics.log RB=1362648 BUG=LIKAFKA-18357 G=Kafka-Code-Reviews R=dolin A=dolin ---------------
…ave been throttled and rename the delay queue size metric TICKET = LI_DESCRIPTION = Add throttle-count metric and rename delay queue size sensor. BUG=LIKAFKA-4424 EXIT_CRITERIA = MANUAL ["when throttle-count metric is no longer needed"]
TICKET = LI_DESCRIPTION = avoid duplicate app-info registrations RB=877780 BUG=LIKAFKA-8529 G=Kafka-Code-Reviews A=jkoshy EXIT_CRITERIA = MANUAL [""]
… progress after committing to a deleted topic TICKET = LI_DESCRIPTION = New consumer throws a KafkaException when trying to commit to a topic that has been deleted. MirrorMaker.commitOffsets doesn't attempt to catch the KafkaException and just kills the process. This hotfix just catches KafkaException in MirrorMaker.commitOffsets until we make a cleaner long-term fix. RB=1120784 BUG=LIKAFKA-12852 G=Kafka-Code-Reviews A=jqin EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
TICKET = KAFKA-6588 LI_DESCRIPTION = Introduces a new metric that is useful to monitor log compaction threads in the broker. Too much overhead in open-source to introduce this metric. Hence, the hotfix. EXIT_CRITERIA = MANUAL ["when this metric is no longer needed. KAFKA-6588 (Won’t Fix)"]
…We need to do this to avoid committing old offsets. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["NONE"]
… to new consumer TICKET = LIKAFKA-14632 LI_DESCRIPTION = adds "poll-interval", "last-poll-seconds-ago" and "last-heartbeat-received-seconds-ago" sensors to consumer RB=1198496 G=Kafka-Code-Reviews R=tpalino,kambroff A=dolin EXIT_CRITERIA = MANUAL ["none right now"]
…e memory than total memory TICKET = LI_DESCRIPTION = RB=1186874 G=Kafka-Code-Reviews A=dolin,jqin EXIT_CRITERIA = MANUAL ["Moving this upstream would probably involve dropping some of the features of this patch."]
…rogress in processing request TICKET = LI_DESCRIPTION = This patch adds a new config called request.processing.max.time.ms. By default its value is Long.MaxValue to be backward compatible. Network threads will log error message and call Runtime.getRuntime.halt(1) if one of the following is true: - New request can not be inserted into requestChannel after request.processing.max.time.ms - requestChannel.lastEnqueueTimeMs - requestChannel.lastDequeueTimeMs > request.processing.max.time.ms This guarantees that Kafka broker will shutdown if no request was dequeued from requestChannel for more than request.processing.max.time.ms. RB=844988 G=Kafka-Code-Reviews A=jkoshy EXIT_CRITERIA = MANUAL ["NONE"]
TICKET = LIKAFKA-11492 LI_DESCRIPTION = captures a heap dump and dies if kafka broker fails to do any IO activity for a configurable amount of time EXIT_CRITERIA = MANUAL ["likely never"]
…use on halt TICKET = LI_DESCRIPTION = Allows failing process (in this case, KafkaServer.scala) to generate heap dump and exit status code, before exiting. I think this codepath is also used by KMM (venice). RB=1340022 EXIT_CRITERIA = MANUAL ["As long as we use KafkaServer.scala"]
… Long.MaxValue TICKET = LI_DESCRIPTION = RB=1137541 G=Kafka-Code-Reviews A=jkoshy EXIT_CRITERIA = MANUAL ["NONE"]
TICKET = KAFKA-7711 LI_DESCRIPTION = Add a bounded flush() API and timeout if producer is unable to flush all the batch records in a limited time. EXIT_CRITERIA = TICKET [KAFKA-7711]
… for LeaderAndIsr, Update_metadata, Stop_replica requests Add controller sensors for requests queue and remote times. Added kafka sensors to display LeaderAndIsr, Stop_replica, and Update_metadata request times (queue time, remote time) on InGraph. Displaying time for 999, 99, 75, 50, Avg, Max percentiles. TICKET = LI_DESCRIPTION = RB=1326056 G=kafka-reviewers R=dolin,luwang A=dolin,luwang EXIT_CRITERIA = MANUAL [""]
… creation Add max.message.bytes, min.insync.replicas, min.compaction.lag.ms config defaults for __consumer_offsets topic during create time. TICKET = LI_DESCRIPTION = RB=1342860 G=Kafka-Code-Reviews R=okaraman A=okaraman EXIT_CRITERIA = MANUAL ["None"]
lmr3796
pushed a commit
that referenced
this pull request
Oct 22, 2021
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Oct 22, 2021
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Oct 27, 2021
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Oct 28, 2021
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Nov 19, 2021
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Nov 19, 2021
TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - ~~Avoid appending to the time index during shutdown if the time index has not yet be initialized~~ This is covered in apache#8346 and apache#10960 EXIT_CRITERIA = TICKET [KAFKA-8667, KAFKA-8668] The patch is a sqaush of the 2 commits: == This is the 1st commit [155b4f8] == [LI-HOTFIX] Reduce lock retention and improve broker shutdown time: TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - Avoid appending to the time index during shutdown if the time index has not yet be initialized RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee == This is the commit #2 [c53fffd] == [LI-HOTFIX] Update fetcher thread idle flag in addPartitions TICKET = KAFKA-8667 LI_DESCRIPTION = This patch fixes in bug introduced by “[LI-HOTFIX] Reduce lock retention and improve broker shutdown time” HOTFIX where the fetcher thread idle flag is not set in addPartitions, which can cause idle fetcher thread not shutdown in time. RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee
lmr3796
pushed a commit
that referenced
this pull request
Jan 1, 2022
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Feb 9, 2022
TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - ~~Avoid appending to the time index during shutdown if the time index has not yet be initialized~~ This is covered in apache#8346 and apache#10960 EXIT_CRITERIA = TICKET [KAFKA-8667, KAFKA-8668] The patch is a sqaush of the 2 commits: == This is the 1st commit [155b4f8] == [LI-HOTFIX] Reduce lock retention and improve broker shutdown time: TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - Avoid appending to the time index during shutdown if the time index has not yet be initialized RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee == This is the commit #2 [c53fffd] == [LI-HOTFIX] Update fetcher thread idle flag in addPartitions TICKET = KAFKA-8667 LI_DESCRIPTION = This patch fixes in bug introduced by “[LI-HOTFIX] Reduce lock retention and improve broker shutdown time” HOTFIX where the fetcher thread idle flag is not set in addPartitions, which can cause idle fetcher thread not shutdown in time. RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee
lmr3796
pushed a commit
that referenced
this pull request
Mar 25, 2022
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Mar 25, 2022
TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - ~~Avoid appending to the time index during shutdown if the time index has not yet be initialized~~ This is covered in apache#8346 and apache#10960 EXIT_CRITERIA = TICKET [KAFKA-8667, KAFKA-8668] The patch is a sqaush of the 2 commits: == This is the 1st commit [155b4f8] == [LI-HOTFIX] Reduce lock retention and improve broker shutdown time: TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - Avoid appending to the time index during shutdown if the time index has not yet be initialized RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee == This is the commit #2 [c53fffd] == [LI-HOTFIX] Update fetcher thread idle flag in addPartitions TICKET = KAFKA-8667 LI_DESCRIPTION = This patch fixes in bug introduced by “[LI-HOTFIX] Reduce lock retention and improve broker shutdown time” HOTFIX where the fetcher thread idle flag is not set in addPartitions, which can cause idle fetcher thread not shutdown in time. RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee
lmr3796
pushed a commit
that referenced
this pull request
Jun 2, 2022
…eiving offset commit exception TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Jun 2, 2022
TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - ~~Avoid appending to the time index during shutdown if the time index has not yet be initialized~~ This is covered in apache#8346 and apache#10960 EXIT_CRITERIA = TICKET [KAFKA-8667, KAFKA-8668] The patch is a sqaush of the 2 commits: == This is the 1st commit [155b4f8] == [LI-HOTFIX] Reduce lock retention and improve broker shutdown time: TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - Avoid appending to the time index during shutdown if the time index has not yet be initialized RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee == This is the commit #2 [c53fffd] == [LI-HOTFIX] Update fetcher thread idle flag in addPartitions TICKET = KAFKA-8667 LI_DESCRIPTION = This patch fixes in bug introduced by “[LI-HOTFIX] Reduce lock retention and improve broker shutdown time” HOTFIX where the fetcher thread idle flag is not set in addPartitions, which can cause idle fetcher thread not shutdown in time. RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)