apache#10585)

The command used by our private CI is ./gradlew cleanTest xxx:test. It does not re-run test when we use unitTest and integrationTest to replace test. The root cause is that we don't offer test output (unitTest and integrationTest) to cleanTest task and so it does not delete related test output.

Reviewers: Ismael Juma <ismael@juma.me.uk>

…raft module (apache#10791)

The command `./gradlew raft:integrationTest`  can't run any integration test since `org.junit.jupiter.api.Tag` does not work for jqwik engine (see jqwik-team/jqwik#36 (comment)). 

Reviewers: Ismael Juma <ismael@juma.me.uk>

Fix examples under security.html so they use the right bash icon (`>`
instead of `$`) and also uses the right tool for showing code listings.

Reviewers: Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>

…pache#10765)

As specified in KIP-743, this PR removes the built-in metrics
in Streams that are superseded by the refactoring proposed in KIP-444.

Reviewers: Guozhang Wang <wangguoz@gmail.com>, Luke Chen <showuon@gmail.com>

The broker shouldn't assume create access to the chroot. There are
deployement scenarios where the chroot is already created is the only
znode which the broker can access.

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ron Dagostino <rdagostino@confluent.io>

Also:
* Remove `ZookeeperTopicService`
* Remove `TopicCommandWithZKClientTest`
* Fix a topic create validation bug
* Adjust existing tests

Reviewers: Ismael Juma <ismael@juma.me.uk>

This patch adds `Admin` support for the `listTransactions` API, which was added by [KIP-664](https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions). Similar to `listConsumerGroups`, the new `listTransactions` API is intended to be sent to all brokers. 

Reviewers: David Jacot <djacot@confluent.io>

…nsumerInterceptor interface (apache#10801)

Co-authored-by: “KahnCheny” <“kahn.cheny@gmail.com”>

Reviewers: Luke Chen <showuon@gmail.com>, David Jacot <djacot@confluent.io>

Async profiler 2.0 outputs html5 flame graph files
and supports simultaneous collection of cpu,
allocation and lock profiles in jfr format.

Updated the readme to include an example of the
latter and verified that the Readme commands
work with async profiler 2.0.

Release notes:
* 1.28: https://mail.openjdk.java.net/pipermail/jmh-dev/2021-March/003171.html
* 1.29: https://mail.openjdk.java.net/pipermail/jmh-dev/2021-March/003218.html
* 1.30: https://mail.openjdk.java.net/pipermail/jmh-dev/2021-May/003237.html
* 1.31: https://mail.openjdk.java.net/pipermail/jmh-dev/2021-May/003286.html
* 1.32: https://mail.openjdk.java.net/pipermail/jmh-dev/2021-May/003307.html

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, David Jacot <djacot@confluent.io>, Luke Chen <showuon@gmail.com>

…moved option (apache#10806)

Reviewers: Luke Chen <showuon@gmail.com>, Ismael Juma <ismael@juma.me.uk>

Co-authored-by: shenwenbing <shenwenbing@qianxin.com>

…fig (apache#10797)

The trogdor ConsumeBenchWorker allows several consumption tasks to be run in parallel, the number is configurable using the threadsPerWorker config. If one of the consumption tasks completes executing successfully due to maxMessages being consumed, then, the consumption task prematurely notifies the doneFuture causing the entire ConsumeBenchWorker to halt. This becomes a problem when more than 1 consumption task is running in parallel, because the successful completion of 1 of the tasks shuts down the entire worker while the other tasks are still running. When the worker is shut down, it kills all the active consumption tasks, though they have not consumed maxMessages yet. This commit defers notification of the doneFuture to the CloseStatusUpdater thread, which is already responsible for tracking the status of the tasks and updating their status when all of the tasks complete.

Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>

…che#10808)

They were both deprecated in Apache Kafka 2.4 and it's a straightforward change
to use the non deprecated variants.

Reviewers: David Jacot <djacot@confluent.io>

…ormance (apache#10552)

I did code refactor/optimization, keep the same algorithm in this PR.

Originally, With this setting:
topicCount = 50;
partitionCount = 800;
consumerCount = 800;
We complete in 10 seconds, after my code refactor, the time down to 100~200 ms

With the 1 million partitions setting:
topicCount = 500;
partitionCount = 2000;
consumerCount = 2000;
No OutOfMemory will be thrown anymore. The time will take 4~5 seconds.

Reviewers: Vahid Hashemian <vahid.hashemian@gmail.com>, Guozhang Wang <wangguoz@gmail.com>

…pache#10787)

Since KafkaEventQueue is a generic data structure not specific to metadata, move it
into the server-common module.

Reviewers: Ismael Juma <ismael@juma.me.uk>, David Arthur <mumrah@gmail.com>

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Matthias J. Sax <matthias@confluent.io>

…#10664)

Refactored logConfig to be passed appropriately when using shutDownWhenFull or emitEarlyWhenFull. Removed the constructor that doesn't accept a logConfig parameter so you're forced to specify it explicitly, whether it's empty/unspecified or not.

Co-authored-by: Bruno Cadonna <cadonna@apache.org>

Reviewers: Walker Carlson <wcarlson@confluent.io>, Bruno Cadonna <cadonna@apache.org>

This is part 2 of KIP-730. Part 1 was in apache#10504.

This PR adds QuorumController support for handling AllocateProducerIDs requests
and managing the state of the latest producer ID block in the controller by committing
this state to the metadata log.

Reviewers: Colin P. McCabe <cmccabe@apache.org>

Style fixes to KafkaRaftClient

Reviewers: Luke Chen <showuon@gmail.com>

…nsCommand (apache#10471)

Also remove zookeeper dependent methods and tests.

Reviewers: Ismael Juma <ismael@juma.me.uk>

1. When register state stores, add the store to globalStateStores before calling any blocking calls that may throw errors, so that upon closing we would close the stores as well.
2. Remove the other list as a local field, and call topology.globalStateStores when needed to get the list.

Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Bruno Cadonna <cadonna@apache.org>

Reviewers: Jun Rao <junrao@gmail.com>

Having the `testChrootExistsAndRootIsLocked` test in a separate `ZookeeperTestHarness` isn't enough to prevent the ACL changes to the ZK root from affecting other integration tests. So instead, let's use a dedicated znode for this test. It still works because `makeSurePersistentPathExists` uses `createRecursive`, which will recurse and act the same for the root or a given znode.

 Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>

…he#10817)

Reviewers: Luke Chen <showuon@gmail.com>, Jason Gustafson <jason@confluent.io>

…er (apache#9441)

Co-authored-by: Jason Gustafson<jason@confluent.io>
Reviewers: Jason Gustafson<jason@confluent.io>, Guozhang Wang <wangguoz@gmail.com>