[Fork]KAFKA-19167: Move RemoteLogManagerConfig out of KafkaConfig#2
[Fork]KAFKA-19167: Move RemoteLogManagerConfig out of KafkaConfig#2apalan60 wants to merge 8 commits into
Conversation
There was a problem hiding this comment.
Summary of Changes
Hello @apalan60, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request refactors the RemoteLogManagerConfig class by extracting its instantiation and management logic from KafkaConfig. The goal is to improve modularity and enable more flexible, dynamic configuration management for remote log features by introducing a singleton pattern for RemoteLogManagerConfig.
Highlights
- Refactoring
RemoteLogManagerConfig: TheRemoteLogManagerConfigclass has been refactored to be a singleton, accessible via a new static factory methodRemoteLogManagerConfig.of(config). It now internally manages itsAbstractConfigreference, allowing for dynamic updates to its configuration. - Decoupling from
KafkaConfig: The direct dependency ofKafkaConfigonRemoteLogManagerConfighas been removed.KafkaConfigno longer instantiates or holds aRemoteLogManagerConfigobject, simplifying its structure. - Widespread Usage Update: All existing code paths that previously accessed
config.remoteLogManagerConfighave been updated to use the newRemoteLogManagerConfig.of(config)static factory method. This change propagates across various server components, log management, and configuration validation logic. - Enhanced Dynamic Configuration Support: The new design for
RemoteLogManagerConfigexplicitly supports dynamic updates to its underlying configuration, which is validated by new dedicated test cases.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Explore these optional code suggestions:
|
||||||||||||||
There was a problem hiding this comment.
Code Review
This pull request refactors RemoteLogManagerConfig to use a singleton pattern with a factory method, which is a good improvement for managing this configuration centrally. The changes are applied consistently across the codebase.
I've found a few issues:
- A critical compilation error in
LogConfigTest.scaladue to a typo. - An opportunity to improve the implementation of the singleton factory method in
RemoteLogManagerConfig.javato make it more readable and avoid side effects in an atomic operation. - A section in
DynamicBrokerConfig.scalawith repetitive code that could be cleaned up for better readability and minor performance improvement.
Overall, the changes are in the right direction. Addressing these points will improve the quality and robustness of the code.
| val logProps = new Properties() | ||
| def validateCleanupPolicy(): Unit = { | ||
| LogConfig.validate(util.Map.of, logProps, kafkaConfig.extractLogConfigMap, kafkaConfig.remoteLogManagerConfig.isRemoteStorageSystemEnabled) | ||
| LogConfig.validate(util.Map.of, logProps, kafkaConfig.extractLogConfigMap, RemoteLogManagerConfig.of(kafkaConfig)isRemoteStorageSystemEnabled) |
There was a problem hiding this comment.
There seems to be a typo here. A dot . is missing between the method call RemoteLogManagerConfig.of(kafkaConfig) and the property access isRemoteStorageSystemEnabled. This will cause a compilation error.
This issue is repeated on lines 314, 317, 334, 343, 345, 347, 367, 371, 391, 395, 410, and 413. Please fix all occurrences.
| LogConfig.validate(util.Map.of, logProps, kafkaConfig.extractLogConfigMap, RemoteLogManagerConfig.of(kafkaConfig)isRemoteStorageSystemEnabled) | |
| LogConfig.validate(util.Map.of, logProps, kafkaConfig.extractLogConfigMap, RemoteLogManagerConfig.of(kafkaConfig).isRemoteStorageSystemEnabled) |
| if (RemoteLogManagerConfig.of(newConfig).remoteLogManagerCopierThreadPoolSize() != RemoteLogManagerConfig.of(oldConfig).remoteLogManagerCopierThreadPoolSize()) | ||
| remoteLogManager.resizeCopierThreadPool(RemoteLogManagerConfig.of(newConfig).remoteLogManagerCopierThreadPoolSize()) | ||
|
|
||
| if (newRLMConfig.remoteLogManagerExpirationThreadPoolSize() != oldRLMConfig.remoteLogManagerExpirationThreadPoolSize()) | ||
| remoteLogManager.resizeExpirationThreadPool(newRLMConfig.remoteLogManagerExpirationThreadPoolSize()) | ||
| if (RemoteLogManagerConfig.of(newConfig).remoteLogManagerExpirationThreadPoolSize() != RemoteLogManagerConfig.of(oldConfig).remoteLogManagerExpirationThreadPoolSize()) | ||
| remoteLogManager.resizeExpirationThreadPool(RemoteLogManagerConfig.of(newConfig).remoteLogManagerExpirationThreadPoolSize()) | ||
|
|
||
| if (newRLMConfig.remoteLogManagerFollowerThreadPoolSize() != oldRLMConfig.remoteLogManagerFollowerThreadPoolSize()) | ||
| remoteLogManager.resizeFollowerThreadPool(newRLMConfig.remoteLogManagerFollowerThreadPoolSize()) | ||
| if (RemoteLogManagerConfig.of(newConfig).remoteLogManagerFollowerThreadPoolSize() != RemoteLogManagerConfig.of(oldConfig).remoteLogManagerFollowerThreadPoolSize()) | ||
| remoteLogManager.resizeFollowerThreadPool(RemoteLogManagerConfig.of(newConfig).remoteLogManagerFollowerThreadPoolSize()) | ||
|
|
||
| if (newRLMConfig.remoteLogReaderThreads() != oldRLMConfig.remoteLogReaderThreads()) | ||
| remoteLogManager.resizeReaderThreadPool(newRLMConfig.remoteLogReaderThreads()) | ||
| if (RemoteLogManagerConfig.of(newConfig).remoteLogReaderThreads() != RemoteLogManagerConfig.of(oldConfig).remoteLogReaderThreads()) | ||
| remoteLogManager.resizeReaderThreadPool(RemoteLogManagerConfig.of(newConfig).remoteLogReaderThreads()) |
There was a problem hiding this comment.
The repeated calls to RemoteLogManagerConfig.of(newConfig) and RemoteLogManagerConfig.of(oldConfig) make the code less readable and slightly less efficient. It would be cleaner to store the results of these calls in local variables, similar to the previous implementation. This improves readability and avoids redundant method calls.
val newRLMConfig = RemoteLogManagerConfig.of(newConfig)
val oldRLMConfig = RemoteLogManagerConfig.of(oldConfig)
if (newRLMConfig.remoteLogManagerCopierThreadPoolSize() != oldRLMConfig.remoteLogManagerCopierThreadPoolSize())
remoteLogManager.resizeCopierThreadPool(newRLMConfig.remoteLogManagerCopierThreadPoolSize())
if (newRLMConfig.remoteLogManagerExpirationThreadPoolSize() != oldRLMConfig.remoteLogManagerExpirationThreadPoolSize())
remoteLogManager.resizeExpirationThreadPool(newRLMConfig.remoteLogManagerExpirationThreadPoolSize())
if (newRLMConfig.remoteLogManagerFollowerThreadPoolSize() != oldRLMConfig.remoteLogManagerFollowerThreadPoolSize())
remoteLogManager.resizeFollowerThreadPool(newRLMConfig.remoteLogManagerFollowerThreadPoolSize())
if (newRLMConfig.remoteLogReaderThreads() != oldRLMConfig.remoteLogReaderThreads())
remoteLogManager.resizeReaderThreadPool(newRLMConfig.remoteLogReaderThreads())| public static RemoteLogManagerConfig of(final AbstractConfig config) { | ||
| return INSTANCE_REF.updateAndGet(existing -> { | ||
| if (existing == null) { | ||
| return new RemoteLogManagerConfig(config); | ||
| } | ||
| if (existing.configRef.get() != config) { | ||
| existing.update(config); | ||
| } | ||
| return existing; | ||
| }); | ||
| } |
There was a problem hiding this comment.
The implementation of the of method using updateAndGet with a lambda that has side effects (existing.update(config)) can be hard to reason about and is generally discouraged. The function passed to updateAndGet may be re-applied on contention, which could lead to unexpected behavior with side effects.
A simpler and clearer approach would be to separate the instance creation from the configuration update. You can use updateAndGet with a side-effect-free lambda for atomic initialization, and then call the update method. This makes the logic more straightforward.
public static RemoteLogManagerConfig of(final AbstractConfig config) {
// Atomically create the instance if it doesn't exist.
RemoteLogManagerConfig instance = INSTANCE_REF.updateAndGet(existing -> existing == null ? new RemoteLogManagerConfig(config) : existing);
// Update the config. This is thread-safe as configRef is an AtomicReference.
// The last writer wins, which is acceptable for dynamic config updates.
instance.update(config);
return instance;
}|
/review
" |
|
Manual-approval option for PR-Agent is disabled. You can enable it via a configuration file |
PR Type
Enhancement
Description
Refactor
RemoteLogManagerConfigto use singleton patternReplace direct constructor usage with factory method
Update all references across codebase to use new pattern
Add dynamic configuration update capability
Changes diagram
Changes walkthrough 📝
10 files
Convert to singleton with factory methodUpdate to use factory methodUpdate to use factory methodUpdate to use factory methodUpdate to use factory methodUpdate to use factory methodUpdate to use factory methodUpdate to use factory methodRemove direct RemoteLogManagerConfig fieldUpdate to use factory method9 files
Add singleton behavior testsUpdate test to use factory methodUpdate test to use factory methodUpdate tests to use factory methodUpdate test to use factory methodUpdate tests to use factory methodUpdate tests to use factory methodUpdate test to use factory methodUpdate tests to use factory method