feat(clp-rust-utils): Add utility function to create MySQL connection pools from CLP-package's database config.#1538
Conversation
WalkthroughThese changes introduce MySQL database connectivity support to the clp-rust-utils library by adding the sqlx dependency, creating a new database module with MySQL pooling functionality, extending the error type to handle sqlx errors, and refactoring module visibility to make package configuration directly accessible. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes
Pre-merge checks and finishing touches✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 4
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (10)
components/clp-rust-utils/Cargo.toml(1 hunks)components/clp-rust-utils/src/database.rs(1 hunks)components/clp-rust-utils/src/database/mysql.rs(1 hunks)components/clp-rust-utils/src/error.rs(1 hunks)components/clp-rust-utils/src/job_config.rs(1 hunks)components/clp-rust-utils/src/job_config/search.rs(1 hunks)components/clp-rust-utils/src/lib.rs(1 hunks)components/clp-rust-utils/src/package_config.rs(1 hunks)components/clp-rust-utils/src/package_config/config.rs(1 hunks)components/clp-rust-utils/src/package_config/credentials.rs(1 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: haiqi96
Repo: y-scope/clp PR: 0
File: :0-0
Timestamp: 2025-07-29T14:04:13.769Z
Learning: User haiqi96 requested creating a GitHub issue to document a bug fix from PR #1136, which addressed MySQL compatibility issues with invalid SQL CAST operations in the WebUI component.
Learnt from: junhaoliao
Repo: y-scope/clp PR: 0
File: :0-0
Timestamp: 2025-10-22T21:02:31.113Z
Learning: Repository y-scope/clp: Maintain deterministic CI/builds for Rust; add a check to verify Cargo.lock is in sync with Cargo.toml without updating dependencies (non-mutating verification in clp-rust-checks workflow).
Learnt from: LinZhihao-723
Repo: y-scope/clp PR: 549
File: components/core/tests/test-ir_encoding_methods.cpp:1180-1186
Timestamp: 2024-10-01T07:59:11.208Z
Learning: In the context of loop constructs, LinZhihao-723 prefers using `while (true)` loops and does not consider alternative loop constructs necessarily more readable.
Learnt from: LinZhihao-723
Repo: y-scope/clp PR: 549
File: components/core/tests/test-ir_encoding_methods.cpp:1180-1186
Timestamp: 2024-10-08T15:52:50.753Z
Learning: In the context of loop constructs, LinZhihao-723 prefers using `while (true)` loops and does not consider alternative loop constructs necessarily more readable.
📚 Learning: 2025-10-22T21:02:31.113Z
Learnt from: junhaoliao
Repo: y-scope/clp PR: 0
File: :0-0
Timestamp: 2025-10-22T21:02:31.113Z
Learning: Repository y-scope/clp: Maintain deterministic CI/builds for Rust; add a check to verify Cargo.lock is in sync with Cargo.toml without updating dependencies (non-mutating verification in clp-rust-checks workflow).
Applied to files:
components/clp-rust-utils/src/package_config.rscomponents/clp-rust-utils/Cargo.tomlcomponents/clp-rust-utils/src/package_config/config.rs
📚 Learning: 2025-04-26T02:21:22.021Z
Learnt from: LinZhihao-723
Repo: y-scope/clp PR: 856
File: components/core/src/clp/ffi/ir_stream/search/utils.cpp:258-266
Timestamp: 2025-04-26T02:21:22.021Z
Learning: In the clp::ffi::ir_stream::search namespace, the design principle is that callers are responsible for type checking, not the called functions. If control flow reaches a function, input types should already be validated by the caller.
Applied to files:
components/clp-rust-utils/src/job_config.rs
🧬 Code graph analysis (5)
components/clp-rust-utils/src/lib.rs (1)
integration-tests/tests/fixtures/integration_test_config.py (1)
package_config(24-28)
components/clp-rust-utils/src/database/mysql.rs (1)
components/clp-py-utils/clp_py_utils/clp_config.py (1)
Database(162-249)
components/clp-rust-utils/src/package_config/credentials.rs (1)
components/clp-py-utils/clp_py_utils/clp_config.py (1)
Database(162-249)
components/clp-rust-utils/src/package_config/config.rs (1)
components/clp-py-utils/clp_py_utils/clp_config.py (4)
Package(135-159)Database(162-249)ResultsCache(327-341)StorageEngine(95-97)
components/clp-rust-utils/src/job_config/search.rs (1)
components/job-orchestration/job_orchestration/scheduler/job_config.py (2)
AggregationConfig(61-66)SearchJobConfig(85-103)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: package-image
- GitHub Check: rust-checks (ubuntu-22.04)
- GitHub Check: lint-check (ubuntu-24.04)
- GitHub Check: rust-checks (ubuntu-24.04)
🔇 Additional comments (11)
components/clp-rust-utils/src/database.rs (1)
1-1: LGTM!Clean module declaration following Rust conventions.
components/clp-rust-utils/src/lib.rs (2)
2-2: LGTM!Appropriately exposes the new database module as part of the public API.
5-5: LGTM!Appropriately exposes the new package_config module as part of the public API.
components/clp-rust-utils/src/database/mysql.rs (2)
25-30: Verify MySqlConnectOptions configuration is complete.The current implementation sets the core connection parameters. If additional options (ssl, compression, autocommit) are added to
DatabaseConfig, ensure they are properly applied here. For example:let mut mysql_options = sqlx::mysql::MySqlConnectOptions::new() .host(&config.host) .port(config.port) .database(&config.name) .username(&credentials.user) .password(credentials.password.expose_secret()); // Apply optional SSL certificate if configured if let Some(ref ssl_cert) = config.ssl_cert { mysql_options = mysql_options.ssl_ca(ssl_cert); }
8-19: Documentation looks good.The function documentation clearly describes the return value and error conditions. The reference to
sqlx::mysql::MySqlPoolOptions::connect_within the errors section is helpful.components/clp-rust-utils/src/error.rs (1)
11-12: LGTM!The
SqlxErrorvariant follows the existing error handling pattern and properly enables automatic error conversion fromsqlx::Errorusing the#[from]attribute. This integrates well with thecreate_mysql_poolfunction's error handling.components/clp-rust-utils/src/job_config.rs (1)
4-5: Code changes are approved.The
search.rsmodule exists and contains appropriate public exports (constants, types, structs, and enums). The module re-export pattern follows standard Rust conventions and is correctly integrated intojob_config.rs.components/clp-rust-utils/src/package_config.rs (2)
3-3: No issues found—the Config re-export is valid.Verification confirms that
config.rsexists and defines theConfigstruct (lines 4–13) with theDatabasefield (line 17) that the re-export assumes. The type is properly structured and available for use inmysql.rs.
1-7: Default paths are correct and consistent with CLP package conventions.Verification confirms that
"etc/clp-config.yml"and"etc/credentials.yml"match the default paths used throughout the CLP package (e.g.,components/clp-py-utils/clp_py_utils/clp_config.pyand deployment scripts). The module structure follows Rust conventions and the paths align with existing CLP configuration standards.components/clp-rust-utils/Cargo.toml (2)
15-15: No issues detected — dependency version is current and secure.✓
sqlx 0.8.6is the latest stable version
✓ No security vulnerabilities affect this version (advisory applies only to versions ≤ 0.8.0)
✓Cargo.lockhas been updated and is in sync with the workspace configuration
11-15: Cargo.lock has been updated correctly; however, fix the Rust edition.All new dependencies (num_enum 0.7.5, rmp-serde 1.3.0, secrecy 0.10.3, serde 1.0.228, sqlx 0.8.6) are present in Cargo.lock with correct versions and no unintended changes to other packages.
However,
components/clp-rust-utils/Cargo.tomlspecifiesedition = "2024", which is not a valid Rust edition. Valid editions are2015,2018, and2021. This will prevent compilation. Update to a valid edition.⛔ Skipped due to learnings
Learnt from: junhaoliao Repo: y-scope/clp PR: 0 File: :0-0 Timestamp: 2025-10-22T21:01:31.391Z Learning: In y-scope/clp, for the clp-rust-checks workflow (GitHub Actions) and the Taskfile target deps:lock:check-rust, we should verify Rust Cargo.lock is in sync with Cargo.toml using a non-mutating method (e.g., cargo metadata --locked / cargo check --locked) to keep CI deterministic and avoid updating dependencies during validation.Learnt from: junhaoliao Repo: y-scope/clp PR: 0 File: :0-0 Timestamp: 2025-10-22T21:02:31.113Z Learning: Repository y-scope/clp: Maintain deterministic CI/builds for Rust; add a check to verify Cargo.lock is in sync with Cargo.toml without updating dependencies (non-mutating verification in clp-rust-checks workflow).Learnt from: LinZhihao-723 Repo: y-scope/clp PR: 1448 File: components/clp-rust-utils/Cargo.toml:4-4 Timestamp: 2025-10-20T22:20:45.733Z Learning: Rust edition 2024 is a valid edition value in Cargo.toml files. It was released with Rust 1.85 on February 20, 2025.
hoophalab
left a comment
There was a problem hiding this comment.
We need to resolve conflicts, but it looks good overall.
I'll test the function after the conflicts are resolved.
| config: &DatabaseConfig, | ||
| credentials: &DatabaseCredentials, | ||
| max_connections: u32, | ||
| ) -> Result<sqlx::MySqlPool, crate::Error> { |
There was a problem hiding this comment.
I'm wondering whether to return the raw sqlx::Error directly or wrap it in our clp_rust_utils::Error type. My thinking is that returning sqlx::Error directly might simplify error handling for the function user, as they won't have to unpack or convert from clp_rust_utils::Error, but wrapping it in clp_rust_utils::Error might be a more common practice. What are your thoughts?
| ) -> Result<sqlx::MySqlPool, crate::Error> { | |
| ) -> Result<sqlx::MySqlPool, sqlx::Error> { |
There was a problem hiding this comment.
Since this is a high-level API, I suspect users will actually need to look into the error and do some special handling. Instead, they might just log the error and exit. In this case, they probably don't need to understand the details of the error. Thus, I prefer to return a general library error to (1) hide the error details, (2) allow users to handle it as a general library error.
There was a problem hiding this comment.
Actionable comments posted: 3
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (5)
components/clp-rust-utils/src/clp_config.rs(1 hunks)components/clp-rust-utils/src/clp_config/package.rs(0 hunks)components/clp-rust-utils/src/database/mysql.rs(1 hunks)components/clp-rust-utils/src/error.rs(1 hunks)components/clp-rust-utils/src/lib.rs(1 hunks)
💤 Files with no reviewable changes (1)
- components/clp-rust-utils/src/clp_config/package.rs
🧰 Additional context used
🧠 Learnings (4)
📓 Common learnings
Learnt from: haiqi96
Repo: y-scope/clp PR: 0
File: :0-0
Timestamp: 2025-07-29T14:04:13.769Z
Learning: User haiqi96 requested creating a GitHub issue to document a bug fix from PR #1136, which addressed MySQL compatibility issues with invalid SQL CAST operations in the WebUI component.
Learnt from: junhaoliao
Repo: y-scope/clp PR: 0
File: :0-0
Timestamp: 2025-10-22T21:02:31.113Z
Learning: Repository y-scope/clp: Maintain deterministic CI/builds for Rust; add a check to verify Cargo.lock is in sync with Cargo.toml without updating dependencies (non-mutating verification in clp-rust-checks workflow).
Learnt from: LinZhihao-723
Repo: y-scope/clp PR: 549
File: components/core/tests/test-ir_encoding_methods.cpp:1180-1186
Timestamp: 2024-10-01T07:59:11.208Z
Learning: In the context of loop constructs, LinZhihao-723 prefers using `while (true)` loops and does not consider alternative loop constructs necessarily more readable.
Learnt from: LinZhihao-723
Repo: y-scope/clp PR: 549
File: components/core/tests/test-ir_encoding_methods.cpp:1180-1186
Timestamp: 2024-10-08T15:52:50.753Z
Learning: In the context of loop constructs, LinZhihao-723 prefers using `while (true)` loops and does not consider alternative loop constructs necessarily more readable.
📚 Learning: 2025-11-03T16:17:40.214Z
Learnt from: hoophalab
Repo: y-scope/clp PR: 1535
File: components/clp-rust-utils/src/clp_config/package/config.rs:47-61
Timestamp: 2025-11-03T16:17:40.214Z
Learning: In the y-scope/clp repository, the `ApiServer` struct in `components/clp-rust-utils/src/clp_config/package/config.rs` is a Rust-native configuration type and does not mirror any Python code, unlike other structs in the same file (Config, Database, ResultsCache, Package) which are mirrors of Python definitions.
Applied to files:
components/clp-rust-utils/src/clp_config.rscomponents/clp-rust-utils/src/lib.rs
📚 Learning: 2025-10-22T21:02:31.113Z
Learnt from: junhaoliao
Repo: y-scope/clp PR: 0
File: :0-0
Timestamp: 2025-10-22T21:02:31.113Z
Learning: Repository y-scope/clp: Maintain deterministic CI/builds for Rust; add a check to verify Cargo.lock is in sync with Cargo.toml without updating dependencies (non-mutating verification in clp-rust-checks workflow).
Applied to files:
components/clp-rust-utils/src/clp_config.rs
📚 Learning: 2025-01-29T22:16:52.665Z
Learnt from: haiqi96
Repo: y-scope/clp PR: 700
File: components/core/src/clp/streaming_archive/ArchiveMetadata.cpp:29-46
Timestamp: 2025-01-29T22:16:52.665Z
Learning: In C++ code, when throwing exceptions, prefer throwing the original return code (RC) from lower-level operations instead of a generic error code to preserve error context and aid in debugging.
Applied to files:
components/clp-rust-utils/src/database/mysql.rs
🧬 Code graph analysis (1)
components/clp-rust-utils/src/database/mysql.rs (1)
components/clp-py-utils/clp_py_utils/clp_config.py (1)
Database(162-249)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
- GitHub Check: package-image
- GitHub Check: rust-checks (ubuntu-24.04)
- GitHub Check: rust-checks (ubuntu-22.04)
- GitHub Check: rust-checks (macos-15)
- GitHub Check: lint-check (macos-15)
🔇 Additional comments (5)
components/clp-rust-utils/src/lib.rs (1)
2-2: LGTM!The addition of the database module appropriately expands the public API surface to expose the new MySQL connection pool functionality.
components/clp-rust-utils/src/clp_config.rs (1)
1-1: Verify that the breaking API change is documented.Changing the
packagemodule visibility to public and removing the wildcard re-export (pub use package::*;) is a breaking change. Users who previously accessed types likeConfigdirectly fromclp_configwill now need to use the full pathclp_config::package::config::Config.Ensure this breaking change is documented in the PR description, changelog, or migration guide.
components/clp-rust-utils/src/error.rs (2)
6-9: Document the breaking changes from error variant renames.The renames (
MsgpackEncodeError→MsgpackEncode,IoError→Io) improve consistency with Rust conventions by removing the redundant "Error" suffix. However, these are breaking changes for any code that pattern-matches on these variants.Ensure these breaking changes are documented alongside the API changes in
clp_config.rs.
11-12: LGTM!The new
Sqlxerror variant correctly enables automatic conversion fromsqlx::Errortocrate::Error, supporting error propagation in the new database module.components/clp-rust-utils/src/database/mysql.rs (1)
1-6: LGTM!The imports are correct and follow security best practices by using
ExposeSecretfrom thesecrecycrate to safely handle password credentials.
| /// Creates a new `MySQL` connection pool using the provided configuration and credentials. | ||
| /// | ||
| /// # Return | ||
| /// | ||
| /// A newly created `MySQL` connection pool configured with the specified maximum number of | ||
| /// connections. | ||
| /// | ||
| /// # Errors | ||
| /// | ||
| /// Returns an error if: | ||
| /// | ||
| /// * Forwards [`sqlx::mysql::MySqlPoolOptions::connect_with`]'s errors on failure. |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Consider expanding the error documentation.
While the documentation correctly references the underlying sqlx errors, consider adding a brief description of common failure scenarios (e.g., network connectivity issues, invalid credentials, database not found) to help users understand when this function might fail.
🤖 Prompt for AI Agents
In components/clp-rust-utils/src/database/mysql.rs around lines 8 to 19, expand
the function's error documentation to enumerate common failure scenarios in
addition to referencing sqlx errors: add short bullets describing network
connectivity/timeouts, invalid credentials/authentication failures, database or
host not found, TLS/Certificate issues, and connection pool exhaustion; keep the
reference to sqlx::mysql::MySqlPoolOptions::connect_with and mention that those
underlying errors occur for these cases so callers know what to expect.
| pub async fn create_mysql_pool( | ||
| config: &DatabaseConfig, | ||
| credentials: &DatabaseCredentials, | ||
| max_connections: u32, | ||
| ) -> Result<sqlx::MySqlPool, crate::Error> { | ||
| let mysql_options = sqlx::mysql::MySqlConnectOptions::new() | ||
| .host(&config.host) | ||
| .port(config.port) | ||
| .database(&config.name) | ||
| .username(&credentials.user) | ||
| .password(credentials.password.expose_secret()); | ||
|
|
||
| Ok(sqlx::mysql::MySqlPoolOptions::new() | ||
| .max_connections(max_connections) | ||
| .connect_with(mysql_options) | ||
| .await?) | ||
| } |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Consider adding connection timeout configuration.
The current implementation doesn't specify a connection timeout, which could lead to indefinite hangs if the database is unreachable. Consider adding .connect_timeout(Duration::from_secs(30)) or a similar timeout to MySqlConnectOptions for better resilience.
Apply this diff to add a connection timeout:
let mysql_options = sqlx::mysql::MySqlConnectOptions::new()
.host(&config.host)
.port(config.port)
.database(&config.name)
.username(&credentials.user)
- .password(credentials.password.expose_secret());
+ .password(credentials.password.expose_secret())
+ .connect_timeout(std::time::Duration::from_secs(30));📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| pub async fn create_mysql_pool( | |
| config: &DatabaseConfig, | |
| credentials: &DatabaseCredentials, | |
| max_connections: u32, | |
| ) -> Result<sqlx::MySqlPool, crate::Error> { | |
| let mysql_options = sqlx::mysql::MySqlConnectOptions::new() | |
| .host(&config.host) | |
| .port(config.port) | |
| .database(&config.name) | |
| .username(&credentials.user) | |
| .password(credentials.password.expose_secret()); | |
| Ok(sqlx::mysql::MySqlPoolOptions::new() | |
| .max_connections(max_connections) | |
| .connect_with(mysql_options) | |
| .await?) | |
| } | |
| pub async fn create_mysql_pool( | |
| config: &DatabaseConfig, | |
| credentials: &DatabaseCredentials, | |
| max_connections: u32, | |
| ) -> Result<sqlx::MySqlPool, crate::Error> { | |
| let mysql_options = sqlx::mysql::MySqlConnectOptions::new() | |
| .host(&config.host) | |
| .port(config.port) | |
| .database(&config.name) | |
| .username(&credentials.user) | |
| .password(credentials.password.expose_secret()) | |
| .connect_timeout(std::time::Duration::from_secs(30)); | |
| Ok(sqlx::mysql::MySqlPoolOptions::new() | |
| .max_connections(max_connections) | |
| .connect_with(mysql_options) | |
| .await?) | |
| } |
🤖 Prompt for AI Agents
In components/clp-rust-utils/src/database/mysql.rs around lines 20 to 36, the
MySqlConnectOptions lacks a connection timeout; modify the options builder to
include .connect_timeout(Duration::from_secs(30)) (or a configurable Duration)
in the chain when constructing mysql_options, and ensure std::time::Duration is
imported or referenced so the code compiles; keep the rest of the pool creation
unchanged.
| let mysql_options = sqlx::mysql::MySqlConnectOptions::new() | ||
| .host(&config.host) |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Consider adding localhost socket connection handling.
The Python implementation in components/clp-py-utils/clp_py_utils/clp_config.py (Lines 183-185) supports transforming "localhost" to "127.0.0.1" when disable_localhost_socket_connection is enabled. This prevents MySQL from using Unix domain sockets and forces TCP connections.
Consider whether this feature should be added for parity with the Python implementation, either as a function parameter or as part of DatabaseConfig.
… pools from CLP-package's database config. (y-scope#1538) Co-authored-by: hoophalab <200652805+hoophalab@users.noreply.github.com>
Description
NOTE: This PR depends on #1535.
Checklist
breaking change.
Validation performed
Summary by CodeRabbit
New Features
Chores