Conversation
📝 WalkthroughWalkthroughAdds support for an alternative API key (alt_api_key) across the codebase. ChannelService exposes a new public field and updated constructor; all call sites and tests were updated. ServiceConfig gains alt_api_key and its validation now considers both keys. AuthKeys now holds alt_read_write and alt_jwt_parser; get_jwt_parser returns primary and alternative parsers and validation/authorization logic tries both. Snapshot recovery and snapshot-transfer flows prefer the primary API key but fall back to the alternative. Tests and RPC handler extraction updated to exercise alternate-key scenarios. Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 inconclusive)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
src/settings.rs (1)
315-321: Unnecessary.clone()calls onOption<String>.Using
.as_ref()instead of.clone()avoids allocating new strings just to check emptiness.🔎 Proposed fix
- let all_keys_are_empty = self.service.api_key.clone().unwrap_or_default().is_empty() - && self - .service - .alt_api_key - .clone() - .unwrap_or_default() - .is_empty(); + let all_keys_are_empty = self.service.api_key.as_deref().unwrap_or_default().is_empty() + && self + .service + .alt_api_key + .as_deref() + .unwrap_or_default() + .is_empty();tests/consensus_tests/test_shard_snapshot_transfer.py (1)
132-132:assert_project_rootis deprecated and does nothing.Per the relevant code snippet at
tests/consensus_tests/utils.py:111-113, this function is now a no-op. Consider removing this call.src/common/auth/mod.rs (1)
121-146: Well-structured dual-parser JWT validation logic.The approach of trying both parsers and taking the first successful claim is correct for rolling key rotation. The error handling appropriately returns the first error when no parser can decode the token.
Minor typo: "JTW" should be "JWT" on lines 143 and 148.
🔎 Proposed fix for typos
- // JTW parser exists, but can't decode the token + // JWT parser exists, but can't decode the token if let Some(error) = errors.into_iter().next() { return Err(error); } - // No JTW parser configured + // No JWT parser configured Err(AuthError::Unauthorized(
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (14)
lib/collection/src/shards/channel_service.rslib/collection/src/shards/transfer/snapshot.rslib/collection/tests/integration/common/mod.rslib/collection/tests/integration/continuous_snapshot_test.rslib/collection/tests/integration/snapshot_recovery_test.rslib/storage/tests/integration/alias_tests.rssrc/common/auth/mod.rssrc/consensus.rssrc/main.rssrc/settings.rssrc/tonic/api/snapshots_api.rstests/consensus_tests/auth_tests/test_jwt_validation.pytests/consensus_tests/auth_tests/utils.pytests/consensus_tests/test_shard_snapshot_transfer.py
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs
📄 CodeRabbit inference engine (.github/review-rules.md)
**/*.rs: Prefer explicitSomeType::from(x)over implicitx.into()in Rust
In new code, don't usetransmute_from_u8,transmute_to_u8,transmute_from_u8_to_slice,transmute_from_u8_to_mut_slice,transmute_to_u8_slice- usebytemuckorzerocopycrates instead
Prefer explicit exhaustive matches over catch-all_arm in Rust match expressions, so we don't miss handling new enum variants when they are added later (exceptions: tests and benchmarks, or when sure new variants won't affect logic)
Prefer explicit field ignoring using: _over using..in Rust struct destructuring (exceptions: tests and benchmarks, or when sure new fields won't affect logic)
Files:
src/tonic/api/snapshots_api.rslib/collection/tests/integration/continuous_snapshot_test.rslib/collection/src/shards/transfer/snapshot.rslib/collection/tests/integration/common/mod.rssrc/settings.rslib/collection/src/shards/channel_service.rslib/storage/tests/integration/alias_tests.rssrc/consensus.rssrc/common/auth/mod.rssrc/main.rslib/collection/tests/integration/snapshot_recovery_test.rs
🧠 Learnings (1)
📚 Learning: 2025-12-23T22:40:56.292Z
Learnt from: coszio
Repo: qdrant/qdrant PR: 7729
File: tests/consensus_tests/auth_tests/test_jwt_access.py:381-381
Timestamp: 2025-12-23T22:40:56.292Z
Learning: The `/cluster/telemetry` endpoint in tests/consensus_tests/auth_tests/test_jwt_access.py intentionally allows collection-scoped read tokens (coll_r defaults to True) because it mimics the `/telemetry` endpoint pattern where content is filtered based on access level, rather than following the `/cluster` pattern which denies collection-scoped access.
Applied to files:
tests/consensus_tests/auth_tests/test_jwt_validation.py
🧬 Code graph analysis (8)
src/tonic/api/snapshots_api.rs (2)
src/tonic/api/mod.rs (1)
validate_and_log(27-31)src/actix/api/snapshot_api.rs (1)
recover_shard_snapshot(421-452)
lib/collection/tests/integration/continuous_snapshot_test.rs (1)
lib/collection/src/shards/channel_service.rs (1)
new(37-50)
lib/collection/tests/integration/common/mod.rs (1)
lib/collection/src/shards/channel_service.rs (1)
new(37-50)
tests/consensus_tests/auth_tests/test_jwt_validation.py (1)
tests/consensus_tests/auth_tests/utils.py (1)
encode_jwt(43-44)
lib/collection/src/shards/channel_service.rs (1)
lib/storage/src/content_manager/toc/mod.rs (1)
new(94-214)
lib/storage/tests/integration/alias_tests.rs (1)
lib/storage/src/content_manager/toc/mod.rs (1)
new(94-214)
src/common/auth/mod.rs (2)
lib/collection/src/shards/channel_service.rs (1)
new(37-50)src/tonic/auth.rs (1)
new(93-97)
tests/consensus_tests/test_shard_snapshot_transfer.py (1)
tests/consensus_tests/utils.py (1)
assert_project_root(112-114)
🪛 Ruff (0.14.10)
tests/consensus_tests/auth_tests/test_jwt_validation.py
65-65: Found useless expression. Either assign it to a variable or remove it.
(B018)
65-65: Undefined name s
(F821)
tests/consensus_tests/auth_tests/utils.py
12-12: Possible hardcoded password assigned to: "ALT_SECRET"
(S105)
tests/consensus_tests/test_shard_snapshot_transfer.py
132-132: assert_project_root may be undefined, or defined from star imports
(F405)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: rust-tests (macos-latest)
- GitHub Check: rust-tests (windows-latest)
- GitHub Check: rust-tests (ubuntu-latest)
- GitHub Check: e2e-tests
- GitHub Check: integration-tests
- GitHub Check: test-shard-snapshot-api-s3-minio
- GitHub Check: test-consistency
- GitHub Check: integration-tests-consensus
- GitHub Check: test-consensus-compose
- GitHub Check: lint
- GitHub Check: Test Python bindings
🔇 Additional comments (17)
tests/consensus_tests/auth_tests/utils.py (1)
12-12: LGTM! Alternative secret properly configured for testing.The addition of
ALT_SECRETand its environment variable configuration correctly enables testing of the alternative API key functionality. The static analysis warning about hardcoded passwords is a false positive since this is test code with intentional test credentials.Also applies to: 24-24
src/tonic/api/snapshots_api.rs (1)
262-285: LGTM! Explicit destructuring improves code clarity.The refactoring to explicitly destructure the request fields into local variables before use is a good practice that makes the code more readable and maintainable. The functional behavior remains unchanged.
tests/consensus_tests/auth_tests/test_jwt_validation.py (1)
62-63: LGTM! Alternative secret validation properly tested.The test correctly validates JWT functionality with both the primary and alternative secrets, ensuring the alternative API key path works as expected.
Also applies to: 79-79
lib/collection/tests/integration/continuous_snapshot_test.rs (1)
83-83: LGTM! Constructor call updated correctly.The
ChannelService::newcall has been properly updated to include the thirdalt_api_keyparameter asNone, matching the new constructor signature while maintaining existing test behavior.src/consensus.rs (1)
1489-1489: LGTM! Test constructor calls updated correctly.Both
ChannelService::newcalls in the test module have been properly updated to include the thirdalt_api_keyparameter asNone, maintaining existing test behavior while conforming to the new constructor signature.Also applies to: 1514-1514
lib/collection/src/shards/transfer/snapshot.rs (1)
239-253: LGTM! Alternative API key fallback implemented correctly.The code properly derives
local_api_keyby preferring the primaryapi_keyand falling back toalt_api_key. The comment clearly explains the intent, and the implementation usingas_deref().or()is idiomatic Rust for this pattern. This enables the remote instance to authenticate with either key during snapshot recovery.src/main.rs (1)
351-355: LGTM! Alternative API key properly wired from configuration.The
ChannelServiceinitialization correctly passes bothapi_keyandalt_api_keyfrom the service settings, enabling the rolling key rotation feature in production deployments.lib/collection/tests/integration/common/mod.rs (1)
104-104: LGTM! Test helper functions updated correctly.Both
ChannelService::newcalls in the test utility functions have been properly updated to include the thirdalt_api_keyparameter asNone, maintaining existing test behavior while conforming to the new constructor signature.Also applies to: 139-139
lib/collection/tests/integration/snapshot_recovery_test.rs (1)
80-80: LGTM!The
ChannelService::newcalls are correctly updated to include the newalt_api_keyparameter asNone, aligning with the extended constructor signature.Also applies to: 140-140
lib/storage/tests/integration/alias_tests.rs (1)
94-94: LGTM!Constructor call correctly updated to include the new
alt_api_keyparameter.lib/collection/src/shards/channel_service.rs (1)
30-33: LGTM!The new
alt_api_keyfield is well-documented and properly integrated into the struct, constructor, andDefaultimplementation for rolling key rotation support.src/settings.rs (1)
38-40: LGTM!The new
alt_api_keyfield is properly added with clear documentation explaining its purpose for rolling key rotation.tests/consensus_tests/test_shard_snapshot_transfer.py (2)
125-127: Potential issue: reusingtmp_pathacross multiple cluster starts may cause port or state conflicts.Each call to
shard_snapshot_transfer_with_api_keystarts a new cluster using the sametmp_path. This could lead to residual files or port collisions between runs. Consider using unique subdirectories for each scenario.🔎 Proposed fix
- shard_snapshot_transfer_with_api_key(tmp_path, env, headers) - shard_snapshot_transfer_with_api_key(tmp_path, alt_env, headers) - shard_snapshot_transfer_with_api_key(tmp_path, alt_env, headers_alt) + shard_snapshot_transfer_with_api_key(tmp_path / "primary", env, headers) + shard_snapshot_transfer_with_api_key(tmp_path / "alt_env_primary_header", alt_env, headers) + shard_snapshot_transfer_with_api_key(tmp_path / "alt_env_alt_header", alt_env, headers_alt)
131-189: Test helper logic looks correct for API key validation.The helper properly threads headers through all API calls (collection creation, point upserts, cluster info, and point counts), ensuring authentication is validated throughout the shard transfer flow.
src/common/auth/mod.rs (3)
52-67: LGTM!The
get_jwt_parserfunction correctly returns a tuple of optional parsers for both primary and alternative API keys, enabling rolling key rotation support.
202-212: LGTM - secure key comparison for both primary and alternative keys.The
can_writefunction correctly uses constant-time comparison (ct_eq) for bothread_writeandalt_read_writekeys, preventing timing attacks during authentication.
73-91: LGTM!The
try_createfunction correctly initializes both JWT parsers and all key fields, properly handling the case where only some keys are configured.
src/settings.rs
Outdated
| let any_api_key_is_short = self.service.api_key.clone().unwrap_or_default().len() | ||
| < JWT_RECOMMENDED_SECRET_LENGTH | ||
| || self.service.alt_api_key.clone().unwrap_or_default().len() | ||
| < JWT_RECOMMENDED_SECRET_LENGTH; |
There was a problem hiding this comment.
Logic issue: any_api_key_is_short will trigger false warnings when only one key is configured.
When alt_api_key is None (common case), unwrap_or_default() returns an empty string with length 0, which is always < JWT_RECOMMENDED_SECRET_LENGTH. This will cause a warning even when only the primary api_key is configured with proper length.
The check should only validate length for keys that are actually set:
🔎 Proposed fix
- let any_api_key_is_short = self.service.api_key.clone().unwrap_or_default().len()
- < JWT_RECOMMENDED_SECRET_LENGTH
- || self.service.alt_api_key.clone().unwrap_or_default().len()
- < JWT_RECOMMENDED_SECRET_LENGTH;
+ let any_api_key_is_short = self
+ .service
+ .api_key
+ .as_ref()
+ .is_some_and(|k| k.len() < JWT_RECOMMENDED_SECRET_LENGTH)
+ || self
+ .service
+ .alt_api_key
+ .as_ref()
+ .is_some_and(|k| k.len() < JWT_RECOMMENDED_SECRET_LENGTH);🤖 Prompt for AI Agents
In src/settings.rs around lines 323 to 326, the current any_api_key_is_short
check uses unwrap_or_default() so a None alt_api_key becomes an empty string and
always triggers the "short" warning; change the logic to only validate lengths
for keys that are actually set (i.e., check Option::as_ref().map_or(false, |k|
k.len() < JWT_RECOMMENDED_SECRET_LENGTH) for each key and combine with ||), so
only present keys are length-checked and absent keys do not cause false
warnings.
There was a problem hiding this comment.
@generall You're welcome! Happy to help catch that. 👍
🧠 Learnings used
Learnt from: generall
Repo: qdrant/qdrant PR: 7006
File: docs/redoc/master/openapi.json:13645-13652
Timestamp: 2025-08-10T18:26:12.443Z
Learning: In qdrant, docs/redoc/master/openapi.json is auto-generated from the REST schemas. To change field docs, edit lib/api/src/rest/schema.rs (e.g., add doc comments or #[schemars(description = ...)]). Specifically, UpdateVectors.update_filter lacked a description and should state: "If specified, only update vectors for points that match this filter; points not matching the filter are left unchanged."
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
src/settings.rs (1)
315-321: Consider avoiding unnecessary clones.The
.clone().unwrap_or_default()pattern allocates newStringinstances unnecessarily. Using.as_deref()with.map_or()would be more efficient.🔎 Proposed refactor
- let all_keys_are_empty = self.service.api_key.clone().unwrap_or_default().is_empty() - && self - .service - .alt_api_key - .clone() - .unwrap_or_default() - .is_empty(); + let all_keys_are_empty = self + .service + .api_key + .as_deref() + .map_or(true, str::is_empty) + && self + .service + .alt_api_key + .as_deref() + .map_or(true, str::is_empty);
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/settings.rs
🧰 Additional context used
📓 Path-based instructions (1)
**/*.rs
📄 CodeRabbit inference engine (.github/review-rules.md)
**/*.rs: Prefer explicitSomeType::from(x)over implicitx.into()in Rust
In new code, don't usetransmute_from_u8,transmute_to_u8,transmute_from_u8_to_slice,transmute_from_u8_to_mut_slice,transmute_to_u8_slice- usebytemuckorzerocopycrates instead
Prefer explicit exhaustive matches over catch-all_arm in Rust match expressions, so we don't miss handling new enum variants when they are added later (exceptions: tests and benchmarks, or when sure new variants won't affect logic)
Prefer explicit field ignoring using: _over using..in Rust struct destructuring (exceptions: tests and benchmarks, or when sure new fields won't affect logic)
Files:
src/settings.rs
🧠 Learnings (3)
📚 Learning: 2025-08-10T18:26:12.443Z
Learnt from: generall
Repo: qdrant/qdrant PR: 7006
File: docs/redoc/master/openapi.json:13645-13652
Timestamp: 2025-08-10T18:26:12.443Z
Learning: In qdrant, docs/redoc/master/openapi.json is auto-generated from the REST schemas. To change field docs, edit lib/api/src/rest/schema.rs (e.g., add doc comments or #[schemars(description = ...)]). Specifically, UpdateVectors.update_filter lacked a description and should state: "If specified, only update vectors for points that match this filter; points not matching the filter are left unchanged."
Applied to files:
src/settings.rs
📚 Learning: 2025-08-10T18:25:16.206Z
Learnt from: generall
Repo: qdrant/qdrant PR: 7006
File: docs/redoc/master/openapi.json:10626-10634
Timestamp: 2025-08-10T18:25:16.206Z
Learning: In Qdrant, docs/redoc/master/openapi.json is autogenerated from the Rust REST schema definitions (e.g., lib/api/src/rest/schema.rs). To change OpenAPI descriptions, update the Rust struct field doc comments/attributes rather than editing the JSON directly.
Applied to files:
src/settings.rs
📚 Learning: 2025-08-10T18:26:33.017Z
Learnt from: generall
Repo: qdrant/qdrant PR: 7006
File: docs/redoc/master/openapi.json:10393-10401
Timestamp: 2025-08-10T18:26:33.017Z
Learning: In qdrant, docs/redoc/master/openapi.json is auto-generated. Any description changes must be made in the source (e.g., lib/api/src/rest/schema.rs and/or lib/api/src/grpc/proto/points.proto), not edited directly in the JSON.
Applied to files:
src/settings.rs
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: rust-tests (macos-latest)
- GitHub Check: rust-tests (ubuntu-latest)
- GitHub Check: rust-tests (windows-latest)
- GitHub Check: lint
- GitHub Check: e2e-tests
- GitHub Check: test-consistency
- GitHub Check: integration-tests
- GitHub Check: test-consensus-compose
- GitHub Check: integration-tests-consensus
- GitHub Check: test-shard-snapshot-api-s3-minio
🔇 Additional comments (2)
src/settings.rs (2)
38-40: LGTM! Clear field addition with helpful documentation.The new
alt_api_keyfield is well-documented and properly typed for optional rolling key rotation support.
323-333: LGTM! The key length validation logic correctly handles optional keys.The use of
.flatten()properly filters outNonekeys before computing the minimum length, preventing false warnings when only one key is configured. This addresses the concern from the previous review.When both keys are
None,min_lengthbecomes 0, but theall_keys_are_emptycheck at line 337 is evaluated first, so the warning at line 340 is correctly skipped.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/consensus_tests/test_shard_snapshot_transfer.py (1)
138-138: Remove the deprecated function call.According to the deprecation notice in
utils.py,assert_project_root()is no longer needed as paths are resolved relative to__file__. The function body is now justpass. Consider removing this call here and in other test functions (lines 42, 205, 305).Note: Static analysis correctly flags this as potentially undefined due to the star import pattern on line 7.
🔎 Verification script to find all usages
#!/bin/bash # Find all calls to the deprecated assert_project_root function rg -n "assert_project_root\(\)" --type=py tests/consensus_tests/
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/consensus_tests/test_shard_snapshot_transfer.py
🧰 Additional context used
🧬 Code graph analysis (1)
tests/consensus_tests/test_shard_snapshot_transfer.py (1)
tests/consensus_tests/utils.py (1)
assert_project_root(112-114)
🪛 Ruff (0.14.10)
tests/consensus_tests/test_shard_snapshot_transfer.py
138-138: assert_project_root may be undefined, or defined from star imports
(F405)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: test-consensus-compose
- GitHub Check: lint
- GitHub Check: test-shard-snapshot-api-s3-minio
- GitHub Check: integration-tests
- GitHub Check: e2e-tests
- GitHub Check: Test Python bindings
- GitHub Check: test-consistency
- GitHub Check: rust-tests (ubuntu-latest)
- GitHub Check: rust-tests (windows-latest)
- GitHub Check: integration-tests-consensus
- GitHub Check: rust-tests (macos-latest)
🔇 Additional comments (3)
tests/consensus_tests/test_shard_snapshot_transfer.py (3)
108-124: Good test setup for API key rotation scenarios.The configuration correctly tests three rotation states: (1) primary key only, (2) both keys configured with primary key usage, and (3) both keys configured with alternate key usage. This validates the zero-downtime rotation workflow described in the PR objectives.
125-135: Well-structured test scenarios with proper isolation.Creating separate directories for each scenario ensures proper test isolation. The three invocations systematically exercise the API key rotation workflow.
137-195: Excellent refactor to enable multi-scenario testing.The helper function extraction eliminates code duplication and enables systematic testing of different API key configurations. All API interactions correctly thread the
headersparameter through the call chain.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tests/consensus_tests/test_shard_snapshot_transfer.py (2)
104-145: Consider more descriptive test names.The numeric suffixes (_1, _2, _3) don't convey the distinct scenarios being tested. More descriptive names would improve test clarity:
test_shard_snapshot_transfer_with_single_api_key(test 1: only primary key)test_shard_snapshot_transfer_with_primary_key_when_both_configured(test 2: both keys, using primary)test_shard_snapshot_transfer_with_alternate_key(test 3: both keys, using alternate)
148-148: Remove deprecated function call.The
assert_project_root()function is deprecated and no longer performs any action. Based on the documentation inutils.py, paths are now resolved relative to__file__and this assertion is unnecessary.🔎 Proposed fix
def shard_snapshot_transfer_with_api_key(tmp_path: pathlib.Path, env, headers): - assert_project_root() - # seed port to reuse the same port for the restarted nodes peer_api_uris, peer_dirs, bootstrap_uri = start_cluster(tmp_path, N_PEERS, 20000, extra_env=env, headers=headers)
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/consensus_tests/test_shard_snapshot_transfer.py
🧰 Additional context used
🧬 Code graph analysis (1)
tests/consensus_tests/test_shard_snapshot_transfer.py (1)
tests/consensus_tests/utils.py (1)
assert_project_root(112-114)
🪛 Ruff (0.14.10)
tests/consensus_tests/test_shard_snapshot_transfer.py
107-107: Local variable alt_api_key is assigned to but never used
Remove assignment to unused variable alt_api_key
(F841)
148-148: assert_project_root may be undefined, or defined from star imports
(F405)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (11)
- GitHub Check: test-shard-snapshot-api-s3-minio
- GitHub Check: rust-tests (macos-latest)
- GitHub Check: rust-tests (ubuntu-latest)
- GitHub Check: test-consensus-compose
- GitHub Check: rust-tests (windows-latest)
- GitHub Check: integration-tests-consensus
- GitHub Check: test-consistency
- GitHub Check: e2e-tests
- GitHub Check: integration-tests
- GitHub Check: Test Python bindings
- GitHub Check: lint
🔇 Additional comments (1)
tests/consensus_tests/test_shard_snapshot_transfer.py (1)
151-205: LGTM! Comprehensive API key authentication.The helper function correctly threads the
headersparameter through all API operations, ensuring proper authentication throughout the shard transfer flow. The test validates:
- Cluster startup with API key configuration
- Collection operations with authentication
- Shard transfer with authenticated requests
- Data consistency verification across all nodes
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
* alt api key * rm typo * fix key length check logic * differnt dirs for tests * fix tests * rm unused
* alt api key * rm typo * fix key length check logic * differnt dirs for tests * fix tests * rm unused
In distributed deployment, if we need to rotate API key of the service, it is only possible with downtime, as all instances need to switch to a new API key at once.
This PR introduces an ability to specify two API keys, which are acting identically. With two API keys it is possible to perform rolling update of the service with zero downtime of the whole cluster.
Update should be performed in the following steps:
alt_api_key(no downtime, as we can restart one peer at a time)api_key, old any can be safely removed now (again, no downtime)WARN: JWT tokens are not automatically migrated to a new API key, they require re-creation with new key.