fix(clp-package): Connect directly to the MongoDB replica (results cache) to avoid failed host resolution of in-container hostname (fixes #1519). by junhaoliao · Pull Request #1518 · y-scope/clp

junhaoliao · 2025-10-28T22:11:58Z

Description

This change updates the results-cache MongoDB client initialization in
components/clp-package-utils/clp_package_utils/scripts/native/search.py to pass
directConnection=True when reading aggregation outputs. This is needed because the results cache is provisioned as a single-node replica with the in-container hostname; without directConnection=True, the Mongo server would serve as a proxy and try to resolve the given host in the initialization before making connections.

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

task
cd build/clp-package
./sbin/start-clp.sh
./sbin/compress.sh ~/samples/hive-24hr
./sbin/search.sh --count 
# observed `tags: [] count: 42092` printed in in the console in roughly 2 seconds

Summary by CodeRabbit

Chores
- Optimised database connection configuration for improved reliability and performance during data operations.

Note: This release contains primarily internal infrastructure improvements with no direct user-facing changes.

…reading query aggregation results from the result cache.

coderabbitai · 2025-10-28T22:12:19Z

Walkthrough

The MongoDB client instantiation in the search script is modified to enable direct connection mode by passing directConnection=True parameter to MongoClient. The remainder of the database interaction logic remains unchanged.

Changes

Cohort / File(s)	Summary
MongoDB connection configuration `components/clp-package-utils/clp_package_utils/scripts/native/search.py`	Adds `directConnection=True` parameter to MongoClient instantiation to establish direct connections to MongoDB instead of topology-aware connections

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Verify that directConnection=True aligns with the deployment topology and expected connection behaviour
Confirm that no other parts of the MongoDB driver usage require adjustment due to this connection mode change

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The pull request title "Connect directly to the MongoDB replica (results cache) to avoid failed host resolution of in-container hostname" accurately and fully captures the main change in the pull request: passing `directConnection=True` to the MongoClient constructor. The title is clear and specific, communicating both what is being changed (direct connection to MongoDB) and why it is necessary (to avoid host resolution failures for the in-container hostname). The title is neither vague nor generic, and it directly relates to the actual code modification shown in the raw summary.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between cf28e78 and 5ed4e13.

📒 Files selected for processing (1)

components/clp-package-utils/clp_package_utils/scripts/native/search.py (1 hunks)

🧰 Additional context used

🧬 Code graph analysis (1)

components/clp-package-utils/clp_package_utils/scripts/native/search.py (1)

components/clp-py-utils/clp_py_utils/clp_config.py (1)

get_uri (336-337)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)

GitHub Check: package-image
GitHub Check: lint-check (macos-15)
GitHub Check: lint-check (ubuntu-24.04)

coderabbitai · 2025-10-28T22:15:02Z

    if do_count_aggregation is None and count_by_time_bucket_size is None:
        return
-    with pymongo.MongoClient(results_cache.get_uri()) as client:
+    with pymongo.MongoClient(results_cache.get_uri(), directConnection=True) as client:


⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

LGTM! Correct parameter for single-node replica set.

The addition of directConnection=True is the appropriate solution for connecting to a single-node MongoDB replica set with an in-container hostname. This parameter instructs the MongoDB driver to bypass automatic topology discovery and connect directly to the specified host, preventing hostname resolution issues during client initialization.

To ensure consistency across the codebase, verify that all other MongoClient instantiations for the results cache also use directConnection=True:

🏁 Script executed:

#!/bin/bash # Description: Find all MongoClient instantiations that use results_cache to ensure consistent directConnection usage # Search for MongoClient calls with results_cache rg -nP --type=py -C3 'MongoClient\([^)]*results_cache'

Length of output: 3029

Apply directConnection=True consistently across all results_cache MongoClient instantiations.

The change at line 80 is correct, but verification reveals two other MongoClient instantiations for results_cache that are missing directConnection=True:

components/clp-py-utils/clp_py_utils/initialize-results-cache.py:133

components/job-orchestration/job_orchestration/scheduler/query/query_scheduler.py:877

These locations connect to the same single-node replica set and should use the same directConnection=True parameter to avoid hostname resolution issues during initialization.

🤖 Prompt for AI Agents

In components/clp-package-utils/clp_package_utils/scripts/native/search.py around line 80 and in components/clp-py-utils/clp_py_utils/initialize-results-cache.py around line 133 and components/job-orchestration/job_orchestration/scheduler/query/query_scheduler.py around line 877, the MongoClient instantiations that use results_cache.get_uri() must include directConnection=True; update those calls to pass directConnection=True to pymongo.MongoClient (preserving any existing kwargs) so all connections to the single-node replica set use the same directConnection behavior and avoid hostname resolution issues.

kirkrodrigues · 2025-10-28T22:55:50Z

@coderabbitai, can you open an issue about the problem that this PR fixes?

coderabbitai · 2025-10-28T22:56:31Z

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

@kirkrodrigues, I'll create an issue documenting the problem that this PR fixes.

🏁 Script executed:

#!/bin/bash
# Create a GitHub issue documenting the MongoDB connection problem

gh issue create \
  --repo y-scope/clp \
  --title "MongoDB client fails to initialize without directConnection parameter for single-node replica results cache" \
  --body "## Problem Description

The results-cache MongoDB client in \`components/clp-package-utils/clp_package_utils/scripts/native/search.py\` fails to properly initialize when connecting to read query aggregation results from the result cache.

## Root Cause

The results cache is provisioned as a single-node replica with an in-container hostname. Without specifying \`directConnection=True\` in the MongoDB client initialization, the Mongo server acts as a proxy and attempts to resolve the given host during initialization before making connections. This behaviour prevents proper connection establishment.

## Impact

This affects the ability to read aggregation outputs from the results cache, preventing query results from being retrieved properly.

## Resolution

Fixed by PR #1518

---
**Reported by:** @kirkrodrigues  
**Related PR:** https://github.com/y-scope/clp/pull/1518" \
  --assignee kirkrodrigues

Length of output: 42

@kirkrodrigues, I've created issue #1519 to document the MongoDB connection problem that this PR fixes. The issue describes the root cause (MongoDB server acting as a proxy and attempting to resolve the in-container hostname without directConnection=True), the impact on reading query aggregation results, and references this PR as the resolution.

You can view it here: #1519

kirkrodrigues

For the PR title, how about:

fix(clp-package): Connect directly to the MongoDB replica (results cache) to avoid failed host resolution of in-container hostname (fixes #1519).

…solve dev-env startup issue. (#1598)

…che) to avoid failed host resolution of in-container hostname (fixes y-scope#1519). (y-scope#1518)

…to resolve dev-env startup issue. (y-scope#1598)

fix(clp-package): Add directConnection=True to MongoClient() for …

5ed4e13

…reading query aggregation results from the result cache.

junhaoliao requested a review from a team as a code owner October 28, 2025 22:11

junhaoliao requested a review from hoophalab October 28, 2025 22:12

junhaoliao requested a review from kirkrodrigues October 28, 2025 22:12

coderabbitai Bot reviewed Oct 28, 2025

View reviewed changes

coderabbitai Bot mentioned this pull request Oct 28, 2025

sbin/search.sh --count fails due to MongoDB replica error #1519

Closed

Merge branch 'main' into direct-connection

857179e

kirkrodrigues approved these changes Oct 29, 2025

View reviewed changes

junhaoliao merged commit 56ebca1 into y-scope:main Oct 29, 2025
20 checks passed

davemarco mentioned this pull request Nov 13, 2025

fix(webui): Apply direct MongoDB replica connection as in #1518 to resolve dev-env startup issue. #1598

Merged

3 tasks

davemarco added a commit that referenced this pull request Nov 19, 2025

fix(webui): Apply direct MongoDB replica connection as in #1518 to re…

0f18e59

…solve dev-env startup issue. (#1598)

junhaoliao deleted the direct-connection branch May 7, 2026 19:46

junhaoliao added a commit to junhaoliao/clp that referenced this pull request May 17, 2026

fix(clp-package): Connect directly to the MongoDB replica (results ca…

a117bad

…che) to avoid failed host resolution of in-container hostname (fixes y-scope#1519). (y-scope#1518)

junhaoliao pushed a commit to junhaoliao/clp that referenced this pull request May 17, 2026

fix(webui): Apply direct MongoDB replica connection as in y-scope#1518 …

e2611a9

…to resolve dev-env startup issue. (y-scope#1598)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(clp-package): Connect directly to the MongoDB replica (results cache) to avoid failed host resolution of in-container hostname (fixes #1519).#1518

fix(clp-package): Connect directly to the MongoDB replica (results cache) to avoid failed host resolution of in-container hostname (fixes #1519).#1518
junhaoliao merged 2 commits into
y-scope:mainfrom
junhaoliao:direct-connection

junhaoliao commented Oct 28, 2025 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 28, 2025 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Oct 28, 2025

Uh oh!

kirkrodrigues commented Oct 28, 2025

Uh oh!

coderabbitai Bot commented Oct 28, 2025

Uh oh!

kirkrodrigues left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

junhaoliao commented Oct 28, 2025 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Pre-merge checks and finishing touches

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

kirkrodrigues commented Oct 28, 2025

Uh oh!

coderabbitai Bot commented Oct 28, 2025

Uh oh!

kirkrodrigues left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

junhaoliao commented Oct 28, 2025 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 28, 2025 •

edited

Loading