fix(query-scheduler): Gracefully terminate reducer connections that do not attempt a connection handshake (fixes #1386). by junhaoliao · Pull Request #1389 · y-scope/clp

junhaoliao · 2025-10-06T21:54:52Z

Description

This PR adds graceful handling for empty reducer connections to the query scheduler, by ignoring such benign noises.

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

Built and started the package.
Ran docker ps to find out the instance id of the query-scheduler.
Ran docker exec <query-scheduler-id> bash -c '< /dev/tcp/localhost/7000' and observed the command exited successfully.
Ran docker logs <query-scheduler-id> and observed no exception logs were printed as before.

Summary by CodeRabbit

Bug Fixes
- Treat reducer connections that close without sending data as benign to avoid spurious errors or crashes.
- Short-circuit connection handling when no message is received and log these cases at debug level to reduce noisy error logs.
- Updated docstrings to clarify return behavior and error handling for closed connections.
Reliability
- Improved resilience of connection handling for smoother job orchestration under intermittent network conditions.

y-scope#1386).

coderabbitai · 2025-10-06T21:55:19Z

Walkthrough

Change _recv_msg_from_reducer to return Optional[bytes] and handle asyncio.IncompleteReadError: if the exception's partial data is empty, log a debug message and return None; if partial data exists, re-raise the exception. Reducer handler short-circuits on None. Signature changed; no other public APIs altered.

Changes

Cohort / File(s)	Summary of changes
Reducer connection handling `components/job-orchestration/job_orchestration/scheduler/query/reducer_handler.py`	Update `_recv_msg_from_reducer` return type to `Optional[bytes]`; catch `asyncio.IncompleteReadError`, return `None` and log when `err.partial` is empty, re-raise when non-empty; short-circuit reducer connection handler when received message is `None`; update docstrings.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Reducer
  participant Scheduler as Scheduler::ReducerHandler

  Note over Scheduler: Await initial reducer read via _recv_msg_from_reducer

  Reducer->>Scheduler: Open connection / send data (or close)
  alt Read completes with data (bytes returned)
    Scheduler-->>Reducer: Continue normal processing
  else asyncio.IncompleteReadError raised
    alt err.partial length == 0
      Note over Scheduler: Debug log and return None (short-circuit)
      Scheduler-->>Reducer: Close handling (no error)
    else err.partial length > 0
      Note over Scheduler: Re-raise exception (propagate error)
      Scheduler-->>Reducer: Error propagated
    end
  end

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related issues

Clean up logs when no bytes are sent in an opened connection to query scheduler #1386 — Matches the change to suppress noisy empty-connection IncompleteReadError occurrences by returning early on zero-length partial.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title succinctly describes the primary change to the query scheduler by indicating that reducer connections without a handshake will be gracefully terminated, follows Conventional Commit conventions, and clearly references the related issue, making it both specific and informative.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ba34c27 and f256f14.

📒 Files selected for processing (1)

components/job-orchestration/job_orchestration/scheduler/query/reducer_handler.py (1 hunks)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)

GitHub Check: package-image
GitHub Check: rust-checks (ubuntu-24.04)
GitHub Check: rust-checks (ubuntu-22.04)
GitHub Check: rust-checks (macos-15)
GitHub Check: lint-check (ubuntu-24.04)

gibber9809

LGTM. For PR title how about:
fix(query-scheduler): Gracefully terminate reducer connections that do not attempt a connection handshake (fixes #1386).

…o not attempt a connection handshake (fixes y-scope#1386). (y-scope#1389)

fix(query-scheduler): Handle empty reducer connections gracefully (fixes

f256f14

y-scope#1386).

junhaoliao requested a review from a team as a code owner October 6, 2025 21:54

junhaoliao requested a review from gibber9809 October 6, 2025 21:55

coderabbitai Bot reviewed Oct 6, 2025

View reviewed changes

Comment thread components/job-orchestration/job_orchestration/scheduler/query/reducer_handler.py Outdated

junhaoliao added 2 commits October 7, 2025 00:24

narrow exception catch range

9650265

narrow exception catch range

daee6cc

gibber9809 approved these changes Oct 8, 2025

View reviewed changes

junhaoliao changed the title ~~fix(query-scheduler): Handle empty reducer connections gracefully (fixes #1386).~~ fix(query-scheduler): Gracefully terminate reducer connections that do not attempt a connection handshake (fixes #1386). Oct 9, 2025

junhaoliao merged commit b8f7b75 into y-scope:main Oct 9, 2025
24 checks passed

junhaoliao deleted the graceful-reducer-handler branch May 7, 2026 19:46

junhaoliao added a commit to junhaoliao/clp that referenced this pull request May 17, 2026

fix(query-scheduler): Gracefully terminate reducer connections that d…

ce36860

…o not attempt a connection handshake (fixes y-scope#1386). (y-scope#1389)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(query-scheduler): Gracefully terminate reducer connections that do not attempt a connection handshake (fixes #1386).#1389

fix(query-scheduler): Gracefully terminate reducer connections that do not attempt a connection handshake (fixes #1386).#1389
junhaoliao merged 3 commits into
y-scope:mainfrom
junhaoliao:graceful-reducer-handler

junhaoliao commented Oct 6, 2025 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 6, 2025 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

gibber9809 left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

junhaoliao commented Oct 6, 2025 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related issues

Pre-merge checks and finishing touches

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gibber9809 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

junhaoliao commented Oct 6, 2025 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 6, 2025 •

edited

Loading

gibber9809 left a comment •

edited

Loading