Skip to content

fix(pool): two fixes for closed connection handling#3764

Merged
ndyakov merged 11 commits into
redis:masterfrom
cxljs:fix-pool-closed-conn
Apr 24, 2026
Merged

fix(pool): two fixes for closed connection handling#3764
ndyakov merged 11 commits into
redis:masterfrom
cxljs:fix-pool-closed-conn

Conversation

@cxljs

@cxljs cxljs commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

Closes #3703

  1. In putConn(): prevent CLOSED state conns from being added to idleConns

  2. In Filter(): ensure closed conns are removed from idleConns during failover


Note

Medium Risk
Changes connection pool close/removal behavior and metrics emission paths, which can affect connection lifecycle correctness under concurrency (especially during failover). Risk is moderated by added targeted sentinel failover tests and mostly localized changes to pooling/metrics codepaths.

Overview
Improves closed-connection handling in the pool, especially around failover and concurrent closes.

Conn.Close() is now idempotent and ensures onClose runs at most once. The pool introduces a per-conn atomic closeReason (plus new CloseReasonFailover) so an in-use conn closed by another goroutine can later emit the correct ConnectionClosed metric reason when it’s returned.

ConnPool.Filter() is updated to properly distinguish idle vs in-use conns: idle conns are removed from idleConns/pool and closed immediately with failover metrics, while in-use conns are marked with closeReason and closed without corrupting idle lists. Metrics emission for CloseConn is refactored into recordConnectionMetrics().

Adds a new serial/ordered sentinel failover test that asserts exactly one failover close metric across different MaxActiveConns settings, and marks the UniversalClient test suite as Serial to reduce inter-test interference.

Reviewed by Cursor Bugbot for commit 7500d29. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread internal/pool/pool.go Outdated
Comment thread internal/pool/pool.go Outdated
@ndyakov

ndyakov commented Apr 7, 2026

Copy link
Copy Markdown
Member

@cxljs other than the notes from the CI, I don't have anything against this. We haven't tested the new metrics collector with sentinel (where this reason will be used), have you tested it?

@cxljs

cxljs commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

the notes from the CI

I saw the CI comment. Since these past few days are a special holiday in my country, I haven’t had time to address it yet. I’ll take care of it as soon as possible.

We haven't tested the new metrics collector with sentinel (where this reason will be used), have you tested it?

Thanks for the reminder. I’ll test it.

@cxljs cxljs force-pushed the fix-pool-closed-conn branch from 03cf31f to efc92ea Compare April 15, 2026 10:09
Comment thread internal/pool/pool.go Outdated
Comment thread internal/pool/pool.go Outdated
@cxljs cxljs force-pushed the fix-pool-closed-conn branch 6 times, most recently from 1f9f39f to 8f52534 Compare April 16, 2026 11:47
ndyakov
ndyakov previously approved these changes Apr 20, 2026

@ndyakov ndyakov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good to me.

Comment thread internal/pool/pool.go Outdated
Comment thread universal_test.go
@cxljs cxljs force-pushed the fix-pool-closed-conn branch 3 times, most recently from 7bdd70a to 63846e9 Compare April 20, 2026 16:40
Comment thread sentinel_test.go
@cxljs cxljs force-pushed the fix-pool-closed-conn branch 2 times, most recently from 7f955e7 to fb7aa18 Compare April 21, 2026 08:12
Comment thread sentinel_test.go
@cxljs cxljs force-pushed the fix-pool-closed-conn branch 3 times, most recently from 97d30b4 to e56de38 Compare April 21, 2026 12:07
Comment thread internal/pool/pool.go
Comment thread internal/pool/pool.go
@cxljs cxljs force-pushed the fix-pool-closed-conn branch from e5b8407 to d529d4b Compare April 21, 2026 12:25
Comment thread internal/pool/pool.go
Comment thread internal/pool/pool.go
@cxljs cxljs force-pushed the fix-pool-closed-conn branch 2 times, most recently from d9e99f4 to d978e6d Compare April 23, 2026 05:07

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit d978e6d. Configure here.

Comment thread internal/pool/conn.go
@cxljs cxljs force-pushed the fix-pool-closed-conn branch from a76d7fb to 9ecde4d Compare April 23, 2026 15:24

@ndyakov ndyakov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thank you @cxljs!

@ndyakov ndyakov merged commit d9d7694 into redis:master Apr 24, 2026
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Zombie connection in ConnPool.idleConns

2 participants