Add adaptive throttling to registry cache to prevent penetration when no stale cache exists by Copilot · Pull Request #2823 · micro/go-micro

Copilot · 2026-01-09T11:54:35Z

Enhanced Cache Penetration Protection for Registry

Problem

While PR #2822 added singleflight protection, it's insufficient when:

Cache is empty (no stale data to return)
etcd is overloaded/failing
Multiple sequential requests arrive after singleflight completes

Current behavior: Each singleflight attempt fails → returns error → next request tries again → continuous etcd hammering

Solution: Adaptive Request Throttling

Add rate limiting when registry is in error state to prevent cache penetration:

Changes Summary

Registry Cache Throttling (registry/cache/)

Added MinimumRetryInterval option (default 5s)
Track last failed attempt time per service
Throttle requests within retry interval when no cache exists
Clear throttling state on successful lookup
Comprehensive test coverage (5 new tests)

Bug Fix (events/stream_test.go)

Fixed WaitGroup race condition causing "negative WaitGroup counter" panics
Moved wg.Add() calls before goroutine starts

Documentation

Updated README with feature overview and usage examples
Documented throttling behavior and configuration options

Original prompt

This section details on the original issue you should resolve

<issue_title>[BUG] etcd authentication performance issue and registry cache penetration</issue_title>
<issue_description>## Describe the bug
The backgound is that we need to enable etcd server auth due to some security concerns.

Etcd server has a special authentication design which causes a serious performance problem of /etcdserverpb.Auth/Authenticate API.

From our observation, a normal 3-nodes etcd cluster whose spec like 64C 256G HDD can only support less than around 100 QPS for authentication requests.

In current default go-micro registry plugin setting and gRPC server setting, gRPC server will do a registry by using KeepAliveOnce every RegisterInterval(default 30s). KeepAliveOnce will use /etcdserverpb.Auth/Authenticate once to instablish the stream.

In our Prod env, we have a k8s cluster which contains over 4000 service pods results in a steady /etcdserverpb.Auth/Authenticate QPS around 110.

When we start to enable the etcd auth, the etcd cluster can not handle such /etcdserverpb.Auth/Authenticate QPS causes the business KeepAliveOnce failed and deregister from etcd server after registryTTL.

The upstream Watch the change and Delete downstream server nodes from registry cache which result in an empty cache finally.

When the cache has been cleared, causes another cache penetration problem which is that a gRPC call will call etcd to get downstream nodes from etcd when cache is empty or invalid, but etcd server does not have the info in this time because downstream can not keep the registry hearbeat due to the /etcdserverpb.Auth/Authenticate problem.

The result is that all gRPC request penetrate to etcd and failed at last.

We wanna handle these two problems:

Limit the request to etcd when cache is empty to avoid the penetration issue.

Use "KeepAlive" instead of "KeepAliveOnce" to address /etcdserverpb.Auth/Authenticate QPS issue.

To Reproduce

Steps to reproduce the behavior:

create a 4000+ service pods which use default go-micro registry settings.

preconfigure the etcd username and passwd.

enable etcd cluster auth.

Environment

Go Micro version:

github.com/go-micro/plugins/v4/client/grpc v1.2.1

github.com/go-micro/plugins/v4/registry/etcd v1.2.0

github.com/go-micro/plugins/v4/server/grpc v1.2.0

go-micro.dev/v4 v4.9.0

go.etcd.io/etcd/client/v3 v3.5.2

Go version: 1.18

OS: Ubuntu 20.04

Plugins used:

etcd registry

Logs

server side monitor

Resources

Documentation

Examples

API Reference
</issue_description>

<agent_instructions>The user is now saying single flight protection is not enough</agent_instructions>

Comments on the Issue (you are @copilot in this section)
@asim Could you look at this PR and see if it satisfies your requirements?
#2822</comment_new>
<comment_new>@asim
This PR #2822 is merged. If you are satisfied with the fix please close the issue.</comment_new>

Fixes [BUG] etcd authentication performance issue and registry cache penetration #2821

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Implement rate limiting mechanism to protect etcd when: - Cache is empty (no stale data available) - Registry is failing/overloaded - Multiple sequential requests arrive Changes: - Add MinimumRetryInterval option (default 5s) - Track last failed attempt time per service - Throttle requests within retry interval when no cache exists - Clear throttling on successful lookup - Add comprehensive tests validating throttling behavior Co-authored-by: asim <17530+asim@users.noreply.github.com>

asim · 2026-01-09T12:05:02Z

@copilot unit test fails

asim · 2026-01-09T12:06:26Z

@copilot Unit test fails:

START| Stream
START| Stream/memory
START| Stream/memory/TestMissingTopic
START| Stream/memory/TestConsumeTopic
| panic: sync: negative WaitGroup counter
| goroutine 22 [running]:
| sync.(*WaitGroup).Add(0xc00018a540, 0xffffffffffffffff)
| /opt/hostedtoolcache/go/1.24.11/x64/src/sync/waitgroup.go:64 +0x19a
| sync.(*WaitGroup).Done(0xc00018a540)
| /opt/hostedtoolcache/go/1.24.11/x64/src/sync/waitgroup.go:89 +0x2e
| go-micro.dev/v5/events.runTestStream.func2.1()
| /home/runner/work/go-micro/go-micro/events/stream_test.go:70 +0x505
| created by go-micro.dev/v5/events.runTestStream.func2 in goroutine 21
| /home/runner/work/go-micro/go-micro/events/stream_test.go:57 +0x32e
FAIL | go-micro.dev/v5/events 0.032s

The WaitGroup.Add() calls were happening after goroutines started, causing a race where Done() could be called before Add(), resulting in "negative WaitGroup counter" panics. Fixes: - Move wg.Add(1) before goroutine in TestConsumeTopic - Move wg.Add(2) before first goroutine in TestConsumeGroup Co-authored-by: asim <17530+asim@users.noreply.github.com>

Copilot · 2026-01-09T12:15:10Z

@copilot Unit test fails:

START| Stream...

Fixed in 2850d50. The WaitGroup.Add() calls were racing with goroutine execution - moved them before the goroutine starts.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

224.0.0.251
- Triggering command: REDACTED, pid is -1 (packet block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Document the new adaptive throttling feature that prevents cache penetration when registry is failing and no stale cache exists. Includes: - Feature overview - Usage examples with configuration options - Explanation of throttling behavior - Example scenario demonstrating protection Co-authored-by: asim <17530+asim@users.noreply.github.com>

Initial plan

37a38d2

Copilot AI assigned Copilot and asim Jan 9, 2026

Copilot started work on behalf of asim January 9, 2026 11:55 View session

Copilot AI changed the title ~~[WIP] Fix etcd authentication performance issue~~ Add adaptive throttling to registry cache to prevent penetration when no stale cache exists Jan 9, 2026

Copilot AI requested a review from asim January 9, 2026 12:02

Copilot finished work on behalf of asim January 9, 2026 12:02

Copilot started work on behalf of asim January 9, 2026 12:05 View session

Copilot stopped work on behalf of asim due to an error January 9, 2026 12:06
The session was cancelled by the user.

Copilot started work on behalf of asim January 9, 2026 12:06 View session

Copilot finished work on behalf of asim January 9, 2026 12:19

asim marked this pull request as ready for review January 9, 2026 12:27

asim merged commit 2869cc1 into master Jan 9, 2026
2 checks passed

asim mentioned this pull request Jan 9, 2026

[BUG] etcd authentication performance issue and registry cache penetration #2821

Closed

Copilot AI mentioned this pull request Jan 13, 2026

Enhance cache rate limiting to prevent registry overload during rolling deployments #2824

Merged

7 tasks

Copilot AI mentioned this pull request Feb 3, 2026

Apply rate limiting before singleflight to prevent goroutine blocking on etcd timeout #2841

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add adaptive throttling to registry cache to prevent penetration when no stale cache exists#2823

Add adaptive throttling to registry cache to prevent penetration when no stale cache exists#2823
asim merged 4 commits intomasterfrom
copilot/fix-etcd-authentication-performance

Copilot AI commented Jan 9, 2026 •

edited

Loading

Uh oh!

asim commented Jan 9, 2026

Uh oh!

asim commented Jan 9, 2026

Uh oh!

Copilot AI commented Jan 9, 2026 •

edited

Loading

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Copilot AI commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Enhanced Cache Penetration Protection for Registry

Problem

Solution: Adaptive Request Throttling

Changes Summary

To Reproduce

Environment

Logs

Resources

Comments on the Issue (you are @copilot in this section)

Uh oh!

asim commented Jan 9, 2026

Uh oh!

asim commented Jan 9, 2026

Uh oh!

Copilot AI commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Jan 9, 2026 •

edited

Loading

Copilot AI commented Jan 9, 2026 •

edited

Loading