Skip to content

[Bug] RateLimiter lock contention when use precise publish rate limiter #21442

@Shawyeok

Description

@Shawyeok

Search before asking

  • I searched in the issues and found nothing similar.

After enabled the preciseTopicPublishRateLimiterEnable configuration in broker.conf, if one topic on the broker exceeds the publish rate limit (for example, 1MB/s), it will cause lock contention, leading to an increase in the 99th percentile write latency for other topics that have not reached the rate limit.
image

The profile shows that there is lock content inside the method org.apache.pulsar.broker.service.PrecisPublishLimiter#tryAcquire.
image
pulsar-3.0.1.lock.profile.008.html.zip

Minimal reproduce step

Noticed in a fork of 2.8.1, reproduced in 3.0.1 on Linux CentOS 7.9 3.10.0-1160.99.1.el7.x86_64

pulsar-admin namespaces create public/ratelimited
pulsar-admin namespaces set-publish-rate -b 1048576 public/ratelimited
pulsar-admin namespaces policies public/ratelimited
pulsar-perf produce persistent://public/default/no-rate-limit
pulsar-perf produce -r 1600 -ioThreads 4 -n 100 persistent://public/ratelimited/topic-1

What did you expect to see?

One topic exceeds publish rate limit should not effect other topics write latency.

What did you see instead?

Another topic write latency also increased.

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

Labels

type/bugThe PR fixed a bug or issue reported a bug

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions