Skip to content

[Bug] Performance improvement opportunity in caching parsed message metadata #24644

@lhotari

Description

@lhotari

Search before reporting

  • I searched in the issues and found nothing similar.

Read release policy

  • I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

User environment

master branch version of Pulsar, 34f8657

Issue Description

There's a opportunity to significantly reduce memory allocations in Pulsar by caching parsed message metadata. In a simple profiling test case in #24623 I noticed that
there's a lot of allocations coming from org.apache.pulsar.common.protocol.Commands.peekAndCopyMessageMetadata.

Image

The broker cache can be modified to support caching the message metadata together with the entries so that unnecessary parsing and allocations can be avoided.

Error messages


Reproducing the issue

With #24623 changes, enabling the BrokerEntryCacheMultiBrokerTest.testTailingReadsRollingRestart test
by modifying the @Test(enabled = false) annotation to @Test(invocationCount = 5) and then running

export LIBASYNCPROFILER_PATH=/full/path/to/async-profiler/lib/libasyncProfiler.so (or .dylib on MacOS)
mvn -PtestAsyncProfiler test -pl pulsar-broker -Dtest=BrokerEntryCacheMultiBrokerTest

Additional information

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bugThe PR fixed a bug or issue reported a bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions