Skip to content

Optimization: osiris_log: add reader config option for read-ahead limit#198

Merged
michaelklishin merged 1 commit intorabbitmq:mainfrom
amazon-mq:configure-read-ahead
Nov 3, 2025
Merged

Optimization: osiris_log: add reader config option for read-ahead limit#198
michaelklishin merged 1 commit intorabbitmq:mainfrom
amazon-mq:configure-read-ahead

Conversation

@the-mikedavis
Copy link
Copy Markdown
Collaborator

@the-mikedavis the-mikedavis commented Nov 3, 2025

Previously the READ_AHEAD_LIMIT was a constant fixed at 4096. For some usage patterns setting the limit higher helps consumer throughput when the chunk size is medium/small - specifically streams published at medium throughput being consumed over a TLS connection.

See rabbitmq/rabbitmq-server#14877

I will make a follow-up in the server to add some Cuttlefish config for this.

@the-mikedavis the-mikedavis self-assigned this Nov 3, 2025
@the-mikedavis
Copy link
Copy Markdown
Collaborator Author

the-mikedavis commented Nov 3, 2025 •
edited
Loading

Maybe it makes more sense for this to be passed in the config map, so it would be consistent with the read_ahead option?

Edit: yeah that would be nicer so we could pass in values like "32KB" in config and all of that parsing logic lives in Rabbit.

@the-mikedavis the-mikedavis changed the title osiris_log: Read read-ahead limit from application environment osiris_log: Add reader config option for read-ahead limit Nov 3, 2025
@the-mikedavis
osiris_log: Add reader config option for read-ahead limit
10ddb6a 
Previously the `READ_AHEAD_LIMIT` was a constant fixed at 4096. For some
usage patterns setting the limit higher helps consumer throughput when
the chunk size is medium/small - specifically streams published at
medium throughput being consumed over a TLS connection.
@the-mikedavis the-mikedavis mentioned this pull request Nov 3, 2025
Merged
@lukebakken lukebakken self-requested a review November 3, 2025 20:50
@michaelklishin michaelklishin changed the title osiris_log: Add reader config option for read-ahead limit Optimization: osiris_log: add reader config option for read-ahead limit Nov 3, 2025
@michaelklishin michaelklishin added this to the 1.10.1 milestone Nov 3, 2025
@michaelklishin michaelklishin merged commit 0134cf0 into rabbitmq:main Nov 3, 2025
4 checks passed
@michaelklishin
Copy link
Copy Markdown
Contributor

michaelklishin commented Nov 3, 2025

Note: the default read-ahead limit is still 4096 with this PR but it now can be tweaked and eventually we might discover a more optimal default.

@the-mikedavis the-mikedavis deleted the configure-read-ahead branch November 3, 2025 22:06
@mergify mergify bot mentioned this pull request Nov 3, 2025
Merged
@kjnilsson
Copy link
Copy Markdown
Contributor

kjnilsson commented Nov 4, 2025

I'm thinking this would have been an opportunity to combine the on and limit fields into 1 where a limit of 0 equals no read-ahead. Unfortunrately this was merged and by the look of things tagged before I noticed there was a PR.

@the-mikedavis
Copy link
Copy Markdown
Collaborator Author

the-mikedavis commented Nov 4, 2025

Ah yeah we could keep these both in the same option. Let me send some follow-up changes. We probably don't need to care about the semantic versioning so much in this dependency. But we could have the stream.read_ahead option in RabbitMQ accept either a boolean or a number of bytes, so it would be backwards compatible with 4.2.0.

@the-mikedavis the-mikedavis mentioned this pull request Nov 4, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@michaelklishin michaelklishin michaelklishin approved these changes

@lukebakken lukebakken Awaiting requested review from lukebakken

Assignees

@the-mikedavis the-mikedavis

Labels

None yet

Projects

None yet

Milestone

1.10.1

Development

Successfully merging this pull request may close these issues.

3 participants

@the-mikedavis @michaelklishin @kjnilsson