Add an optimized QueueLeaves implementation for single-leaf batches in the PostgreSQL storage backend#3769
Merged
roger2hk merged 5 commits intogoogle:masterfrom Apr 23, 2025
Conversation
…n the PostgreSQL storage backend
cf90692 to
761fd19
Compare
Contributor
|
/gcbrun |
roger2hk
approved these changes
Apr 4, 2025
2 tasks
Contributor
|
/gcbrun |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The current QueueLeaves implementation for PostgreSQL is optimised for sequencing large batches, but the approach it takes (temporary tables, and table locks to prevent concurrent operations on the LeafData and Unsequenced tables) imposes overheads that make it perform fairly poorly when sequencing small batches. We've found that this is a pain point for our PostgreSQL-based Tiger and Elephant CT log shards that are now seeing a significant rate of submissions.
AIUI, Trillian's CT persona only ever calls QueueLeaves for batches containing a single-leaf. i.e., one QueueLeaves operation per add-chain or add-pre-chain request.
This PR incorporates an alternative "QueueLeaf" implementation for PostgreSQL that's used for and optimised for single-leaf batches. We deployed this to our Tiger and Elephant log shards about a week ago, and the results have been very promising.
Checklist