backport: performance improvements for the event query API (#7319)#9334
backport: performance improvements for the event query API (#7319)#9334cmwaters merged 12 commits intotendermint:mainfrom
Conversation
Rework the implementation of event query parsing and execution to improve performance and reduce memory usage. Previous memory and CPU profiles of the pubsub service showed query processing as a significant hotspot. While we don't have evidence that this is visibly hurting users, fixing it is fairly easy and self-contained. Updates tendermint#6439. Typical benchmark results comparing the original implementation (PEG) with the reworked implementation (Custom): ``` TEST TIME/OP BYTES/OP ALLOCS/OP SPEEDUP MEM SAVING BenchmarkParsePEG-12 51716 ns 526832 27 BenchmarkParseCustom-12 2167 ns 4616 17 23.8x 99.1% BenchmarkMatchPEG-12 3086 ns 1097 22 BenchmarkMatchCustom-12 294.2 ns 64 3 10.5x 94.1% ``` Components: * Add a basic parsing benchmark. * Move the original query implementation to a subdirectory. * Add lexical scanner for Query expressions. * Add a parser for Query expressions. * Implement query compiler. * Add test cases based on OpenAPI examples. * Add MustCompile to replace the original MustParse, and update usage.
|
Please do not merge it to |
|
I just want to clarify that this PR isn't actually a reimplementation of adr075. It's just the performance improvements that Michael did in #7319 which AFAIK wasn't part of the ADR which actually involved adding a new API. |
yes, since this is prerequisites of the rest backports. |
Do you plan to do the rest of the commits in this PR or as a separate PR? (I think I would prefer them separated) |
yes, to keep as minimal as possible. |
|
Let me clarify my comment. This effort (i.e. porting ADR075) looks like a good candidate for a feature branch. Hence I was saying to hold on this PR. If this PR is the first of a series of backports that are needed to get ADR075 on
I don't think it's a good idea to backport the whole set of PRs for ADR075 directly on |
That would work, but which feature branch I should based on? |
I'll come back to you shortly on that |
@sergio-mena |
|
After some internal discussion, we've decided that it's worth merging this PR directly on |
|
Thanks for contributing @mmsqe! |
I take a free ride, thanks @creachadair |
@sergio-mena May I ask any update on feature branch of backport adr075? |
Sorry, @mmsqe, for my late reply (had to deal with personal issues). We have created branch We will be reviewing all those PRs and merging them into Once the backport is complete, and all CI tests pass, we will proceed to merge If you have any questions/comments/concerns don't hesitate to contact me over Slack, or email (sergio@informal.systems). |
…t#7319) (tendermint#9334) * Performance improvements for the event query API (tendermint#7319) Rework the implementation of event query parsing and execution to improve performance and reduce memory usage. Previous memory and CPU profiles of the pubsub service showed query processing as a significant hotspot. While we don't have evidence that this is visibly hurting users, fixing it is fairly easy and self-contained. Updates tendermint#6439. Typical benchmark results comparing the original implementation (PEG) with the reworked implementation (Custom): ``` TEST TIME/OP BYTES/OP ALLOCS/OP SPEEDUP MEM SAVING BenchmarkParsePEG-12 51716 ns 526832 27 BenchmarkParseCustom-12 2167 ns 4616 17 23.8x 99.1% BenchmarkMatchPEG-12 3086 ns 1097 22 BenchmarkMatchCustom-12 294.2 ns 64 3 10.5x 94.1% ```
…t#7319) (tendermint#9334) * Performance improvements for the event query API (tendermint#7319) Rework the implementation of event query parsing and execution to improve performance and reduce memory usage. Previous memory and CPU profiles of the pubsub service showed query processing as a significant hotspot. While we don't have evidence that this is visibly hurting users, fixing it is fairly easy and self-contained. Updates tendermint#6439. Typical benchmark results comparing the original implementation (PEG) with the reworked implementation (Custom): ``` TEST TIME/OP BYTES/OP ALLOCS/OP SPEEDUP MEM SAVING BenchmarkParsePEG-12 51716 ns 526832 27 BenchmarkParseCustom-12 2167 ns 4616 17 23.8x 99.1% BenchmarkMatchPEG-12 3086 ns 1097 22 BenchmarkMatchCustom-12 294.2 ns 64 3 10.5x 94.1% ```
PR checklist
CHANGELOG_PENDING.mdupdated, or no changelog entry neededdocs/) and code comments, or nodocumentation updates needed