This repository was archived by the owner on Aug 23, 2023. It is now read-only.
Performance optimizations for meta tag queries#1517
Merged
Conversation
also changes the default value for tag query workers from 50 to 5. actually 50 has always been relatively high, but since we now also create sub-queries from the expressions associated with meta tags it is way too high. also changes the id selector so that it doesn't unnecessarily deduplicate results when it is called by a sub-query, because sub-queries don't evaluate meta tags and only if meta tags get evaluated duplicates are possible.
since we still always need to filter by the from timestamp, we are now doing this in the id selector by calling a new method on the tag query context called newerThanFrom. to make this change work it was necessary to pass the result channel into getIds(), instead of returning it from it. this resulted in a bit of rewiring of the channels and especially where channels get closed. to make it possible to close the result channel when the id selector is finished, getIds() is now a blocking method.
61eeb13 to
2ec3d6e
Compare
Contributor
Author
|
Rebased onto the latest master |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR does 4 things, in 4 different commits, so I recommend reviewing them separately:
TagQueryWorkers. When building the initial result set, if the query expression includes a meta tag, then we instantiate a sub-query in a go routine for each of the involved meta records. If a meta tag has a really large number of meta records associated with it (f.e.dc=dc1resulting inhost=host1,host=host2,host=host3etc) then we don't want to start all those go routines for the sub queries at once. So this introduces a gate to limit their concurrency..getIds()when calling it and.getIds()is now a blocking method.master.