Use window functions for paginated count #2527
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A short explanation of the proposed change
Use the
COUNT(*) OVER()window function to get the total results in the same query as selecting the actual objectsAn explanation of the use cases your change solves
Running
SELECT COUNT(*)for getting total results, followed bySELECT *means 2 potentially expensive queries. The count is just used to populate thetotal_resultsfield in the pagination block of the list responses in the V3 API. This does everything in one query resulting in a ~40-50% query saving (DB caching meant the second query wasn't quite as expensive as the first but still a lot, especially for queries that required temp files/had too many results to efficiently cache)The pagination extension does 3 main things:
SELECT COUNT(*)last_page?,current_pageAs we don't use the dataset extensions (only the records are returned) then by switching to using the
COUNT(*) OVER()window function, we can save the extra calculation query and by just doing the limit/offset ourselves we can remove this extension entirely, saving the overhead of extending the dataset.Note: Whilst investigating this PR, @johha noticed that the sequel paginator wasn't respecting the position ordering of buildpacks. This has been addressed in a second commit in this PR.
Links to any other associated PRs
#2525 had the same end goal of improving the efficiency of this COUNT but this PR does it better and more generally
I have reviewed the contributing guide
I have viewed, signed, and submitted the Contributor License Agreement
I have made this pull request to the
mainbranchI have run all the unit tests using
bundle exec rakeI have run CF Acceptance Tests