Skip to content

keyset pagination for dbal extractor #1598

@norberttech

Description

@norberttech

Currently, we primarily use the dbal limit/offset paginator, which functions effectively as long as it's applied to tables with evenly distributed rows across a timeline.

Whenever we would like to use on a tables that comes with 100k+ rows with the same timestamp column value, each next page will take longer.

keyset pagination is not using offset, instead it operates on a indexed and sorted column like:

SELECT * FROM table WHERE id > :id SORT BY id ASC LIMIT :limit

And in order to jump to the next page we need to know the previous page id from the last row.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Status

    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions