Skip to content

DateFacet suggestions are incredibly slow against large tables #2407

@simonw

Description

@simonw

I thought the limit 100 would help here, but it turns out that for a column on a huge table (e.g. 20m rows) which has no dates in at all that glob has to scan through every value of the column every time, just to figure out that there are NOT 100 matches!

# Does this column contain any dates in the first 100 rows?
suggested_facet_sql = """
select date({column}) from (
{sql}
) where {column} glob "????-??-*" limit 100;
""".format(
column=escape_sqlite(column), sql=self.sql
)

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions