Skip to content

Consolidate optimizer passes to improve planning speed #15045

New issue
New issue
Open
Open
Consolidate optimizer passes to improve planning speed#15045
Labels
enhancementNew feature or request
@alamb

Description

@alamb
alamb
opened on Mar 6, 2025

Is your feature request related to a problem or challenge?

@jayzhan211 removed the UnwrapCastInComparison optimizer (and combined it with the simplifier) In

Doing so seemed to make a non trivial difference for planning speed:

I think the improvement came from reducing the number of Optimizer passes (and this rewrites/copies of the plan and all expressions) that happened

Here were my mesurements about speed
Image

Describe the solution you'd like

I would like to make planning faster by potentially combining other passes from this list:

datafusion/datafusion/optimizer/src/optimizer.rs

Lines 243 to 272 in 43ecd9b

let rules: Vec<Arc<dyn OptimizerRule + Sync + Send>> = vec![
Arc::new(EliminateNestedUnion::new()),
Arc::new(SimplifyExpressions::new()),
Arc::new(UnwrapCastInComparison::new()),
Arc::new(ReplaceDistinctWithAggregate::new()),
Arc::new(EliminateJoin::new()),
Arc::new(DecorrelatePredicateSubquery::new()),
Arc::new(ScalarSubqueryToJoin::new()),
Arc::new(ExtractEquijoinPredicate::new()),
Arc::new(EliminateDuplicatedExpr::new()),
Arc::new(EliminateFilter::new()),
Arc::new(EliminateCrossJoin::new()),
Arc::new(CommonSubexprEliminate::new()),
Arc::new(EliminateLimit::new()),
Arc::new(PropagateEmptyRelation::new()),
// Must be after PropagateEmptyRelation
Arc::new(EliminateOneUnion::new()),
Arc::new(FilterNullJoinKeys::default()),
Arc::new(EliminateOuterJoin::new()),
// Filters can't be pushed down past Limits, we should do PushDownFilter after PushDownLimit
Arc::new(PushDownLimit::new()),
Arc::new(PushDownFilter::new()),
Arc::new(SingleDistinctToGroupBy::new()),
// The previous optimizations added expressions and projections,
// that might benefit from the following rules
Arc::new(SimplifyExpressions::new()),
Arc::new(UnwrapCastInComparison::new()),
Arc::new(CommonSubexprEliminate::new()),
Arc::new(EliminateGroupByConstant::new()),
Arc::new(OptimizeProjections::new()),

Describe alternatives you've considered

Some potential candidates to try consolidating:

  • EliminateNestedUnion + EliminateOneUnion
  • EliminateJoin and EliminateJoin

You can run the planning benchmarks like

cargo bench --bench sql_planner

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions