Skip to content

[Doris On ES] Support ES alias and wildcard index  #3967

@wuyunfeng

Description

@wuyunfeng

Is your feature request related to a problem? Please describe.
ES users create indexes by hour, day, month etc such as log_2020_6_20、 log_2020_6_21、log_2020_6_22 .., At this time, they can easily search all those index by using aliases and wildcards to match those indexes

Currently Doris On ES does not support create external table with aliases or wildcard index such as log_*, if you create such table would cause unpredictable results, this may annoy many Doris On ES users which is what we don't want to see

Describe the solution you'd like
After this PR #3454 (comment) was merged into Doris (This provides great convenience for supporting alias and wildcard indexes, although I am not satisfied with the implementation inside, I will rename and refactor later)

  1. obtain the mapping from {index}/_mapping API
  2. fetch all resolved index shard partitions from {index}/_search_shards
  3. split all shard partitions with corresponding index, Doris BE execute all _search concurrently

_mapping and _search_shards support wildcard and aliases indexName

Additional context

Please Note: we just validate the first index from _mapping API, so DOE's user should guarantee all involved index has same schema or fields at least now( this maybe changed in the future)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions