Skip to content

Add Statement, QueryExecution and QueryManager#845

Merged
penghuo merged 5 commits intoopensearch-project:feature/maximus-m1from
penghuo:feature/maximus/statement
Oct 28, 2022
Merged

Add Statement, QueryExecution and QueryManager#845
penghuo merged 5 commits intoopensearch-project:feature/maximus-m1from
penghuo:feature/maximus/statement

Conversation

@penghuo
Copy link
Copy Markdown
Collaborator

@penghuo penghuo commented Sep 19, 2022

New abstraction

QueryManager is the high level interface of the core engine, Parser parse raw query as the Plan and sumitted to QueryManager.

  1. AstBuilder analyze raw query string and create Statement.
  2. QueryPlanFactory create Plan for different Statement.
  3. QueryManager execute Plan.

Screen Shot 2022-10-25 at 5 05 46 PM

Core engine define the interface of QueryManager. Each execution engine should provide the implementation of QueryManager which bind to execution enviroment.
QueryManager manage all the submitted plans and define the following interface

  • submit: submit queryexecution.
  • cancel: cancel query execution.
  • get: get query execution info of specific query.

image

Parser parse raw query as Statement and create AbstractPlan. Each AbstractPlan decide how to execute the query in QueryManager.

image (1)

QueryService is the low level interface of core engine, each Plan decide how to execute the query and use QueryService to analyze, plan, optimize and execute the query.

Screen Shot 2022-10-25 at 5 07 29 PM

Change of existing logic

  1. Remove the schedule logic in NIO thread. After the change,
    a. Parser will be executed in NIO thread.
    b. QueryManager decide query execution strategy. e.g. OpenSearchQueryManager schedule the QueryExecution running in sql-worker thread pool.

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@penghuo penghuo marked this pull request as ready for review September 19, 2022 20:08
@penghuo penghuo requested a review from a team as a code owner September 19, 2022 20:08
Metrics.getInstance()
.getNumericalMetric(MetricName.PPL_FAILED_REQ_COUNT_CUS)
.increment();
return channel ->
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the schedule logic. In OpenSearch, QueryManager will schedule the query in sql-worker thread.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Oct 3, 2022

Codecov Report

❌ Patch coverage is 98.97959% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 94.92%. Comparing base (63f3449) to head (26fc997).
⚠️ Report is 45 commits behind head on feature/maximus-m1.

Files with missing lines Patch % Lines
...opensearch/sql/ppl/parser/AstStatementBuilder.java 83.33% 0 Missing and 1 partial ⚠️

❌ Your project status has failed because the head coverage (94.92%) is below the target coverage (99.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@                   Coverage Diff                    @@
##             feature/maximus-m1     #845      +/-   ##
========================================================
- Coverage                 97.58%   94.92%   -2.66%     
- Complexity                 3177     3203      +26     
========================================================
  Files                       307      325      +18     
  Lines                      7898     8612     +714     
  Branches                    513      634     +121     
========================================================
+ Hits                       7707     8175     +468     
- Misses                      190      382     +192     
- Partials                      1       55      +54     
Flag Coverage Δ
query-workbench 62.76% <ø> (?)
sql-engine 97.58% <98.97%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@penghuo penghuo marked this pull request as draft October 12, 2022 06:04
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo force-pushed the feature/maximus/statement branch from 16eb89e to 2080313 Compare October 26, 2022 00:06
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo changed the title Add Statement, QueryExecution and QueryManager [Feature] Add Statement, QueryExecution and QueryManager Oct 26, 2022
@penghuo penghuo self-assigned this Oct 26, 2022
@penghuo penghuo added the enhancement New feature or request label Oct 26, 2022
@penghuo penghuo marked this pull request as ready for review October 26, 2022 01:02
@penghuo penghuo changed the title [Feature] Add Statement, QueryExecution and QueryManager Add Statement, QueryExecution and QueryManager Oct 26, 2022
@penghuo penghuo added this to the Maximus M1 - Phase 1 milestone Oct 26, 2022
Signed-off-by: Peng Huo <penghuo@gmail.com>
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo merged commit 30fcd79 into opensearch-project:feature/maximus-m1 Oct 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Development

Successfully merging this pull request may close these issues.

3 participants