Skip to content

Add Streaming Plan Impl#1068

Merged
penghuo merged 13 commits intoopensearch-project:2.xfrom
penghuo:hp/feature/maximus/streamingplan
Nov 23, 2022
Merged

Add Streaming Plan Impl#1068
penghuo merged 13 commits intoopensearch-project:2.xfrom
penghuo:hp/feature/maximus/streamingplan

Conversation

@penghuo
Copy link
Copy Markdown
Collaborator

@penghuo penghuo commented Nov 11, 2022 •
edited
Loading

Description

Major Change

  1. Add StreamingPlan implementation which is the high level abstration of a streaming query. Parser should parse streaming query and submit to QueryManager.
  2. Add add(Split) interface to PhysicalOperator
  3. Add StreamingSource asStreamingSource() on Table. Table which could be used as StreamingSource could implement this interface.

Minor Change

  1. Add cancel interface to QueryManager.
  2. Add StreamingIT which run streaming query on source of local filesystem.

FAQ

  1. Could QueryService and ExecutionEngine been combined as single interface. seems they could be combined.

Todo in Phase 2

  1. [FEATURE] Implement cancel interface of QueryManager in OpenSearchQueryManager. #1071
  2. [FEATURE] Manage async query status. #1072

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@penghuo
add streaming plan
de56e89 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Nov 11, 2022 •
edited by codecov bot
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.78%. Comparing base (e280866) to head (ed263a8).
⚠️ Report is 475 commits behind head on 2.x.

❌ Your project status has failed because the head coverage (95.78%) is below the target coverage (99.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files 
@@             Coverage Diff              @@
##                2.x    #1068      +/-   ##
============================================
- Coverage     98.28%   95.78%   -2.51%     
- Complexity     3450     3461      +11     
============================================
  Files           345      357      +12     
  Lines          8580     9295     +715     
  Branches        547      669     +122     
============================================
+ Hits           8433     8903     +470     
- Misses          142      334     +192     
- Partials          5       58      +53
Flag Coverage Δ
query-workbench 62.76% <ø> (?)
sql-engine 98.29% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@penghuo penghuo changed the title Hp/feature/maximus/streamingplan Add Streaming Source Impl Nov 12, 2022
@penghuo penghuo changed the title Add Streaming Source Impl Add Streaming Plan Impl Nov 14, 2022
@penghuo
fix thread leak
cd0238a 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo
fix build issue
fdbd364 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo
enable prometheus test
373a853 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo
disable flaky test
71652dc 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo
fix ut
804f39c 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo
add DefaultQueryManager to testFixtures
4c7a6f1 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo
ad fs storage engine to testFixtures
3d3fbfd 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo
fix style issue
79f4bf6 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo mentioned this pull request Nov 16, 2022
Closed
43 tasks
@penghuo penghuo changed the base branch from feature/maximus-m1 to 2.x November 16, 2022 20:52
@penghuo penghuo marked this pull request as ready for review November 16, 2022 20:52
@penghuo penghuo requested a review from a team as a code owner November 16, 2022 20:52
@penghuo penghuo added the enhancement New feature or request label Nov 16, 2022
@penghuo penghuo added this to the Maximus M1 - Phase 1 milestone Nov 16, 2022
@penghuo penghuo self-assigned this Nov 16, 2022
dai-chen
dai-chen previously approved these changes Nov 17, 2022
View reviewed changes
Copy link
Copy Markdown
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes!

Yury-Fridlyand
Yury-Fridlyand previously approved these changes Nov 18, 2022
View reviewed changes
@penghuo
address comments
085adff 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo
Merge branch '2.x' into hp/feature/maximus/streamingplan
d1d7a25 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo dismissed stale reviews from Yury-Fridlyand and dai-chen via d1d7a25 November 19, 2022 00:46
@penghuo
fix build issue
ed263a8 
Signed-off-by: Peng Huo <penghuo@gmail.com>
@penghuo penghuo merged commit 42143b9 into opensearch-project:2.x Nov 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dai-chen dai-chen dai-chen approved these changes

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand

Assignees

@penghuo penghuo

Labels

enhancement New feature or request

Projects

Object Storage (S3) Data Ingestion through Streaming Query
Status: Done

Milestone

Maximus M1 - Phase 1

Development

Successfully merging this pull request may close these issues.

4 participants

@penghuo @codecov-commenter @dai-chen @Yury-Fridlyand