Skip to content

[main]Changes in DataSourceService and DataSourceMetadataStorage interface#1408

Merged
vamsimanohar merged 2 commits intoopensearch-project:mainfrom
vamsimanohar:datasource-metadata-interface-main
Mar 8, 2023
Merged

[main]Changes in DataSourceService and DataSourceMetadataStorage interface#1408
vamsimanohar merged 2 commits intoopensearch-project:mainfrom
vamsimanohar:datasource-metadata-interface-main

Conversation

@vamsimanohar
Copy link
Copy Markdown
Member

@vamsimanohar vamsimanohar commented Mar 7, 2023

Description

Definitions

  • Datasource : Datasource class holds basic datasource info(name, connector), storage engine and other objects created specific to a datasource.
  • DatasourceMetadata: DatasourceMetadata class carries all the metadata required for a datasource.
  • DatasourceService: This class acts as interface for datasources and datasource metadata management.
  • DatasourceMetadataStorageService: This class acts as interface for datasource metadata storage.

usecases

  • CRUD APIs for datasources.
  • Analyzer class
    • Fully Qualified Name Resolution requires names of all datasources. [Queries GetAllDatasourceMetadata]
    • DataSourceTable class requires metadata(name, connector) of all datasources for show datasources command. [Queries GetAllDatasourceMetadata]
    • Requires storage engine to implement and execute queries. [Queries GetAllDatasourceMetadata]

code flow for each usecase.

Datasource bootstrapping during plugin startup [Internal usecase for storage engine creation]

Screen Shot 2023-03-06 at 5 56 34 PM

Create datasource [REST API]

Screen Shot 2023-03-06 at 6 09 58 PM

Get datasource along with storage engine and other artifacts [Internal use case only]

Screen Shot 2023-03-06 at 6 13 39 PM

Get datasource [REST API]

Screen Shot 2023-03-06 at 6 21 26 PM

Delete datasource [REST API]

Screen Shot 2023-03-06 at 6 21 44 PM

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
…taStorage

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
@vamsimanohar vamsimanohar requested a review from a team as a code owner March 7, 2023 23:45
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

Merging #1408 (1d08187) into main (0ccb20a) will increase coverage by 0.00%.
The diff coverage is 100.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff            @@
##               main    #1408   +/-   ##
=========================================
  Coverage     98.38%   98.38%           
- Complexity     3694     3698    +4     
=========================================
  Files           343      343           
  Lines          9113     9121    +8     
  Branches        586      586           
=========================================
+ Hits           8966     8974    +8     
  Misses          142      142           
  Partials          5        5           
Flag Coverage Δ
sql-engine 98.38% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ain/java/org/opensearch/sql/analysis/Analyzer.java 100.00% <100.00%> (ø)
...ensearch/sql/datasource/DataSourceServiceImpl.java 100.00% <100.00%> (ø)
...anner/physical/datasource/DataSourceTableScan.java 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@vamsimanohar vamsimanohar changed the title Datasource metadata interface main [main]Changes in DataSourceService and DataSourceMetadataStorage interface Mar 8, 2023
@dai-chen dai-chen added the maintenance Improves code quality, but not the product label Mar 8, 2023
@vamsimanohar vamsimanohar added enhancement New feature or request backport 2.x and removed maintenance Improves code quality, but not the product labels Mar 8, 2023
@vamsimanohar vamsimanohar merged commit 98fb969 into opensearch-project:main Mar 8, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Mar 8, 2023
…rface (#1408)

* Added DataSourceMetadataStorage interface

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

* Added new interface methods in DataSourceService and DataSourceMetadataStorage

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

---------

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
(cherry picked from commit 98fb969)
vamsimanohar added a commit that referenced this pull request Mar 8, 2023
…rface (#1408) (#1414)

* Added new interface methods in DataSourceService and DataSourceMetadataStorage

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
(cherry picked from commit 98fb969)

Co-authored-by: vamsi-amazon <reddyvam@amazon.com>
matthewryanwells pushed a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Mar 10, 2023
…rface (opensearch-project#1408)

* Added DataSourceMetadataStorage interface

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

* Added new interface methods in DataSourceService and DataSourceMetadataStorage

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>

---------

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
Signed-off-by: Matthew Wells <matthew.wells@improving.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.x enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants