Add _routing to SQL includes list (#277) by acarbonetto · Pull Request #1771 · opensearch-project/sql

acarbonetto · 2023-06-23T00:14:04Z

Description

Add the _routing reserved word as a metafield accessible as an identifier from any opensearch index.

Example:

select _id, _index, _routing, int0 from calcs limit 5

Example response:

        [
            "1",
            "calcs",
            "[xoyIAOdqQsS74x2yKSgiEQ][calcs][0]",
            1
        ],

Issues Resolved

Partially resolves: sql#1478

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com>

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Yury-Fridlyand

I think we need an IT to prove that filtering by _routing works fine, for example:

select *, _id from <index> where _routing = 'a'
select *, _id from <index> where _routing = 'b'

The test should propagate the data properly into different shards and get their names using REST API.

codecov · 2023-06-23T00:37:20Z

Codecov Report

Merging #1771 (1c0b35d) into main (fa840e0) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##               main    #1771   +/-   ##
=========================================
  Coverage     97.32%   97.32%           
- Complexity     4458     4459    +1     
=========================================
  Files           388      388           
  Lines         11050    11052    +2     
  Branches        790      791    +1     
=========================================
+ Hits          10754    10756    +2     
  Misses          289      289           
  Partials          7        7

Flag	Coverage Δ
sql-engine	`97.32% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...search/sql/opensearch/storage/OpenSearchIndex.java	`100.00% <ø> (ø)`
...ch/sql/opensearch/response/OpenSearchResponse.java	`100.00% <100.00%> (ø)`

acarbonetto · 2023-06-23T00:44:49Z

I think we need an IT to prove that filtering by _routing works fine, for example:
select *, _id from <index> where _routing = 'a'
select *, _id from <index> where _routing = 'b'
The test should propagate the data properly into different shards and get their names using REST API.

Right now routing shard names are randomly assigned, so I cannot filter by shard name. Yet.
In the following PR, I will be able to assign documents to specific shards, and will be able to filter by routing shard using exactly what you were suggesting.

Note: this is also why a doctest with _routing doesn't work, because the documents are assigned to random shards.

Yury-Fridlyand · 2023-06-23T00:58:42Z

Yes, but you can get shard names with REST API or CAT API

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

integ-test/src/test/java/org/opensearch/sql/sql/IdentifierIT.java

dai-chen

Thanks for the changes!

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

…uting Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

…uting

* Add _routing to SQL includes list Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update IT index Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update doctest Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Add WHERE clause IT tests Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Fix IT test Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Update documentation for _routing Signed-off-by: acarbonetto <andrewc@bitquilltech.com> --------- Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: acarbonetto <andrewc@bitquilltech.com> (cherry picked from commit 876a9c8)

* Add _routing to SQL includes list Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update IT index Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update doctest Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Add WHERE clause IT tests Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Fix IT test Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Update documentation for _routing Signed-off-by: acarbonetto <andrewc@bitquilltech.com> --------- Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: acarbonetto <andrewc@bitquilltech.com> (cherry picked from commit 876a9c8) Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com>

* Add _routing to SQL includes list Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update IT index Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update doctest Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Add WHERE clause IT tests Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Fix IT test Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Update documentation for _routing Signed-off-by: acarbonetto <andrewc@bitquilltech.com> --------- Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: acarbonetto <andrewc@bitquilltech.com> (cherry picked from commit 876a9c8)

* Add _routing to SQL includes list Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update IT index Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update doctest Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Add WHERE clause IT tests Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Fix IT test Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Update documentation for _routing Signed-off-by: acarbonetto <andrewc@bitquilltech.com> --------- Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: acarbonetto <andrewc@bitquilltech.com> (cherry picked from commit 876a9c8) Co-authored-by: Andrew Carbonetto <andrewc@bitquilltech.com>

* Add _routing to SQL includes list Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update IT index Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> * Update doctest Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Add WHERE clause IT tests Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Fix IT test Signed-off-by: acarbonetto <andrewc@bitquilltech.com> * Update documentation for _routing Signed-off-by: acarbonetto <andrewc@bitquilltech.com> --------- Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com> Signed-off-by: acarbonetto <andrewc@bitquilltech.com> Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com>

acarbonetto added 3 commits June 16, 2023 14:23

Add _routing to SQL includes list

1fd3a57

Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com>

Update IT index

da1973f

Signed-off-by: Andrew Carbonetto <andrewc@bitquilltech.com>

Merge branch 'dev-metafields-routing' into integ-metafields-routing

e3de72b

acarbonetto requested review from GumpacG, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, penghuo, pjfitzgibbons, ps48, rupal-bq, seankao-az and vamsimanohar as code owners June 23, 2023 00:14

Update doctest

e3e02b0

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Yury-Fridlyand previously approved these changes Jun 23, 2023

View reviewed changes

acarbonetto changed the title ~~Integ metafields routing~~ Add _routing to SQL includes list (#277) Jun 23, 2023

MitchellGale approved these changes Jun 23, 2023

View reviewed changes

Add WHERE clause IT tests

9eb5cbd

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

acarbonetto dismissed Yury-Fridlyand’s stale review via 9eb5cbd June 23, 2023 21:18

Fix IT test

1d89d9d

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Yury-Fridlyand previously approved these changes Jun 24, 2023

View reviewed changes

integ-test/src/test/java/org/opensearch/sql/sql/IdentifierIT.java Outdated Show resolved Hide resolved

Yury-Fridlyand added the backport 2.x label Jun 24, 2023

dai-chen reviewed Jun 26, 2023

View reviewed changes

integ-test/src/test/java/org/opensearch/sql/sql/IdentifierIT.java Show resolved Hide resolved

dai-chen previously approved these changes Jun 26, 2023

View reviewed changes

Update documentation for _routing

ede3956

Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

acarbonetto dismissed stale reviews from dai-chen and Yury-Fridlyand via ede3956 June 27, 2023 20:58

acarbonetto added 2 commits June 27, 2023 14:58

Merge remote-tracking branch 'upstream/main' into integ-metafields-ro…

be90b0f

…uting Signed-off-by: acarbonetto <andrewc@bitquilltech.com>

Merge remote-tracking branch 'upstream/main' into integ-metafields-ro…

68745b7

…uting

forestmvey approved these changes Jun 28, 2023

View reviewed changes

Yury-Fridlyand approved these changes Jun 28, 2023

View reviewed changes

acarbonetto added the v2.9.0 v2.9.0 label Jul 4, 2023

Merge remote-tracking branch 'upstream/main' into integ-metafields-ro…

1c0b35d

…uting

MaxKsyunz approved these changes Jul 11, 2023

View reviewed changes

MaxKsyunz merged commit 876a9c8 into opensearch-project:main Jul 11, 2023

MaxKsyunz deleted the integ-metafields-routing branch July 11, 2023 06:32

opensearch-trigger-bot bot mentioned this pull request Jul 11, 2023

[Backport 2.x] Add _routing to SQL includes list (#277) #1831

Merged

penghuo added the backport 2.9 label Jul 11, 2023

opensearch-trigger-bot bot mentioned this pull request Jul 11, 2023

[Backport 2.9] Add _routing to SQL includes list (#277) #1843

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add _routing to SQL includes list (#277)#1771

Add _routing to SQL includes list (#277)#1771
MaxKsyunz merged 10 commits intoopensearch-project:mainfrom
Bit-Quill:integ-metafields-routing

acarbonetto commented Jun 23, 2023

Uh oh!

Yury-Fridlyand left a comment

Uh oh!

codecov bot commented Jun 23, 2023 •

edited

Loading

Uh oh!

acarbonetto commented Jun 23, 2023

Uh oh!

Yury-Fridlyand commented Jun 23, 2023

Uh oh!

Uh oh!

Uh oh!

dai-chen left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

acarbonetto commented Jun 23, 2023

Description

Issues Resolved

Check List

Uh oh!

Yury-Fridlyand left a comment

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jun 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

acarbonetto commented Jun 23, 2023

Uh oh!

Yury-Fridlyand commented Jun 23, 2023

Uh oh!

Uh oh!

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecov bot commented Jun 23, 2023 •

edited

Loading