Skip to content

Add X-Opaque-Id to search request metadata for query insights#13374

Merged
msfroh merged 5 commits intoopensearch-project:mainfrom
ansjcy:poc-tenant-labeling
Jun 7, 2024
Merged

Add X-Opaque-Id to search request metadata for query insights#13374
msfroh merged 5 commits intoopensearch-project:mainfrom
ansjcy:poc-tenant-labeling

Conversation

@ansjcy
Copy link
Copy Markdown
Member

@ansjcy ansjcy commented Apr 25, 2024 •
edited
Loading

Description

Customized and labeling for search queries in top n queries

  • Added the ability to send customized labels in a search query.
  • Use top n queries service in query insights plugin to read those tags

A simple demo test of the changes:

Steps

  • Preparation: First spin up a test cluster, enable security and query insights plugin, and index one document.
curl -X PUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d'
{
    "persistent" : {
        "search.insights.top_queries.latency.enabled" : "true",
        "search.insights.top_queries.latency.window_size" : "1m",
        "search.insights.top_queries.latency.top_n_size" : 20
    }
}'

curl -X POST "localhost:9200/my-index-0/_doc/?pretty" -H 'Content-Type: application/json' -d'
{
  "@timestamp": "2099-11-15T13:12:00",
  "message": "this is document 1",
  "user": {
    "id": "cyji"
  }
}'
  • Do search query on the index with a customized label with X-Opaque-Id
curl -H "X-Opaque-Id: customized-id" -X GET "localhost:9200/my-index-*/_search?size=1000&pretty" -H 'Content-Type: application/json' -d '{}'
  • Then query top n queries API, validate the customized label my_tag is there in the top n queries results
✗ curl -X GET "localhost:9200/_insights/top_queries?pretty"
{
  "top_queries" : [
    {
      "timestamp" : 1717620404073,
      "phase_latency_map" : {
        "expand" : 0,
        "query" : 52,
        "fetch" : 2
      },
      "total_shards" : 1,
      "labels" : {
        "X-Opaque-Id" : "customized-id"  // <--------- user provided label 
      },
      "node_id" : "NJZTMuTfTOe-5nyyE1SMtg",
      "source" : "{\"size\":1000}",
      "indices" : [
        "my-index-*"
      ],
      "search_type" : "query_then_fetch",
      "latency" : 3525
    }
  ]
}

Deprecated for the alternative approach: A simple demo test of the changes:

Steps

  • Preparation: First spin up a test cluster, enable query insights plugin, and index one document.
curl -X PUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d'
{
    "persistent" : {
        "search.insights.top_queries.latency.enabled" : "true",
        "search.insights.top_queries.latency.window_size" : "1m",
        "search.insights.top_queries.latency.top_n_size" : 20
    }
}'

curl -X POST "localhost:9200/my-index-0/_doc/?pretty" -H 'Content-Type: application/json' -d'
{
  "@timestamp": "2099-11-15T13:12:00",
  "message": "this is document 1",
  "user": {
    "id": "cyji"
  }
}'
  • Do search query on the index with a customized label
curl -X GET "localhost:9200/my-index-*/_search?size=1000&pretty" -H 'Content-Type: application/json' -d '{
  "query" : {
    "term": {
      "user.id": "chenyang"
    }
  },
  "labels": {
    "my_tenant_tag": "customized-tenant-id"
  }
}'
  • Then query top n queries API, validate the customized label my_tenant_tag and default labels injected by the default rules are there in the top n queries results
curl -X GET "localhost:9200/_insights/top_queries?pretty"
{
  "top_queries" : [
    {
      "timestamp" : 1714010081376,
      "user_name" : "chenyang",  // <--------- default label 
      "phase_latency_map" : {
        "expand" : 0,
        "query" : 26,
        "fetch" : 1
      },
      "search_type" : "query_then_fetch",
      "source" : "{\"size\":1000,\"query\":{\"term\":{\"user.id\":{\"value\":\"cyji\",\"boost\":1.0}}},\"labels\":{\"my_tenant_tag\":\"customized-tenant-id\",\"user_tenant\":\"t1\",\"user_backend_roles\":[\"br1\",\"b2\"],\"user_name\":\"chenyang\",\"remote_address\":\"1.2.3.4\",\"user_roles\":[\"r1\",\"r2\"]}}",
      "customized_tag" : "customized-tenant-id",  // <--------- User customized label
      "total_shards" : 1,
      "indices" : [
        "my-index-*"
      ],
      "node_id" : "ZeIqCaZtQ3GPDdRwLuXorg",
      "latency" : 38
    }
  ]
}

Detailed Results

res

Related Issues

#13341

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ansjcy ansjcy marked this pull request as draft April 25, 2024 02:03
This was referenced Apr 25, 2024
Open
Open
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 25, 2024

❌ Gradle check result for 62b17bf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@ansjcy ansjcy mentioned this pull request May 2, 2024
Open
5 tasks
@ansjcy ansjcy changed the title [DRAFT] POC for customized and rule-based labeling for search queries Customized and rule-based labeling for search queries May 20, 2024
@ansjcy ansjcy marked this pull request as ready for review May 20, 2024 23:15
ansjcy added a commit to ansjcy/OpenSearch that referenced this pull request Jun 7, 2024
@ansjcy
Add X-Opaque-Id to search request metadata for query insights (opense…
dcb30c3 
…arch-project#13374)

---------

Signed-off-by: Chenyang Ji <cyji@amazon.com>
(cherry picked from commit 9d3cf43)
Signed-off-by: Chenyang Ji <cyji@amazon.com>
@ansjcy ansjcy mentioned this pull request Jun 7, 2024
Merged
3 tasks
jed326 pushed a commit that referenced this pull request Jun 7, 2024
@ansjcy
Add X-Opaque-Id to search request metadata for query insights (#13374) (
34af8c7 
#14078)

---------


(cherry picked from commit 9d3cf43)

Signed-off-by: Chenyang Ji <cyji@amazon.com>
parv0201 pushed a commit to parv0201/OpenSearch that referenced this pull request Jun 10, 2024
@ansjcy @parv0201
Add X-Opaque-Id to search request metadata for query insights (opense…
1d3ab4a 
…arch-project#13374)


---------

Signed-off-by: Chenyang Ji <cyji@amazon.com>
@ansjcy ansjcy mentioned this pull request Jun 10, 2024
Closed
This was referenced Jun 13, 2024
Open
Open
Closed
Closed
Open
Open
Open
Open
Open
Open
Open
@finnegancarroll finnegancarroll mentioned this pull request Jun 14, 2024
Merged
3 tasks
@prudhvigodithi prudhvigodithi mentioned this pull request Jun 14, 2024
Closed
@BrewTestBot BrewTestBot mentioned this pull request Jun 26, 2024
Merged
This was referenced Jun 27, 2024
Closed
Open
kkewwei pushed a commit to kkewwei/OpenSearch that referenced this pull request Jul 24, 2024
@ansjcy @kkewwei
Add X-Opaque-Id to search request metadata for query insights (opense…
eb540e7 
…arch-project#13374) (opensearch-project#14078)

---------

(cherry picked from commit 9d3cf43)

Signed-off-by: Chenyang Ji <cyji@amazon.com>
Signed-off-by: kkewwei <kkewwei@163.com>
wdongyu pushed a commit to wdongyu/OpenSearch that referenced this pull request Aug 22, 2024
@ansjcy
Add X-Opaque-Id to search request metadata for query insights (opense…
3ebe340 
…arch-project#13374)


---------

Signed-off-by: Chenyang Ji <cyji@amazon.com>
This was referenced Sep 6, 2024
Closed
Open
@opensearch-ci-bot opensearch-ci-bot mentioned this pull request Sep 24, 2024
Closed
@prudhvigodithi prudhvigodithi mentioned this pull request Oct 16, 2024
Open
@msfroh msfroh mentioned this pull request Sep 23, 2025
Open
This was referenced Feb 18, 2026
Open
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jainankitk jainankitk jainankitk left review comments

@msfroh msfroh msfroh approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz

@andrross andrross Awaiting requested review from andrross

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE

@dblock dblock Awaiting requested review from dblock

@dbwiddis dbwiddis Awaiting requested review from dbwiddis

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@gbbafna gbbafna Awaiting requested review from gbbafna

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal

@mch2 mch2 Awaiting requested review from mch2

@nknize nknize Awaiting requested review from nknize

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19

@reta reta Awaiting requested review from reta

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159

@sachinpkale sachinpkale Awaiting requested review from sachinpkale

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli

@shwetathareja shwetathareja Awaiting requested review from shwetathareja

@sohami sohami Awaiting requested review from sohami

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah

+2 more reviewers

@deshsidd deshsidd deshsidd left review comments

@kaushalmahi12 kaushalmahi12 kaushalmahi12 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

backport 2.x Backport to 2.x branch backport-failed v2.15.0 Issues and PRs related to version 2.15.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ansjcy @jainankitk @msfroh @deshsidd @owaiskazi19 @kaushalmahi12