[GRPC] Add DocumentService and Bulk GRPC endpoint v1 by karenyrx · Pull Request #17727 · opensearch-project/OpenSearch

karenyrx · 2025-03-30T02:51:38Z

Description

Implement DocumentService GRPC service, starting with the Bulk endpoint.

The goal is to maintain parity with the HTTP APIs.

The overall flow of how this works is:

The protobuf definitions are based off the OpenSearch API spec (the source of truth). (In the future, they will be auto-generated whenever the spec change.)
The compiled and generated protobuf Java code is published to the Sonatype Maven repo.
The protobuf jars are imported to Core repo as a Gradle dependency.
The Core implementation is done by tracing through the Bulk REST API implementation. It adds a correspondingly named utility class as the original class in the plugins/transport-grpc folder, with the suffix *ProtoUtils. Javadocs are added above each proto conversion method to easily reference the original REST method. The methods follow the HTTP side closely.

Test Plan

Start local grpc server

export JAVA_HOME=/opt/jvm/jdk-21
export PATH=$JAVA_HOME/bin:$PATH
which java

./gradlew run -PinstalledPlugins="['transport-grpc']" --info -Dtests.opensearch.aux.transport.types="[experimental-transport-grpc]" -Drepos.mavenLocal

Copy the raw protobufs to a local folder, and change path in below sample request accordingly:

grpcurl -import-path ~/opensearch/opensearch-protobufs -d @ -proto ~/opensearch/opensearch-protobufs/document_service.proto -plaintext localhost:9400 org.opensearch.protobufs.services.DocumentService/Bulk < bulk_grpc.json
{// payload}

Test 1: Testing by Passing all possible Request parameter fields

{
  "index": "my_index",
  "source": {
    "bool_value": true,
    "string_array": []
  },
  "source_excludes": [
    "field1",
    "field2"
  ],
  "source_includes": [
    "field3",
    "field4"
  ],
  "pipeline": "my_ingest_pipeline",
  "refresh": "REFRESH_TRUE",
  "require_alias": false,
  "routing": "1",
  "timeout": "1m",
  "type": "_doc",
  "wait_for_active_shards": {
    "int32_value": 1
  },
  "source_type": "SOURCE_TYPE_UNSPECIFIED",
  "request_body":[
    {
      "index":{
        "id":"2",
        "index":"my_index",
        "require_alias": false,
        "routing": "0",
        "pipeline" : "my_ingest_pipeline",
        "op_type" : "OP_TYPE_UNSPECIFIED",
        "version_type": "VERSION_TYPE_UNSPECIFIED"
      },
      "doc":"eyJ0aXRsZSI6Ik9wZW5TZWFyY2ggQmFzaWNzIn0=",
      "source": {
        "fetch" : true
      }
    },
    {
      "create":{
        "id":"1",
        "index":"my_index",
        "require_alias": false,
        "pipeline" : "my_ingest_pipeline",
        "routing": "1"
      },
      "doc":"eyJ0aXRsZSI6Ik9wZW5TZWFyY2ggQmFzaWNzIn0=",
      "source": {
        "fetch" : true
      }
    },
    {
      "update": {
        "id": "2",
        "index": "my_index",
        "routing": "0",
        "require_alias": false,
        "if_primary_term": 1,
        "if_seq_no":0
      },
      "doc": "eyJrZXkxIjoidmFsdWUyIn0K"
    },
    {
      "delete": {
        "id": "2",
        "index": "my_index",
        "routing": "0",
        "if_primary_term": 1,
        "if_seq_no": 1
      }
    }
  ]
}

Example successful response:

  "bulkResponseBody": {
    "errors": false,
    "items": [
      {
        "index": {
          "id": {
            "string": "2"
          },
          "index": "my_index",
          "status": 201,
          "primaryTerm": "1",
          "result": "created",
          "seqNo": "0",
          "shards": {
            "successful": 1,
            "total": 2
          },
          "version": "1",
          "forcedRefresh": true
        }
      },
      {
        "create": {
          "id": {
            "string": "1"
          },
          "index": "my_index",
          "status": 201,
          "primaryTerm": "1",
          "result": "created",
          "seqNo": "0",
          "shards": {
            "successful": 1,
            "total": 2
          },
          "version": "1",
          "forcedRefresh": true
        }
      },
      {
        "update": {
          "id": {
            "string": "2"
          },
          "index": "my_index",
          "status": 200,
          "primaryTerm": "1",
          "result": "updated",
          "seqNo": "1",
          "shards": {
            "successful": 1,
            "total": 2
          },
          "version": "2",
          "forcedRefresh": true,
          "get": {
            "found": true,
            "seqNo": "1",
            "primaryTerm": "1",
            "source": "e30="
          }
        }
      },
      {
        "delete": {
          "id": {
            "string": "2"
          },
          "index": "my_index",
          "status": 200,
          "primaryTerm": "1",
          "result": "deleted",
          "seqNo": "2",
          "shards": {
            "successful": 1,
            "total": 2
          },
          "version": "3",
          "forcedRefresh": true
        }
      }
    ],
    "took": "87",
    "ingestTook": "0"
  }
}

Verified parity with the HTTP response:

{"took":76,"ingest_took":0,"errors":false,"items":[{"index":{"_index":"my_index","_id":"2","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"create":{"_index":"my_index","_id":"1","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"update":{"_index":"my_index","_id":"2","_version":2,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1,"get":{"_seq_no":1,"_primary_term":1,"found":true,"_source":{}},"status":200}},{"delete":{"_index":"my_index","_id":"2","_version":3,"result":"deleted","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1,"status":200}}]}

Example error response (submitting the same "create" operation twice):

{
  "bulkResponseBody": {
    "errors": true,
    "items": [
      {
        "create": {
          "id": {
            "string": "3"
          },
          "index": "my_index1",
          "status": 409,
          "error": {
            "type": "version_conflict_engine_exception",
            "reason": "[3]: version conflict, document already exists (current version [1])",
            "stackTrace": "[my_index1/TxTenbTvQyySzhcM-vzrLg][[my_index1][0]] VersionConflictEngineException[[3]: version conflict, document already exists (current version [1])]\n"
          }
        }
      }
    ],
    "took": "2"
  }
}

Future work

Write an integration test for bulk request submitted E2E
Offline/manual parity testing of the GRPC vs HTTP Bulk requests and responses
Switch from using the SNAPSHOT jar to a jar from the central Maven repo
Fixing the Bulk Spec (PR: Fix Bulk API schemas opensearch-api-specification#843)
Converting HTTP status codes to GRPC ones in the response-side

Related Issues

#16784

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-03-30T02:57:01Z

❌ Gradle check result for 21dac0c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-30T03:03:49Z

❌ Gradle check result for 6a341bd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-31T17:00:19Z

❌ Gradle check result for 1485676: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-31T20:41:42Z

❌ Gradle check result for d763e8d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-31T20:59:10Z

❌ Gradle check result for 772f42e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-31T21:22:43Z

❌ Gradle check result for 8292a05: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-31T21:51:03Z

❌ Gradle check result for 62453b1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-31T22:57:51Z

❌ Gradle check result for b3a261c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Karen Xu <karenxyr@gmail.com>

github-actions · 2025-04-01T00:04:34Z

❌ Gradle check result for 5d83d26: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-03T21:02:30Z

❌ Gradle check result for e4faba2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Karen Xu <karenxyr@gmail.com>

Signed-off-by: Karen X <karenxyr@gmail.com>

andrross

What's the deal with the bouncy castle license files that have been removed?

github-actions · 2025-04-03T23:05:07Z

❕ Gradle check result for 09dd19d: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2025-04-03T23:05:38Z

Codecov Report

Attention: Patch coverage is 87.77090% with 79 lines in your changes missing coverage. Please review.

Project coverage is 72.39%. Comparing base (5487374) to head (3417719).
Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
...est/document/bulk/BulkRequestParserProtoUtils.java	79.06%	18 Missing and 18 partials ⚠️
...response/common/OpenSearchExceptionProtoUtils.java	80.72%	12 Missing and 4 partials ⚠️
...o/request/common/FetchSourceContextProtoUtils.java	83.72%	2 Missing and 5 partials ⚠️
...oto/response/document/get/GetResultProtoUtils.java	80.00%	4 Missing and 2 partials ⚠️
...onse/document/bulk/BulkItemResponseProtoUtils.java	85.71%	2 Missing and 3 partials ⚠️
...o/request/document/bulk/BulkRequestProtoUtils.java	86.36%	2 Missing and 1 partial ⚠️
...g/opensearch/plugin/transport/grpc/GrpcPlugin.java	80.00%	1 Missing and 1 partial ⚠️
...uest/document/bulk/ActiveShardCountProtoUtils.java	87.50%	2 Missing ⚠️
...rt/grpc/proto/request/common/ScriptProtoUtils.java	96.96%	0 Missing and 1 partial ⚠️
...ponse/document/common/DocumentFieldProtoUtils.java	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main   #17727    +/-   ##
==========================================
  Coverage     72.38%   72.39%            
- Complexity    66063    66264   +201     
==========================================
  Files          5358     5384    +26     
  Lines        306485   307127   +642     
  Branches      44402    44548   +146     
==========================================
+ Hits         221854   222344   +490     
- Misses        66435    66585   +150     
- Partials      18196    18198     +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Karen Xu <karenxyr@gmail.com>

karenyrx · 2025-04-04T05:21:55Z

What's the deal with the bouncy castle license files that have been removed?

can't seem to reproduce how i got it, but have removed it manually for now
(some combination of ./gradlew precommit + modifying build.gradle files in the test/framework/ directory + rebasing main?)

Signed-off-by: Karen Xu <karenxyr@gmail.com>

github-actions · 2025-04-04T05:46:47Z

❌ Gradle check result for 08e419e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Karen Xu <karenxyr@gmail.com>

github-actions · 2025-04-04T14:00:39Z

❕ Gradle check result for 3417719: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

karenyrx changed the title ~~[GRPC] DocumentService and Bulk GRPC endpoint v1~~ [GRPC] Add DocumentService and Bulk GRPC endpoint v1 Mar 30, 2025

finnegancarroll reviewed Mar 31, 2025

View reviewed changes

Comment thread .../org/opensearch/plugin/transport/grpc/proto/request/document/bulk/BulkRequestProtoUtils.java Outdated

karenyrx force-pushed the bulk_grpc_2 branch from d763e8d to 772f42e Compare March 31, 2025 20:52

karenyrx force-pushed the bulk_grpc_2 branch 3 times, most recently from 782097d to 8292a05 Compare March 31, 2025 21:08

karenyrx force-pushed the bulk_grpc_2 branch from 62453b1 to b3a261c Compare March 31, 2025 22:23

Add GRPC DocumentService and Bulk endpoint

5d83d26

Signed-off-by: Karen Xu <karenxyr@gmail.com>

karenyrx force-pushed the bulk_grpc_2 branch from b3a261c to 5d83d26 Compare March 31, 2025 23:06

karenyrx marked this pull request as ready for review April 1, 2025 00:09

karenyrx requested review from Bukhtawar, CEHENKLE, anasalkouz, andrross, ashking94, bugmakerrrrrr, cwperks, dblock, dbwiddis, gbbafna and jainankitk as code owners April 1, 2025 00:09

karenyrx added 2 commits April 3, 2025 21:22

update SHAs

5c30159

Signed-off-by: Karen Xu <karenxyr@gmail.com>

Merge branch 'main' into bulk_grpc_2

09dd19d

Signed-off-by: Karen X <karenxyr@gmail.com>

andrross reviewed Apr 3, 2025

View reviewed changes

Comment thread libs/core/src/main/java/org/opensearch/OpenSearchException.java

Comment thread plugins/transport-grpc/build.gradle

andrross reviewed Apr 3, 2025

View reviewed changes

Comment thread .gitignore Outdated

amberzsy reviewed Apr 4, 2025

View reviewed changes

Comment thread .../org/opensearch/plugin/transport/grpc/proto/request/document/bulk/BulkRequestProtoUtils.java

address comments and write more UTs

7dd6d21

Signed-off-by: Karen Xu <karenxyr@gmail.com>

amberzsy reviewed Apr 4, 2025

View reviewed changes

Comment thread ...ain/java/org/opensearch/plugin/transport/grpc/proto/response/common/ObjectMapProtoUtils.java

delete extra sha

08e419e

Signed-off-by: Karen Xu <karenxyr@gmail.com>

karenyrx force-pushed the bulk_grpc_2 branch from f9e65da to 08e419e Compare April 4, 2025 05:23

fix netty4grpcserver unit tests

3417719

Signed-off-by: Karen Xu <karenxyr@gmail.com>

finnegancarroll reviewed Apr 4, 2025

View reviewed changes

Comment thread .../org/opensearch/plugin/transport/grpc/proto/request/common/FetchSourceContextProtoUtils.java

andrross approved these changes Apr 4, 2025

View reviewed changes

finnegancarroll approved these changes Apr 4, 2025

View reviewed changes

Comment thread .../org/opensearch/plugin/transport/grpc/proto/request/common/FetchSourceContextProtoUtils.java

andrross merged commit 98568e8 into opensearch-project:main Apr 4, 2025
32 of 38 checks passed

peterzhuamazon mentioned this pull request Apr 8, 2025

[RELEASE] Release version 3.0.0 opensearch-project/opensearch-build#3747

Closed

66 tasks

This was referenced Apr 24, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SimpleSecureNetty4TransportTests #18071

Open

[AUTOCUT] Gradle Check Flaky Test Report for AzureBlobStoreRepositoryTests #14291

Open

BrewTestBot mentioned this pull request May 6, 2025

opensearch 3.0.0 Homebrew/homebrew-core#222665

Merged

This was referenced Jan 7, 2026

Add gRPC/Protobuf Transport feature documentation tkykenmt/opensearch-feature-explorer#39

Merged

docs: add gRPC Transport & Services report for v3.0.0 tkykenmt/opensearch-feature-explorer#321

Merged

Conversation

karenyrx commented Mar 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Plan

Test 1: Testing by Passing all possible Request parameter fields

Future work

Related Issues

Check List

Uh oh!

github-actions Bot commented Mar 30, 2025

Uh oh!

github-actions Bot commented Mar 30, 2025

Uh oh!

Uh oh!

github-actions Bot commented Mar 31, 2025

Uh oh!

github-actions Bot commented Mar 31, 2025

Uh oh!

github-actions Bot commented Mar 31, 2025

Uh oh!

github-actions Bot commented Mar 31, 2025

Uh oh!

github-actions Bot commented Mar 31, 2025

Uh oh!

github-actions Bot commented Mar 31, 2025

Uh oh!

github-actions Bot commented Apr 1, 2025

Uh oh!

github-actions Bot commented Apr 3, 2025

Uh oh!

andrross left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 3, 2025

Uh oh!

codecov Bot commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

karenyrx commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 4, 2025

Uh oh!

github-actions Bot commented Apr 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

karenyrx commented Mar 30, 2025 •

edited

Loading

codecov Bot commented Apr 3, 2025 •

edited

Loading

karenyrx commented Apr 4, 2025 •

edited

Loading