Skip to content

[Pull-based Ingestion] Support consumer reset in Resume API#18332

Merged
andrross merged 3 commits intoopensearch-project:mainfrom
varunbharadwaj:vb/offsetupdate
May 28, 2025
Merged

[Pull-based Ingestion] Support consumer reset in Resume API#18332
andrross merged 3 commits intoopensearch-project:mainfrom
varunbharadwaj:vb/offsetupdate

Conversation

@varunbharadwaj
Copy link
Copy Markdown
Contributor

Description

This PR is a follow up to ingestion management API effort, to support consumer reset in the resume API. This PR includes the following changes.

  1. Support shard pointer reset in resume API.
  2. Include batchStartPointer in GetIngestionState API response.
  3. Add metric to track number of duplicate messages that are skipped on rewind.
  4. Rename reset state enum from rewind to reset. Example: REWIND_BY_OFFSET to RESET_BY_OFFSET. This will be a backward incompatible change, applicable from 3.1.0 release onwards.

Related Issues

Resolves #17442

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Indexing Indexing, Bulk Indexing and anything related to indexing labels May 19, 2025
@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 5bde213: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@varunbharadwaj varunbharadwaj changed the title [Pull-based Ingestion] Support consumer reset in Resume API [WIP][Pull-based Ingestion] Support consumer reset in Resume API May 19, 2025
@varunbharadwaj varunbharadwaj force-pushed the vb/offsetupdate branch 2 times, most recently from 0bd390a to cf6b842 Compare May 20, 2025 06:59
@varunbharadwaj varunbharadwaj changed the title [WIP][Pull-based Ingestion] Support consumer reset in Resume API [Pull-based Ingestion] Support consumer reset in Resume API May 20, 2025
@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for cf6b842: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 812d39c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 38fc627: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 38fc627: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 0a3b372: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 0a3b372: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 0a3b372: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for a862103: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@varunbharadwaj varunbharadwaj force-pushed the vb/offsetupdate branch 2 times, most recently from d7c4b92 to 6948986 Compare May 20, 2025 22:56
@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for 6948986: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

✅ Gradle check result for 98e0207: SUCCESS

Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>
Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>
Signed-off-by: Varun Bharadwaj <varunbharadwaj1995@gmail.com>
@github-actions
Copy link
Copy Markdown
Contributor

❌ Gradle check result for c6f578b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Copy Markdown
Contributor

✅ Gradle check result for c6f578b: SUCCESS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Enhancement or improvement to existing feature or request Indexing Indexing, Bulk Indexing and anything related to indexing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Ingestion management APIs for pull-based ingestion

3 participants