[improve] configure whether function consumer should skip to latest by nlu90 · Pull Request #17214 · apache/pulsar

nlu90 · 2022-08-22T23:11:43Z

(If this PR fixes a github issue, please add Fixes #<xyz>.)

Fixes #

(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)

Master Issue: #

Motivation

In certain failure cases, the function needs to skip all the content between the last successfully Acked message and the latest message in the topic in order for quick recovery.

Modifications

Providing a boolean config for function submission cmd
PulsarSource will call consumer.seek(MessageId.latest) if the skip flag is set

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below or label this PR directly.

Need to update docs?

doc-required
(Your PR needs to update docs and you will update later)
doc-not-needed
(Please explain why)
doc
(Your PR contains doc changes)
doc-complete
(Docs have been already added)

liudezhi2098 · 2022-08-23T00:41:50Z

/pulsarbot run-failure-checks

github-actions · 2022-11-08T02:14:55Z

The pr had no activity for 30 days, mark with Stale label.

liudezhi2098 · 2022-11-11T07:24:14Z

@nlu90 help handle Conflicting files.

github-actions · 2022-12-13T02:05:24Z

The pr had no activity for 30 days, mark with Stale label.

nlu90 · 2023-01-30T18:23:57Z

/pulsarbot run-failure-checks

codecov-commenter · 2023-02-07T01:57:32Z

Codecov Report

Merging #17214 (a94a2b2) into master (644be5f) will increase coverage by 28.98%.
The diff coverage is 35.71%.

@@              Coverage Diff              @@
##             master   #17214       +/-   ##
=============================================
+ Coverage     32.42%   61.40%   +28.98%     
- Complexity     6347    23020    +16673     
=============================================
  Files          1644     1833      +189     
  Lines        123712   143586    +19874     
  Branches      13486    16890     +3404     
=============================================
+ Hits          40109    88170    +48061     
+ Misses        77694    47233    -30461     
- Partials       5909     8183     +2274

Flag	Coverage Δ
inttests	`26.39% <0.00%> (+1.45%)`	⬆️
systests	`26.74% <35.71%> (+1.08%)`	⬆️
unittests	`57.85% <21.42%> (+40.31%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...java/org/apache/pulsar/admin/cli/CmdFunctions.java	`47.45% <0.00%> (+47.45%)`	⬆️
...ar/functions/source/MultiConsumerPulsarSource.java	`89.47% <0.00%> (-2.20%)`	⬇️
...r/functions/source/SingleConsumerPulsarSource.java	`93.33% <0.00%> (+4.04%)`	⬆️
...he/pulsar/functions/utils/FunctionConfigUtils.java	`62.87% <33.33%> (+22.58%)`	⬆️
...apache/pulsar/common/functions/FunctionConfig.java	`96.42% <100.00%> (+0.06%)`	⬆️
...ulsar/functions/instance/JavaInstanceRunnable.java	`54.63% <100.00%> (+0.38%)`	⬆️
...he/pulsar/functions/source/PulsarSourceConfig.java	`81.81% <100.00%> (+1.81%)`	⬆️
...r/metadata/api/extended/MetadataStoreExtended.java	`40.00% <0.00%> (-10.00%)`	⬇️
...ce/ConsistentHashingStickyKeyConsumerSelector.java	`76.92% <0.00%> (-3.85%)`	⬇️
.../apache/pulsar/client/impl/TopicMessageIdImpl.java	`82.60% <0.00%> (-2.01%)`	⬇️
... and 1231 more

michaeljmarshall · 2023-02-24T03:17:50Z

    bool cleanupSubscription = 11;
    SubscriptionPosition subscriptionPosition = 12;
    uint64 negativeAckRedeliveryDelayMs = 13;
+    bool skipToLatest = 14;


@nlu90 - was this discussed on the mailing list? This is a protobuf change. We have a general rule that those go through the PIP process. While there could be exceptions, we should always post about these kinds of changes on the mailing list.

@michaeljmarshall As we discussed via slack, I'll email the mailing list about the function proto change.
But it should be backward compatible and won't cause any issue for previous users.

momo-jun · 2023-02-27T06:18:28Z

Hi @nlu90, I'm checking in to follow up with the doc updates. Did you have any plans to update the docs?

…latest (apache#17214)" This reverts commit bf982f4.

nlu90 · 2023-02-27T20:59:54Z

Hi @nlu90, I'm checking in to follow up with the doc updates. Did you have any plans to update the docs?

@momo-jun Let me know where I can update the docs.

momo-jun · 2023-02-28T02:49:37Z

@nlu90 shall we add skipToLatest to here? Maybe @freeznet @liudezhi2098 can also advise.
BTW, the pulsar docs have been moved to the site repo.

github-actions Bot added the doc-required Your PR changes impact docs and you will update later. label Aug 22, 2022

configure whether function consumer should skip to latest

3e3aa03

nlu90 force-pushed the neng/function-consume-skip-to-latest branch from 0ebc332 to 3e3aa03 Compare August 22, 2022 23:13

nlu90 requested review from freeznet and liudezhi2098 August 22, 2022 23:19

liudezhi2098 approved these changes Oct 4, 2022

View reviewed changes

freeznet approved these changes Oct 8, 2022

View reviewed changes

github-actions Bot added the Stale label Nov 8, 2022

github-actions Bot removed the Stale label Nov 12, 2022

github-actions Bot added the Stale label Dec 13, 2022

nlu90 added 3 commits January 27, 2023 13:42

Merge branch 'master' into neng/function-consume-skip-to-latest

7ef95df

remove tailing whitespace

993bd83

check skipToLatest null

a94a2b2

github-actions Bot removed the Stale label Jan 31, 2023

codelipenghui assigned nlu90 Feb 7, 2023

codelipenghui added this to the 3.0.0 milestone Feb 7, 2023

codelipenghui closed this Feb 7, 2023

codelipenghui reopened this Feb 7, 2023

nlu90 changed the title ~~[WIP] configure whether function consumer should skip to latest~~ [feat] configure whether function consumer should skip to latest Feb 7, 2023

nlu90 changed the title ~~[feat] configure whether function consumer should skip to latest~~ [improve] configure whether function consumer should skip to latest Feb 7, 2023

nlu90 merged commit bf982f4 into apache:master Feb 24, 2023

michaeljmarshall reviewed Feb 24, 2023

View reviewed changes

michaeljmarshall added a commit to michaeljmarshall/pulsar that referenced this pull request Feb 27, 2023

Revert "[improve] configure whether function consumer should skip to …

6c36169

…latest (apache#17214)" This reverts commit bf982f4.

michaeljmarshall mentioned this pull request Feb 27, 2023

Revert "[improve] configure whether function consumer should skip to latest (#17214)" #19651

Closed

1 task

codelipenghui deleted the neng/function-consume-skip-to-latest branch March 14, 2023 00:34

momo-jun mentioned this pull request Mar 15, 2023

[feat][doc] Add new configs skipToLatest and compressionType apache/pulsar-site#461

Merged

4 tasks

momo-jun added doc-complete Your PR changes impact docs and the related docs have been already added. and removed doc-required Your PR changes impact docs and you will update later. labels Mar 22, 2023

flowchartsman mentioned this pull request Aug 22, 2023

skipToLatest breaks function creation if < pulsar 3.0 streamnative/pulsar-admin-go#43

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improve] configure whether function consumer should skip to latest#17214

[improve] configure whether function consumer should skip to latest#17214
nlu90 merged 4 commits into
apache:masterfrom
streamnative:neng/function-consume-skip-to-latest

nlu90 commented Aug 22, 2022

Uh oh!

liudezhi2098 commented Aug 23, 2022

Uh oh!

github-actions Bot commented Nov 8, 2022

Uh oh!

liudezhi2098 commented Nov 11, 2022

Uh oh!

github-actions Bot commented Dec 13, 2022

Uh oh!

nlu90 commented Jan 30, 2023

Uh oh!

codecov-commenter commented Feb 7, 2023 •

edited

Loading

Uh oh!

michaeljmarshall Feb 24, 2023

Uh oh!

nlu90 Feb 27, 2023

Uh oh!

momo-jun commented Feb 27, 2023

Uh oh!

nlu90 commented Feb 27, 2023

Uh oh!

momo-jun commented Feb 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

nlu90 commented Aug 22, 2022

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

liudezhi2098 commented Aug 23, 2022

Uh oh!

github-actions Bot commented Nov 8, 2022

Uh oh!

liudezhi2098 commented Nov 11, 2022

Uh oh!

github-actions Bot commented Dec 13, 2022

Uh oh!

nlu90 commented Jan 30, 2023

Uh oh!

codecov-commenter commented Feb 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

michaeljmarshall Feb 24, 2023

Choose a reason for hiding this comment

Uh oh!

nlu90 Feb 27, 2023

Choose a reason for hiding this comment

Uh oh!

momo-jun commented Feb 27, 2023

Uh oh!

nlu90 commented Feb 27, 2023

Uh oh!

momo-jun commented Feb 28, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecov-commenter commented Feb 7, 2023 •

edited

Loading