Skip to content

Conversation

@merlimat
Copy link
Contributor

@merlimat merlimat commented Mar 8, 2017

Motivation

First part of changes needed for #281. Tool to access the data will come in separate PR.

Modifications

Added config switch to enable protobuf binary format for ManagedLedger and ManagedCursor z-nodes.

@merlimat merlimat added the type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages label Mar 8, 2017
@merlimat merlimat added this to the 1.17 milestone Mar 8, 2017
@merlimat merlimat self-assigned this Mar 8, 2017
@merlimat
Copy link
Contributor Author

merlimat commented Mar 8, 2017

Note: this PR is based on a branch that contains the changes for #276. I will rebase once that PR gets merged.

@merlimat merlimat force-pushed the enable-binary-format branch from e50a3cb to 9d75a5a Compare March 8, 2017 22:35
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if persisting into zk fails? Shouldn't we store individualDeletedPosition in both the case Test and Binary

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you're writing in text format, we don't put the current position + individually deleted message ranges into the z-node, but fallback to append to current ledger, where the info is stored in binary format

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I understood that on TextFormat we only write in ledger as we can't write in znode. However, I meant: if we are writing in Binary format and on cursor-close if znode-update fails then we won't have latest individualDeletedMessages list in ledger, which is fine and we were not handling it before as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, that's the same behavior as if the ledger.addEntry() fails, we fail the cursor.close() operation and then it will recover from the cursor ledger next time.

The reason to write directly on the z-node was to speed-up the cursor recovery by needing just one zk read to be completed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should fail callback rather throwing RuntimeException as it is also being called from non-try-catch-async-callback

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ups, forgot that line that was there for debug

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you think is it fine to combine two ledger/cursor parse method to:
parseInfoFromText(final byte[] data, final Message.Builder builder)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just wanted to make sure that we first try to deserialize in the format in which we expect the data to be in, to avoid always catching exceptions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For 1.18 release, when the only option will be to write in binary, we can collapse the 2 method and always try to deserialize first in binary and then in text.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same way here to combine two methods?

@rdhabalia
Copy link
Contributor

@merlimat
Copy link
Contributor Author

I created that branch, so that this PR would only show intended changes. I'll rebase on master now and remove that temp branch.

@merlimat merlimat force-pushed the enable-binary-format branch from 9d75a5a to 9ac741d Compare March 10, 2017 19:50
@merlimat merlimat changed the base branch from persist-individually-deleted-messages-br to master March 10, 2017 19:50
@merlimat
Copy link
Contributor Author

@rdhabalia Updated

Copy link
Contributor

@rdhabalia rdhabalia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@merlimat merlimat force-pushed the enable-binary-format branch from 9ac741d to 61c0f57 Compare March 10, 2017 22:38
@merlimat merlimat merged commit 146738c into apache:master Mar 10, 2017
dlg99 added a commit to dlg99/pulsar that referenced this pull request May 28, 2024
…footer of the chunked data (apache#282)

(cherry picked from commit 6e72ecb)
dlg99 added a commit to dlg99/pulsar that referenced this pull request Sep 9, 2024
…footer of the chunked data (apache#282)

(cherry picked from commit 6e72ecb)
dlg99 added a commit to dlg99/pulsar that referenced this pull request Sep 16, 2024
…footer of the chunked data (apache#282)

(cherry picked from commit 6e72ecb)
dlg99 added a commit to dlg99/pulsar that referenced this pull request Sep 24, 2024
…footer of the chunked data (apache#282)

(cherry picked from commit 6e72ecb)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants