-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Enable ML binary format from broker configuration #282
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Note: this PR is based on a branch that contains the changes for #276. I will rebase once that PR gets merged. |
e50a3cb to
9d75a5a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what if persisting into zk fails? Shouldn't we store individualDeletedPosition in both the case Test and Binary
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you're writing in text format, we don't put the current position + individually deleted message ranges into the z-node, but fallback to append to current ledger, where the info is stored in binary format
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I understood that on TextFormat we only write in ledger as we can't write in znode. However, I meant: if we are writing in Binary format and on cursor-close if znode-update fails then we won't have latest individualDeletedMessages list in ledger, which is fine and we were not handling it before as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, that's the same behavior as if the ledger.addEntry() fails, we fail the cursor.close() operation and then it will recover from the cursor ledger next time.
The reason to write directly on the z-node was to speed-up the cursor recovery by needing just one zk read to be completed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should fail callback rather throwing RuntimeException as it is also being called from non-try-catch-async-callback
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ups, forgot that line that was there for debug
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you think is it fine to combine two ledger/cursor parse method to:
parseInfoFromText(final byte[] data, final Message.Builder builder)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just wanted to make sure that we first try to deserialize in the format in which we expect the data to be in, to avoid always catching exceptions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For 1.18 release, when the only option will be to write in binary, we can collapse the 2 method and always try to deserialize first in binary and then in text.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same way here to combine two methods?
|
just FYI: https://github.com/yahoo/pulsar/tree/persist-individually-deleted-messages-br should we delete branch? |
|
I created that branch, so that this PR would only show intended changes. I'll rebase on master now and remove that temp branch. |
9d75a5a to
9ac741d
Compare
|
@rdhabalia Updated |
rdhabalia
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
9ac741d to
61c0f57
Compare
…footer of the chunked data (apache#282) (cherry picked from commit 6e72ecb)
…footer of the chunked data (apache#282) (cherry picked from commit 6e72ecb)
…footer of the chunked data (apache#282) (cherry picked from commit 6e72ecb)
…footer of the chunked data (apache#282) (cherry picked from commit 6e72ecb)
Motivation
First part of changes needed for #281. Tool to access the data will come in separate PR.
Modifications
Added config switch to enable protobuf binary format for ManagedLedger and ManagedCursor z-nodes.