Skip to content

[Issue 4804][Schema] encode null key/value in KeyValue #6384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
nlu90 wants to merge 4 commits into from
Closed

[Issue 4804][Schema] encode null key/value in KeyValue #6384

nlu90 wants to merge 4 commits into from

Conversation

@nlu90
Copy link
Member

@nlu90 nlu90 commented Feb 20, 2020

Fixes #4804

Motivation

Allow specifying null key or null value in KeyValue Schema

Modifications

  • encode null key/value information in KeyValue

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): no
  • The public API: no
  • The schema: yes
  • The default values of configurations: no
  • The wire protocol: no
  • The rest endpoints: no
  • The admin cli options: no
  • Anything that affects deployment: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@nlu90 nlu90 requested a review from sijie February 20, 2020 19:14
@sijie sijie added this to the 2.6.0 milestone Feb 22, 2020
sijie
sijie requested changes Feb 24, 2020
View reviewed changes
Copy link
Member

@sijie sijie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nlu90 can you add unit tests? It would be great to cover different serialization format as well.

pulsar-client-api/src/main/java/org/apache/pulsar/common/schema/KeyValue.java
ByteBuffer byteBuffer = ByteBuffer.allocate(4 + keyBytes.length + 4 + valueBytes.length);
byteBuffer.putInt(keyBytes.length).put(keyBytes).putInt(valueBytes.length).put(valueBytes);

int keyEncodeLength = 1 + (null == keyBytes ? 0 : 4 + keyBytes.length);
Copy link
Member

@sijie sijie Feb 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that it is breaking the backward compatibility. Because it introduces a new format of how key/value is stored. Can we introduce versioning to the format and use message properties to indicate which version of the forrmat that is used for serializing key/value pairs?

@codelipenghui
Copy link
Contributor

codelipenghui commented May 19, 2020

@gaoran10 Could you please help take a look at this PR? Looks related to #6379.

@gaoran10 gaoran10 mentioned this pull request Jun 4, 2020
Merged
1 task
codelipenghui pushed a commit that referenced this pull request Jun 4, 2020
@gaoran10
Support KeyValue Schema Use Null Key And Null Value (#7139)
1d5ff26 
Fixes #4804  

Thanks, @nlu90, work at #6384.

### Motivation

Currently, the KeyValue schema doesn't handle `null` key and `null` value well.
@codelipenghui
Copy link
Contributor

codelipenghui commented Jun 4, 2020

close via #7139

cdbartholomew pushed a commit to kafkaesque-io/pulsar that referenced this pull request Jul 24, 2020
@gaoran10 @cdbartholomew
Support KeyValue Schema Use Null Key And Null Value (apache#7139)
6bc8af2 
Fixes apache#4804  

Thanks, @nlu90, work at apache#6384.

### Motivation

Currently, the KeyValue schema doesn't handle `null` key and `null` value well.
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Aug 24, 2020
@gaoran10 @huangdx0726
Support KeyValue Schema Use Null Key And Null Value (apache#7139)
6d08a44 
Fixes apache#4804  

Thanks, @nlu90, work at apache#6384.

### Motivation

Currently, the KeyValue schema doesn't handle `null` key and `null` value well.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sijie sijie sijie requested changes

Assignees

@nlu90 nlu90

Labels

area/client

Projects

None yet

Milestone

2.6.0

Development

Successfully merging this pull request may close these issues.

KeyValue schema doesn't handle null key and null value well

3 participants

@nlu90 @codelipenghui @sijie