[improve][broker] Upgrade avro version to 1.12.0 #24617

hangc0276 · 2025-08-09T03:15:25Z

Fixes #xyz

Main Issue: #xyz

PIP: #xyz

Motivation

Upgrade the Avro version.

Modifications

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository:

github-actions · 2025-08-09T03:15:50Z

@hangc0276 Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

horizonzy

LGTM.

lhotari · 2025-08-09T19:09:35Z

@hangc0276 It seems that there are test failures:

AvroSchemaTest.testTimestampWithJsonDef: pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java#L515Can not set java.time.Instant field org.apache.pulsar.client.impl.schema.AvroSchemaTest$TimestampPojo.value to java.lang.Long

CI - Unit - Pulsar ClientProcess completed with exit code 1.
TopicSchemaTest.testGetSchema: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/schema/TopicSchemaTest.java#L48Namespace part "Request$ServiceRequest" is invalid: Illegal character in: Request$ServiceRequest
CI - Unit - OtherProcess completed with exit code 1.
SchemaDataValidatorTest.testJsonSchemaTypeWithJsonSchemaData: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/schema/validator/SchemaDataValidatorTest.java#L134Invalid schema definition data for JSON schema

AvroSchemaTest.testTimestampWithJsonDef: pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java#L515
Can not set java.time.Instant field org.apache.pulsar.client.impl.schema.AvroSchemaTest$TimestampPojo.value to java.lang.Long
CI - Unit - Pulsar Client
Process completed with exit code 1.
TopicSchemaTest.testGetSchema: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/schema/TopicSchemaTest.java#L48
Namespace part "Request$ServiceRequest" is invalid: Illegal character in: Request$ServiceRequest
CI - Unit - Other
Process completed with exit code 1.
SchemaDataValidatorTest.testJsonSchemaTypeWithJsonSchemaData: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/schema/validator/SchemaDataValidatorTest.java#L134
Invalid schema definition data for JSON schema

Technoboy- · 2025-08-12T04:06:30Z

@coderzc For AvroSchemaTest.testTimestampWithJsonDef, in Avro 1.11.4, it use Unsafe.putObject(object, offset, value) to set the object value when decoding. But in 1.12.0, it changed to use Field.set(object, value), the later one could check the object type, so it could throw below error

Can not set java.time.Instant field org.apache.pulsar.client.impl.schema.AvroSchemaTest$TimestampPojo.value to java.lang.Long

can we delete this assertion, as it may not involve break changes ?

pulsar/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java

Lines 510 to 517 in 277835a

    
           AvroSchema<TimestampPojo> schemaWithJsonDefNoClassLoader = 
        
                   AvroSchema.of(SchemaDefinition.<TimestampPojo>builder() 
        
                   .withJsonDef(schemaDefinition) 
        
                   .withJSR310ConversionEnabled(false).build()); 
        
           TimestampPojo decodeWithJsonNoClassLoader = schemaWithJsonDefNoClassLoader.decode(encode); 
        
           Assert.assertNotEquals(decodeWithJsonNoClassLoader, decodeWithPojo); 
        
           Assert.assertNotEquals(Instant.class, decodeWithJsonNoClassLoader.getValue().getClass());

coderzc · 2025-08-12T05:55:46Z

@coderzc For AvroSchemaTest.testTimestampWithJsonDef, in Avro 1.11.4, it use Unsafe.putObject(object, offset, value) to set the object value when decoding. But in 1.12.0, it changed to use Field.set(object, value), the later one could check the object type, so it could throw below error
Can not set java.time.Instant field org.apache.pulsar.client.impl.schema.AvroSchemaTest$TimestampPojo.value to java.lang.Long
can we delete this assertion, as it may not involve break changes ?

pulsar/pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java

Lines 510 to 517 in 277835a

AvroSchema<TimestampPojo> schemaWithJsonDefNoClassLoader =

AvroSchema.of(SchemaDefinition.<TimestampPojo>builder()

.withJsonDef(schemaDefinition)

.withJSR310ConversionEnabled(false).build());

TimestampPojo decodeWithJsonNoClassLoader = schemaWithJsonDefNoClassLoader.decode(encode);

Assert.assertNotEquals(decodeWithJsonNoClassLoader, decodeWithPojo);

Assert.assertNotEquals(Instant.class, decodeWithJsonNoClassLoader.getValue().getClass());

I think we can delete this assertion, this should be a misuse, If you want to customize the conversion, you can set the classLoader.

Technoboy- · 2025-08-13T08:03:23Z

@gaoran10 Please help review the fix of SchemaDataValidatorTest

codecov-commenter · 2025-08-13T08:08:00Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.31%. Comparing base (3d57c60) to head (a93db06).
⚠️ Report is 33 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #24617       +/-   ##
=============================================
+ Coverage     38.80%   74.31%   +35.50%     
- Complexity    12984    33165    +20181     
=============================================
  Files          1824     1881       +57     
  Lines        142741   146850     +4109     
  Branches      16389    16866      +477     
=============================================
+ Hits          55392   109129    +53737     
+ Misses        79916    29056    -50860     
- Partials       7433     8665     +1232

Flag	Coverage Δ
inttests	`26.65% <50.00%> (-0.20%)`	⬇️
systests	`23.26% <16.66%> (-0.04%)`	⬇️
unittests	`73.80% <100.00%> (+39.18%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...r/service/schema/JsonSchemaCompatibilityCheck.java	`47.05% <100.00%> (+17.64%)`	⬆️
...che/pulsar/client/impl/schema/util/SchemaUtil.java	`80.55% <100.00%> (+8.33%)`	⬆️
.../apache/pulsar/io/elasticsearch/JsonConverter.java	`71.55% <100.00%> (+71.55%)`	⬆️
...g/apache/pulsar/io/kinesis/json/JsonConverter.java	`67.96% <100.00%> (+67.96%)`	⬆️

... and 1380 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java

Co-authored-by: Jiwe Guo <technoboy@apache.org>

hangc0276 · 2025-08-17T21:26:35Z

When I cherry-pick this PR to branch-4.0, this project build failed with following exception:

[ERROR] Rule 0: org.apache.maven.plugins.enforcer.EnforceBytecodeVersion failed with message: 
[ERROR] Found Banned Dependency: org.apache.avro:avro-protobuf:jar:1.12.0 
[ERROR] Found Banned Dependency: org.apache.avro:avro:jar:1.12.0 
[ERROR] Use 'mvn dependency:tree' to locate the source of the banned dependencies.

Looks like related to branch-4.0 pulsar client enforced bytecode version and the maxJdkVersion is set to 8

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-enforcer-plugin</artifactId>
        <version>${maven-enforcer-plugin.version}</version>
        <executions>
          <execution>
            <id>enforce-bytecode-version</id>
            <goals>
              <goal>enforce</goal>
            </goals>
            <configuration>
              <rules>
                <enforceBytecodeVersion>
                  <maxJdkVersion>${pulsar.client.compiler.release}</maxJdkVersion>
                  <ignoredScopes>
                    <ignoreScope>test</ignoreScope>
                  </ignoredScopes>
                </enforceBytecodeVersion>
              </rules>
            </configuration>
          </execution>
        </executions>

Co-authored-by: Jiwe Guo <technoboy@apache.org>

This reverts commit 36cd2c2.

Technoboy- · 2025-08-18T03:40:50Z

When I cherry-pick this PR to branch-4.0, this project build failed with following exception:

[ERROR] Rule 0: org.apache.maven.plugins.enforcer.EnforceBytecodeVersion failed with message: 
[ERROR] Found Banned Dependency: org.apache.avro:avro-protobuf:jar:1.12.0 
[ERROR] Found Banned Dependency: org.apache.avro:avro:jar:1.12.0 
[ERROR] Use 'mvn dependency:tree' to locate the source of the banned dependencies.

Looks like related to branch-4.0 pulsar client enforced bytecode version and the maxJdkVersion is set to 8

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-enforcer-plugin</artifactId>
        <version>${maven-enforcer-plugin.version}</version>
        <executions>
          <execution>
            <id>enforce-bytecode-version</id>
            <goals>
              <goal>enforce</goal>
            </goals>
            <configuration>
              <rules>
                <enforceBytecodeVersion>
                  <maxJdkVersion>${pulsar.client.compiler.release}</maxJdkVersion>
                  <ignoredScopes>
                    <ignoreScope>test</ignoreScope>
                  </ignoredScopes>
                </enforceBytecodeVersion>
              </rules>
            </configuration>
          </execution>
        </executions>

Reverted from branch-4.0.

Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit 36cd2c2)

This reverts commit 36cd2c2. (cherry picked from commit 80e68e1)

Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit 36cd2c2)

This reverts commit 36cd2c2. (cherry picked from commit 80e68e1)

This reverts commit 36cd2c2.

Co-authored-by: Jiwe Guo <technoboy@apache.org>

Upgrade avro version to 1.12.0

1144439

hangc0276 requested review from codelipenghui, horizonzy, lhotari, poorbarcode and zymap August 9, 2025 03:15

hangc0276 self-assigned this Aug 9, 2025

github-actions bot added the doc-label-missing label Aug 9, 2025

hangc0276 added dependencies Pull requests that update a dependency file release/4.0.7 doc-not-needed Your PR changes do not impact docs ready-to-test and removed doc-label-missing labels Aug 9, 2025

github-actions bot added doc-label-missing doc-not-needed Your PR changes do not impact docs and removed doc-not-needed Your PR changes do not impact docs doc-label-missing labels Aug 9, 2025

horizonzy approved these changes Aug 9, 2025

View reviewed changes

poorbarcode approved these changes Aug 9, 2025

View reviewed changes

lhotari approved these changes Aug 9, 2025

View reviewed changes

dao-jun approved these changes Aug 11, 2025

View reviewed changes

fix

df90798

Technoboy- added this to the 4.1.0 milestone Aug 12, 2025

Technoboy- added 3 commits August 12, 2025 15:56

fix

7e7627f

fix

7c7029f

fix json converter

d3ea0ad

Technoboy- added 2 commits August 13, 2025 12:54

fix json converter

ef0a23f

fix kinesis json converter

a93db06

gaoran10 reviewed Aug 13, 2025

View reviewed changes

pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java Show resolved Hide resolved

pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java Show resolved Hide resolved

gaoran10 approved these changes Aug 13, 2025

View reviewed changes

Technoboy- merged commit baa288a into apache:master Aug 14, 2025
50 of 51 checks passed

Technoboy- added a commit to Technoboy-/pulsar that referenced this pull request Aug 14, 2025

[improve][broker] Upgrade avro version to 1.12.0 (apache#24617)

f3cdb1a

Co-authored-by: Jiwe Guo <technoboy@apache.org>

Technoboy- added the cherry-picked/branch-4.0 label Aug 14, 2025

Technoboy- added a commit that referenced this pull request Aug 18, 2025

[improve][broker] Upgrade avro version to 1.12.0 (#24617)

36cd2c2

Co-authored-by: Jiwe Guo <technoboy@apache.org>

Technoboy- added a commit that referenced this pull request Aug 18, 2025

Revert "[improve][broker] Upgrade avro version to 1.12.0 (#24617)"

80e68e1

This reverts commit 36cd2c2.

manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Aug 21, 2025

[improve][broker] Upgrade avro version to 1.12.0 (apache#24617)

76a8513

Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit 36cd2c2)

manas-ctds pushed a commit to datastax/pulsar that referenced this pull request Aug 21, 2025

Revert "[improve][broker] Upgrade avro version to 1.12.0 (apache#24617)"

6da8616

This reverts commit 36cd2c2. (cherry picked from commit 80e68e1)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Aug 26, 2025

[improve][broker] Upgrade avro version to 1.12.0 (apache#24617)

4342fb7

Co-authored-by: Jiwe Guo <technoboy@apache.org> (cherry picked from commit 36cd2c2)

srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Aug 26, 2025

Revert "[improve][broker] Upgrade avro version to 1.12.0 (apache#24617)"

b7837c4

This reverts commit 36cd2c2. (cherry picked from commit 80e68e1)

Technoboy- added a commit to Technoboy-/pulsar that referenced this pull request Sep 10, 2025

Revert "[improve][broker] Upgrade avro version to 1.12.0 (apache#24617)"

b4ec416

This reverts commit 36cd2c2.

nborisov pushed a commit to nborisov/pulsar that referenced this pull request Sep 12, 2025

Revert "[improve][broker] Upgrade avro version to 1.12.0 (apache#24617)"

9a22722

This reverts commit 36cd2c2.

KannarFr pushed a commit to CleverCloud/pulsar that referenced this pull request Sep 22, 2025

[improve][broker] Upgrade avro version to 1.12.0 (apache#24617)

55514df

Co-authored-by: Jiwe Guo <technoboy@apache.org>

walkinggo pushed a commit to walkinggo/pulsar that referenced this pull request Oct 8, 2025

[improve][broker] Upgrade avro version to 1.12.0 (apache#24617)

8aded6b

Co-authored-by: Jiwe Guo <technoboy@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[improve][broker] Upgrade avro version to 1.12.0 #24617

[improve][broker] Upgrade avro version to 1.12.0 #24617

Uh oh!

hangc0276 commented Aug 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Aug 9, 2025

Uh oh!

horizonzy left a comment

Uh oh!

lhotari commented Aug 9, 2025

AvroSchemaTest.testTimestampWithJsonDef: pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java#L515Can not set java.time.Instant field org.apache.pulsar.client.impl.schema.AvroSchemaTest$TimestampPojo.value to java.lang.Long

Uh oh!

Technoboy- commented Aug 12, 2025 •

edited

Loading

Uh oh!

coderzc commented Aug 12, 2025

Uh oh!

Technoboy- commented Aug 13, 2025

Uh oh!

codecov-commenter commented Aug 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hangc0276 commented Aug 17, 2025

Uh oh!

Technoboy- commented Aug 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

[improve][broker] Upgrade avro version to 1.12.0 #24617

[improve][broker] Upgrade avro version to 1.12.0 #24617

Uh oh!

Conversation

hangc0276 commented Aug 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

Uh oh!

github-actions bot commented Aug 9, 2025

Uh oh!

horizonzy left a comment

Choose a reason for hiding this comment

Uh oh!

lhotari commented Aug 9, 2025

AvroSchemaTest.testTimestampWithJsonDef: pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java#L515Can not set java.time.Instant field org.apache.pulsar.client.impl.schema.AvroSchemaTest$TimestampPojo.value to java.lang.Long

Uh oh!

Technoboy- commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderzc commented Aug 12, 2025

Uh oh!

Technoboy- commented Aug 13, 2025

Uh oh!

codecov-commenter commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hangc0276 commented Aug 17, 2025

Uh oh!

Technoboy- commented Aug 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

hangc0276 commented Aug 9, 2025 •

edited

Loading

Technoboy- commented Aug 12, 2025 •

edited

Loading

codecov-commenter commented Aug 13, 2025 •

edited

Loading