In what version(s) of Spring for Apache Kafka are you seeing this issue?
spring-kafka 4.0.5
Describe the bug
Value serializer mappings are not honored (have a non-deterministic behavior) when multiple classloaders are involved.
Consider a property-based (i.e. via spring.kafka.* properties) kafka template initialization. Property-based seems to be important here because beans are initialized in a lazy manner.
Mappings (i.e. spring.kafka.producer.properties.spring.json.type.mapping entries) are created and cached on template initialization. Also, mappings are stored/indexed based on the class bytes (not the class string).
So, if the template happens to be initialized under classloader A, then a mapping is created for the class in question with the class bytes bound to classloader A. If at a later time the the template is accessed using a message class that happens to be bound to another classloader, looking up the mapping fails and the resulting message is not the expected one.
One case of different classloaders is when using spring dev tools (good thing it's local only...). In the sample project below, the force-init in a different classloader happens when a kafka write is triggered inside a reactive chain which in turn involves a custom-built mongo client. Using the default mongo client as built by spring, resolves the issue. Using custom kafka beans (i.e. not based on properties but programmatically configuring them) also resolves the issue.
The problem here is the non-deterministic behavior. Because kafka template is initialized lazily, whoever initializes first, also "owns" the class mappings.
Not necessarily expecting a fix but i'd like to hear your thoughts ☔
On a sidenote, this might look like a weird edge case but it has caused real pain in a real project in real life 🚑
To Reproduce
- use properties to create kafka configuration
- configure a mapping for produced messages, e.g.
NotificationEvent:com.example.demo.NotificationEvent
- force the initialization of kafka beans under a different classloader
Expected behavior
- messages should have a header of:
[__TypeId__ :: NotificationEvent]
Actual behavior
- messages have the (fallback) FQN header:
[__TypeId__ :: com.example.demo.NotificationEvent]
Sample
https://github.com/diamondT/demo-kafka-mappings
./mvnw clean spring-boot:run
In what version(s) of Spring for Apache Kafka are you seeing this issue?
spring-kafka 4.0.5
Describe the bug
Value serializer mappings are not honored (have a non-deterministic behavior) when multiple classloaders are involved.
Consider a property-based (i.e. via
spring.kafka.*properties) kafka template initialization. Property-based seems to be important here because beans are initialized in a lazy manner.Mappings (i.e.
spring.kafka.producer.properties.spring.json.type.mappingentries) are created and cached on template initialization. Also, mappings are stored/indexed based on the class bytes (not the class string).So, if the template happens to be initialized under classloader A, then a mapping is created for the class in question with the class bytes bound to classloader A. If at a later time the the template is accessed using a message class that happens to be bound to another classloader, looking up the mapping fails and the resulting message is not the expected one.
One case of different classloaders is when using spring dev tools (good thing it's local only...). In the sample project below, the force-init in a different classloader happens when a kafka write is triggered inside a reactive chain which in turn involves a custom-built mongo client. Using the default mongo client as built by spring, resolves the issue. Using custom kafka beans (i.e. not based on properties but programmatically configuring them) also resolves the issue.
The problem here is the non-deterministic behavior. Because kafka template is initialized lazily, whoever initializes first, also "owns" the class mappings.
Not necessarily expecting a fix but i'd like to hear your thoughts ☔
On a sidenote, this might look like a weird edge case but it has caused real pain in a real project in real life 🚑
To Reproduce
NotificationEvent:com.example.demo.NotificationEventExpected behavior
[__TypeId__ :: NotificationEvent]Actual behavior
[__TypeId__ :: com.example.demo.NotificationEvent]Sample
https://github.com/diamondT/demo-kafka-mappings
./mvnw clean spring-boot:run