KAFKA-13418: Support key updates with TLS 1.3#11966
Conversation
|
I intend to cherry-pick this change to the 3.2, 3.1 and 3.0 branches. cc @cadonna |
|
|
||
| private String blockingRequest(String node, String s) throws IOException { | ||
| selector.send(createSend(node, s)); | ||
| selector.poll(1000L); |
There was a problem hiding this comment.
Removed since it's redundant.
| } | ||
| } | ||
|
|
||
| public SecurityProtocol securityProtocol() { |
| } | ||
| selector.send(createSend(node, node + "-" + 0)); | ||
| selector.poll(0L); | ||
| server.renegotiate(); |
There was a problem hiding this comment.
Won't this test fail with Java 8 where we use TLSv1.2 as default?
There was a problem hiding this comment.
Good point. I'll verify that it fails (Jenkins should show it) and update the test to be a TLS 1.3 test that only runs if Java is 11 or newer.
There was a problem hiding this comment.
We did indeed see the Java 8 failure before my last change:
Build / JDK 8 and Scala 2.12 / org.apache.kafka.common.network.DefaultTlsSelectorTest.testKeyUpdate()
Build / JDK 8 and Scala 2.12 / org.apache.kafka.common.network.DefaultTlsSelectorTest.testKeyUpdate()
Build / ARM / org.apache.kafka.common.network.DefaultTlsSelectorTest.testKeyUpdate()
Build / ARM / org.apache.kafka.common.network.DefaultTlsSelectorTest.testKeyUpdate()
…th Java 11 or newer
rajinisivaram
left a comment
There was a problem hiding this comment.
@ijuma Thanks for the PR, LGTM
|
Various builds passed, there was one flake:
|
Key updates with TLS 1.3 trigger code paths similar to renegotiation with TLS 1.2. Update the read/write paths not to throw an exception in this case (kept the exception in the `handshake` method). With the default configuration, key updates happen after 2^37 bytes are encrypted. There is a security property to adjust this configuration, but the change has to be done before it is used for the first time and it cannot be changed after that. As such, it is best done via a system test (filed KAFKA-13779). To validate the change, I wrote a unit test that forces key updates and manually ran a producer workload that produced more than 2^37 bytes. Both cases failed without these changes and pass with them. Note that Shylaja Kokoori attached a patch with the SslTransportLayer fix and hence included them as a co-author of this change. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com> Co-authored-by: Shylaja Kokoori
Key updates with TLS 1.3 trigger code paths similar to renegotiation with TLS 1.2. Update the read/write paths not to throw an exception in this case (kept the exception in the `handshake` method). With the default configuration, key updates happen after 2^37 bytes are encrypted. There is a security property to adjust this configuration, but the change has to be done before it is used for the first time and it cannot be changed after that. As such, it is best done via a system test (filed KAFKA-13779). To validate the change, I wrote a unit test that forces key updates and manually ran a producer workload that produced more than 2^37 bytes. Both cases failed without these changes and pass with them. Note that Shylaja Kokoori attached a patch with the SslTransportLayer fix and hence included them as a co-author of this change. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com> Co-authored-by: Shylaja Kokoori
Key updates with TLS 1.3 trigger code paths similar to renegotiation with TLS 1.2. Update the read/write paths not to throw an exception in this case (kept the exception in the `handshake` method). With the default configuration, key updates happen after 2^37 bytes are encrypted. There is a security property to adjust this configuration, but the change has to be done before it is used for the first time and it cannot be changed after that. As such, it is best done via a system test (filed KAFKA-13779). To validate the change, I wrote a unit test that forces key updates and manually ran a producer workload that produced more than 2^37 bytes. Both cases failed without these changes and pass with them. Note that Shylaja Kokoori attached a patch with the SslTransportLayer fix and hence included them as a co-author of this change. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com> Co-authored-by: Shylaja Kokoori
@tombentley FYI |
Key updates with TLS 1.3 trigger code paths similar to renegotiation with TLS 1.2. Update the read/write paths not to throw an exception in this case (kept the exception in the `handshake` method). With the default configuration, key updates happen after 2^37 bytes are encrypted. There is a security property to adjust this configuration, but the change has to be done before it is used for the first time and it cannot be changed after that. As such, it is best done via a system test (filed KAFKA-13779). To validate the change, I wrote a unit test that forces key updates and manually ran a producer workload that produced more than 2^37 bytes. Both cases failed without these changes and pass with them. Note that Shylaja Kokoori attached a patch with the SslTransportLayer fix and hence included them as a co-author of this change. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com> Co-authored-by: Shylaja Kokoori
…cs-12-may-2022 * apache-kafka/3.0: (14 commits) fix: make sliding window works without grace period (#kafka-13739) (apache#11980) KAFKA-13794: Follow up to fix producer batch comparator (apache#12006) KAFKA-13794; Fix comparator of `inflightBatchesBySequence` in `TransactionManager` (apache#11991) KAFKA-13748: Do not include file stream connectors in Connect's CLASSPATH and plugin.path by default (apache#11908) KAFKA-13418: Support key updates with TLS 1.3 (apache#11966) KAFKA-13770: Restore compatibility with KafkaBasedLog using older Kafka brokers (apache#11946) KAFKA-13761: KafkaLog4jAppender deadlocks when idempotence is enabled (apache#11939) KAFKA-13759: Disable idempotence by default in producers instantiated by Connect (apache#11933) MINOR: Fix `ConsumerConfig.ISOLATION_LEVEL_DOC` (apache#11915) KAFKA-13750; Client Compatability KafkaTest uses invalid idempotency configs (apache#11909) ...
…e#11966) LI_DESCRIPTION=This may improve p99 inter-broker latency under TLS1.3 -- Original message -- Key updates with TLS 1.3 trigger code paths similar to renegotiation with TLS 1.2. Update the read/write paths not to throw an exception in this case (kept the exception in the `handshake` method). With the default configuration, key updates happen after 2^37 bytes are encrypted. There is a security property to adjust this configuration, but the change has to be done before it is used for the first time and it cannot be changed after that. As such, it is best done via a system test (filed KAFKA-13779). To validate the change, I wrote a unit test that forces key updates and manually ran a producer workload that produced more than 2^37 bytes. Both cases failed without these changes and pass with them. Note that Shylaja Kokoori attached a patch with the SslTransportLayer fix and hence included them as a co-author of this change. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com> Co-authored-by: Shylaja Kokoori
…e#11966) LI_DESCRIPTION=This may improve p99 inter-broker latency under TLS1.3 -- Original message -- Key updates with TLS 1.3 trigger code paths similar to renegotiation with TLS 1.2. Update the read/write paths not to throw an exception in this case (kept the exception in the `handshake` method). With the default configuration, key updates happen after 2^37 bytes are encrypted. There is a security property to adjust this configuration, but the change has to be done before it is used for the first time and it cannot be changed after that. As such, it is best done via a system test (filed KAFKA-13779). To validate the change, I wrote a unit test that forces key updates and manually ran a producer workload that produced more than 2^37 bytes. Both cases failed without these changes and pass with them. Note that Shylaja Kokoori attached a patch with the SslTransportLayer fix and hence included them as a co-author of this change. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com> Co-authored-by: Shylaja Kokoori
…e#11966) (#485) LI_DESCRIPTION=This may improve p99 inter-broker latency under TLS1.3 -- Original message -- Key updates with TLS 1.3 trigger code paths similar to renegotiation with TLS 1.2. Update the read/write paths not to throw an exception in this case (kept the exception in the `handshake` method). With the default configuration, key updates happen after 2^37 bytes are encrypted. There is a security property to adjust this configuration, but the change has to be done before it is used for the first time and it cannot be changed after that. As such, it is best done via a system test (filed KAFKA-13779). To validate the change, I wrote a unit test that forces key updates and manually ran a producer workload that produced more than 2^37 bytes. Both cases failed without these changes and pass with them. Note that Shylaja Kokoori attached a patch with the SslTransportLayer fix and hence included them as a co-author of this change. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com> Co-authored-by: Shylaja Kokoori Co-authored-by: Ismael Juma <ismael@juma.me.uk>
Key updates with TLS 1.3 trigger code paths similar to renegotiation with TLS 1.2.
Update the read/write paths not to throw an exception in this case (kept the exception
in the
handshakemethod).With the default configuration, key updates happen after 2^37 bytes are encrypted.
There is a security property to adjust this configuration, but the change has to be
done before it is used for the first time and it cannot be changed after that. As such,
it is best done via a system test (filed KAFKA-13779).
To validate the change, I wrote a unit test that forces key updates and manually ran
a producer workload that produced more than 2^37 bytes. Both cases failed without
these changes and pass with them.
Note that Shylaja Kokoori attached a patch with the SslTransportLayer fix and hence
included them as a co-author of this change.
Co-authored-by: Shylaja Kokoori
Committer Checklist (excluded from commit message)