Conversation
Add kernel TLS (kTLS) offload support as a PoC, togglable via DOTNET_SYSTEM_NET_SECURITY_KTLS=1 environment variable. When SslStream wraps a NetworkStream on Linux, this enables OpenSSL's kTLS integration by using socket BIOs (SSL_set_fd) instead of memory BIOs. After the TLS handshake, the kernel handles encryption/decryption of application data, potentially with hardware offload. Changes: - Native PAL: Add SSL_set_fd, kTLS query, and blocking SSL I/O functions with internal poll loop for non-blocking socket support - OpenSSL shim: Add SSL_set_fd, SSL_get_fd, SSL_get_wbio, SSL_get_rbio as lightup functions - Managed interop: P/Invoke declarations, SafeSslHandle.CreateForKtls - Interop.OpenSsl: AllocateSslHandleForKtls, DoSslHandshakeKtls, KtlsRead, KtlsWrite - SslStream.IO: Detection of NetworkStream + env var, kTLS handshake path, kTLS read/write paths using Task.Run with blocking native calls Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace blocking Task.Run + poll() approach with proper async I/O: - Handshake: non-blocking SSL_do_handshake loop with zero-byte reads on InnerStream (via TIOAdapter) for WANT_READ - Read: non-blocking SSL_read with zero-byte reads for WANT_READ - Write: non-blocking SSL_write with Task.Yield() for WANT_WRITE This avoids blocking threadpool threads and integrates with .NET's epoll-based async socket infrastructure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With socket BIOs on non-blocking sockets, OpenSSL may return SSL_ERROR_SYSCALL with errno=EAGAIN instead of SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE. This happens because ERR_clear_error() is called before SSL operations, leaving the error queue empty, so SSL_get_error() falls through to SSL_ERROR_SYSCALL. Handle this in the managed kTLS code via IsKtlsWantRead() helper that checks Marshal.GetLastPInvokeError() for EAGAIN, avoiding changes to the shared native PAL functions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Debug Console.WriteLine statements between SSL_read/SSL_write P/Invoke calls and the subsequent errno check were corrupting the saved errno. Console.WriteLine internally calls Interop.Sys.Write (SetLastError=true), which overwrites Marshal.GetLastPInvokeError() before IsKtlsWantRead could read it. This caused genuine EOF (ret=0 + SSL_ERROR_SYSCALL + errno=0) to be misinterpreted as EAGAIN/WANT_READ, leading to an infinite wait on a zero-byte read that would never complete. Fix: - Remove all debug Console.WriteLine statements from kTLS paths - Capture errno immediately after SSL P/Invoke calls via out parameters in TryKtlsRead/TryKtlsWrite, before any other P/Invoke can run - Pass captured errno to IsKtlsWantRead instead of reading it later - Handle ret=0 + SSL_ERROR_SYSCALL + errno=SUCCESS as EOF explicitly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove unused blocking wrappers (DoSslHandshakeKtls, KtlsRead, KtlsWrite) from Interop.OpenSsl.cs and their P/Invoke declarations (SslDoHandshakeBlocking, SslReadBlocking, SslWriteBlocking) from Interop.Ssl.cs. These were from the initial blocking I/O approach and are no longer called since switching to the async pattern. Also remove: empty static constructor, unused 'ns' local variable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three performance improvements for the kTLS PoC: 1. Fix busy-spin on WANT_READ: Zero-byte reads on Socket complete immediately (SocketAsyncContext fast-paths them without a syscall). Replace with 1-byte MSG_PEEK recv which properly waits via epoll for data availability without consuming it from the kernel buffer. Applies to both handshake and read paths. 2. Fix busy-spin on WANT_WRITE: Replace Task.Yield() (immediate reschedule = CPU burn) with Task.Delay(1) to give the socket send buffer time to drain. WANT_WRITE is rare in practice. 3. Enable SSL_CTX caching and TLS session resume: The kTLS path now reuses cached SSL_CTX handles and sets TLS sessions for repeat connections, matching the behavior of the normal SslStream path. This avoids full TLS handshakes on subsequent connections to the same host. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The two functions were nearly identical — the only difference was SafeSslHandle.Create vs CreateForKtls (memory BIOs vs socket BIO). Add an optional socketFd parameter (default -1) to AllocateSslHandle and branch at the Create call. This eliminates ~120 lines of duplicated SSL configuration code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With blocking sockets (the default for new Socket instances), SSL_do_handshake blocks the thread pool thread during recv/send calls. In loopback test scenarios where both client and server handshakes run as tasks on the thread pool, this causes thread pool starvation and hangs. Set socket.Blocking = false before the handshake. This is necessary because SSL_do_handshake bypasses SocketAsyncContext (calling recv/send directly on the fd), so we can't rely on SocketAsyncContext's lazy non-blocking initialization. With non-blocking sockets, the handshake loop properly returns WANT_READ/WANT_WRITE and uses peek-based async readiness waiting. All 4957 tests pass with DOTNET_SYSTEM_NET_SECURITY_KTLS=1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Only set socket.Blocking = false when using the async handshake path (AsyncReadWriteAdapter). For the sync path, blocking is expected by the caller. The JIT constant-folds the typeof comparison. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two improvements to the kTLS read path based on benchmark trace analysis: 1. Buffer filling: After a successful SSL_read, continue reading to fill the caller's buffer instead of returning immediately. This avoids extra peek+SSL_read syscall round trips when multiple TLS records are already buffered in the kernel. For a 64KB response (~4 TLS records), this reduces from 8 syscalls to ~6 and eliminates 3 async iterations. 2. Connection close handling: Catch SocketException from the MSG_PEEK recv used for readiness notification. When the peer closes a kTLS connection, recv() may return ECONNRESET instead of clean EOF. The exception is caught and the loop continues to SSL_read which determines the actual TLS-level status (SSL_ERROR_ZERO_RETURN for clean closure, or a real error). This eliminates ~276K exceptions/15s seen in h11-get-close benchmarks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove PollForSsl, SslDoHandshakeBlocking, SslReadBlocking, and SslWriteBlocking from pal_ssl.c. These were part of the initial kTLS implementation but are no longer used since async I/O was implemented in managed code using MSG_PEEK readiness notification. Also remove the unused errno.h and poll.h includes, and fix entrypoints.c ordering to keep kTLS-related entries sorted alphabetically. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ring - When kTLS TX is active, bypass SSL_write and use Socket.SendAsync directly — the kernel encrypts transparently on send() - Add 32KB read-ahead buffer to amortize SSL_read (recvmsg) calls, reducing epoll_wait syscalls per request - Track kTLS TX state via _ktlsTx field set after handshake - Remove unused _ktlsRx field (socket-direct reads don't work due to NewSessionTicket records requiring OpenSSL's record layer) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the custom _ktlsReadBuffer with the existing _buffer (SslBuffer) infrastructure. SSL_read returns already-decrypted plaintext, so we Commit + OnDecrypted(0, ret, ret) to treat it as decrypted data in _buffer, reusing the same CopyDecryptedData consume path. When the caller's buffer is >= ReadBufferSize, read directly into it to avoid a copy (zero-copy fast path for large reads). This eliminates 4 fields (_ktlsReadBuffer, offset, count, buffer size constant) and significantly improves large-response scenarios: - H1.1 64KB: -9.2% -> +1.6% (neutral) - H2 64KB: -34.2% -> -26.0% Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The direct socket read bypass for kTLS RX doesn't work with TLS 1.3 because OpenSSL must process post-handshake messages (NewSessionTicket) that arrive as non-application-data records on the socket. Bypassing SSL_read causes these records to be mixed into the data stream. Instead, keep SSL_read for all kTLS paths (TX and RX). The kernel decryption benefit of kTLS RX still applies transparently when OpenSSL calls recv/recvmsg on the kTLS socket. Added a buffer-fill loop after successful SSL_read: since each SSL_read returns at most one TLS record (~16KB), we immediately retry to fill more of the read-ahead buffer or user buffer without re-waiting. This amortizes the MSG_PEEK wait cost for large responses. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones |
|
copilot-generated report: Kernel TLS (kTLS) Offloading — Performance ReportExecutive SummaryThis report presents the performance characteristics of kernel TLS (kTLS) offloading Three modes are compared:
Key findings:
Test Configuration
Machines
Each profile uses separate client and server machines on a dedicated network. Throughput — perf-lin (12 cores)
Throughput — citrine-lin (28 cores)
Throughput — gold-lin (56 cores)
AnalysisWhere kTLS helps: H1.1 small-payload keep-alivekTLS shows a consistent +6–8% throughput improvement for H1.1 GET keep-alive with DirectOpenSSL (socket BIO without kTLS) is 3–5% slower than Standard, which Where kTLS regresses: H2 GET 64KB (−32% to −51%)HTTP/2 with large response bodies has a fundamental regression caused by a Linux kernel A 64KB HTTP/2 response is split across ~4 TLS records (each up to 16KB). Without kTLS, This is compounded by HTTP/2's multiplexing: flow control windows and stream processing Where kTLS regresses: H1.1 GET 64KB on high-core machines (−12%)On citrine-lin (28 cores), the client CPU reaches 97% with kTLS (vs 78% Standard), On perf-lin (12 cores), this scenario shows no regression because the CPU is not Where kTLS regresses: connection-close (−10% to −24%)Each kTLS connection requires additional POST scenarios: at parityH1.1 POST 64KB and H2 POST 64KB show no meaningful difference (within ±2%) across Gold-lin (56 cores) — NotesThe gold-lin machine (aspnet-gold-lin) runs benchmark jobs inside Docker containers. The gold-lin machine has a Mellanox ConnectX-6 Dx NIC ( Known Limitations and Future Optimizations
|
There was a problem hiding this comment.
Pull request overview
This PR updates the Linux/OpenSSL kTLS proof-of-concept in SslStream to avoid bypassing SSL_read on RX (to correctly handle TLS 1.3 post-handshake records like NewSessionTicket), while still attempting to benefit from kernel kTLS offload when OpenSSL reads from a kTLS-enabled socket.
Changes:
- Add new native OpenSSL interop exports for socket-BIO + kTLS enablement/status checks.
- Add a Linux/OpenSSL kTLS handshake/read/write path in
SslStream(env-var gated), including a post-SSL_readloop to fill buffers without re-waiting. - Add managed P/Invokes and SafeSslHandle support for creating an OpenSSL
SSL*bound directly to a socket fd.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/native/libs/System.Security.Cryptography.Native/pal_ssl.h | Declares new native exports for setting fd, enabling kTLS, and querying kTLS send/recv. |
| src/native/libs/System.Security.Cryptography.Native/pal_ssl.c | Implements the new native exports using OpenSSL APIs. |
| src/native/libs/System.Security.Cryptography.Native/opensslshim.h | Adds lightup entries for additional SSL symbols needed by the new exports. |
| src/native/libs/System.Security.Cryptography.Native/entrypoints.c | Registers the new exports for managed interop. |
| src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Ssl.cs | Adds P/Invokes and SafeSslHandle support for kTLS socket-BIO creation. |
| src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.OpenSsl.cs | Extends SSL handle allocation for optional socket-fd based creation; exposes GetSslError internally. |
| src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs | Adds the Linux/OpenSSL kTLS handshake and I/O paths, including buffer-fill logic on reads. |
| src/libraries/System.Net.Security/src/System.Net.Security.csproj | Makes System.Console a non-conditional dependency (currently used for debug logging). |
Comments suppressed due to low confidence (3)
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:1279
- Using Task.Delay(1) as a stand-in for socket writability can turn sustained backpressure into a tight wake/sleep loop (high CPU, poor scalability). Instead of polling delays, consider using a proper readiness mechanism (e.g., an async wait on writability if available, or a backoff strategy / Socket.Poll-based wait on a dedicated thread) so this doesn’t busy-wait under load.
if (error == Interop.Ssl.SslErrorCode.SSL_ERROR_WANT_WRITE ||
IsKtlsWantRead(error, errno)) // SYSCALL+EAGAIN can also mean socket buffer full
{
// Wait briefly for socket buffer to drain. Socket doesn't expose
// an async writability wait, but WANT_WRITE is rare and short-lived.
await Task.Delay(1, cancellationToken).ConfigureAwait(false);
continue;
src/native/libs/System.Security.Cryptography.Native/pal_ssl.c:1322
- CryptoNative_SslSetFd doesn’t validate inputs or follow the existing pattern of asserting non-null parameters used by neighboring exports (e.g., CryptoNative_SslStapleOcsp asserts ssl != NULL). Please add consistent asserts (at least
assert(ssl != NULL)) for easier debugging and to match established conventions in this file.
int32_t CryptoNative_SslSetFd(SSL* ssl, int32_t fd)
{
ERR_clear_error();
#ifdef FEATURE_DISTRO_AGNOSTIC_SSL
if (SSL_set_fd_ptr == NULL)
{
return 0;
}
#endif
return SSL_set_fd(ssl, fd);
}
src/libraries/System.Net.Security/src/System.Net.Security.csproj:472
- System.Net.Security.csproj now references System.Console unconditionally. This increases the baseline dependency surface for the library and appears to be solely to support the new Console.WriteLine debug logging. Please remove the Console.WriteLine usage and revert this reference to its prior conditional form (or otherwise avoid taking an unconditional System.Console dependency).
<ProjectReference Include="$(LibrariesProjectRoot)System.Collections\src\System.Collections.csproj" />
<ProjectReference Include="$(LibrariesProjectRoot)System.Collections.Concurrent\src\System.Collections.Concurrent.csproj" />
<ProjectReference Include="$(LibrariesProjectRoot)System.Collections.NonGeneric\src\System.Collections.NonGeneric.csproj" />
<ProjectReference Include="$(LibrariesProjectRoot)System.Console\src\System.Console.csproj" />
| while (true) | ||
| { | ||
| Memory<byte> readTarget = useReadAhead ? _buffer.AvailableMemory : buffer; | ||
| int ret = TryKtlsRead(sslHandle, readTarget, out Interop.Ssl.SslErrorCode error, out Interop.Error errno); | ||
|
|
There was a problem hiding this comment.
The kTLS read path calls SSL_read without taking the existing _handshakeLock used by EncryptData/DecryptData to serialize OpenSSL operations. This can allow concurrent SSL_read/SSL_write on the same SSL* from different threads, which isn’t consistent with the rest of SslStream’s synchronization model and can lead to racey failures/corruption. Please wrap SSL_read/SSL_write (and any related OpenSSL state access) in the same lock used elsewhere, or otherwise ensure thread-safety is preserved.
| while (totalWritten < buffer.Length) | ||
| { | ||
| ReadOnlyMemory<byte> remaining = buffer.Slice(totalWritten); | ||
| int ret = TryKtlsWrite(sslHandle, remaining, out Interop.Ssl.SslErrorCode error, out Interop.Error errno); |
There was a problem hiding this comment.
The kTLS write path calls SSL_write in a loop without taking the existing _handshakeLock used for OpenSSL operations elsewhere. This can run concurrently with SSL_read on another thread and violates the established synchronization pattern in SslStream. Please add appropriate locking or another mechanism to ensure the OpenSSL SSL* is not accessed concurrently.
| int ret = TryKtlsWrite(sslHandle, remaining, out Interop.Ssl.SslErrorCode error, out Interop.Error errno); | |
| Interop.Ssl.SslErrorCode error; | |
| Interop.Error errno; | |
| int ret; | |
| lock (_handshakeLock) | |
| { | |
| ret = TryKtlsWrite(sslHandle, remaining, out error, out errno); | |
| } |
| int32_t CryptoNative_SslGetKtlsSend(SSL* ssl) | ||
| { | ||
| #ifndef OPENSSL_NO_KTLS | ||
| return BIO_get_ktls_send(SSL_get_wbio(ssl)); | ||
| #else | ||
| (void)ssl; | ||
| return 0; | ||
| #endif | ||
| } | ||
|
|
||
| int32_t CryptoNative_SslGetKtlsRecv(SSL* ssl) | ||
| { | ||
| #ifndef OPENSSL_NO_KTLS | ||
| return BIO_get_ktls_recv(SSL_get_rbio(ssl)); |
There was a problem hiding this comment.
These kTLS helpers call BIO_get_ktls_send/recv and SSL_get_wbio/rbio without any distro-agnostic/lightup safety. In FEATURE_DISTRO_AGNOSTIC_SSL scenarios the runtime may load an OpenSSL lib that lacks these symbols (or where SSL_get_wbio/rbio weren’t found), which can lead to missing-symbol load failures or null-function-pointer calls. Please add proper LIGHTUP_FUNCTION entries for BIO_get_ktls_send/recv (and null checks for SSL_get_wbio_ptr/SSL_get_rbio_ptr and the BIO_get_ktls_* ptrs), returning 0 when unavailable.
src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs
Show resolved
Hide resolved
| { | ||
| _ktlsTx = Interop.Ssl.SslGetKtlsSend(sslHandle) != 0; | ||
| _ktlsRx = Interop.Ssl.SslGetKtlsRecv(sslHandle) != 0; | ||
| System.Console.WriteLine($"kTLS handshake completed: kTLS recv: {(_ktlsRx ? 1 : 0)}, kTLS send: {(_ktlsTx ? 1 : 0)}"); |
There was a problem hiding this comment.
This handshake path logs kTLS status with System.Console.WriteLine. Console output from a networking library is not appropriate and can’t be controlled by consumers. Please remove this and use existing tracing/telemetry (e.g., NetEventSource/Activity/Telemetry) if you need observability.
| System.Console.WriteLine($"kTLS handshake completed: kTLS recv: {(_ktlsRx ? 1 : 0)}, kTLS send: {(_ktlsTx ? 1 : 0)}"); | |
| if (NetEventSource.Log.IsEnabled()) | |
| { | |
| NetEventSource.Info(this, $"kTLS handshake completed: kTLS recv: {(_ktlsRx ? 1 : 0)}, kTLS send: {(_ktlsTx ? 1 : 0)}"); | |
| } |
kTLS: use Socket.ReceiveAsync for kTLS RX instead of recvmsg Replace the custom recvmsg+cmsg P/Invoke with plain Socket.ReceiveAsync for the kTLS RX read path. On a kTLS RX socket, recv() transparently returns decrypted application data from the kernel. When a non-application TLS record (NewSessionTicket, KeyUpdate) is at the queue head, recv() returns EIO — we catch this and use SSL_read to consume the control record, then retry. The fill loop uses synchronous socket.Receive() on the non-blocking socket to gather additional TLS records without waiting, breaking on WouldBlock. This removes the need for the native CryptoNative_KtlsRecvMsg function and all associated recvmsg/cmsg infrastructure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> kTLS: replace Task.Delay(1) with Socket.Poll for WANT_WRITE Use Socket.Poll(SelectMode.SelectWrite) instead of Task.Delay(1) to wait for socket writability when SSL_do_handshake or SSL_write returns SSL_ERROR_WANT_WRITE. Poll uses the poll() syscall which returns as soon as the socket is writable, avoiding the minimum 1ms scheduling delay of Task.Delay. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> kTLS: add Debug.Fail for unexpected WANT_WRITE paths WANT_WRITE was never observed across 400K+ requests on both local and ASP.NET perf lab benchmarks (loopback and real network). Add Debug.Fail to both handshake and write WANT_WRITE handlers so we notice immediately if this assumption is ever violated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> kTLS: remove fill loop from KtlsSocketReadAsync The fill loop added a synchronous recv() call after each async ReceiveAsync. For small responses this always returned EAGAIN, causing a SocketException throw/catch per request (visible as 0.13% EH.DispatchEx in CPU profiles). Unlike SSL_read which returns one TLS record per call (due to recvmsg with cmsg), plain recv() on a kTLS socket already returns all available decrypted data in a single call. The fill loop provided no benefit while doubling the recv syscall count. CPU profile improvement (H1.1 keepalive 0B): - recv exclusive: 15.97% → 1.09% - Active CPU: 18.46% → 2.43% (vs 6.30% for non-kTLS baseline) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> kTLS: use non-throwing SAEA-based ReceiveAsync for kTLS RX reads Replace try/catch SocketException pattern with a cached KtlsReceiveEventArgs (SAEA + IValueTaskSource<int>) that returns -1 on error instead of throwing. This eliminates the exception throw/catch overhead on every NewSessionTicket/KeyUpdate record (typically 1-2 per TLS 1.3 connection). The KtlsReceiveEventArgs instance is lazily allocated, cached on SslStream._ktlsRecvArgs, and disposed in CloseInternal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> kTLS: add DirectOpenSSL experiment mode Add DOTNET_SYSTEM_NET_SECURITY_DIRECT_OPENSSL=1 mode that uses SSL_set_fd (direct socket BIO) for OpenSSL I/O without enabling kTLS offload. This allows measuring the overhead of SslStream's memory-BIO abstraction layer vs direct OpenSSL socket I/O. In DirectOpenSSL mode: - Handshake: same as kTLS (SSL_do_handshake on socket fd) - Reads: SSL_read with MSG_PEEK readability wait (no kTLS RX) - Writes: SSL_write directly on socket fd (no kTLS TX) - No kernel crypto offload, no Socket.SendAsync/ReceiveAsync bypass Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> kTLS: replace custom SAEA with Socket.ReceiveAsync + try/catch Remove KtlsReceiveEventArgs (custom SAEA implementing IValueTaskSource<int>) and use Socket.ReceiveAsync(Memory<byte>) with a filtered catch for the rare EIO exception from kTLS non-application TLS records (NewSessionTicket). The custom SAEA caused thread pool lock contention (+0.44pp CPU at 100 connections) due to the public SocketAsyncEventArgs completion path using different scheduling than the internal AwaitableSocketAsyncEventArgs. The catch is filtered to SocketError.SocketError (unmapped errno, i.e. EIO) so genuine socket errors like ConnectionReset propagate normally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
For kTLS and DirectOpenSSL modes, SSL_shutdown attempted I/O directly on the non-blocking socket via socket BIO, causing EAGAIN to surface as SSL_ERROR_SYSCALL and be treated as a fatal error. Fix: keep quiet shutdown enabled for socket BIO handles so SSL_shutdown sets internal flags without attempting I/O. Skip GenerateToken in CreateShutdownToken since there is no output memory BIO to read from. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributes to #66224.
This experiment special-cases
SslStreamwhen used withNetworkStreamto make OpenSSL talk directly do the underlying socket. This allows us to enable kTLS on supported platforms which will offload encryption/decryption of outgoing/incoming data to the in-kernel TLS implementation, and, if supported by NIC, even to specialized networking hardware.The functionality in the PoC is gated by
DOTNET_SYSTEM_NET_SECURITY_KTLS=1