Non-conformance Bug affecting usability of the library

OS
Linux, Ubuntu 20
develop branch, commit c20c632
Affects both clients and servers

Non-conformance

According to the RFC, decrypt_error alerts should be generated when:

A handshake cryptographic operation failed, including being unable
to correctly verify a signature or validate a Finished message.

None of these happen in the below capture. Therein, the client receives an unencrypted HelloVerifyRequest with epoch 1 is received. The message is unencrypted. The fact that it has epoch 1 should mean that it should not be processed/should instead be discarded/ rejected.

The issue also affects servers. In the capture below, an ClientKeyExchange with epoch 1 elicits a decrypt_error response from the server for no good reason.

Abruptly terminating the handshake on receiving a record with an epoch for which a session has not been established has a more severe implication. In case Finished (epoch 1) is received before ChangeCipherSpec (epoch 0) or ClientKeyExchange (epoch 0), the TinyDTLS client/server will abruptly terminate the handshake. This can happen in real world due to reordering. A capture of this is shown below exposing the problem on clients. Notice that the ChangeCipherSpec of the server is sent before Finished, to mimic this scenario. Finished appears as an "Encrypted Handshake Message" in the capture, as it cannot be decrypted by Wireshark.

Steps to Reproduce
I attached files necessary for reproduction using DTLS-Fuzzer, a Java-based tool for testing DTLS libraries. Also included in the archive the captures of the interaction show above. DTLS-Fuzzer requires the JDK for Java 8. On Ubuntu, this can be installed by running:
sudo apt-get install openjdk-8-jdk

Unpack the archive, cd to resulting folder and run bash reproduce.sh test_sequence_decrypt or bash reproduce.sh test_sequence_reorder, while running an instance of Wireshark on the side. The reproduction script will:

• setup eclipse TinyDTLS and DTLS-Fuzzer
• launch the TinyDTLS client using the 'tests/dtls-client utility;
• launch DTLS-Fuzzer to execute the input sequences 'test_sequence_decrypt' or 'test_sequence_reorder' which expose the two problems.

I also added the test sequence 'test_sequence_handshake'. This is similar to the reordering sequence, but without the reordering. By executing this sequence via bash reproduce.sh test_sequence_handshake, you should complete the handshake with the client.

Note that the problems affect both clients and servers. On request, I can provide reproduction scripts also for servers. I am guessing fixing the problem for clients will also do it for servers.

Thanks!

reproduction.tar.gz