Verification of security properties of the TLS protocol version 1.3

Abstract

This paper presents the experience of verifying server implementations of the TLS protocol version 1.3. TLS is one of the most popular cryptographic protocols designed to create secure data transmission channels-one of the main mechanisms for protecting information in modern networks. The protocol provides the necessary functionality for its tasks: confidentiality of transmitted data, data integrity, and authentication of the parties. In the new version 1.3 of the TLS architecture was significantly redesigned, eliminating a number of shortcomings of previous versions that were identified both during the development of implementations and during their operation. We used a new test suite for verifying implementations of the TLS 1.3 for compliance with Internet specifications, developed on the basis of the RFC8446 specification, using UniTESK technology and mutation testing methods. To test implementations for compliance with formal specifications, UniTESK technology is used, which provides testing automation tools based on the use of finite state machines. The states of the system under test define the states of the state machine, and the test effects are the transitions of this machine. When a transition is made, the specified effect is passed to the implementation under test, after which the implementation's reactions are registered and a verdict is automatically given on whether the observed behavior matches the specification. Mutation methods are used to detect nonstandard behavior of the system under test (termination due to a fatal error, "suspension", memory access errors) by transmitting incorrect data, such situations often remain outside the requirements of the specifications. Any changes are made to messages generated based on the developed protocol model. The protocol model allows one to make changes to the data flow at any stage of network exchange, which allows the test scenario to pass through all significant protocol states and in each such state to test the implementation in accordance with the specified program. This approach has proven effective in our previous projects when testing network protocols, providing detection of various deviations from the specification and other errors.