Skip to main content

Configure SSL handshake with mutual authentication

Establishing a secure SSL/TLS session often involves mutual authentication, where both the client and server authenticate each other. This topic outlines the steps required to set up an SSL handshake with mutual authentication using NanoSSL.

Requirements

Incorporating mutual authentication into your SSL handshake requires specific build configurations. Ensure the following flags are included in your build settings:

  • __ENABLE_MOCANA_SSL_MUTUAL_AUTH_SUPPORT__

  • __ENABLE_MOCANA_SSL_CIPHER_SUITES_SELECT__

These flags activate the mutual authentication support and the selection of specific SSL cipher suites.

Configuration steps

To establish a mutual authentication mechanism during the SSL handshake process, follow these instructions:

  1. Activate mutual authentication

    Inside your server configuration code, such as ssl_example.c, the server should demand mutual authentication by setting the appropriate session flag:

    status = SSL_setSessionFlags(connectionInstance, SSL_FLAG_REQUIRE_MUTUAL_AUTH);

    This command configures the server to request a certificate from the client, initiating the mutual authentication process.

  2. Set up a callback for invalid certificates

    Implement a callback function to handle cases where the client may provide an invalid certificate, such as a zero-length certificate:

    SSL_setInvalidCertCallback(connectionInstance, SSL_EXAMPLE_invalidCertCallback);

    This callback gives the server application the discretion to allow or deny the connection in case of certificate errors during the handshake.

  3. Configure cipher suites on the client side

    Use the SSL_enableCiphers function to specify the cipher suites that the client is permitted to use during the handshake:

    // Ensure the __ENABLE_MOCANA_SSL_CIPHER_SUITES_SELECT__ flag is set
SSL_enableCiphers(clientInstance, desiredCipherList);

    The server will then choose an appropriate cipher from the list provided by the client. This selection process complies with the client’s certificate response mechanism as outlined in the relevant RFCs.

    • If the server sends a certificate request message, the client must respond. The client’s response can either be a valid certificate or, in certain cases, a zero-length certificate.

    • The server’s role is to issue the certificate request message and evaluate the client’s response to proceed with establishing a secure connection.

In this section: