How do I configure mutual authentication?

Question: How do I configure either the NanoSSL client or NanoSSL server for mutual authentication?

Answer: Both the NanoSSL client and NanoSSL server must be built with the __ENABLE_MOCANA_SSL_MUTUAL_AUTH_SUPPORT__ flag. By default, mutual authentication is enabled for the NanoSSL server, however, may be explicitly controlled at runtime using the SSL_ioctl function to set the SSL_FLAG_REQUIRE_MUTUAL_AUTH flag.

If the NanoSSL client is built with mutual authentication, it processes the certificate request from the SSL server, and replies with a valid certificate based on the lookup in the certificate store. If no certificate is found in the certificate store, or if SSL_ioctl is used to set SSL_FLAG_NO_MUTUAL_AUTH_REPLY at runtime, the client replies with a zero length (null) certificate. For any specific requirement, if the NanoSSL client must enforce the runtime mutual authentication certificate request to be always required from the server, set SSL_FLAG_REQUIRE_MUTUAL_AUTH using an SSL_ioctl call, so if the client does not receive the certificate request the connection is closed.

When NanoSSL server sends a certificate request to the client, the remote client may respond with a valid certificate or empty certificate. In case of an empty certificate, the server invokes the registered funcPtrInvalidCertCallback application callback to determine whether to allow the connection to continue or terminate. When SSL_FLAG_REQUIRE_MUTUAL_AUTH is enabled, the NanoSSL server always expects a certificate response from the client (even though it may include an empty certificate).