HTTPS on the ESP32 - Part 2, client

In the previous part we've taken a look at a HTTPS server implementation for the ESP32, and although technically it works well it has some drawbacks which in my opinion do not really make it look suitable for serious use cases - speed during connection initialization being the most prominent. Today we're going to see how the reverse direction works out: ESP32 making HTTPS calls to gateways or backend services, which is the basis for secure device to cloud communication. Let's get started.

The Arduino Core for ESP32 contains ports of Arm's Mbed TLS and openssl, but they're buried deeper within the SDK, so they typically do not come into appearance when coding for the Arduino Framework with all these WiFiClients, digitalRead, loop and setup. And regarding HTTPS as a client, this isn't necessary. Wherever you'd make a HTTP call (or in general open a tcp connection, you can use the WiFiClient class. Additionally, the ESP32 libraries include a WiFiClientSecure, which interfaces with mbedtls to establish a secure TLS-Connection.

On the surface, WiFiClientSecure is used equal to WiFiClient, as it inherits from this class. What's necessary in addition is to set the CA certificate of the CA that signed the server's certificate, so that the device can be sure to connect to a server, which received a certificate from the respective CA. (at least better than nothing..) One important aspect is that - at least in my tests - the ESP seems to be unable to deal with self-signed certificates or a self-created CA. So for tests i've been using a server certificate signed by LetsEncrypt.

There's demo code available for WiFiClientSecure, which makes connection to howsmyssl.com, and it works well. To play around with it, i've taken the code and redirected it my own HTTP server.

So first we need a server to connect to.

Server side

The server code is written in go, packaged as a docker container, and given a server key and certificate. server.go has a single endpoint which prints out some basic TLS information it can obtain from the http.Request:

package main

import (
    "fmt"
    "strings"
    "net/http"
    "log"
)

func TLSInfoServer(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/plain")

    if r.TLS != nil {
	fmt.Fprintf(w, "TLS Version: 0x%x\n", r.TLS.Version)
	fmt.Fprintf(w, "Cipher Suite: 0x%x\n", r.TLS.CipherSuite)

	if len(r.TLS.PeerCertificates) > 0 {
		cn := strings.ToLower(r.TLS.PeerCertificates[0].Subject.CommonName)
		fmt.Fprintf(w , "CN: %s\n", cn)
	}
     }

}

func main() {
    http.HandleFunc("/", TLSInfoServer)
    err := http.ListenAndServeTLS(":443", "cert.pem", "privkey.pem", nil)
    if err != nil {
        log.Fatal("ListenAndServe: ", err)
    }
}

Let's put this in a simple (yes, unoptimized) Dockerfile:

FROM debian:9

RUN apt-get update -y -q && apt-get install -y golang

RUN mkdir /app
WORKDIR /app

ADD server.go /app/server.go

CMD [ "go", "run", "server.go" ]

It assumes that the server's key and cert are on the host and can be mounted into the container:

$ docker build -t gohttps:latest .
$ docker run -ti --mount type=bind,source=privkey.pem,target=/app/privkey.pem --mount type=bind,source=cert.pem,target=/app/cert.pem -p 0.0.0.0:6565:443 gohttps:latest

The code does not produce output, just errors to stderr. Let's look at the client code.

ESP32 https client

I'm goint to take the WiFiClientSecure.ino demo sketch from arduino-esp32 as a basis:

$ pio init -b nodemcu-32s --ide vscode
$ curl -L https://raw.githubusercontent.com/espressif/arduino-esp32/master/libraries/WiFiClientSecure/examples/WiFiClientSecure/WiFiClientSecure.ino >src/main.cpp

To get more debug messages on the serial console of the ESP32, add the following to platformio.ini:

build_flags = -DCORE_DEBUG_LEVEL=ARDUHAL_LOG_LEVEL_VERBOSE

Within src/main.cpp, change lines #11 and #12 to match your WiFi SSID and password. If you happen a have a server certificate signed by LetsEncrypt and its recent X3 CA, you do not need to change lines 21-47 which contain the CA certificate in PEM format. Otherwise, from the CA certificate of your servers' CA, openssl x509 -in <file> -text and copy/paste the base64-encoded certificate part into the source code.

Change line #14 to match your server name, change line #80 to match your port (it's 6565 in my case). Change lines #85 and #86 to match the server name, or delete #86 and change #85 to GET / HTTP/1.1 which should do as well. Compiling, flashing and testing yields:

$ pio run -t upload
$ pio device monitor -b 115200

(...)

Attempting to connect to SSID: <YourWiFi>
.Connected to <YourWiFi>

Starting connection to server...
[V][ssl_client.cpp:52] start_ssl_client(): Free heap before TLS 153488
[V][ssl_client.cpp:54] start_ssl_client(): Starting socket
[V][ssl_client.cpp:90] start_ssl_client(): Seeding the random number generator
[V][ssl_client.cpp:99] start_ssl_client(): Setting up the SSL/TLS structure...
[V][ssl_client.cpp:112] start_ssl_client(): Loading CA cert
[V][ssl_client.cpp:147] start_ssl_client(): Setting hostname for TLS session...
[V][ssl_client.cpp:162] start_ssl_client(): Performing the SSL/TLS handshake...
[D][ssl_client.cpp:173] start_ssl_client(): Protocol is TLSv1.2 Ciphersuite is TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256
[D][ssl_client.cpp:175] start_ssl_client(): Record expansion is 29
[V][ssl_client.cpp:181] start_ssl_client(): Verifying peer X.509 certificate...
[V][ssl_client.cpp:190] start_ssl_client(): Certificate verified.
[V][ssl_client.cpp:205] start_ssl_client(): Free heap after TLS 114320
Connected to server!
headers received
TLS Version: 303
Cipher Suite: c02f

The last three lines have been returned by the server, so it successfully connected using HTTPS. The hex constants can be looked up in the golang package docs, and 303 as a TLS Version means TLSv1.2, c02f as Cipher Suite equals to TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, which is identical to the one within debug output. Feel free to include other fields from the tls.ConnectionState struct in the go code.

Adding client side certificates

The go ouput does not list peer certificates, as the client did not present any - as the server did not request any! Let's try to change that by first generating a key and self-signed certificate, to be used by the ESP32 device:

$ openssl genrsa -out server.key 2048
Generating RSA private key, 2048 bit long modulus
.....................+++
..............................................................................................+++
e is 65537 (0x10001)


$ openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:DE
State or Province Name (full name) []:NRW
Locality Name (eg, city) []:Somewhere
Organization Name (eg, company) []:My
Organizational Unit Name (eg, section) []:Own
Common Name (eg, fully qualified host name) []:esp32.local
Email Address []:

The code needs both key and certificate in PEM format, as a single string. I use sed to bring it into shape:

$ cat server.crt | sed -e 's/\(.*\)/\"\1\\n\" \\/g'
$ cat server.key | sed -e 's/\(.*\)/\"\1\\n\" \\/g'

Paste the output to src/main.cpp to lines #50 and #51. Go to #76/#77 to activate them. Caution, the example code passes the cert as the key and vice versa. So correct this to:

  client.setCertificate(test_client_cert); // for client verification
  client.setPrivateKey(test_client_key);	// for client verification

Now, back again to the go server part. We have to make the HTTPS server request certificates, so change main() to:

func main() {
    tlsConfig := &tls.Config{
      // NoClientCert
      // RequestClientCert
      // RequireAnyClientCert
      // VerifyClientCertIfGiven
      // RequireAndVerifyClientCert
      ClientAuth: tls.RequestClientCert,
    }
    server := &http.Server{
      Addr:      ":443",
      TLSConfig: tlsConfig,
    }

    http.HandleFunc("/", TLSInfoServer)
    err := server.ListenAndServeTLS("cert.pem", "privkey.pem")
    if err != nil {
        log.Fatal("ListenAndServe: ", err)
    }
}

This sets up a tls.Config struct with ClientAuth set to tls.RequestClientCert. The snippet contains other options, such as RequireAndVerifyClientCert (which would require to additionally set up CA certificates). We'll start with the simple setting to ask the client for (any) certificate.

Build and run the container again, compile and flash the ESP code. It should output something similar to this:

Starting connection to server...
[V][ssl_client.cpp:52] start_ssl_client(): Free heap before TLS 153488
[V][ssl_client.cpp:54] start_ssl_client(): Starting socket
(..)
[V][ssl_client.cpp:244] send_ssl_data(): Writing HTTP request...
[V][ssl_client.cpp:244] send_ssl_data(): Writing HTTP request...
headers received
TLS Version: 0x303
Cipher Suite: 0xc02f
CN: esp32.local
[V][ssl_client.cpp:213] stop_ssl_socket(): Cleaning SSL connection.
[V][ssl_client.cpp:213] stop_ssl_socket(): Cleaning SSL connection.

and now it includes the line CN: esp32.local, which means that on the server side we can check for valid certificates and get information such as users and/or device IDs out of certificates. What still needs to be done is some investigation into self-signed certificates, which in my case caused Mbed TLS errors during connection initialization.

Wrapping up

Again, things technically work out. A TLS handshake still takes a significant amount of time. Placing calls to millis() in the ESP32 sketch yields ~2.2 seconds necessary for the client.connect() function call. Again too much to make multiple calls. But a valid basis for an initial connection followed by a protocol upgrade to e.g. websockets. The TCP connection can remain open for some subsequent calls. I'm curious about throughput, i still have to test that. But it's a valid basis to start further experiments from :)