Reusing Web Thing API's websocket messages for MQTT, using MicroPython

The Web of Things standardization effort by W3C also touches Web Of Things Architectures and Deployments. Devices / Web Things typically have different technical communication stacks, from ZigBee to TCP/IP, HTTPS, MQTT and so on. A single specification is probably not suited to express all these possible interactions, so there are so called Protocol Bindings, which are mappings of the interactions models (Properties, Actions, Events, ...) to underlying protocols (such as HTTP). As this is repetitive stuff, so called Binding Templates serve as blueprints.

The Web Thing API of Mozilla IoT is much more pragmatic - and it leads to useable software such as the gateway, the plugins and code for some embedded maker boards. As of now, it's targeted at HTTP(S) and WebSockets, and the modules at THNG:STRUCTION support HTTP (without websockets).

Now what if i wanted to have an MQTT implementation of communication being similar to the Web Thing API? This is not (yet) specified and implemented, but probably not hard to code as a first example. Let's try it.

Interaction patterns

There are several interaction patterns defined within the Web of Things space: Properties, Actions and Events. Properties are related to the device in terms of hardware capabilities, e.g. a temperature sensor, or they could be virtual values of some form, e.g. a threshold to compare the temperature to. Properties can be read and/or written. If they change, others could be notified about the change. Actions are triggered from outside, and affect the device. Actions do not need to run immediately after triggering, and they can run for longer time, e.g. the (slow) closing of a valve. When an action is started or stopped, others might want to know by notification or query. Whenever the device detects some form of change, it might create an event. Others might register for events, or just query the recent events that occured.

A pure HTTP implementation does all interactions using HTTP request. Setting a property, reading the event list etc. are all HTTP requests. Using websockets, notifications become possible, because the websocket connection remains open and the device can just send out data to notify the other end.

MQTT poor man's binding for web things

Now what about MQTT? At boot time, typically the device connects to an MQTT broker instance. If it wants to emit data, it publishes to an MQTT topic. If it wants to receive data or be notified about something, it subscribes to a topic. Both MQTT broker instance and topic names can be combined within URLs, or in the case of a Thing Description, in href elements. Let's look at the interactions:

Properties

  • Setting a property could be implemented by the device by subscribing to a topic, checking for upcoming messages if they comply to setProperty message style.
  • Getting a property is done by a device actively informing others about changes, by publishing a propertyStatus message to a topic.

That's close to the Web Socket API, since it can use the same kind of messages.

Actions

  • Device subsribes to a topic and waits for an incoming requestAction method.
  • Whenever an action starts or finishes, the device publishes an actionStatus message to a topic.

Events

  • Whenever an event is raised, the device publishes an event message to a topic.

There is an addEventSubscription message in the Web Thing API spec, to register to specific event types. But as the device publishes all events anyway, this subscription can possibly be better handled with MQTT mechanisms at the client side. One thing that's missing is the ability to query a Things' ThingDescription, the json snippet that describes the whole thing. I could imagine an announce message, which is published on a topic every time the device connects to an MQTT broker, possibly published to an alternative device topic as well.

Good thing about this are the re-use of message types already in the spec, and that all of these messages can appear on the same topic. Which means that a topic has to be solely for a single device, otherwise clients and devices don't know which messages to listen for. But that's ok.

To sum it up, a Web Thing API device with the poor man's mqtt binding needs to subscribe to a single topic to set properties and to receive action requests. It publishes property changes, action status updates and events to a single topic. For all these interactions, the same topic can be used. Sounds easy for a first start, let's try it out.

First implementation

I happen to have a WiPy from Pycom around, an ESP32 board suitable for development in Micropython. It has builtin WiFi, and libraries for MQTT are around. PyCom has one at their github library repository. Luckily, Dave Hylands and Michael Stegemann of Mozilla-IoT have a (Micro-)Python implementation of the WebThing API. It's primarily for HTTP, but it already has classes for the whole meta model behind, i.e. Values, Properties, Actions, Events and the Thing itself. It can be found at github.com/mozilla-iot/webthing-upy.

For using this in a PyMakr project, i clone both repos from above, set up some directories and copy files over:

$ mkdir suppl
$ git clone https://github.com/mozilla-iot/webthing-upy.git
$ git clone https://github.com/pycom/pycom-libraries.git

$ mkdir upy-webthing-mqtt-demo
$ cd upy-webthing-mqtt-demo
$ mkdir mqtt upy webthing

$ cp ../webthing-upy/upy/* ./upy/
$ cp ../webthing-upy/webthing/* ./webthing
$ cd ../pycom-libraries/lib/mqtt/mqtt.py ./mqtt

$ # sample demo code, as gists: main.py and thingmqttbinding.py
$ curl https://gist.githubusercontent.com/aschmidt75/706fa7f5360868d6ed45d2bcd9cd0bad/raw/1c887b6f94272730195c5f57c5e63914de683e7a/main.py >main.py
$ curl https://gist.githubusercontent.com/aschmidt75/b341a90c507b861c7b73b7b919b50b38/raw/59d39cdbe08e50405f8aeda2743cdf741d501217/thingmqttbinding.py >thingmqttbinding.py

webthing/ and upy/ contain the WebThing API implementation, mqtt/ the MQTT client class. The last two files are demo code snippets which i uploaded as single gists:

  • thingmqttbinding.py is the glue between the MQTT client class and WebThing classes, and
  • main.py is a sample that exposes the WiPy's onboard LED and the S1 button of a Pycom Expansion board as w WebThing.

We still need a boot.py to connect the WiPy to a WiFi network, this one works for Pycom's WiPy:

from network import WLAN

import machine
import pycom
import sys

pycom.heartbeat(False)
pycom.wifi_on_boot(False)

print("Connecting to WiFi...")

wlan = WLAN(mode=WLAN.STA)

wlan.connect("<YOUR_SSID>", auth=(WLAN.WPA2, "<YOUR_PWD>"), timeout=5000)

while not wlan.isconnected():  
    machine.idle()
    
print(wlan.ifconfig()[0])

# include libs
sys.path.append('/flash/upy')
sys.path.append('/flash/webthing')
sys.path.append('/flash/mqtt')

Add the above as boot.py and fill in your WiFi credentials.

As an MQTT broker, i use Eclipse Mosquitto and its client tools. After starting the broker, i launch mosquitto_sub to subscribe to a topic named pycom. This is also defined in main.py, and it's the topic where the device publishes and subscribes to:

$ mosquitto_sub -t pycom

Once the device boots, it connects to WiFi and to Mosquitto broker, and immediately announces itself on the topic:

{"messageType": "announce", "data": {"@context": "https://iot.mozilla.org/schemas", "description": "-", "@type": ["OnOffSwitch", "Light"], "name": "PycomDemoThing", "properties": {"on": {"label": "Builtin LED On/Off", "description": "WiPy built-in color LED turned on of off", "@type": "OnOffProperty", "href": "/properties/on", "type": "boolean"}, "S1": {"label": "Builtin S1 button", "description": "Button S1 of Pycom Expansion board", "@type": "OnOffProperty", "href": "/properties/S1", "type": "boolean"}}, "actions": {"test": {"properties": {"duration": {"unit": "seconds", "minimum": 0, "type": "number"}}, "href": "/actions/test", "label": "Test"}}, "events": {"test": {"href": "/events/test", "type": "string", "description": "something has happened."}}, "links": [{"rel": "properties", "href": "/properties"}, {"rel": "actions", "href": "/actions"}, {"rel": "events", "href": "/events"}, {"rel": "alternate", "href": "pycom"}], "href": "/"}}

When button S1 is pressed, an event message is published:

{"messageType": "event", "data": {"test": {"data": "button_pressed", "timestamp": "1970-01-01T00:00:24+00:00"}}}

Turning on the LED:

$ mosquitto_pub -t pycom -m '{ "action": "setProperty", "on": true }'

Every 60 seconds, the device publishes a propertyStatus message, together with the values of all registered properties:

{"messageType": "propertyStatus", "data": {"on": true, "S1": false}}

Wrapping up

So far, things work out well. I could reuse all WebSocket messages as MQTT messages, and added an announce message to distribute the ThingDescription. In a HTTPS-based scenario, a device typically needs to announce itself using mDNS or SSDP, with MQTT this comes a bit more natural, since the device starts the interaction by connecting to a broker. Some drawbacks remain: The WebThing/UPY library is focuesed on HTTP and Websockets, and the Thing Description always contains href elements, which in a MQTT context are not really necessary. But from the perspective of energy consumption, using MQTT as a protocol the device is better off, as it can decide when and how much to sleep, and to start interaction for a limited timeframe (as long as the broker buffers messages).

On the whole, a good basis for a THNG:STRUCTION module :)