Connecting to AWS IoT Core using Python and MQTT

Andrew Smith Posted on

Eclipse Paho is an open source implementation of the MQTT publish/subscribe messaging protocol that’s frequently used within IoT applications.

Here it’s used to send messages from a Python 3 device client to AWS IoT Core, triggering an IoT Core rule that inserts the message into an AWS DynamoDB table. The implementation used is described in this post.

The scenario being modelled is that of an IoT application receiving data from multiple environmental sensors. Each sensor sends data such as temperature, humidity and wind direction, together with a device ID and timestamp:

IoTSensorDiagram

The overall steps for implementing this are:

1. Create an AWS IoT Thing for each physical device
2. Create X.509 device certificates and an IoT Core policy
3. Attach the policy to the certificates and the certificates to the Things
4. Activate device messaging and test using IoT Core
5. Create a table for the messages in AWS DynamoDB
6. Implement an IoT Core rule to write inbound messages to the DynamoDB table
7. Check that inbound messages are being written to the table

Detailed documentation for the AWS specifics above is provided here – so only a summary is given below. More focus is given to the Python MQTT implementation used within the device client.

1. Create an AWS IoT Thing for each physical device

The AWS IoT Core GUI is used to create 2 Things – Env_Sensor_1 and Env_Sensor_2. These are logical representations of the physical devices that we will use:

Things1

2. Create X.509 device certificates and an IoT Core policy

Communication between devices and AWS IoT Core is protected through the use of X.509 certificates. A device presents its certificate when sending messages to IoT Core. These certificates can be generated by IoT Core or supplied by the user. In this instance we generate them via IoT Core (one for each device):

Certs

Certificate creation results in files for the certificate itself and private and public keys. The certificate and private key files are downloaded to the device, together with an AWS IoT Core root CA certificate (available here).

An IoT Core policy now needs to be created. Whilst X.509 certificates are used to authenticate a device with IoT Core, IoT Core policies are used to authorise the device to perform IoT Core operations, such as publishing to MQTT topics. Policies are attached to certificates.

The policy created is shown below. It is restricted to only allow devices to connect and publish to IoT Core, where for connecting the MQTT client ID must match the name of an IoT Core Thing, and for publishing the MQTT topic name must be “sensor/data”:

Policy

3. Attach the policy to the certificates and the certificates to the Things

The IoT Core policy now needs to be attached to the 2 certificates. The result is shown here:

PolicyCerts

Finally the Things we created earlier are attached to their respective certificates. Below we see one of these Things attached to its certificate:

CertThing

4. Activate device messaging and test using IoT Core

Devices are being emulated via a Python 3 script running on a Red Hat Enterprise Linux 8 AWS EC2 VM. Two directories were created, one for each device being emulated.

Prior to running the script the 3 certificate related files mentioned in step 2 above were copied for each device to a device subdirectory on the VM:

LinuxCerts

Python 3 was then installed using yum install python3, followed by the Eclipse Paho MQTT library using pip3 install paho-mqtt .

The Python script used to emulate a device is as follows:

PythonScript1

The first section above shows the creation of an MQTT connection using mqtt.Client(). Variables are then set up that define the custom IoT Core endpoint and port for the application, as well as the location of the certificate and private key files. These are passed into the subsequent two library calls, which results in a connection to IoT Core.

PythonScript2

The second section above shows an infinite loop that transmits simulated sensor data every 5 seconds. The key statement here is mqttc.publish(), which publishes a JSON string containing sensor data to an MQTT topic called “sensor/data”.

This script is available in my GitHub repository here. (Note that it is a modified version of the script originally presented here.)

The script is run from the Linux command line on the VM using: python3 env_sensor_1.py

The AWS IoT Core GUI has a test facility where it’s possible to subscribe to an MQTT topic via an MQTT client they have implemented. After subscribing to the “sensor/data” MQTT topic used in the script above and after having run the script on the VM for one of the devices, the test GUI shows that messages are being successfully received from the device:

IoTTest

5. Create a table for the messages in AWS DynamoDB

A table called SensorData is created in DynamoDB, with the MQTT payload device id (unique per device) as the partition key and the MQTT payload timestamp as the sort key:

DynDBSensorData

6. Implement an IoT Core rule to write inbound messages to the DynamoDB table

Finally we create a rule that runs each time a message is received, that inserts the message into the DynamoDB SensorData table. The SELECT statement at the beginning identifies the MQTT topic that we want to filter on, and the action that follows is simply inserting into DynamoDB:

SensorDataRule

Within the rule action, ${deviceid} is entered into the partition key value field. This instructs the rule to take the value of the deviceid attribute from the MQTT message and write it to the DeviceID attribute in the DynamoDB table. ${datetime} is entered into the sort key value field. This writes the value of the datetime attribute into the DateTime attribute. “Payload” is entered into the “write message data to this column” field, which inserts the message payload into the Payload attribute.

SensorDataRule1

Following this, the option to create a default IAM role is selected, which allows the rule to insert items to DynamoDB. Inspection of this role after creation shows that it has a trust relationship with iot.amazonaws.com (i.e. IoT Core can assume this role) and that it has permissions to perform writes (only) to a DynamoDB table with the name SensorData.

7. Check that inbound messages are being written to the table

To check that the rule is working, the contents of the DynamoDB SensorData table can be viewed. Table items inserted by the rule are seen here:

SensorDataDynDBTest