Twitter icon
Facebook icon
LinkedIn icon
Google icon
Reddit icon
StumbleUpon icon
Del.icio.us icon

The Things Network V2 Azure IoT Hubs and IoT Central Gateway

Added to IoTplaybook or last updated on: 01/10/2021
The Things Network V2 Azure IoT Hubs & IoT Central Gateway

Story

Background

After a customer abandoned a proof of concept (PoC) project I had a RAK WirelessWisGate Developer D0+ gateway, RAK WisNode 7200 Track LiteDragino LHT65 LoRaWAN Temperature & Humidity Sensor, and a selection of Seeeduino LoRaWAN and Seeeduino LoRaWAN/W GPS devices in a box under my desk.

Over the last couple of years, I have built a series of Windows 10 IoT Core field gateway applications for connecting LoRa devices to Azure IoT Hubs, Azure IoT Central, and Adafruit.IO. These had worked well but now I needed a Microsoft Azure cloud-based solution to connect my LoRaWAN devices attached to the The Things Network(TTN) to applications running in Microsoft Azure.

Another side project I had been working on libraries for.NET nanoFramework and GHI Electronics TinyCLR devices to enable LoRaWAN connectivity with a RAK811 LPWAN module (in Wisduino form factor for PoC).

In New Zealand, there are two nationwide networks (Spark IoTKotahiNet) but I had been wanting to explore the functionality of TTN and The Things Industries which looked to have sufficient diagnostic functionality for my purposes.

I have assumed that if you are reading this project story you are familiar with developing applications for Microsoft Azure and especially the IoT services. The configuration of TTN Applications and Devices has been covered in detail in several other Hackster.IO projects so I won't repeat it here.

This project is a summary of a series of posts on my blog where I cover the construction of the solution in significantly more detail.

Things used in this project

Hardware components

Seeeduino LoRaWAN
Seeeduino LoRaWAN
 
× 1

Seeed

Seeeduino LoRaWAN W/GPS
Seeeduino LoRaWAN W/GPS
 
× 1

Seeed

WisNode Track D RAK7205
RAKwireless WisNode Track D RAK7205
 
× 1

Rak Wireless

WisNode Track Lite RAK7200
RAKwireless WisNode Track Lite RAK7200
 
× 1

Rak Wireless

 
Dragino LHT65 LoRaWAN Temperature & Humidity Sensor
 
× 1

Dragino

WisGate Developer D0 / D0+ RAK7246 / RAK7246G
RAKwireless WisGate Developer D0 / D0+ RAK7246 / RAK7246G
 
× 1

Rak Wireless

 
Seeed Grove - Temperature&Humidity Sensor (High-Accuracy & Mini)
 
× 1

Seeed

Software apps and online services

 
Microsoft Azure IoT Hub
 
  Azure
 
Microsoft Azure IoT Central
 
  IoT Central
 
Microsoft Azure Functions
 
  Azure
 
Microsoft Azure Key Vault
 
  Key Vault
The Things Network
The Things Network
 
  The Things

In the beginning

I initially connected the RAK WisGate Developer Gateway and configured the RAK7200 Track Lite device. In the TTN Application Device data tab, I could see uplink messages being received from the device and most of the payload getting decoded which was a good start.

Getting Connected

configured my Arduino IDE so that I could access the Seeeduino, LoRaWAN examples, then compile and download them to my devices. I tested modified versions of the Activation By Personalisation(ABP) and Over the Air Activation(OTAA) examples with my local gateway to confirm my device configuration was good. The device code worked a second time after I remembered that I needed to turn the power on for my Seeeduino Grove I2Ctemperature and humidity sensor connector.

Deserialising the TTN messages

I configured a TTN HTTP Integration for one of my TTN Applications so it POSTed uplink messages to an Azure Function with an HTTP trigger endpoint.

TTN Application HTTP Integration Configuration
TTN Application HTTP Integration Configuration

I used JSON2Csharp and a sample Uplink payload I downloaded from the TTN website to generate an initial version of some C# classes to de-serialize the uplink messages.

I had some issues with the generated code due to JSON2CSharp not being able to determine whether a numeric field was an integer or unsigned long.

The TTN documentation indicated that the payload_fields property was populated when an uplink message was successfully decoded. The TTN has a built-in decoder for Cayenne Low Power Payload(LPP) messages which is partially supported by the RAK7200 Wisnode Track Light (a custom decoder/encoder with enhanced functionality is also available).

I used a 3rd party library (CayenneLPP library from Electronic Cats) on my Seeeduino LoRaWAN devices to encode the payload which contained temperature and humidity information.

Outstupiding Myself

Unpacking the payload_fields property caused me some pain. I tried many different approaches but they all failed. After much experimentation, I found that using a C# object was the simplest approach (even though post-processing the field was more complex).

public class PayloadV4
{
   public string app_id { get; set; }
   public string dev_id { get; set; }
   public string hardware_serial { get; set; }
   public int port { get; set; }
   public int counter { get; set; }
   public bool is_retry { get; set; }
   public string payload_raw { get; set; }
   //public JsonObject payload_fields { get; set; }
   //public JObject payload_fields { get; set; }
   //public JToken payload_fields { get; set; }
   //public JContainer payload_fields { get; set; }
   //public dynamic payload_fields { get; set; }
   public Object payload_fields { get; set; }
   public MetadataV4 metadata { get; set; }
   public string downlink_url { get; set; }
}

I also had to add some code to my PoC application to unpack the RAK Wisnode 7200 Tracker accelerometer, gyroscope, and location values which had nested fields.

I then used the Microsoft.Azure.Devices.Client library to connect to an Azure IoT Hub or Azure IoT Central (generating the connection string with DPS-KeyGen) and uploaded telemetry messages which I could see in Azure IoT explorer.

Azure IoT Explorer displaying message payload
Azure IoT Explorer displaying message payload

Processing messages and failures with Azure Storage Queues

For my HTTP Integration, I need to reliably forward uplink messages to an Azure IoT Hub or Azure IoT Central so I used an Azure Storage queue to provide an elastic buffer between my Azure Function HTTPTrigger endpoint and the message processor.

My solution needed to be robust and not lose any messages even when portions of the system are unavailable because of failures or sudden spikes in inbound traffic.

The code for receiving The Things Network(TTN) HTTP integration JSON messages used an Azure Function HTTPTrigger. (secured with an APIKey) and then put them into an Azure Storage Queue for processing.

This code was intentionally kept as small and as simple as possible so there was less to go wrong. After some experimentation, it took less than two dozen lines of C# to create a secure endpoint to receive uplink messages and put them in an Azure Storage queue.

By storing the raw uplink event JSON from TTN the application can recover if they can’t be de-serialized, (message format has changed or generated class issues) When the queue processor can't process an uplink event message (throws an exception) it will end up in the poison message queue after being retried a few times (just in case the failure is transient).

The uplink message queue processor uses an Azure Function Queue trigger to pull messages from the queue, and provision the device if required, retrieve the Azure IoT Hub/Azure IoT central connection string or use the cached DeviceClient.

"Automagic" Provisioning

For development and testing being able to provision an individual device is really useful, though for Azure IoT Central it is not easy (especially with the deprecation of DPS-KeyGen). With an Azure IoT Hub device connection strings are available in the portal which is convenient but not terribly scalable.

Azure IoT Hub Device connection configuration
Azure IoT Hub Device connection configuration

Azure IoT Hub Is integrated with, and Azure IoT Central forces the use of the Device Provisioning Service(DPS). The DPS is designed to support the management of 1000’s of devices that required some custom applications for stress and soak testing.

My HTTP Integration for The Things Network(TTN) is intended to support many devices and integrate with Azure IoT Central. The DPS supports device attestation with a Trusted Platform Module(TPM) but this approach was not possible for my application. My TTN Application Integration uses Group enrollments with Symmetric Key Attestation

DPS Enrollment Management
DPS Enrollment Management

Azure IoT Hub integration

Though you can provision individual devices in your Azure IoT Hub the Azure Device Provisioning Service(DPS) is the preferred approach.

DPS Group Enrollment blade
DPS Group Enrollment blade

The scopeID and the primary/secondary enrollment key are configured in the Azure Key Vault where they can be securely accessed by the Azure QueueTrigger function.

For more complex deployments The Things Network(TTN) devices can be provisioned using different GroupEnrollment keys based on the applicationID (or the applicationID + port number). An example of this would be a tracking device on a truck reporting location data with one port number and cargo temperature and humidity on another so telemetry events can be routed to the right application.

Then as the first uplink message for a device is processed it is “automagically” created in the DPS and Azure IoT Hub.

Azure DPS device configuration
Azure DPS device configuration

Azure IoT Central Integration

The Things Network(TTN) HTTP integration uplink messages had to be provisioned then post-processed so they could be displayed by Azure IoT Central.

The first step is to copy the IDScope, and one of the primary/ secondary keys from the Administration\Device connection and store them in the Azure Key Vault.

Azure IoT Hub devices configuration
Azure IoT Hub devices configuration

For more complex deployments The Things Network (TTN) devices can be provisioned using different GroupEnrollment keys based on the applicationid (or the applicationID + port number) in the first Uplink message which initiates provisioning.

Shortly after the first uplink message from a TTN device is processed, it will be listed in the “Unassociated devices” blade.

Unassociated devices blade
Unassociated devices blade

The device can then be associated with an Azure IoT Central Device Template.

Unassociated devices blade showing recently associated device
Unassociated devices blade showing recently associated device

The device template provides for the mapping of uplink message payload_fields to device properties. In this example, the payload field has been generated by the TTN Application Integration Cayenne Low Power Protocol(LPP) decoder. Many LoRaWAN devices use LPP to minimize the size of the network payload.

Azure IoT Central Device template blade
Azure IoT Central Device template blade

Once the device has been associated with a template a user-friendly device name etc. can be configured.

Azure IoT Central Device properties blade
Azure IoT Central Device properties blade

Azure IoT Central has mapping functionality that can be used to display the location of a device.

Azure Device information
Azure Device information

The format of the location payload generated by the TTN LPP decoder is different from the one required by Azure IoT Central. I have added temporary code (“a cost-effective modification to expedite deployment” aka. a hack) to format the TelemetryEvent payload so it can be displayed.

if (token.First is JValue)
{
   // Temporary dirty hack for Azure IoT Central compatibility
   if (token.Parent is JObject possibleGpsProperty)
   {
      if (possibleGpsProperty.Path.StartsWith("GPS", StringComparison.OrdinalIgnoreCase))
      {
         if (string.Compare(property.Name, "Latitude", true) == 0)
         {
            jobject.Add("lat", property.Value);
         }
         if (string.Compare(property.Name, "Longitude", true) == 0)
         {
            jobject.Add("lon", property.Value);
         }
         if (string.Compare(property.Name, "Altitude", true) == 0)
         {
            jobject.Add("alt", property.Value);
         }
      }
   }
   jobject.Add(property.Name, property.Value);
}

After configuring a device template, associating some devices with the template, and modifying each device’s properties I could create a dashboard to view the temperature and humidity information returned by my Seeeduino LoRaWAN devices.

Azure IoT Central dashboard
Azure IoT Central dashboard

Keeping Secrets Secret

The Application Integration configuration contains sensitive information like Device Provision Service(DPS) Group Enrollment Symmetric Keys and Azure IoT Hub connection strings.

The Azure Key Vault is intended for securing sensitive information like connection strings so I added one to my resource group.

Azure Key Vault overview and basic metrics
Azure Key Vault overview and basic metrics

I wrote a wrapper that resolves configuration settings based on the The Things Network(TTN) application identifier and port information in the uplink message payload. The resolve methods start by looking for configuration for the applicationId and port (separated by a – ), then the applicationId and then finally falling back to a default value.

This functionality is used for AzureIoTHub connection strings, DPS IDScopes, DPS Enrollment Group Symmetric Keys, and is also used to format the cache keys.

The values of Azure function configuration settings (like Azure Storage Connection strings) are replaced by a reference to the secret in the Azure Key Vault.

Azure Function configuration value replacement
Azure Function configuration value replacement

In the Azure Key Vault “Access Policies” I configured an “Application Access Policy” so my Azure TTNAzureIoTHubMessageV2Processor function identity can retrieve secrets.

Go Biggly or go home

In my initial implementations, I used a ConcurrentDictionary to store Azure IoT Hub connections to reduce the number of calls to the Device Provisioning Service(DPS). After some testing, I replaced it with a.Net ObjectCache which is in the System.Runtime.Caching namespace.

I was using a cache to store Azure IoT Hub connections to reduce the number of calls to the Device Provisioning Service(DPS) but the number of connections was still too high.

Number of connections with no pooling
Number of connections with no pooling

So after some research, I decided to enable Advanced Message Queuing Protocol(AMQP) connection pooling.

return DeviceClient.Create(result.AssignedHub,
   authentication,
   new ITransportSettings[]
   {
      new AmqpTransportSettings(TransportType.Amqp_Tcp_Only)
      {
         PrefetchCount = 0,
         AmqpConnectionPoolSettings = new AmqpConnectionPoolSettings()
         {
            Pooling = true,
         }
      }
   }
);

After this, the number of connections was significantly reduced

Number of connections with pooling
Number of connections with pooling

The "Markitecture" diagram

The desired architecture in a nice Visio diagram
The desired architecture in a nice Visio diagram

Oops I painted myself into a corner

After scale and soak testing of the handling of uplink messages I realized that some of the design and implementation choices I had made meant it wasn't going to be easy to process downlink messages.

If your application only needs to receive messages from LoRaWAN devices this solution would be ideal.

To support downlink messages I'm most probably going to have to transition to the MQTT Data API and remove some of the advanced configuration options.

This is going to take a while so if you're interested keep an eye on my blog where I'll be posting about my progress.

Code

Azure IoT The Things Network Integration source code

Source code for all the projects I wrote as part of my Azure IoT The Things Network Integration

KiwiBryn / AzureIoTTheThingsNetworkIntegration

A sample integration of The Things Network and Azure IoT Hubs/Central — Read More

Latest commit to the master branch on 11-21-2020 - Download as zip

Credits

Bryn Lewis

Bryn Lewis

 

Hackster.io

This content is provided by our content partner Hackster.io, an Avnet developer community for learning, programming, and building hardware. Visit them online for more great content like this.

This article was originally published at Hackster.io. It was added to IoTplaybook or last modified on 01/10/2021.