Twitter icon
Facebook icon
LinkedIn icon
Google icon
Reddit icon
StumbleUpon icon
Del.icio.us icon

M&M Counter Powered by Azure Cognitive Services & Azure IoT

Added to IoTplaybook or last updated on: 10/11/2019
M&M Counter Powered by Azure Cognitive Services & Azure IoT

Story

It was a rainy afternoon and I was searching for inspiration for a project to introduce high school students to machine learning-based object classification and detection.

Then, as I reached across to a bowl of M&M on my desk I realised that counting the number and colours remaining was a good sample object detection problem.

At the same time in the living room I could hear my son playing with his Lego and I realised that identifying pieces of Lego was a good example of an object classification problem.

Things used in this project

Hardware components

Raspberry Pi 3 Model B
Raspberry Pi 3 Model B
 
× 1

Newark

Adafruit

ModMyPI

SparkFun

Raspberry Pi 2 Model B
Raspberry Pi 2 Model B
 
× 1

Newark

Adafruit

CPC

ModMiPI

DragonBoard 410c
Qualcomm DragonBoard 410c
 
× 1

Arrow

 
Microsoft Generic web Camera
Check you camera on https://docs.microsoft.com/en-us/windows/iot-core/learn-about-hardware/h... even if not on list give it a go.
× 1

Microsoft

 

 

Seeed Grove Starter Kit for 96Boards
 
× 1

Seeed

Grove Base Hat for Raspberry Pi
Seeed Grove Base Hat for Raspberry Pi
 
× 1

Seeed

 
Seeed Grove - Green LED
 
× 1

Seeed

 
Seeed Grove - Universal 4 Pin Buckled 5cm Cable (5 PCs Pack)
 
× 1

Seeed

Software apps and online services

Visual Studio 2017
Microsoft Visual Studio 2017
 
 

Microsoft

Windows 10 IoT Core
Microsoft Windows 10 IoT Core
 
  Microsoft
Microsoft Azure
Microsoft Azure
   

The.Net Core based UWP client application runs on a Windows 10 IoT Core device and is remotely configured (model type, model to run & threshold etc.) using Azure IoT Hub device twin.

Azure IoT Central Device twin settings
Azure IoT Central Device twin settings

 

Azure IoT Central Commands
Azure IoT Central Commands

The images are uploaded to Azure Cognitive Services Custom Vision Service for processing and then the results are post processed (filtered & aggregated) on the device to make them easier to use in Azure IoT Central.

19-08-14 05:26:14 Timer triggered              
Prediction count 33              
Tag:Blue 0.0146500813              
Tag:Blue 0.61186564              
Tag:Blue 0.0923164859              
Tag:Blue 0.7813785              
Tag:Brown 0.0100603029              
Tag:Brown 0.128318727              
Tag:Brown 0.0135991769              
Tag:Brown 0.687322736              
Tag:Brown 0.846672833              
Tag:Brown 0.1826635              
Tag:Brown 0.0183384717              
Tag:Green 0.0200069249              
Tag:Green 0.367765248              
Tag:Green 0.011428359              
Tag:Orange 0.678825438              
Tag:Orange 0.03718319              
Tag:Orange 0.8643157              
Tag:Orange 0.0296728313              
Tag:Red 0.02141669              
Tag:Red 0.7183208              
Tag:Red 0.0183610674              
Tag:Red 0.0130951973              
Tag:Red 0.82097              
Tag:Red 0.0618815944              
Tag:Red 0.0130757084              
Tag:Yellow 0.04150853              
Tag:Yellow 0.0106579047              
Tag:Yellow 0.0210028365              
Tag:Yellow 0.03392527              
Tag:Yellow 0.129197285              
Tag:Yellow 0.8089519              
Tag:Yellow 0.03723789              
Tag:Yellow 0.74729687              
Tag valid:Blue 2              
Tag valid:Brown 2              
Tag valid:Orange 2              
Tag valid:Red 2              
Tag valid:Yellow 2              
05:26:17 AzureIoTHubClient SendEventAsync start              
05:26:18 AzureIoTHubClient SendEventAsync finish               

I used Azure IoT Central to store client configuration, display the number of M&M remaining and the bowl and plot some basic consumption/self control KPIs.

Plot of M&M consumption/restraint
Plot of M&M consumption/restraint

The object detection training process requires at least 15 images per tag, so for 6 different colours that's more than 90 images, which took a while. The object classification model worked pretty well but I would need to upload more training images with different lighting (LED vs. Sun etc.) so the model copes with the varying lighting in my home office.

Loading training images
Loading training images

I iterated the model several times to get the object detection model reliable enough to use in a classroom environment.

Testing the model after adding additional images
Testing the model after adding additional images

The remote configuration should make it easier for students to train and test their own models. Future applications include counting wildfowl on the school stream, and identifying the type/growth stage of plants.

Every so often the contrast on the webcam (on hardware compatibility list) goes wrong and the images are washed out.

Sample of image with constrast problem
Sample of image with constrast problem

The remote configuration should make it easier for students to train and test their own models. Future applications include counting wildfowl on the school stream, and identifying the type/growth stage of plants.

I used a Seeed Studio 96Boards mezzanine on the Dragonboard 410c and Grove Base HAT on the Raspberry PI 2/3 device. These are not required if the digital input to initiate photos and the LED to indicate an image is being processed are not required

I'm looking at building a versions which supports the DragonBoard 410C TPM, disconnected image analysis using a "compact" model downloaded to the device and initiating the image processing in the cloud in an Azure function.

Code

Solution for all my Windows 10 IoT Core Camera applications

Open with Visual Studio refresh Nugets, build and deploy

KiwiBryn / Azure.ML.IoTCore.Camera

A series of projects for taking photos using a USB webcam attached to a Windows 10 IoT Core device and uploading them to Azure for processing with a Machine Learning model — Read More

Latest commit to the master branch on 8-11-2019 Download as zip

Credits

Bryn Lewis

Bryn Lewis

19 projects • 41 followers

Microsoft MVP Embedded, maker, husband & father of two.

Hackster.io

This content is provided by our content partner Hackster.io, an Avnet developer community for learning, programming, and building hardware. Visit them online for more great content like this.

This article was originally published at Hackster.io. It was added to IoTplaybook or last modified on 10/11/2019.
Verticals: