Twitter icon
Facebook icon
LinkedIn icon
Google icon
Reddit icon
StumbleUpon icon icon

AI for Good - Workplace Safety

Added to IoTplaybook or last updated on: 12/11/2019
AI for Good - Workplace Safety


Workplace Safety

AI-based - Custom Vision Workplace Safety detection system.

Use Case

Ability to detect vision-based compliance. In an environment where safety is important for the workforce to work, there are compliance rules in place to wear proper gear to safeguard the environment and keep the humans working safe and secured. For example, depending on what manufacturing company the human causality is one thing everyone wants to avoid. So, they insist on wearing proper vests, hard hats, and safety glass or lab coats and glass to protect. In some cases, maybe a mask and complete covered suits for chemical spills, etc. So, the idea here is to detect humans and then see if they are wearing a vest, hard hats, and safety glass. Usually, in manufacturing there are lines where folks can walk and that would be the next future work. Ability to detect humans and compliance and alert management with reporting and Realtime alerts. Also, the ability to detect forklifts and alert humans on the way to the forklift operator. The system just not only detects the objects but also the ability to store the information for further reporting and analysis.

For example: usually, the plants have to provide yearly or quarterly reports to OSHA auditors to make sure if there is human causality and what actions were taken not repeat it. Having the picture when the objects were detected and when not detected is super helpful to analyze the data by the auditors. That makes it very simple and easy for auditors. But the main purpose is the plant or factory can be running and there is no downtime or pull workforce to go through the auditing process. Usually, there is downtime when auditing happens which in this case will reduce and increase productivity and uptime.

It is necessary to detect and provide a report and also it is important to store the data for historical purposes to able to do auditing and also learn from the data. The historical data can be combined with other productivity data and find insights as well. Mostly likely pushing the data in data lake make sense. It is also very important to know how the system is performing. So we need to collect the telemetry and store in Azure SQL and Blob for further processing. We can generate monthly reports or weekly reports on how many compliance issues were raised. We can also analyze the data and find if the model is performing well or find where is it not.

The scenario can be customized to other use cases like hospitals, chemical plants, and various other heavy machinery and mining industries as well.

Here is the Vision AI developer kit page click here

Things used in this project

Hardware components

Qualcomm Vision AI Developer Kit
× 1


Vision AI Dev Kit
Microsoft Vision AI Dev Kit
× 1 Azure Github

Software apps and online services

Microsoft Azure
Microsoft Azure
Azure IoT Edge
Microsoft Azure IoT Edge

Azure IoT Edge

Visual Studio 2017
Microsoft Visual Studio 2017

Visual Studio


Architecture Explained

  • IoHub – To collect detected objects and send them to the cloud to further process using a downstream system. Serves as the gateway to pump data out of the IoT devices and send it to the cloud. Also, device management capability and secured data transmission are all provided by the IoT Hub.
  • Stream Analytics – To read the data from the incoming event hub in IoT hub and parse the JSON and write back into Azure SQL and blob storage. The incoming data set is in JSON format and that has to deflate to a structured format to be able to analyse. Also here we can do windowing based calculation if needed based on the use case requirements.
  • Azure SQL – The detected object data with time at which object was detected are store for creating charts for web site and power bi based reporting. The data is only kept for 6 months to 1-year time frame for reporting and then the cleanup job is created to delete the old records. It is used for Reporting and Dashboarding. Also used for downstream business systems if needed.
  • Blob Storage – the same data going into Azure SQL is stored in a blob for long term storage. Data older than one year or more are kept for auditing records and compliance. Preferably the images should also be stored here so that compliance and auditing can be performed when needed. Storing this data can also be moved to cold storage if needed to keep it for the long term. More on the long term and auditing purposes. Ability to analyze the model results based on historical data.
  • Web App – Dashboard uses Azure SQL data to display the information on the page. It also uses historical data as well from Azure SQL but limited to the data stored in SQL.
  • Camera Tagging Module - Here is another module to take pictures and tag them and send them to custom vision service and also send them to Blob Storage. Blob storage is used for long term and historical data analysis. In a practical use case, we wanted the ability to take real-world pictures and then use them for training custom vision models and make the model more accurate.

To get started to add the module to vision kit follow the below link

For example, in a real factory or plant or hospital or any other scenario, unless we have pictures, it becomes hard to build a model. With the above tagging module, we can take the real world pictures and use that for training. The above module is based on manually taking pictures so that there is control over the picture taken and storage.

Create a custom vision model using pictures available using custom vision service

  • Log into and create a compact object detection project.
  • Upload the images with various tags.
  • Tag each image with the correct tag and bounding box.
  • Train the model and download the model as an AI vision kit deploy files.
  • For details please follow other articles in the vision ai documentation

Deploy the model to Vision Kit

  • Download the model file and upload it to blob storage where we can access it for module twin.
  • You can zip the model and upload.
  • The ModelUrlZip property is assigned in the module twin to download the new vision model created. Now you should see the new model displayed the bounding box when you wear a hard hat or vest or safety glass. The model should also predict a person as well.
  • ""

For more details go Github page here.





Workplace Safety code

balakreshnan / Workplace Safety

AI based - Custom Vision Workplace Safety detection system. — Read More

Latest commit to the master branch on 11-11-2019 - Download as zip


Balamurugan Balakreshnan Balamurugan Balakreshnan

1 project • 3 followers



This content is provided by our content partner Avnet, a global technology solutions provider with end-to-end ecosystem capabilities. Visit them online for more great content like this.

This article was originally published at Avnet. It was added to IoTplaybook or last modified on 12/11/2019.