InnovateFPGA

With the rapid emergence of IoT-based technologies in the recent era, the need for high-performance edge computing to handle data and run AI-based algorithms has increased. The objective of moving the computation from the cloud to the edge devices for compute-intensive tasks, provides increased reconfigurability, low latency, and scalability. In this project, we use Intel’s DE-10 Nano SoC FPGA, by leveraging the ARM Processor to run Linux Server, Docker containers, etc., required to host the IoT system for the premise, we offload the AI-based object detection task from the HPC to the FPGA to accelerate it. The system design uses the ARM Processor to run the Linux Server and connect to the home network hosting the Home-assistant framework, NodeRED, Grafana, and other services. On the other hand, every switch box in the house is equipped with a Wi-Fi-based Custom ESP8266 based Relay switches running Tasmota firmware which is connected to the same network. Similar edge deployments are made for getting temperature, water level in tanks, etc., All these data are communicated to the Home Assistant server running on the HPC. Using Azure Event hub integration we send the data to the Azure IoT Hub. The security system consists of CCTV Cameras which are connected to DVR and using ONVIF the stream is captured in our application for object detection and processing. We use the FPGA for Accelerating the YOLOv2 object detection algorithm. We process the results and in the event of an intruder/vehicle/motion detection, the necessary alarms/lightings/or any other actions like email/SMS/mobile alerts, etc. can be set up according to the end-user requirements. The use of FPGA in this kind of security system allows to train new data sets for uses cases like but not limited to automatic distinguished detection of house members to perform actions like opening the garage doors/preventing false-positive security alarms/ lighting based on motion detection etc., We believe that by using SoC FPGAs especially DE-10 Nano, along with Microsoft Azure IoT, we would be able to accomplish a reconfigurable, scalable and low-cost IoT setup that can be deployed in real-time.

Project Proposal

1. High-level project introduction and performance expectation

Purpose of the design

Application scope and target users

This can be deployed as IoT Hubs and Processing elements for the CCTVs in places ranging from domestic homes to large industrial buildings. This eliminates the need for a central unit to handle all the data related to IoT and security systems. With suitable modifications, the same can be deployed in a distributed fashion minimizing power consumption and central point of failure. In addition to deployments pertaining to IoT and security systems, this can also be deployed in a wide variety of applications ranging from toll gates, traffic signals etc., for various use cases like detection of vehicle numbers moving at a high speed and logging data.

High Level Functional Description

The functionality of the deployment can be divided into services on the ARM Processor and the acceleration carried out by the FPGA. The HPC runs a version of Linux, hosting the Home Assistant Server. This is connected to the internal network/ internet through the on board ethernet port. In addition to running the basic functionality of a web server/ MQTT Broker, it also handles the databases facilitated by InfluxDB, NodeRED service is run to enhance the flexibility in the deployment of automations and configurations. A history of the resource utilisation is continuously monitored and logged in a time series database which is visualised by Grafana. This provides insights on the usage metric, intrusion frequency, etc, and upon processing the data allows to arrive at an estimate of the power consumed. The ARM processor receives the live stream from the NVR, through the ONVIF protocol. This stream is embedded into the home assistant dashboard. In addition the image frames are processed with YoLo Framework where certain CNN algorithms are accelerated using the FPGA,there by reducing the time to action for intruder alert, family member detection, turning on/off lights etc.,

The DE10-Nano Development Kit presents a robust hardware design platform built around the Intel System-on-Chip (SoC) FPGA, which combines the latest dual-core Cortex-A9 embedded cores with industry-leading programmable logic for ultimate design flexibility. Users can now leverage the power of tremendous re-configurability paired with a high-performance, low-power processor system. Altera’s SoC integrates an ARM-based hard processor system (HPS) consisting of processor, peripherals and memory interfaces tied seamlessly with the FPGA fabric using a high-bandwidth interconnect backbone. The DE10-Nano development board is equipped with high-speed DDR3 memory, analog to digital capabilities, Ethernet networking, and much more that promise many exciting applications.

Design

Leveraging the ARM Processor and the FPGA, DE-10 nano provides flexibility in the deployments with the Processor handling the web services, controllers while the FPGA is dedicated to accelerate image processing in real time scenarios demanding less latency to take quick actions. In this project, we use Intel’s DE-10 Nano SoC FPGA, by leveraging the ARM Processor to run Linux Server, Docker containers, etc., required to host the IoT system for the premise, we offload the AI-based object detection task from the HPC to the FPGA to accelerate it. The system design uses the ARM Processor to run the Linux Server and connect to the home network hosting the Home-assistant framework, NodeRED, Grafana, and other services.

On the other hand, every switch box in the house is equipped with a Wi-Fi-based Custom ESP8266 based Relay switches running Tasmota firmware which is connected to the same network. Similar edge deployments are made for getting temperature, water level in tanks, etc., For the camera streams 8 CCTV cameras are connected to a NVR and the home network. The stream can be imported using the web url in the python script and the Image detection is carried out. All these data are communicated to the Home Assistant server running on the HPC. Using Azure Event hub integration we send the data to the Azure IoT Hub. The security system consists of CCTV Cameras which are connected to DVR and using ONVIF the stream is captured in our application for object detection and processing. We use the FPGA for Accelerating the YOLOv2 object detection algorithm. We process the results and in the event of an intruder/vehicle/motion detection, the necessary alarms/lightings/or any other actions like email/SMS/mobile alerts, etc. can be set up according to the end-user requirements. The use of FPGA in this kind of security system allows to train new data sets for uses cases like but not limited to automatic distinguished detection of house members to perform actions like opening the garage doors/preventing false-positive security alarms/ lighting based on motion detection etc., We believe that by using SoC FPGAs, especially DE-10 Nano, along with Microsoft Azure IoT, we would be able to accomplish a reconfigurable, scalable and low-cost IoT setup that can be deployed in real-time.

Intel FPGA Virtues in this Project:

Communication between the Processor and FPGA is involved.
Maximum efficiency can be achieved by pipelining
Linux Operating System running on the ARM HPC manages Home Assistant , web server related processes and cloud interfacing.
Offloading acceleration specific programs to be run on the FPGA and returning the results to ARM HPC, is easy because of the tight integration in the form of an SoC FPGA
Generally, neural networks are trained on GPUs. There always exists a tradeoff between performance and energy and power utilization. Intel FPGA offers a great balance to this tradeoff and enables it to perform well for edge applications.
Lesser power consumption and latency by using embedded FPGAs

2. Block Diagram

3. Expected sustainability results, projected resource savings

Expected Sustainability Results

It is seen that FPGA provides reduced power consumption compared to the traditional approach. These savings and sustainability results would reflect in the costs cut down in eliminating the need for a high performance server, with high performance GPUs for handling the Server as well as to perform the hardware acceleration. Also the ARM processor running Linux can send data over the network/internet and a unified dashboard can be built with the data from Azure IoT Hub. This provides flexibility of data access from any device.

Deployment of IoT and Intelligent Security systems in large deployments pose a huge challenge in terms of power, performance and cost. Hence, by using SoC FPGAs, a distributed cluster like topology can be deployed with accelerated compute capabilities harnessed from the programmable logic. This system is interfaced to the cloud using the Azure IoT service, eliminating the need for a reliable on premise high performance server and compute units.

Existing Implementation

This setup has been developed and deployed previously in a house with over 30 IoT devices and 8 CCTV Cameras with the HomeAssistant Server and Image Processing running on Raspberry pi 4. However, the use of Raspberry Pi hugely bottlenecks the versatility of the Home Assistant framework and our own custom integrations. The intruder detection system functioned with a huge latency around 10-15 seconds, due to the performance overhead caused by running the home assistant server, and inputting all the 8 CCTV Streams into the Raspberry Pi from the DVR. Also, there was a slow down in running NodeRED, VSCode Server, InfluxDB, Grafanna due to the excessive CPU utilization caused by running the object detection frameworks on Raspberry pi. Though the software stack is flexible and versatile, the utility has been bottlenecked by the hardware.

Expected Improvements

By deploying the IoT system on DE-10 nano, the ARM Processor would handle the Home Assistant server and its associated services completely without the overhead of running complex CNN Algorithms for image processing on the CPU for intruder/face detection etc. This is accelerated with the FPGA thereby reducing the hardware bottleneck. We expect to reduce the latency of intrusion detection and face recognition to a latency <2 seconds.

This implementation then can be expanded on a large scale, and a cluster of DE-10 Nano’s communicating with Azure IoT can be used to build a cloud dashboard in addition to a local Home Assistant Dashboard for every such node. This would eliminate the need for a central power hungry infrastructure prone to failures and provision for arranging redundant systems thereby resulting in power and cost savings. Ultimately, the use of such a distributed deployment results in a lesser carbon footprint providing a sustainable intelligent IoT and security solutions.

4. Design Introduction

5. Functional description and implementation

High Level Functional Description

Convolutional Neural Network Inference:

Testing and deploying neural network models for inference on FPGA platforms using RTL design methodology is a herculean task. Generally models are trained on high end data centers using one of the many available deep learning frameworks such as Tensorflow, PyTorch, Keras, Caffe, MXNet, etc. Intel's tools such as OpenVINO make our life easier. It provides tools that convert trained models to reduce the memory footprint of the model using optimization techniques like quantization and graph optimization. It also provides dedicated inference APIs that are optimized for specific hardware platforms. By using this toolkit, we make the best use of the underlying Intel Hardware.

We planned to train the YOLO or MobileNetV2 model on a high-end GPU machine using either Pytorch or Tensorflow. For a dataset to train, open-source datasets like Wider Face or CelebA can be used.

Next we generate an Intermediate Representation(IR) file using Intel OpenVINO model optimizer. Finally we run our CNN model on the OpenVINO Inference Engine.

Source: OpenVINO

There is yet another way to run models using Blueoil which is an open source deep learning platform for low-bit computation.

Generally, neural networks are trained on GPUs. There always exists a tradeoff between performance and energy and power utilization. Intel FPGA offers a great balance to this tradeoff and enables it to perform well for edge applications.

6. Performance metrics, performance to expectation

Existing Implementation

One of the IoT Dashboard in Home Assistant

Intrusion detection within home assistant

Expected Improvements

7. Sustainability results, resource savings achieved

Software Stack

Hardware (described in Section 2)

AI based Smart Home Assistant and Security System