EM046 » Smart home system on DE10-Nano
The development object is a hardware and software complex for the device management and access system management for the smart home system. The project is divided into components: a neural network for facial recognition, a server, a mobile application for Android, a control system for DE10-Nano modules.
Implemented a mobile application that supports communication with external devices de10-Nano. For communication applications and de10-Nano created the server, dispatching requests to switch on the devices. Added face registration to provide additional functionality – "smart lock". Developed and trained neural network that allows you to determine the registered user.
Further development of the work is to improve the quality of user recognition, increase the range of signal transmission, expanding the list of managed modules and the functionality of the application.
GitHub: links below.
Goal: The task set before us is part of the rapidly developing sphere "Internet of things" (IoT) technologies – the Internet of things. Our goal is to develop a hardware and software complex for managing devices and access control system for the smart home system.
The advantage of our system is a comprehensive approach: ensuring user security, remote control of any device entered in the system of a smart home, the separation of user rights.
Application: The system is designed for everyday use. With a user-friendly interface, the user can easily remotely control household appliances. The facial recognition authentication system provides additional security and protection against intruders. This will allow you not to worry about the fact that your home can be managed by someone without your knowledge. Assigning permissions allows an administrator to create and assign new user groups. For example, you can limit the list of devices used for a child.
Target audience: At this stage of development, the system is a prototype of a future full-fledged set of sensors, applications and servers. The end users of the complex are the owners of private houses, apartments or studios. Now such companies as Apple, Amazon, Google, REDMOND, Xiaomi are popular in the smart home market. Our solution involves lower costs, and as a result to make the smart home system more accessible to people with less income. In addition, the implemented security system (“smart lock”) and the division of access rights deserve additional trust of buyers.
*System flow diagram:
*Smart lock working system block diagram:
The most important advantages of the SoC DE10-Nano board is the FPGA and HPS available on a single chip. This provides an opportunity to implement high-level processing algorithms for Smart house control, video processing, information from sensors and Internet connections at the ARM core HPS level. Due to this, the FPGA subsystem can be used to control the low-level functions of Smart things, and also as an acceleration of calculations, heavy for a general-purpose processor core. The organization of the chip itself also forces this, where part of the pins is accessible only through HPS.
Using Intel FPGA chips provides an ability to quickly apply changes in the design of the system and modify the project. The DE10-Nano board has a wide range of interfaces and tools which facilitate a design of such control units. In addition, part of the functionality of the control module can be implemented using the high-level Open Vino / OpenCL libraries provided by Intel.
Creating a system for monitoring printing process on SoC allows a real-time execution of parallel data processing from sensors and provides a high-level accuracy and reliability of Smart Home system functioning.
The project consists of a server with a neural network for face recognition, an Android mobile application, and a DE10-Nano board. For work, a certain logic of component interaction is used.
Each new user needs to download and install a mobile application on their smartphone. When using for the first time, you need to register, to do this, select the corresponding button on the application start page.
To register, you must specify a username, username and password, then the application will offer to go through the process of initializing the face by scanning the face through the front camera of the smartphone. In the future, to enter the application will be used previously set login and password.
After authorization in the application, the user enters the main screen with a list of devices available for management. The list of devices is limited by the access rights of each individual user, and can be expanded or changed from the administrator account. A request to enable the selected device is sent to the server, where it is processed and then transmitted to the DE10-Nano.
The request received from the server is interpreted on the DE10-Nano as a call to the module corresponding to the sensor to receive data. If the device must transmit data, such as temperature / humidity, then the FPGA received it sends it back to the server, and from there back to the user device.
In addition to devices operating at the request of the user, there is a smart lock, the operation of which is based on the following logic. A smart lock consists of a motion sensor, a distance sensor, a USB web-camera, a servo drive, two diodes.
The system starts with motion detection, after which the motion sensor starts reading data. If the user stands at a fairly close distance, the green diode turns on and the camera starts taking photos. Photos are sent to the server, and there, if the user was recognized as registered in the system, then a “200” response is sent to the DE10-Nano, otherwise “400” is negative. Depending on the received answer, the green or red diodes turn on: green when the user is successfully recognized, red - when the failure. Also, the operation of the servo, simulating the lock devices, depends on the answer. At 200 - the servo rotates 150 °, at 400 - 25 °.
The hardware part of the project consists of a server organized on the basis of the computer to which the camera is connected, DE10-Nano boards to which Smart Lock is connected, HC-SR04 rangefinder sensors, DHT11 temperature and humidity sensor, MG90S servo drive, SR505 motion sensor, indicator LEDs and relays.
To communicate with the DE10-Nano server, a uart connection module was implemented, consisting of an RX module for receiving data by the FPGA card and a TX module for transmitting data. The baud rate can be set to the following values 9600, 19200, 38400, 57600, 115200 baud.
The SR505 motion sensor, when motion is detected, returns a 5V digital signal, so a dog level converter from 5V to 3.3V was used to connect it to the DE10-Nano.
For the HC-SR04 rangefinder sensors, a module has been developed that implements the following actions: first, a high signal level is applied to the TRIG sensor contact for 10 microseconds to generate an ultrasonic wave. Then there is a wait when a high signal level does not appear on the ECHO pin of the sensor and the time is counted until the signal applies a low value. Further, to translate the obtained value into the distance to the obstacle in centimeters, the obtained value is multiplied by a coefficient of 0.01723.
To work with servo-drive, a frequency divider module was implemented, which receives a 50 MHz clock signal at the input, and generates a clock signal with a frequency of 26 kHz at the output. At this frequency, a PWM signal of the required duty cycle is generated.
The project took up 30% of the crystal resources, which means a large reserve for expanding the system.
Applications running at the kernel level have enough response to talk about a real-time system.
System power consumption:
It is also at a sufficient level, which allows to ensure with the help of: 1 LiPo battery with 14.1 V power supply, continuous operation of the system with sensors and a video camera for 5 hours.
When powered from the electric socket - the system worked for more than 100 hours in active operation without failures.
The project consists of several parts
For comfortable interaction with the server and management of smart home devices, an intuitive interface has been developed that allows:
- to authorize the User;
- to register the User;
- to get data on current temperature and humidity;
- to get a list of all available devices for this user;
- to manage devices (at this stage of the project, for example, the simulation of turning on and off the device was implemented).
In addition to the standard registration through the login and password added face registration for the further work of the smart lock. The application interface can be found in the screenshots:
during the work on the project, the “smart lock” module was added. It consists of a security camera, motion sensor, distance sensor, diodes and the lock itself. The logic is as follows:
Database and neural network:
a console interface for interacting with the application was developed to work with the database and neural network. The interaction is carried out by passing arguments when the main server file is started. The first argument is the module with which the interaction will take place. Possible values are “recognition” and “database”. The following arguments pass commands to the module and arguments to the commands:
Server / client:
for communication and smart devices implemented the server that stores the network and the database. The server performs user recognition by photo and acts as a connectable object between the application and the FPGA. The server stores a database of users and various devices, with its help authorization and registration of new users, as well as differentiation of their roles, so that each user can use only those devices for which he has rights.
communication with the application takes place through the transmission of requests through the server. For example, when you select a device, a socket is created in the application to which the id of device is sent. The socket is sent to the server, processed and transmitted to de10-Nano, where the id is identified and starts the function necessary to turn on the device.