Annual: 2019

EM029 »
Anthropomorphic robot on FPGA
📁Other: Robotic
👤Edward Rzayev
 (National Research University Higher School of Economics)
📅Oct 08, 2019
Regional Final



👀 2837   💬 4

EM029 » Anthropomorphic robot on FPGA

Description

The goal of this project is to obtain the final result in the form of a fully working and debugged anthropomorphic robot that performs the stated basic and additional functions such as helping people with visual limitations to navigate in space (to warn about the nearest obstacles); a robot guide who will tell the information about objects near him; robot nanny who can take care of the sleep and daytime activities of the child during the absence of parents; companion robot, which can become a friend of man and a good conversationalist.

In the process of development it will be made an application for smartphones that allows you to contact with the robot using Wi-Fi or Bluetooth connections. It was already made a developed neural network that can determine the boundaries of objects based on the received images from the camera.

Project Proposal

1. High-level Project Description

https://github.com/EdwardRzayev/EdwardRobot

URL:https://www.youtube.com/watch?v=cgOC615KqRg

The future is coming, and soon the integration of anthropomorphic robots into our daily lives will not be something special. Robots are already used in various social aspects of human life, for example, as consultants and lecturers at universities. The goal of our team is to develop an anthropomorphic robot based on FPGA.

The main feature of the developed robot will be social interaction with people. The robot will be used as a client between a person and a computer. FPGA is planned to be used as a control device for servos, as well as a device for receiving and transmitting information from sensors to a computer or smartphone. The HPS subsystem will provide processing of a part of high-level functions, as well as high-speed interfaces that connect to an external server directly on the robot.

The scope of the robot is limited only by its physical structure and the imagination of the developer. For example, a robot can be used as a robot guide (a laser pointer will be built in his hands, with which he can point to various objects near him and tell their stories or any information about them). Also, the robot can be used as a robot assistant for people with disabilities. Also examples of the use of the robot are: a robot nanny (for remote child care), a companion robot, etc.

Currently, the body of the anthropomorphic robot is designed and optimized for the DE10-Nano. The research has been done in order to develop a robot head to place a camera in it. At the moment, a neural network has also been developed, which is planned to be used in the future for pattern recognition on a PC. We are also planning to create a speech synthesizer on the FPGA.

A detailed 3D-model of the robot and a virtual environment for remote control of the robot and simulation of its movements have been developed.

 

                   

2. Block Diagram

System layout:

System block diagram:

3. Intel FPGA Virtues in Your Project

As an interface for convenient work with the robot, it is planned to create a mobile application for devices with the Android operating system (installed on Arm HPS), as they occupy more than 75% of the market of smartphones and tablets. This application will allow the user to control the robot quickly and intuitively and monitor its status (for example, battery charge or the quality of communication between the robot and its control device). As a basis for the robot, the board for programming the DE10-Nano will be used, since, at its small size, it allows you to use all the advantages of SoC, including combining high performance ARM HPS core and FPGA flexibility. The robot will receive commands and send data for further processing via Bluetooth or Wi-Fi networks using the HPS core, which is optimized for this type of task. Also it is planned to implement in the application the display of a real-time video directly from the camera of the robot, along with information about which objects the robot has recognized and identified. For this, a neural network has already been developed, which is able to determine the boundaries of the identified objects. Due to the ability of the HPS core to locally implement powerful data processing algorithms and parallelize their execution at the hardware level, for stable operation of the robot, it was decided to create a server that will handle this computer vision task. Finally, the user's device will receive a video with images identified in it. In addition, the application will allow you to control the mechanical movements of the robot using the virtual joystick shown in the figure.

As one of the features of the robot, it will be implemented a functionality that allows people with visual impairment to use the robot as a companion who can inform the user about the impending obstacles (for example, a slippery floor or stairs). Also, the robot will be able to report on those objects, which it will identify. The robot will reproduce the commands through the speakers or through the device controlling it. The control of the robot voice commands is in the development stage.

4. Design Introduction

The main goal of this project is to develop an anthropomorphic robot that performs various types of tasks, starting with the identification of objects from the outside world to serve as an assistant to people with visual impairments and ending with the function of an interlocutor robot. The World Health Organization estimates that about 1.3 billion people suffer from any vision problems. With regard to distance vision, mild visual impairment is observed in 188.5 million people, from moderate to severe - in 217 million people, while 36 million people are affected by blindness. As for near vision, 826 million people live with this kind of visual impairment [1]. Thus, the development of this robot implies an approach to solving an important world problem.

 To achieve these goals, namely real-time image processing, a powerful computing base is required, for the implementation of the project FPGA was chosen, which has such advantages as low power consumption and high speed of information processing, which makes it possible to use a high-precision neural network in mobile conditions. Also, with the help of a servo motor shield connected to the FPGA, direct control of the servos takes place.

Deep Learning algorithm:

1) Image Preprocessing and Model Architecture:

To process the input video stream from the camera of an anthropomorphic robot to identify objects in the images, it was decided to compress the input image to a resolution of 320x240 pixels. This was done in order to significantly reduce the necessary computing power for object recognition, since the anthropomorphic robot must work stably and without system hangs. Then, from the input data, the center of the image is 224x224 in size. Such a move allows you to use pre-trained convolutional layers of various architectures that can create the necessary features for the subsequent identification of objects. In our case, we tested two architectures: Faster-RCNN [2] and YOLO [3]. The first consumed too much computing resources and we could not get it to work without interruptions in the robot, in connection with this an alternative decision was made. YOLO algorithm works much faster than its competitor, which is the main and decisive criterion for choosing an architecture. Naturally, we had to make small changes to the parameters and characteristics of the neural network, however, this allowed us to use the capabilities of the DE10-NANO programmable card to the maximum: often repeated operations were optimized for speed, and with the ability to completely parallelize all the calculations, the system gained stability.

2) YOLO:

YOLO is more effective than RCNN, since in this algorithm the image passes through the layers of the neural network once, while its competitor's image is divided into many smaller ones and classification is already done on them. YOLO at the output returns 5 values, these are the X, Y coordinates, the length along these two axes (for constructing the bounding box) and the probability that one of the classes is in this area of the image.

Comparison of different architectures for detecting objects at VOC 2007 [4]:

5. Function Description

The robot we create have a number of functions:

  1. Processing data from the camera, microphone and ultrasonic sensor for the interaction of the robot with a person and the surrounding space using a neural network
    1. The neural network will use FPGA as an accelerator
  2. The implementation of the movement of the robot. Management via android application
    1. All movements of the robot have both a low-level and a high-level implementation. In the first case, the data is processed and transmitted to the servos using FPGA, and in the second, these rotations of the servos are converted into normal movements like a step, this is implemented using the API on the ARM core
  3. Interaction with the environment
    1. For orientation in space, the robot uses an ultrasonic sensor, a camera, as well as buttons on the hands and feet to prevent collisions with objects that for some reason have not been noticed before.
  4. User interaction
    1. To communicate with the user, our robot uses a built-in microphone and speaker, and it is also able to respond to touch using the touch buttons on the head and body.
  5. User can program robot
    1. The user, using a mobile application, is able to independently create behavioral algorithms for our robot: whether it is a dance or unusual movements due to the specifics of the environment.

 

6. Performance Parameters

In this section, we describe two main design parameters, which, in our opinion, are the most important and will affect the functionality of the total project as a whole.

Robot core

    In modern robotics, there are a huge number of tools. Their choice depends on the tasks of the created robot. Therefore, we faced a problem of choosing the main working base of an anthropomorphic robot.

There were several options: a microcontroller, ASIC, FPGA or FPGA board with an integrated ARM core.

The microcontroller has a number of advantages that had to be considered when choosing a control system for the robot. We considered arduino microcontrollers. Firstly, the microcontroller is easy to program, which greatly affects the speed of development. Secondly, Modern microcontrollers are capable of processing a large number of servos, which is enough to create the movements of an anthropomorphic robot. And thirdly, connecting via bluetooth allows you to create robot control through external devices. And finally, the microcontroller has low power consumption. Despite all the pluses, we rejected this option because of the minuses that are peculiar to it. The most important drawback is the speed of signal processing on the microcontroller. The robot we are developing must quickly process a large data stream from various sensors, after which all this data must be processed by a neural network. Using a microcontroller would make it impossible to implement an analysis of the surrounding space, as well as social functionality in view of the limitations of the platform.

 A microcomputer - so we can call the option we are considering using raspberry pi, is a good option that no longer has such a critical drawback for us as a microcontroller. The microcomputer is built on the basis of a microprocessor, this allows you to increase the amount of processed information, it becomes possible to qualitatively take data from sensors and process them, but there is still the problem of optimal use of the resources of such a solution. Space analysis and communication with a person is realized through neural networks; for them, the process of parallelizing calculations per cycle is very important. It is impossible to perform a huge number of operations on the microprocessor at the same time, all of them will be performed sequentially, which still affects the speed of the neural network and the analysis of data from sensors in general.

The next option for the implementation of the core was an application-specific integrated circuit (ASIC). Such circuits allow you to create systems to solve a specific problem. Historically, earlier FPGAs were slower, less energy efficient, and basically performed fewer functions than they were installed in ASIC counterparts over the same period. Early FPGA is a dynamic device that loses not only its current state, but also its design configuration when the power is turned off. However, in order to test the benefits of the FPGA configuration over the commonly used ASIC, it is important to mention that modern FPGAs have competed with ASIC solutions, providing many advanced features such as lower power consumption, faster speeds , lower costs, increased flexibility, minimal implementation and enhanced re-tuning capabilities. In addition, for some integrated circuits, it may take from 6 to 10 ASICs to implement, for example, a controller, while the same design can be achieved using only one FPGA chip. [7] It is worth considering that FPGA reprogrammability is very important for us to work on a project, which is impossible to do with ASIC, we would have to constantly buy new chips. As a result, between these two options we prefer FPGA.

The FPGA board allows you to parallelize computations and data processing enough for optimal operation of neural networks, and the need to create software for working with a neural network can take a huge amount of time, so we came to a hybrid solution between FPGA and microprocessor - SoC FPGA. This solution combines the diversity of FPGA resources and the performance of a general-purpose microprocessor, the flexibility of configuring I / O ports and support for multi-threaded applications, and such platforms are well suited for those systems that require middleware and a full protocol stack, as well as synchronization circuits [8], which is necessary for our anthropomorphic robot.

We stopped on a hybrid FPGA solution with an ARM core.

FPGA-based platforms still have a number of problems, the main one being device heating.

Cooling system

 One of the main problems of the project is that the electronic components are pretty tightly placed in the robot body. At the same time, there are serious restrictions on the size of the case, since the weight and stability of the entire structure, the reliability of the connections, the power of the servos, and, accordingly, the power consumption and power consumption and battery life.

This, among other things, determines the choice of De10-Nano, as one of the most compact motherboards, with SoC.

The simulation and field testing of various options for the placement of the control board and Servo Motor Card allow us to evaluate the temperature regime of their work:

Device

Temperature, °C

 

 

 

Average

Max

Environmental

DE10-Nano

45

65

22

Servo Motor Kit

23

25

22

Battery

23

26

22

Battery

23

26

22

The screenshot from the Autodesk Fusion 360 program clearly shows the DE10-Nano board with an aluminum radiator and cooler integrated on the chip, as well as a pair of purge coolers.

The aluminum radiator in the aggregate was chosen for two reasons:

  1. It is cheaper than a similar radiator made of copper.
  2. The difference between the heat dissipation abilities is insignificant.

The cooler, located directly above the radiator, dissipates heat on the sides of the board, and purge coolers displace warm air from the robot body.

After temperature measurements on the DE10-Nano with the installed cooling system, the following data were obtained:

Average temperature = 35 ° C

Max temperature = 49 ° C

Environmental temperature = 22 ° C

Thus, we were able to reduce the above indicators, which allowed us to achieve an improvement in the work of the crystal during throttling measurements.

7. Design Architecture

At the moment, there is the following plan for the division of work and the connection of logical modules among themselves:

1) 3D model: it is a model of the robot body, it is based on the required degrees of freedom of the manipulators and on the list of available equipment.

2) Electronics: circuit for connecting components to the board.

3) Low-level software: a set of wrappers over components, which is a set of classes in C ++ with which the manager will interact (item 4)

4) The connecting link, it is the manager, it is responsible for the communication of other components with each other, which makes the modules as independent as possible. It is also a “transfer point” for signals and slots from QT.

5) Neural network: Its tasks include image processing and, based on it, send signals to either servo drives (through module 7) or the application.

6) Application: UI component, a mobile application for controlling the robot.

7) Motion programming module: an important key component; through it the creation of new commands for the robot takes place, it stores functions and only it has the right to call them. Custom function files are also associated only with this module.

8) The module for working with packages: a server, which is a transshipment point between the application and the link.

The algorithm for calling a primitive function:

One way to control the robot is an Android application.

Application features include:

  1. Sensor Performance Display
  2. Running programs on the robot
  3. Direct control of robot elements

The core of the application architecture is the MVVM design pattern.

 

References

[1].https://www.who.int/ru/news-room/fact-sheets/detail/blindness-and-visual-impairment

[2]. https://arxiv.org/pdf/1506.01497.pdf

[3].https://arxiv.org/pdf/1709.05943.pdf

[4].https://medium.com/@jonathan_hui/object-detection-speed-and-accuracy-comparison-faster-r-cnn-r-fcn-ssd-and-yolo-5425656ae359

[5].http://www.atari-forum.com/viewtopic.php?t=32542

[6].http://www.terasic.com.tw/wiki/DE10-Nano_heat_issue

[7]. Robotic Contriller: ASIC verus FPGA - A Review - Article  in Journal of Computational and Theoretical Nanoscience · January 2018, Falih Alkhafaji, W. Z. Wan Hasan

[8].  Microprocessor vs FPGA - http://www.russianelectronics.ru/developer-r/review/2189/doc/56299/

[9].https://github.com/EdwardRzayev/EdwardRobot

[10].https://www.youtube.com/watch?v=cgOC615KqRg



4 Comments

Luis Aljure
i like your robots, nice project keep going!!
🕒 Jul 01, 2019 02:06 AM
EM029🗸

Thank you very much for your interest in our project. We will continue to work on developing our robot.
🕒 Jul 02, 2019 09:50 PM
Doreen Liu
So cool! Looking forward to your work!
🕒 Jun 27, 2019 02:06 PM
EM029🗸
Thanks for your comment. I hope we will succeed!
🕒 Jun 28, 2019 05:46 AM