EM020 » Smart Glasses for the Blind People
Todays, because of importance of performance, efficiency, cost and product presentation time, the influence of artificial intelligence (AI) in smart manufacturing is rapidly growing. Thus, we attempt to present a new type of smart glasses for the blind people by using AI with machine learning to improve their quality of life. In fact, our proposal can help visually-impaired people, specifically, completely blind people to identify person, detect objects, cross the street and generally, help them to navigate and find orientation without any assistance.
Our smart assistive device is very high efficient because of using FPGA implementation. In this system, we use a DE10-Nano SoC FPGA kit and a camera as main hardware. Also, we use an audio system and other software/hardware peripherals to complete the requirements of the target device.
The world health organization (WHO) reported that there are 285 million visually-impaired people in the world. Among them, there are 39 million who are totally blind and unfortunately all these numbers to be doubled by 2020 . Hence, despite the different devices, the existence of a high-speed and low cost device with a high flexibility is necessary to improve the blinds’ quality of life.
Here, we propose a smart wearable device for the blind people to detect objects and identify familiar person in different places by using Field-Programmable Gate Array (FPGA), AI-based algorithm like Convolutional Neural Network (CNN) and image processing. In this design, we focus on one of the biggest problems of blind people; Detect and identify people and objects in order to have an easy communication with their surrounding environment and navigation.
According to the structure of this device, the type of any visited object or person will be determined by sending the camera frames to FPGA kit, and then by making smart decision in AI unit. Thus, if the object is human, the engine will search in a picture database to find a familiar match and announce the result to the user by an audio system. Since, the project has image processing, making smart decision and audio system units; we use Intel FPGA as a best kind of control platform at parallelizing tasks.
The DE10-Nano kit integrates the capabilities of a Cyclone FPGA and a dual core ARM-based hard processor system (HPS). High-speed DDR3 SDRAM is applied to store the familiar person as a database to identify them among other people. Also, by training our device, it can detect different type of object like human, car, traffic light, sign, crosswalk and pedestrian-bridge, leading to understand the type of obstacle in front of the blind user with an audio system.
Application area and target users
This smart wearable system will be used to help visually-impaired people, specifically, who are totally blind to detect object, recognize person, and navigate without other aids. Moreover, by using of this type of smart glasses, our target users (blind people) will be able to identify their familiar person like their friends among other people.
In the future, one of the important thing that we want to add this design is ability to delete and add people to our database of familiar person. For example, if he/she doesn’t see one of his/her for a long time like 5 years, our device will delete that person. And if the blind user see a person a lot, like new friends in the university or school, our system will be able to add these new pictures to the database. Also, we can add a text processing unit to these glasses for text-to-speech applications such as reading newspapers or studying. Furthermore, by adding a GPS unit into the system, visually-impaired people will be able to find places such as bus/metro stations, and generally, they can easily travel from one place to another without the help of others.
Fig.1. Smart glasses block diagram
Fig.2. State transition graph
Fig.3. System flow diagram
According to figures of (1), (2) and (3), our proposed design consists of four main parts: take a video frame, a machine learning unit, an image processing part and an audio system peripheral.
Take a Video: In this part, camera takes the video from surrounding environment of the blind user.
Machine Learning: training the neural network with CNN algorithms, the machine learning unit analyzes the camera frame and extracts the features in order to detect object, and recognize the type of object as a car, human and etc.
Image processing: In this unit, people’s photo will be searched in the system database to find a match with the photos of user’s known people.
Audio system: The result of image processing unit and machine learning unit will be announced to the user by a speaker. Of course, in noisy places, we can add a vibration motor or buzzer to this design for warning to user in the future.
The most important difference between our proposed smart glasses with other similar devices is the use of a large family and friends’ images database to help blind users to communicate better with other people.
Our main goal in this project is the awareness of the blind person in dealing with other people and also, navigation and finding the way, especially to cross the street. So that, it can be announced to the blind user when the AI system identifies a familiar person in different places; or the device sees an object such as car, traffic light, pedestrian-bridge, crosswalk and sign.
Using of DE10-Nano FPGA kit in this project is so important, because in compared with other implementation platforms, it can perform tasks in parallel that reduces the run-time and improves the performance and speed of the system.
In our proposed design, we use different parts such as machine learning unit, image processing unit and so on, to identify people and detect objects. Therefore, FPGA is the best choice to perform tasks in parallel. Our project is based on Neural Network and requires real-time object detection and face recognition. So, using FPGA kit allows us to control of parallelism and pipeline of the board for energy-efficient CNN implementation. Also, compared with any other microcontroller, FPGA offers better performance, flexibility, low latency and cost to design our smart glasses [2, 3].
Because of using DE10-Nano FPGA kit, we have a high-bandwidth to send frames to the AI unit, and also, the system can process images at high-speed and make real-time decision as a result, cause to reduce the response time. Main advantages of using FPGA board in our design, is shown in figure (4).
Fig.4. Advantages of using FPGA
The world Health Organization (WHO) reported that there are 285 million visually-impaired people in the world. Among these people, 39 million who are blind, and unfortunately, all these numbers are estimated to be doubled by 2020. The need for assistive devices for communicate with surrounding world has increased by visually-impaired people. All the systems, services, devices and appliances that are used by disabled people to help in their daily lives, make their activities easier, and provide a safe mobility .
In fact, the ideal device should satisfy the main and basic needs of these users. Since the technology is in advance every day, we proposed a smart wearable device based on Artificial Intelligence (AI). Between different algorithms of machine learning in AI, we use convolutional neural network (CNN) due to its effectiveness in object detection and image recognition.
On the other hand, despite of different devices to help blind people, existence of a high-speed, low power and low cost device with a high flexibility is necessary to improve their quality of life. Because of this, we choose Field Programmable Gate Array (FPGA) as a platform to implement our proposed smart glasses. In fact, FPGA is widely considered as a promising platform for convolutional neural network (CNN) acceleration.
In this design, we focused to detect objects, especially in streets like cars and identify familiar person, as one of the biggest problems for the blind people.
As mentioned in the previous section, we proposed a smart glasses for the blind people to help them navigate easily and communicate with their surrounding environment. By using of this device, the blind user can understand dangers without others help, especially when he/she is near the street or cross the roads. So, this design can detect important thing of outdoor like car, crosswalk and traffic light, then, based on the result of detection, announces to the user.
The structure of a typical object detection algorithm and an example is shown in figure (5) and figure (6), respectively.
Fig.5. Structure of a typical object detection algorithm 
Fig.6. Example of object detection
The main difference of our design with others is that we use a database of individuals that are familiar for blind users. According to this, when the neural network detects the human, searches in its database of pictures, then, if the captured image is matched, it says ‘name of the detected person’ to the user; On the other hand, it says ‘unfamiliar’. The procedure of face recognition in our design is shown in figure (7).