What problem: Our mental health is as important as our physical health. To evaluate a person's overall behaviour, especially for the elderly and the differently-abled people, it is necessary to accurately analyse both physical and mental behaviour. Through the various camera, inertial and location sensors, it is possible to build a complete model which is able to describe the overall behaviour of human beings with details including emotional, physical and environmental aspects. In this project, we aim to address this issue and try to solve it. Furthermore, we will incorporate the advantage of edge computing and build a near-sensor solution that can monitor the overall well being of an individual.

What and how: In this project, we approach solving the problem by designing an efficient Deep Neural Network architecture.
First, in order to perform emotion analysis and correctly classify various human emotions of sadness, happiness or anger, a Convolutional Neural Network (CNN) based model would be designed. Information from camera sensors would be fed into this CNN model, which would correctly classify human emotions.
Next, we would design a hybrid network that employs the advantages of spatially deep CNN and temporally deep Long Short Term Memory (LSTM). This hybrid CNN-LSTM network would be used for analysing various complex human activities including walking, sitting and lying down - all of which would be collected using inertial sensors attached to the human body.
Lastly, a final deep neural network (DNN) model would be designed, which would combine the processed information of an individual's emotional analysis and activity recognition. This NN model would be crucial to accurately analyse the overall well being of the individual, for example, whether the person is having a problem while walking or not. When an individual is found to be in any distress - physical or mental, the responsible caregiver would be alerted by a signal or a message sent to him/her.

Today, a majority of the DNNs are implemented on the cloud. This design will remove any drawbacks associated with communication links to the cloud and will use the benefits of edge computing. The concepts of stochastic computing (SC) will be used in order to reduce the hardware resource utilisation of the DNN implementation. SC also reduces the complexity of the individual operations of the neural networks. For example, a multiplier can be implemented by only an AND gate or an XNOR gate and an adder can be implemented by a MUX in SC. This significantly improves this implementation and makes it ideal for edge computing.

Today, a majority of the DNNs are implemented on the cloud. Our design will remove any drawbacks associated with communication links to the cloud and will use the benefits of edge computing.
According to our knowledge, there is no existing autonomous solution that can accurately monitor the overall well being of an individual in their daily lives. In our project, we will design a DNN model which can compute and analyse the information near the sensor and not depend on cloud computing. Our model will be extremely useful in monitoring the health of any individual - especially the elderly and the differently-abled. They would be under constant supervision even if the designated caregiver is not physically present right next to the individual.

The prebuilt face and object detection models will be used . We would use the smart camera and these prebuilt algorithms would help us to correctly analyse the emotions and locality of the individual under supervision. Another advantage of using the provided kit with the FPGA board is to be able to parallekize our CNN and hybrid CNN-LSTM models, as there is no data dependency between them. The video processing based emotion analysis can be done much faster compared to CPU or GPU implementations by using the concepts of pipelining. The SD card would be loaded with sensor signal data and these data would then be processed by the neural networks. The CNN model will be used to process the data from the camera sensor and analyse the video frames and detect people, their emotions and also their locality. The inertial sensor data will be fed to the CNN- LSTM model that will correctly analyse human activities. The final DNN would then use the already processed information and provide a complete description combination for our models to run faster and better.

Project Proposal

1. High-level project introduction and performance expectation

2. Block Diagram

3. Expected sustainability results, projected resource savings