Mental and behavioural problems are increasing part of the health problems the world over. The burden of illness resulting from psychiatric and behavioural disorders is enormous. Although it remains grossly under-represented by conventional public health statistics, which focus on mortality rather than morbidity or dysfunction. The psychiatric disorders account for 5 of 10 leading causes of disability as measured by years lived with a disability. The overall DALYs burden for neuropsychiatric disorders increased to 15% in the year 2020. At the international level, mental health is receiving increasing importance as reflected by the WHO focus on mental health as the theme for the World Health Day (4th October 2001), World Health Assembly (15th May 2001) and the World Health Report 2001 with Mental Health as the focus.

Let’s consider Covid-19 pandemic situation, opportunities to monitor psychosocial needs and deliver support during direct patient encounters in clinical practice are greatly curtailed in this crisis by large-scale home confinement. Psychosocial services, which are increasingly delivered in primary care settings, are being offered by means of telemedicine. In the context of Covid 19, psychosocial assessment and monitoring should include queries about Covid-19 related stressors (such as exposures to infected sources, infected family members, loss of loved ones, and physical distancing), secondary adversities (economic loss, for example), psychosocial effects (such as depression, anxiety, psychosomatic preoccupations, insomnia, increased substance use, and domestic violence), and indicators of vulnerability (such as pre existing physical or psychological conditions). Some patients will need referral for formal mental health evaluation and care, while others may benefit from supportive interventions designed to promote wellness and enhance coping (such as psychoeducation or cognitive behavioral techniques). In light of the widening economic crisis and numerous uncertainties surrounding this pandemic, suicidal ideation may emerge and necessitate immediate consultation with a mental health professional or referral for possible emergency psychiatric hospitalization.

We believe that Big Problems have Simple Solutions. If someone is always monitoring you and giving you timely recommendations/suggestions it can help in a very positive way. In this project we are trying to solve this problem with a lot of positivity which is actually very important in anyone's life. Our project consists of a device connected to human body which consists of sensors which monitor different parameters from human body, gas sensor inside the living room to monitor the pollution/toxic gases in the environment, a recommender system which uses the captured images of human/video and uses ML algorithms uses Cloud and Cloud connectivity kit to monitor human behavior. Data from sensors and machine learning algorithms are put together and timely recommendation is given to the people.

Demo Video

Project Proposal

1. High-level project introduction and performance expectation

Mental health is a widely discussed topic in today's times. It is the need of the hour. Mental health is not just a mere absence of mental illnesses, but also includes many aspects of an individual and his/her life. It encompasses all the aspects of cognitive, social, emotional and behavioral well-being. Thus a better mental health leads to a better daily functioning, better relationships and a sense of life satisfaction. Mental health and physical health are closely connected. Mental health plays a major role in people's ability to maintain good physical health. Mental illnesses, such as depression and anxiety, affect people's ability to participate in health-promoting behaviors. In this project we hope to provide a technological solution to promote mental health and wellbeing with the help of an SoC FPGA, sensors, internet data and the Cloud.

This idea of ours mainly consists of three working units. The first unit is the data acquisition unit, consisting of sensors, web scrapers and user input interfaces. Data is quintessential to this project and data will be collected on as many relevant parameters as possible, for example, speech, behaviour and facial images of the patient, ambient light intensity, temperature, air quality, blood pressure, pulse rate, internet usage data etc. The second part is a Terasic DE10-Nano FPGA Cloud Connectivity Kit which will behave as the 'brains' of the entire operation, providing cloud connectivity and doing computations. The DE-10 Nano kit will also be responsible for consolidating data from the Analog Devices plug-in cards as well as other sensors and data sources. The third unit is Microsoft Azure Cloud which provides remote control, monitoring and data storage facilities. Machine learning models may also be trained and deployed using Azure. Data is to be analyzed carefully to classify symptoms of various targeted mental health conditions and the parameters most relevant to this classification process are to be zeroed-in upon. Based on user health and mental state, timely positive suggestions are to be given using a recommendation system. This helps patients by detecting issues in lifestyle or environment of the patient . Neuroscientists and clinicians around the world are using machine learning to develop treatment plans for potential patients and to identify some of the key markers for mental health disorders before they may set in. One of the benefits is that machine learning helps clinicians predict who may be at risk of a particular disorder. What makes machine learning so helpful today is that in the past, the understanding of diagnoses was based on group averages and statistics over populations. Machine learning gives clinicians the opportunity to personalize diagnosis for each individual. In this way we expect this project to be very useful because lots of problems related to mental well-being are neglected. Our project is a positive step towards sustainable mental health.

The scope of this project is vast as mental health issues are a huge problem but are often neglected. If there is someone, or in this case, something which is there to monitor you and give positive suggestions, a person struggling with mental health issues may be greatly helped. Doctors can use these systems to monitor the patient, the data acquired in the cloud can be used for further analysis and research purposes. The data will also be used to fine-tune the machine learning model for prediction and classification of mental health disorders, and even to improve the lifestyle of people without any mental health disorders. We also expect accurate data readings from ADC pins which will be used for monitoring patients biomarkers and environmental parameters accurately.

2. Block Diagram

Figure 1: High Level Block Diagram

3. Expected sustainability results, projected resource savings

● UN's Sustainability Goal 3: Good Health and Well Being

● Increase in individual mental wellbeing leads to physical well being which inturn leads to mental and physical well being of society and enabling a sustainable life.

● Manasik Swasthya Yantram as a product.

● Reduces workload, stress, anxiety, depression etc.

● Eases the work of a doctor and saves a lot of time.

● Continuously monitors patients which is not possible by a doctor.

● It is quite a cost effective way to monitor people, among most of the IOT devices DE10 nano allows FPGA and Embedded system properties together which allows to run ML algorithms smoothly and interface sensors easily.

● Data acquired allows detailed research of the human mind, this data can be used for different analysis purposes.

● It saves a lot of resources for testing, counselling in hospitals.

● It allows clinicians the opportunity to personalize.

Some research papers we referred are,

[1] Mental health-related stigma in health care and mental health-care settings Claire Henderson, Jo Noblett, Hannah Parke, Sarah Clement, Alison Caffrey, Oliver Gale-Grant, Beate Schulze, Benjamin Druss, Graham Thornicroft

[2] A new healthcare diagnosis system using an IoT-based fuzzy classifier with FPGA Sambit Satpathy, Prakash Mohan, Sanchali Das, Swapan Debbarma

[3] Real-time data analysis for medical diagnosis using FPGA-accelerated neural networks Ahmed Sanaullah, Chen Yang, Martin C. Herbordt

[4] FPGA-based system for continuous monitoring of three vital signs of human body Aminuddin Rizal, M. Riyadi, Darjat (ICITACEE)

4. Design Introduction

Our design focuses on building a system that can monitor a person's mental health. Mental Health being an integral part of any human-being needs to be taken care of. We, through our project want to prove that physical health conditions of a person and the environmental conditions that he is surrounded with, have an unsaid and profound impact on his mental well being. Thus, we in our project have successfully shown the underlying link that exists between these parameters. 

In a world that is expanding in technology, and post the Covid-19 pandemic which has certainly changed the way we live life, has definitely opened the gates to opportunities for us, as the problem that we have tried to solve through this project is real and significant. The application scope is wide.

The new normal of living life has paved the path for the deterioration of our mental health. Our project can definitely aid the connection between people suffering from mental health conditions like depression and anxiety with doctors treating them. We also provide recommendations to people suffering from depression and anxiety through recommender system.

5. Functional description and implementation

As shown in the high level block diagram in section 2, our proposed model Manasik Swasthya Yantram consists of 3 data acquisition units namely, Glove; which contains various sensors like Temperature, GSR, Accelerometer, LDR and Air quality sensors, Camera; which does the continuous monitoring of the user. Mobile phone; which does the user survey, also provides "Wear the glove" instruction on continuous detection of sadness, anxiety through video monitoring and gives user interface for providing recommendations to the user.

Working: Camera is fixed in the living room, on continuous detection of traits like sadness and anxiety using CNN ML model with high confidence, "Wear the glove" message is sent. Person wears the glove and sensor parameters are recorded. Sensor data goes through the LSTM ML model, providing prediction if the person is stressed or not. The predictions from Sensor model and Video model are correlated and mental disorder is predicted. Based on predictions, timely recommendation is given to the user and in severe conditions a doctor could be notified. Detailed description available below under Hardware, Software parts and Correlation, Classification, Mental Health Disorder Prediction and Recommendation sections.

MANASIK SWASTHYA YANTRAM

SOC Based Mental Health Monitoring and Recommender System

Figure 2: Working prototype using DE10 NANO and Sensors, Camera for monitoring the user and Mobile phone

Figure 3: Final Glove Prototype

Hardware Part

Sensor Integration

This is the core of our project where the sensor integration, namely, GSR sensor, Temperature Sensor, Ambient light LDR sensor, Air quality sensors and onboard accelerometer integration is done. Sensors are connected to the 2X5 ADC headers onboard. SPI protocol is used for reading sensor values where DE10 Nano acts as Master and sensors are Slaves. The sensors are fixed in appropriate positions on the glove to obtain accurate reading. These readings are used as test data for the Sensor ML model in the software part.

Figure 4: Sensor integration using 2X5 ADC header

HPS for automation and cloud connectivity 

ADC sensor values can be accessed through SDCard booted linux, which makes the MSY an independent system which can program itself and flash the FPGA SDRAM automatically. The same linux provides Azure cloud connectivity.

Custom PCB for connecting sensors to ADC

DE10 Nano ADC header contains single VCC and Gnd pins. 4 sensors that are used with the ADC can't be connected at once, a simple custom PCB provides common VCC and Gnd for each sensor.

Figure 5: Custom PCB

Software Part

Sensor ML Model

WESAD dataset^[1] is used for modeling sensor ML model. Accelerometer, Temperature and EDA data are used for the model. The standard deviation, mean, maximum and minimum values of each trait are taken into consideration to build the LSTM model. Live stress detection using sensor data uses the trained LSTM model.

Figure 6: Sensor LSTM model

Video Emotion Recognition CNN Model

Video dataset is collected from social media platforms like youtube and stock videos are used as dataset for the video emotion recognition model. Data collected contains 5 traits namely, Angry, Anxiety, Happy, Neutral and Sad which is converted to Gray scale initially which goes into the CNN model. Real time emotion detection uses the trained CNN model.

Figure 7: Video Emotion Detection CNN model

User Survey

The user survey consists of the PHQ-9( Patient Health Questionnaire) assessment which is recommended by DSM-5(Diagnostic and Statistical Manual for Mental Disorders) to measure the severity of mental disorders like Anxiety and Depression and also to monitor the progress of clinical treatment. This assessment is found to be of immense help in characterizing patient status and improving patient care. For Depression and Anxiety, a set of 9 and 10 questions are to be answered by the user.

Figure 8: User Survey, Questionnaire with the user

Correlation, Classification, Mental Health Disorder Prediction and Recommendation

Classification, Correlation and Disorder Prediction

Video emotion recognition part gives classification in 5 traits. Real time stress monitoring gives binary results, stressed and not stressed based on the number of times the model gives 1 and 0 values. 1's concentration is more, the person is stressed else not stressed. Correlation and mental disorder prediction is done using the below shown table.

​​​​​Figure 9: Classification of emotion into 5 traits, correlation with sensor model based on 1's concentration and mental health prediction.

Recommendations

The recommendation system consists of research based solutions for people suffering from depression and anxiety. The recommendations have been primarily drawn from NIMH (National Institute of Mental Health - one of the leading agencies for research on mental disorders) and NHS (National Health Service - one of the largest healthcare systems in the world).

​​Figure 10: Recommendations to users based on correlation from video, sensor predictions. Anxiety, Moderately Anxious, Highly Anxious, Depressed, Moderately Depressed and Highly Depressed categories.

6. Performance metrics, performance to expectation

1) Resource Consumption

The sensor interfacing model uses < 25 % of memory and total pins on the FPGA board, uses only 3 DSP blocks and logic utilization is only 4 %. Considering the power consumption, overheating and effect on sensor values (especially temperature sensor), the resource consumption is ideal (Result was unexpected but in a positive way).

Figure 11: Resource consumption

Figure 12: RTL

2) Sensor Readings from the Glove

Channel voltage values from the ADC pins are converted to real sensor values using appropriate conversion formulas. Values from the sensors are verified against the real values and plotted. 

Figure 13: Sensor values on putty terminal window

Figure 14: Temperature and EDA sensor values plotted

3) Video Emotion Recognition ML Model - Accuracy with validation data

The video emotion recognition using CNN gives 94 % accuracy after 25 epochs. With real time video we expect upto 85 % accuracy.

Figure 15: Video emotion detection ML model accuracy after 25 epochs

4) Video Emotion Recognition ML Model - Confidence while real time monitoring

Real time video emotion recognition provides confidence upto 85 % even with low quality video with 5 traits (Angry, Anxiety, Happy, Neutral, Sad) recognised.

Figure 16: Real time video emotion recognition

5) Sensor ML Model - Accuracy with validation data

The sensor ML model using LSTM gives upto 98 % accuracy (for Temp, GSR and Acc) with the validation data from WESAD dataset.

Figure 17: Sensor ML model accuracy with validation data

6) Sensor ML Model - Prediction of mood while real time monitoring

Real time stress monitoring gives binary results, stressed and not stressed based on the number of times the model gives 1 and 0 values. Based on the percentage values of 1's and 0's mood can be detected. If the number of 1's is high, again based on percentage value, the user is classified as stressed, moderately stressed and highly stressed. And if the 0 concentration is very high, again based on the percentage value, the user is classified as depressed, moderately depressed and highly depressed.

Here in the below figure the person is normal based on 1's concentration.

Figure 18: Real time stress prediction

7) Correlation

By plotting the values, correlation between parameters acceleration, body temperature and electrodermal activity is observable and proved by sensor ML model.

Figure 19: Correlation of temperature, eda and accelerometer values

Summary of performance to expectation

Figure 20: Summary of performance metrics and expectation

7. Sustainability results, resource savings achieved

• UN SDG Goal 3: Good Health and Well Being 

• WHO - World Health Assembly - Promoting well being at all ages

• The relationship between mental health and sustainable development

• Systematic integration of Positive Psychology Interventions(PPIs)

• Give people ability to adapt to avoid mental illness and go towards mental well being.

• Reduce suicides, the final stage of mental health issue

• Ability to provide high value care  in the face of environmental, economic and social constraints.

Figure 21: MSY

● As responsible world citizen, only people with good mental well-being and physical well- being can contribute effectively to the society and environment. In this direction our project aims to build a society where mental health is given equal importance like physical health.

● "Manasik Swasthya Yantram - MSY" as a technological solution for mental health enabled with DE10 Nano.

● Less resource consumption on FPGA.

● Low power usage.

● It saves a lot of resources for testing, counselling in hospitals.

● Improvisation of machine learning models with more training and test validation.

● Eases the work of a doctor and saves a lot of time.

● Continuously monitors patients which is not possible by a doctor.

● User need not wear glove all the time.

● Data acquired allows detailed research of the human mind, this data can be used for different analysis purposes.

Future works

● Addition of more traits like pulse, heartbeat, EMG etc.

8. Conclusion

Mental health is equally important as physical health. Manasik Swasthya Yantram (MSY) is a step towards "Sustainable Mental Health" enabled with DE10 Nano SoC FPGA. MSY can make a big difference in the life of people who often neglect their state of mental health. MSY is easy to use. Our target users involve Businessmen, Engineers, Blue collar workers, Doctors, Multinational companies etc. This makes MSY relevant in the present day society. Especially after COVID 19 pandemic, MSY has increased its relevance.

References

^[1] Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger and Kristof Van Laerhoven. 2018. Introducing WESAD, a multimodal dataset for Wearable Stress and Affect Detection. In 2018 International Conference on Multimodal Interaction (ICMI ’18), October 16–20, 2018, Boulder, CO, USA. ACM, New York, NY, USA, 9 pages.

^[2] Mental health-related stigma in health care and mental health-care settings Claire Henderson, Jo Noblett, Hannah Parke, Sarah Clement, Alison Caffrey, Oliver Gale-Grant, Beate Schulze, Benjamin Druss, Graham Thornicroft

^[3] A new healthcare diagnosis system using an IoT-based fuzzy classifier with FPGA Sambit Satpathy, Prakash Mohan, Sanchali Das, Swapan Debbarma

^[4] Real-time data analysis for medical diagnosis using FPGA-accelerated neural networks Ahmed Sanaullah, Chen Yang, Martin C. Herbordt

^[5] FPGA-based system for continuous monitoring of three vital signs of human body Aminuddin Rizal, M. Riyadi, Darjat (ICITACEE)

^[6] A Mobile Mental Health Monitoring System: A Smart Glove, L. Fraiwan, Tasnim Basmaji, O. Hassanin