Annual: 2019

AS003 »
A Mind Reading and Intervention System
📁Machine Learning
👤Hong Lin
 (University of Houston-Downtown)
📅Jun 28, 2019
Regional Final

41


👀 584   💬 4

AS003 » A Mind Reading and Intervention System

Description

Existing physiological reading systems, e.g., those used in patient monitoring, are ineffective for daily practices. This project aims to simulate an environment for daily physiological tracking using a FPGA DE10 board with physiological sensors. On a single board computer, we will an affordable reusable, expandable, and wireless, machine that can monitor a user’s temperature, ECG heart activity and EEG brain waves. With the integrated sensors and coding, the device should be capable of live streaming and exporting collected data on a local web server for rendering.
For the collection of EEG brain waves signals, a hand-made headset will be connected to the FPGA board via a Bluetooth module. Data will be sent and collected from the EEG headset using a Python Library.
The hosting microcomputer will be made capable of configuring and programming the required html, PHP, and python files. Overall, the data rendering software simulates a professional medical interface and is available to both the mobile devices and internet browser. The user should be able to connect the device to a remote server via the internet wirelessly, attach the reusable sensors to his/her body, and download the information gathered.
This project aims to challenge the affordability and accessibility of existing healthcare oriented monitoring equipment. From this point on, a system with the ability to collect data, and perform machine learning tasks based on the collected data is desirable. Ideally, with the computational power provided by the remote server, such a system will be able to diagnose the user’s mental states based on the knowledge gathered with the machine learning power and the organized data collection and processing.
A virtual reality system on mobile phone will be connected to the DE10 board to render mind intervention activities based on the diagnosis of the user’s mental state. Continual diagnosis and intervention will be studied to find the best routine for certain type of mental health problems.
The FPGA DE10 board will exert its power in this project, especially in the stages of machine learning for brain state recognition and rendering of virtual reality scenes. Those machine learning and virtual reality tasks will be handled using packages that run on full-fledged operating systems supported by FPGA DE10.

Demo Video

  • URL: https://youtu.be/_G4AlaXKspY

  • Project Proposal

    1. High-level Project Description

    Existing physiological reading systems, e.g., those used in patient monitoring, are ineffective for daily practices. This project aims to simulate an environment for daily physiological tracking using a FPGA DE10 board with physiological sensors. On a single board computer, we implement an affordable, reusable, expandable, and wireless, machine that can monitor a user's EEG brain waves.  With the integrated sensors and coding, the device is capable of live streaming collected data, performing data processing and modeling, and exporting brain state data to a mobile app for mental intervention. Figure 1 shows the system architecture.

    For the collection of EEG brain waves signals, a hand-made headset is connected to the mobile phone via a Bluetooth module. Data are collected from the EEG headset and sent to the FPGA board via the mobile phone. The design of the EEG headset is shown in Figure 2.

    The FPGA microcomputer is made capable of configuring and programming the required python files for EEG data preprocessing, feature extraction, and brain state classification using a deep learning model (See Figure 3 for the data processing and modeling flowchart). The data rendering software on the mobile phone simulates a professional medical interface through which the user is able to connect the EEG device to the phone, relay the EEG data to the FPGA board, and render the brain state curve on the phone (See Figure 4 for the snapshot of the app) . 

    This project aims to challenge the affordability and accessibility of existing healthcare oriented monitoring equipment.  With the computational power provided by FPGA, such a system is able to diagnose the user's mental states based on the knowledge gathered with the machine learning power and the organized data collection and processing, and to render mind intervention activities based on the diagnosis of the user's mental state. Continual diagnosis and intervention can be studied to find the best routine for certain type of mental health problems. In this project, we implement a sleep inducing program that can help the user to get into sleep more easily and deeply.

    The sleep inducing technique used in this project is taking the advantage of Delta waves. Scientists have proven that the 1HZ Delta waves (Figure 5) can be converted to sounds and those sounds have the effect of deepening a person's sleep. Our system monitors the user's sleep state and adjust the volume of Delta wave sounds together with a user chosen background music according to the user's sleep depth.

    The FPGA DE10 board exerts its power in this project, especially in the stages of machine learning for brain state recognition and rendering of brain state information. FPGA DE10 enables a portable and onsite machine learning system while maintaining sufficient computational power to run packages on a full-fledged operating system. In particular, the data processing method used in our project, the emsemble empirical mode decomposition (EEMD), has high time and space complexity. Thereafter, our system extracts 20 features, including power spectral intensities (PSIs), relative intensity ratios PSIs (RIR PSIs), Petrosian Fractal Dimension, Higuchi Fractal Dimension, Hjorth Parameters, Spectral Entropy, SDV Entropy, Fisher Information, Approximate Entropy, Detrended Fluctuation Analysis, and Hurst Exponent. FPGA is the probably the only feasible scheme for implementing such an onsite real-time system with sufficient computational power.

    Our sleep mental intervention system can be used by general population who have a need for improved sleep quality, and patients in rehabilitation centers or hospitals who want to adopt a non-medication approach for sleep disorders and insomnia.

     

    2. Block Diagram

    3. Intel FPGA virtues in Your Project

    We aims to design a portable device that provides onsite real-time functionality for our mental intervention services. The FPGA DE10 board is the best fit to in this project, for its portability and computational power. Both the data processing method used in our project, the emsemble empirical mode decomposition (EEMD), and the feature extraction methods have high time and space complexity. EEMD method decomposes the raw EEG waves into a set of Intrinsic Mode Functions (IMFs), which can be used to remove low frequency drift artifacts. IMFs can also be used directly as input data to the following data processing and analysis stages. Our system extracts 20 features, including power spectral intensities (PSIs), relative intensity ratios PSIs (RIR PSIs), Petrosian Fractal Dimension, Higuchi Fractal Dimension, Hjorth Parameters, Spectral Entropy, SDV Entropy, Fisher Information, Approximate Entropy, Detrended Fluctuation Analysis, and Hurst Exponent. 

    FPGA is capable of implementing such an onsite real-time system with sufficient computational power. Figure 6 illustrates a service log that shows the timestamps (in seconds) of the server's responses to EEG data uploading in every second. Our EEG acquisition device sends 128 samples of EEG reading every time to the server. It reads a sample whenever it detects voltage reading changes. While slow brain activities entails longer data collection time for 128 samples, statistically, it collects 128 samples in every second. Figure 6 indicates that the server's response time to data uploading allows real-time processing of input data.

    4. Design Introduction

    Existing physiological reading systems, e.g., those used in patient monitoring, are ineffective for daily practices. This project aims to simulate an environment for daily physiological tracking using a FPGA DE10 board with physiological sensors. On a single board computer, we implement an affordable, reusable, expandable, and wireless, machine that can monitor a user's EEG brain waves.  With the integrated sensors and coding, the device is capable of live streaming collected data, performing data processing and modeling, and exporting brain state data to a mobile app for mental intervention. Figure 1 shows the system architecture.

    For the collection of EEG brain waves signals, a hand-made headset is connected to the mobile phone via a Bluetooth module. Data are collected from the EEG headset and sent to the FPGA board via the mobile phone. The design of the EEG headset is shown in Figure 2.

    The FPGA microcomputer is made capable of configuring and programming the required python files for EEG data preprocessing, feature extraction, and brain state classification using a deep learning model (See Figure 3 for the data processing and modeling flowchart). The data rendering software on the mobile phone simulates a professional medical interface through which the user is able to connect the EEG device to the phone, relay the EEG data to the FPGA board, and render the brain state curve on the phone (See Figure 4 for the snapshot of the app). 

    This project aims to challenge the affordability and accessibility of existing healthcare oriented monitoring equipment.  With the computational power provided by FPGA, such a system is able to diagnose the user's mental states based on the knowledge gathered with the machine learning power and the organized data collection and processing, and to render mind intervention activities based on the diagnosis of the user's mental state. Continual diagnosis and intervention can be studied to find the best routine for certain type of mental health problems. In this project, we implement a sleep inducing program that can help the user to get into sleep more easily and deeply.

    The sleep inducing technique used in this project is taking the advantage of Delta waves. Scientists have proven that the 1HZ Delta waves (Figure 5) can be converted to sounds and those sounds have the effect of deepening a person's sleep. Our system monitors the user's sleep state and adjust the volume of Delta wave sounds together with a user chosen background music according to the user's sleep depth.

    The FPGA DE10 board exerts its power in this project, especially in the stages of machine learning for brain state recognition and rendering of brain state information. FPGA DE10 enables a portable and onsite machine learning system while maintaining sufficient computational power to run packages on a full-fledged operating system. In particular, the data processing method used in our project, the emsemble empirical mode decomposition (EEMD), has high time and space complexity. Thereafter, our system extracts 20 features, including power spectral intensities (PSIs), relative intensity ratios PSIs (RIR PSIs), Petrosian Fractal Dimension, Higuchi Fractal Dimension, Hjorth Parameters, Spectral Entropy, SDV Entropy, Fisher Information, Approximate Entropy, Detrended Fluctuation Analysis, and Hurst Exponent.FPGA is the probably the only feasible scheme for implementing such an onsite real-time system with sufficient computational power.

    Our sleep mental intervention system can be used by general population who have a need for improved sleep quality, and patients in rehabilitation centers or hospitals who want to adopt a non-medication approach for sleep disorders and insomnia.

    5. Function Description

    EEG headset:

    The BBC micro:bit is a pocket-sized, programmable computer, which is turned into a galvanic skin response (GSR) device. A program written in MicroPython was created to test the micro:bit onboard 3.3 V power supply. The board reads the signal in a default analog range of 0 to 1023. The signal was converted to volts by assigning the analog value of 1023 to 3.3 V. The formula used to convert analog reading to volts was: Vout = Vmax × AO / AOmax, where

    • Vout is the output voltage, measured in volts (V)
    • AO is the numeric analog output from the Micro:Bit
    • Vmax is the maximum voltage of 3.3 V
    • AOmax is the maximum analog value of 1023

    A voltage divider (Figure 2) using two resistors was built to lower the 3.3 V output to the standard GSR voltage of roughly 0.5 V. The resistors R1 and R2 were 270K Ω and 56K Ω, respectively. The equation used to determine the resisted voltage output: Vout = Vin × R2 / (R1 + R2), where

    • Vin is the source voltage, measured in volts (V)
    • R1 is the resistance of the 1st resistor, measured in Ohms (Ω)
    • R2 is the resistance of the 2nd resistor, measured in Ohms (Ω)
    • Vout is the output voltage, measured in volts (V)

    Two electrodes were created using copper wires and placed on the forehead of the subject (Figure 2). The micro:bit is connected to a mobile phone via blue tooth.

     

    Mobile app:

    The mobile app reads EEG data from the headset and sends every 128 readings to the server on the FPGA board. It then receives server responses to the EEG data uploading and render them in a sleep depth curve. The server responds to each EEG uploading with probabilities of the corresponding brain state being “awake”, “light sleep”, and “deep sleep”.

    The mobile app uses the 1HZ Delta waves to generate sounds as intervention means to induce sleep. Our system also uses meditation to help the user relax and get into a tranquil state. The app asks the user to lie in an “Auspicious Lying Poise”, which is believed to be a poise with mind-body goodness. The user has the chance to choose a music to be played while he/she is trying to get into sleep. The music is branded with the Delta waves and the volume is adjusted according to the depth of the sleep state.

     

    EEG data processing and analysis:

    This is the most challenging and time-consuming part of the system. For every 128-sample uploading, the FPGA board uses the procedure depicted in Figure 3 to process the data and analyze the data to evaluate the sleep depth based on predictive machine learning models.

    Empirical Mode Decomposition (EMD) transforms wave forms into a series of components called Intrinsic Mode Functions (IMFs). Some recording artifacts such as low frequency drift can be identified by examining the IMFs. Ensemble Empirical Mode Decomposition (EEMD) is a noise-assisted method to improve shifting and generate a better EEG data from a selected set of IMFs. Low frequency drift can be removed by eliminating IMFs that show consistent increasing/decreasing tendency. We then either compute the summary of remaining IMFs as the output of this process, or use the remaining IMFs directly as features in the steps onward.

    Feature extraction is computed based on an open source Python module “PyEEG”. Particularly, Fast Fourier transform was used to extract the PSI, such as delta (0.5-4Hz), theta (4-7Hz), alpha (8-12Hz), beta (12-30Hz), and gamma (30-100Hz). RIR computed from PSIs were deltaRIR, thetaRIR, alphaRIR, betaRIR, and gammaRIR. Though Neurosky data also used Fast Fourier transformation to transform its data, the range of frequency is just in the interval [0.5Hz, 50Hz]. Other features also extracted were PFD, HFD, Hjorth Parameters, Spectral Entropy, SDV Entropy, Fisher Information, Approximate Entropy, Detrended Fluctuation Analysis, and Hurst Exponent.

    The effectiveness of using EMD and features mentioned above has been proven in experiments. We compared the model accuracy across different combinations of data processing the feature extractions. Herein, we use term “filtered data” to refer to the data obtained by filtering the raw EEG data by EEMD, with the low frequency IMFs being eliminated and remaining IMFs combined; term “extracted data” the features extracted from either directly from the raw EEG data or filtered data after EEMD; “decomposed data” the data obtained by EEMD, with the low frequency IMFs being eliminated and remaining IMFs used individually (without combination) in the following process;  “normalized data” the data or features normalized by the min-max normalization. We compared the results from 4 cases. In case 1, raw data were filtered, extracted, and normalized prior to being used to train the model; in case 2, raw data were extracted and normalized prior to being used to train the model; in case 3, raw data were decomposed and normalized prior to being used to train the model; in case 4, raw data were filtered and normalized before used to train the model. Figure 7 (left) shows the 4 cases of experiments and 9 (right) the accuracy of the predictions made by the models trained by different machine learning methods for each corresponding case.

    The experimental results support the importance of feature extraction in obtaining good performance from the model. Fourier Fast Transformation and other features such as PFD, HFD, Hjorth Parameters, Spectral Entropy, SDV Entropy, Fisher Information, Approximate Entropy, DFA, and Hurst Exponent all contribute to the very good performance of models. The results also verify that the low frequency drifts in the raw data affect the model accuracy. As part of the measuring artifacts, the low frequencies need to be removed from the input data, which is done after EEMD in our experiment. After removing low frequency drift, we can choose to combine the remaining IMFs to create filtered data, or use IMFs individually as processed data for further process and analysis. Our experiment supports that the latter derives better model performance.

    6. Performance Parameters

    We aims to design a portable device that provides onsite real-time functionality for our mental intervention services. The FPGA DE10 board is the best fit to in this project, for its portability and computational power. Both the data processing method used in our project, the emsemble empirical mode decomposition (EEMD), and the feature extraction methods have high time and space complexity. EEMD method decomposes the raw EEG waves into a set of Intrinsic Mode Functions (IMFs), which can be used to remove low frequency drift artifacts. IMFs can also be used directly as input data to the following data processing and analysis stages. Our system extracts 20 features, including power spectral intensities (PSIs), relative intensity ratios PSIs (RIR PSIs), Petrosian Fractal Dimension, Higuchi Fractal Dimension, Hjorth Parameters, Spectral Entropy, SDV Entropy, Fisher Information, Approximate Entropy, Detrended Fluctuation Analysis, and Hurst Exponent. 

    FPGA is capable of implementing such an onsite real-time system with sufficient computational power. Figure 6 illustrates a service log that shows the timestamps (in seconds) of the server's responses to EEG data uploading in every second. Our EEG acquisition device sends 128 samples of EEG reading every time to the server. It reads a sample whenever it detects voltage reading changes. While slow brain activities entails longer data collection time for 128 samples, statistically, it collects 128 samples in every second. Figure 6 indicates that the server's response time to data uploading allows real-time processing of input data.

     

    7. Design Architecture

    The design architecture, function modules, and software flowchart of our system are shown in Figure 8.



    4 Comments

    Luis Aljure
    excellent project Hong, keep going!!
    🕒 Jun 30, 2019 08:48 PM
    AS003🗸
    Many thanks!
    🕒 Jul 01, 2019 12:56 AM
    Mandy Lei
    Please upload your design Block Diagram and add the detailed contents of the Intel FPGA virtues in Your Project.
    🕒 Jun 26, 2019 03:32 AM
    AS003🗸
    Thanks for your reminder! We uploaded the required materials.
    🕒 Jul 01, 2019 12:57 AM

    Please login to post a comment.