AS015 » Flex Force Smart Glove for Measuring Sensorimotor Stimulation
Each year approximately 795,000 people suffer from stroke which makes it the leading cause of permanent disability in the country. Research have shown that the human brain is capable of self-reorganizing especially after limb stimulation is employed resulting in re-establishment of neural pathways that control volitional movement. Traditional physical therapy which involves one-on-one interaction with a therapist is a conventional method used for stimulating sensorimotor activities, while robotic-assisted rehabilitation can increase the effectiveness of the repetitive exercises used in rehabilitation. Furthermore, to properly quantify the effectiveness of both conventional and robotic-assisted rehabilitation, sensorimotor measurements need to be acquired and analyzed. Thus, the invention of the Flex Force Smart Glove (FFSG) allows for a complete nonintrusive design used to acquire and analyze sensorimotor information obtained from the human hand. The novel nonintrusive FFSG design will be powered by the Intel Altera FPGA and will incorporate all the sensors needed to measure the force and rotation of the human wrist and fingers. The device will provide the individual with a low-cost wearable smart glove to keep track of their rehabilitation process, which they can take home to facilitate further rehabilitation.
According to the 2013 update report of the American Heart Association (AHA) , each year approximately 795,000 people suffer from stroke, which makes it the leading cause of permanent disability in the country. Recent advances in development of robotic devices have demonstrated the feasibility and increased benefits of robotic-assisted rehabilitation for stroke-related or joint injuries of the upper extremities -. Research shows that the human brain is capable of self-reorganizing (i.e. plasticity), especially after limb stimulation is employed, resulting in re-establishment of neural pathways that control volitional movement. Traditional physical therapy, which involves one-on-one interaction with a therapist is a conventional method used for stimulating sensorimotor activities, while robotic-assisted rehabilitation can increase the effectiveness of the repetitive exercises used in rehabilitation. Furthermore, to properly quantify the effectiveness of both conventional and robotic-assisted upper-limb rehabilitation, sensorimotor measurements need to be acquired and analyzed. Thus, the invention of the Flex Force Smart Glove (FFSG) allows for a complete nonintrusive design used to acquire and analyze sensorimotor information obtained from the human hand.
Figure 1: Initial proof-of-concept of the Flex force device (image to left) that will utilize the Altera DE10-Nano Kit.
In this proposal, we will develop the FFSG prototype based on an Altera field-programmable-gate-array (FPGA). More specifically, our design will be utilizing the versatile DE10-Nano kit as illustrated in Figure 1. Altera FPGA’s allow for fast parallel signal processing of multiple stimulus in the digital domain that can also be re-programed to achieve different logic operations, which makes them very suitable for this application. The Altera FPGA will be the back-bone of the FFSG device and will be used for sensor processing, data fusion and Fugl-Meyer Assessments (FMA). FMA is metric used to assess the sensorimotor impairment in individuals who have had stroke, it includes scoring for motor function, sensory function, range of motion and more. The proposed FFSG device will incorporate all the sensors needed to measure the force and rotation of the human wrist along with the force and position of each finger.
The FFSG device can also be adopted as a mobile wearable device for other non-medical related applications. Some of these applications include, digital eyeglasses (i.e. Google glass), smart phones, smart watches, fashion design, augmented reality, pattern recognition, and much more. The FFSG is also a wearable device that will have applications in 3D spatial interfacing (i.e., controlling 3D objects), non-medical related robotics, gaming, and sports. As a 3D spatial interface, the FFSG device can be used to create, control, or even edit computer-aided design (CAD) models. Furthermore, the FFSG device can be used in robotic control applications by incorporating hand gestures. The FFSG device will also find its place in sporting applications such as, skiing, biking, golfing, and running.
Our project Github repository can be found here:
Figure 2: Illustration showing a schematic view of the Smart Glove design, which can be used to facilitate further rehabilitation in the clinic or at home.
A high-level block diagram of the FFSG design concept is shown in Figure 2. We will incorporate up to eight resistive/capacitive pressure and flex sensors that will be sampled by a multi-channel ADC and streamed via SPI to the Altera DE10-Nano. The pressure and flex sensors will be placed on key pressure points along the hand to measure force exerted during rehabilitation and exercises. We will also incorporate up to five inertial measurement unit (IMU) sensors that will measure the orientation of the fingers. The G sensor on the DE10-Nano will be used to measure the orientation of the hand/wrist. We will also incorporate a custom arduino shield with voltage divider circuits for interfacing with the pressure and force sensors. This shield will also contain an additional IMU for wrist sensing. These sensors will interface to the FPGA via SPI/I2C interfaces. The FFSG design is further extended to utilize a low-power wireless interface, i.e. Bluetooth, to allow for real-time monitoring of sensorimotor function at home or in the clinic. During clinical upper-limb rehabilitation, the FFSG will measure and acquire sensorimotor information from the human hand along with clinical results from commercial conventional and robotic-rehabilitation process. This will then provide the patient with real-time recovery information that they can take home with them for further rehabilitation exercises.
We believe that our project inherits all of the virtues of FPGAs, this includes (1) boosting performance, (2) adapting to changes and (3) expanding I/O.
Although the DE10-Nano gives us access to a dual-core ARM Cortex-A9 processor, we will be offloading key processing functions unto the FPGA such as implementing Finite Infinite Response (FIR) filters to remove out-of-band sampling noise from pressure sensors. In addition we will also be using the FPGA to compute the Kinematic features of the finger and hand movements, which require real-time integration of the five IMU-gyro sensor data. This type of computation takes advantage of the fast parallel processing of the FPGA without utilizing too much of the FPGA. Finally, we will performing real-time classification of different hand exercises on the FPGA by utilizing a feed forward neural network. We estimated based on the sensors and processing we will be using that this will utilize close to 12,700 logic elements, 70 virtual pins, 7000 registers, 10 9-bit multiplier elements and 1 PLL on a Cyclone IV FPGA.
Adapting to Changes:
The FPGA will be used to compute the Kinematic features of the human hand, which will change over time. A similar computation on an ARM processor will require up to three integrations in real-time, which will limit the amount of processing cycles left for other operations. Thus, performing these calculations over time on the FPGA will free the ARM processor for other user related tasks. In addition we will also be performing real-time classification of different hand exercises on the FPGA by incorporating a feed forward neural network.
We will be creating custom FPGA IP cores for Kinematic computation and hand exercise classification from the sampled pressure data. These custom interfaces and IP cores will add new functionalities that are not native to the processor.
A nonintrusive and noninvasive Flex Force Smart Glove (FFSG) design is presented that allows for acquisition and processing of sensorimotor information obtained from the human hand. The novel FFSG design is powered by the Intel Altera FPGA and incorporates all the sensors needed to measure the force and rotation of the human wrist and fingers. Quaternion-based Kalman filters are used to fuse the raw sensor data from five finger joints and one wrist joint to provide detailed orientation information. In addition, feed forward neural network filters are used to classify possible hand exercises that can be further used to facilitate rehabilitation through exercise sessions. The novel design will allow for a unified way to quantify the effectiveness of both conventional and robotic-assisted rehabilitation.
After consulting Physical therapist and medical doctors in the area and obtaining crucial feedback, we were able to go from the original proof-of-concept to the first working prototype of the smart glove as shown in Figure 3. By leveraging 3D printing technology, we were able to create a glove out of a flexible material that houses the DE10-Nano, five IMU sensors (each with 9 degrees of freedom), one pressure sensor that has been placed in the center of the hand and a custom Arduino shield that contains an additional IMU along with resistor divider circuits for interfacing with the analog sensors. The glove design was chosen to mimic a workout glove that allows the fingers to be exposed. The remaining sensors are fitted unto a cotton glove that allows for measurement and acquisition of the flex and force movements of the finger. This glove is fitted with five flex sensors and two pressure sensors that have been placed at key points on the glove.
Figure 3: Image to the left shows our original proof of concept that uses a flexible PCB. The image to the right is our first working prototype based on the DE10-Nano and uses a flexible 3D printed material for the base glove.
In this section, we will go over the different performance parameters and metrics used in our FFSG design and evaluate those metrics under different exercise sessions. The FPGA has been configured to interface with all sensors described in the previous section, which is accessed by the ARM processor running on the DE10-Nano. A real-time routine has been setup that streams all of the sensor data along with the results of a neural network running on the FPGA to a local PC host.
Figure 4: introduces the main FFSG graphical user interface, which allows for visualization of the gloves orientation based on the pitch, roll and yaw measurements streamed from the DE10-Nano.
The main graphical user interface runs on a local PC host and takes all of the raw and converted measurements from the hand and displays it accordingly. The GUI also allows for observation of the flex and pressure sensors as illustrated in Figure 5.
Figure 5: Sample display of the real-time pressure data, showing the results of slight thumb pressure force being exerted unto a ball.
In the top-left of the plot, we see a time history plot of the three force sensors, the center, thumb and middle. By applying a slight force unto a green exercise ball, we see that this is reflected in the time-history plot. This is also reflected in the histogram plot to the right of the same Figure. A 3D hand model is also created that takes the real-time sampled data from the ADC on the DE10-Nano and create a 3D rotation based on the flex angle. For a demo of this please refer to our video demonstration. Other tabs within the GUI allow for real-time visualization of the IMU data, the IMU wrist, thumb, index, middle, ring and pinky information can all be displayed accordingly.
In this sub-section, we will perform different hand exercises and demonstrates the gloves ability to detect possible hand exercises that can be used in an interactive gaming session. We will begin by describing some of the different hand exercises that we have trained the neural network running on the FPGA to classify. These are: Power Grip, Ball Roll, Finger Tip, Cup and Remote Curl. We will start by demonstrating the gloves ability to detect the Finger-tip exercise. Depending on the hand and finger orientation, the classifier will highlight this exercise as one possible exercise. After selecting it and performing the exercise we report the results in Figure 6.
Figure 6: Result of the finger-tip grip exercise. Image of the exercise is shown in the top left using a green exercise ball. A 3D model of the ball and hand is shown to the right in an interactive gaming session. Detailed sensor breakdown during the exercise session is shown in the bottom plot, which mainly consist of thumb and index interactions.
From this figure we see the interactive feedback provided to the user during the gaming session. This includes the time remaining to complete the exercise, the number of reps completed and a 3D model of the object to be used in the exercise. Once the exercise begins we have 45 seconds. We point out that this exercise is a force-based exercised, meaning that we are scored based on applying the right amount of force on the ball and releasing the force to complete one cycle. We repeat this process and try to get as many force-based cycles as possible. Afterwards we are giving the final score and a feedback to encourage the user to do better if needed. More detailed real-time results of the sensor data captured during the exercise session can be seen in the bottom plot of Figure 6, which shows the variations of the thumb pressure sensor data during the exercise session. Furthermore, we introduce the 3D finger model that has been generated in the bottom left of the same plot. The only variation that is noticeable here is the flexing of the index finger, which varies more than that of the thumb during finger-tip grip exercises. The next exercise that we report is the power grip exercise, which is shown in Figure 7.
Figure 7: Result of the power grip exercise. Image of the exercise is shown in the top left using a green exercise ball. A 3D model of the ball and hand is shown to the right in an interactive gaming session. Detailed sensor breakdown during the exercise session is shown in the bottom plot, which mainly consist of hand-center and middle finger pressure interactions.
From this figure we see a similar interactive feedback provided to the user during the power grip exercise session. We note in the 3D model of the hand (right plot), the color code is now red instead of blue as before. This is because the center pressure sensor result also controls the color coding of the hand model being displayed. So for high hand pressure this is red and for no hand pressure this is blue. The detailed real-time results is shown in the bottom plot of Figure 7, which shows the variations in the hand-center and middle finger pressure sensors during the power-grip exercise. The flex sensor results show an increase in flex angle of all the sensors except the middle finger during the power grip exercise. The result show that the power grip exercise utilizes different finger kinematics than the finger-tip grip as shown earlier. The next exercise that we report is the ball roll exercise, which is shown in Figure 8.
Figure 8: Result of the ball roll exercise. Image of the exercise is shown in the top left using a green exercise ball. A 3D model of the ball and hand is shown to the right in an interactive gaming session. Detailed sensor breakdown during the exercise session is shown in the bottom plot, which mainly consist of hand-center pressure interactions.
From this figure we see a similar interactive feedback provided to the user during the ball roll exercise session. We note in the 3D model of the hand (right plot), the exercise follows the users hand interactions with the ball during the ball roll exercise. Furthermore, this is also a force-based reps as before, since we are exerting force unto the object and releasing the force to complete a cycle of exercise. The detailed real-time results is shown in the bottom plot of Figure 8, which shows the variations mainly due to hand-center pressure sensors during the ball roll exercise. The flex sensor results show an almost flat response during the entire exercise since all of the fingers are in the straight position during the exercise. We will now move from a pressure-based exercise to a rotation-based exercise. The next exercise that we report is the cup exercise, which is shown in Figure 9.
Figure 9: Result of the cup exercise. Image of the exercise is shown in the top left using a blue exercise cup. A 3D model of the cup and hand is shown to the right in an interactive gaming session. Detailed wrist IMU sensor breakdown during the exercise session is shown in the bottom plot, which consist of gyro, acceleration and magnetometer variations.
From this figure we see a similar interactive feedback provided to the user during the cup exercise session. We note in the 3D model of the hand (right plot), the exercise now follows the users hand interactions with the cup during the exercise. This is a rotation-based exercise, meaning that the user needs to rotate the cup from an initial point to an end point to complete a full range of motion. The detailed real-time inertial measurement unit (IMU) results are shown in the bottom plot of Figure 9, which shows the variations are due to gyro, acceleration and XY magnetometer measurements. The plot also shows a real-time plot of the wrist orientation and results of a Kalman filter running on the DE10-Nano. The Kalman filter running on the DE10-Nano works by taking the raw sensor data (gyro, accel, and mag) and performs a Quaternion-based Kalman filter that fuses all of the sensor data to produce the pitch, roll and yaw measurements. Given the power of the DE10-Nano, we currently have six Kalman filters running, to compute the orientation of wrist, thumb, index, middle, ring and pinky measurements. The final exercise that we report is the remote curl exercise, which is shown in Figure 10.
Figure 10: Result of the remote curl exercise. Image of the exercise is shown in the top left using an exercise remote. A 3D model of the remote and hand is shown to the right in an interactive gaming session. Detailed pressure/flex sensor breakdown during the exercise session is shown in the bottom plot, which mainly consist of middle-finger force variations and occasional thumb interactions.
From this figure we see a similar interactive feedback provided to the user during the remote curl exercise session. We note in the 3D model of the hand (right plot), the exercise now follows the users hand interactions with the remote curl during the exercise. This is also a rotation-based exercise, meaning that the user needs to rotate the remote from an initial point to an end point to complete a full range of motion. The wrist IMU measurements is similar to that of the cup, so we report that of the pressure/flex sensors in the bottom plot of Figure 9, which shows the variations are due to middle-finger and hand-center force measurements. With a majority of the force contribution is due to the middle-finger. We could also display other sensor data, such as the thumb, index, middle, ring and pinky IMUs but for the sake of clarity and simplicity we leave those out. The figures 6-10 all demonstrate a working glove prototype that is able to acquire, measure and compute the desired Fugl-Meyer assessment (FMA) scores depending on the exercise.
The design architecture of the FFSG design is shown in Figure 11.
Figure 11: Design architecture. The FPGA components are shown on the bottom left, which consist of SPI modules for interfacing with the IMU and ADC. It also contains two custom routines for classification of hand exercises. The HPS components are shown on the bottom right, which interfaces with all of the FPGA routines through Avalon Memory Mapped Slave interfaces.
From this figure, we see the FPGA components, which uses SPI routines to interface with all six IMUs and the ADC. This is implemented as a Qsys project such that the addresses are properly mapped to possible address available to the HPS. The FPGA components also includes custom neural network routines for classification of different hand exercises based on different sets: pressure-based exercise and rotation-based exercise. The neural network have been pre-trained based on recorded exercises performed on the glove. The neural network performance for both sets are shown in Figure 12.
Figure 12: Neural network performance of the two sets. The sets are designed with identical parameters: inputs layer size of 80, hidden layer size of 10 and output layer size of 3. The pressure-based set reached it best validation after 22 epochs, while the rotation-based set reached its best validation after 27.
The design uses a two-layer feed-forward network, with sigmoid hidden and softmax output neurons that can classify vectors arbitrarily well given enough neurons in its hidden layer. The network was trained by employing a scaled conjugate gradient backpropagation. The neural network was designed to have an input layer size of 80, which represents to total number of raw sensor types from the glove. The design used one hidden layer with 10 total neurons, the final output layer consists of 3 neurons. The results show that the pressure-based set reached best validation after 22 epochs, while the rotation-based set took an additional 5 epochs before meeting best validation. We also employed a Quaternion-based Kalman filter that takes the raw IMU sensor data and fuses them for pitch, roll and yaw estimates of each joint: wrist, thumb, index, middle, ring and pinky. The Kalman filter design is shown in Figure 13
Figure 13: Kalman filter Quaternion calculations performed on the DE10-nano.
The high level detailed design architecture for both FPGA and HPS is further shown in Figure 14, which consist of the RTL for the FPGA and a state machine for the HPS. At the current state the HPS code is about 128 KB in size. The FPGA RTL code utilization breakdown is shown below:
Logic Utilization: 7,438 / 41,910 (18%)
Total registers: 13432
Total pins: 260 / 314 (83%)
Total block memory bits: 1,198,848 / 5,662,720 (21%)
Total DSP Blocks: 1/112 (< 1%)
Total PLLs: 1/6 (17%)
Total DLLs: 1/4 (25%)
Figure 14: Detailed design architecture, showing the top RTL entity for the FPGA and the main state machine running on the HPS.
Finally, the design for the custom Arduino shield used to interface with all the analog sensors is shown in Figure 15. It also includes an additional IMU (LSM9DS) for wrist sensing, the same sensor type is also used in each finger IMU. We plan on upgrading this design in the future for better IMU performance and an additional 8-channel ADC to increase our sensing capabilities.
Figure 15: Design of the custom Arduino shield, which contains resistor dividers and an additional IMU for wrist motion sensing.
This project will contribute to the ultimate goal of helping patients suffering from stroke or joint-related injuries, more specifically injuries affecting the upper extremities. The result of this project will lead to a low-cost wearable device that patients can use to track their rehabilitation progress. Furthermore, the project will result in a body of knowledge that both medical doctors and physical therapists can leverage. For example, Fugl-Meyer assessment (FMA) scores can be computed from the acquired sensorimotor information to assess the physical performance of a patient. This project also has a huge commercial potential for non-medical related applications. These applications includes, 3D spatial interfacing, robotics, gaming and sports. We are excited about the potential of the FFSG device and are continually working to improve the glove based on constructive medical and community feedback.