EM032 » MultiWave: A Fully Customizable Mobile Function Generator For Haptic Feedback In VR
In human-computer interaction, haptic feedback is one of the most important modalities. In particular, modern devices such as virtual reality headsets use vibratory feedback to increase immersion in virtual environments. So far, the main problem has been that with an increasing number of actuators, signal generation is limited to a few possibilities. As a result, various effects cannot be precisely represented at the expense of immersion.
With our project we want to develop a mobile haptic full body feedback device, which we call MultiWave.
Due to the almost unlimited possibilities of FPGAs, we will develop a high-precision independent multichannel function generator that can produce any periodic signal for haptic feedback on every output. All possible signal parameters such as frequency, function, amplitude and phase between the outputs will be freely configurable in real time. An expansion shield for the DE10 nano platform is to be built, which supplies the system with power and has Bluetooth and wifi connection as well as a wired USB/UART interface for controlling the FPGA, which generates the output signals.
The aim is to simulate the collision with objects in virtual environments.
In a demonstrator VR game, various objects such as bushes, water, explosions, etc. are to be correctly replicated by the built full body suit and the user experience will be maximized.
Furthermore we will create a python API to control the FPGA. In a graphical user interface made with our API, all parameters and the number of outputs should be freely configurable even without prior knowledge of the system. In the style of a simple drawing program, the grid of actuators are to be controllable almost in real time in order to explore haptic effects on the skin even without programming knowledge.
Haptic feedback is a well explored field in Human-Computer Interaction. Single vibration motors can achieve different effects on the human skin by variation of function, frequency, or amplitude. For instance, a sine wave function modulated on the eigenfrequency of the motor causes an effect of "roughness." More complex effects can be created by using multiple actuators at the same time. As an example, at a small distance between two vibration motors, only one actuator is felt, which can be shifted by varying the amplitude. In this way, a feeling like a moving instect can be created on the skin. With periodic signals, the phase of the control signals is another function parameter that can generate effects such as water waves at low sinusoidal frequencies.
A challenge beyond the capabilities of dedicated hardware, microcontrollers, and even small computers such as the Raspberry Pi is the control of a large number of actuators simultaneously and precisely in all signal parameters. However, exactly these requirements are given for virtual environments to realistically represent collisions with different objects and to interact with the scene. Furthermore, signal parameters have to be processed in real time to avoid delays in the felt sensation. Only modern FPGAs, such as on the DE10 Nano kit, are able to master such requirements due to their immense possibilities.
In practical use, such systems are not only limited to VR games. In the literature also mobile applications are known, e.g., to improve the posture during sports or to learn a musical instrument. Through more precise control with more actuators it might be possible to achieve much better learning effects. Furthermore, such function generators are also of interest for medical applications. As an example HASOMED distributes the RehaMove 3 which stimulates the muscles with four electrodes. Also their system is very limited in the precise control of the function parameters in only 16 steps. It is known from the literature that by changing the signal parameters different effects, like itching, can be evoked. For the Innovate FPGA Design Contest we limit ourselves to vibratory feedback for safety and ethical reasons.
In order to create a system for mobile applications, that extends known prototypes from the literature and products on the market, we developed an expansion board for the DE-10 Nano which offers enough power to supply the system over longer periods of time completely without power supplies. Since the FPGA outputs cannot operate actuators, our hardware also includes MOSFETS, which provide the power for the vibration. The necessary Bluetooth, Wi-Fi and USB/UART interfaces are implemented via a microcontroller installed on the expansion board. To give haptic feedback on the upper body, the actuators are mounted on velcro tape on the suit that can be flexibly attached according to the specific user's body anatomy.
As a demonstrator we created an interactive virtual environment in Unity with different objects to test sensations on the skin. In addition, we developed a Python3 API that allows an easy use of the system for research purposes. In order to explore effects on the body even without programming skills, a demonstrator in the style of a simple painting program is integrated into the software that easily controls all actuators in almost realtime. MultiWave is intended to become a general platform for scientific purposes and applications in the field of virtual reality with haptic feedback. Also navigation of blind people is a desired target user group for the body suit.
Step up/Step down converters are used for the output signals and the DE10 nano kit to provide constant output power independent of battery charge. The used ESP-32 microcontroller is programmed with Arduino software to allow easy customizations.
PWM-generator: Two simple counters that compare the number of clock ticks to the desired output value and count through the 1024 steps of the signal. Additionally all signal paramters are stored here. Each output has its own PWM generator.
Update Logic: Handles reqeusts from the PWM generators further to the memories to avoid simultaneous reads. At this point changes of the phase between two actuators can be set.
Memories: The memory reader reads the desired output function sample and calculates all parameters for the PWM value according to the desired function parameters. Divisions are replaced by shift operations with magic numbers to achieve a high performance and to eliminate critical paths.
Predefined Function Memories: Except for the DC and rectangle generator all functions are stored in ROMs.
Custom Memories: Dual port Random Access Memories that can be set by the user via UART.
External Communication: Handles external user requests via UART. See section 5 for additional information about the state machine that implements the protocol.
PLL: Not included in block diagram. Generates the 100MHz clock signal that drives all units and generates the PWM frequency.
Top Level parameters: In order to ensure easy adjustments on the FPGA, the number of ports, the number of custom memories, the clock frequency, the PWM steps and PWM frequency as well as the baud rate of the UART interface are freely configurable before compilation.
Example of a sine function as pulse width modulation signal with 2 bit resolution. The red line represents the analog output value of the signal generated in black.
We achieve a resolution of 10 bits and resolve the signals in 1024 steps at 100MHz internal clock signal. This results in a minimum Ton and Toff time of 5 nanoseconds. In order to adjust the final output frequency, each step is repeated accordingly. In order to reach the final output frequency as accurately as possible, each step is extended by an adjusted number of clock ticks.
Counting clock ticks and resolving the signal in 1024 steps results in a frequency error that can be approximated by polynomial regression. We achieve a maximum error of 1.25 Hz which however has no negative effect on the perceived output signal as these differences are not noticeable at higher frequencies.
According to the current state of the art for VR suits, body regions are augmented with haptic feedback by microcontrollers directly or with shift registers as well as multiplexers. This means that microcontrollers can only achieve relatively inaccurate control and with additional hardware costs. This is especially the case if all signal parameters, i. e., frequency, amplitude, signal form, and phase between the actuators, have to be controlled. The phase between the actuators represents a particularly great challenge because individual hardware modules have to be synchronized with each other. By applying multiplexers or external DACs only time-shifted signals can be generated. Latest innovations in haptic actuators like the Lofelt wave technology unit require high-precision output signals with controlled phase shift when a large number of these actuators are used in a grid to generate accurate feedback. In most cases the hardware also does not have enough memory to provide a large number of custom functions. Additional memory would have to be installed, which results in additional costs. Furthermore, this leads to additional weight, which is perceived as disturbing especially in mobile applications.
As a solution to these challenges, a central unit has to take over the complete control, which also has to be resource efficient. To ensure precise control, each actuator requires its own independent signal. FPGAs like the DE10 or DE0 Nano kit are particularly suitable for such tasks due to their large number of outputs, hardware resources, and their parallelism. The almost instantaneous adjustment of all parameters prevents delays of the effects and increases the immersion in a virtual reality environment.
Due to the large amount of integrated hardware ressources in FPGAs, the entire system can in the future be extended to obstacle detection and collision avoidance, by applying cameras and other hardware to develop a navigation system for blind people.
Due to the ever increasing computing power of computers, the representation of virtual realities is becoming more and more realistic. However, there is up to now a lack of accurate haptic feedback during interaction and collision with objects and environments. This leads to an unnatural perception of sensations which results in a lack of immersion. To counteract this challenge with vibratory feedback, all signal parameters must be freely and precisely adjustable. With our project we want to increase the possibilities to maximize the immersion in VR.
While microcontrollers and other hardware approaches, that modern haptic suits apply, are limited in their possibilities, FPGAs have immense parallel power which is required to realize simultaneous control. Mobile use is also an important factor for VR environments. Since the DE10-Nano (and especially the DE0 Nano) kit is very small and has a large number of outputs, it is ideally suited to meet all requirements in this area.
Although VR is also of interest in medical research and covers many application areas of simulation, we focus our contribution on immersion. The most important target group here are gamers who want to have the sensation of being in the game. For this purpose we created with Unity a small game called "Sensation Island" which transfers the user to an interactive island where he or she can interact freely with the environment.
In the following a brief overview will be given on all units involved in the project.
We have decided to put our design on a warning vest to ensure maximum safety in mobile applications. All 76 used actuators can be flexibly adjusted in their position via velcro tape. The DE10 Nano is clipped to the trousers pocket and takes over the complete control of all actuators. The additional ESP-32 on the PCB enables wireless control via bluetooth or WiFi. The actuators are integrated in neoprene to prevent the vibration from propagating over the vest. In addition a grid of 80 actuators was built to test the FPGA implementation and sensations.
In the FPGA, all functions are stored in RAMs and are accessed from each output at each step. A simple communication protocol via USB/UART ensures an easy setup of all function parameters. The PWM frequency, number of freely rewritable memories for custom functions, UART baud rate and number of outputs can be set via top level parameters.
The 80 GPIO pins on the DE10 Nano control the outputs via Pulse Width Modulation (PWM) almost in real time in the frequency Range 0Hz to 512Hz which covers the maximum felt sensation at 260Hz.
A PWM is a square wave signal with constant period duration that oscillates between two different voltage levels (0V and 3.3V). The ratio of switch-on time to switch-off time can vary and forms the duty cycle. For our implementation we use a 100 MHz clock frequency to generate a precise output signal with a 10 bit resolution.
Recent technological advances, such as inside-out tracking, enable the mobile use of VR headsets. We use the Oculus Quest VR headset and built a virtual island with different haptic effects e.g. waterfalls and collision with virtual objects. Due to the limited computing power, some limitations had to be made in graphics resolution.
To test effects, a software was created together with an API in Python 3. A pattern designer can be used to test various effects with regard to all functional parameters. Additionally all actuators can be controlled with strokes in the style of a simple drawing program.
To realize this functionality on the Cyclone FPGA each output has a PWM generator consisting of simple counters which are updated after each of the 1024 discrete function steps. The functions are stored in the internal memories and the amplitude of the signal is calculated when a sample is requested. We eliminated critical paths through the divider by using magic numbers that change a constant division to a shift operation.
In order to control several simultaneous requests of the outputs, an update logic is switched between the generators and the memories with the computing unit. Furthermore, a UART connection is implemented via two pins, which processes the incoming commands and changes the parameters within the PWM generators. At this point the phase can also be set to another output by setting the internal counters. All other parameters can be set asynchronously between all requests.
The following picture shows the state diagram of our communication protocol. Each command must be completed within five sceonds.
In case of errors The FPGA will return an Error Code.
Commands are started with an ASCII "M" followed by binary values.
Commands starting with an ASCII "C" write values to a desired internal FPGA memory
N is the number of outputs and . N=80
K is the number of internal memories. K=250
P is the resolution of the function. P=1024
The frequency range is adjustable from 0Hz to 512Hz, which covers the usable range for haptic feedback.
Higher frequencies can be achieved by storing a certain number of periods in a custom memory at the expense of accuracy..
Before we started to implement effects in VR we created a Python3 Software and API for testing different Effects. In the picture below actuators are grouped and a pattern was tested to create water waves with our pattern designer. We identified that low sine waves at 1Hz provide a very good representation of water when the colored groups have a phase shift between eachother.
The testing was performed on a grid of 80 actuators which can be attachted via velcro trape to different objects such as chairs. The vibration motors are embedded in 10mm neoprene in order not to propagate the vibration over the object. We power the entire prototype including the development board via a power supply for stationary use.
We have decided to place the vibration motors on a warning vest so that maximum safety can be guaranteed in mobile use. The size is freely adaptable to the shape of the user. The vibration motors are attached variably on velcro tape on the warning west side that is close to the body. Also here 10mm neoprene was used, so that the vibration is not propagated over the vest. The DE10 Nano is clipped onto the trouser pocket and can be powered over one hour at full vibration intensitiy with the batteries on the backside of the development board. The complete prototype can be worn under the jacket for mobile use. Also in the future there are many possibilities to use this design for blind navigation. On the right, the view of the vibration motors is shown on the vest. The actuators are embedded in neoprene and can be freely re-attached on the Velcro tape.
In the virtual environment, the user is transferred to a lonely island. Various objects are placed there with which the user can interact. For example, the user can feel the water waves on his body when he goes into the sea. In addition, there is a waterfall and a bomb hidden in a box, which actuates the whole body suit. As a little extra the user can fly over the island when standing on a "magic" stone. On a table different objects were placed to test different "hit" sensations. The following picture shows the island from a live recording on the Oculus Quest. As it can be seen. As can be seen, due to the limited computing power, only low graphics can be displayed. We have also tried to implement better graphics, but this leads to large drops in the FPS rate.
During creating collisions, we have also noticed limitations which can be attributed to the limited computing power. An exact model of the human body was not possible, as this also led to frame rate breakdowns. We decided to use a simple capsule model for the torso as seen in the figure below which also gave good results in the sensation. Whenever an object hits its boundaries the corresponding vibration motors are actuated.
Since effects up to 260Hz are felt particularly well, at least this frequency range must be covered. W cover the frequency range of 1 to 512Hz with 10 bits. A frequency of 0Hz can be achieved by intensity control of a constant signal. Higher frequencies can be implemented by storing more then one desired wave in a custom memory
Actuator intensities can be differentiated in up to five steps. However, in order to feel an exact movement between actuators, a more precise control is required, whereby we regulate the actuators in at least 100 intensity levels. By using a gain factor from 0 to 2.55, signals can also be changed by clipping.
Signals are resolved in 1024 steps to provide high accuracy operation that is especially important for alternative actuators such as bone conductors to minimize noise. The phase can be shifted by between 0 and 359 degrees in relation to any actuator. We tested each functionality with ModelSim and also with an oscilloscope.
When using this system for auditory feedback, noise has to be eliminated by the 100MHz clock signal which generates the PWM frequency. A simple second or third order low pass with 100nF and 1kOhm can suppress noise. We tested this by using a MAX 98306 stereo amplifier and a voice coil on the output.
On the Cyclone V FPGA 80 actuators require less than 40% of the ressources. Over 200 outputs can be implemented and over 300 custom functions designed. The following table gives an overview of the ressource utilization.
The biggest advantage of using the DE Nano board family for this project is the small size, which allows fast prototyping, but still offers the possibility to develop a prototype that is almost similar to a final product. This has made it already possible to test the prototype in mobile use without large cost expenditure and to identify challenges with virtual realities in mobile use. We were able to find out that people are interested in this technique but are not yet accepting VR headsets in their daily lives. This means that people keep a particularly large distance when a person wears a headset. Furthermore, in mobile use, it should be borne in mind that means of transportation and the use nearby roads should be avoided generally. The use in the tram also brings further risks, as the user cannot estimate when the tram starts to depart. The risk of falling is too high, which is why standing use should be avoided here. In contrast, the use in a park is optimal as long as there are no animals in the vicinity, as they cannot estimate the danger. For our tests, an experimenter was always present, who pointed out possible dangers. Here, too, the user could be informed in the future by adequate feedback with the body suit.
In the following we will give some short design schemes of our prototype.
The body suit was created on the basis of a warning vest to minimize risks in mobile use. It is also possible to wear the system without VR headset when cycling. The prototype can be adapted to the size of the user and all actuators can be freely repositioned. The DE development board is attached to the trousers pocket with a clip and does not restrict the freedom of movement. All actuators are embedded in 10mm neoprene to prevent the vibration from propagating over the vest. In addition, this minimizes vibration noise, which brings advantages in quiet environments.
This standard circuit of N-channel MOSFETs for vibration motors suppresses electromagnetic interferences by the capacitor. The flyback diode prevents against voltage spikes from the coils and the resistor to ground reduces quiescent current. A fast switching MOSFET is recommended to ensure a high output signal quality.
Motor driver board with ESP32, USB/Uart and additional Arduino header. The 3.3V are transformated to 5V to drive the DE development boards.
Mobile Power board for LR18650 batteries that provides 12A and 3.3V. An additional LED indicates the battery status. The size of both boards was optimized and tested on a DE0 Nano and DE10 Nano development board. All boards were created with Eagle 7.7 and manufactured by Elecrow as a 4-Layer design.
We have developed and tested a VR body suit for mobile use. Based on a warning vest with 76 vibration motors it could be shown that FPGAs are able to outperform today's comparable VR body suit systems in their precision. We found out that accurate phase control provides advantages for immersion in virtual worlds. Furthermore, we have created expansion boards for the DE Nano family, which allows a complete mobile use with wireless connection powered by batteries. In addition we tested our prototype in mobile use can could indentify challenges for future research.
In the future the entire system can be used to investigate vibration effects on the skin in detail. This is not only advantageous in the multimedia area but also enables the implementation of a navigation system for blind people. Furthermore, on the basis of MultiWave pattern for electrical muscle stimulation can be examined in detail what might brings in the future advantages in rehabilitation.