EM032 » MultiWave: A Fully Customizable Mobile Function Generator For Haptic Feedback In VR
In human-computer interaction, haptic feedback is one of the most important modalities. In particular, modern devices such as virtual reality headsets use vibratory feedback to increase immersion in virtual environments. So far, the main problem has been that with an increasing number of actuators, signal generation is limited to a few possibilities. As a result, various effects cannot be precisely represented at the expense of immersion.
With our project we want to develop a mobile haptic full body feedback device, which we call MultiWave.
Due to the almost unlimited possibilities of FPGAs, we will develop a high-precision independent multichannel function generator that can produce any periodic signal for haptic feedback on every output. All possible signal parameters such as frequency, function, amplitude and phase between the outputs will be freely configurable in real time. An expansion shield for the DE10 nano platform is to be built, which supplies the system with power and has Bluetooth and wifi connection as well as a wired USB/UART interface for controlling the FPGA, which generates the output signals.
The aim is to simulate the collision with objects in virtual environments.
In a demonstrator VR game, various objects such as bushes, water, explosions, etc. are to be correctly replicated by the built full body suit and the user experience will be maximized.
Furthermore we will create a python API to control the FPGA. In a graphical user interface made with our API, all parameters and the number of outputs should be freely configurable even without prior knowledge of the system. In the style of a simple drawing program, the grid of actuators are to be controllable almost in real time in order to explore haptic effects on the skin even without programming knowledge.
Haptic feedback is a well discovered field in human-computer interaction. Single vibration motors can achieve different effects on the human skin by variation of function, frequency or amplitude. For instance, a sine wave function modulated on the eigenfrequency of the motor causes an effect of "roughness". More complex effects can be created by using multiple actuators at the same time. As an example, at a small distance between two vibration motors, only one actuator is felt, which can be shifted by varying the amplitude. In this way, a feeling like a moving instect can be created on the skin. With periodic signals, the phase of the control signals is another function parameter that can generate effects such as water waves at low sinusoidal frequencies.
A challenge beyond the capabilities of dedicated hardware, microcontrollers and even small computers such as the Raspberry Pi is the simultaneous control of a large number of actuators simultaneously and precisely in all signal parameters. However, exactly these requirements are set for virtual environments to represent collisions with different objects and to interact with the scene. Furthermore, signal parameters have to be processed in real time to avoid delays in the felt sensation. For this reason, only modern FPGAs such as on the DE10 Nano kit are able to master such requirements due to their immense possibilities.
In practical use, such systems are not only limited to VR games. In the literature also mobile applications are known e.g. to improve the posture during sports or to learn a musical instrument. By more precise control with more actuators it might be possible to achieve much better learning effects. Furthermore, such function generators are also of interest for medical applications. As an example HASOMED distributes the RehaMove 3 which stimulates the muscles with four electrodes. Also this system is very limited in the precise control of the function parameters by only 16 steps. Also here it is known from the literature that by changing the signal parameters different effects like itching can be perceived. For the Innovate FPGA Design Contest we will limit ourselves to vibratory feedback due to ethical reasons.
In order that our system can be used mobile, like in the literature, we will develop an expansion board for the DE-10 Nano which offers enough power to supply the system over longer periods of time completely without power cables. Since the FPGA outputs cannot operate actuators, our hardware also includes MOSFETS which provide the power for the vibration. The necessary Bluetooth, Wi-Fi and USB/UART interfaces are implemented via a microcontroller installed on the expansion board. To give haptic feedback on the whole body, the actuators are mounted on velcro tape that can be fastened anywhere on the body.
As a demonstrator we will create an interactive virtual environment in Unity with objects like bushes, water and animals. In addition, we will develop a python API that allows an easy use of the system for research purposes. In order to explore effects on the body even without programming skills a demonstrator in the style of a simple painting program will be developed that easily controls all Actuators in almost realtime. MultiWave is intended to become a general platform for scientific purposes and applications in the field of virtual reality with haptic feedback.
Step up/Step down converters are used for the output signals and the DE10 nano kit to provide constant output power independent of battery charge.
The used ESP-32 microcontroller is programmed with Arduino software to allow easy customizations.
FPGA Design Overview
PWM-generator: Two simple counters that compare the number of clock ticks to the desired output value and count through the 1024 steps of the signal. Additionally all signal paramters are stored here. Each output has its own PWM generator.
Update Logic: Handles reqeusts from the PWM generators further to the memories to avoid simultaneous reads. At this point changes of the phase between two actuators can be set.
Memories: The memory reader reads the desired output function sample and calculates all parameters for the PWM value according to the desired function parameters. Divisions will be replaced by multiplications with magic numbers to achieve a high performance and to elimite critical paths.
Predefined Function Memories: Except for the DC generator all functions are stored in ROMs.
Custom Memories: Random Access Memories that can be set by the user via UART.
External Communication: Handles external user requests via UART. See below for additional information for the state machine that implements the protocol.
PLL: Not included in block diagram. Generates the 100MHz clock signal that drives all units.
Top Level parameters: In order to ensure easy adjustments on the FPGA, the number of ports, the number of custom memories, the clock frequency, the PWM steps and PWM frequency as well as the baud rate of the UART interface will be freely configurable before compilation.
FPGA Communication states
In case of errors The FPGA will return an Error Code.
Commands are started with an ASCII "M" followed by binary values.
Commands starting with an ASCII "C" write values to a desired internal FPGA memory.
N is the number of outputs and . N=80
K is the number of internal memories. K=250
P is the resolution of the function. P=1024
The frequency range is adjustable from 0Hz to 512Hz, which covers the usable range for haptic feedback.
Higher frequencies can be achieved by storing a certain number of periods in a custom memory at the expense of accuracy..
The DE10-Nano kit is mounted on the green velcro belt.
The white numbers represent the mounted actuators on each velcro tape.
A total of 80 vibration motors are controlled via the GPIO and Arduino header of the DE10-Nano kit.
Example of a sine function as pulse width modulation signal with 2 bit resolution. The red line represents the analog output value of the signal generated in black.
We want to achieve a resolution in 10 bits and resolve the signals in 1024 steps at 100MHz internal clock signal. This results in a minimum Ton and Toff time of 5 nanoseconds. In order to adjust the final output frequency, each step is repeated accordingly. In order to reach the final output frequency as accurately as possible, each step is extended by an adjusted number of clock ticks.
Counting clock ticks and resolving the signal in 1024 steps results in a frequency error that can be approximated by polynomial regression. We expect a maximum error of 1.25 Hz which however has no negative effect on the perceived output signal as these differences are not noticeable at higher frequencies.
According to the current state of the art for VR suits, body regions are augmented with haptic feedback by microcontrollers directly or with shift registers as well as multiplexers. This means that a relatively inaccurate control can only be achieved with high hardware costs. Especially if all signal parameters frequency, amplitude, signal form and phase between the actuators have to be controlled. The phase between the actuators represents a particularly great challenge because individual hardware modules have to be synchronized with each other. Even high-precision output signals are hardly possible because they require dedicated PWM hardware. However, this is necessary to adjust the strength of the feedback to the user individually. In most cases the hardware also does not have enough memory to provide a large number of own functions. Additional memory would have to be installed, which results in additional costs. Furthermore, this results in additional weight which is perceived as disturbing especially in mobile applications.
As a solution to these challenges, a central unit has to take over the complete control, which must also be resource efficient. To ensure precise control, each actuator requires its own independent signal. Therefore, FPGAs like the DE10 Nano kit are particularly suitable for such tasks due to their large number of outputs, hardware resources and their parallelism. The almost instantaneous adjustment of all parameters prevents delays of the effects and increases the immersion in a virtual reality.