Immersive Multimodal Virtual Reality Experiences - Using Visual and Auditory Stimuli to Improve Tactile Experiences

Abstract

Haptic feedback, such as the sensation of `being touched', is an essential part of how we experience our environment. Yet, it is often disregarded in current virtual reality (VR) systems. In addition to the technical challenge of creating such tactile experiences, there are also human aspects that are not fully understood, especially with respect to how humans integrate multimodal stimuli. In this research, we proved that the visual stimuli in a VR setting can influence how vibrotactile stimuli are perceived. In particular, we identified how visual cues that are generally associated with the characteristic of weight, namely size and falling speed, influence tactile perception, whereas a similar effect could not be achieved for a temperature-related visual cue, namely color. Such an effect was also not achieved for impact in a realistic precipitation scenario with other typical representative visual and auditory indications, however the results did demonstrate that fundamentally consistent cues, as were used in the case of weight, are necessary to trigger such a response rather than only real-world knowledge. Our results have technical implications -- for example, suggesting that a rather simple vibration motor may be sufficient to create a complex tactile experience such as perceiving weight when correctly presented with multimodal stimuli -- and relevance for practical implementations -- for example, indicating that vibration intensities need to be `exaggerated' to achieve certain effects.