Real-Time Vocal Auralisation in XR: from Multimodal Perception to Applications in Virtual Performance Spaces

Abstract

This thesis explores the sensory modalities that contribute to the perception of acoustic environments in eXtended Reality (XR), focusing on visual input and the auditory acoustic response (auralisation). Using vocal interaction as first-person auditory feedback, it examines key factors of audio-visual multimodality in Immersive Virtual Environments (IVEs). As multimodal research and applications grow with advances in immersive technologies, auditory feedback has been shown to be a key dimension in audio-visual contexts. Prior research has highlighted the role of sound auralisation in supporting perceptual factors in IVEs and shown that altering the quality of one modality can influence overall multimodal perception. Moreover, self-produced vocal interaction and real-time auralisation have also been found to affect musical performance. However, most research has focused on sound sources within the environment, perceived by the user from a third-person perspective. This viewpoint is also reflected in previous work on sound auralisation and audio-visual reconstruction of virtual spaces. To this end, this work investigates self-produced and embodied auditory feedback in multimodal IVEs, its effect on perceptual constructs and psychophysiological responses, and how visual and audio rendering fidelity influence perceived quality and audio-visual integration. In addition, it evaluates the plausibility and usability of a virtual performance space and how it shapes singers� overall experience. Findings show a higher sense of presence and its related subscales when real-time vocal auralisation is present, and demonstrate that rendering fidelity in both the audio and visual domains influences overall perceived quality and audio-visual integration. Moreover, high-fidelity audio-visual representations can provide meaningful environments for vocal training and performance, enabling singers to understand spatial distribution and acoustics while supporting a psychological shift that enhances vocal training in IVEs. Thus, through perceptual, practical, and technical contributions, it highlights the role of real-time vocal auralisation in shaping audio-visual experiences in IVEs.