Vocal communication of emotion: Exploring the contribution of local and global shifts to the voice source and the impact of the visual context

Abstract

This thesis explores aspects of the mapping of voice quality to affect and the impact of the visual context in emotion perception. The first phase of the study involves a thorough analysis of recorded acted data through manual inverse filtering and model matching to the Liljencrants-Fant model, used as a basis to create contour visualizations and estimate the changes to the voice source related to emotion expression. Five affects were explored: happy, angry, bored, sad and relaxed. This analysis was then followed by the development of a novel methodology to apply acoustic modifications to the voice source parameters f0, EE and RD to a neutral synthesized baseline, in terms of global (utterance-wide), local (aligned to the prosodic structure of the sentence, i.e., pitch accents), and combined (global + local) transformations. These transformations were applied to a neutral baseline using the Voice Source Generator System (Wang & Gobl, 2023) and presented to groups of listener judges in the form of perception tests to evaluate the effectiveness of the modifications system.

Four main studies were conducted. The first two focus on the vocal stimuli and aimed to investigate whether global, local or combined changes were more effective in cueing the intended affect, and whether individual affects were well differentiated (especially within activation categories). The results of these voice-only experiments were mixed. In the first experiment, which applied the affect-related transformations to the original recorded neutral baseline, transformations for the high activation affects were the most successful, while the low activation affects were mostly only identified in terms of its activation category. The types of transformations also varied in effectiveness for each affect. In the second experiment, which used a different neutral baseline produced by an Irish TTS voice, the combined and global stimuli were the most effective, with the local stimuli performing much more poorly in comparison.

The third and fourth experiments investigated the effect of the visual context in the perception of emotion. A set of animated facial expressions were created for each emotion, and listeners were asked to judge the emotion perceived for the following sets of stimuli: voice-only (using the combined shifts); voice with congruent visuals; and voice with incongruent visuals. Types of incongruent combinations varied between experiments, either switched across activation categories (e.g., happy voice with sad visuals), or within activation categories (e.g., happy voice with angry visuals). Results from this set of experiments indicate that, while the addition of congruent facial expressions does not seem to affect identification rates, it did improve differentiation between affects, particularly in terms of valence. The incongruent stimuli influenced the perception of the emotion in the voice, but for the most part the voice was still dominant. This may indicate that different affects rely more on different information channels to be successfully identified.

The shifts and methodology proposed here were relatively successful across most affects, indicating potential success for implementation in terms of emotion expression.