Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance
Citation:
Ylva Ferstl, Michael Neff, Rachel McDonnell, Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance, Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, IVA 2020, International Conference on Intelligent Virtual Agents (IVA), 2020, 2020Download Item:
3383652.3423882.pdf (Published (author's copy) - Peer Reviewed) 337.0Kb
Abstract:
Gesture behavior is a natural part of human conversation. Much work has focused on removing the need for tedious hand-animation to create embodied conversational agents by designing speech-driven gesture generators. However, these generators often work in a black-box manner, assuming a general relationship between input speech and output motion. As their success remains limited, we investigate in more detail how speech may relate to different aspects of gesture motion. We determine a number of parameters characterizing gesture, such as speed and gesture size, and explore their relationship to the speech signal in a two-fold manner. First, we train multiple recurrent networks to predict the gesture parameters from speech to understand how well gesture attributes can be modeled from speech alone. We find that gesture parameters can be partially predicted from speech, and some parameters, such as path length, being predicted more accurately than others, like velocity. Second, we design a perceptual study to assess the importance of each gesture parameter for producing motion that people perceive as appropriate for the speech. Results show that a degradation in any parameter was viewed negatively, but some changes, such as hand shape, are more impactful than others. A video summarization can be found at https://youtu.be/aw6-_5kmLjY.
Author's Homepage:
http://people.tcd.ie/ramcdonnDescription:
PUBLISHED
Author: Mc Donnell, Rachel
Other Titles:
Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, IVA 2020International Conference on Intelligent Virtual Agents (IVA)
Type of material:
Conference PaperCollections:
Availability:
Full text availableKeywords:
Gesture behavior, speech-driven gesture generators, parameters characterizing gesture, Speech gestures, Perception, Gesture modelling, Machine Learning (ML)DOI:
http://dx.doi.org/10.1145/3383652.3423882Licences:
Related items
Showing items related by title, author, creator and subject.
-
Egocentric Gesture Recognition for Head-Mounted AR devices
Smolic, Aljosa; Chalasani, Tejo; Ondrej, Jan (2018)Natural interaction with virtual objects in AR/VR environments makes for a smooth user experience. Gestures are a natural extension from real world to augmented space to achieve these interactions. Finding discriminating ... -
Two Yellow to Cerulean Blue
Fitzgerald, Mary,(b.1956) (2009-01-12) -
Untitled
Johnston, Roy (Irish painter, born 1936) (2009-02-28)