Alignment in a Multimodal Interlingual Computer-Mediated Map Task Corpus

Abstract

This work presents an assessment of interlocutor alignment using a semi-automated method in the context of multimodal interlingual (English-Portuguese) computer-mediated interactions. We study the adaptation phenomenon (also known as convergence behaviour and alignment behaviour) by looking at verbal repetition at different levels of linguistic representation. Since alignment behaviour has already been analysed in direct human-to-human and in human-to-agent dialogues, one may wonder whether the same behaviour is observed in interlingual computer-mediated communication. First, we compare repetitions patterns in task-oriented dialogues of human-to-human communication (HCRC Edinburgh Map Task corpus) and interlingual computer-mediated human-to-human communication (ILMT-s2s corpus), for eye-contact and no eye-contact scenarios. Secondly, we study the relation between the cognitive state of the subject, and the alignment process in interlingual computer-mediated communication settings. Results show that above chance repetitions, signalling verbal alignment, are present in both direct human-to-human communication and interlingual computer-mediated interactions, and that interlingual computer-mediated setting yields significantly more self-repetitions than direct human-to-human interactions. Also, in interlingual computer-mediated communication, a lack of alignment cues for long sequences correlated with a high amount of negative cognitive states in the eye-contact setting, implying a potential lack of mutual understanding.