Supporting Wizard of Oz experimentation for language technology applications

Abstract

Wizard of OZ (WOZ) is a well-established method used by researchers and product designers to simulate the functionality and user experience of future systems. Using a human wizard to mimic possible operations is particularly useful in situations where extensive engineering effort would otherwise be required to explore a given design space. The method has been widely used in connection with speech and language technologies, but advances in sensor technology and pattern recognition as well as new application areas such as human-robot interaction have made WOZ increasingly relevant to the design of interactive systems. Even though it is recognised as a valuable prototyping technique, surprisingly little effort has been devoted to exploring WOZ from a wizard perspective. The goal of this thesis is to expand upon the existing knowledge by presenting a systematic investigation and analysis of the potential design space for WOZ prototyping as well as an empirical exploration of appropriate support features for wizards through an iteratively developed web-based prototyping platform. The presented research employed a systematic analysis approach in which the construction and consequent evaluation of a maturing system created new insight into WOZ support and some of its challenges. As part of these evaluations, WOZ characteristics such as wizard workload, wizard consistencies as well as the creation of WOZ experiments, were analysed. The results show that wizards gradually adapt to their task, as would be expected, resulting in faster response times. Consistency, however, is more difficult to achieve and therefore requires additional support. Furthermore, it was found that researchers/designers are generally able to create WOZ experiments, even without upfront training, validating our efforts of providing a platform for more generic support. In summary, this thesis expands the existing body of knowledge on WOZ prototyping by presenting a structured analysis of the method. Informed by both the literature and a set of investigations (including a series of interviews with researchers from industry and academia), it demonstrates the multitude of motivations for using WOZ, highlights its challenges and investigates potential solutions. Being undertaken alongside the development of a new WOZ prototyping tool, this thesis furthermore explores technical problems and presents possibilities for more generic tool support. Even though the presented work has a strong focus on evaluating language technologies, we believe that the generated insight is applicable to the whole domain of WOZ prototyping, and therefore should be seen as a substantial contribution to the field of Human-Computer Interaction.