Advancing Listenability Assessment for Language Learning

Abstract

This thesis advances research on automated listenability assessment in language learning contexts. Listenability is defined as the comprehensibility of spoken materials for an intended listener. The work is motivated by the increasing availability of online spoken materials and the need for scalable methods that can support their evaluation, organisation, and adaptation for listening practice in a target language. The thesis addresses this need through complementary studies that formalise listenability as a construct, evaluate the feasibility of automated text-based assessment, and investigate accent similarity as a perceptually grounded dimension of spoken material assessment.

The thesis first presents a scoping review of research on listenability assessment, with particular attention to automation. The review examines the factors that affect listenability, how these factors have been operationalised, and how technological developments have shaped assessment systems. The synthesis leads to the Listenability Framework, which conceptualises listenability in terms of Content, Delivery, Listener, and Context. It also introduces Listenability Factor Grids, which summarise potential operationalisations of listenability factors at different levels of abstraction. The review shows that Content-Based Factors have received the greatest attention, largely because transcripts are readily available, while Delivery-Based, Listener-Based, and Context-Based Factors remain less fully represented in automated assessment.

The first empirical study evaluates six text-based comprehensibility formulae against human-assigned difficulty levels for Voice of America Learning English materials. Two-way and three-way classification experiments use comprehensibility scores as features in logistic regression models. The results show that these formulae can discriminate between difficulty levels, especially when level differences are large. The study also evaluates their feasibility within a fully automated pipeline using automatic speech recognition and natural language processing. The findings indicate that such integration is feasible, but that robustness depends strongly on the quality of upstream speech recognition and language processing components.

The second and third empirical studies examine accent similarity for listenability assessment. A MUSHRA-style listening activity collects accent similarity judgements from non-expert listeners across native and non-native English accents. The results reveal meaningful perceptual structure, including differences between hard- and soft-decision raters, effects of reference accent and stimulus type, and broad accent groupings associated with perceived nativeness. These findings provide the basis for a modelling study that uses a SCOREQ-based approach to generate utterance-level accent embeddings from which similarity scores can be calculated. The model reproduces major perceptual patterns observed in the listening study, including relative similarity ordering and native/non-native grouping patterns, although finer-grained accent distinctions are less consistently preserved.

Taken together, the thesis clarifies the structure of listenability as a multidimensional construct, demonstrates the utility and limitations of established text-based approaches in automated assessment, and introduces accent similarity as a promising speech-based and perceptually grounded direction for future work. These contributions provide a structured basis for developing listenability assessment systems that are more transparent, scalable, and aligned with contemporary language learning.