A Proposal for the Evaluation of Adaptive Information Retrieval Systems using Simulated Interaction
Item Type:Conference Paper
Citation:Catherine Mulwa, Wei Li, Seamus Lawless, and Gareth Jones, A Proposal for the Evaluation of Adaptive Information Retrieval Systems using Simulated Interaction, Workshop on the Simulation of Interaction in Automated Evaluation of Interactive Information Retrieval, SimInt 2010, at the 33rd Annual ACM SIGIR Conference, SIGIR, Geneva, Switzerland., July 19th-23rd, ACM, 2010, 5 - 6
Paper21CameraReady.pdf (Published (publisher's copy) - Peer Reviewed) 100.5Kb
The Centre for Next Generation Localisation (CNGL) is involved in building interactive adaptive systems which combine Information Retrieval (IR), Adaptive Hypermedia (AH) and adaptive web techniques and technologies. The complex functionality of these systems coupled with the variety of potential users means that the experiments necessary to evaluate such systems are difficult to plan, implement and execute. This evaluation requires both component-level scientific evaluation and user- based evaluation. Automated replication of experiments and simulation of user interaction would be hugely beneficial in the evaluation of adaptive information retrieval systems (AIRS). This paper proposes a methodology for the evaluation of AIRS which leverages simulated interaction. The hybrid approach detailed combines: (i) user-centred methods for simulating interaction and personalisation; (ii) evaluation metrics that combine Human Computer Interaction (HCI), AH and IR techniques; and (iii) the use of qualitative and quantitative evaluations. The benefits and limitations of evaluations based on user simulations are also discussed.
Type of material:Conference Paper
Availability:Full text available