A Proposal for the Evaluation of Adaptive Information Retrieval Systems using Simulated Interaction
Citation:
Catherine Mulwa, Wei Li, Séamus Lawless, and Gareth Jones, A Proposal for the Evaluation of Adaptive Information Retrieval Systems using Simulated Interaction, Workshop on the Simulation of Interaction in Automated Evaluation of Interactive Information Retrieval, SimInt 2010, at the 33rd Annual ACM SIGIR Conference, SIGIR, Geneva, Switzerland., July 19th-23rd, ACM, 2010, 5 - 6Download Item:
Paper21CameraReady.pdf (Published (publisher's copy) - Peer Reviewed) 100.5Kb
Abstract:
The Centre for Next Generation Localisation (CNGL) is involved in building interactive adaptive systems which combine Information Retrieval (IR), Adaptive Hypermedia (AH) and adaptive web techniques and technologies. The complex functionality of these systems coupled with the variety of potential users means that the experiments necessary to evaluate such systems are difficult to plan, implement and execute. This evaluation requires both component-level scientific evaluation and user- based evaluation. Automated replication of experiments and simulation of user interaction would be hugely beneficial in the evaluation of adaptive information retrieval systems (AIRS). This paper proposes a methodology for the evaluation of AIRS which leverages simulated interaction. The hybrid approach detailed combines: (i) user-centred methods for simulating interaction and personalisation; (ii) evaluation metrics that combine Human Computer Interaction (HCI), AH and IR techniques; and (iii) the use of qualitative and quantitative evaluations. The benefits and limitations of evaluations based on user simulations are also discussed.
Sponsor
Grant Number
Science Foundation Ireland (SFI)
07/CE/I1142
Author's Homepage:
http://people.tcd.ie/mulwachttp://people.tcd.ie/selawles
http://people.tcd.ie/msharp
http://people.tcd.ie/vwade
Description:
PUBLISHEDGeneva, Switzerland.
Author: MULWA, CATHERINE
Other Titles:
Workshop on the Simulation of Interaction in Automated Evaluation of Interactive Information Retrieval, SimInt 2010, at the 33rd Annual ACM SIGIR Conference, SIGIRAdvisor:
LAWLESS, SEAMUSWADE, VINCENT
SHARP, MARY
Publisher:
ACMType of material:
Conference PaperCollections:
Availability:
Full text availableSubject (TCD):
Intelligent Content & Communications , Smart & Sustainable PlanetLicences: