An investigation of and a hybrid recommender system for evaluating adaptive E-Learning systems

Abstract

A key problem with research in the field of adaptive systems is the inconsistency of evaluation applied to such systems. A fact that is well established by expert evaluators is that adaptive systems cannot be evaluated as if they were non-adaptive. Several researchers acknowledge that evaluation of such systems is a difficult, demanding endeavour due to the complex nature of such systems. One major problem is the understanding of the adaptation mechanism of the system, what is improved by the adaptation, and what might have been the situation if a different kind of adaptation had occurred. Furthermore, when the evaluation of an adaptive system indicates a problem, such as user dissatisfaction and non-use of adaptive features, it is impossible to pinpoint the source of these problems, whether wrong user model, problems with the adaptation theory, wrong adaptation strategy, inappropriate method or evaluation techniques (methods, metrics, and criteria). It is important that evaluators of these systems use correct evaluation techniques. This thesis investigates evaluations of adaptive E-Learning systems developed from 2000 to date and addresses the fact that it is difficult to identify the evaluation objective, the evaluation approach, and the range of evaluation choices. This evidence-based study examines what people have evaluated in adaptive system s and what evaluation techniques they used, and then maps those techniques to different evaluation approaches and techniques. Based on the results of these investigations, there is clear evidence that many design choices are being made during evaluations of adaptive E-Learning systems. For an expert evaluator this is tricky; for a novice evaluator it is much more difficult. Researchers need more advice on their evaluation options in order to attain their goal. They need support in their decision-making. To support these evaluators, the candidate has specified, designed and developed a web-based evaluation framework for supporting evaluators of adaptive systems (EFEx). In addition the candidate has designed and implemented a focused crawling system for evaluation studies of adaptive E-Learning systems. The major contribution of this thesis is a novel hybrid (case-based and knowledge-based) recommendation service built on an evaluation educational dataset. A recommendation technology is used to enhance the appropriateness of suggestions for evaluation techniques for adaptive systems. A hybrid (case- study and user-centred) evaluation approach was taken to evaluate and validate the thesis. In addition a detailed analysis of the different aspects of the research is presented, outlining and addressing the identified challenges encountered by evaluators of adaptive systems.