How People Understand Causal and Counterfactual Explanations in the Context of eXplainable Artificial Intelligence (XAI)

Abstract

In this thesis, we present the results of 13 experiments (n = 2,756) investigating the cognitive processes underlying causal explanation evaluation and the role of counterfactual explanations during explanation selection and risky decision-making. The experiments examined key questions in cognitive psychology relevant to understanding explanations, especially counterfactual explanations. They also explored ways to test and generate better explanations of potential relevance for the field of eXplainable Artificial Intelligence (XAI). We address three questions: What information influences people's selection of causal explanations? What are the underlying cognitive processes when people read counterfactual explanations about how things could have turned out differently? What types of tasks and explanations are helpful in people's understanding and satisfaction with automated decisions? The first six experiments (Experiments 1, 2a, 2b, 3a, 3b and 4) investigated a novel strategy people follow when choosing between competing causal explanations. The results showed that people preferred explanations with consequent-controllable elements, such as "the symptoms are caused by a disease that can be treated with medication", than explanations without consequent-controllable elements, such as "the symptoms are caused by a disease that has no known treatment". These experiments also showed simplicity had a stronger impact on causal explanation selection, as we found that participants preferred simple explanations without consequent-controllable elements over complex explanations with consequent-controllable elements. Moreover, we showed that counterfactual explanations can reduce people's strong preference for the simple explanation. These findings contributed to the debate about what guides the cognitive processes underlying causal explanation selection. The results also corroborated the mental model theory of counterfactual thinking. The second series of two experiments (Experiments 5 and 6) showed that in a mock AI decision support app setting, counterfactual explanations (about how things could have turned out differently in the past) and prefactual explanations (about how things could turn out differently in the future) were equally effective in helping human users understand the app. They were also equally effective in terms of user satisfaction with the explanations. The results also showed that there was a difference in the objective measurement of accuracy when participants were asked to make predictive inferences (from causes to effects) compared to diagnostic inferences (from effects to causes). These findings added to the current methods of user testing and showed the importance of testing different explanation types and task types in XAI. The third series of five experiments (Experiments 7 to 11) showed that for risky decisions (in the risky choice framing effect scenario), participants switched from the typical decision to a non-typical one proposed by an Artificial Intelligence (AI) system when they were given a counterfactual explanation. Their baseline tendency to switch was low when the non-typical decision was proposed by an AI system compared to a human expert. The results corroborated previous findings on algorithmic aversion. They also supported the mental model theory of counterfactual thinking, as participants were able to consider alternatives when given counterfactual explanations. All 13 experiments further contribute to our understanding of the cognitive processes underlying explanation evaluation and the proposition that counterfactual explanations prompt people to simulate alternative realities. The results have important implications for the psychological understanding of causal explanation and counterfactual thinking. The implications also extend to the current user testing methods and assumption of what makes a good explanation in XAI.