Counterfactual and Semifactual Explanations for Reinforcement Learning

Abstract

Reinforcement learning (RL) is a trial-and-error machine learning paradigm where an agent interacts with an environment and receives rewards for its actions. RL aims to learn an optimal policy – a mapping from the observed states to actions maximizing the long-term reward. Deep reinforcement learning (DRL) algorithms use neural networks to learn the policy, increasing their scalability. However, the opacity of neural networks makes the decisions of DRL agents difficult to understand and hinders their applicability to high-risk tasks. Explainable RL (xRL) investigates methods for explaining the behaviour of RL agents.

Counterfactuals and semifactuals are powerful user-friendly explanations. Counterfactuals explain an outcome by contrasting it with the closest possible world – a similar scenario resulting in a different outcome. Semifactuals reinforce an outcome by offering an alternative scenario with the same outcome, in the form of a most distant neighbour. While extensively researched in supervised learning, there are only a few approaches developing these explanations for RL. Moreover, existing approaches in RL use the same feature-based metrics as used in supervised learning to select, evaluate, and search for these explanations. However, RL differs from supervised learning in its sequential and stochastic execution. Given the differences between the two frameworks, a question arises if the same definitions and methods for generating these explanations can be directly translated from supervised to RL. In this thesis, we aim to develop counterfactual and semifactual explanations for RL, by redefining the concepts of the closest possible world and the most distant neighbour in stochastic and sequential tasks and offering algorithms for counterfactual and semifactual search.

The first contribution of this thesis is an in-depth analysis of the two frameworks which identifies: 1) sequential execution 2) stochastic conditions, and 3) complex explanation types as the three main differences between supervised and RL frameworks, concluding that the same definitions of counterfactual and semifactual explanations cannot be directly translated to RL. We propose a set of five RL-specific counterfactual requirements (validity, reachability, recency, plausibility and certainty of outcome) and six RL-specific semifactual requirements (validity, reachability, recency, plausibility, unexpectedness, and argument strength) that ensure informative explanations in RL. We formally redefine counterfactual and semifactual explanations for stochastic and sequential RL tasks based on the defined requirements.

The second contribution of this thesis is the implementation of RACCER, the first RL-specific algorithm for counterfactual generation. RACCER implements a set of metrics – validity, temporal distance, fidelity and stochastic certainty to evaluate the counterfactual requirements. Additionally, RACCER implements three algorithms to optimize these metrics – RACCER-HTS (using heuristic tree search), RACCER-Advance and RACCER-Rewind (based on evolutionary algorithms).

The third contribution of this thesis is the implementation of SGRL, the first semifactual generation algorithm for RL. SGRL implements five metrics – validity, temporal distance, fidelity, stochastic uncertainty and exceptionality that can be used to evaluate the RL-specific semifactual requirements. Additionally, SGLR implements two algorithms SGRL-Advance and SGRL-Rewind which use evolutionary algorithms to optimize these metrics and find semifactuals.

We evaluate RACCER and SGRL in five RL environments. Firstly, we evaluate them in standard, increasingly complex benchmark environments: frozen lake, stochastic grid-world, highway driving, and farm environment. We also conduct a case study using CitiBikes, a real-world simulated bike-sharing task. We find that RACCER and SGRL perform better on RL-specific counterfactual and semifactual metrics, producing more diverse and realistic explanations than the baseline approaches. Through a user study, we also find that counterfactual explanations generated by RACCER help users better understand the behaviour of RL agents compared to the counterfactuals generated by a baseline method. In contrast, a user study conducted on semifactual explanations finds that, while users rate highly explanations generated by SGRL, no significant impact has been found on users’ perception and understanding of the agent compared to the baseline. A case study in large-scale CitiBikes tasks finds that while RACCER and SGRL approaches generate more realistic explanations, they struggle to scale to environments with large action spaces.