Prominent theories in cognitive science propose that humans understand and represent the
knowledge of the world through causal relationships. In making sense of the world, we build
causal models in our mind to encode cause-effect relations of events and use these to
explain why new events happen by referring to counterfactuals—things that did not happen.
In this paper, we use causal models to derive causal explanations of the behaviour of model-
free reinforcement learning agents. We present an approach that learns a structural causal …

This thesis investigates methods for explaining and understanding how and why
reinforcement learning agents select actions, from a causal perspective. Understanding the
behaviours, decisions and actions exhibited by arti cially intelligent agents has been a
central theme of interest since the inception of agent research. As systems grow in
complexity, the agents' underlying reasoning mechanisms can become opaque and the
intelligibility towards humans can be diminished, which can have negative consequences in …