The challenges of providing explanations of AI systems when they do not behave like users expect

Abstract

Explanations in artificial intelligence (AI) ensure that users of complex AI systems understand why the system behaves as it does. Expectations that users may have about the system behaviour play a role since they co-determine appropriate content of the explanations. In this paper, we investigate user-desired content of explanations when the system behaves in unexpected ways. Specifically, we presented participants with various scenarios involving an automated text classifier and then asked them to indicate their preferred explanation in each scenario. One group of participants chose the type of explanation from a multiple-choice questionnaire, the other had to answer using free text. Participants show a pretty clear agreement regarding the preferred type of explanation when the output matches expectations: most do not require an explanation at all, while those that do would like one that explains what features of the input led to the output (a factual explanation). When the output does not match expectations, users also prefer different explanations. Interestingly, there is less of an agreement in the multiple-choice questionnaire. However, the free text responses indicate slightly favour an explanation that describes how the AI system's internal workings led to the observed output (i.e., a mechanistic explanation). Overall, we demonstrate that user expectations are a significant variable in determining the most suitable content of explanations (including whether an explanation is needed at all). We also find different results, especially when the output does not match expectations, depending on whether participants answered via multiple-choice or free text. This shows a sensitivity to precise experimental setups that may explain some of the variety in the literature.