Why did the AI make that decision? Towards an explainable artificial intelligence (XAI) for autonomous driving systems

•End-to-end explainable model is developed for reasoning actions and explanations.•Traditional driving decision generation task is recast as image-captioning task to enable higher-level explainability.•Proposed fully Transformer-based model outperforms baseline models in both action and explanation predictions.•Application of the proposed explainable models is discussed from perspectives of users and AV developers.