Explainability in process outcome prediction: Guidelines to obtain interpretable and faithful models

•Comparative analysis of state-of-the art models for process outcome prediction.•Explainability defined by its properties interpretability and faithfulness.•Model-agnostic explainability metrics tailored to the dimensions of process data.•Guidelines to obtain eXplainable Models for Outcome Prediction (X-MOP).