Pattern Discovery and Disentanglement: A New Interpretable Machine Learning Paradigm for Relational Datasets

Abstract In machine learning (ML) on relational datasets, association patterns in the data, paths in decision trees, and weights between layers of the neural network coming from multiple underlying sources are often entangled, masking the pattern-to-source relation, weakening prediction and defying explanation. This paper presents a revolutionary ML paradigm: Pattern Discovery and Disentanglement (PDD), which disentangles associations and provides an All-in-One knowledge framework and computational platform, capable of a) disentangling patterns to associate with distinct sources/classes; b) discovering rare/imbalanced groups, detecting anomalies and rectifying discrepancies to improve class association, pattern/entity clustering; c) organizing knowledge for interpretability/traceability for causal exploration with statistical support. Results from base studies validate such capabilities. The explainable knowledge reveals pattern-source relations, entity characteristics and underlying factors for causal inference, clinical study, and practice, addressing the major concern of interpretability, trust and reliability when applying ML to healthcare --- a step towards closing the AI chasm.