Hierarchical Variational Auto-Encoding for Unsupervised Domain Generalization

We address the task of domain generalization, where the goal is to train a predictive model such that it is able to generalize to a new, previously unseen domain. We choose a generative approach within the framework of variational autoencoders and propose an unsupervised algorithm that is able to generalize to new domains without supervision. We show that our method is able to learn representations that disentangle domain-specific information from class-label specific information even in complex settings where domain structure is not observed during training. Our interpretable method outperforms previously proposed generative algorithms for domain generalization and achieves competitive performance compared to state-of-the-art approaches, which rely on observing domain-specific information during training, on the standard domain generalization benchmark dataset PACS. Additionally, we proposed weak domain supervision which can further increase the performance of our algorithm in the PACS dataset. 1 Background and Motivation One big challenge of deploying a neural network model in real world use-cases is domain shift. In many real world applications, data seen by a deployed model is drawn from a distribution that is different from the training distribution and often unknown at train time. Domain Generalization aims at training a model from a set of domains (i.e. related distributions) such that the model is able to generalize to a new, unseen domain at test time. Domain generalization is relevant for a variety of tasks, ranging from personalized medicine, where each patient corresponds to a domain, to predictive maintenance in the context of industrial AI. In the latter usecase, domains can represent different factories where an industrial asset (e.g. a tool machine or a turbine) is operated, or different workers operating the asset. In addition to these discrete domains, domain shift can manifest itself in a continuous manner, where for example the data distribution seen by an industrial asset can change due to wear and tear or due to maintenance procedures. Similarly, domain sub-structures are not always observable during training due to data privacy concerns (in particular when patient data is used). In ∗LMU Munich, smilesun.east@gmail.com, work partially done during intern at Siemens AG. †Siemens AG, buettner.florian@siemens.com these latter scenarios, it is difficult to train standard domain generalization algorithms since they are based on the notion of clearly separable domains that are observable during model training. In many of these use cases, interpretability and human oversight of machine learning models is key. Generative models allow for learning disentangled representations that correspond to specific and interpretable factors of variation, thereby facilitating transparent predictions. We propose a new generative model that solves domain generalization problems in an interpretable manner without requiring domain labels during training. We build on previous work using autoencoder-based models for domain generalization [Kingma and Welling, 2013, Ilse et al., 2019] and propose a Hierarchical Domain Unsupervised Variational Auto-encoding that we refer to as HDUVA. Our major contributions include: • We present an unsupervised algorithm for domain generalization that is able to learn in setting with incomplete or hierarchical domain information. Our algorithm only need to use extended ELBO as model selection criteria, instead of relying on the validation set for early stopping. • Our method is able to learn representations that disentangle domain-specific information from classlabel specific information without domain supervision even in complex settings. • Our algorithm generates interpretable domain predictions that reveal connections between domains. • We constructed several hierarchical and sequential domain generalization benchmark datasets with doubly colored mnist for the domain generalization community.