On the Necessity of Disentangled Representations for Downstream Tasks

A disentangled representation encodes generative factors of data in a separable and compact pattern. Thus it is widely believed that such a representation format benefits downstream tasks. In this paper, we challenge the necessity of disentangled representation in downstream applications. Specifically, we show that dimension-wise disentangled representations are not necessary for downstream tasks using neural networks that take learned representations as input. We provide extensive empirical evidence against the necessity of disentanglement, covering multiple datasets, representation learning methods, and downstream network architectures. Moreover, our study reveals that the informativeness of representations best accounts for downstream performance. The positive correlation between informativeness and disentanglement explains the claimed usefulness of disentangled representations in previous works.