Approximations to the Fisher Information Metric of Deep Generative Models for Out-Of-Distribution Detection
CoRR(2024)
摘要
Likelihood-based deep generative models such as score-based diffusion models
and variational autoencoders are state-of-the-art machine learning models
approximating high-dimensional distributions of data such as images, text, or
audio. One of many downstream tasks they can be naturally applied to is
out-of-distribution (OOD) detection. However, seminal work by Nalisnick et al.
which we reproduce showed that deep generative models consistently infer higher
log-likelihoods for OOD data than data they were trained on, marking an open
problem. In this work, we analyse using the gradient of a data point with
respect to the parameters of the deep generative model for OOD detection, based
on the simple intuition that OOD data should have larger gradient norms than
training data. We formalise measuring the size of the gradient as approximating
the Fisher information metric. We show that the Fisher information matrix (FIM)
has large absolute diagonal values, motivating the use of chi-square
distributed, layer-wise gradient norms as features. We combine these features
to make a simple, model-agnostic and hyperparameter-free method for OOD
detection which estimates the joint density of the layer-wise gradient norms
for a given data point. We find that these layer-wise gradient norms are weakly
correlated, rendering their combined usage informative, and prove that the
layer-wise gradient norms satisfy the principle of (data representation)
invariance. Our empirical results indicate that this method outperforms the
Typicality test for most deep generative models and image dataset pairings.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要