Tradeoffs of Diagonal Fisher Information Matrix Estimators
CoRR(2024)
摘要
The Fisher information matrix characterizes the local geometry in the
parameter space of neural networks. It elucidates insightful theories and
useful tools to understand and optimize neural networks. Given its high
computational cost, practitioners often use random estimators and evaluate only
the diagonal entries. We examine two such estimators, whose accuracy and sample
complexity depend on their associated variances. We derive bounds of the
variances and instantiate them in regression and classification networks. We
navigate trade-offs of both estimators based on analytical and numerical
studies. We find that the variance quantities depend on the non-linearity with
respect to different parameter groups and should not be neglected when
estimating the Fisher information.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要