On Least Squares Estimation in Softmax Gating Mixture of Experts
CoRR(2024)
摘要
Mixture of experts (MoE) model is a statistical machine learning design that
aggregates multiple expert networks using a softmax gating function in order to
form a more intricate and expressive model. Despite being commonly used in
several applications owing to their scalability, the mathematical and
statistical properties of MoE models are complex and difficult to analyze. As a
result, previous theoretical works have primarily focused on probabilistic MoE
models by imposing the impractical assumption that the data are generated from
a Gaussian MoE model. In this work, we investigate the performance of the least
squares estimators (LSE) under a deterministic MoE model where the data are
sampled according to a regression model, a setting that has remained largely
unexplored. We establish a condition called strong identifiability to
characterize the convergence behavior of various types of expert functions. We
demonstrate that the rates for estimating strongly identifiable experts, namely
the widely used feed forward networks with activation functions
sigmoid(·) and tanh(·), are substantially faster than
those of polynomial experts, which we show to exhibit a surprising slow
estimation rate. Our findings have important practical implications for expert
selection.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要