Breaking the Curse of Dimensionality with Distributed Neural Computation
CoRR(2024)
摘要
We present a theoretical approach to overcome the curse of dimensionality
using a neural computation algorithm which can be distributed across several
machines. Our modular distributed deep learning paradigm, termed neural
pathways, can achieve arbitrary accuracy while only loading a small number of
parameters into GPU VRAM. Formally, we prove that for every error level
ε>0 and every Lipschitz function f:[0,1]^n→ℝ, one can
construct a neural pathways model which uniformly approximates f to
ε accuracy over [0,1]^n while only requiring networks of
𝒪(ε^-1) parameters to be loaded in memory and
𝒪(ε^-1log(ε^-1)) to be loaded during the
forward pass. This improves the optimal bounds for traditional non-distributed
deep learning models, namely ReLU MLPs, which need
𝒪(ε^-n/2) parameters to achieve the same accuracy. The
only other available deep learning model that breaks the curse of
dimensionality is MLPs with super-expressive activation functions. However, we
demonstrate that these models have an infinite VC dimension, even with bounded
depth and width restrictions, unlike the neural pathways model. This implies
that only the latter generalizes. Our analysis is validated experimentally in
both regression and classification tasks, demonstrating that our model exhibits
superior performance compared to larger centralized benchmarks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要