ALTO: An Efficient Network Orchestrator for Compound AI Systems
arxiv(2024)
摘要
We present ALTO, a network orchestrator for efficiently serving compound AI
systems such as pipelines of language models. ALTO achieves high throughput and
low latency by taking advantage of an optimization opportunity specific to
generative language models: streaming intermediate outputs. As language models
produce outputs token by token, ALTO exposes opportunities to stream
intermediate outputs between stages when possible. We highlight two new
challenges of correctness and load balancing which emerge when streaming
intermediate data across distributed pipeline stage instances. We also motivate
the need for an aggregation-aware routing interface and distributed
prompt-aware scheduling to address these challenges. We demonstrate the impact
of ALTO's partial output streaming on a complex chatbot verification pipeline,
increasing throughput by up to 3x for a fixed latency target of 4 seconds /
request while also reducing tail latency by 1.8x compared to a baseline serving
approach.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要