H2O-Danube3 Technical Report
arxiv(2024)
Abstract
We present H2O-Danube3, a series of small language models consisting of
H2O-Danube3-4B, trained on 6T tokens and H2O-Danube3-500M, trained on 4T
tokens. Our models are pre-trained on high quality Web data consisting of
primarily English tokens in three stages with different data mixes before final
supervised tuning for chat version. The models exhibit highly competitive
metrics across a multitude of academic, chat, and fine-tuning benchmarks.
Thanks to its compact architecture, H2O-Danube3 can be efficiently run on a
modern smartphone, enabling local inference and rapid processing capabilities
even on mobile devices. We make all models openly available under Apache 2.0
license further democratizing LLMs to a wider audience economically.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined