Nyonic Technical Report
arxiv(2024)
摘要
This report details the development and key achievements of our latest
language model designed for custom large language models. The advancements
introduced include a novel Online Data Scheduler that supports flexible
training data adjustments and curriculum learning. The model's architecture is
fortified with state-of-the-art techniques such as Rotary Positional
Embeddings, QK-LayerNorm, and a specially crafted multilingual tokenizer to
enhance stability and performance. Moreover, our robust training framework
incorporates advanced monitoring and rapid recovery features to ensure optimal
efficiency. Our Wonton 7B model has demonstrated competitive performance on a
range of multilingual and English benchmarks. Future developments will
prioritize narrowing the performance gap with more extensively trained models,
thereby enhancing the model's real-world efficacy and adaptability.GitHub:
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要