PhoGPT: Generative Pre-training for Vietnamese
CoRR(2023)
摘要
We open-source a state-of-the-art 4B-parameter generative model series for
Vietnamese, which includes the base pre-trained monolingual model PhoGPT-4B and
its chat variant, PhoGPT-4B-Chat. The base model, PhoGPT-4B, with exactly 3.7B
parameters, is pre-trained from scratch on a Vietnamese corpus of 102B tokens,
with an 8192 context length, employing a vocabulary of 20480 token types. The
chat variant, PhoGPT-4B-Chat, is the modeling output obtained by fine-tuning
PhoGPT-4B on a dataset of 70K instructional prompts and their responses, along
with an additional 290K conversations. We demonstrate its strong performance
compared to previous closed-source and open-source 7B-parameter models. Our
PhoGPT models are available at: https://github.com/VinAIResearch/PhoGPT
更多查看译文
关键词
pre-training
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要