Distinguishing the Correctness of Knowledge Makes Knowledge Transfer Better

Using the Large Language Model(LLM) for in-context learning to solve Natural Language Processing(NLP) tasks has become one of the most popular and effective methods. There has been a sea of works to elicit knowledge from LLM to perform commonsense reasoning, but those methods consume a lot of time and space. Our work considers how to transfer the generated knowledge from LLM to a small model for knowledge generation. We find that the incorporation of wrong knowledge is important in knowledge transfer, which has been neglected by previous work. We propose different filter methods to different generated knowledge to distinguish the correctness of knowledge, and use both correct and wrong knowledge in Contrastive-Learning for knowledge transfer to improve the ability of small models to generate knowledge. In this paper, we first figure out what kind of prompts in in-context learning can better motivate LLM to generate knowledge that has higher generalization and is more helpful in answering questions. Then, we compare various filtering methods for knowledge correctness determination. At last, we use Contrastive-Learning based knowledge generation for transferring knowledge from LLM to the small model. In this way, the knowledge generated by the small model are not only richer but also more correct, which boost reasoning tasks with performance improved up to 1.7% on the CommonsenseQA and 3.2% on the OpnebookQA comparing the knowledge generated by simply fine-tuned on all knowledge.