Distractor re-ranking for automatic quiz generation

Automatic quiz generation tackles the problem of generating questions from free-form texts. An important part of this process is generating good distractors as multiple choice options. Current methods rely on similarity search methods using unsupervised word2vec method. In our project, we leverage existing multiple choice choice questions written in textbooks to add a layer of supervision to existing similarity search methods. First, an unsupervised word2vec model is used to extract an initial list of candidates. Second, a supervised re-ranker, trained on the above-mentioned textbook questions, is used to choose the top-k multiple choice distractors. Three ranking approaches were explored A point-wise ranking SVM, a list-wise ranking neural network and a ranking Generative Adversarial Network (GAN). We evaluate our methods on the SciQ dataset. Qualitative results show that our models are able to generate good distractors and our quantitative results showed that the list-wise ranking neural net performed the best. Future work will focus on improving the syntactic match of distractors and dealing with cases where the chosen distractors could be correct answers to the question.