Evaluation for Scenario Question Answering Systems

Scenario Question Answering is a relatively new direction in Question Answering (QA) research that presents a number of challenges for evaluation. In this paper, we propose a comprehensive evaluation strategy for Scenario QA, including a methodology for building reusable test collections for Scenario QA and metrics for evaluating system performance over such test collections. Using this methodology, we have built a test collection, which we have made available for public download as a service to the research community. It is our hope that widespread availability of quality evaluation materials fuels research in new a pproaches to the Scenario QA task. In this paper, we discuss the unique evaluation challenges associated with Scenario QA, a form of Question Answer- ing where the user input can include background informa- tion and questions with multiple parts, representing a com- plex information need. We propose an evaluation strat- egy and metrics for the Scenario QA task, and present a methodology for building a Scenario QA test collection. We report on a successful application of this process at our site and demonstrate how to evaluate Scenario QA system responses with the test collection we have built. Our test collection is available for public download for research pur- poses, and constitutes our contribution to evaluation mate- rials for the community at large. As access to quality eval- uation for Scenario QA improves, we hope to see an accel- eration in research into the Scenario QA task.