A New Evaluation Method: Evaluation Data and Metrics for Chinese Grammatical Error Correction

Abstract As a fundamental task in natural language processing (NLP), Chinese Grammatical Error Correction (CGEC) [1–3] has gradually received widespread attention and become a research hotspot. However, one obvious deficiency of the existing CGEC evaluation systems is that the evaluation values of the same error correction models are signif- icantly influenced by the Chinese word segmentation (CWS) results or different language models. However, it is expected that these met- rics should be independent of the CWS results and language models for a fair evaluation. To this end, we propose three novel eval- uation metrics for CGEC in two dimensions: reference-based and reference-less. What’s more, according to these three evaluation met- rics, we build a new evaluation metric that can comprehensively evaluate the CGEC model from multiple dimensions. We deeply eval- uate and analyze the reasonableness and validity of the proposed metrics, and we expect them to become a new standard for CGEC.