AI-Assisted Human Evaluation of Machine Translation
CoRR(2024)
Abstract
Annually, research teams spend large amounts of money to evaluate the quality
of machine translation systems (WMT, inter alia). This is expensive because it
requires detailed human labor. The recently proposed annotation protocol, Error
Span Annotation (ESA), has annotators marking erroneous parts of the
translation. In our work, we help the annotators by pre-filling the span
annotations with automatic quality estimation. With AI assistance, we obtain
more detailed annotations while cutting down the time per span annotation by
half (71s/error span → 31s/error span). The biggest advantage of
ESA^AI protocol is an accurate priming of annotators (pre-filled
error spans) before they assign the final score as opposed to starting from
scratch. In addition, the annotation budget can be reduced by up to 24
filtering of examples that the AI deems to be very likely to be correct.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined