Human-in-the-Loop Synthetic Text Data Inspection with Provenance Tracking
arxiv(2024)
Abstract
Data augmentation techniques apply transformations to existing texts to
generate additional data. The transformations may produce low-quality texts,
where the meaning of the text is changed and the text may even be mangled
beyond human comprehension. Analyzing the synthetically generated texts and
their corresponding labels is slow and demanding. To winnow out texts with
incorrect labels, we develop INSPECTOR, a human-in-the-loop data inspection
technique. INSPECTOR combines the strengths of provenance tracking techniques
with assistive labeling. INSPECTOR allows users to group related texts by their
transformation provenance, i.e., the transformations applied to the original
text, or feature provenance, the linguistic features of the original text. For
assistive labeling, INSPECTOR computes metrics that approximate data quality,
and allows users to compare the corresponding label of each text against the
predictions of a large language model. In a user study, INSPECTOR increases the
number of texts with correct labels identified by 3X on a sentiment analysis
task and by 4X on a hate speech detection task. The participants found grouping
the synthetically generated texts by their common transformation to be the most
useful technique. Surprisingly, grouping texts by common linguistic features
was perceived to be unhelpful. Contrary to prior work, our study finds that no
single technique obviates the need for human inspection effort. This validates
the design of INSPECTOR which combines both analysis of data provenance and
assistive labeling to reduce human inspection effort.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined