Smaller Language Models are Better Black-box Machine-Generated Text Detectors
CoRR(2023)
Abstract
With the advent of fluent generative language models that can produce
convincing utterances very similar to those written by humans, distinguishing
whether a piece of text is machine-generated or human-written becomes more
challenging and more important, as such models could be used to spread
misinformation, fake news, fake reviews and to mimic certain authors and
figures. To this end, there have been a slew of methods proposed to detect
machine-generated text. Most of these methods need access to the logits of the
target model or need the ability to sample from the target. One such black-box
detection method relies on the observation that generated text is locally
optimal under the likelihood function of the generator, while human-written
text is not. We find that overall, smaller and partially-trained models are
better universal text detectors: they can more precisely detect text generated
from both small and larger models. Interestingly, we find that whether the
detector and generator were trained on the same data is not critically
important to the detection success. For instance the OPT-125M model has an AUC
of 0.81 in detecting ChatGPT generations, whereas a larger model from the GPT
family, GPTJ-6B, has AUC of 0.45.
MoreTranslated text
Key words
text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined