Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text
Date:
Abstract
In this talk I describe our work investigating the ability of humans to detect when text was generated by large language models. Through the use of our website https://roft.io we conduct the largest-ever study of human detectability and release a dataset of over 20,000 human annotations of generated text. We find that while humans are not typically good at detection there is substantial variance in their skill with some participants being much more accurate than others. We find that, using our site, participants were able to improve their detection skill over time and that this finding held across LMs and genres. Finally, we show which models were most effective at fooling our participants and which errors were most useful for accurate detection.
Location
This talk was given on February 2nd 2023 at the Walter E. Washington Convention Center in Washington D.C. as part of the 37th AAAI Conference on Artificial Intelligence. A version of this talk was also given on March 2nd 2023 at Brown University in Providence, RI.