Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022
ACM Transactions on Graphics(2023)
Abstract
This paper reports on the second GENEA Challenge to benchmark data-driven
automatic co-speech gesture generation. Participating teams used the same
speech and motion dataset to build gesture-generation systems. Motion generated
by all these systems was rendered to video using a standardised visualisation
pipeline and evaluated in several large, crowdsourced user studies. Unlike when
comparing different research papers, differences in results are here only due
to differences between methods, enabling direct comparison between systems. The
dataset was based on 18 hours of full-body motion capture, including fingers,
of different persons engaging in a dyadic conversation. Ten teams participated
in the challenge across two tiers: full-body and upper-body gesticulation. For
each tier, we evaluated both the human-likeness of the gesture motion and its
appropriateness for the specific speech signal. Our evaluations decouple
human-likeness from gesture appropriateness, which has been a difficult problem
in the field.
The evaluation results show some synthetic gesture conditions being rated as
significantly more human-like than 3D human motion capture. To the best of our
knowledge, this has not been demonstrated before. On the other hand, all
synthetic motion is found to be vastly less appropriate for the speech than the
original motion-capture recordings. We also find that conventional objective
metrics do not correlate well with subjective human-likeness ratings in this
large evaluation. The one exception is the Fréchet gesture distance (FGD),
which achieves a Kendall's tau rank correlation of around -0.5. Based on the
challenge results we formulate numerous recommendations for system building and
evaluation.
MoreTranslated text
Key words
animation,gesture generation,embodied conversational agents,evaluation paradigms
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined