Evaluation is Key: A Survey on Evaluation Measures for Synthetic Time Series

Abstract Synthetic data generation describes the process of learning the underlying distribution of a given real dataset in a model, which is, in turn, sampled to produce new data objects still adhering to the original distribution. This approach often finds application where circumstances limit the availability or usability of real-world datasets, for instance, in health care due to privacy concerns. While image synthesis has received much attention in the past, time series are arguably even more relevant for many practical (e.g., industrial) applications. By now, numerous different generative models and measures to evaluate time series syntheses have been proposed. However, when it comes to what characterizes high-quality synthetic time series and how to quantify quality, no consensus has yet been reached among researchers. Hence, we propose this comprehensive survey on evaluation measures for time series generation to assist users in evaluating synthetic time series. We provide brief descriptions or - where applicable - precise definitions and also a multidimensional analysis of their properties, applicability, and usage. In order to facilitate the selection of the most suitable measures, we provide a quick guide combined with many tables and figures. Notably, during our study, we found that there is currently no generally accepted approach for an evaluation procedure, including what measures to use. We believe this situation hinders progress and may even erode evaluation standards to a ``do as you like''-approach to synthetic data evaluation. Therefore, this survey is a preliminary step to advance the field of synthetic data evaluation.