How to quantify between-study heterogeneity in single-group evidence synthesis? - It depends!

Abstract Background: Random-effects meta-analysis models account for between-study heterogeneity by estimating and incorporating the heterogeneity variance parameter tau2. Numerous estimators for tau2 have been proposed, but no widely accepted guidance exists on when to best use which meta-analysis method. Especially in the context of observational studies, systematic evaluations and comparisons of the various meta-analysis methods are lacking. Since between-study heterogeneity is of crucial importance, particularly in meta-analysis of observational studies, where control groups typically do not exist, considerable attention should be paid to its estimation. This study aims to investigate the advantages of different meta-analysis methods for typical situations through comprehensive simulations in a neutral comparison study and with an empirical application. Methods: We compared seven methods for random-effects meta-analysis. The methods were selected with a focus on methodological diversity and availability and were evaluated both empirically and in a simulation study. We simulated typical meta-analysis scenarios for continuous and binary outcomes in a single-group meta-analysis setting. Results: Specific study characteristics, such as the number of studies included in a meta-analysis, the amount of heterogeneity in the data and binary outcome data with rare events, strongly affected the performance of the heterogeneity variance estimator. Moreover, we discovered that most heterogeneity variance estimators produce zero heterogeneity estimates under all simulated conditions, even though heterogeneity was present. The estimated overall effect was found to be relatively robust regarding different methods in the empirical application and in our simulation study. Conclusions: Although different meta-analysis methods produce substantially different heterogeneity variance estimates, too little attention is paid to selecting a suitable meta-analysis method in research applications. Based on our literature review, we conclude that the awareness about different heterogeneity variance estimators and their properties needs to be strengthened in practice. Our simulation study showed that all evaluated heterogeneity estimators were imprecise and often failed to estimate the true amount of heterogeneity. The estimation is particularly imprecise in situations where the meta-analysis contained few studies or when the binary outcomes included rare events. As it is rarely appropriate to rely on a single heterogeneity variance estimator, therefore we suggest careful consideration and evaluation of a wider range of plausible estimators in a sensitivity analysis before drawing a final conclusion about the meta-analysis results.