Bridging Big Data: Procedures for Combining Non-equivalent Cognitive Measures from the ENIGMA Consortium

Investigators in the cognitive neurosciences have turned to Big Data to address persistent replication and reliability issues by increasing sample sizes, statistical power, and representativeness of data. While there is tremendous potential to advance science through open data sharing, these efforts unveil a host of new questions about how to integrate data arising from distinct sources and instruments. We focus on the most frequently assessed area of cognition - memory testing - and demonstrate a process for reliable data harmonization across three common measures. We aggregated raw data from 53 studies from around the world which measured at least one of three distinct verbal learning tasks, totaling N = 10,505 healthy and brain-injured individuals. A mega analysis was conducted using empirical bayes harmonization to isolate and remove site effects, followed by linear models which adjusted for common covariates. After corrections, a continuous item response theory (IRT) model estimated each individual subjects latent verbal learning ability while accounting for item difficulties. Harmonization significantly reduced inter-site variance by 37% while preserving covariate effects. The effects of age, sex, and education on scores were found to be highly consistent across memory tests. IRT methods for equating scores across AVLTs agreed with held-out data of dually-administered tests, and these tools are made available for free online. This work demonstrates that large-scale data sharing and harmonization initiatives can offer opportunities to address reproducibility and integration challenges across the behavioral sciences. ### Competing Interest Statement Dr. Arango has been a consultant to or has received honoraria or grants from Acadia, Angelini, Biogen, Boehringer, Gedeon Richter, Janssen Cilag, Lundbeck, Medscape, Menarini, Minerva, Otsuka, Pfizer, Roche, Sage, Servier, Shire, Schering Plough, Sumitomo Dainippon Pharma, Sunovion and Takeda. Dr. Brodtmann serves on the editorial boards of Neurology and International Journal of Stroke. Dr. Diaz-Caneja has received honoraria from Exeltis and Angelinii. Dr. Giza: consultant for NBA, NFL, NHLPA, Los Angeles Lakers; Advisory Board: Highmark Interactive, Novartis, MLS, NBA, USSF; Medicolegal 1-2 cases annually. Dr. Soares: ALKERMES (Research Grant), ALLERGAN (Research Grant), ASOFARMA (Consultant), ATAI (Stock), BOEHRINGER Ingelheim (Consultant), COMPASS (Research Grant), JOHNSON & JOHNSON (Consultant), LIVANOVA (Consultant), PFIZER (Consultant), PULVINAR NEURO LLC (Consultant), RELMADA (Consultant), SANOFI (Consultant), SUNOVIAN (Consultant). Dr. Thompson received partial research support from Biogen, Inc., for research unrelated to this manuscript. Dr. Yatham has been on speaker or advisory boards for, or has received research grants from, Alkermes, Abbvie, Canadian Institutes of Health Research, Sumitomo Dainippon Pharma, GlaxoSmithKline, Intracellular Therapies, Merck, Sanofi, Sequiris, Servier, and Sunovion, over the past 3 years, all outside this work. The collection of this cohort was partially supported by an investigator-initiated research grant from Biogen (US). Biogen had no role in the analysis or writing of this manuscript. Eisai (JP) and Life Molecular Imaging for research unrelated to this manuscript. Dr. Wylie has received research support from the NJ Commission for brain injury research, from the Dept of Veterans' Affairs, from Biogen, from Bristol, Myers, Squibb, from Genetech, and has served on advisory boards for the CDMRP and the VA. All of these activities are unrelated to this research. The views expressed in this article are those of the author(s) and do not reflect the official policy of the Department of Army/Navy/Air Force, Department of Defense, or U.S. Government.