Response to "Comment on 'A Permutation Test-Based Approach to Strengthening Inference on the Effects of Environmental Mixtures: Comparison between Single-Index Analytic Methods'".

Vol. 131, No. 1 Response to LetterOpen AccessResponse to “Comment on ‘A Permutation Test-Based Approach to Strengthening Inference on the Effects of Environmental Mixtures: Comparison between Single-Index Analytic Methods’”is accompanied byA Permutation Test-Based Approach to Strengthening Inference on the Effects of Environmental Mixtures: Comparison between Single-Index Analytic Methodsis a reply to letterComment on “A Permutation Test-Based Approach to Strengthening Inference on the Effects of Environmental Mixtures: Comparison between Single-Index Analytic Methods” Drew B. Day, Sheela Sathyanarayana, Kaja Z. LeWinn, Catherine J. Karr, W. Alex Mason, and Adam A. Szpiro Drew B. Day Address correspondence to Drew B. Day, Center for Child Health, Behavior, and Development, Seattle Children’s Research Institute, M/S Cure-3, P.O. Box 5371, Seattle, WA 98145 USA. Telephone: (206) 884-1798. Email: E-mail Address: [email protected] https://orcid.org/0000-0003-2606-2050 Center for Child Health, Behavior, and Development, Seattle Children’s Research Institute, Seattle, Washington, USA Search for more papers by this author , Sheela Sathyanarayana Center for Child Health, Behavior, and Development, Seattle Children’s Research Institute, Seattle, Washington, USA Department of Pediatrics, University of Washington, Seattle, Washington, USA Department of Environmental and Occupational Health Sciences, University of Washington, Seattle, Washington, USA Search for more papers by this author , Kaja Z. LeWinn Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, California, USA Search for more papers by this author , Catherine J. Karr Department of Pediatrics, University of Washington, Seattle, Washington, USA Department of Environmental and Occupational Health Sciences, University of Washington, Seattle, Washington, USA Department of Epidemiology, University of Washington, Seattle, Washington, USA Search for more papers by this author , W. Alex Mason Department of Preventive Medicine, University of Tennessee Health Sciences Center, Memphis, Tennessee, USA Search for more papers by this author , and Adam A. Szpiro Department of Biostatistics, University of Washington, Seattle, Washington, USA Search for more papers by this author Published:3 January 2023CID: 018002https://doi.org/10.1289/EHP12517AboutSectionsPDF ToolsDownload CitationsTrack Citations ShareShare onFacebookTwitterLinked InReddit We appreciate the careful consideration by Keil et al. of our paper1 comparing single-index mixture exposure models in simulations, including a novel permutation test for weighted quantile sum regression (WQSr) and quantile g-computation (QGC). We agree that mean absolute error (MAE) and mean absolute percent error (MAPE) assess accuracy and not bias. Therefore, we should have stated that the component-specific coefficients estimated by QGC were far less accurate that those of the WQSr models in our simulations. We appreciate the opportunity to make this correction.However, we differ with Keil et al.’s characterization of QGC being more accurate than WQSr in their simulation scenario with 1 signal and 13 noise components. We reproduced this simulation using the permutation test version of WQSr instead of the training/test split version, which we recommended against using in our paper due to low power. Signal component MAPE was 11.6% for both WQSr and QGC, and MAE for all components was 0.018 for WQSr and 0.026 for QGC, indicating that WQSr is more accurate overall.Keil et al. noted that WQSr is well-suited to the condition of unidirectional mixture effects used in our simulations.1 It remains unclear how it will perform when run in both directions for the case of bidirectional mixture effects. Similarly, performance of QGC has not been characterized with complex bidirectional mixture effects having more than two signal components.Keil et al. argued our treatment of failed WQSr iterations (i.e., those with no detectable signal in the positive direction) underestimated error. Imputing zero for failed WQSr iterations accurately represents the result but may underestimate error for versions of WQSr with high failure rates, such as repeated holdout (RH). We adopted this approach partly to ensure a fair evaluation of the RH versions of WQSr, although we recommended avoiding these models because of their high failure rates. We recommended the permutation test WQSr model, for which less than or equal to 1 percent≤1% of simulations failed, so our conclusions would not be affected by alternative approaches.In our alcohol/caffeine mixture analogy, Keil et al. noted that the exposures counteract each other and that QGC can provide that information in addition to direction-specific effects. However, it is not straightforward to characterize beneficial and adverse effects using QGC in complex situations with more than two components, such as a mixture of endocrine-disrupting chemicals with pro- and antiandrogenic mechanisms. It would be difficult to ascertain the beneficial and adverse effects of such a mixture with QGC.WQSr and QGC are both appealing choices depending on the aims of a given analysis. QGC is well-suited to estimating an overall (i.e., positive plus negative) mixture effect. It also provides estimates of individual component coefficients, although our simulations suggest these may be inaccurate. WQSr is well suited to providing separate estimates of the beneficial and adverse (i.e., positive and negative) effects of the mixture. We believe that in many contexts it is critical to separately detect and characterize beneficial and adverse mixture effects, which is why we recommended WQSr with a permutation test to maximize power.AcknowledgmentsThis research was supported by National Institutes of Health grants 1UG3OD023271-01 and 4UH3OD023271-03. Additional funding for the Conditions Affecting Neurocognitive Development and Learning in Early Childhood (CANDLE) study was provided by the Urban Child Institute.Reference1. Day DB, Sathyanarayana S, LeWinn KZ, Karr CJ, Mason WA, Szpiro AA. 2022. Permutation test-based approach to strengthening inference on the effects of environmental mixtures: comparison between single-index analytic methods. Environ Health Perspect 130(8):87010, PMID: 36040702, 10.1289/EHP10570. Link, Google ScholarThe authors declare they have no actual or potential competing financial interests.FiguresReferencesRelatedDetailsRelated articlesA Permutation Test-Based Approach to Strengthening Inference on the Effects of Environmental Mixtures: Comparison between Single-Index Analytic Methods30 August 2022Environmental Health PerspectivesComment on “A Permutation Test-Based Approach to Strengthening Inference on the Effects of Environmental Mixtures: Comparison between Single-Index Analytic Methods”3 January 2023Environmental Health Perspectives Vol. 131, No. 1 January 2023Metrics About Article Metrics Publication History Manuscript received30 November 2022Manuscript accepted13 December 2022Originally published3 January 2023 Financial disclosuresPDF download License information EHP is an open-access journal published with support from the National Institute of Environmental Health Sciences, National Institutes of Health. All content is public domain unless otherwise noted. Note to readers with disabilities EHP strives to ensure that all journal content is accessible to all readers. However, some figures and Supplemental Material published in EHP articles may not conform to 508 standards due to the complexity of the information being presented. If you need assistance accessing journal content, please contact [email protected]. Our staff will work with you to assess and meet your accessibility needs within 3 working days.