Response to A. Eleuteri regarding "A radiomics-based decision support tool improves lung cancer diagnosis in combination with the Herder score in large lung nodules"

Dear Editor, We would like to thank Dr. Eleuteri for such a detailed letter providing statistical feedback on our publication. The efforts to improve the statistical analysis of predictive models are to be commended, and we hope that the letter proves helpful for the research community. Whilst thorough statistical and bioinformatics input was obtained throughout the study design, set up and analysis, we acknowledge that statistical approaches can generate significant debate, and expertise spanning traditional statistics and machine-learning is still evolving at many institutions. As Dr. Eleuteri points out, there are known limitations to the split-sample methodology, including loss of training data. However, the approach remains widely-adopted by the machine-learning community as a useful method for estimating performance on unseen data. Indeed, many studies use three splits – training (for model fitting), validation (for hyperparameter tuning) and testing (for internal validation/estimation of performance on unseen data), which we agree seems unnecessary when cross-validation is employed. However, we maintain that a test set split was important, as although an external test set was available, it was obtained from public repositories without clinical data and would not have allowed comparisons against the Brock and Herder scores on unseen data. We thank Dr. Eleuteri for also raising the issue of data clustering. Although we looked for effects of scan vendor on the LN-RPV and found no significant interaction, we did not perform in-depth covariance clustering analysis. Whilst this omission could have implications for our study findings, it does not necessarily invalidate them. He mentions the worst-case scenario of type-1 errors, but given that all patients came from within the same healthcare service, it is reasonable to assume that this omission does not invalidate our findings. We note the ongoing debate within the statistical community regarding appropriate contexts for fixed versus random effects models, and the potential drawbacks including increased model complexity.1Hamaker E.L. Muthén B. The fixed versus random effects debate and how it relates to centering in multilevel modeling [Internet].Psychol Methods. 2020; 25 ([cited 2023 May 12]. Available from:): 365-379https://pubmed.ncbi.nlm.nih.gov/31613118/Crossref PubMed Scopus (108) Google Scholar,2Harrison X.A. Donaldson L. Correa-Cano M.E. et al.A brief introduction to mixed effects modelling and multi-model inference in ecology [Internet].PeerJ. 2018; 6 ([cited 2023 May 12]. Available from: pmc/articles/PMC5970551/)Crossref Scopus (1003) Google Scholar This may partially explain why linear mixed effects models are not widely reported in the radiomics literature, but we agree such approaches could have strengthened the analysis and reduced the risk of Type 1 errors. The next criticism raised was the feature selection process. Though stepwise feature selection approaches incorporating univariable testing are widely adopted,3Wang X. Xie T. Luo J. et al.Radiomics predicts the prognosis of patients with locally advanced breast cancer by reflecting the heterogeneity of tumor cells and the tumor microenvironment [Internet].Breast Cancer Res. 2022; 24 ([cited 2023 May 12]. Available from:): 1-17https://breast-cancer-research.biomedcentral.com/articles/10.1186/s13058-022-01516-0Crossref PubMed Scopus (8) Google Scholar, 4Cozzi L. Dinapoli N. Fogliata A. et al.Radiomics based analysis to predict local control and survival in hepatocellular carcinoma patients treated with volumetric modulated arc therapy [Internet].BMC Cancer. 2017; 17 ([cited 2023 May 12]. Available from:): 1-10https://bmccancer.biomedcentral.com/articles/10.1186/s12885-017-3847-7Crossref PubMed Scopus (69) Google Scholar, 5Parr E. Du Q. Zhang C. et al.Radiomics-based outcome prediction for pancreatic cancer following stereotactic body radiotherapy [internet].Cancers. 2020; 12 ([cited 2023 May 12]. Available from:): 1051https://www.mdpi.com/2072-6694/12/4/1051/htmCrossref PubMed Scopus (27) Google Scholar Dr. Eleuteri rightly points out the potential limitations including a risk of bias. We have repeated the feature selection process with this step omitted, and can confirm the LN-RPV features were identical, meaning no deleterious effects have occurred. Regarding the choice of correction method for multiple testing, methods including Bonferroni, Benjamini-Hochberg or Holm correction can be justified according to individual preference. Again, we repeated the analysis using the Holm test, which returned fewer variables at the univariable testing stage but did not change the features selected by the LN-RPV LASSO. Overall, we agree that more robust feature selection approaches, such as bootstrap resampled LASSO models, could strengthen future analysis.6Compter I. Verduin M. Shi Z. et al.Deciphering the glioblastoma phenotype by computed tomography radiomics.Radiother Oncol. 2021; 160: 132-139Summary Full Text Full Text PDF PubMed Scopus (5) Google Scholar To clarify the comment regarding the nnUNet model being fed into the radiomics model as a predictor, we think this may be a misunderstanding. The nnUNet model was used to generate duplicate segmentation masks, to compare against the values generated manually. This allowed us to test the reproducibility of the model by fitting it both to manual and automated segmentation masks and calculating inter-reader reliability. We believe our reporting of the model performance, in terms of ROC-curves, AUCs, and Youden-index threshold-derived accuracy metrics allowed us to sufficiently estimate the performance and show clinical benefit when integrated with existing decision tools. We accept that reporting of additional metrics, including the Brier score, net benefit and decision–curve plots could provide additional useful information. Regarding our approach to missing data, Dr. Eleuteri suggests it is safer to use imputation when we cannot guarantee if data are missing at random or completely at random. We believe this approach is debatable, and opted to assume the PET scans were negative in order to model the lowest possible Herder score, accepting that a very avid-PET could potentially increase the Herder score above the 70% threshold.7Hughes R. Heron J. Sterne J. Tilling K. Accounting for missing data in statistical analyses: multiple imputation is not always the answer.Int J Epidemiol. 2019; 48: 1294-1304https://doi.org/10.1093/ije/dyz032Crossref PubMed Scopus (259) Google Scholar This was a decision taken partially on clinical grounds. In summary, we thank Dr Eleuteri for his letter and hope it will prove useful to the research community. We accept that elements of our statistical analysis could have been bolstered by additional complex approaches, but believe their omission does not necessarily invalidate the study findings. The authors would also like to point out that complexity comes with a cost, in particular it can hamper understanding for the targeted audiences. As Confucius said, “Life is really simple, but we insist on making it complicated.”? With kind regards, BH drafted this response letter. All authors named in the byline contributed to its content, reviewed and edited the letter and approved its final version. The named authors have responded on behalf of all the authors of the original article. The other authors have not directly contributed to this letter due to the nature of the expertise required for the response and for necessary promptness of turnaround. Dr Lee is funded by the Royal Marsden NIHR BRC and Royal Marsden Cancer Charity. RL's institution receives compensation for time spent in a secondment role for the lung health check program and as a national specialty lead for the National Institute of Health and Care Research. He has received research funding from CRUK, Innovate UK (co-funded by GE Healthcare, Roche and Optellum), SBRI (co-applicant in grants with QURE.AI), RM Partners Care Alliance and NIHR (co-applicant in grants with Optellum). He has received honoraria from CRUK. The other authors have no conflict of interest to report. Letter to the Editor: “A radiomics-based decision support tool improves lung cancer diagnosis in combination with the Herder score in large lung nodules”Dear Editor, Full-Text PDF Open AccessA radiomics-based decision support tool improves lung cancer diagnosis in combination with the Herder score in large lung nodulesThe model accurately segments and classifies large lung nodules, and may improve upon existing clinical models. Full-Text PDF Open Access