Multicenter Validation of a Machine Learning Algorithm for Diagnosing Pediatric Patients with Multisystem Inflammatory Syndrome and Kawasaki Disease

Background Multisystem inflammatory syndrome in children (MIS-C) is a novel disease identified during the COVID-19 pandemic characterized by systemic inflammation following SARS-CoV-2 infection. Delays in diagnosing MIS-C may lead to more severe disease with cardiac dysfunction or death. Most pediatric patients recover fully with anti-inflammatory treatments, but early detection of MIS-C remains a challenge given its clinical similarities to Kawasaki disease (KD) and other acute childhood illnesses. Methods We developed KIDMATCH (KawasakI Disease vs Multisystem InflAmmaTory syndrome in CHildren), a deep learning algorithm for screening patients for MIS-C, KD, or other febrile illness, using age, the five classical clinical KD signs, and 17 laboratory measurements prospectively collected within 24 hours of admission to the emergency department from 1448 patients diagnosed with KD or other febrile illness between January 1, 2009 and December 31, 2019 at Rady Children’s Hospital. For MIS-C patients, the same data was collected from 131 patients between May 14, 2020 to June 18, 2021 at Rady Children’s Hospital, Connecticut Children’s Hospital, and Children’s Hospital Los Angeles. We trained a two-stage model consisting of feedforward neural networks to distinguish between MIS-C and non MIS-C patients and then KD and other febrile illness. After internally validating the algorithm using 10-fold cross validation, we incorporated a conformal prediction framework to tag patients with erroneous data or distribution shifts, enhancing the model generalizability and confidence by flagging unfamiliar cases as indeterminate instead of making spurious predictions. We externally validated KIDMATCH on 175 MIS-C patients from 16 hospitals across the United States. Findings KIDMATCH achieved a high median area under the curve in the 10-fold cross validation of 0.988 [IQR: 0.98-0.993] in the first stage and 0.96 [IQR: 0.956-0.972] in the second stage using thresholds set at 95% sensitivity to detect positive MIS-C and KD cases respectively during training. External validation of KIDMATCH on MIS-C patients correctly classified 76/83 (2 rejected) patients from the CHARMS consortium, 47/50 (1 rejected) patients from Boston Children’s Hospital, and 36/42 (2 rejected) patients from Children’s National Hospital. Interpretation KIDMATCH has the potential to aid frontline clinicians with distinguishing between MIS-C, KD, and similar febrile illnesses in a timely manner to allow prompt treatment and prevent severe complications. Funding Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Heart, Lung, and Blood Institute, Patient-Centered Outcomes Research Institute, National Library of Medicine ### Competing Interest Statement The authors have declared no competing interest. ### Funding Statement This work was supported by NIH grants (R61HD105590 to J.C.B, R01HL140898 to J.C.B. and A.H.T) and a grant from the Patient-Centered Outcomes Research Institute (CER-1602-3447 to J.C.B). J.Y.L. was supported by a predoctoral fellowship from the National Library of Medicine (T15LM011271). ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: Written consent or assent as appropriate was obtained from parents and subjects and the study was approved by the Institutional Review Boards (IRB) of the University of California San Diego (UCSD), Connecticut Children's Medical Center, Children's Hospital Los Angeles, Boston Children's Hospital, and Children's National Hospital. UCSD served as the central IRB of record for the CHARMS study participants. I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable. Yes De-identified data has been compiled from multiple sites across the United States. Requests for data will require approval from UCSD and partner institutions independently.