Evaluating clinical trial inclusion/exclusion criteria from claims using generative artificial intelligence

e13566 Background: Increasing enrollment in oncology clinical trials (CTs) requires identification of trial-eligible patients from routine health data. However, identification of CT-eligible patients is limited by inadequate access to EHR and its relevant clinical features to evaluate inclusion/exclusion (I/E) criteria in large populations. Artificial Intelligence (AI) applied to large-scale administrative claims may accelerate pre-screening of large patient cohorts against common I/E criteria related to stage and performance status (PS). Methods: This retrospective cohort study included 92,895 adult patients from ConcertAI’s real-world data: linked claims and deeply curated de-identified EHRs sourced from data-sharing agreements with 900+ community oncology clinics in the US. Patients had solid malignancies and an oncology encounter between January 6, 2014, and August 12, 2022. Patients with ≥1 claim record within one week of a PS entry and ≥25 claims within three years of a clinical encounter were included. We trained a Generative AI Transformer Model to predict good PS (ECOG 0-1 or Karnofsky 70-100) and stage IV disease at baseline, both of which are common inclusion criteria for therapeutic CTs in advanced malignancies. Model features included 7,745 timestamped claim codes and 8 time-independent demographic and socioeconomic variables. A state-of-the-art Generative AI Pre-trained Transformer Model was fine-tuned via compute-efficient transfer-learning to obtain predictive models for both PS and stage at baseline. The train/validation/test patient split was 95%/2%/3% with 3,039 patients in the PS test set and 2,718 in the stage IV test set. A comparator model was built to predict stage IV at baseline if a C77-9 ICD-100CM secondary malignancy code or cross-walked ICD-9-CM was in the claim record. Results: Among 92,895 patients, 48,331 (52%) were women, 63,620 (68%) were ≥65 years old, 54,495 (59%) were White, and 6,603 (7%) were Black. 38,229 (41%) were confirmed stage IV at baseline and 64,177 (69%) had good PS. In the test set predicting good PS, area under the receiver operating characteristic curve (AUROC) was 0.8 and under the precision-recall curve (AUPRC) was 0.89 for the Generative AI model. This corresponded to an Equal Error Rate (EER) operating point with good precision and recall (both 0.82). Performance metrics for the stage IV endpoint were also acceptable (AUROC 0.84, AUPRC 0.87, EER operating point 0.77). Comparator stage IV precision was 0.73 and recall was 0.47. Conclusions: Generative AI can leverage claims-only data to evaluate common pre-screening I/E criteria in large cohorts and outperforms models based on limited diagnosis codes. Rather than building one model per criterion per indication, Generative AI enables a systematic and reliable solution that can be readily scaled in future large-scale CT screening.