Exploring ChatGPT's potential in the context of colon cancer patient education: A comparative analysis (Preprint)

BACKGROUND ChatGPT is a large language model capable of generating human-like conversation. It has demonstrated promise as a tool for medical education for both professionals and patients. Previous research in medical oncology and colon cancer showed a glimpse of its application on topics like colonoscopy, colorectal surgery, and guideline-based treatment. OBJECTIVE To evaluate ChatGPT's performance as a source of patient medical education for colon cancer METHODS A set of twenty non-expert questions were prepared and fed to ChatGPT three times. Later, generated responses were evaluated by two doctors for accuracy, simplicity (0-10), and consistency (0,1). Mean, median and standard deviation were calculated for both accuracy and simplicity scores along with the Intraclass Correlation Coefficient and confidence interval for inter-rater agreement assessment. For consistency, rate, cohen's kappa, standard error, and confidence interval were calculated. RESULTS Accuracy: Mean = 8.4, Median = 8.5, SD = 1.7. ICC: Avg. measures for absolute agreement = 0.7 (95% CI 0.25 to 0.88), for consistency = 0.74 (95% CI 0.34 to 0.9). Simplicity: Mean = 8.55, Median = 9, SD = 1.69. ICC: Avg. measures for absolute agreement = 0.65 (95% CI 0.12 to 0.86), for consistency = 0.72 (95% CI 0.28 to 0.89). Consistency: rate = 67.5%, Cohen's Kappa: 0.66 (SE = 0.18, 95% CI 0.31 to 1.0). CONCLUSIONS In this study, we assessed ChatGPT's capabilities of answering patients' questions about colon cancer. Findings showed significant and promising results of answers' accuracy, simplicity, and consistency in multiple trials. However, there is room for improvement. As ChatGPT continues to gain popularity among users, research studies on the impact of this technology on patient outcomes are needed urgently to guide clinical application.