HeartBERT : A language model pre-trained on anopen source dataset for cardiac text mining

Abstract The advent of the Transformer has significantly altered the course of research in Natural Language Processing (NLP) within thedomain of deep learning, making Transformer-based studies the mainstream in subsequent NLP research. There has alsobeen considerable advancement in domain-specific NLP research, including the development of specialized language modelsfor medical. These medical-specific language models were trained on medical data and demonstrated high performance. Whilethese studies have treated the medical field as a single domain, in reality, medical is divided into multiple departments, eachrequiring a high level of expertise and treated as a unique domain. Recognizing this, our research focuses on constructinga model specialized for cardiology within the medical sector. Our study encompasses the creation of open-source datasets,training, and model evaluation in this nuanced domain.