Next Generation of Electronic Medical Record Search Engines to Support Chart Reviews: A Systematic User Study and Future Research Direction

Objective Little research has been done on the user-centered document ranking approach, especially in a crowdsourcing chart review environment. As the starting point of designing and implementing the next generation of Electronic Medical Record (EMR) search engines, a systematic user study is needed to better understand the users' needs, challenges, and future research directions of EMR search engines. Materials and Methods One primary observation during the user study is the need for a ranking method to better support the so-called "early stopping" reviewing strategy (i.e., reviewing only a subset of EMRs of one patient to make the final decision) during the clinical chart reviews. The authors proposed two novel user-centered ranking metrics: "critical documents" and "negative guarantee ratio," to better measure the power of a ranking method in supporting the “early stopping” requirements during clinical chart reviews. Results The evaluation results show that i) traditional information retrieval metrics, such as the precision-at-K, have limitations in guiding the design and development of EMR search engines to better support clinical chart reviews; ii) there is not a global optimal ranking method that fits the needs of different chart reviews and different users; iii) a learning-to-rank approach cannot guarantee a stable and optimal ranking for different chart reviews and different users; and iv) A user-centered ranking metric, such as the negative guarantee ratio (NGR) metric is able to measure the “early-stopping” performance of ranking methods. Conclusions User-centered ranking metrics can better measure the power of ranking methods in supporting clinical chart reviews. Future research should explore more user-centered ranking metrics and evaluate their impact on real-world EMR search engines.