A Comparison of the Content and Consistency of Methodological Quality and Transferability Checklists for Reviewing Model-Based Economic Evaluations

Objectives The aim of this study was to examine whether and how the content of six checklists (Caro, Consensus on Health Economic Criteria [CHEC]-Extended, European Network of Health Economic Databases [EURONHEED], National Institute for Health and Care Excellence [NICE], Philips, Welte) affect the consistency in findings on methodological quality and transferability, using 10 model-based economic evaluations of genetic-guided pharmacotherapy for venous thromboembolism. Methods Each checklist was categorised by domain (structure, data, consistency, etc.) and type of assessment (presence vs. appropriateness) and was applied to each study by two independent reviewers who agreed on ratings via consensus, and discussion with a third reviewer when necessary. Methodological quality scores and rankings were examined using Spearman correlation tests, with subgroup analyses for domains and types of assessment. We compared overall ratings of transferability qualitatively, including how content may affect what is considered ‘transferable’. Results The checklists had similar proportions of items judging presence and appropriateness, but varying proportions of items across domains. For methodological quality, ranking consistencies were the highest between CHEC-Extended-Philips, Philips-NICE and NICE-Caro, with similar consistencies for domains and type of assessment. For transferability, NICE and Caro identified the same study, which scored high on EURONHEED, as transferable to the UK, while Welte, which considered methodological quality, identified none as transferable. Conclusions We found that the choice of checklist can affect findings on study quality and decisions about whether study results are transferable, indicating that different checklists may shortlist different sets of studies in formulating policy recommendations, leading to different policy decisions. Our systematic approach for evaluating the content of methodological quality and transferability checklists of economic evaluations can be extended to other checklists.